Assignment 4 · Data

What you build

This assignment is the course’s data chapter made concrete: everything before the model. The work splits into three layers.

First, a toolkit of filtering primitives, each an independently tested function: extracting text from raw HTML bytes with Resiliparse (handling non-UTF-8 encodings), language identification with fastText’s lid.176.bin model, masking PII (emails, US-style phone numbers, IPv4 addresses) with sentinel strings, NSFW and toxic-speech detection using the Dolma project’s pretrained fastText classifiers, a subset of the Gopher quality heuristics (document length 50–100,000 words, mean word length 3–10 characters, at most 30% of lines ending in an ellipsis, at least 80% of words containing an alphabetic character), and a quality classifier you train yourself — following the WebText recipe, with Wikipedia-linked URLs as positives and random Common Crawl as negatives.

Second, deduplication: exact line deduplication with hashed line counts (kills repeated boilerplate like headers and menus), then fuzzy document-level deduplication with MinHash signatures and locality-sensitive hashing over n-gram Jaccard similarity.

Third, the open-ended part: compose your primitives into a pipeline over a provided set of English-filtered Common Crawl WET files, tokenize the output with the GPT-2 tokenizer, and train a GPT-2-small-shaped model (~430M parameters) with the provided training script. The goal is minimum validation perplexity on the C4 100 domains subset of the Paloma benchmark — and you are explicitly forbidden from touching the model or training procedure. Only the data moves.

Which lectures feed it

Lectures 13–14 cover data: what goes into pretraining corpora, how the well-known datasets (C4, Gopher/MassiveText, The Pile, Dolma, RefinedWeb) were filtered, and why data curation is where much of the real leverage in language modeling lives. Rewatch them when deciding which filters to apply and how aggressively — the assignment gives you the primitives, but the lectures give you the judgment about precision/recall trade-offs in filtering. The Gopher and RefinedWeb papers referenced in the handout are the primary sources behind the quality rules and the dedup-normalization choices.

The training half leans on everything from Assignment 1: the cs336_basics package in the repo is the staff’s optimized version of the model you built there, with a multi-GPU training script included.

Order of attack

The handout’s own sequence is the right one, and it starts with your eyes, not code:

1. Look at the data (look_at_cc). Browse a raw WARC file and its WET counterpart with zcat ... | less, annotate 25 records. This calibrates every threshold decision you make later. (Warning from the staff: the files are unfiltered internet — skip anything you’d rather not read.)
2. Primitives, in handout order: text extraction, language ID, PII masking, harmful content, Gopher filters, quality classifier. Each has an adapter in tests/adapters.py and a uv run pytest -k test_... target, and most also ask you to run the filter on real WARC-extracted text and report error rates and suitable confidence thresholds — 20-example manual inspections are a recurring deliverable.
3. Deduplication: exact line dedup first (a two-pass counting scheme), then MinHash + LSH.
4. The pipeline (filter_data, inspect_filtered_data, tokenize_data, train_model): filter the provided WET files, report per-filter keep rates, inspect kept and discarded examples, iterate, then tokenize and train.

What’s hard: phone-number masking (formats are wildly diverse — the handout only asks for robustness on common US formats), the quality classifier (at 15 points the biggest single item, and deliberately under-specified — you choose the training data and threshold), and the sheer throughput problem of the final pipeline. The handout pushes concurrent.futures/multiprocessing hard and includes a parallelization skeleton; you’re asked to report your pipeline’s runtime on the ~2,500 provided WET files and extrapolate to the full Common Crawl.

Compute, testing, and submission

The final training run is fixed: 8 B200 GPUs via Modal, data parallelism, batch size 128 per device, context length 512, 16,384 steps — about 2^33 ≈ 8.6B training tokens, finishing in roughly two hours with the staff setup. Tokenized data must be a np.uint16 array written with tofile so the provided scripts/train.py can read it.

Two rules worth internalizing early: you may use the Paloma C4 100 validation data to design filters or train classifiers, but no validation text may ever enter your training set; and the training script and config are untouchable. Grading is 71 points across 13 problems, mixing tested code, written analyses of your filters’ real-world behavior, and the final validation loss with its learning curve. Submission is writeup.pdf plus code.zip on Gradescope.