Data: Sources, Datasets
Percy opens the two-lecture data unit by arguing that data is the most important thing to get right in a language model — and the one thing companies won't talk about. The lecture traces where training data actually comes from (crawlers, robots.txt, copyright, fair use), tours the workhorse sources, and walks the genealogy of pre-training datasets from BERT's scraped books to DCLM's 240-trillion-token pool, where the real differentiator is how you filter.
Data is the most important thing — and nobody will tell you theirs
You know how to train a model given data; the last lecture covered how to tell a good model from a bad one. This week’s question is what to train on, and Percy’s opening claim is that data is the most important thing to get right in language models. His evidence is what companies disclose: the Llama 3 paper is fully transparent about architecture — it’s an open-weight model, it has to be — and even describes training procedures, but on data it says essentially nothing beyond “a variety of data sources.” Two good reasons for the secrecy: data is the competitive secret sauce, and data is where the copyright liability lives.
Data enters at every stage of the pipeline. Pre-training consumes raw documents from the web; mid-training shifts to higher-quality data that enhances specific capabilities (and things like long context); post-training uses chat transcripts or RL environments and becomes task-specific. The lines blur in practice, but the trend across stages is constant: from large amounts of low-quality data to small amounts of high-quality data. “Base model” usually means after pre-training plus mid-training, instruct/chat models come after post-training — though increasingly the largest models ship with no base model at all, just the final Qwen3.5-397B-A17B. Open models like AI2’s OLMo let you see every stage.
”Trained on the entire internet” doesn’t type-check
The hallway claim that models are “trained on the internet” isn’t even well-formed: the web is a set of live servers you send requests to, and you can’t train on live servers unless you’re an RL agent. What actually happens is that a crawler discovers pages from a seed set and downloads them. And a crawler cannot get everything:
- Dynamic content. Many sites are apps now — the URL underspecifies the content, and you’d need to click buttons and submit forms (Discord, wandb).
- Authentication. Facebook, X, LinkedIn, the New York Times — huge amounts of content sit behind walled gardens. If you’re Meta you already have Facebook’s data; anyone else can’t touch it.
- Technical restrictions.
robots.txt(voluntary good-citizenship, not law) tells crawlers what’s off-limits — nytimes.com disallows OAI-SearchBot, PerplexityBot, ClaudeBot, and friends. Cloudflare-style bot detection, CAPTCHAs, IP and country blocks, rate limits. - Legal restrictions. Terms of service may prohibit bots outright, and you may have no license to copy the pages for training.
These restrictions are tightening. The “Consent in Crisis” paper (Longpre et al.) examined robots.txt and terms of service for URLs in common datasets: full robots.txt restrictions were roughly flat until 2023, then jumped to nearly 50% of sites by mid-2023, and most pages now carry terms that forbid AI use. The internet you can legally crawl is much smaller than it was in 2020. Crawlers also misbehave — Percy shows a site complaining that Anthropic’s crawler hit it a million times in 24 hours — and even before copyright, server load costs the host money and degrades service.
Finally there are shadow libraries — LibGen, Z-Library, Anna’s Archive, Sci-Hub — which disregard copyright and bypass paywalls (LibGen had roughly 4 million books in 2019; Sci-Hub roughly 88 million papers in 2022). Takedown orders get circumvented by moving servers between countries. Advocates say this frees what should be free; legally, it’s piracy.
Everything is copyrighted — so what can you train on?
Copyright sits inside intellectual property law, whose goal is to incentivize the creation of intellectual goods — the spirit is not to say no to everything. Copyright goes back to England’s Statute of Anne in 1709; in the US the operative law is the Copyright Act of 1976, protecting “original works of authorship fixed in any tangible medium of expression.” Collections aren’t copyrightable unless the arrangement is creative (a telephone directory isn’t), and copyright protects expression, not ideas — you can’t copyright the quicksort algorithm, only an implementation of it. The 1976 Act moved the bar from “published” to “fixed”: no registration required, so putting something on your website copyrights it. Registration ($65) is only needed to sue. Copyright lasts 75 years, then the work joins the public domain.
The upshot: basically everything on the internet is copyrighted. Using a copyrighted work takes one of two doors. Door one is a license — “a promise not to sue,” granted by licensor to licensee. Creative Commons (created in 2001 by Lessig and Eldred) makes copyrighted work act like public domain — Wikipedia, OpenCourseWare, Khan Academy — and if the content isn’t CC, you can pay: Google licensed Reddit, OpenAI licensed Shutterstock and Stack Exchange. Door two is fair use.
Watching a movie and writing a summary is fair use; reimplementing an algorithm (the idea, not the code) is fair use; Google Books showing snippets was ruled fair use after the eleven-year Authors Guild v. Google fight — a precedent people reach for when thinking about LM training. Note that copyright is not about verbatim memorization, which may surprise an ML audience raised on N-gram-overlap papers: plots and characters (Harry Potter) are copyrightable, while parody is likely fair use. “A lot of copyright is about semantics… and also about the economics.”
For language models: the mere act of copying data is potentially a violation before you train anything; training itself has a transformative flavor — the model uses data as a means to learn about the world, not to rehost the expression; but LMs clearly can affect markets for original work, which is factor four working against you. And terms of service are yet another layer: YouTube hosts CC-licensed videos whose download-by-bot the ToS still forbids.
The lawsuits are where this is getting resolved. The New York Times sued OpenAI in 2023 after prompting ChatGPT into near-verbatim articles (pending). Authors v. Anthropic produced the landmark 2025 summary judgment: training on the plaintiffs’ books was fair use — but pirating millions of books was not, even though Anthropic had also bought and scanned the same books (itself fair use, but no absolution for the earlier sin). Anthropic paid 3,000 a book. Kadrey v. Meta went the same way — training ruled fair use, the torrenting allegation still pending. Summary: training has so far been deemed fair use in narrow, specific rulings; piracy is clearly illegal; the area is active and evolving.
The workhorse sources
Common Crawl, a non-profit running since 2007, is the crawl most of the field builds on: a roughly monthly crawl adds 3–5 billion pages (300 billion so far; the April 2026 crawl alone is 2.19 billion pages, 372.2 TB). Crawling is graph traversal — pop a URL from the queue, download, push its hyperlinks — run in parallel under selection, politeness, and re-visit policies. Dumps come in two formats: WARC (raw HTTP responses) and WET (text, extracted lossily). The extraction matters: the DCLM ablation shows trafilatura and resiliparse beat the official WET files on downstream model quality.
Wikipedia (2001; 67 million articles across 361 language editions) contains no original thought — everything cited, articles gated by notability — and a tiny core of Wikipedians does most of the work (Steven Pruitt: 5 million edits). Crucially, Wikipedia publishes periodic dumps every few weeks; you download the tarball, you don’t crawl. Fun adversarial aside from Carlini: because dumps happen on a known cadence, you can inject malicious edits right before the snapshot — the vandalism gets reverted, but the dump keeps it, enabling poisoning attacks that attach negative sentiment to trigger phrases. Even “high-quality” sources can carry bad content.
GitHub (2008; 420M+ repositories, 28M public) matters for coding ability and — folklore — general reasoning. Repository data comes via the git protocol, metadata (issues, PRs, comments) via GitHub Archive’s hourly event-stream snapshots; Software Heritage aggregates repositories across GitHub, GitLab, Bitbucket, and more. Training on public permissively-licensed repos (MIT, Apache) is allowed. arXiv (1991; about 3 million submissions) offers metadata under CC0, author-chosen licenses per paper (most are Creative Commons), optional LaTeX source, and bulk download from S3 — again, no crawling needed.
A genealogy of pre-training datasets
With sources in hand, the story of 2018–2024 is a string of answers to one question: Common Crawl is mostly junk for natural language, so how do you cut a large and high-quality corpus out of it?
| Dataset | Year | Filtering idea | Size |
|---|---|---|---|
| BERT data | 2018 | curated: Wikipedia + BooksCorpus | — |
| WebText (GPT-2) | 2019 | Reddit outlinks, 3+ karma | 40 GB |
| CCNet | 2019 | looks-like-Wikipedia LM score | — |
| C4 (T5) | 2019 | manual rules | 806 GB / 156B tokens |
| GPT-3 data | 2020 | quality classifier | 570 GB / 400B tokens |
| The Pile | 2021 | 22 curated domains | 825 GB / ~275B tokens |
| MassiveText (Gopher) | 2021 | manual rules | 10.5 TB |
| LLaMA / RedPajama v1 | 2023 | Wikipedia-reference classifier + rules | 1.2T tokens |
| RefinedWeb / FineWeb | 2023–24 | rules only, web is all you need | 5T / 15T tokens |
| Dolma | 2024 | rules, avoid model-based filtering | 3T tokens |
| DCLM-baseline | 2024 | fastText quality classifier | 3.8T of a 240T pool |
| Nemotron-CC | 2024 | classifier ensemble + synthetic rephrasing | 6.3T tokens |
A few of these deserve the story behind the row. BERT trained on Wikipedia plus BooksCorpus — 7K self-published books scraped from Smashwords in the “innocent days”; it violated Smashwords’ terms of service and has since been taken down. GPT-2’s WebText used a clever quality surrogate: pages linked from Reddit posts with at least 3 karma. CCNet (Facebook) automated quality another way — keep documents that look like Wikipedia under a KenLM 5-gram model — aimed especially at low-resource languages. C4 went pure rules: keep lines ending in punctuation with five-plus words, drop pages under three sentences, drop pages with bad words, drop pages containing a curly brace (no code — 2019 wasn’t thinking about code models), keep confident English. The C4 paper also rebuilt a WebText-like corpus from 12 dumps and got only 17 GB against WebText’s 40 GB — evidence Common Crawl is incomplete.
The Pile was EleutherAI’s grassroots answer to GPT-3 — volunteers on Discord curating 22 domains: Pile-CC (WARC via jusText), PubMed Central, arXiv from LaTeX, the Enron emails (500K emails released in the 2002 investigation — one of the only email corpora that exists), Project Gutenberg (copyright-cleared books; PG-19), Stack Exchange (Q&A that looks conspicuously like supervised data — “not everything is super magically emergent”), and Books3: 196K books from the shadow library Bibliotik. LLaMA, one of the last non-open models to document its data, trained on Books3 and announced it in the paper — which is exactly what got Meta sued, and a big part of why nobody talks about data anymore. RefinedWeb and FineWeb then argued web data is all you need, deliberately avoiding ML-based filtering to avoid its biases; Dolma took the same rules-not-models stance at 3T tokens.
Nemotron-CC (NVIDIA) pushed back: DCLM and FineWeb-Edu filter too aggressively, removing ~90% of the data, and you need more tokens. So: jusText extraction (returns more tokens than trafilatura), an ensemble of classifiers (prompt Nemotron-340B-instruct to score documents for educational value, distill into a fast model; plus the DCLM classifier), and a real lean into synthetic data — rephrase low-quality documents with an LM, and have an LM generate tasks (QA pairs, summaries, key-information extraction) from high-quality ones. Result: 6.3T tokens, 1.1T in the high-quality subset. For reference Llama 3 trained on 15T and Qwen3 on 36T — though quoted token counts often include repeats, so two epochs doubles the number.
Code at scale, and the fully-licensed experiment
The Stack set out in 2022 to build the good code dataset: repository names from GitHub Archive (2015–2022), 137M repositories cloned — 51B files, of which 5B unique — kept only permissive licenses via go-license-detector, near-deduped with MinHash, for 3.1 TB of code. Stack v2 (2024) added the metadata: issues, comments, and PRs from GitHub Archive, repositories from Software Heritage, documentation crawled from PyPI, npm, and devdocs.io — then stripped binaries, malware, and bot traffic, redacted PII, and subsampled the PRs. Two nice ideas: low-resource languages like Nim get paired with their LLVM intermediate representation, so the model can learn the mapping through a shared low-level language it has lots of data for; and pull requests get linearized into XML-like token sequences — the PR, its diffs, comment events, review states — with a design decision about how much surrounding context to include per diff. The model learns not just code but the software development process around code.
Common Pile asks the risk-averse question: if fair use is unsettled and your policy is “if I don’t know it’s OK, it’s a no,” how far can you get on permissively licensed data alone? Scouring the internet — Stack v2, government proceedings (surprisingly permissive), wikis, some news sites, academic papers, forums, public-domain works, educational resources — yielded 8 TB. It’s harder than it sounds: license laundering means anyone can slap CC-BY on copyrighted work; collection licenses (Dolma is ODC-By) don’t extend to the individual works inside; and they forewent synthetic data entirely, since generator models were themselves trained on unlicensed data — using their output “is a little bit of data laundering if you are really honest.” The resulting models do decently against 2023-era baselines like the first Llama and MPT, but can’t yet compete with Qwen-class models — though Percy doesn’t think this is the final word.
The closing summary: data does not fall from the sky. Someone runs the live service, someone produces the raw data (a dump or a crawler), and someone decides how to process it — and filtering, transformation, and deduplication are where language models actually differentiate, since the architectures are roughly the same transformer. Unlike the first-principles parts of this course, data processing today runs on vibes — a classifier here, a threshold there — which means there are many opportunities to improve, starting with assignment 4.
Check yourself
Conceptual exercises in the lecture’s spirit — reason through the layers rather than compute.
For each scenario, name every layer that blocks (or permits) the use: (a) a bot bulk-downloads Creative Commons–licensed videos from YouTube; (b) a crawler downloads nytimes.com articles while ignoring robots.txt; (c) a lab downloads Wikipedia’s periodic dump and trains on it; (d) a lab trains on Books3.
Show solution
(a) The CC license clears copyright, but YouTube’s terms of service prohibit downloading by bot — the ToS layer blocks even properly licensed content. (b) robots.txt is voluntary, so ignoring it isn’t illegal by itself, but it violates good-citizen norms; the NYT’s terms of service likely prohibit it, and the articles are copyrighted with no license — so ToS and copyright both bite (this is essentially the NYT v. OpenAI fact pattern, plus rudeness). (c) Clean on every layer: Wikipedia is Creative Commons (license clears copyright) and the dump is the sanctioned access path (no crawling, no ToS issue). (d) Books3 came from the shadow library Bibliotik — the copies are pirated, and Authors v. Anthropic established that acquiring pirated copies is illegal regardless of whether the training itself would be fair use. The dataset has been taken down.
A startup scrapes thousands of copyrighted cookbooks and trains a model that, on request, generates complete recipes in the distinctive voice of specific living cookbook authors, sold via subscription. Walk the four fair-use factors and say which cut for and against the startup.
Show solution
Factor 1 (purpose and character): commercial, which disfavors fair use; training is arguably transformative, but generating in an author’s voice looks more reproductive than transformative — mixed, leaning against. Factor 2 (nature of the work): recipes have factual ingredient lists but cookbooks contain creative expression (voice, narrative); creative works get more protection — against. Factor 3 (amount used): whole books were copied for training — against (remember, copying is potentially a violation before any training happens). Factor 4 (market effect): the product directly substitutes for buying the authors’ cookbooks and undercuts their market — strongly against, and this is the factor tied to copyright’s economic purpose. The existing rulings (Anthropic, Meta) found training fair use in narrow instances; a product mimicking specific living authors and displacing their sales is exactly where those narrow rulings stop helping.
C4 is 806 GB of text and 156 billion tokens. Using that ratio, roughly how many tokens would you expect from Common Pile’s 8 TB of permissively licensed data? Why might the true number differ, and why should you be careful comparing it to a headline like “Qwen3 trained on 36T tokens”?
Show solution
C4 gives bytes per token. At that rate, trillion tokens. The true number shifts with content mix — Common Pile includes lots of code, whose bytes-per-token differs from prose — and with the tokenizer. And headline training-token counts often include repeats: two epochs over a corpus doubles the reported tokens without adding a single unique token, so 36T trained tokens is not a claim about 36T unique tokens.
You want a DCLM-style fastText filter for a math-focused pre-training corpus. Choose positive and negative example sets in the spirit of DCLM’s choices, and name one bias risk of your classifier and one reason rules-only pipelines (RefinedWeb, Dolma) rejected this approach entirely.
Show solution
Following DCLM’s template — positives should exemplify the target distribution, negatives should be the raw web: positives could be Math Stack Exchange answers sorted by votes (Q&A with quality metadata, like DCLM’s ELI5) plus strong model-generated math explanations (like OpenHermes); negatives should be random documents from a loosely filtered web corpus such as RefinedWeb. Bias risk: the classifier keeps only documents resembling the positives — e.g. formatted Q&A and textbook prose — and may discard valid but differently styled math (forum arguments, code-heavy notebooks, non-English proofs). That narrowing is precisely why RefinedWeb and Dolma avoided ML-based filtering: a learned filter injects the biases of whatever you declared “good,” while manual rules are cruder but more transparent about what they exclude.