Stanford · Hashimoto & Liang

Stanford CS336 · Language Modeling from Scratch

0 / 23 complete

Part 1 — Basics

01 Overview, Tokenization
02 PyTorch (einops), Resource Accounting
03 Architectures, Hyperparameters
04 Attention Alternatives, Mixture of Experts
★ Assignment 1 · Basics

Part 2 — Systems

05 GPUs, TPUs
06 Kernels, Triton, XLA
07 Parallelism I
08 Parallelism II
★ Assignment 2 · Systems

Part 3 — Scaling, Inference, Evaluation

09 Scaling Laws I
10 Inference
11 Scaling Laws II
12 Evaluation
★ Assignment 3 · Scaling

Part 4 — Data

13 Data: Sources, Datasets
14 Data: Filtering, Deduplication
★ Assignment 4 · Data

Part 5 — Post-Training & Alignment

15 Mid/Post-Training: SFT, RLHF
16 Post-Training: RLVR
17 Alignment, Multimodality
★ Assignment 5 · Alignment & Reasoning RL

Part 6 — Guest Lectures

18 Guest Lecture: Dan Fu

Find it on the board

Search the course

Every lesson, definition, and problem is indexed. Try “four subspaces” or “pivot”.