Lecture 17 · Part 5

Alignment, Multimodality

Percy sets aside the planned reinforcement-learning material to survey how multimodal models are actually built. The through-line is that transformers speak tokens, so every modality must be converted into them. CLIP and SigLIP learn image encoders from noisy web captions, the LLaVA and Qwen-VL series stitch those encoders onto pretrained language models with progressively better adapters and resolution handling, and Chameleon shows the elegance and the pain of discretizing everything.

▶ Lecture video Executable lecture (.py) ↗

The world is multimodal; transformers speak tokens

Percy had one lecture left and had planned more reinforcement learning, but decided the class “would be a little bit incomplete without saying something about multimodality” — it’s pervasive in every major model. So this is a one-lecture overview of a topic that could be a whole course.

Language models already map any text to any text — any language, code, poetry, even DNA. But the world also has images, audio, and video, and the North Star is the omni model: input any combination of modalities, output any combination. This lecture builds pieces of that, focused almost entirely on ingesting images.

The motivation for how is blunt: “transformers work really well,” and despite people’s best efforts to try other things, they remain the best we have at scale across all of these modalities. Transformers speak tokens — discrete or continuous — so everything must be converted into tokens.

Two questions frame everything: how do you input non-text data into the transformer, and how do you generate it out? This lecture is mostly question one; generation appears only at the end.

CLIP: semantics from noisy captions

Rewind to 2021. Language modeling had entered the foundation-model era by scraping noisy internet text; vision was still ResNets trained on carefully annotated datasets like ImageNet. The OpenAI researchers asked the obvious transfer question: can you leverage the vast supply of images paired with captions the same way? CLIP — Contrastive Language-Image Pre-Training — was the answer.

Take a batch of image–text pairs (around 32,768 of them). Encode every image into a vector $I_1 \ldots I_N$ and every text into $T_1 \ldots T_N$ . The objective: the dot product between $I_1$ and its aligned $T_1$ should beat $I_1$ against every other text, and symmetrically $T_1$ against every other image — $2N$ softmax classification problems over an $N \times N$ similarity matrix.

The data came from searching roughly 500K queries and mining about 20K pairs per query: 400 million image–text pairs, never released. OpenCLIP replicated the idea on the released LAION-5B dataset (5 billion captioned images — filtered, in a nice bit of bootstrapping, using CLIP itself). Preprocessing is heuristic: images arrive in arbitrary resolutions, neural nets “don’t like things to be dynamic,” so resize the shorter side to 336 pixels and center-crop a square. Fine for classification, where the object sits in the middle; a decision we’ll pay for later.

The headline result: on ImageNet, zero-shot CLIP — score each class label’s text against the image, pick the highest dot product — outperformed a ResNet trained on 1.2 million hand-annotated ImageNet images. Many hours of Mechanical Turk labor, matched by organic web data. And the captions are extremely noisy (a photo of a dog rarely bothers to say “a dog”), which makes it somewhat surprising and interesting that it works at all.

One ablation foreshadows the lecture’s ending: they tried predicting the caption from the image with a language model instead of ranking, and it was much less compute efficient. Modeling the exact token sequence of a caption isn’t important for getting a rough semantic representation of an image. The downsides: CLIP’s design decisions were made with classification in mind, so it’s not fine-grained; it needs huge batches (batch size 1 clearly doesn’t work); and the softmax couples the whole batch, so nothing decomposes the way language-model training does.

SigLIP: a simpler loss that parallelizes

Google’s SigLIP (Sigmoid Loss for Language Image Pre-training) is best thought of as an improved CLIP. Where CLIP does multi-class classification against all alternatives, SigLIP asks a binary question per pair: aligned or not? Diagonal entries of the similarity matrix are positives, off-diagonal entries are negatives.

Code walkthrough — CLIP's softmax vs SigLIP's sigmoid

CLIP, as shown in the lecture: normalize, take scaled dot products, then two cross-entropies over the $N \times N$ matrix — every example competes with the whole batch.

I_e = l2_normalize(image_encoder(images))      # n x d
T_e = l2_normalize(text_encoder(texts))        # n x d
logits = I_e @ T_e.T * exp(t)                  # n x n, temperature t
labels = arange(n)
loss = (cross_entropy(logits, labels, axis=0) +
        cross_entropy(logits, labels, axis=1)) / 2

SigLIP keeps the same matrix but scores each entry independently:

logits = I_e @ T_e.T * t + b
labels = 2 * eye(n) - 1                        # +1 diagonal, -1 elsewhere
loss = -sum(log_sigmoid(labels * logits)) / n

Because each entry is its own binary problem, the loss decomposes — which unlocks a DDP-style parallelization: each device computes losses on its local image–text pairs, then the text embeddings rotate around the devices until every off-diagonal block is covered.

The data is WebLI: on the order of a billion pairs scraped from the internet, augmented with OCR-extracted text from images, filtered to the top 10 percent, spanning 100 languages. Efficiency was the headline: CLIP took 10 days on 256 TPUv3s; SigLIP took 5 days on 32 TPUv4s — and per-chip, v4s are actually slower in FLOP/s, so this is a real algorithmic win. SigLIP also decouples batch size from the loss function: it beats CLIP below 16K batch size (where CLIP’s loss itself degrades), and pushing to 1M helps nothing — 32K was essentially the critical batch size.

LLaVA: stitching an encoder onto a language model

Now build a VLM. Both families in this lecture follow one template, and it’s mid-training/post-training in flavor: take an existing image encoder, take an existing LLM, stitch them together rather than training from scratch.

LLaVA (2023) made it concrete when GPT-4 was doing visual reasoning behind closed doors: CLIP ViT-L/14 as the encoder, Vicuna (LLaMA fine-tuned on shared ChatGPT conversations) as the decoder, and a single linear projection $W$ mapping image vectors into the text embedding space. The data is synthesized: take MS COCO’s human-annotated captions and bounding boxes, prompt GPT-4 to generate conversations, detailed descriptions, and complex reasoning about each image — 158K examples. Training has two stages: freeze everything and train only $W$ (alignment), then train $W$ plus the language model. The paper’s signature example: shown a man ironing on the back of a minivan and asked what’s unusual, the model explains it — even without the prompt hinting at unusualness.

LLaVA OneVision (2024) is the same recipe with every part upgraded — SigLIP encoder, Qwen-2 72B decoder, two-layer MLP projector — and more ambition: multiple images and video (a sampled sequence of frames, up to 32).

OneVision deliberately balances the modalities so video doesn’t drown everything: a single image gets high resolution (up to 9 crops plus the base), multiple images get base resolution each, video frames get even fewer tokens. The data philosophy is quality over quantity — heavily task-targeted, unabashedly distilling GPT-4 (“this is what you do if you don’t have an annotation budget”) — and training goes easier-to-harder in three stages. The payoff they highlight is transfer between modalities: trained on single-image chart data, it answers questions across a table image and a chart image; single-image OCR plus multi-image relational reasoning generalizes to GUI-agent screenshots; visual prompting (a circle drawn on one image) transfers to highlighting a player across video frames. And the LLaVA series open-sources the data, not just weights, so you can actually study it.

The Qwen-VL series: dynamic resolution and deeper fusion

Qwen-VL (2023) fits the pattern: an OpenCLIP ViT encoder (14 by 14 patches), an adapter of one cross-attention layer with 2D positional encodings mapping every image to a fixed 256 tokens, and special tokens <img>, <box>, <ref> so the model can output bounding boxes (find Spider-Man and the Hulk, get boxes back — it never outputs pixels). Three stages: large-scale low-quality data with the LM frozen and, unlike LLaVA, the vision encoder trained (1.4 billion examples); higher-quality task data with everything trained; instruction tuning with the visual encoder frozen.

Qwen2-VL’s main idea is dynamic resolution: instead of a fixed 256 tokens, an image costs what it costs. Each 224-by-224 region is encoded with a ViT (675M, initialized from DFN), every 2-by-2 group of patch vectors is compressed into one, giving 66 tokens per region; video is sampled at 2 frames per second up to a 16,384-token cap. A detailed photo might be 11,000 tokens; a tiny image of an equation, 8. It also introduces multimodal RoPE (MRoPE): each patch’s position is a (time, height, width) triple, RoPE is computed per axis and concatenated.

Qwen3-VL (the last stop, “a pretty good model”) upgrades everything again: Qwen-3 dense and MoE language models up to 235B-A22B, 256K context for long video, SigLIP-2 encoder. Three smaller changes matter. First, interleaved MRoPE: the blocked layout gave time all the low frequencies and height all the high ones, so they interleave the axes across frequency bands — [t w h t w h …] instead of [t t t w w w h h h]. Second, explicit video timestamps as actual tokens rather than positions, so the model can refer to “what happened after two seconds.” Third, a square-root-normalized per-token loss so long video examples don’t dominate short single-image ones. The adapter also deepens: DeepStack injects the vision encoder’s stack of embeddings directly into multiple layers of the language model’s residual stream — deep fusion instead of treating the encoder as a black box emitting a vector sequence. Pre-training runs four stages (adapter, then progressively 8K, 32K, 256K sequence lengths); post-training is SFT on long chain-of-thought data, distillation, and RL. On the benchmark table it trades wins with Gemini, GPT-5, and Opus 4.1.

Chameleon: what if everything were discrete tokens?

VLMs inject continuous image embeddings into a language model, so they can only ever generate text. Meta’s Chameleon (2024) takes the aesthetically appealing route — Percy admits “maybe this reflects that I’m a language person” — map everything into discrete tokens, and analysis and generation of images become ordinary autoregressive modeling. Prompt “I’m bored, can you show me some birds?” and the model emits interleaved text tokens and image tokens.

With images now “text,” training is just language model training — no adapter, no vision encoder at inference-time stitching. Stage 1 (80 percent of training) is large-scale and unsupervised: 2.9T text tokens, 1.5T text/image tokens, 400B interleaved. Stage 2 mixes in high-quality data. But it wasn’t stable.

Diffusion models then arrived as the better image generator, and the VQ-VAE flavor faded. Percy’s closing read on the frontier: models are touted as natively multimodal with zero disclosed detail, and his speculation is a continuous encoder for understanding (you don’t want to lose information) plus diffusion for generation (it micro-optimizes fine detail). There’s a real asymmetry between understanding and generating a modality — CLIP-style semantics suffice for one, fine-grained detail is required for the other — and no universal encoder exists. Five-year-old CLIP-family ideas are still the go-to. No homework on this one, but “I encourage you to play around with training some of these models.”

Back of the chalk tray — what to remember

Transformers speak tokens, so multimodality reduces to tokenization: convert every modality into discrete or continuous tokens that carry semantic units of information.
CLIP learns image semantics from 400M noisy web caption pairs via a contrastive softmax over the batch; zero-shot it beat a ResNet trained on 1.2M hand-labeled ImageNet images.
SigLIP replaces the softmax with per-pair sigmoids, decoupling the loss from batch size and enabling a rotate-the-embeddings parallelization — 5 days on 32 TPUv4 vs CLIP’s 10 days on 256 TPUv3.
The standard VLM is vision encoder + adapter + LLM, trained in stages from adapter alignment to full fine-tuning; successive generations upgrade parts (linear to MLP to cross-attention to DeepStack) without changing the template.
Resolution handling is the recurring battle: AnyRes and dynamic resolution let token cost scale with image content, and video makes multimodality a long-context problem.
Discretizing everything (Chameleon, VQ-VAE) is elegant but loses information and destabilizes training — the entropy of image tokens is nothing like text — so frontier practice is likely continuous encoders in, diffusion out.

Check yourself

Exercises in the lecture’s spirit — objectives, token budgets, and napkin arithmetic.

Problem 17.1 contrastive objectives

For a batch of $N$ image–text pairs, how many classification problems does CLIP’s objective set up, and over how many classes each? How many does SigLIP set up? Use the difference to explain why SigLIP admits the rotating-embeddings parallelization and why CLIP’s loss changes when the batch size changes.

Show solution

CLIP: $2N$ softmax problems (one per image over all $N$ texts, one per text over all $N$ images), each an $N$ -way classification. SigLIP: $N^2$ independent binary problems, one per cell of the similarity matrix ( $N$ positives on the diagonal, $N^2 - N$ negatives). Because SigLIP’s loss is a sum of independent per-cell terms, a device holding a block of images can accumulate its share of the loss as text embeddings rotate past — each block contributes additively. CLIP’s softmax normalizes over the entire batch, so no cell’s loss is computable without every other cell in its row/column; the batch size is baked into the loss itself, which is why shrinking it gives a different (and worse) objective rather than just a noisier estimate.

Problem 17.2 AnyRes token accounting

A vision encoder takes 336-by-336 inputs and emits 576 tokens per encoded view. Under AnyRes, how many tokens does a 1008-by-672 document image produce, counting the downsampled base view plus all crops? If the system caps images at 4,000 tokens, does it need to interpolate?

Show solution

Crops: $1008/336 = 3$ across and $672/336 = 2$ down, so $3 \times 2 = 6$ crops. Plus one downsampled view of the whole image: 7 encoded views total. Tokens: $7 \times 576 = 4032$ . That exceeds the 4,000 cap, so yes — the token grid gets reduced slightly by bilinear interpolation. This is exactly the OneVision trade: fine-grained detail (needed for OCR) costs tokens linearly in area, and the budget forces a choice between resolution and context.

Problem 17.3 video context budget

A Qwen2-VL-style model samples video at 2 frames per second and encodes each 224-by-224 region into 66 tokens after 2-by-2 compression. For 672-by-448 video frames, how many seconds of video fit in the 16,384-token visual budget? What two design levers from the lecture would extend it?

Show solution

Regions per frame: $(672/224) \times (448/224) = 3 \times 2 = 6$ , so $6 \times 66 = 396$ tokens per frame, and at 2 fps that is 792 tokens per second. Budget: $16384 / 792 \approx 20.7$ — about 20 seconds. Levers from the lecture: spend fewer tokens per frame (the OneVision move — video gets lower resolution than single images precisely to keep modalities comparable) and grow the context window itself (Qwen3-VL’s 256K-length training stages exist mostly for long video). Explicit timestamp tokens don’t extend the budget but make the sparse frames referable.

Problem 17.4 discretization loses bits

Chameleon’s VQ-VAE encodes a 512-by-512 RGB image (8 bits per channel) into 1,024 tokens from a codebook of 8,192. Compute the compression ratio in bits, and use it to explain the lecture’s claim that this pipeline can’t do OCR on small print, while a dynamic-resolution continuous encoder can.

Show solution

Raw image: $512 \times 512 \times 3 \times 8 \approx 6.3 \times 10^6$ bits. Tokens: $\log_2 8192 = 13$ bits each, so $1024 \times 13 \approx 1.33 \times 10^4$ bits. Ratio: about $470\times$ compression — each token summarizes a 16-by-16 pixel region with 13 bits. A character of small print occupies a few such regions, and 13 bits per region cannot preserve stroke-level shape, so “a J looks like an I”: the information is gone before the language model ever sees it. A continuous encoder with AnyRes-style dynamic resolution instead spends more tokens (each a high-dimensional continuous vector) on exactly the regions that carry fine detail, so the detail survives to the transformer. The fixed discrete budget is the price Chameleon pays for being able to generate images autoregressively.