Large Language Models: From Transformers to Frontier Models

Build a modern large language model from the ground up: tokens and embeddings, the attention mechanism, positional encodings, and the efficiency and scaling ideas behind today's frontier systems — KV caching, grouped-query and latent attention, mixture-of-experts, multi-token prediction and low-precision training.

Undergrad to Professional · English · 42 chapters

• Course Overview
• What a Language Model Is — A large language model is a next-token predictor: given some text, it puts a probability on every possible next token. Everything else is built on that one idea.
• How a Modern LLM Is Built — A map of the whole course: the base transformer everyone shares, and the four pressures — memory, compute, scale and precision — that produced every modern upgrade to it.
• Tokenization: Text into Tokens — Before a model can do arithmetic on language, text is chopped into tokens from a fixed vocabulary. Subword tokenization is the quiet compromise that makes the vocabulary both small and complete.
• Embeddings: Tokens into Vectors — Each token ID is swapped for a learned vector. Those vectors form the 'residual stream' the whole model reads from and writes to, and their geometry carries meaning.
• Inside a Transformer Block — The tall stack is one block repeated. Each block does two things — mix information across positions (attention) and refine each position (a feed-forward network) — wrapped in residual connections and normalization that keep a deep stack trainable.
• From the Top of the Stack to the Next Token — The final vector becomes a score for every token, softmax turns scores into probabilities, and a sampling rule picks one. Temperature and top-p decide how adventurous that pick is.
• Why Attention — The meaning of a word depends on the other words around it. Attention is the mechanism that lets each token gather, on demand, exactly the context it needs — and unlike older approaches it does so in parallel and across any distance.
• Self-Attention: Queries, Keys and Values — Give each token three learned roles — a query, a key and a value. Match every query against every key to get weights, then blend the values by those weights. That is self-attention, in one formula.
• Scaling the Scores — One small divisor — √dₖ — sits inside the softmax. Without it, dot products in high dimensions grow large, softmax saturates, and gradients vanish. Here is why that factor is there.
• Causal Attention: Don't Peek Ahead — A model that predicts the next token must never see it. A triangular mask on the attention scores enforces this, and it is exactly what lets a model learn from every position of a sentence at once.
• Multi-Head Attention — One attention head can only focus one way at a time. Running several heads in parallel — each with its own queries, keys and values — lets a token attend to different things at once, then combine what they found.
• The Shapes of Multi-Head Attention — Following the dimensions — batch, sequence, heads, head-dimension — turns multi-head attention from a diagram into something you could implement. The key trick is that heads are just a reshaped view of one big projection.
• What the Heads Learn — and Whether We Need Them All — Different heads specialise into recognisable jobs, but many are redundant and the keys and values they store dominate memory at generation time. That tension sets up the efficient-attention module.
• The Problem of Order — Attention treats its input as a bag of tokens — scramble the words and the maths is unchanged. Since meaning depends on order, we must inject position by hand. This chapter pins down exactly what a position signal has to do.
• First Attempts: Integer and Binary — The naive fix — tag each token with its position number — fails because the numbers grow without bound. Fixing that with binary digits accidentally reveals the idea that powers every modern scheme: position as a set of oscillations at different speeds.
• Sinusoidal Positional Encodings — Replace the binary staircase with smooth sine and cosine waves whose frequencies slow down across the dimensions. The famous formula is just that idea — and the sine/cosine pairing hides a rotation that makes relative position easy.
• Rotary Positional Encodings (RoPE) — RoPE acts on the two dissatisfactions with sinusoidal encoding: it injects position inside attention by rotating the query and key vectors, with an angle set by position. Magnitudes are preserved, embeddings stay pure, and relative distance falls out for free.
• The KV Cache: Speed at a Price — Generating text one token at a time repeats enormous amounts of work. Caching each past token's keys and values removes the repetition and turns quadratic cost into linear — but the cache itself grows until, at long context, it dominates memory.
• Multi-Query Attention — The boldest way to shrink the cache: let all heads share a single set of keys and values. The cache drops by a factor equal to the number of heads — but the diversity that made multi-head attention powerful collapses with it.
• Grouped-Query Attention — The practical middle ground: split the heads into a handful of groups, and let each group share one set of keys and values. A dial between multi-head quality and multi-query thrift — and the scheme most open models actually ship.
• Multi-Head Latent Attention — Instead of choosing how many heads share keys and values, compress them. Store one small latent vector per token and let every head expand its own keys and values from it. The cache shrinks like multi-query's, yet head diversity survives — the best of both worlds.
• Latent Attention Meets RoPE — Rotary position depends on where a token sits, so it cannot be folded into fixed weights — and that breaks latent attention's absorption trick. The fix is to split each query and key into a content half that stays compressible and a small position half that carries RoPE.
• Why Mixture-of-Experts — The feed-forward layer holds most of a model's parameters and runs in full for every token. Mixture-of-experts breaks that: many expert networks, but a router sends each token to only a couple — so capacity grows while the work per token stays small.
• The Router: Choosing Experts — A small gating network scores every expert for each token, keeps the top few, turns their scores into weights, and blends only those experts' outputs. That tiny mechanism is what makes sparse activation work.
• Keeping the Experts Balanced — Left alone, routing collapses onto a few experts. The classic cures add a balancing loss and cap each expert's intake; a newer cure nudges the router with a per-expert bias and no extra loss at all — balancing the load without disturbing the real objective.
• Sharper Experts: Fine-Grained and Shared — Two refinements make experts specialise better: slice them smaller and more numerous so each can own a narrow skill, and set a few experts to always-on so they absorb the common knowledge every token needs — freeing the rest to specialise.
• Beyond Next-Token Prediction — Training a model to predict the next few tokens at once — not just the immediate one — gives a denser learning signal, better data efficiency, a nudge toward planning, and a path to faster generation. The objective changes; the model does not have to.
• How Multi-Token Prediction Is Built — Predicting several tokens means several prediction modules on top of the shared trunk. Making each module depend on the previous one — a causal chain rather than independent heads — is the design that makes the modern version work better.
• Why Low Precision — Every number in a model takes up bits, and fewer bits means less memory and faster arithmetic. Quantization trades a little numerical precision for large savings — the art is in which formats to use and how the trade is managed.
• Making Low Precision Work — FP8 is too fragile to use naively. Four engineering ideas make it safe: run only the heavy matrix multiplies in low precision while keeping sensitive parts high, scale in small blocks so one outlier can't spoil a tensor, accumulate sums in high precision, and compute scales on the fly.
• Pruning and Distillation — Quantization shrinks a model by storing each number in fewer bits. The other two levers remove numbers altogether — pruning cuts parts of a trained network, distillation trains a smaller model to imitate a larger one — and each trades accuracy, effort and hardware support differently.
• Model Alignment — Supervised fine-tuning teaches a model to imitate good answers, but imitation is not the same as being helpful. Alignment adds a second signal — human preferences — through RLHF or its simpler descendant, DPO.
• Fine-Tuning and LoRA — Updating every weight to teach a model a new task is slow, memory-hungry and produces a full model copy each time. Low-rank adaptation freezes the original weights and learns a small correction instead — at a fraction of the cost, and often with no loss of quality.
• Synthetic Data Generation — Post-training runs on data that often does not exist: instructions nobody wrote, rare cases nobody recorded, records nobody may share. Generating that data — with a model, a simulator or a statistical recipe — is now a standard step, and it has failure modes of its own.
• Perplexity — The oldest measure of a language model asks one question: how surprised was it by text it had never seen? Perplexity turns that surprise into a single number — and it is the same number the model was trained to minimise.
• BLEU and ROUGE — Perplexity scores the model's probabilities; these two score what it actually wrote. BLEU asks how much of the output was right, ROUGE asks how much of the reference was covered — and both are blind to meaning.
• Elo Ratings for Language Models — When there is no reference answer to match against, the only reliable signal left is which of two outputs a person prefers. A rating system borrowed from chess turns thousands of those local judgements into one global ranking.
• Hallucinations and Jailbreaks — A model that invents a citation and a model that is talked past its own safety training are failing in related ways: both are cases of competing internal pressures, where the drive to produce fluent, obliging text wins over the drive to be accurate or to decline.
• Model Interpretability — A trained model is a few hundred billion numbers that work. Interpretability is the effort to find out how — by probing activations, isolating features and intervening on them — and its most unsettling finding is that a model's stated reasoning need not match the computation that produced its answer.
• Anatomy of a Frontier Model — A walk through a modern large language model with every part named — from a token entering the tokenizer to the trained, quantized, sparse, long-context system that predicts the next word. Everything in the course, assembled into one picture.
• Where to Go Next — The architecture is behind you. This closing chapter is about what comes after it — the training stages that turn a predictor into an assistant, the systems built around it, and which of our other courses to open next.