Course Overview

If you have ever wondered what is actually going on inside ChatGPT, Claude or Gemini when you type a question, this course is our attempt to answer that honestly and from the ground up. Underneath all the noise sits one almost disappointing idea: a very good next-word guesser, trained at enormous scale. Everything else — turning text into numbers, letting every word weigh up every other word through the attention mechanism, teaching a model which word came first, and then the efficiency tricks that separate a textbook model from a frontier one — exists to make that one idea work well enough to astonish you. We are going to build all of it, piece by piece.

We start from first principles and code each idea up conceptually before layering on the next. By the end of this course you will be able to read a modern model's architecture description and recognise every block in it: the residual stream, multi-head attention and its memory-saving variants (multi-query, grouped-query and latent attention), rotary positional encodings, the mixture-of-experts feed-forward layer, multi-token prediction, and the low-precision arithmetic that makes training affordable.

The course is written for a wide audience. An undergraduate who has met gradient descent and matrix multiplication can follow every chapter; a postgraduate or working engineer will find the later modules on efficient attention, sparsity and quantization go all the way to what current systems actually do.

What you should already know

You will be comfortable if you have seen:

• Vectors and matrices — multiplication, dot products, transposes.
• The basics of neural networks — a neuron, a weight, a non-linear activation, and training by gradient descent and backpropagation.

If any of that is rusty, our Deep Learning course covers it from scratch, and we link to the exact lessons where they are needed — for example the backpropagation algorithm, output and loss functions and activation functions.

How the course is organised

The modules move in one straight line, each building on the last:

1. Orientation — what a language model is and how a modern one is put together.
2. The transformer: journey of a token — the end-to-end path from text to a predicted next word.
3. Attention from first principles — self-attention, scaling, and causal masking.
4. Multi-head attention — many attention heads working in parallel.
5. Positional information — why order must be injected, up to rotary encodings (RoPE).
6. Efficient attention at inference — the KV cache and the family of tricks that shrink it.
7. Mixture-of-experts — scaling the feed-forward layer without scaling the cost.
8. Beyond next-token training — multi-token prediction.
9. Numerical precision and model compression — low precision, pruning and distillation.
10. Post-training — alignment and adaptation, the stage that turns a predictor into an assistant.
11. Evaluating language models — perplexity, overlap metrics, and preference-based ranking.
12. Behaviour, failure and interpretability — hallucinations, jailbreaks, and looking inside the model.
13. Putting it together — the anatomy of a frontier model, and where to go next.

Every chapter ends with a few questions to check understanding. Many chapters also link to an experiment in the Transformer Lab, where you can run the idea yourself: tokenization, embeddings, attention, positional encodings, sampling, the KV cache, mixture-of-experts and quantization.