Transformer Lab

Tokenize with BPE, watch attention heads, rotate positions with RoPE, size a KV cache, route tokens to experts and quantize weights.

About the Transformer Lab

The Transformer Lab takes apart the architecture behind ChatGPT, Claude and Gemini, one piece at a time. Each experiment isolates a single idea, such as how text is split into tokens or why attention scores are divided by the square root of the head size, and lets you change the numbers and see the effect immediately.

It suits students who have met the word "attention" and want to know what it actually computes, and engineers preparing for interviews where KV cache sizes and mixture-of-experts routing come up. You need no maths beyond multiplying and adding, and nothing to install: everything runs in your browser.

A good order is the one below, from the input side of a model to the output side, then the efficiency tricks that make large models affordable. Each experiment links to the lesson in the Large Language Models course that explains it in full.

Experiments in this lab

Tokenization (BPE): Learn merges from text and watch words become tokens.
Embeddings: Similar meanings sit close together; relations become directions.
Self-attention: Queries, keys and values, with a causal mask.
Why divide by √d: Large heads blow up the scores; scaling keeps softmax soft.
Multi-head attention: Several heads, several patterns, same parameter count.
Sinusoidal and RoPE: Give each position a signature, or rotate by position.
Temperature, top-k, top-p: Turn logits into a choice, from greedy to creative.
KV cache: MHA to MLA: How much memory long context costs, and how to cut it.
Mixture of experts: Route each token to a few experts and keep them balanced.
Quantization: INT8, FP8 and INT4, and what an outlier does to them.

Large Language Models — The course this simulator goes with
RAG Lab — Chunk a handbook, search it by meaning, fuse BM25 with dense retrieval, rerank, measure recall and NDCG, and keep the index fresh.
Linear Algebra Lab — Drag vectors to see a span collapse, warp the grid with a matrix, hunt eigenvectors by hand, run Gram–Schmidt step by step, and watch a circle become an ellipse under the SVD.
Machine Learning Lab — Find principal components, run k-means and EM step by step, tame overfitting with ridge and lasso, and compare kNN, trees, naive Bayes, SVMs and boosting.
Deep Learning Lab — Build neural networks from scratch. Explore 3D error surfaces, optimizers, backpropagation, overfitting and convolutions.
Image Processing Lab — Sampling, filtering, Fourier transforms, segmentation, SIFT and HOG, applied to a live image you can change.
All simulators