A Brief History of Deep Learning

From the first look at a neuron in 1871 to today's large language models: the ideas, the hype, the winters and the breakthroughs, with simulators to try the key ideas yourself.

Undergrad · English · 10 chapters

• From Biology to the Neuron — How scientists argued for decades about what the brain is made of, and why the answer still shapes neural networks.
• The First Artificial Neurons — The McCulloch-Pitts neuron of 1943 and Rosenblatt's perceptron of 1958: a yes/no decision machine, and a machine that learns its own weights.
• Hype, XOR and the First Winter — Why a simple logic function embarrassed the perceptron in 1969, what Minsky and Papert really said, and how funding dried up.
• Gradient Descent and Backpropagation — An idea from 1847 astronomy, rediscovered several times, finally made deep networks trainable in the 1980s.
• Universal Approximation: Power on Paper — The 1989 result that a network with one hidden layer can approximate almost any function, and what that promise does and does not say.
• Why Deep Networks Stayed Hard — Backpropagation and universal approximation existed by 1989, yet deep networks barely worked for almost two decades. The reasons, and the bright spots.
• Data, GPUs and AlexNet — 2006 to 2012: better training tricks, graphics cards and a huge image dataset come together, and one result changes the field.
• Architectures, Attention and the Transformer — 2013 to 2017: deeper networks, networks that generate, networks that pay attention, and the design that took over.
• Scale and Large Language Models — 2018 to today: pre-trained Transformers, scaling laws, and the path from research models to everyday assistants.
• Lessons from the History — The whole timeline in one picture, the patterns that keep repeating, and where to go next.