Deep Learning: Fundamentals to Advanced Concepts

Learn the complete deep learning with this course starting from basics to current day attention networks.

Undergrad · English · 52 chapters

• Course Overview
• From Biology to the Neuron — How scientists worked out what the brain is made of, and the four parts of a biological neuron that artificial neurons borrow.
• From the First Model to Deep Learning — The McCulloch-Pitts neuron, the perceptron, the AI winter and the revival, and the map of this course.
• The McCulloch-Pitts Neuron — The first computational neuron: add up binary inputs, compare with a threshold, and build logic gates.
• What a Neuron Draws: Lines, Planes and Linear Separability — A threshold neuron splits its input space in two. Seeing that split explains what it can compute and where it will fail.
• From MP Neuron to Perceptron — Weights, real-valued inputs and a learnable bias: what Rosenblatt's perceptron added, and why Boolean examples still matter.
• Finding Weights by Hand — Turn a truth table into inequalities, solve them for OR, and see why this does not scale.
• Errors and Error Surfaces — Count mistakes as a function of the weights, and meet the error surface that will return throughout the course.
• The Perceptron Learning Algorithm — A short loop that finds separating weights from examples: add the point when a positive is missed, subtract it when a negative is wrongly accepted.
• Why the Update Works: Dot Products and Angles — The geometry behind add and subtract, then a full worked run of the algorithm on AND.
• Does It Always Converge? (Optional) — A short proof that on separable data the perceptron algorithm makes only a bounded number of corrections.
• Where One Perceptron Fails — XOR cannot be computed by a single perceptron, and real-world data often has the same problem.
• A Network of Perceptrons — A hidden layer of perceptrons can compute any Boolean function, including XOR. The price is size, which grows exponentially.
• From Step to Sigmoid — Why the perceptron's all-or-nothing decision is too harsh for real problems, and the smooth sigmoid neuron that replaces it.
• The Supervised Learning Setup — Data, a model with parameters, a loss function and a learning algorithm: the four pieces behind almost every machine learning system.
• Guesswork and the Error Surface — Fit a sigmoid to two points by trial and error, measure each guess with the loss, and see that you were walking on an error surface without a map.
• Taylor Series: Approximating a Curve Locally — The calculus tool that tells us what a small move will do to the loss: replace a curve by a line or polynomial near a point.
• Why Move Against the Gradient — Use the Taylor series to find the best direction for a small step, and arrive at the gradient descent update rule.
• Gradient Descent for a Sigmoid Neuron — Work out the two partial derivatives of the squared-error loss, turn them into code, and watch the loss fall on the toy problem.
• Sigmoid Towers in One Dimension — Steep sigmoids act like steps, two steps make a tower, and enough towers trace any curve: the idea behind universal approximation.
• Sigmoid Towers in Two Dimensions — Extend the tower trick to two inputs with four sigmoids, a sum and a final sigmoid, and see how the cost grows with the number of inputs.
• Output Functions and Loss Functions — Matching the output function and the loss to the problem: linear output with squared error for regression, softmax with cross-entropy for classification.
• From One Neuron to a Network — Gradient descent stays the same when we scale up. What changes is how hard it is to compute the gradient, and why we need backpropagation.
• Feedforward Networks: Notation and the Forward Pass — The layer-by-layer structure of a feedforward network, the symbols we will use, and a worked forward pass with real numbers.
• Backpropagation Intuition: The Chain Rule on a Thin Network — Use a network with one neuron per layer to see how the chain rule gives every gradient, why we work backwards, and why gradients fade.
• The Gradient at the Output Layer — Derive the starting point of backpropagation for softmax with cross-entropy: the gradient is simply the prediction minus the target.
• Gradients of Weights and Biases: The Outer Product — Given the gradient at a layer's pre-activation, the weight gradient is the product of that gradient with the layer's input, and the bias gradient is the gradient itself.
• Sending the Gradient Back Through a Hidden Layer — How the gradient at one layer's pre-activation turns into the gradient at the layer before it: multiply by the transposed weights, then by the activation's slope.
• The Backpropagation Algorithm — The forward and backward passes as pseudocode and working Python, a numerical gradient check, and a network that learns XOR.
• Recap: Derivatives and Gradient Descent in Networks — The vocabulary, the four formulas and a debugging checklist, in one place.
• Reading Contour Plots — Contour maps show a loss surface from above. Learn to read them, and see why plain gradient descent zigzags in a narrow valley.
• Momentum — Let the update build up speed by remembering past gradients: faster on flat ground, calmer across steep walls, with a risk of overshooting.
• Nesterov Accelerated Gradient — Look before you leap: measure the gradient where momentum is about to carry you, so you can brake before overshooting.
• Stochastic, Mini-Batch and Batch Gradient Descent — How many examples should each update use? One, a few dozen, or all of them: a trade-off between noise, speed and cost.
• Scheduling the Learning Rate — Why one fixed rate is rarely best, and the common ways to change it over training: decay, line search, warm-up and cosine annealing.
• AdaGrad: A Learning Rate for Every Parameter — Scale each parameter's step by the history of its own gradients, so rarely updated parameters take big steps and frequently updated ones take small steps.
• RMSProp and AdaDelta — Fix AdaGrad's ever-shrinking rate by forgetting old gradients, and see one way to remove the learning rate altogether.
• Adam and AdaMax — Combine momentum's memory of the gradient with RMSProp's per-parameter scaling, fix the start with bias correction, and see the max-norm variant.
• Choosing an Optimizer — All the update rules side by side, a runnable test bed to compare them, honest results on two small problems, and practical advice.
• Underfitting, Overfitting and Test Error — A model that fits its training data perfectly can still fail on new data. How to measure that gap, and why model complexity decides it.
• Bias and Variance — Split a model's test error into three parts: its systematic mistake, its sensitivity to the training sample, and unavoidable noise.
• L2 Regularization and Weight Decay — Add a penalty on the size of the weights to trade a little bias for a large drop in variance, and see that it amounts to shrinking the weights at every step.
• Noise, Data Augmentation and Parameter Sharing — Regularize by changing the data rather than the loss: add noise to inputs, create new examples, share weights, and soften the labels.
• Ensembles: Bagging and Model Averaging — Average many models that make different mistakes, and the variance drops. Bagging trains the models on resampled data.
• Early Stopping — Watch the error on held-out data while training, and stop when it starts to rise. The cheapest regularizer there is.
• Dropout — Randomly switch off hidden neurons during training. It prevents neurons from relying on each other and acts like averaging a huge number of networks.
• Regularization Summary and the Loss Landscape — All the regularization tools side by side, how to combine them, and a picture of what they do to the loss surface.
• Initialization: Starting Training Well — Why the starting weights decide whether signals and gradients survive a deep network, and the variance rules that keep them healthy.
• Unsupervised Pre-training and Autoencoders — How deep networks were first made trainable in 2006: learn useful features from unlabeled data, then fine-tune.
• Transfer Learning — Pre-training showed that a network starts better from learned weights than from random ones. Transfer learning is that idea taken to its conclusion: begin from a model trained on someone else's much larger problem, and adapt it to yours.
• Activation Functions: From Sigmoid to GELU — What goes wrong with sigmoid and tanh, how ReLU and its relatives fix it, and where GELU and SiLU fit.
• Maxout: Learning the Activation Function — Instead of picking a fixed activation, let each unit take the maximum of several learned linear functions.