Gradient descent: learning step by step

Why we need it

Linear regression with one feature has an exact formula. Most models, including neural networks with millions of weights, do not. Gradient descent is the general method that trains almost all of them. If you understand it on one weight, you understand the core of deep learning.

The loss function

A loss function turns "how wrong is the model?" into one number. For regression, a common choice is the mean squared error. Training means finding the weights that make the loss as small as possible. Picture the loss as a valley: each possible weight is a position, and the height is the loss. We want the lowest point.

The gradient

The gradient is the slope of the loss at the current weight: how fast the loss rises if the weight increases a little. If the gradient is positive, increasing the weight makes things worse, so we decrease it. If it is negative, we increase it. The update rule is always: new weight = old weight − learning rate × gradient.

A tiny worked example

Take the loss f(w) = (w − 3)². Its lowest point is at w = 3. Its gradient is 2(w − 3). Start at w = 0 with a learning rate of 0.1. The gradient is 2 × (0 − 3) = −6, so the new w is 0 − 0.1 × (−6) = 0.6. Next, the gradient is 2 × (0.6 − 3) = −4.8, and w becomes 1.08. Each step moves closer to 3, and the steps get smaller as the slope flattens.

w = 0.0
lr = 0.1
for step in range(5):
    grad = 2 * (w - 3)
    w = w - lr * grad
    print(step + 1, round(w, 4))

The learning rate

• Too small: training is very slow, and may stop before reaching the bottom.
• Too large: steps jump over the valley, the loss bounces around or even grows to infinity (it "diverges"). For f(w) = (w − 3)², any learning rate above 1.0 diverges.
• Just right: the loss drops quickly and settles. In practice people try values like 0.1, 0.01 and 0.001.

Batch, stochastic and mini-batch

An epoch is one full pass over the training data. Batch gradient descent computes the gradient on all examples before each step: accurate but slow for big data. Stochastic gradient descent (SGD) uses one example per step: fast but noisy. Mini-batch gradient descent uses small groups, like 32 or 256 examples, and is what almost every deep learning library uses.

Common mistakes

• Features on very different scales (area in thousands, rooms in single digits) make the valley long and thin, so descent zig-zags. Fix: scale features (module 8).
• Watching only the final loss. Fix: plot the loss after every epoch; it should go down and level off.
• Forgetting the minus sign and climbing uphill. The update subtracts the gradient.