Gradient Descent — Tutorial

Minimising the loss surface

Gradient descent finds the minimum of a loss function by repeatedly moving in the direction of steepest descent. Imagine a hilly landscape; you are blindfolded and want to reach the lowest valley. At each step you feel the slope under your feet and take one step downhill.

Formally: θ ← θ − α · ∇L(θ), where θ are the model parameters, α is the learning rate, and ∇L is the gradient of the loss with respect to θ.

Learning rate — the critical hyperparameter

Too large a learning rate: you overshoot the minimum and oscillate or diverge. Too small: training converges extremely slowly, and you may get stuck in a shallow local minimum.

Common practice: start with lr=1e-3 for Adam, lr=0.01 for SGD. Use a learning rate schedule (cosine annealing, warmup) to decrease lr as training progresses. Tools like lr_finder in fastai can sweep learning rates automatically.

Variants: SGD, Mini-batch, Adam

Vanilla GD computes the gradient over the entire dataset — accurate but slow for large datasets.

Stochastic GD (SGD) computes it on one sample at a time — noisy but fast per step.

Mini-batch GD uses a batch of 32–512 samples — the standard in practice. The noise from batching acts as regularisation.

Adam (Adaptive Moment Estimation) maintains per-parameter learning rates using estimates of first and second gradient moments. It converges faster than SGD in most settings and is the default for deep learning.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Gradient Descent — Tutorial.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path