ModelRefs / Bias-Variance Tradeoff — Tutorial

Bias-Variance Tradeoff — Tutorial

Diagnose underfitting and overfitting and apply the right fix. Covers The fundamental tension in learning, Diagnosing from learning curves.

Overview

Diagnose underfitting and overfitting and apply the right fix

Level: Advanced. Estimated reading time: 30 minutes.

The fundamental tension in learning

Total expected error = Bias² + Variance + Irreducible noise.

Bias: systematic error — the model's average prediction differs from the true value. High bias = underfitting. The model is too simple to capture the data's pattern.

Variance: sensitivity to training set fluctuations — train on a different sample and you get a very different model. High variance = overfitting. The model learned the noise in the training data, not the signal.

You cannot eliminate both simultaneously with a fixed amount of data — increasing model complexity reduces bias but raises variance.

Diagnosing from learning curves

High bias symptoms: training accuracy ≈ test accuracy, but both are too low. Adding more data barely helps because the model's capacity is the bottleneck.

High variance symptoms: training accuracy >> test accuracy. Large gap between them. Adding more training data will help — it lets the model distinguish signal from noise.

Learning curves plot train and validation error as a function of training set size. A converging gap means good generalisation; a persistent large gap means high variance; curves that converge but at high error means high bias.

Regularisation as the fix

Regularisation adds a penalty for model complexity to the loss: L_reg = L + λ·Ω(θ).

L2 (Ridge): Ω = Σ wᵢ². Shrinks all weights toward zero. Reduces variance without forcing sparsity.

L1 (Lasso): Ω = Σ |wᵢ|. Drives some weights exactly to zero — feature selection. Sparse models.

Dropout (neural nets): randomly zero activations during training — acts like ensembling many sub-networks. Reduces co-adaptation.

Early stopping: stop training when validation loss stops improving — prevents the model from memorising training data.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Bias-Variance Tradeoff — Tutorial.