Bias-Variance tradeoff

The core tension in every modelling decision

Why

This is the single most important concept in applied ML. Every modelling decision — choosing model complexity, adding regularisation, collecting more data, applying dropout — is a move along the bias-variance spectrum. Not being able to explain this clearly means not understanding why models fail and how to fix them.

It's asked directly ("explain bias-variance tradeoff") and indirectly ("why is your model overfitting?", "when would you use L2 regularisation?", "how would you fix underfitting?"). All are the same question in disguise.

Intuition

Bias is the error from wrong assumptions. A linear model fitting a sine wave has high bias — it's systematically wrong no matter how much data you give it. Not paying attention to the data closely enough.

Variance is the error from being too sensitive to the training data. A deep unpruned decision tree memorises every training point but fails on new data — paying too much attention to noise.

The tradeoff: making a model more complex (lower bias) almost always makes it more sensitive to training data (higher variance). The sweet spot captures real patterns without capturing noise.

Explanation

The decomposition

E[(y - ŷ)²] = Bias²(ŷ) + Variance(ŷ) + Irreducible noise Bias(ŷ) = E[ŷ] - f(x) — how wrong the average prediction is Variance(ŷ) = E[(ŷ - E[ŷ])²] — how much predictions vary across datasets Irreducible = Var(ε) — noise in the data — cannot be reduced

The irreducible noise is a floor — even a perfect model cannot go below it. Your job is to minimise Bias² + Variance.

High bias vs high variance

High bias — underfitting

Training error is high. Val ≈ train (both bad). Fix: more complex model, add features, reduce regularisation, train longer.

High variance — overfitting

Training error is low. Val >> train (large gap). Fix: L1/L2 regularisation, dropout, early stopping, more data, simpler model.

The "more data" rule — critical nuance

More data reduces variance — a high-variance model stabilises as the training set grows, narrowing the train/val gap. But more data does not fix bias. If your model is fundamentally wrong (linear model on quadratic data), 10× more data won't help — the architecture needs to change.

Diagnostic: plot learning curves (train/val error vs dataset size). Both plateau high = bias problem. Large gap between them = variance problem.

Where models sit on the spectrum

High bias ←————————————————————→ High variance Linear regression Unpruned decision tree Naive Bayes k-NN (k=1) Logistic regression Deep neural net (unregularised) Regularisation moves a model LEFT (more bias, less variance): Ridge (L2), Lasso (L1), dropout, weight decay, early stopping

← Previous Next →