Why loss functions exist — deriving them from probability
MLE and MAP are the theoretical frameworks that explain why we minimise the losses we do. Most engineers use MSE and cross-entropy every day without knowing why — but understanding that they are maximum likelihood estimators is what separates someone who reasons from first principles from someone who follows recipes.
They also explain regularisation from a Bayesian perspective — L2 regularisation is exactly MAP estimation with a Gaussian prior.
MLE asks: "What parameters make the data I observed most probable?" You find the parameters that best explain the data, with no assumptions about what the parameters should look like beforehand.
MAP asks the same question but with a twist: "What parameters make the data most probable, while also being themselves plausible given my prior beliefs?" It's MLE plus a prior — a regularised version of MLE that prevents extreme parameter values.
Given observed data D = {x₁, x₂, ..., xₙ} and a model with parameters θ, MLE finds the θ that maximises the probability of observing D:
In practice: take the log (monotonic, doesn't change argmax):
Assume: outputs = true value + Gaussian noise: y = θᵀx + ε, where ε ~ N(0, σ²)
The Gaussian noise assumption directly gives you MSE. It's not arbitrary — it's the statistically correct loss for Gaussian-distributed outputs.
Assume: outputs follow a Bernoulli distribution with p = σ(θᵀx)
The Bernoulli assumption directly gives binary cross-entropy. Extend to K classes with Categorical → get categorical cross-entropy.
MAP adds a prior P(θ) — your belief about plausible parameter values before seeing data:
The extra term log P(θ) acts as a regulariser — it penalises parameter values that are improbable under the prior.
Regularisation is not an ad hoc trick — L2 (prefer small weights) corresponds to believing weights are normally distributed around zero before seeing any data. The strength λ is the inverse of the prior's variance — a tight prior (small σ²) gives strong regularisation.