The shape of your data determines your model — and your loss function
Every assumption you make when building a model is implicitly a distributional assumption. When you use MSE loss, you're assuming Gaussian noise. When you use cross-entropy, you're assuming Bernoulli or Categorical outputs. When you initialise weights from a normal distribution, you're using the Gaussian.
Questions like "why cross-entropy for classification?" and "what distribution does softmax output?" both require knowing these distributions cold. Generative models (VAEs, diffusion) are entirely defined by distributional choices.
A probability distribution is a function that assigns probabilities to every possible outcome. Different phenomena follow different shapes: coin flips are binary (Bernoulli), class labels are categorical, measurement errors cluster around zero (Gaussian), event counts in time are Poisson.
Choosing the right distribution for your model is choosing the right language to describe your data. Using the wrong one is like measuring distance in seconds — the math will work but the answers will be wrong.
The bell curve. Symmetric around the mean. Ubiquitous because of the Central Limit Theorem: the mean of many independent random variables converges to Gaussian regardless of the original distribution.
In ML: noise in linear regression is assumed Gaussian → MSE is the right loss. Weight initialisation often uses N(0, σ²). Latent variables in VAEs are Gaussian.
Single binary trial. Coin flip is the canonical example. In ML: the output of sigmoid in binary classification is p — the model's estimate of P(y=1|x). Binary cross-entropy loss is derived from the Bernoulli likelihood.
Generalises Bernoulli to K classes. In ML: softmax outputs a Categorical distribution — the K numbers sum to 1 and represent the model's probability distribution over classes. Categorical cross-entropy loss is derived from the Categorical likelihood.