The mathematical engine behind backpropagation
A neural network is a composition of functions — layer 1 feeds into layer 2, feeds into layer 3, and so on. To compute how the loss at the end depends on weights at layer 1, you need the chain rule. Backpropagation is literally "apply the chain rule through a computational graph." This is the single most important rule to understand if you want to explain how neural networks learn.
The chain rule handles composed functions: if y depends on u, and u depends on x, then how does y change when x changes? Answer: multiply the rates. If u doubles when x increases by 1, and y triples when u doubles — then y increases by 6 when x increases by 1. You chain the rates together by multiplying.
In a neural network: loss depends on the output layer, which depends on the hidden layer, which depends on the weights. The chain rule lets you trace this dependency all the way back.
If y = f(u) and u = g(x), then:
The derivative of the outer function times the derivative of the inner function. For deeper compositions:
A chain of multiplications — one factor per function in the composition.
To find ∂L/∂w₁ (how loss depends on weight in layer 1):
Each factor is a local derivative — easy to compute at each node. Backprop walks this chain from right to left, accumulating products. This is the full mechanism of backpropagation.
Backprop multiplies many numbers together as it propagates backward. Two problems arise: