Backpropagation

Computational graphs and autodiff — how frameworks compute every gradient in one backward pass

Why

Every framework call to .backward() is backprop. What matters isn't the calculus — it's understanding what the framework is doing under the hood: how a graph of tensor operations turns into a single efficient gradient computation.

This page covers the deep-learning framing: computational graphs and automatic differentiation. The chain-rule derivation itself is covered in Calculus → Backpropagation — read that first if you want the math from scratch.

Intuition

Picture the forward pass as a DAG: each node is an operation (matmul, add, activation), each edge carries a tensor. Autodiff attaches a "local gradient recipe" to every node when it runs forward. The backward pass then walks the graph in reverse, multiplying local gradients together via the chain rule to get the gradient of the loss with respect to every parameter — all in one traversal.

Nothing is symbolic or numeric-differencing here — it's exact derivatives, computed mechanically, reusing the same graph the forward pass built.

Explanation

Forward pass builds the graph, backward pass walks it

Forward: x → [matmul W1] → [+b1] → [ReLU] → [matmul W2] → ŷ → [loss] → L each op caches what it needs for its local derivative Backward: L → dL/dŷ → dL/d(matmul W2) → dL/d(ReLU) → dL/d(+b1) → dL/d(matmul W1) → dL/dx each op receives the upstream gradient, multiplies by its local derivative, passes the product further back (chain rule)

Memory cost scales with depth because every cached forward activation must be kept alive until its node is visited on the way back — this is why very deep nets are memory-hungry and why techniques like gradient checkpointing (recompute instead of cache) exist.

Why "one backward pass" is the whole point

Naively, you could estimate each weight's gradient by perturbing it and re-running the forward pass (finite differences) — that's one forward pass per parameter, infeasible at millions of parameters. Backprop instead reuses shared intermediate gradients: a gradient computed once at a shared node is reused by every path that depends on it, so the total cost of computing all gradients is about the same as one extra forward pass.

Autodiff in practice (PyTorch-style)

requires_grad=True — marks a tensor so operations on it are recorded onto the graph
loss.backward() — traverses the graph in reverse, accumulates into .grad on every leaf tensor
optimizer.step() — applies the update rule (see Optimization) using those accumulated gradients
optimizer.zero_grad() — clears .grad before the next forward pass; gradients accumulate by default, a classic bug source
dynamic graphs (PyTorch) rebuild the graph every forward call, so control flow (loops, conditionals) can depend on data — static graphs (early TensorFlow) trade that flexibility for ahead-of-time optimization

← Previous Next →