Backpropagation

The algorithm that makes gradient descent practical

Why

Gradient Descent needs the gradient of the loss with respect to every weight. A network with millions of weights can't compute each gradient independently — that would take millions of forward passes. Backpropagation computes all gradients in a single backward pass by reusing intermediate computations. This is why deep learning is computationally feasible at all.

Intuition

Think of backprop as a blame assignment algorithm. The network makes a prediction, computes a loss, then works backward through each layer asking: "how much did this weight contribute to the error?" Weights that contributed more get a larger gradient — they're more responsible and need a larger correction.

The key insight: each layer only needs to know two things — the error signal coming from the layer above, and its own local derivative. It can compute its contribution without knowing anything about the rest of the network.

Explanation

Forward pass vs backward pass

Forward pass: input → layer 1 → layer 2 → ... → output → loss (compute activations and cache them) Backward pass: loss → layer N → layer N-1 → ... → layer 1 (compute gradients using chain rule + cached activations)

The forward pass caches intermediate activations because the backward pass needs them to compute local derivatives. This is why memory usage scales with depth — you need to store all activations until the backward pass is complete.

The four equations of backprop

For a layer with weights W, bias b, input x, pre-activation z = Wx + b, activation a = σ(z):

δᴸ = ∇ₐL ⊙ σ'(zᴸ) ← error at output layer δˡ = (Wˡ⁺¹)ᵀ · δˡ⁺¹ ⊙ σ'(zˡ) ← error propagated backward ∂L/∂Wˡ = δˡ · (aˡ⁻¹)ᵀ ← gradient for weights ∂L/∂bˡ = δˡ ← gradient for biases

⊙ is element-wise multiplication. Each layer passes δˡ backward to the layer below — this is the "error signal." Each layer also computes its own weight gradients from δˡ and its input activations.

Computational graph & automatic differentiation

Modern frameworks (PyTorch, JAX) implement backprop via automatic differentiation on a computational graph. Every operation in the forward pass registers a backward function. When you call loss.backward(), PyTorch traverses the graph in reverse and accumulates gradients automatically. You never write backprop by hand.

requires_grad=True: tells PyTorch to track operations on this tensor
loss.backward(): triggers the backward pass, populates .grad on all tracked tensors
optimizer.step(): applies the update rule using the accumulated gradients
optimizer.zero_grad(): clears gradients before the next forward pass (critical — PyTorch accumulates by default)

← Previous Next →