NexForay

Gradient

The vector of all partial derivatives — direction of steepest ascent

Why

The gradient is what every optimizer actually uses. When you call loss.backward() in PyTorch, it computes the gradient of the loss with respect to every parameter. The gradient vector is the complete answer to "which direction makes the loss increase the fastest?" — and you walk in the opposite direction to minimize it.

Intuition

Think of the gradient as a compass for a hilly landscape with millions of dimensions. It points uphill — in the direction of steepest increase. Flip it and you get the direction of steepest descent. Each component tells you the slope in one parameter's direction.

The gradient always points perpendicular to contour lines (lines of equal loss). To descend most efficiently, follow the negative gradient.

Explanation
Definition

For a function f(w₁, w₂, ..., wₙ) with n parameters, the gradient is a vector of all partial derivatives:

∇f = [∂f/∂w₁, ∂f/∂w₂, ..., ∂f/∂wₙ]

The gradient has the same shape as the parameter vector — one number per parameter. For a network with 10 million weights, the gradient is a 10-million-dimensional vector.

Key properties
  • ∇f at a point points in the direction of steepest increase of f
  • −∇f points in the direction of steepest decrease — this is the descent direction
  • The magnitude ||∇f|| tells you how steep the slope is at that point
  • At a minimum (or maximum or saddle): ∇f = 0 — all partial derivatives are zero
  • The gradient is perpendicular to the level curves (contour lines) of f
Gradient of cross-entropy loss

For logistic regression with sigmoid output σ(z) and binary cross-entropy loss:

L = -[y·log(σ(z)) + (1-y)·log(1-σ(z))] ∂L/∂z = σ(z) - y ← prediction minus true label

Beautifully simple: the gradient is just how wrong your prediction is. If σ(z) = 0.9 and y = 0, gradient = 0.9 — large, because you were very wrong. This is why cross-entropy + sigmoid is the standard for binary classification.

Jacobian & Hessian
Jacobian matrix
Generalization of the gradient when the output is also a vector. Matrix of all partial derivatives of all outputs w.r.t. all inputs. Shape: (m×n) for m outputs, n inputs. Used in autograd frameworks.
Hessian matrix
Matrix of all second partial derivatives. Captures curvature. Used by second-order optimizers. Too expensive for large neural nets — n² entries for n parameters.
← Previous Next →