Derivatives

Rate of change — the slope at a single point

Why

Training a model means minimizing a loss function. To minimize it, you need to know which direction makes it decrease — and that direction is given by the derivative. Without derivatives there is no gradient descent, no backprop, no training. Every optimizer (SGD, Adam, RMSProp) is entirely built on derivatives. This is the most foundational concept in ML training.

Intuition

A derivative answers one question: if I nudge the input by a tiny amount, how much does the output change? Geometrically, it's the slope of the curve at a single point — the steepness of the tangent line there.

Imagine you're standing on a hilly landscape (your loss surface). The derivative at your current position tells you: how steep is the ground right here, and in which direction does it slope? That's all you need to know to take one step downhill.

Explanation

Formal definition

The derivative of f at point x is the limit of the slope of the secant line as the two points get infinitely close:

f'(x) = df/dx = lim[h→0] (f(x+h) - f(x)) / h

You rarely need to compute limits by hand — what matters is the meaning: derivative = instantaneous rate of change = slope of tangent at x.

If f'(x) > 0 → function is increasing at x. If f'(x) < 0 → decreasing. If f'(x) = 0 → flat — could be a minimum, maximum, or saddle point.

Derivatives you must know cold

d/dx [xⁿ] = n·xⁿ⁻¹ (power rule) d/dx [eˣ] = eˣ (exponential — derivative of itself) d/dx [ln x] = 1/x (log) d/dx [sin x] = cos x d/dx [c] = 0 (constant — zero slope) d/dx [c·f(x)] = c·f'(x) (constant factor pulls out)

Activation function derivatives you'll need:

sigmoid: σ(x) = 1/(1+e⁻ˣ) d/dx [σ(x)] = σ(x)·(1 - σ(x)) ← elegant, self-referential ReLU: f(x) = max(0, x) d/dx [ReLU] = 1 if x > 0, else 0 ← subgradient at x=0 is 0

Second derivative — curvature

The second derivative f''(x) measures how the slope is changing — curvature of the function.

f''(x) > 0 at a critical point → local minimum (concave up, like a bowl)
f''(x) < 0 at a critical point → local maximum (concave down, like a hill)
f''(x) = 0 → inflection point

Second-order optimizers (Newton's method) use the Hessian matrix — the matrix of all second partial derivatives — to take more informed steps. Computationally too expensive for large models, but theoretically more efficient than first-order methods.

← Previous Next →