Perceptrons, layers, and why nonlinearity is what makes depth meaningful
Every architecture — CNNs, RNNs, transformers — is built from the same primitive: a linear transform followed by a nonlinear activation. Understanding the primitive itself matters more than the frameworks built on top of it.
The single most common trap question is "why not just stack linear layers?" — if you can't answer it, you don't yet understand why depth helps at all.
A single neuron is a weighted vote: multiply each input by how much it matters (weight), add a bias to shift the threshold, and squash the result through an activation function. Stack neurons into a layer, stack layers into a network, and each layer learns to re-represent the input in a way that makes the next layer's job easier.
Without the nonlinear squashing step, none of that re-representation matters — the whole network collapses to a single linear function, no matter how many layers you add.
Weights control how strongly each input influences a neuron; the bias shifts the decision boundary independent of the input. Training means adjusting W and b so the network's output matches the target.
The activation function breaks that collapse — it's what lets each additional layer add real representational power instead of being algebraically absorbed into the layer before it.
ReLU won out over sigmoid/tanh for hidden layers mainly because it keeps gradients alive through many layers — this is the single biggest practical reason deep networks became trainable at all.