Artificial Neurons

From a biological loose analogy to a trainable unit — and why "error" is what makes learning possible

Why

Every deep learning conversation traces back to this unit. Two things matter: knowing the artificial neuron is a loose, stripped-down analogy rather than a copy of biology, and knowing precisely why the MP neuron couldn't learn while the perceptron could.

The perceptron's limits — and the "error surface" framing of training — are also the setup for gradient descent and backpropagation, so getting this right pays off on every later topic.

Intuition

A biological neuron collects signals from other neurons and only "fires" once the combined signal crosses a threshold. Artificial neurons borrow that idea in stripped-down math form: take a weighted sum of inputs, compare it to a threshold, fire or don't. The whole history since 1943 is about making that threshold decision trainable, and making "firing" smooth enough to compute a gradient on.

Explanation

The biological inspiration

Dendrites → inputs (x1, x2, ...) Synapse strength → weight (w1, w2, ...) Cell body → weighted sum of inputs Axon fires → activation / step function decides output

The mapping is loose on purpose — real neurons spike in time, have thousands of dynamic synapses, and involve chemistry with no clean math analogue. The artificial neuron keeps only the shape of the idea (combine inputs, decide whether to fire) so it can be built out of arithmetic instead of biology. Know the analogy well enough to state its limits — that's usually the real question.

The MP neuron and its limitations

MP Neuron (McCulloch-Pitts, 1943): y = 1 if Σxi ≥ θ else 0 binary inputs, hand-set weights and threshold

The MP neuron proved that networks of simple threshold units could compute logic functions (AND, OR, NOT) — a foundational result. But it had no learning rule: a person had to hand-pick the threshold θ for every task. It also only handles binary inputs with implicitly equal weight per input, and — like every single-layer threshold unit — it can only separate linearly separable data, so it can't compute XOR no matter how θ is tuned.

The perceptron neuron

Perceptron (Rosenblatt, 1958): z = w·x + b y = 1 if z ≥ 0 else 0 Learning rule, per mistake: w ← w + η(target - y)x b ← b + η(target - y)

The perceptron generalizes the MP neuron two ways: real-valued weights per input (not just on/off), and a bias term that shifts the threshold instead of hand-fixing it. Its real contribution is the learning rule — weights are nudged from mistakes, and the perceptron convergence theorem guarantees this finds a separating boundary if the data is linearly separable. It still can't solve XOR (still one linear boundary), and its step function has zero gradient almost everywhere, which is why it can't be trained with gradient descent directly — later networks swap the step for a smooth activation to fix exactly this.

Error and the error surface

Per-example error (perceptron rule): e = target - y General loss over a dataset, e.g. mean squared error: L(w, b) = (1/n) Σ (target_i - prediction_i)² Error surface: the shape of L plotted as a function of (w, b) — a landscape in weight-space, with a minimum where predictions best match targets.

The perceptron's mistake-driven update is implicitly walking down this surface, even though it uses a discrete "right or wrong" signal instead of a smooth slope. Once you frame training as "minimize a function over an error surface," the next question answers itself: replace the discrete rule with a differentiable loss, and follow the slope — the gradient — downhill instead of reacting to individual mistakes. That reframing is exactly what gradient descent and backpropagation formalize next.

← Previous Next →