Recurrence and gating — how a network carries information forward across a sequence
A CNN or dense layer has no notion of order or variable length — it expects a fixed-size input processed all at once. Text, audio, time series, and sensor logs are sequences where order carries meaning and length varies per example. RNNs process one step at a time and carry a hidden state forward, so the network's output at any point can depend on everything it has seen so far. "Why not just use a CNN with a wide window instead?" is a common follow-up — a fixed window caps how far back the model can look; recurrence in principle doesn't.
Think of the hidden state as a running summary the network updates one token at a time — like reading a sentence left to right and keeping a mental note of what matters so far. Each new input gets blended into that summary using the same learned weights at every step, which is what lets an RNN handle sequences of any length with a fixed number of parameters.
The same three weight matrices are used at every step, so the model doesn't need separate parameters per position — this weight sharing across time is the sequence-model analogue of a CNN's weight sharing across space.
Training an RNN means unrolling it across all time steps and backpropagating through that whole chain (BPTT). Because the same weight matrix is multiplied in at every step, gradients flowing back through many steps get repeatedly scaled by the same factor — if that factor is less than 1 the gradient vanishes, if greater than 1 it explodes. In practice this means plain (vanilla) RNNs struggle to learn dependencies more than a few dozen steps apart, which motivated gated architectures.
LSTMs add a separate cell state plus forget, input, and output gates that learn when to keep, overwrite, or expose information — the cell state's update is closer to additive than repeatedly multiplicative, which keeps gradients from vanishing as fast. GRUs simplify this to two gates (reset and update) and merge the cell and hidden state, training faster with fewer parameters while covering most of the same use cases. Both are still the standard answer to "how do you handle long-range dependencies without a transformer?"