In deep RNNs, gradients shrink exponentially through time — early timesteps get near-zero updates.

In deep RNNs, gradients shrink exponentially through time — early timesteps get near-zero updates. LSTM introduces a cell state with additive updates (not multiplicative) and three gates: a forget gate that decides what old information to remove, an input gate that decides what new information to store, and an output gate that decides what to expose as output. The forget gate can stay near 1 to preserve gradient flow, avoiding vanishing — e.g. in a long paragraph, if the subject is "the report," the cell state can hold onto that information across many intervening tokens until it's needed again. GRU is a simpler, lighter variant with only two gates instead of three: an update gate that controls how much past information to keep, and a reset gate that controls how much past information to forget. GRU drops the separate cell state and output gate that LSTM has, which makes it faster and cheaper to train while still handling long dependencies far better than a vanilla RNN — the trade-off is slightly less expressive control than LSTM's three-gate design.