Recurrent Neural Networks — Tutorial

Sequences and hidden state

Standard feedforward networks process fixed-size inputs independently. An RNN processes a sequence one element at a time, maintaining a hidden state h that carries information from previous steps:

h_t = tanh(W_h · h_{t-1} + W_x · x_t + b)

The same weights (W_h, W_x, b) are used at every timestep — weight sharing over time, analogous to CNN's weight sharing over space. The hidden state is the network's "memory."

For sequence classification (sentiment analysis), use the final hidden state h_T to make a prediction. For sequence-to-sequence tasks (translation), use all hidden states as a context for decoding.

A deep RNN stacks multiple RNN layers: h¹_t → h²_t → h³_t. Each layer processes the sequence of hidden states from the layer below.

The vanishing gradient problem

Training RNNs with backpropagation through time (BPTT) requires multiplying gradients across each timestep. For a sequence of 100 steps, the gradient of the loss with respect to h_1 involves the product of 99 Jacobians: ∂L/∂h_1 = (∂h_T/∂h_{T-1}) × ... × (∂h_2/∂h_1) × ∂L/∂h_T.

If each factor is < 1 (which is typical with tanh), this product approaches 0 exponentially. The gradient vanishes before reaching early timesteps — the network cannot learn dependencies between tokens far apart in the sequence.

If each factor is > 1, the gradient explodes. Gradient clipping (clip_grad_norm_(model.parameters(), max_norm=1.0)) prevents explosion but cannot fix vanishing.

The practical consequence: vanilla RNNs cannot reliably learn dependencies longer than ~10 steps. LSTMs and GRUs were designed specifically to address this.

LSTMs: gated memory cells

LSTM (Long Short-Term Memory, Hochreiter & Schmidhuber 1997) adds an explicit memory cell c_t alongside the hidden state h_t. Three gates control information flow:

Forget gate f_t: how much of the previous cell state to keep (sigmoid → 0–1) Input gate i_t: how much new information to write to the cell Output gate o_t: how much of the cell to expose as the hidden state

The cell state update: c_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t The hidden state: h_t = o_t ⊙ tanh(c_t)

The cell state flows through time with only element-wise multiplication (by f_t) — no matrix multiply. This gives gradients a "highway" that doesn't collapse over long sequences. LSTMs can learn dependencies across hundreds of timesteps.

GRU (Gated Recurrent Unit) is a simpler variant with 2 gates instead of 3 — about 25% fewer parameters, similar performance, faster to train.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Recurrent Neural Networks — Tutorial.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path