LSTMs & GRUs — Tutorial

The vanishing gradient problem in RNNs

Vanilla RNNs maintain a hidden state h_t that is updated at each time step: h_t = tanh(W_h · h_{t-1} + W_x · x_t). The problem: when backpropagating through time (BPTT) over many steps, gradients are multiplied by the weight matrix W_h at every step. If the largest eigenvalue of W_h is less than 1, gradients shrink exponentially — the network cannot learn dependencies spanning more than ~10 tokens. If eigenvalue > 1, gradients explode.

This makes vanilla RNNs useless for most real sequential tasks. The LSTM (Long Short-Term Memory, Hochreiter & Schmidhuber 1997) and GRU (Gated Recurrent Unit, Cho et al. 2014) solve this through gating mechanisms that create additive (not multiplicative) paths for gradient flow — similar in spirit to the residual connections in ResNets.

LSTM: the cell state and four gates

The LSTM adds a cell state c_t alongside the hidden state h_t. The cell state is a "conveyor belt" — information can flow through it unchanged via additive updates, avoiding the vanishing gradient problem.

Four gates control information flow (all are sigmoid or tanh activations):

Forget gate: f_t = σ(W_f · [h_{t-1}, x_t] + b_f). Decides what to erase from cell state. Output in [0,1] — 0 = forget completely, 1 = keep completely.

Input gate: i_t = σ(W_i · [h_{t-1}, x_t] + b_i). Decides which new values to write.

Candidate values: g_t = tanh(W_g · [h_{t-1}, x_t] + b_g). New content to potentially write.

Cell update: c_t = f_t ⊙ c_{t-1} + i_t ⊙ g_t. The cell state is updated additively.

Output gate: o_t = σ(W_o · [h_{t-1}, x_t] + b_o). Decides what to expose as hidden state: h_t = o_t ⊙ tanh(c_t).

The gradient path through c_t is additive — the forget gate can be near 1 for many steps, letting gradients flow undiminished.

GRU and when to use each

The GRU (Gated Recurrent Unit) simplifies the LSTM by merging the cell and hidden states and using only two gates:

Reset gate: r_t = σ(W_r · [h_{t-1}, x_t]). Controls how much of the previous state to use when computing new content.

Update gate: z_t = σ(W_z · [h_{t-1}, x_t]). Blends old and new hidden state: h_t = (1-z_t) ⊙ h_{t-1} + z_t ⊙ tanh(W · [r_t ⊙ h_{t-1}, x_t]).

GRU has ~25% fewer parameters than LSTM with comparable performance on most tasks. Use GRU when data is limited or training speed matters.

When to use LSTMs/GRUs today: time-series forecasting, anomaly detection on sensor streams, character-level text generation, audio classification, and sequence labelling (POS tagging, NER) where the sequence length is < 500. For long sequences (text > 512 tokens, long documents), Transformers dominate. But for low-latency sensor/time-series on embedded hardware, RNNs are still preferred — they process one step at a time with constant memory.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LSTMs & GRUs — Tutorial.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path