ModelRefs / Gradient Checkpointing — AI Glossary

Gradient Checkpointing — AI Glossary

A memory-compute trade-off that recomputes intermediate activations during the backward pass instead of storing them.

Overview

Standard backpropagation stores all forward activations for gradient computation, consuming O(n_layers) memory. Gradient checkpointing (Chen et al. 2016) stores only a subset (checkpoints) and recomputes the rest, reducing activation memory by √n at the cost of ~33% extra compute. Essential for training >100B parameter models on limited hardware.

Reference details

Topic	training
Also known as	activation checkpointing, rematerialization
Last reviewed	2026-06-24

Example: Store a few, recompute the rest

Backpropagation normally keeps every layer's forward activations so the backward pass can use them, which scales with depth. Checkpointing keeps only a subset and recomputes the others on the way back. Segmenting an 80-layer model at roughly √80 ≈ 9 means holding about 9 checkpoints and re-deriving at most 9 layers at a time — on the order of 18 activation sets instead of 80. The price is one extra forward pass over the recomputed segments, commonly around a third more compute. It is the cleanest trade in the training stack: a large, predictable memory saving for a modest, predictable slowdown.

Commonly confused with

This targets activation memory, which grows with batch size and sequence length. Sharded strategies like ZeRO and FSDP target parameter, gradient and optimizer memory. They are complementary, not alternatives — a run can be limited by either, and the fix differs, so measure which one is actually filling the device before choosing.

When to use it

Reach for it when:

Long sequences or large batches, where activations dominate memory
Fitting a training run onto the hardware you have rather than acquiring more
As the first lever when out-of-memory happens during the backward pass

Reach for something else when:

When compute-bound and memory is comfortable — you would pay the slowdown for nothing
Inference, which does not retain activations for a backward pass
Before checking whether the optimizer states, not the activations, are the constraint

Primary source

Chen et al., Training Deep Nets with Sublinear Memory Cost (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Gradient Checkpointing — AI Glossary.

Frequently asked questions

What is Gradient Checkpointing?

A memory-compute trade-off that recomputes intermediate activations during the backward pass instead of storing them.

Is Gradient Checkpointing the same as activation checkpointing?

Yes — activation checkpointing, rematerialization are common aliases for Gradient Checkpointing.

What concepts are related to Gradient Checkpointing?

Closely related concepts include mixed precision, deepspeed, fsdp.