ModelRefs / Scaling Law — AI Glossary
Scaling Law — AI Glossary
Empirical power-law relationships between model size, training compute, dataset size, and loss that predict model performance. Also called neural scaling law.
Overview
Kaplan et al. (OpenAI 2020) showed loss scales predictably as a power law of parameters and data. Hoffmann et al. (DeepMind 2022, 'Chinchilla') refined optimal compute allocation: tokens ≈ 20× parameters. Scaling laws guide model design budgets but break down for emergent capabilities and instruction-tuned models.
Reference details
| Topic | architecture |
|---|---|
| Also known as | neural scaling law |
| Last reviewed | 2026-06-24 |
Related terms
Example: The point is to predict before you spend
Training compute is approximately 6ND floating-point operations for N parameters and D training tokens: a 7B model on 140B tokens is about 6 × 7e9 × 1.4e11 ≈ 5.9e21 FLOPs. Nobody discovers the right N and D by trying them at that scale. You train a ladder of small models, fit loss as a power law of compute, and extrapolate to choose the size of the run you can only afford once. Kaplan et al. fitted such a law in 2020; Hoffmann et al. refitted it in 2022 with the token budget varied properly and moved the compute-optimal allocation substantially towards more data — the same method, a different answer, because the first fit had held one variable too still.
Commonly confused with
A scaling law predicts loss, not capability. Loss falls smoothly; downstream benchmark accuracy can stay flat and then move sharply, which is what claims about emergent abilities are usually describing. The law is also fitted to one architecture, dataset and optimizer — it is an empirical curve, not a physical constant, and changing the recipe means refitting it.
When to use it
Reach for it when:
- Allocating a fixed training budget between model size and training tokens
- Extrapolating from a ladder of small runs before committing to a large one
- Sanity-checking a published training recipe against the compute it claims to use
Reach for something else when:
- Predicting benchmark scores or specific capabilities from loss alone
- Extrapolating far outside the fitted range, or across an architecture change
- Justifying that bigger is always better — inference cost, not training loss, usually decides the size you ship
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Scaling Law — AI Glossary.
Frequently asked questions
What is Scaling Law?
Empirical power-law relationships between model size, training compute, dataset size, and loss that predict model performance.
Is Scaling Law the same as neural scaling law?
Yes — neural scaling law are common aliases for Scaling Law.
What concepts are related to Scaling Law?
Closely related concepts include chinchilla, emergence, parameter count.