ModelRefs / Layer Normalization — AI Glossary
Layer Normalization — AI Glossary
A normalization technique applied across features of a single sample, stabilizing training of deep transformers. Also called LayerNorm or RMSNorm.
Overview
LayerNorm (Ba et al. 2016) normalizes across the hidden dimension per token, unlike batch norm which normalizes across the batch. Pre-LN (norm before sub-layer) is preferred in modern LLMs for training stability at scale. RMSNorm (a simplified variant) is used in LLaMA and Mistral.
Reference details
| Topic | architecture |
|---|---|
| Also known as | LayerNorm, RMSNorm |
| Last reviewed | 2026-06-24 |
Related terms
Primary source
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Layer Normalization — AI Glossary.
Frequently asked questions
What is Layer Normalization?
A normalization technique applied across features of a single sample, stabilizing training of deep transformers.
Is Layer Normalization the same as LayerNorm?
Yes — LayerNorm, RMSNorm are common aliases for Layer Normalization.
What concepts are related to Layer Normalization?
Closely related concepts include residual connection, transformer, feed forward network.