ModelRefs / Feed-Forward Network (FFN) — AI Glossary
Feed-Forward Network (FFN) — AI Glossary
The position-wise fully connected sub-layer in a transformer block, responsible for most of the model's stored knowledge. Also called FFN or MLP layer.
Overview
Each transformer layer contains a two-layer MLP applied identically to each token position. The FFN typically expands the hidden dimension 4× before projecting back. SwiGLU (used in LLaMA, PaLM) and GeGLU variants outperform ReLU FFNs. Mechanistic interpretability research suggests FFN layers act as key-value memory stores.
Reference details
| Topic | architecture |
|---|---|
| Also known as | FFN, MLP layer, position-wise FFN |
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Feed-Forward Network (FFN) — AI Glossary.
Frequently asked questions
What is Feed-Forward Network (FFN)?
The position-wise fully connected sub-layer in a transformer block, responsible for most of the model's stored knowledge.
Is Feed-Forward Network (FFN) the same as FFN?
Yes — FFN, MLP layer, position-wise FFN are common aliases for Feed-Forward Network (FFN).
What concepts are related to Feed-Forward Network (FFN)?
Closely related concepts include transformer, residual connection, layer normalization.