ModelRefs / Mixture of Experts (MoE) — AI Glossary
Mixture of Experts (MoE) — AI Glossary
A neural architecture routing each token to a small subset of expert sub-networks, decoupling parameter count from active compute.
Overview
MoE models (Mixtral 8x7B, GPT-4 estimated, Gemini 1.5) replace each FFN layer with N expert networks and a gating router that selects top-k experts per token. Active parameters per forward pass ≈ dense_params / N, enabling 3–8× parameter scale-up without proportional compute increase. Training challenges: load balancing across experts, router collapse.
Reference details
| Topic | architecture |
|---|---|
| Also known as | MoE, sparse MoE |
| Last reviewed | 2026-06-24 |
Related terms
Example: Sparse in compute, dense in memory
Take eight experts with the top two routed per token. Each token runs through 2 / 8 = 25% of the expert parameters, so the arithmetic per token is close to a model a quarter the size. The memory is not sparse at all: any expert may be selected by the next token, so all eight must be resident. You get the compute bill of a small model and the VRAM bill of a large one. That single asymmetry explains most practical MoE decisions — why they are attractive to serve at scale and awkward to run on one accelerator, and why quoting a single parameter count for them is meaningless.
Commonly confused with
Total and active parameters are two different numbers and neither substitutes for the other: total predicts memory, active predicts compute and latency. An MoE is also not an ensemble — an ensemble runs several models and combines outputs, while a router here picks a subset of sub-networks *inside* one forward pass, and only those run.
When to use it
Reach for it when:
- Serving at scale, where the compute saving repays the memory cost across many requests
- Reasoning about why a very large model serves faster than its parameter count suggests
- Reading model cards, where total and active counts must both be stated to mean anything
Reach for something else when:
- Memory-constrained deployment — a dense model of the active size is the fair comparison
- Assuming quality tracks total parameters; the active path does the work
- Fine-tuning without accounting for routing, which can collapse onto a few experts
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Mixture of Experts (MoE) — AI Glossary.
Frequently asked questions
What is Mixture of Experts (MoE)?
A neural architecture routing each token to a small subset of expert sub-networks, decoupling parameter count from active compute.
Is Mixture of Experts (MoE) the same as MoE?
Yes — MoE, sparse MoE are common aliases for Mixture of Experts (MoE).
What concepts are related to Mixture of Experts (MoE)?
Closely related concepts include feed forward network, model architecture, parameter count.