ModelRefs / Model Distillation — AI Glossary
Model Distillation — AI Glossary
Training a smaller student model to mimic a larger teacher model's outputs, transferring capability at lower cost.
Overview
Distillation transfers capability from frontier models into smaller, cheaper variants (GPT-4o-mini, Claude Haiku, Gemini Flash). Token-level KL divergence on teacher logits is the most common objective. Black-box distillation uses teacher completions as supervised fine-tuning data.
Reference details
| Topic | training |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Model Distillation — AI Glossary.
Frequently asked questions
What is Model Distillation?
Training a smaller student model to mimic a larger teacher model's outputs, transferring capability at lower cost.
What concepts are related to Model Distillation?
Closely related concepts include fine tuning, quantization, base model.