ModelRefs / Tokenizer — AI Glossary
Tokenizer — AI Glossary
The component that converts text into tokens and back, using a fixed vocabulary of sub-word pieces. Three related things get used interchangeably.
Overview
Tokenizers (BPE, SentencePiece, Tiktoken) define the model's vocabulary. Different families use different tokenizers, which is why token counts for the same prompt vary across providers. Multilingual prompts may tokenize 2–3× less efficiently than English.
Reference details
| Topic | core |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Commonly confused with
Three related things get used interchangeably. The token is the unit, byte-pair encoding is one algorithm for deciding what the units are, and the tokenizer is the component that applies a specific learned vocabulary to turn text into IDs and back. Two tokenizers using the same algorithm still produce different token counts for the same text, so the vocabulary is the thing that has to match, not the algorithm.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Tokenizer — AI Glossary.
Frequently asked questions
What is Tokenizer?
The component that converts text into tokens and back, using a fixed vocabulary of sub-word pieces.
What concepts are related to Tokenizer?
Closely related concepts include token, llm.