ModelRefs / SentencePiece — AI Glossary
SentencePiece — AI Glossary
A language-agnostic subword tokenizer supporting BPE and unigram LM algorithms, used in T5, LLaMA, and Gemma. Also called SPM.
Overview
SentencePiece (Kudo & Richardson, Google 2018) tokenizes directly from raw text without pre-tokenization, making it language-agnostic. It supports both BPE and unigram language model algorithms. Used in T5, LLaMA 1/2/3, Gemma, and multilingual models. Packaged as a standalone C++ library with Python bindings.
Reference details
| Topic | architecture |
|---|---|
| Also known as | SPM |
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to SentencePiece — AI Glossary.
Frequently asked questions
What is SentencePiece?
A language-agnostic subword tokenizer supporting BPE and unigram LM algorithms, used in T5, LLaMA, and Gemma.
Is SentencePiece the same as SPM?
Yes — SPM are common aliases for SentencePiece.
What concepts are related to SentencePiece?
Closely related concepts include byte pair encoding, tiktoken, tokenizer.