ModelRefs / Common Crawl — AI Glossary
Common Crawl — AI Glossary
A publicly available archive of petabyte-scale web text, the primary raw source for most large language model pretraining corpora.
Overview
Common Crawl has crawled the web monthly since 2008, accumulating ~400 TB of compressed WARC data. Directly unusable: requires aggressive filtering (language detection, quality scoring, deduplication, PII removal) to produce usable training data. C4, The Pile, Dolma, and FineWeb are Common Crawl derivatives widely used in research.
Reference details
| Topic | training |
|---|---|
| Last reviewed | 2026-06-24 |
Related terms
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Common Crawl — AI Glossary.
Frequently asked questions
What is Common Crawl?
A publicly available archive of petabyte-scale web text, the primary raw source for most large language model pretraining corpora.
What concepts are related to Common Crawl?
Closely related concepts include pretraining data, data deduplication, data augmentation.