ModelRefs / Quantization — AI Glossary

Quantization — AI Glossary

Compressing model weights and activations to lower-precision formats (INT8, INT4, FP4) to reduce memory and accelerate inference.

Overview

GPTQ, AWQ, and GGUF are common post-training quantization schemes. INT4 typically loses 1–3 points on benchmarks while cutting memory 4× and improving throughput 2–3×. QLoRA combines quantization with fine-tuning for memory-efficient adaptation.

Reference details

Topic	inference
Last reviewed	2026-06-24

Example: Where the memory goes

A 7B-parameter model at FP16 needs roughly 7e9 × 2 bytes ≈ 14 GB for weights alone, before the KV cache. At INT4 that is about 3.5 GB, which is the difference between needing a data-centre GPU and fitting on a consumer card. Quality loss is usually small but is never zero, and it lands unevenly across tasks.

weights_bytes = params * bytes_per_param
FP16: 7e9 * 2   = 14.0 GB
INT8: 7e9 * 1   =  7.0 GB
INT4: 7e9 * 0.5 =  3.5 GB   # KV cache is on top of this

Commonly confused with

Quantization is not distillation or pruning. The architecture and parameter count are unchanged — only the numeric precision of the stored values differs. Distillation trains a smaller model; pruning removes weights; quantization keeps every weight at lower precision.

When to use it

Reach for it when:

Memory is the binding constraint on which model you can serve
You are serving open-weight models on your own hardware
A small, measured quality loss is an acceptable trade for capacity

Reach for something else when:

You have not measured the loss on your own task — averages hide task-specific damage
The workload is precision-sensitive, such as exact numeric extraction
You are using a hosted API, where the provider already made this decision

Referenced by

This term is used by the following ModelRefs references:

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Quantization — AI Glossary.

Frequently asked questions

What is Quantization?