ModelRefs / LoRA (Low-Rank Adaptation) — AI Glossary

LoRA (Low-Rank Adaptation) — AI Glossary

A PEFT method that injects small trainable rank-decomposition matrices into frozen model weight layers. LoRA is not quantization and the two are independent.

Overview

LoRA cuts trainable parameters by 100–1,000× and produces tiny (MB-scale) adapter files that can be hot-swapped at inference time or merged into the base model. QLoRA combines LoRA with 4-bit quantization to fine-tune 65B models on a single consumer GPU.

Reference details

Topic	training
Also known as	Low-Rank Adaptation
Last reviewed	2026-06-24

Example: Why the adapter is so small

A weight matrix of 4096 × 4096 holds about 16.8M parameters. LoRA freezes it and trains two matrices of 4096 × 8 and 8 × 4096 instead — roughly 65.5K parameters, about 0.4% of the original. That is the whole idea: the update is assumed to be low-rank, so you train the small factors and leave the large matrix alone.

# rank r = 8 on one 4096 x 4096 layer
full   = 4096 * 4096          # 16,777,216 trainable
lora   = 4096 * 8 + 8 * 4096  #     65,536 trainable  (~0.39%)

Commonly confused with

LoRA is not quantization and the two are independent. LoRA reduces how many parameters you train; quantization reduces the precision each weight is stored in. QLoRA is simply both at once — a 4-bit base model with LoRA adapters trained on top.

When to use it

Reach for it when:

You want several task-specific variants sharing one base model
Adapters need to be swapped, versioned or rolled back independently
Full fine-tuning will not fit in available memory

Reach for something else when:

The change is large and general — a big domain shift may need full fine-tuning
Prompting or retrieval has not been tried; this is the more expensive lever
You cannot evaluate the result; a quietly worse adapter is easy to ship

Referenced by

This term is used by the following ModelRefs references:

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LoRA (Low-Rank Adaptation) — AI Glossary.

Frequently asked questions

What is LoRA (Low-Rank Adaptation)?

A PEFT method that injects small trainable rank-decomposition matrices into frozen model weight layers.

Is LoRA (Low-Rank Adaptation) the same as Low-Rank Adaptation?

Yes — Low-Rank Adaptation are common aliases for LoRA (Low-Rank Adaptation).

What concepts are related to LoRA (Low-Rank Adaptation)?

Closely related concepts include peft, fine tuning, quantization.