ModelRefs / LoRA & PEFT — Tutorial

LoRA & PEFT — Tutorial

Fine-tune a 7B model on a consumer GPU — parameter-efficient methods that change 0.1% of weights. Covers The memory problem with full fine-tuning.

Overview

Fine-tune a 7B model on a consumer GPU — parameter-efficient methods that change 0.1% of weights

Level: Expert. Estimated reading time: 40 minutes.

The memory problem with full fine-tuning

Full fine-tuning a 7B parameter model in fp32 requires roughly 28GB just for the weights. Add gradients (28GB), Adam optimiser states (56GB), and activations — you're looking at 120GB+ GPU memory. That's 2–4 A100s.

PEFT (Parameter-Efficient Fine-Tuning) methods update only a tiny fraction of parameters while keeping most of the model frozen. The frozen weights stay in memory but need no gradient storage or optimiser states, reducing memory to a fraction of full fine-tuning.

Key insight: for most adaptation tasks, you don't need to change all 7 billion weights. The language model already knows grammar, reasoning, and world knowledge. You need to shift a small subspace of its representations — which a low-rank update can express very efficiently.

LoRA: Low-Rank Adaptation

LoRA (Hu et al., 2021) freezes the pretrained weight matrices and adds a trainable low-rank decomposition alongside each one:

W_new = W_frozen + α · (B · A)

Where W is the original d×k matrix, A is r×k (randomly initialised), B is d×r (initialised to zero so the update starts at zero), and r << d. The rank r controls the number of trainable parameters.

For a 7B model with r=16, updating all attention projection matrices adds ~17M trainable parameters (0.25% of total). Training cost is comparable to a 17M parameter model despite using the full 7B as the base.

After training, merge the LoRA weights back: W_new = W_frozen + BA, giving zero-overhead inference. Or keep them separate and hot-swap adapters at runtime.

QLoRA (Dettmers et al., 2023) quantises the frozen base to 4-bit (NF4), reducing GPU memory to ~6GB for a 7B model while maintaining near full fine-tuning quality.

Which layers to adapt and key hyperparameters

Target modules: LoRA is typically applied to the query (q_proj), key (k_proj), value (v_proj), and output projection (o_proj) matrices of every attention layer. Some practitioners also add feed-forward layers.

Key hyperparameters: r (rank): 8–64 for most tasks. Higher r = more parameters = more flexibility. Start at 16. alpha (α): controls the effective learning rate. Common: alpha = 2 × r, so lora_alpha=32 with r=16. dropout: 0.05–0.1 for regularisation.

Practical rule: if validation loss plateaus with systematic errors, increase r. If overfitting, reduce r or increase dropout.

The PEFT library wraps this: get_peft_model(model, lora_config) inserts the LoRA matrices and freezes everything else in two lines.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to LoRA & PEFT — Tutorial.