QLoRA: fine-tuning a big model on one GPU — and the knobs that matter
QLoRA fine-tunes a large model by freezing a 4-bit-quantized base and training a tiny adapter. It put fine-tuning within reach of a single GPU. Here's how it works and which hyperparameters to tune.
Fine-tuning a large model used to mean a cluster of expensive GPUs, because you had to update every one of its billions of parameters. QLoRA changed that — it makes fine-tuning a big model possible on a single GPU, and it's become the default way most teams customise open models.
LoRA, then QLoRA
LoRA (Low-Rank Adaptation) freezes the base model's weights and instead trains small 'adapter' matrices — a tiny fraction of the parameters — that capture the task-specific change. QLoRA adds quantization: the frozen base is stored in 4-bit precision (NF4), which slashes memory enough that a large model both fits and fine-tunes on one GPU. You train the adapter; the base never moves.
Why it works so well
- The adapters capture the task-specific delta, which is genuinely small — you don't need to move the whole model.
- The base knowledge stays intact, so there's less catastrophic forgetting than full fine-tuning.
- Memory drops enough to democratise fine-tuning — from a data centre to a single rented GPU.
The hyperparameters that matter
- Rank (r) — the adapter's capacity; higher means more expressive and more trainable parameters. Start low (8-16) and raise only if you need to.
- Alpha — a scaling factor for the adapter, commonly set around twice the rank.
- Target modules — which layers get adapters; the attention projection layers are the usual choice.
- Learning rate — typically higher than full fine-tuning; a key dial to get right.
- Dropout, epochs, and batch size — the standard levers to balance fit against overfitting.
Practical notes
Start small — low rank, few epochs — and watch for overfitting, which comes fast on small datasets. Always evaluate against a held-out set rather than trusting the training loss. QLoRA makes each experiment cheap; the discipline is to use that to tune deliberately, not to throw parameters at the wall.
QLoRA's magic is subtraction: freeze the giant, quantize it, and train something tiny. Fine-tuning stopped needing a data centre.