QLoRA: Efficient Finetuning of Quantized LLMs

In plain words

Fine-tuning adapts a pre-trained language model to a new task or style. Doing it the classic way means updating every weight, which for a 65-billion-parameter model needs hundreds of gigabytes of GPU memory, far beyond a single card.

LoRA already reduced the number of weights you train to a tiny add-on. But the frozen base model still has to sit in memory at full 16-bit precision. QLoRA shrinks that frozen base to 4 bits per weight, using a number format (NF4) tuned to how neural network weights are actually distributed. It even compresses the small bookkeeping numbers that the 4-bit format needs. During training, each layer is temporarily expanded back to 16 bits for the computation, and only the small add-on is updated.

The result: a 65B model can be fine-tuned on one 48 GB GPU, and the authors find no quality loss compared with 16-bit fine-tuning, provided the add-ons are attached to every linear layer. Their Guanaco chatbot, trained this way, came close to ChatGPT on the benchmark they used.

TL;DR

QLoRA is LoRA on top of a frozen base model stored in 4 bits. Weights are quantized blockwise (block size 64) to a new 4-bit NormalFloat (NF4) data type. The per-block FP32 absmax constants are then quantized again to FP8 (block size 256), a step called "double quantization". Every matmul dequantizes the weight to BF16. Gradients flow through the frozen weights into BF16 LoRA adapters, and only the adapters are updated. Optimizer states sit in NVIDIA unified memory ("paged optimizers"), so occasional spikes from long sequences spill to CPU RAM instead of causing out-of-memory errors. The authors report no loss against 16-bit LoRA or full finetuning, but only when adapters are placed on all linear layers, not only on q/v. Their chatbot family, Guanaco 65B, reaches 99.3% of ChatGPT on the Vicuna benchmark under GPT-4 judging. Official code uses Hugging Face transformers + peft + bitsandbytes.

Problem

Regular 16-bit finetuning of LLaMA 65B needs more than 780 GB of GPU memory §1. Existing LLM quantization methods help at inference time but "break down during training" §1. LoRA cuts the number of trainable parameters, but most training memory goes to activation gradients and the frozen base weights, not to the adapters. For LLaMA 7B (FLAN v2, batch size 1, LoRA ≈0.2% of weights), the LoRA parameters take 26 MB, LoRA input gradients take 567 MB (18 MB per sequence with gradient checkpointing), and the 4-bit base model takes 5,048 MB §2. The aim is to shrink the frozen base model to 4 bits without losing task performance. Assumption: pretrained weights are roughly zero-mean normal, so a fixed normal-quantile code fits them §3 app. F.

Method

Section numbering follows the compiled paper: §1 Introduction, §2 Background, §3 QLoRA Finetuning, §4 QLoRA vs. Standard Finetuning, §5 Pushing the Chatbot State-of-the-art (§5.1 setup, §5.2 evaluation, §5.3 Guanaco), §6 Qualitative Analysis, §7 Related Work, §8 Limitations. Appendices: A (QLoRA vs standard setup; A.1 hyperparameters, A.2 Super-NaturalInstructions), B (chatbot setup; B.1 datasets, B.2 hyperparameters, B.3 ablations, B.4 data size vs quality), C (human evaluation), D (pairwise GPT-4), E (NF4 values), F (normality), G (memory). Equations 1 to 6 are numbered in order: absmax quantization, dequantization, LoRA, NF quantiles, QLoRA linear layer, doubleDequant.

Storage vs compute dtype §3. There is one storage dtype (NF4) and one compute dtype (BF16). Whenever a weight is used, it is dequantized to BF16 and the matmul runs in 16-bit.

Blockwise absmax quantization §2 eq. 1 eq. 2. Flatten the tensor and cut it into n=(b\times h)/B contiguous blocks of size B. Quantize each block independently with its own constant c_i:

\mathbf{X}^{\text{Int8}} = \text{round}\left(\frac{127}{\text{absmax}(\mathbf{X}^{\text{FP32}})}\mathbf{X}^{\text{FP32}}\right) = \text{round}(c^{\text{FP32}}\cdot\mathbf{X}^{\text{FP32}}), \quad \text{dequant}(c,\mathbf{X}^{\text{Int8}}) = \mathbf{X}^{\text{Int8}}/c

For NF4 the "round" step means mapping each normalized value to the nearest of the 16 code values. The paper does not spell this out; it is implemented in bitsandbytes, which the abstract links.

4-bit NormalFloat §3 eq. 4 app. E. NF4 is quantile quantization for a N(0,1) source, rescaled to [-1,1]. Each bin should receive an equal expected number of values. The code values are midpoints of adjacent normal quantiles:

q_i = \tfrac{1}{2}\left(Q_X\!\left(\tfrac{i}{2^k+1}\right) + Q_X\!\left(\tfrac{i+1}{2^k+1}\right)\right)

A symmetric construction would have no exact zero. The paper therefore builds an asymmetric code: it estimates 2^{k-1} quantiles for the negative side and 2^{k-1}+1 for the positive side, merges the two sets and drops the duplicate zero §3. The resulting 16 values are listed exactly in app. E. Use that list rather than re-deriving it (see Pitfalls):

[-1.0, -0.6961928009986877, -0.5250730514526367, -0.39491748809814453,
 -0.28444138169288635, -0.18477343022823334, -0.09105003625154495, 0.0,
 0.07958029955625534, 0.16093020141124725, 0.24611230194568634,
 0.33791524171829224, 0.44070982933044434, 0.5626170039176941,
 0.7229568362236023, 1.0]

Inputs are brought into [-1,1] by per-block absmax rescaling §3.

Double Quantization §3. The first-level constants c_2^{\text{FP32}} (one per 64 weights) are themselves quantized to 8-bit floats with block size 256. This produces c_2^{\text{FP8}} plus second-level constants c_1^{\text{FP32}}. Because the c_2 values are all positive, their mean is subtracted first so that symmetric quantization can be used. Overhead drops from 32/64=0.5 to 8/64+32/(64\cdot256)=0.127 bits per parameter, a saving of 0.373 bits/param, or about 3 GB on a 65B model §1 §3.

Paged optimizers §3. Optimizer states are allocated in NVIDIA unified memory. When the GPU runs out of memory they are evicted to CPU RAM, and they are paged back in for the optimizer step. In code this is optim='paged_adamw_32bit' qlora.py.

QLoRA layer eq. 5 eq. 6.

\mathbf{Y}^{\text{BF16}} = \mathbf{X}^{\text{BF16}}\,\text{doubleDequant}(c_1^{\text{FP32}}, c_2^{\text{k-bit}}, \mathbf{W}^{\text{NF4}}) + \mathbf{X}^{\text{BF16}}\mathbf{L}_1^{\text{BF16}}\mathbf{L}_2^{\text{BF16}}

\text{doubleDequant}(c_1^{\text{FP32}}, c_2^{\text{k-bit}}, \mathbf{W}^{\text{k-bit}}) = \text{dequant}(\text{dequant}(c_1^{\text{FP32}}, c_2^{\text{k-bit}}), \mathbf{W}^{\text{4bit}}) = \mathbf{W}^{\text{BF16}}

Here \mathbf{L}_1\in\mathbb{R}^{h\times r} and \mathbf{L}_2\in\mathbb{R}^{r\times o}. Generic LoRA also carries a scalar s on the adapter path eq. 3. Only \partial E/\partial \mathbf{L}_i is computed. Backprop through the frozen weight uses the dequantized BF16 \mathbf{W}, and no gradient is stored for \mathbf{W} §3.

Algorithm: QLoRA finetuning  [§3] [app. B.2] [code: qlora.py#L310-L405]
Input: pretrained LM, instruction dataset
1. Load the base model with every linear layer quantized: NF4, block size 64, double quant
   (FP8 constants, block size 256), compute dtype BF16 [§3] [code: qlora.py#L310-L328].
2. Prepare for k-bit training with gradient checkpointing enabled [§3] [code: qlora.py#L376-L377].
3. Attach LoRA (r=64, alpha=16) to ALL linear layers of the base model (code excludes lm_head)
   [app. B.2] [code: qlora.py#L248-L259] [code: qlora.py#L384-L394].
4. Cast LoRA modules to bf16 and norm layers to fp32 (code only) [code: qlora.py#L396-L405].
5. Train with cross-entropy, on the response tokens only where instruction and response
   are separable; source tokens get label -100 [§5.1] [app. B.3] [code: qlora.py#L478-L486].
   For OASST1/HH-RLHF, train on the full selected conversation [§5.1].
6. Optimizer: paged AdamW (32-bit), constant LR, max grad norm 0.3, group-by-length batching
   [app. B.2] [code: qlora.py#L198-L211].
Output: LoRA adapter weights (base model stays in 4-bit).

Hyperparameters

Results

Author-reported.

Data types, no finetuning tab. 2 fig. 3. Mean Pile Common Crawl perplexity over OPT, BLOOM, LLaMA and Pythia models from 125M to 13B:

Matching 16-bit finetuning tab. 3 tab. 4.

Chatbot (Guanaco = QLoRA on OASST1) tab. 6 tab. 7. The Vicuna benchmark has 80 prompts. Scores are relative to ChatGPT, judged by GPT-4 and averaged over both presentation orders.

Elo ratings judged by GPT-4 on Vicuna: Guanaco 65B 1022, Guanaco 33B 992, ChatGPT-3.5 966. On the 953-prompt OA benchmark (GPT-4 judge): ChatGPT-3.5 1015, Guanaco 65B 1008 tab. 7. GPT-4 and human system-level rankings agree moderately (Kendall τ=0.43, Spearman r=0.55); example-level agreement is weaker (Fleiss κ=0.25) §5.3. Training only on the target beats training on source+target, 38.6 vs 37.5 mean MMLU (7B, 52k examples) tab. 10. Dataset choice matters more than size: varying size and epochs moves MMLU by 0.0–0.5, while switching dataset moves it by 1.5–8.0 app. B.4 tab. 11. Training time: 33B Guanaco in under 12 h on a 24 GB consumer GPU; 65B in 24 h on one professional GPU §1 §5.3.

Paper vs code

Checked qlora.py and scripts/finetune_guanaco_*.sh at the pinned commit.

• LoRA dropout for 13B. The paper says 0.1 for models up to 13B app. B.2. The 13B Guanaco script uses 0.05 finetune_guanaco_13b.sh, while the 7B script uses 0.1 finetune_guanaco_7b.sh. The lora_dropout default in code is 0.0 qlora.py.
• Warmup. Not mentioned in the paper. The code uses warmup_ratio=0.03, followed by a constant LR qlora.py.
• OASST1 size and formatting. The paper says 9,209 examples app. B.1. The code loads timdettmers/openassistant-guanaco and documents 9,846 examples qlora.py qlora.py. Each conversation becomes input='', output=text qlora.py, so the loss covers the whole conversation, which matches §5.1. The script sets source_max_len 16, where tab. 9 shows "-".
• Unstated dtype handling (code only). LayerNorm/RMSNorm modules are cast to fp32, LoRA modules to bf16, and lm_head/embed_tokens are kept unquantized and without LoRA qlora.py qlora.py.
• Batch size. tab. 9 batch sizes are reached with per-device batch 1 and 16 gradient-accumulation steps in the Guanaco scripts finetune_guanaco_7b.sh.
• Other defaults match the paper: r=64, alpha=16, max grad norm 0.3, constant LR, group-by-length, NF4 + double quant qlora.py qlora.py.

Pitfalls

• Adapt every linear layer. LoRA on q/v only, the original LoRA default, does not reach full-finetuning quality on large models. The number of adapted layers matters far more than r §4 fig. 2 app. A.1.
• Dropout guidance is self-contradictory. app. A.1 says dropout 0.05 helps small models (7B, 13B) but not large ones (33B, 65B). app. B.2 prescribes 0.1 for ≤13B and 0.05 for 33B/65B. The code splits the difference (see above). Treat dropout as something to tune.
• Do not re-derive NF4 from eq. 4. The text says to estimate 2^k+1 quantiles, eq. 4 produces 2^k midpoints, and the asymmetric negative/positive construction is only described in prose. The offsets used at the tails (Q_X(0) and Q_X(1) are infinite) are not stated. Hard-code the 16 values from app. E.
• Notation traps. In the DQ paragraph, c_2 denotes the first-level (per-64-weight) constants and c_1 the second-level ones. Eq. 5–6 write c_2^{\text{k-bit}} although the text says FP8. Eq. 5 drops LoRA's scalar s from eq. 3, while the code applies peft's alpha/r scaling (16/64). The DQ mean subtraction needs the mean stored and added back on dequantization; this is not counted in the 0.127 bits/param figure §3.
• Paged optimizers are unmeasured. The authors give no hard measurements, because paging only happens on rare long-sequence mini-batches. They report the same speed as regular optimizers for 65B on 48 GB at batch size 16 §4. Slow-downs elsewhere are not characterized. 33B does not quite fit in 24 GB without paging app. G.
• Memory numbers assume small activations. The breakdown is for batch size 1, sequence length 512, gradient checkpointing on app. G fig. 6. Longer sequences or larger batches add significant activation-gradient memory.
• Unproven at 33B/65B vs full finetuning. Equivalence to full 16-bit finetuning is shown only up to 3B (11B against 16-bit full FT on SNI only); 7–65B are compared against 16-bit LoRA only §4 §8.
• Dequantization cost. Every forward and backward matmul dequantizes NF4 to BF16 §3, so throughput depends on the bitsandbytes kernels. The paper claims runtime is not degraded §1 but gives no throughput table.
• Chatbot evaluation caveats. GPT-4 favors whichever response is shown first. Average over both orders §5.2. GPT-4 also rates its own outputs higher than humans do (Elo 1348 vs 1176) §6.2. Vicuna-benchmark CIs are wide and overlap §5.3 tab. 6.
• Normality assumption is approximate. 7.5% of LLaMA-7B hidden units fail a Shapiro-Wilk test at 5%, against an expected 5% false-positive rate app. F.
• Response-only loss. Response-only training helps MMLU tab. 10. Its effect on chatbot benchmarks was not evaluated app. B.3.

Lineage

• Builds on: LoRA. Adapters move to all linear layers, and the frozen base is 4-bit NF4 instead of 16-bit. Builds on blockwise k-bit / quantile quantization and 8-bit optimizers from the authors' earlier bitsandbytes work (Dettmers et al. 2022) §2 §3.
• Compared against: 16-bit full finetuning, 16-bit LoRA, Int8/FP4 QLoRA variants tab. 3 tab. 4; Vicuna 13B, Open Assistant 33B (RLHF), ChatGPT, Bard, GPT-4 tab. 6 tab. 7.
• Related: DoRA, a later LoRA variant.