Paper of the day, 2026-10-26

LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models

Listen to the episode, or read on. The paper: arXiv:2304.01933.

Four ways to bolt a small part onto a frozen giant

LLM-Adapters (arXiv:2304.01933) holds a 13-billion-parameter model completely still and asks one practical question: where should the few trainable weights go, and how many do you need? Teams fine-tune open models every week, usually reaching for whichever adapter they used last time, so a side-by-side test of the four common choices is still worth an engineer's coffee break. The headline results look great. The fine print matters more.

The problem in one picture

Drawing made for wikillm on 2026-10-26 by the AI editor. It is not taken from the paper.

How it works

Think of the base model as a finished engine bolted to a test stand. You can't rebuild it, but you can fit small parts to it. The paper tests four places to fit them.

Prefix-tuning adds a few learned key and value vectors inside every attention layer, as if the model always saw some extra invisible tokens before your prompt. LoRA leaves a weight matrix alone and adds a learned correction next to it. The correction is the product of two thin matrices, so it can only change the layer along a few directions. Series adapters put a small bottleneck right after a sublayer: shrink, apply a non-linearity, expand, add back. Parallel adapters use the same bottleneck but run it beside the sublayer. It reads the same input, and its output is added to the sublayer's output.

The authors tried all four on LLaMA-7B and 13B, BLOOMz-7B and GPT-J-6B. For training data they built two sets. Math10K has about 10K word problems with step-by-step solutions that ChatGPT wrote; answers with wrong final results were dropped. Commonsense170K has about 170K examples taken from the training splits of eight commonsense benchmarks. Each task family gets one model. Training runs three epochs at batch size 16, with learning rate 3e-4 for the adapters and LoRA and 3e-2 for prefix-tuning.

Then comes the placement study, the part with the most to tinker with. On LLaMA-7B with math data, the best LoRA setup covered both attention and MLP layers at rank 32. Both bottleneck adapters did best on the MLP layers with a bottleneck of 256. Prefix-tuning did best with only 10 virtual tokens. The optimizer, warmup and non-linearity are not stated in the paper. The card finds them only in the released code.

Try it yourself

LoRA bets that the change fine-tuning needs is nearly low-rank. You can test that bet on a toy layer in a few minutes. The script below builds a frozen weight and an "ideal" update with a few strong directions and a long faint tail. Then, for each rank, it computes the best possible LoRA pair (B, A) for that update and measures how much of the update is still missed. No training happens here. This is the ceiling: no LoRA of that rank can do better.

Python

import numpy as np
import matplotlib.pyplot as plt

rng = np.random.default_rng(0)
d = 256                                        # width of our toy layer
W0 = rng.normal(0, d ** -0.5, (d, d))          # frozen weight
U, _ = np.linalg.qr(rng.normal(size=(d, d)))   # the "ideal" update dW:
V, _ = np.linalg.qr(rng.normal(size=(d, d)))   # a few strong directions,
power = 1.0 / (1 + np.arange(d)) ** 1.5        # then a long faint tail
dW = (U * power) @ V.T
H = rng.normal(size=(512, d))                  # a batch of layer inputs
want = H @ dW                                  # what fine-tuning should add

Ud, s, Vt = np.linalg.svd(dW)
ranks = [1, 2, 4, 8, 16, 32, 64, 128]
errs, shares = [], []
for r in ranks:
    B, A = Ud[:, :r] * s[:r], Vt[:r]           # d x r and r x d
    out = H @ W0 + H @ B @ A                   # frozen path + low-rank path
    errs.append(np.linalg.norm(out - H @ W0 - want) / np.linalg.norm(want))
    shares.append(2 * d * r / d ** 2)
    print(f"rank {r:3d}: missed update {errs[-1]:.3f}, trainable share {shares[-1]:.1%}")

fig, ax = plt.subplots(1, 2, figsize=(9, 3.5))
ax[0].plot(ranks, errs, "o-"); ax[0].set_xscale("log", base=2)
ax[0].set(xlabel="LoRA rank r", ylabel="share of update missed", title="Best possible fit")
ax[1].plot(shares, errs, "o-"); ax[1].set_xscale("log")
ax[1].set(xlabel="trainable share of a full d x d matrix", title="What you pay")
plt.tight_layout(); plt.show()

What to expect. The missed share falls fast over the first few ranks and then levels off. Each doubling of rank doubles the cost but buys less and less. Look at the last printed line: at the top rank, the two thin matrices together have as many entries as the full matrix, so LoRA has stopped saving anything. The paper's own rank sweep (4, 8, 16, 32) sits in the steep, cheap part of that curve.

Explore. Change the exponent 1.5 to 0.3. Now the update is spread over many directions, and low rank stops being a free lunch. Your toy has just shown when LoRA's bet loses.

What the authors report, and what they leave open

All numbers come from the authors. On commonsense, LLaMA-13B with a parallel adapter averages 81.5 over eight tasks. LoRA gets 80.5, the series adapter 79.5 and ChatGPT 77.0. The gap is 4.5 points, not 4.5 percent. On math, GPT-3.5 averages 70.4 over six sets and LLaMA-13B with LoRA averages 65.4, about 92.9% of it (the paper rounds this a little differently). GSM8K is where the gap is widest: 47.5 against 56.4. Placement matters at small scale too: on LLaMA-7B the parallel adapter scores 61.7 on the MLP layers and 59.3 on attention. More prefix tokens did not help: 10 tokens gave 42.0 and 30 tokens gave 33.1.

Now the caveats, and they are large ones:

Training and test data overlap. Math10K includes MAWPS, and three of the reported math test sets are part of MAWPS. After switching to the MAWPS test set, the repository's updated table shows GPT-3.5 ahead: 87.4 against 83.6, and 51.1 against 63.2 on average over four sets.
Settings were picked on the test sets. The placement and size choices were made on the same math sets used in the final results, and no validation split is described.
The baselines are not like-for-like. The adapters trained on the commonsense training splits, while the baselines ran zero- or few-shot.
Not stated in the paper: seeds, number of runs, trainable parameter counts, memory, training time and decoding settings. A one-point gap between parallel and LoRA tells you very little.

Ask yourself

If the adapter trains on the same benchmarks it is tested on, what is the comparison with a zero-shot ChatGPT actually measuring?
Parallel adapters won on commonsense and LoRA won on math. Is that a real difference between the methods, or noise from single runs?
In your own stack, would you pick an adapter by accuracy alone, or would merging LoRA into the weights at inference outweigh a point or two?