Paper · 2026-10-06 · cs.LG
Are Parameter-Efficient Fine-tuning Methods Really Different?
Compares LoRA, DoRA, PiSSA, MiLoRA, OFT and HRA after per-method learning-rate tuning on Qwen2.5-7B, Llama-3.1-8B and FLUX.2-klein. LoRA-family updates barely shift pretrained singular values or hyperspherical energy, and restoring the singular values keeps task scores. LoRA forgets least; restoring the dominant singular components cuts forgetting most.
Parameter-efficient fine-tuningLow-rank adaptation (LoRA and variants)Catastrophic forgettingText-to-image personalizationpeftloraorthogonal-finetuningsingular-value-decompositioncatastrophic-forgetting
arXivPDFCodeMarkdown15 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Parameter-efficient fine-tuning (PEFT) adapts a large pretrained model by training a small add-on instead of all of its weights. Many variants exist, and each paper usually presents its own as the better idea. This paper runs six of them under matched conditions and asks two questions: do they really change the model in different ways, and which differences matter?
The first answer is that the changes to the model's internal structure are small. The measured shift in the pretrained geometry is small for all six methods, and it is modest for the LoRA-style ones, even though only some of them are designed to keep it intact. Putting back the original strengths of the weight matrices, while keeping the newly learned directions, leaves the task score largely unchanged. The methods differ more in how much of the original behaviour they lose. LoRA loses the least while staying competitive on the task. PiSSA usually loses the most. Restoring the strongest directions of the weights reduces forgetting the most in every method tested, though it can hurt the task score.
TL;DR
Six PEFT methods (LoRA, DoRA, PiSSA, MiLoRA, OFT, HRA) were trained with the same data and a learning-rate search, on two language backbones for maths and code, and on one diffusion backbone for personalisation, with three seeds. Scores often end up similar, but LoRA keeps forgetting lowest most consistently. Geometric preservation does not predict retention. Restoring dominant singular components reduces forgetting most; the effect on the task varies. Results are author-reported.
Problem
Published PEFT comparisons often use one learning rate, a single seed, or different settings per method, so differences can come from tuning rather than from the method §1 §2. A review of 64 method papers in the Hugging Face PEFT library found that only 25 document all applicable comparison conditions together, and only 9 document repeated training with uncertainty attributable to training §2.
OFT and HRA are motivated by keeping pretrained geometry, which is claimed to help generalisation and retention §1. The open questions are whether LoRA-family methods depart much from that geometry in practice and whether tighter preservation improves adaptation or retention §1. The paper's study is limited to the six methods, two language backbones and one diffusion backbone, under the protocol in §3 and app. A.
Method
The study compares six parameterisations of an adapted weight matrix W_0 \in \mathbb{R}^{m \times n} §3 tab. 1:
- LoRA: W_0 + sBA, with a zero initial update and s = \alpha / r tab. 1.
- DoRA: a learned per-row magnitude times the row-normalised LoRA-updated matrix tab. 1.
- PiSSA: the frozen residual W_{\mathrm{res}} plus a LoRA term initialised from the leading singular components of W_0 tab. 1 §3.
- MiLoRA: as PiSSA, but initialised from the trailing singular components tab. 1 §3.
- OFT: W_0 R with R block-diagonal and each block a Cayley transform of a skew-symmetric matrix; the implementation swaps the exact matrix inverse for a short series, so the result is only approximately orthogonal §3 tab. 1.
- HRA: W_0 R with R a product of Householder reflections I - 2 u_i u_i^{\top} with unit vectors u_i §3 tab. 1.
Geometry is measured with relative weight change d_W and relative spectral change d_S §4. Hyperspherical energy (HE) is measured on rows of each matrix app. C.
Geometric preservation. The distance from the adapted matrix to the two-sided orthogonal orbit of W_0 equals the singular-value change, and the restored matrix attains it §4 eq. 1. For a small first-order update E, the singular-value derivative is the diagonal of U_0^{\top} E V_0, and the tangent part of E changes only the bases §4 eq. 2. For an isotropic update with unit Frobenius norm, the expected squared singular-value displacement is 1/\max(m,n), so the root-mean-square ratio d_S/d_W tends to 1/\sqrt{\max(m,n)} §4 eq. 3. The paper extends this to uniformly oriented low-rank updates §4 app. E.
Spectrum restoration. Keep the trained singular vectors and put back the pretrained singular values:
W_R = U_* \operatorname{diag}(s_0) V_*^{\top} \qquad [\text{§4}]
Component groups. For a set B of singular indices, restore the pretrained vectors inside B and keep the adapted vectors outside it eq. 4. Groups are dominant, intermediate and trailing, which are approximately thirds of the spectrum adjusted to nearby spectral gaps §6 app. C app. D.
Hyperspherical energy. For a matrix W with rows w_i and \hat{w}_i = w_i / \lVert w_i \rVert_2 app. C:
H(W) = \sum_{i \neq j} \frac{1}{\lVert \hat{w}_i - \hat{w}_j \rVert_2} \qquad \delta_H = \frac{\lvert H(W_*) - H(W_0) \rvert}{H(W_0)}
Language likelihood (retention). For documents j with valid next-token positions T_j app. A:
\mathrm{NLL}(\Theta) = -\frac{1}{\sum_j \lvert T_j \rvert} \sum_j \sum_{t \in T_j} \log p_\Theta(x_{j,t} \mid x_{j,<t})
Forgetting is \Delta\mathrm{NLL} = \mathrm{NLL}_* - \mathrm{NLL}_0 on the 200-document FineWiki sample §3 app. A.
Diffusion metrics. With normalised DINOv2 features f, CLIP image and text features g and h, and paired base and reference images app. A:
\mathrm{DINO} = \frac{1}{J} \sum_j \cos\bigl(f(I_j^\Theta), f(I_j^{\mathrm{ref}})\bigr)
D_{\mathrm{image}} = \frac{1}{J} \sum_j \frac{1 - \cos\bigl(f(I_j^\Theta), f(I_j^0)\bigr)}{2}
\mathrm{CLIP} = \frac{1}{J} \sum_j \cos\bigl(g(I_j^\Theta), h(c_j)\bigr)
The COCO flow loss for fixed noisy states, with d_j velocity coordinates, K time buckets and rectified-flow target u app. A:
\mathcal{L}_{\mathrm{flow}}(\Theta) = \frac{1}{NK} \sum_{j,k} \frac{w_{j,k}}{d_j} \lVert v_\Theta(y_{j,k}) - u_{j,k} \rVert_2^2
Spectrum restoration of PiSSA and MiLoRA starts each layer from an SVD of the pretrained weight. The code keeps the top (PiSSA) or bottom (MiLoRA) r singular triplets, splits the scaled singular values between B and A, and subtracts the adapter product from the base weight. The code's check is \lVert W_{\mathrm{res}} + sBA - W_0 \rVert / \lVert W_0 \rVert \le 10^{-2} spectral_lora.py.
# Spectral restoration of one weight matrix [§4] [eq. 1]
# Input: W0 (m, n) pretrained; W_star (m, n) adapted
# Output: W_R (m, n) with adapted bases and pretrained singular values
def restore_spectrum(W0, W_star):
# Thin SVD of both matrices, singular values in decreasing order [§4]
U0, s0, V0h = svd(W0)
U_star, s_star, V_starh = svd(W_star)
# Keep the adapted bases, put back the pretrained spectrum [§4]
W_R = U_star @ diag(s0) @ V_starh
return W_R# Restore one spectral band, keep adapted vectors elsewhere [§6] [eq. 4]
# Input: band (set of indices), W0 (m, n), W_star (m, n)
# Output: W_band (m, n)
def restore_band(band, W0, W_star):
# Pretrained and adapted singular triplets [§6]
U0, s0, V0h = svd(W0)
U_star, s_star, V_starh = svd(W_star)
W_band = zeros_like(W0)
for i in range(len(s0)):
# Pretrained vectors inside the band, adapted vectors outside it [eq. 4]
if i in band:
W_band += s0[i] * outer(U0[:, i], V0h[i, :])
else:
W_band += s0[i] * outer(U_star[:, i], V_starh[i, :])
return W_band# PiSSA (principal) or MiLoRA (minor) initialisation of one linear layer
# [§3] [tab. 1] [code: loraoft/spectral_lora.py#L153-L164]
# Input: W0 (m, n); rank r; scaling s = alpha / r
# Output: B (m, r), A (r, n), W_res (m, n) with W_res + s * B @ A ~ W0
def spectral_init(W0, rank, scaling, method):
# Thin SVD of the pretrained weight [code: loraoft/spectral_lora.py#L153]
U, S, Vh = svd(W0)
# PiSSA takes the leading components, MiLoRA the trailing ones [§3]
if method == "pissa":
idx = slice(0, rank)
else:
idx = slice(-rank, None)
# Split the scaled singular values evenly between B and A [code: #L158]
root = sqrt(S[idx] / scaling)
B = U[:, idx] * root
A = root[:, None] * Vh[idx, :]
# Frozen residual keeps the starting model unchanged [§3] [code: #L164]
W_res = W0 - scaling * (B @ A)
return B, A, W_resTraining protocol, language. Qwen2.5-7B and Llama-3.1-8B are trained on 20,000 MetaMath examples for mathematics and 20,000 CodeFeedback examples for coding, each for one epoch with the same seven projections per layer §3 app. A app. B. Methods share data order per seed and approximately matched parameter budgets §3. Learning rates are picked on development data (development accuracy on GSM8K for maths, held-out response NLL for coding); the test score is then reported §3 app. A app. B.
Training protocol, diffusion. FLUX.2-klein-base-4B is personalised to one benchmark cat, with 14 training, 2 validation and 4 test images, and to 28 DreamBooth-style objects with 3 training, 1 validation and 1 or 2 test images app. A. Retention is measured on general prompts and COCO image–caption pairs app. A.
Hyperparameters
Language (mathematics and coding), from app. A and app. B unless noted:
| Name | Value | Where | Notes |
|---|---|---|---|
| Train set | 20,000 MetaMath examples, one epoch | app. A | train.py also removes rewrites of 1,000 dev problems; not described in the paper |
| Optimiser | AdamW, weight decay 0 | app. A | train.py |
| Microbatch × accumulation | 2 × 16 (effective 32) | app. A | train.py |
| Updates | 625 | app. A | |
| Max sequence length | 768 (maths); 1,024 (coding) | app. A app. B | |
| LR schedule | cosine, 3% warmup | app. A | train.py |
| Gradient clipping | global norm 1 | app. A | train.py train.py |
| LR grid | {1, 3, 10, 30, 100} × 1e-5 | §3 app. A | grid.py |
| Seeds | 13, 37, 73 | app. A | grid.py |
| LoRA-family rank | 7 / 14 / 28 (Qwen); 7 / 15 / 30 (Llama) | app. A | grid.py |
| LoRA α; dropout | α = 2r; dropout 0 | app. A | methods.py |
| OFT block size | 32 / 64 / 128, unshared | app. A | Cayley–Neumann with 5 terms methods.py |
| HRA vectors | 16 / 32 / 64 | app. A | No Gram–Schmidt, identity initialisation methods.py |
| Adapted modules | q, k, v, o, up, gate, down | app. A | 196 matrices (Qwen), 224 (Llama); methods.py |
| Learning-rate choice | mean development accuracy, lower LR on ties | app. A | choose_lr.py |
| Test sets | GSM8K 1,319 items; HumanEval+ and MBPP+ | app. A app. B | Coding: one capacity per method, rank 7 for LoRA-family, OFT 32, HRA 16 |
| Retention sample | FineWiki-200, 768-token contexts | app. A | Token-pooled NLL |
Diffusion (FLUX.2-klein-base-4B), from app. A unless noted:
| Name | Value | Where | Notes |
|---|---|---|---|
| Optimiser | AdamW, batch 2, weight decay 1e-4 | app. A | default_training_params.json |
| Gradient clipping | 1 | app. A | default_training_params.json |
| LR schedule | constant | app. A | default_training_params.json (lr_scheduler: null) |
| Updates | 750; validation at step 700 | app. A | default_training_params.json |
| LR grid | {0.5, 1, 3, 5, 10, 30, 50, 100} × 1e-5 (cat); {3, 5, 10} × 1e-5 (28 objects) | app. A | grid.py |
| Seeds | 0, 1, 2 | app. A | |
| Cat capacity pairs | LoRA/DoRA/PiSSA/MiLoRA rank 4/8/16/32; OFT block 32/64/128/256; HRA 16/32/62/124 vectors | app. A | grid.py |
| LoRA α | α = r | app. A | grid.py |
| Resolution; sampling | 512; 20 inference steps; guidance 3.5 | app. A | default_training_params.json |
| Feature encoder | DINOv2 (size not stated in paper) | app. A | Code uses facebook/dinov2-base default_training_params.json |
Results
Author-reported. Development-selected learning rate, held-out test scores, means over three seeds. Standard deviations are in tab. 2 and tab. 3 and are omitted here.
Mathematics, GSM8K test accuracy (%) tab. 2a. S/M/L = smallest to largest capacity for each model.
| Method | Qwen S | Qwen M | Qwen L | Llama S | Llama M | Llama L |
|---|---|---|---|---|---|---|
| LoRA | 83.07 | 83.22 | 83.65 | 70.84 | 71.65 | 71.90 |
| OFT | 83.35 | 82.92 | 83.50 | 71.42 | 71.34 | 71.75 |
| DoRA | 83.27 | 82.64 | 83.12 | 69.95 | 70.91 | 71.97 |
| PiSSA | 83.85 | 83.67 | 83.17 | 70.33 | 69.98 | 70.26 |
| MiLoRA | 83.35 | 83.67 | 83.73 | 70.43 | 71.06 | 70.68 |
| HRA | 83.60 | 83.65 | 83.22 | 67.63 | 70.31 | 71.04 |
Selected learning rates, Qwen S/M/L: LoRA 1e-3, 1e-5, 3e-5; OFT 3e-4, 3e-4, 1e-4; DoRA 3e-4, 3e-4, 1e-4; PiSSA 3e-5, 3e-5, 3e-5; MiLoRA 1e-5, 1e-5, 3e-5; HRA 1e-4, 1e-4, 1e-5. Llama S/M/L: LoRA 3e-4 throughout; OFT 3e-4, 3e-4, 1e-4; DoRA 3e-4 throughout; PiSSA 1e-4, 1e-4, 3e-5; MiLoRA 3e-4 throughout; HRA 1e-3 throughout. tab. 2 app. A
Coding, test pass@1 (%) and forgetting tab. 2b. One capacity per method. HE+ is HumanEval+, MBPP+ is MBPP+. Forgetting is ΔNLL × 10³ (lower is better).
| Method | Qwen HE+ | Qwen MBPP+ | Qwen ΔNLL | Llama HE+ | Llama MBPP+ | Llama ΔNLL |
|---|---|---|---|---|---|---|
| Base | 51.22 | 64.02 | 0 | 31.71 | 50.79 | 0 |
| LoRA | 52.24 | 64.29 | 1.60 | 36.99 | 54.76 | 10.73 |
| OFT | 51.83 | 64.55 | 2.23 | 39.02 | 54.41 | 25.01 |
| DoRA | 52.64 | 64.37 | 1.89 | 37.80 | 56.08 | 13.51 |
| PiSSA | 52.44 | 63.93 | 4.25 | 40.45 | 53.88 | 43.00 |
| MiLoRA | 51.02 | 64.02 | 3.11 | 37.80 | 54.06 | 22.61 |
| HRA | 51.83 | 64.46 | 2.92 | 35.57 | 55.64 | 15.09 |
Selected learning rates: Qwen LoRA 1e-4, OFT 1e-4, DoRA 1e-4, PiSSA 1e-5, MiLoRA 1e-4, HRA 1e-3; Llama LoRA 3e-4, OFT 3e-4, DoRA 3e-4, PiSSA 1e-4, MiLoRA 3e-4, HRA 1e-3. tab. 2b app. B
Diffusion, cat personalisation tab. 3a. Each cell is DINO ↑ / general image drift ↓ / CLIP ↑, all ×100. Rows are the four approximately matched budgets. The HRA column uses 16, 32, 62 and 124 Householder vectors in the four rows.
| Budget | LoRA | OFT | DoRA | PiSSA | MiLoRA | HRA |
|---|---|---|---|---|---|---|
| r4 / b32 | 68.90 / 14.23 / 27.00 | 67.43 / 27.66 / 24.88 | 69.92 / 16.93 / 26.81 | 68.29 / 18.63 / 27.80 | 66.51 / 15.89 / 26.94 | 68.40 / 14.83 / 26.95 |
| r8 / b64 | 68.48 / 14.38 / 28.09 | 68.90 / 15.03 / 27.88 | 68.77 / 14.77 / 27.40 | 69.41 / 18.32 / 28.04 | 65.87 / 14.58 / 28.25 | 70.10 / 13.47 / 26.63 |
| r16 / b128 | 65.78 / 12.87 / 27.99 | 63.11 / 30.83 / 23.23 | 66.38 / 13.83 / 28.27 | 69.98 / 19.69 / 26.83 | 65.14 / 15.38 / 27.07 | 50.89 / 32.82 / 21.10 |
| r32 / b256 | 67.63 / 14.98 / 28.78 | 59.54 / 43.69 / 16.51 | 70.97 / 15.98 / 27.34 | 68.92 / 26.14 / 26.79 | 65.96 / 15.44 / 27.69 | 63.63 / 32.82 / 23.27 |
Selected learning rates per budget row (r4 / r8 / r16 / r32): LoRA 3e-4 / 1e-4 / 1e-4 / 5e-5; OFT 3e-4 / 5e-5 / 1e-4 / 1e-4; DoRA 5e-4 / 3e-4 / 5e-5 / 1e-4; PiSSA 1e-4 / 5e-5 / 5e-5 / 5e-5; MiLoRA 1e-4 / 5e-5 / 1e-4 / 1e-4; HRA 5e-4 / 3e-4 / 5e-4 / 3e-4. tab. 3a
28 objects tab. 3b. Four capacity pairs, LoRA against OFT, averaged over objects and seeds. Test DINO, COCO flow loss (lower is better), general drift, general CLIP.
| Adapter | Test DINO | COCO flow loss | General drift | General CLIP |
|---|---|---|---|---|
| LoRA r4 | 72.95 | 0.8877 | 12.11 | 25.63 |
| OFT b32 | 73.44 | 0.8588 | 17.29 | 23.99 |
| LoRA r8 | 73.94 | 0.8739 | 11.38 | 25.74 |
| OFT b64 | 73.60 | 0.8596 | 19.21 | 23.41 |
| LoRA r16 | 71.88 | 0.8687 | 10.67 | 25.98 |
| OFT b128 | 73.39 | 0.9692 | 43.59 | 12.24 |
| LoRA r32 | 72.84 | 1.0164 | 16.10 | 24.62 |
| OFT b256 | 74.13 | 0.9886 | 44.91 | 11.74 |
Selected learning rates: LoRA r4 1e-4, r8 5e-5, r16 3e-5, r32 1e-4; OFT b32 1e-4, b64 5e-5, b128 1e-4, b256 5e-5. tab. 3b app. A
Geometry §4 app. C.
- Spectral displacement d_S/d_W at the largest capacities is about 11% for Llama MiLoRA and 3–4% for LoRA, DoRA and PiSSA. It is 5.8% for FLUX PiSSA and 8.7% for FLUX MiLoRA §4.
- Mean relative change of \delta_H is below 0.01% for every checkpoint; the largest is 0.00980% §4 app. C.
- At the largest capacities, all restoration points lie within one percentage point of what the paper calls the reconstruction reference fig. 3 §4. The text does not define that reference further.
- Llama LoRA at the largest capacity has an NLL increase of 0.0487 and PiSSA 0.0646, with PiSSA higher in every seed. The text gives these without units §4.
Spectral components §6 fig. 5 app. D.
- Restoring the top components yields the biggest average drop in NLL or image drift for every method, at the largest capacities fig. 5 §6.
- On Llama, the restoration cuts LoRA and DoRA accuracy by roughly 27 points. PiSSA loses almost all recorded accuracy on both models. OFT improves on average with large seed variation §6.
- Keeping only PiSSA's adapted top components holds its language accuracy to within one point of the full restored model; the other methods show no consistent pattern §6.
Overall. LoRA shows the smallest NLL increase in four of the six maths settings, and on both coding backbones, with competitive scores §5. PiSSA has the largest NLL increase on both coding backbones and in half the maths settings §5. In diffusion, LoRA's mean general drift is lowest at three of four budgets and second-lowest at the fourth §5. The authors report that OFT and HRA show no steady edge, in performance or in retention, over the LoRA-family methods §5.
Paper vs code
Official repository: https://github.com/Kuaaannn/PEFT_methods, at commit
14424dabd0f9d0ebfa4a0b5c3795b1ad04e3d616 (branch main, shallow clone). The
link comes from the paper's abstract, so the owner rule is met. Code paths
below are relative to the repository root and shortened to the shortest unique
suffix.
Matched:
- Language grids: the learning rates, seeds, rank and block-size grids, and the 2 × 16 batch, 625 updates, cosine schedule, 3% warmup, clipping 1 and zero weight decay match §3 app. A grid.py train.py train.py.
- Method settings: LoRA α = 2r and zero dropout methods.py; OFT with five Cayley–Neumann terms and unshared blocks methods.py; HRA without Gram–Schmidt methods.py.
- PiSSA and MiLoRA follow the definition in tab. 1 spectral_lora.py.
- Learning-rate choice: lowest LR on ties, development data only choose_lr.py.
- FLUX: batch 2, 750 updates, constant LR, weight decay 1e-4, clipping 1 and validation at step 700 default_training_params.json app. A.
Differs or not stated in paper:
- Training data: the code removes MetaMathQA rewrites of 1,000 development
problems from training (
--group-holdout 1000) train.py. The paper does not describe this, so the development set and the training set do not overlap by construction in the code, but the paper does not say so. - Drift prompts: app. A describes two five-prompt banks for general and class
drift. The evaluator uses one list of ten
drift_image_promptsevaluate.py. I did not find the split in the evaluator I read, so treat the bank definition as unverified. - Dev subset: during training the code scores 300 development items; the full set is scored after training for selection train.py. The paper says the 1,000-item panel is used for selection app. A.
- Repository licence: there is no LICENSE file.
THIRD_PARTY_NOTICES.mdsays the package assigns no blanket licence to original code, and the upstream PEFT snapshots are Apache-2.0. Do not copy code from this repository.
Not checked: the coding harness, the evaluation pipeline, the 28-object harness and the geometry proofs. I did not re-run any experiment.
Pitfalls
- Spectral prediction is conditional. The 1/\sqrt{\max(m,n)} result holds for isotropic updates eq. 3. Trained updates are not isotropic, and the measured 3–11% values §4 are the actual evidence. Do not treat the formula as a bound on a trained update.
- The within-one-point restoration result is narrow. §4 states it for the largest capacities at the selected learning rates. Smaller capacities are in app. C. The "reconstruction" reference is not defined in §4.
- Band boundaries have no numeric rule in the text. The paper says
approximate thirds adjusted to spectral gaps §6 app. C app. D. The code
places each boundary at the largest relative gap within a window of 20% of a
band's width around the nominal third ops.py. A
separate equal-thirds split used by other operators is
np.array_splitops.py. Use the gap-adjusted rule for the dominant, intermediate and trailing results. - The 10/14 DoRA claim is not reproducible from the tables. §5 says DoRA beats LoRA on mean task score in 10 of 14 comparisons, including small Qwen on GSM8K. In tab. 2a the small-Qwen best mean is PiSSA (83.85) and DoRA is 83.27. On coding, DoRA is top on Qwen HE+ and Llama MBPP+ only, OFT is top on Qwen MBPP+ (64.55 against DoRA's 64.37), and PiSSA is top on Llama HE+. The paper does not define "mean task score", so the count depends on an aggregation I could not find.
- "Small budget" for HRA refers to the second row. §5 says HRA has the highest DINO and lowest drift at the small budget. In tab. 3a that is the r8/b64 row. At r4/b32 (the smallest), DoRA is top on DINO and LoRA is lowest on drift.
- Retention numbers have mixed units. Tables use ΔNLL × 10³, while the text quotes 0.0646 and 0.0487 without units §4. Check the unit before comparing the two.
- OFT is only approximately orthogonal. The implementation uses five terms of a series in place of the exact inverse, so the transform is approximately orthogonal. The paper's functional results use the native implementation. Reproduce the same series truncation, not the exact inverse §3 app. C.
- HRA needs an even count in the code. The implementation rejects an odd number of vectors for identity initialisation methods.py. The paper's unit-norm convention for each u_i is in §3.
- Coding and maths use different sequence lengths. Maths uses 768 and coding 1,024 app. A app. B. Do not share a configuration between them.
- Compute. The full grids cover 540 maths, 180 coding, 576 cat and 2,016 object configurations; an HRA extension adds 18 (repository README). Training used A100 GPUs, one GPU per run app. A.
- Results are one setting. §2 and app. F show that few of the 64 eligible PEFT papers document all the controls together. Read the tables as results for these two backbones and this protocol, not as a general ranking of the methods.
Lineage
- Builds on: LoRA (Hu et al.), DoRA (Liu et al.), PiSSA (Meng et al.), MiLoRA (Wang et al.), OFT (Qiu et al.), OFTv2 (Qiu et al., Cayley–Neumann approximation), HRA (Yuan et al.). The paper's bibliography has no arXiv ids for these, so they are named but not linked.
- Compared against: the six methods above; the learning-rate and rescaling study of LoRA by Schulman et al. and the survey of learning-rate reporting by Lee et al. §1.
- Superseded by: not stated.
Source and license
Adapted from “Are Parameter-Efficient Fine-tuning Methods Really Different?” by Yikuan Li, Pinyan Lu, Fanghui Liu (arXiv:2610.09122), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Kuaaannn/PEFT_methods at commit 14424dabd0f9 (none (no LICENSE file; THIRD_PARTY_NOTICES.md assigns no blanket license)). Any quoted code is shown under that license.