Paper · 2026-11-12 · cs.LG, cs.CL
Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off
LoRA updates often use far less of their configured rank. The authors add an SVD-free Rényi-2 effective-rank penalty, -lambda log R, to the language-modeling loss. With a weight of 0.01 it lowers average Wiki perplexity on three models at each tested rank from 16 upward (author-reported), and training perplexity is mostly unchanged.
Low-rank adaptation (LoRA and variants)Parameter-efficient fine-tuningCatastrophic forgettingContinual learningloraeffective-rankregularizationcatastrophic-forgettingparameter-efficient-fine-tuning
arXivPDFCodeMarkdown16 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
LoRA teaches a language model a new task by training a small add-on instead of all of the model's weights. The size of the add-on is set by a number called the rank. The paper's point is that a higher rank does not automatically give the add-on more useful room. Training tends to put most of the change into a few directions, so the rank on paper is much larger than the room actually used.
The authors measure how evenly the change is spread across directions. They add a term to the training loss that rewards a more even spread. In their tests the model keeps more of its general knowledge, measured as perplexity on general-text samples, while its fit to the new task mostly stays the same or gets better. The gains depend on the setup, and they are reported by the authors themselves.
TL;DR
- Standard LoRA updates have an effective rank far below the configured rank, and the gap grows with the rank §1 §2 fig. 1.
- The main method adds a Rényi-2 effective-rank term to the language-modeling loss. It is computed from the LoRA factors without an SVD and works with any parameterization that can be written as a product of two factors §3.3 app. E.
- On three models in mixed-domain runs, a penalty weight of 0.01 gives lower average Wiki perplexity than unpenalized LoRA at each tested rank from 16 upward. Training perplexity changes by at most 0.02 in all but one comparison §4.4 tab. 9 tab. 10 tab. 11.
- The penalty is a log, not the plain form written in the main equation, and it has a steep gradient near zero update. The authors use a late start in single-task runs app. E app. F §4.5.
- Relevant when you want more retention for a given task loss. Not a substitute for tuning lambda, which the authors say is task-specific § Limitations.
Problem
Standard LoRA is usually set up by choosing a rank r and assuming that the adapter can use up to r directions §2. The authors measure the utilization quotient erank(ΔW)/r and find it well below 1. It also falls as r grows, and it is lowest in the late layers §2 fig. 1. This matters for the learning-forgetting trade-off, because it depends on how the update's energy is spread across directions, not only on the rank §1 §2.
Several remedies exist: parameter-efficient variants (VeRA, SVF, TinyLoRA), norm-constrained LoRA (NB-LoRA), and spectral initializations (PiSSA, LoRA-XS). The authors argue that these mostly move the trade-off rather than improve it, because methods that fit the mixture well also forget more §1 fig. 2 tab. 5.
The paper's definition of effective rank is exact, but its gradient needs an SVD and has singularities when singular values approach zero §3.1. The authors therefore optimize a surrogate that does not need an SVD.
Method
Notation. W0 is the pretrained weight of a target layer and ΔW = W - W0 is
the accumulated LoRA update §2. The paper restricts adaptation to the MLP
down_proj modules in its main experiments §4. The per-module update is
written ΔW = γBA with γ = α/r app. E.
Effective rank. The Roy–Vetterli effective rank is erank(ΔW) = exp(H1(p)), where p_i = σ_i / Σ_j σ_j is the normalized singular-value spectrum §2 eq. 1 eq. 2. It equals 1 for a rank-one update and approaches r for a uniform spectrum §2.
Objective. The goal is min L(W0+ΔW) − λ·Cap(ΔW), with Cap taken as the effective rank §3 eq. 3. The authors develop two routes around it.
Route 1: norm-bounded LoRA (NB-LoRA). Since erank ≥ ‖ΔW‖²_S2 / δ² whenever the operator norm of ΔW is at most δ §3.2, they optimize L − λ‖ΔW‖²_S2/δ² under an operator-norm cap eq. 4. The cap is enforced by the NB-LoRA parameterization, which builds ΔW from a diagonal S with operator norm δ and two semi-orthogonal factors obtained from a Cayley map §3.2. They report that this route raises effective rank but can be hard to optimize §3.2 §4.2.
Route 2 (main method): Rényi-2 penalty. The Rényi-2 effective rank is R(ΔW) = exp(H2(q)), with q_i = σ_i² / Σ_j σ_j² §3. It has the closed form R(ΔW) = ‖ΔW‖⁴_F / Tr((ΔW^T ΔW)²) §3.3. The paper proves rank(ΔW) ≥ erank(ΔW) ≥ R(ΔW) ≥ 1 §3.3, so raising R raises a lower bound on the Roy–Vetterli effective rank. The penalized problem is min L(W0+ΔW) − λR(ΔW) §3.3 eq. 6, where λ may be fixed or scheduled.
Logarithmic form (used in every reported penalty run). The experiments subtract λ·H2 = λ·log R rather than λR app. E app. F. For a nonzero update, the gradient of H2 is the gradient of R divided by R, so the penalty is rescaled per module app. E. The reported λ values refer to this log form app. F.
Factor form, no SVD. Let G = (BᵀB)(AAᵀ), an r×r matrix. Then ‖ΔW‖²_F = γ²Tr(G) and Tr((ΔW^T ΔW)²) = γ⁴Tr(G²). Hence R = Tr(G)²/Tr(G²), and H2 = 2 log Tr(G) − log Tr(G²) app. E. The dense update is never formed during training. Storage is Θ(r²) beyond the adapter, and arithmetic is Θ(r²(m+n) + r³) app. E. γ cancels in R, so the penalty value does not depend on α app. E.
# Rényi-2 entropy of one LoRA module, no SVD [app. E] [§3.3]
# Input: A (r, n), B (m, r), gamma = alpha / r (float32)
# Output: h2, the Rényi-2 entropy of the energy spectrum (scalar)
def renyi2_entropy(A, B, gamma, eps=1e-8):
# Reduced r x r Gram matrices; dense (m, n) update never built [app. E]
gram_b = B.T @ B # (r, r)
gram_a = A @ A.T # (r, r)
G = gram_b @ gram_a # (r, r)
# Traces of G and G^2 give the Frobenius and collision terms [app. E]
trace_g = max(trace(G), eps) # Tr(G), clamped below at eps
trace_g2 = max(sum(G * G.T), eps) # Tr(G^2) = sum_ij G_ij G_ji
# gamma cancels in the ratio, so h2 does not depend on alpha [app. E]
return 2 * log(trace_g) - log(trace_g2)# Penalty weight schedule [app. F]
# Input: step, lam (terminal weight), start (S), ramp (W)
# Output: weight applied to the summed entropy
def penalty_weight(step, lam, start, ramp):
# Weight is 0 before the start step [app. F]
if step < start:
return 0.0
# Linear ramp to lam over ramp steps, then constant [app. F]
progress = min(1.0, (step - start) / ramp)
return lam * progress# Training loss with the penalty [§3.3] [app. F]
# Input: batch, adapters (one per targeted module), step
# Output: scalar loss
def training_loss(batch, adapters, step, lam, start, ramp):
# Cross-entropy on assistant tokens only; prompt tokens are masked [app. F]
lm_loss = assistant_cross_entropy(model(batch), batch)
# Sum of per-module Rényi-2 entropies, log form [app. F]
total_h2 = 0.0
for module in adapters:
total_h2 = total_h2 + renyi2_entropy(module.A, module.B, module.gamma)
weight = penalty_weight(step, lam, start, ramp)
# Subtracting the entropy rewards a flatter energy spectrum [app. F]
return lm_loss - weight * total_h2PiSSA and LoRA-XS. The same penalty applies after exposing effective factors. For LoRA-XS, B = Ũ and A = R Ṽᵀ, with fixed truncated-SVD bases Ũ and Ṽ and a trainable core R app. E. For PiSSA, ΔW = γ(BA − B0A0), so the effective factors are B_eff = [B, B0] and A_eff = [A; −A0], of width 2r app. E. For NB-LoRA, the effective (A, B, γ) come from the Cayley construction app. E.
Scheduling. The penalty can start at a step S and ramp for W steps app. F. Mixed-domain runs use S=0 and W=50 app. F. Single-task runs use S=400 and W=200, with stated exceptions app. F. Section 4.5 shows that the start step trades training fit against retention §4.5 tab. 3.
Diagnostics (not in the training loss). The paper logs erank(ΔW) from a QR factorization of γB and an SVD of the r×n matrix R_qr A app. E. It also reports a spectral alignment analysis: a finetuned direction is aligned if its best energy overlap with a pretrained direction is at least 0.9² §5. A soft permutation coupling is computed with Sinkhorn normalization at β=10, and its normalized row entropy is H_soft §5 eq. 7. These diagnostics are descriptive §5.
Hyperparameters
Default SFT protocol for the 0.5B mixed-domain runs and single-task runs. Values are as stated; "not stated" means the paper does not say.
| Name | Value | Where | Notes |
|---|---|---|---|
| base model (main) | Qwen2.5-0.5B (base, not Instruct) | §4 app. F | Also Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct, Pythia-2.8B, OLMo-2 in the appendix sweeps app. G |
| adapted modules | MLP down_proj layers | §4 app. F | Module list for the Llama sweeps: not stated |
| LoRA rank r | 896 (main); sweep 4, 16, 128, 256, 512 | §4 §4.4 | Qwen2.5-3B baselines use r=2048 app. F |
| alpha | 2r = 1792 | app. F | Qwen2.5-3B baselines α=4096 app. F; α for the 3B and Llama sweeps: not stated |
| scaling gamma | α/r = 2 | app. E app. F | Cancels in the penalty ratio app. E |
| optimizer | AdamW | app. F | |
| learning rate (adapters) | 1e-4 | app. F | Constant after warmup in mixed-domain runs |
| weight decay (adapters) | 0.01 | app. F | |
| warmup | linear, 20 optimizer steps | app. F | Then constant peak rate |
| micro-batch / grad accumulation | 1 / 2 (effective batch 2 packed sequences) | app. F | |
| max sequence length | 512 tokens, packed, dynamic padding | app. F | |
| LoRA dropout | 0 | app. F | |
| gradient clipping | none | app. F | |
| precision | base weights bf16, no AMP, no fp32 master weights; penalty and SVD in float32 | app. F app. E | |
| clamp | Tr(G), Tr(G²) ≥ eps = 1e-8 before log | app. F app. E | |
| penalty lambda | {0, 0.01, 0.1, 1.0}; sweep also includes 0.001 for Llama | §4 app. F tab. 11 | lambda is in the log form |
| penalty schedule (mixed) | S=0, W=50 | app. F | Ramp endpoint convention: see Pitfalls |
| penalty schedule (single task) | S=400, W=200 | app. F | Rank exceptions listed in app. F |
| mixed-domain budget | 1e6 supervised target tokens | §4 app. F | 45% GSM8K, 10% WMT19 cs–en, 45% Aya Hausa |
| mixture seed / shuffle buffer | 0 / 50,000 | app. F | |
| evaluation | every 100 optimizer steps | app. F | Terminal state reported, no early stopping |
| eval batches | ≤20 (mixed), ≤50 (single task) | app. F | Probe slices: 2,000 documents each from OLMo-Mix-1124 wiki and starcoder |
| Train PPL | exp of EMA (β=0.99) of assistant-only train loss | app. F | Smoothed proxy, not validation PPL |
| NB-LoRA delta | 1.0 | app. F | Cayley-based |
| VeRA | initial scale 0.1, seed 0 | app. F | |
| TinyLoRA | one shared scalar, seed 0 | app. F | |
| dense FT (mixed) | lr 3e-5, weight decay 0, micro-batch 4, grad accum 2, 20-step warmup | app. F | Not matched to adapters; see Pitfalls |
| dense FT grid | lr {1e-5, 3e-5, 1e-4}, micro-batch {1, 2, 4}, wd {0, 0.01, 0.1}, schedule {linear, warmup-cosine} | app. F | Selected by lowest Train PPL |
| scheduler ablation | ~500 steps, lambda_1=0.1, W=200, S ∈ {0, 50, 100, 400}, Hausa only | §4.5 tab. 3 app. F | Separate short budget |
| hardware | one NVIDIA A40, 48 GiB host memory | app. F | "unless noted otherwise" |
| alignment threshold | epsilon = 0.9 | §5 | Aligned if max overlap ≥ 0.81 |
| Sinkhorn beta | 10 | §5 | Log clamp 1e-12 |
Results
Author-reported. Setup: Qwen2.5-0.5B base, mixed-domain continual-learning proxy (1e6 supervised target tokens), rank 896 with alpha 1792, unless noted §4.1 app. F. Columns: eRank is higher-is-better; Train, StarCoder, Wiki, GSM8K and WMT are perplexities (PPL), lower is better. Means ± sample SD over n training executions. Single runs have n=1. PiSSA excludes one resumed execution with apparent data replay §4.1.
Mixed-domain, Qwen2.5-0.5B, rank 896 tab. 5 fig. 2
| Method | n | #Params | eRank ↑ | Train ↓ | StarCoder ↓ | Wiki ↓ | GSM8K ↓ | WMT ↓ |
|---|---|---|---|---|---|---|---|---|
| Dense FT | 1 | — | 844.5 | 5.643 | 3.640 | 16.563 | 1.817 | 9.223 |
| Standard LoRA | 3 | 123,863,040 | 360.5 ± 5.2 | 4.773 ± 0.093 | 5.815 ± 0.811 | 19.601 ± 1.056 | 1.929 ± 0.044 | 15.018 ± 0.802 |
| LoRA, λ=0.01 | 3 | 123,863,040 | 668.0 ± 1.4 | 4.381 ± 0.007 | 4.143 ± 0.006 | 15.290 ± 0.067 | 1.836 ± 0.010 | 10.156 ± 0.034 |
| LoRA, λ=0.1 | 3 | 123,863,040 | 887.8 ± 0.3 | 4.525 ± 0.016 | 4.098 ± 0.007 | 14.648 ± 0.046 | 1.832 ± 0.008 | 9.517 ± 0.172 |
| LoRA, λ=1.0 | 1 | 123,863,040 | 895.5 | 4.688 | 4.003 | 14.085 | 1.800 | 8.909 |
| PiSSA | 2 | 123,863,040 | 815.7 ± 8.2 | 4.286 ± 0.003 | 5.162 ± 0.102 | 21.565 ± 0.658 | 1.878 ± 0.002 | 13.160 ± 0.020 |
| PiSSA, λ=0.1 | 1 | 123,863,040 | 892.2 | 4.335 | 4.769 | 18.634 | 1.874 | 12.109 |
| LoRA-XS | 3 | 19,267,584 | 566.8 ± 5.0 | 4.578 ± 0.005 | 5.517 ± 0.050 | 21.431 ± 0.584 | 1.901 ± 0.010 | 14.691 ± 0.158 |
| NB-LoRA (δ=1, no penalty) | 3 | 123,863,040 | 583.2 ± 1.5 | 4.396 ± 0.002 | 4.404 ± 0.028 | 15.705 ± 0.073 | 1.809 ± 0.003 | 10.196 ± 0.107 |
| VeRA | 3 | 43,008 | 632.2 ± 0.0 | 11.468 ± 0.001 | 3.832 ± 0.001 | 13.005 ± 0.006 | 2.373 ± 0.000 | 9.309 ± 0.007 |
| SVF | 3 | 21,504 | 783.6 ± 0.1 | 9.694 ± 0.257 | 3.895 ± 0.002 | 13.186 ± 0.006 | 2.061 ± 0.000 | 8.814 ± 0.006 |
| TinyLoRA | 3 | 1 | 681.5 ± 0.2 | 11.549 ± 0.001 | 3.833 ± 0.000 | 13.006 ± 0.008 | 2.388 ± 0.001 | 9.341 ± 0.003 |
Setup notes from the same table: VeRA, SVF and TinyLoRA underfit, so their low retention is not matched-capacity evidence app. F. The dense FT row is a single run with its own optimizer settings app. F.
Rank sweep, Qwen2.5-0.5B tab. 9 fig. 3. Same mixed-domain protocol; λ=0 is standard LoRA.
| Rank | λ | eRank ↑ | Train ↓ | StarCoder ↓ | Wiki ↓ | GSM8K ↓ | WMT ↓ |
|---|---|---|---|---|---|---|---|
| 4 | 0 | 3.014 ± 0.075 | 6.096 ± 0.009 | 3.926 ± 0.004 | 13.615 ± 0.037 | 1.802 ± 0.001 | 8.370 ± 0.045 |
| 4 | 0.01 | 3.999 ± 0.000 | 6.086 ± 0.013 | 3.932 ± 0.009 | 13.575 ± 0.017 | 1.803 ± 0.001 | 8.425 ± 0.062 |
| 4 | 0.1 | 4.000 ± 0.000 | 6.595 ± 0.012 | 3.942 ± 0.016 | 13.564 ± 0.001 | 1.821 ± 0.001 | 8.403 ± 0.068 |
| 16 | 0 | 9.689 ± 0.077 | 5.419 ± 0.006 | 3.953 ± 0.003 | 13.780 ± 0.020 | 1.783 ± 0.002 | 8.382 ± 0.008 |
| 16 | 0.01 | 15.990 ± 0.000 | 5.391 ± 0.005 | 3.950 ± 0.006 | 13.602 ± 0.031 | 1.780 ± 0.001 | 8.310 ± 0.022 |
| 16 | 0.1 | 15.999 ± 0.000 | 5.717 ± 0.004 | 3.941 ± 0.002 | 13.578 ± 0.013 | 1.795 ± 0.000 | 8.332 ± 0.039 |
| 128 | 0 | 74.494 ± 0.391 | 4.605 ± 0.002 | 4.012 ± 0.017 | 14.202 ± 0.047 | 1.767 ± 0.004 | 8.715 ± 0.079 |
| 128 | 0.01 | 126.714 ± 0.074 | 4.611 ± 0.002 | 3.961 ± 0.012 | 13.865 ± 0.032 | 1.763 ± 0.001 | 8.470 ± 0.028 |
| 128 | 0.1 | 127.951 ± 0.001 | 4.749 ± 0.002 | 3.958 ± 0.010 | 13.863 ± 0.005 | 1.767 ± 0.003 | 8.424 ± 0.046 |
| 256 | 0 | 145.032 ± 0.175 | 4.450 ± 0.001 | 4.067 ± 0.006 | 14.674 ± 0.057 | 1.778 ± 0.004 | 9.406 ± 0.064 |
| 256 | 0.01 | 250.540 ± 0.111 | 4.465 ± 0.003 | 3.992 ± 0.010 | 14.145 ± 0.048 | 1.777 ± 0.003 | 8.742 ± 0.069 |
| 256 | 0.1 | 255.777 ± 0.002 | 4.559 ± 0.001 | 3.986 ± 0.013 | 13.975 ± 0.037 | 1.776 ± 0.001 | 8.626 ± 0.058 |
| 512 | 0 | 265.043 ± 0.619 | 4.375 ± 0.009 | 4.183 ± 0.008 | 15.529 ± 0.129 | 1.806 ± 0.005 | 10.292 ± 0.505 |
| 512 | 0.01 | 473.743 ± 0.910 | 4.310 ± 0.132 | 4.056 ± 0.006 | 14.486 ± 0.120 | 1.794 ± 0.007 | 9.321 ± 0.060 |
| 512 | 0.1 | 510.839 ± 0.011 | 4.469 ± 0.001 | 4.025 ± 0.021 | 14.281 ± 0.061 | 1.802 ± 0.004 | 9.056 ± 0.013 |
Cross-model checks from the sweep tables (Wiki PPL, λ=0 → λ=0.01):
- Qwen2.5-3B-Instruct tab. 10: rank 16, 9.148 → 9.102; rank 256, 9.539 → 9.210; rank 512, 9.708 → 9.185. Rank 16 Train PPL: 4.211 → 4.524 ± 0.227, the one exception to "at most 0.02".
- Llama-3.2-3B-Instruct tab. 11: rank 16, 11.532 → 11.315; rank 512, 12.180 → 11.655.
- Llama-3.2-3B-Instruct at rank 3072 tab. 7: standard LoRA Train 3.984, Wiki 18.547, eRank 985.4; λ=0.01 Train 3.837, Wiki 15.924, eRank 1504.2; λ=0.1 Train 3.679, Wiki 13.327, eRank 2725.1.
Scheduling, Aya Hausa, Qwen2.5-0.5B, λ₁=0.1, W=200 tab. 3 §4.5
| Start step S | eRank ↑ | Train PPL ↓ | StarCoder PPL ↓ | Wiki PPL ↓ |
|---|---|---|---|---|
| standard LoRA (no penalty) | 341.4 | 22.604 | 4.054 | 21.822 |
| 0 | 838.2 | 24.737 | 3.792 | 18.917 |
| 50 | 810.0 | 23.860 | 3.753 | 18.700 |
| 100 | 785.1 | 23.651 | 3.756 | 18.824 |
| 400 | 429.8 | 22.410 | 3.861 | 19.768 |
PiSSA and LoRA-XS on single-task GSM8K, Qwen2.5-0.5B, r=896 tab. 2 §4.3
| Method | eRank ↑ | Train PPL ↓ | StarCoder PPL ↓ | Wiki PPL ↓ |
|---|---|---|---|---|
| PiSSA, λ=0 | 829.0 | 1.871 | 4.323 | 23.173 |
| PiSSA, λ=0.01 | 877.9 | 1.869 | 4.233 | 22.255 |
| PiSSA, λ=0.1 | 894.0 | 1.851 | 4.104 | 20.951 |
| PiSSA, λ=1.0 | 895.7 | 1.825 | 3.934 | 19.706 |
| LoRA-XS, λ=0 | 565.0 | 1.892 | 5.066 | 25.479 |
| LoRA-XS, λ=0.01 | 722.0 | 1.904 | 4.625 | 23.195 |
| LoRA-XS, λ=0.1 | 880.4 | 1.896 | 4.371 | 21.398 |
| LoRA-XS, λ=1.0 | 895.0 | 1.847 | 4.134 | 20.121 |
NB-LoRA on Aya Hausa, Qwen2.5-0.5B, r=896 tab. 1 §4.2. Here λ weights the Frobenius capacity term of eq. 4; the NB-LoRA row uses λ=0.
| Method | eRank ↑ | Train PPL ↓ | StarCoder PPL ↓ | Wiki PPL ↓ |
|---|---|---|---|---|
| Dense FT | 805.5 | 20.977 | 4.243 | 23.393 |
| Standard LoRA | 340.5 | 22.580 | 3.998 | 21.098 |
| NB-LoRA | 504.0 | 23.919 | 4.071 | 20.089 |
| λ=0.01 | 527.5 | 23.729 | 4.024 | 19.815 |
| λ=0.1 | 663.7 | 23.923 | 4.108 | 19.331 |
| λ=1.0 | 814.1 | 24.307 | 4.048 | 19.066 |
Main-text summary numbers from the same experiments: erank for standard LoRA rises from 361 to 668, 888 and 896 at λ=0.01, 0.1 and 1.0 §4.1. The text calls LoRA-XS at λ=1 (Train PPL 1.847) the best Train PPL in its block; across the whole GSM8K table, PiSSA at λ=1 is lower at 1.825 tab. 2.
Paper vs code
No official code: the repository the paper links, https://github.com/tomohiro-sawada/lora-capacity-utilization (linked from the abstract), holds only a LICENSE file at commit e08349187c1e115b763c0a3c25f689946f536eab. The LICENSE is MIT. The repository has no adapter, penalty, training, or configuration code. I checked only this linked repository, and no other GitHub search was run. Every implementation detail above therefore comes from the paper text and its appendices, not from code, so nothing could be compared line by line.
Pitfalls
- Penalty form. The main equation uses −λR eq. 6, but every reported run uses −λ log R app. E app. F. The log changes the gradient scale by 1/R, and R lies between 1 and r §4.5. Implement the log form if you want to reproduce the reported λ values.
- Start at zero update. The paper states the Rényi-2 ratio is undefined at ΔW=0 and its gradient grows as the scale falls app. D. The paper does not state the adapter initialization. If it starts at ΔW=0 (standard LoRA's B=0 would do this, but the paper does not say so), Tr(G²) is 0 at step 0 and the clamp at eps=1e-8 decides the gradient app. E. Mixed-domain runs apply the penalty from step 0 app. F.
- Start step trades fit for retention. An early start gives the most retention and the worst training perplexity (Train PPL 22.6 without the penalty, 24.7 at S=0) tab. 3 §4.5. The paper says the mixed-domain Train PPL cost at λ=0.1 may be reduced by a later start, but it did not rerun those sweeps §4.5.
- Ramp endpoint. The paper says the weight rises linearly from 0 to λ over W steps after S app. F. It does not say whether the ramp is (t−S)/W with a full value at S+W. The pseudocode above uses that convention; check against the implementation.
- Dense FT is not matched. The dense baseline uses lr 3e-5, weight decay 0 and micro-batch 4 in the mixed setting, while adapters use lr 1e-4, weight decay 0.01 and micro-batch 1 app. F. Its hyperparameters were chosen by lowest Train PPL, which the paper calls a training proxy rather than validation app. F. Comparisons with dense FT are confounded by these settings.
- Adapter counts are not matched. VeRA, SVF and TinyLoRA use far fewer parameters. The paper says they underfit and should be read as an underfitting regime app. F tab. 5.
- PiSSA has n=2. One resumed run with apparent data replay is excluded §4.1. The two-run SD does not show low population variance §4.1.
- Outlier cell. Qwen2.5-3B at rank 16 with λ=0.01 has Train PPL 4.524 ± 0.227, up from 4.211 tab. 10. The paper gives no explanation, and this is the one exception to the "at most 0.02" claim §4.4.
- Undiscussed λ. The Llama sweep includes λ=0.001, which the text does not discuss tab. 11.
- Rank and λ interact. At rank 4, λ=0.1 pushes erank to its ceiling, but training perplexity rises from 6.10 to 6.60 and retention does not clearly improve §4.4 tab. 9. The paper's Wiki claim is stated for ranks of at least 16; at rank 4 the λ=0.01 change is small (13.615 → 13.575) tab. 9.
- Attribution of the best train value. The text says LoRA-XS at λ=1 has "the best Train PPL in the sweep" §4.3. Within the LoRA-XS block that holds. PiSSA at λ=1 is lower (1.825 vs 1.847) tab. 2.
- Norm control is confounded. NB-LoRA raises erank and lowers Wiki PPL too, and the authors say these experiments do not separate effective-rank maximization from norm control §4.1.
- Different diagnostics. The reported erank is the Roy–Vetterli quantity, computed from the final ΔW with a QR and SVD app. E app. F. The training objective uses R, which is a different surrogate §4.1.
- Rank sweeps used a fixed token budget. The 1e6-token budget covers the Qwen2.5-0.5B sweeps app. F. For the 3B and Llama sweeps, the paper does not state α.
- Table and figure numbers. No compiled PDF was available. Numbers were counted in source order from the LaTeX, and the tables cited here are assumed to match the rendered paper. Check against the PDF before quoting a number.
- Author-reported only. The paper reports means over 1–3 runs for most rows, and several rows are single runs. The authors list benefits as configuration-dependent and do not claim universal improvement §7.
- Stated limits. The authors say the method still needs task-specific tuning of λ, may not help with deliberate underfitting or narrow-domain retention, and was only tested on LoRA-style adapters § Limitations.
Lineage
- Builds on: Roy and Vetterli's effective rank (2007, no arXiv id in the bibliography) §2; Rényi entropies §3.3.
- Compared against: PiSSA (arXiv:2404.02948), LoRA-XS (arXiv:2405.17604), NB-LoRA (arXiv:2501.19050), plus VeRA, SVF and TinyLoRA by name §4 tab. 5.
- Related but different: rank-stabilized scaling and MiLoRA, which the paper says address adjacent problems §6.
- Superseded by: not known from this wiki.
Source and license
Adapted from “Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off” by Tomohiro Sawada, Kartik Goyal (arXiv:2611.17191), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/tomohiro-sawada/lora-capacity-utilization at commit e08349187c1e (MIT (LICENSE file only; repository has no code)). Any quoted code is shown under that license.