wikillm

Paper · 2026-11-12 · cs.LG, cs.CL

Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off

LoRA updates often use far less of their configured rank. The authors add an SVD-free Rényi-2 effective-rank penalty, -lambda log R, to the language-modeling loss. With a weight of 0.01 it lowers average Wiki perplexity on three models at each tested rank from 16 upward (author-reported), and training perplexity is mostly unchanged.

Low-rank adaptation (LoRA and variants)Parameter-efficient fine-tuningCatastrophic forgettingContinual learningloraeffective-rankregularizationcatastrophic-forgettingparameter-efficient-fine-tuning

arXivPDFCodeMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

LoRA teaches a language model a new task by training a small add-on instead of all of the model's weights. The size of the add-on is set by a number called the rank. The paper's point is that a higher rank does not automatically give the add-on more useful room. Training tends to put most of the change into a few directions, so the rank on paper is much larger than the room actually used.

The authors measure how evenly the change is spread across directions. They add a term to the training loss that rewards a more even spread. In their tests the model keeps more of its general knowledge, measured as perplexity on general-text samples, while its fit to the new task mostly stays the same or gets better. The gains depend on the setup, and they are reported by the authors themselves.

TL;DR

  • Standard LoRA updates have an effective rank far below the configured rank, and the gap grows with the rank §1 §2 fig. 1.
  • The main method adds a Rényi-2 effective-rank term to the language-modeling loss. It is computed from the LoRA factors without an SVD and works with any parameterization that can be written as a product of two factors §3.3 app. E.
  • On three models in mixed-domain runs, a penalty weight of 0.01 gives lower average Wiki perplexity than unpenalized LoRA at each tested rank from 16 upward. Training perplexity changes by at most 0.02 in all but one comparison §4.4 tab. 9 tab. 10 tab. 11.
  • The penalty is a log, not the plain form written in the main equation, and it has a steep gradient near zero update. The authors use a late start in single-task runs app. E app. F §4.5.
  • Relevant when you want more retention for a given task loss. Not a substitute for tuning lambda, which the authors say is task-specific § Limitations.

Problem

Standard LoRA is usually set up by choosing a rank r and assuming that the adapter can use up to r directions §2. The authors measure the utilization quotient erank(ΔW)/r and find it well below 1. It also falls as r grows, and it is lowest in the late layers §2 fig. 1. This matters for the learning-forgetting trade-off, because it depends on how the update's energy is spread across directions, not only on the rank §1 §2.

Several remedies exist: parameter-efficient variants (VeRA, SVF, TinyLoRA), norm-constrained LoRA (NB-LoRA), and spectral initializations (PiSSA, LoRA-XS). The authors argue that these mostly move the trade-off rather than improve it, because methods that fit the mixture well also forget more §1 fig. 2 tab. 5.

The paper's definition of effective rank is exact, but its gradient needs an SVD and has singularities when singular values approach zero §3.1. The authors therefore optimize a surrogate that does not need an SVD.

Method

Notation. W0 is the pretrained weight of a target layer and ΔW = W - W0 is the accumulated LoRA update §2. The paper restricts adaptation to the MLP down_proj modules in its main experiments §4. The per-module update is written ΔW = γBA with γ = α/r app. E.

Effective rank. The Roy–Vetterli effective rank is erank(ΔW) = exp(H1(p)), where p_i = σ_i / Σ_j σ_j is the normalized singular-value spectrum §2 eq. 1 eq. 2. It equals 1 for a rank-one update and approaches r for a uniform spectrum §2.

Objective. The goal is min L(W0+ΔW) − λ·Cap(ΔW), with Cap taken as the effective rank §3 eq. 3. The authors develop two routes around it.

Route 1: norm-bounded LoRA (NB-LoRA). Since erank ≥ ‖ΔW‖²_S2 / δ² whenever the operator norm of ΔW is at most δ §3.2, they optimize L − λ‖ΔW‖²_S2/δ² under an operator-norm cap eq. 4. The cap is enforced by the NB-LoRA parameterization, which builds ΔW from a diagonal S with operator norm δ and two semi-orthogonal factors obtained from a Cayley map §3.2. They report that this route raises effective rank but can be hard to optimize §3.2 §4.2.

Route 2 (main method): Rényi-2 penalty. The Rényi-2 effective rank is R(ΔW) = exp(H2(q)), with q_i = σ_i² / Σ_j σ_j² §3. It has the closed form R(ΔW) = ‖ΔW‖⁴_F / Tr((ΔW^T ΔW)²) §3.3. The paper proves rank(ΔW) ≥ erank(ΔW) ≥ R(ΔW) ≥ 1 §3.3, so raising R raises a lower bound on the Roy–Vetterli effective rank. The penalized problem is min L(W0+ΔW) − λR(ΔW) §3.3 eq. 6, where λ may be fixed or scheduled.

Logarithmic form (used in every reported penalty run). The experiments subtract λ·H2 = λ·log R rather than λR app. E app. F. For a nonzero update, the gradient of H2 is the gradient of R divided by R, so the penalty is rescaled per module app. E. The reported λ values refer to this log form app. F.

Factor form, no SVD. Let G = (BᵀB)(AAᵀ), an r×r matrix. Then ‖ΔW‖²_F = γ²Tr(G) and Tr((ΔW^T ΔW)²) = γ⁴Tr(G²). Hence R = Tr(G)²/Tr(G²), and H2 = 2 log Tr(G) − log Tr(G²) app. E. The dense update is never formed during training. Storage is Θ(r²) beyond the adapter, and arithmetic is Θ(r²(m+n) + r³) app. E. γ cancels in R, so the penalty value does not depend on α app. E.

Pseudocode
# Rényi-2 entropy of one LoRA module, no SVD  [app. E] [§3.3]
# Input:  A (r, n), B (m, r), gamma = alpha / r  (float32)
# Output: h2, the Rényi-2 entropy of the energy spectrum (scalar)

def renyi2_entropy(A, B, gamma, eps=1e-8):
    # Reduced r x r Gram matrices; dense (m, n) update never built  [app. E]
    gram_b = B.T @ B                          # (r, r)
    gram_a = A @ A.T                          # (r, r)
    G = gram_b @ gram_a                       # (r, r)
    # Traces of G and G^2 give the Frobenius and collision terms  [app. E]
    trace_g = max(trace(G), eps)              # Tr(G), clamped below at eps
    trace_g2 = max(sum(G * G.T), eps)         # Tr(G^2) = sum_ij G_ij G_ji
    # gamma cancels in the ratio, so h2 does not depend on alpha  [app. E]
    return 2 * log(trace_g) - log(trace_g2)
Pseudocode
# Penalty weight schedule  [app. F]
# Input:  step, lam (terminal weight), start (S), ramp (W)
# Output: weight applied to the summed entropy

def penalty_weight(step, lam, start, ramp):
    # Weight is 0 before the start step  [app. F]
    if step < start:
        return 0.0
    # Linear ramp to lam over ramp steps, then constant  [app. F]
    progress = min(1.0, (step - start) / ramp)
    return lam * progress
Pseudocode
# Training loss with the penalty  [§3.3] [app. F]
# Input:  batch, adapters (one per targeted module), step
# Output: scalar loss

def training_loss(batch, adapters, step, lam, start, ramp):
    # Cross-entropy on assistant tokens only; prompt tokens are masked  [app. F]
    lm_loss = assistant_cross_entropy(model(batch), batch)
    # Sum of per-module Rényi-2 entropies, log form  [app. F]
    total_h2 = 0.0
    for module in adapters:
        total_h2 = total_h2 + renyi2_entropy(module.A, module.B, module.gamma)
    weight = penalty_weight(step, lam, start, ramp)
    # Subtracting the entropy rewards a flatter energy spectrum  [app. F]
    return lm_loss - weight * total_h2

PiSSA and LoRA-XS. The same penalty applies after exposing effective factors. For LoRA-XS, B = Ũ and A = R Ṽᵀ, with fixed truncated-SVD bases Ũ and Ṽ and a trainable core R app. E. For PiSSA, ΔW = γ(BA − B0A0), so the effective factors are B_eff = [B, B0] and A_eff = [A; −A0], of width 2r app. E. For NB-LoRA, the effective (A, B, γ) come from the Cayley construction app. E.

Scheduling. The penalty can start at a step S and ramp for W steps app. F. Mixed-domain runs use S=0 and W=50 app. F. Single-task runs use S=400 and W=200, with stated exceptions app. F. Section 4.5 shows that the start step trades training fit against retention §4.5 tab. 3.

Diagnostics (not in the training loss). The paper logs erank(ΔW) from a QR factorization of γB and an SVD of the r×n matrix R_qr A app. E. It also reports a spectral alignment analysis: a finetuned direction is aligned if its best energy overlap with a pretrained direction is at least 0.9² §5. A soft permutation coupling is computed with Sinkhorn normalization at β=10, and its normalized row entropy is H_soft §5 eq. 7. These diagnostics are descriptive §5.

Hyperparameters

Default SFT protocol for the 0.5B mixed-domain runs and single-task runs. Values are as stated; "not stated" means the paper does not say.

NameValueWhereNotes
base model (main)Qwen2.5-0.5B (base, not Instruct)§4 app. FAlso Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct, Pythia-2.8B, OLMo-2 in the appendix sweeps app. G
adapted modulesMLP down_proj layers§4 app. FModule list for the Llama sweeps: not stated
LoRA rank r896 (main); sweep 4, 16, 128, 256, 512§4 §4.4Qwen2.5-3B baselines use r=2048 app. F
alpha2r = 1792app. FQwen2.5-3B baselines α=4096 app. F; α for the 3B and Llama sweeps: not stated
scaling gammaα/r = 2app. E app. FCancels in the penalty ratio app. E
optimizerAdamWapp. F
learning rate (adapters)1e-4app. FConstant after warmup in mixed-domain runs
weight decay (adapters)0.01app. F
warmuplinear, 20 optimizer stepsapp. FThen constant peak rate
micro-batch / grad accumulation1 / 2 (effective batch 2 packed sequences)app. F
max sequence length512 tokens, packed, dynamic paddingapp. F
LoRA dropout0app. F
gradient clippingnoneapp. F
precisionbase weights bf16, no AMP, no fp32 master weights; penalty and SVD in float32app. F app. E
clampTr(G), Tr(G²) ≥ eps = 1e-8 before logapp. F app. E
penalty lambda{0, 0.01, 0.1, 1.0}; sweep also includes 0.001 for Llama§4 app. F tab. 11lambda is in the log form
penalty schedule (mixed)S=0, W=50app. FRamp endpoint convention: see Pitfalls
penalty schedule (single task)S=400, W=200app. FRank exceptions listed in app. F
mixed-domain budget1e6 supervised target tokens§4 app. F45% GSM8K, 10% WMT19 cs–en, 45% Aya Hausa
mixture seed / shuffle buffer0 / 50,000app. F
evaluationevery 100 optimizer stepsapp. FTerminal state reported, no early stopping
eval batches≤20 (mixed), ≤50 (single task)app. FProbe slices: 2,000 documents each from OLMo-Mix-1124 wiki and starcoder
Train PPLexp of EMA (β=0.99) of assistant-only train lossapp. FSmoothed proxy, not validation PPL
NB-LoRA delta1.0app. FCayley-based
VeRAinitial scale 0.1, seed 0app. F
TinyLoRAone shared scalar, seed 0app. F
dense FT (mixed)lr 3e-5, weight decay 0, micro-batch 4, grad accum 2, 20-step warmupapp. FNot matched to adapters; see Pitfalls
dense FT gridlr {1e-5, 3e-5, 1e-4}, micro-batch {1, 2, 4}, wd {0, 0.01, 0.1}, schedule {linear, warmup-cosine}app. FSelected by lowest Train PPL
scheduler ablation~500 steps, lambda_1=0.1, W=200, S ∈ {0, 50, 100, 400}, Hausa only§4.5 tab. 3 app. FSeparate short budget
hardwareone NVIDIA A40, 48 GiB host memoryapp. F"unless noted otherwise"
alignment thresholdepsilon = 0.9§5Aligned if max overlap ≥ 0.81
Sinkhorn beta10§5Log clamp 1e-12

Results

Author-reported. Setup: Qwen2.5-0.5B base, mixed-domain continual-learning proxy (1e6 supervised target tokens), rank 896 with alpha 1792, unless noted §4.1 app. F. Columns: eRank is higher-is-better; Train, StarCoder, Wiki, GSM8K and WMT are perplexities (PPL), lower is better. Means ± sample SD over n training executions. Single runs have n=1. PiSSA excludes one resumed execution with apparent data replay §4.1.

Mixed-domain, Qwen2.5-0.5B, rank 896 tab. 5 fig. 2

Methodn#ParamseRank ↑Train ↓StarCoder ↓Wiki ↓GSM8K ↓WMT ↓
Dense FT1—844.55.6433.64016.5631.8179.223
Standard LoRA3123,863,040360.5 ± 5.24.773 ± 0.0935.815 ± 0.81119.601 ± 1.0561.929 ± 0.04415.018 ± 0.802
LoRA, λ=0.013123,863,040668.0 ± 1.44.381 ± 0.0074.143 ± 0.00615.290 ± 0.0671.836 ± 0.01010.156 ± 0.034
LoRA, λ=0.13123,863,040887.8 ± 0.34.525 ± 0.0164.098 ± 0.00714.648 ± 0.0461.832 ± 0.0089.517 ± 0.172
LoRA, λ=1.01123,863,040895.54.6884.00314.0851.8008.909
PiSSA2123,863,040815.7 ± 8.24.286 ± 0.0035.162 ± 0.10221.565 ± 0.6581.878 ± 0.00213.160 ± 0.020
PiSSA, λ=0.11123,863,040892.24.3354.76918.6341.87412.109
LoRA-XS319,267,584566.8 ± 5.04.578 ± 0.0055.517 ± 0.05021.431 ± 0.5841.901 ± 0.01014.691 ± 0.158
NB-LoRA (δ=1, no penalty)3123,863,040583.2 ± 1.54.396 ± 0.0024.404 ± 0.02815.705 ± 0.0731.809 ± 0.00310.196 ± 0.107
VeRA343,008632.2 ± 0.011.468 ± 0.0013.832 ± 0.00113.005 ± 0.0062.373 ± 0.0009.309 ± 0.007
SVF321,504783.6 ± 0.19.694 ± 0.2573.895 ± 0.00213.186 ± 0.0062.061 ± 0.0008.814 ± 0.006
TinyLoRA31681.5 ± 0.211.549 ± 0.0013.833 ± 0.00013.006 ± 0.0082.388 ± 0.0019.341 ± 0.003

Setup notes from the same table: VeRA, SVF and TinyLoRA underfit, so their low retention is not matched-capacity evidence app. F. The dense FT row is a single run with its own optimizer settings app. F.

Rank sweep, Qwen2.5-0.5B tab. 9 fig. 3. Same mixed-domain protocol; λ=0 is standard LoRA.

RankλeRank ↑Train ↓StarCoder ↓Wiki ↓GSM8K ↓WMT ↓
403.014 ± 0.0756.096 ± 0.0093.926 ± 0.00413.615 ± 0.0371.802 ± 0.0018.370 ± 0.045
40.013.999 ± 0.0006.086 ± 0.0133.932 ± 0.00913.575 ± 0.0171.803 ± 0.0018.425 ± 0.062
40.14.000 ± 0.0006.595 ± 0.0123.942 ± 0.01613.564 ± 0.0011.821 ± 0.0018.403 ± 0.068
1609.689 ± 0.0775.419 ± 0.0063.953 ± 0.00313.780 ± 0.0201.783 ± 0.0028.382 ± 0.008
160.0115.990 ± 0.0005.391 ± 0.0053.950 ± 0.00613.602 ± 0.0311.780 ± 0.0018.310 ± 0.022
160.115.999 ± 0.0005.717 ± 0.0043.941 ± 0.00213.578 ± 0.0131.795 ± 0.0008.332 ± 0.039
128074.494 ± 0.3914.605 ± 0.0024.012 ± 0.01714.202 ± 0.0471.767 ± 0.0048.715 ± 0.079
1280.01126.714 ± 0.0744.611 ± 0.0023.961 ± 0.01213.865 ± 0.0321.763 ± 0.0018.470 ± 0.028
1280.1127.951 ± 0.0014.749 ± 0.0023.958 ± 0.01013.863 ± 0.0051.767 ± 0.0038.424 ± 0.046
2560145.032 ± 0.1754.450 ± 0.0014.067 ± 0.00614.674 ± 0.0571.778 ± 0.0049.406 ± 0.064
2560.01250.540 ± 0.1114.465 ± 0.0033.992 ± 0.01014.145 ± 0.0481.777 ± 0.0038.742 ± 0.069
2560.1255.777 ± 0.0024.559 ± 0.0013.986 ± 0.01313.975 ± 0.0371.776 ± 0.0018.626 ± 0.058
5120265.043 ± 0.6194.375 ± 0.0094.183 ± 0.00815.529 ± 0.1291.806 ± 0.00510.292 ± 0.505
5120.01473.743 ± 0.9104.310 ± 0.1324.056 ± 0.00614.486 ± 0.1201.794 ± 0.0079.321 ± 0.060
5120.1510.839 ± 0.0114.469 ± 0.0014.025 ± 0.02114.281 ± 0.0611.802 ± 0.0049.056 ± 0.013

Cross-model checks from the sweep tables (Wiki PPL, λ=0 → λ=0.01):

  • Qwen2.5-3B-Instruct tab. 10: rank 16, 9.148 → 9.102; rank 256, 9.539 → 9.210; rank 512, 9.708 → 9.185. Rank 16 Train PPL: 4.211 → 4.524 ± 0.227, the one exception to "at most 0.02".
  • Llama-3.2-3B-Instruct tab. 11: rank 16, 11.532 → 11.315; rank 512, 12.180 → 11.655.
  • Llama-3.2-3B-Instruct at rank 3072 tab. 7: standard LoRA Train 3.984, Wiki 18.547, eRank 985.4; λ=0.01 Train 3.837, Wiki 15.924, eRank 1504.2; λ=0.1 Train 3.679, Wiki 13.327, eRank 2725.1.

Scheduling, Aya Hausa, Qwen2.5-0.5B, λ₁=0.1, W=200 tab. 3 §4.5

Start step SeRank ↑Train PPL ↓StarCoder PPL ↓Wiki PPL ↓
standard LoRA (no penalty)341.422.6044.05421.822
0838.224.7373.79218.917
50810.023.8603.75318.700
100785.123.6513.75618.824
400429.822.4103.86119.768

PiSSA and LoRA-XS on single-task GSM8K, Qwen2.5-0.5B, r=896 tab. 2 §4.3

MethodeRank ↑Train PPL ↓StarCoder PPL ↓Wiki PPL ↓
PiSSA, λ=0829.01.8714.32323.173
PiSSA, λ=0.01877.91.8694.23322.255
PiSSA, λ=0.1894.01.8514.10420.951
PiSSA, λ=1.0895.71.8253.93419.706
LoRA-XS, λ=0565.01.8925.06625.479
LoRA-XS, λ=0.01722.01.9044.62523.195
LoRA-XS, λ=0.1880.41.8964.37121.398
LoRA-XS, λ=1.0895.01.8474.13420.121

NB-LoRA on Aya Hausa, Qwen2.5-0.5B, r=896 tab. 1 §4.2. Here λ weights the Frobenius capacity term of eq. 4; the NB-LoRA row uses λ=0.

MethodeRank ↑Train PPL ↓StarCoder PPL ↓Wiki PPL ↓
Dense FT805.520.9774.24323.393
Standard LoRA340.522.5803.99821.098
NB-LoRA504.023.9194.07120.089
λ=0.01527.523.7294.02419.815
λ=0.1663.723.9234.10819.331
λ=1.0814.124.3074.04819.066

Main-text summary numbers from the same experiments: erank for standard LoRA rises from 361 to 668, 888 and 896 at λ=0.01, 0.1 and 1.0 §4.1. The text calls LoRA-XS at λ=1 (Train PPL 1.847) the best Train PPL in its block; across the whole GSM8K table, PiSSA at λ=1 is lower at 1.825 tab. 2.

Paper vs code

No official code: the repository the paper links, https://github.com/tomohiro-sawada/lora-capacity-utilization (linked from the abstract), holds only a LICENSE file at commit e08349187c1e115b763c0a3c25f689946f536eab. The LICENSE is MIT. The repository has no adapter, penalty, training, or configuration code. I checked only this linked repository, and no other GitHub search was run. Every implementation detail above therefore comes from the paper text and its appendices, not from code, so nothing could be compared line by line.

Pitfalls

  • Penalty form. The main equation uses −λR eq. 6, but every reported run uses −λ log R app. E app. F. The log changes the gradient scale by 1/R, and R lies between 1 and r §4.5. Implement the log form if you want to reproduce the reported λ values.
  • Start at zero update. The paper states the Rényi-2 ratio is undefined at ΔW=0 and its gradient grows as the scale falls app. D. The paper does not state the adapter initialization. If it starts at ΔW=0 (standard LoRA's B=0 would do this, but the paper does not say so), Tr(G²) is 0 at step 0 and the clamp at eps=1e-8 decides the gradient app. E. Mixed-domain runs apply the penalty from step 0 app. F.
  • Start step trades fit for retention. An early start gives the most retention and the worst training perplexity (Train PPL 22.6 without the penalty, 24.7 at S=0) tab. 3 §4.5. The paper says the mixed-domain Train PPL cost at λ=0.1 may be reduced by a later start, but it did not rerun those sweeps §4.5.
  • Ramp endpoint. The paper says the weight rises linearly from 0 to λ over W steps after S app. F. It does not say whether the ramp is (t−S)/W with a full value at S+W. The pseudocode above uses that convention; check against the implementation.
  • Dense FT is not matched. The dense baseline uses lr 3e-5, weight decay 0 and micro-batch 4 in the mixed setting, while adapters use lr 1e-4, weight decay 0.01 and micro-batch 1 app. F. Its hyperparameters were chosen by lowest Train PPL, which the paper calls a training proxy rather than validation app. F. Comparisons with dense FT are confounded by these settings.
  • Adapter counts are not matched. VeRA, SVF and TinyLoRA use far fewer parameters. The paper says they underfit and should be read as an underfitting regime app. F tab. 5.
  • PiSSA has n=2. One resumed run with apparent data replay is excluded §4.1. The two-run SD does not show low population variance §4.1.
  • Outlier cell. Qwen2.5-3B at rank 16 with λ=0.01 has Train PPL 4.524 ± 0.227, up from 4.211 tab. 10. The paper gives no explanation, and this is the one exception to the "at most 0.02" claim §4.4.
  • Undiscussed λ. The Llama sweep includes λ=0.001, which the text does not discuss tab. 11.
  • Rank and λ interact. At rank 4, λ=0.1 pushes erank to its ceiling, but training perplexity rises from 6.10 to 6.60 and retention does not clearly improve §4.4 tab. 9. The paper's Wiki claim is stated for ranks of at least 16; at rank 4 the λ=0.01 change is small (13.615 → 13.575) tab. 9.
  • Attribution of the best train value. The text says LoRA-XS at λ=1 has "the best Train PPL in the sweep" §4.3. Within the LoRA-XS block that holds. PiSSA at λ=1 is lower (1.825 vs 1.847) tab. 2.
  • Norm control is confounded. NB-LoRA raises erank and lowers Wiki PPL too, and the authors say these experiments do not separate effective-rank maximization from norm control §4.1.
  • Different diagnostics. The reported erank is the Roy–Vetterli quantity, computed from the final ΔW with a QR and SVD app. E app. F. The training objective uses R, which is a different surrogate §4.1.
  • Rank sweeps used a fixed token budget. The 1e6-token budget covers the Qwen2.5-0.5B sweeps app. F. For the 3B and Llama sweeps, the paper does not state α.
  • Table and figure numbers. No compiled PDF was available. Numbers were counted in source order from the LaTeX, and the tables cited here are assumed to match the rendered paper. Check against the PDF before quoting a number.
  • Author-reported only. The paper reports means over 1–3 runs for most rows, and several rows are single runs. The authors list benefits as configuration-dependent and do not claim universal improvement §7.
  • Stated limits. The authors say the method still needs task-specific tuning of λ, may not help with deliberate underfitting or narrow-domain retention, and was only tested on LoRA-style adapters § Limitations.

Lineage

  • Builds on: Roy and Vetterli's effective rank (2007, no arXiv id in the bibliography) §2; Rényi entropies §3.3.
  • Compared against: PiSSA (arXiv:2404.02948), LoRA-XS (arXiv:2405.17604), NB-LoRA (arXiv:2501.19050), plus VeRA, SVF and TinyLoRA by name §4 tab. 5.
  • Related but different: rank-stabilized scaling and MiLoRA, which the paper says address adjacent problems §6.
  • Superseded by: not known from this wiki.

Source and license

Adapted from “Maximizing Capacity Utilization of LoRA Adapters for Mitigating the Learning-Forgetting Trade-off” by Tomohiro Sawada, Kartik Goyal (arXiv:2611.17191), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/tomohiro-sawada/lora-capacity-utilization at commit e08349187c1e (MIT (LICENSE file only; repository has no code)). Any quoted code is shown under that license.