Paper · 2026-10-27 · cs.LG
SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection
SimplexLoRA gives each LoRA adapter a scalar weight kept on the n-scaled simplex, so the total rank budget stays fixed. After a short warm-up that projects the weights and resizes adapters, it trains standard LoRA at the new ranks. Author-reported: matches or beats LoRA at equal parameters on 12 tasks and four models, with the largest gains at low ranks.
Low-rank adaptation (LoRA and variants)Parameter-efficient fine-tuningAdaptive rank allocation for LoRAfine-tuningparameter-efficientloraadaptive-rankrank-allocation
arXivPDFCodeMarkdown11 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Standard LoRA gives every adapted layer the same small adapter size. SimplexLoRA lets training decide which layers need more room. Each adapter gets a dial, a single number that says how much it counts. The dials are kept at a fixed total, like slices of one pie, so the overall adapter budget does not grow.
After a short warm-up, the dials are turned into whole-number adapter sizes and the adapters are re-created at those sizes. Then ordinary LoRA training runs on that fixed layout. The authors report that this matches or beats standard LoRA at the same size, and that the gains are usually largest at the smallest sizes.
TL;DR
SimplexLoRA is a LoRA variant that learns a per-adapter scalar weight during a short warm-up, constrained to the n-scaled simplex so that the total rank budget stays fixed. It projects the weights, resizes each adapter by floor(r0 * w_i), and then trains standard LoRA with those ranks. The authors report results on 12 tasks across DeBERTaV3-base, BART-large, Llama 3.1 8B and Qwen3-8B. Differences on some tasks are around 0.001, and the released code departs from the paper in several places (see Paper vs code).
Problem
Standard LoRA uses one rank for every adapted layer, ignoring layer-to-layer differences §1. Earlier adaptive-rank methods add trainable parameters, add inference-time gating or pruning, or change the LoRA pipeline §1. Several use SVD of the adapters at every step, which the authors name as a training-cost overhead §2. The method targets a fixed parameter budget and no extra inference cost §1.
Method
Overview. Each of the n LoRA adapters is scaled by a non-negative weight omega_i, so the adapted layer is W0 + omega_i * A_i B_i. The vector omega is constrained to the n-scaled simplex, whose entries sum to n. With omega equal to the all-ones vector, the method reduces to standard LoRA §3. Weights are trained jointly with the adapters, then projected and used to set ranks, and the rank search is repeated K times §3.1.
- Scaled adapters. Add one scalar weight per adapter, initialised to 1 §3.
- Joint objective. Solve the alternating problem in eq. 1, where L is the loss and the penalty (lambda/2) * ||omega - 1||^2 discourages extreme weights §3 eq. 1.
- Phase 1, warm-up. For T steps, train A, B and omega by gradient methods §3.1.
- Phase 2, projection. Project omega onto the n-scaled simplex in the Euclidean sense: the nearest vector with non-negative entries summing to n §3.1.
- Phase 3, rank update. Set r_i = floor(r0 * omega_i), where r0 is the uniform starting rank. Resize adapter i and re-initialise it with the standard LoRA scheme §3.1 app. B.
- Repeat steps 3 to 5 for K repetitions §3.1.
- Phase 4, standard LoRA. Fix omega to 1, re-initialise A and B at the chosen ranks, and train as ordinary LoRA with any LoRA-compatible optimiser. The warm-up adapter weights are discarded on purpose §3.1.
The authors justify the weighting with a convexity-based bound. They say this is a heuristic: convexity is assumed only to derive the bound, and the non-convex result is a design heuristic, not a guarantee app. A.
# SimplexLoRA: rank search, then standard LoRA [§3] [§3.1] [eq. 1]
# Input: n adapted layers, start rank r0, K repetitions, T steps each
# Output: per-layer ranks; adapters re-initialised; omega fixed to 1
def simplex_lora(layers, r0, K, T, lam):
# One scalar weight per adapter, all starting at 1 [§3]
omegas = ones(n)
for repetition in range(K):
# Phase 1: T steps of standard LoRA training [§3.1]
for step in range(T):
# Penalty pulls omega toward the all-ones vector [§3] [eq. 1]
loss = task_loss(batch) + (lam / 2) * sum((omegas - 1) ** 2)
# Omega: plain gradient step. A, B: optimiser step [§3.1]
omegas = omegas - lr_w * grad(loss, omegas)
adapters = optimiser_step(adapters, grad(loss, adapters))
# Phase 2: Euclidean projection onto the n-scaled simplex [§3.1]
omegas = project_scaled_simplex(omegas, n)
# Phase 3: integer ranks that keep the total budget [§3.1]
ranks = floor(r0 * omegas)
adapters = reinit_adapters(layers, ranks)
# Phase 4: fix omega to 1, then train standard LoRA [§3.1]
omegas = ones(n)
adapters = reinit_adapters(layers, ranks)
return train_lora(layers, adapters)# Euclidean projection onto {w >= 0, sum(w) = n} [§3.1]
# Input: values (n,) ; Output: projected weights (n,), sum = n
def project_scaled_simplex(values, n):
# Find theta so that sum(max(values - theta, 0)) equals n [§3.1]
theta = solve_threshold(values, n)
# Entries below the threshold become exactly zero [§3.1]
return maximum(values - theta, 0)Key equation, the only numbered one in the main text:
\min_{\omega \in n\cdot\Delta_{n-1}} \min_{\mathcal{A},\mathcal{B}} \mathcal{L}(\omega,\mathcal{A},\mathcal{B}) + \tfrac{\lambda}{2}\,\lVert \omega - \mathbf{1} \rVert_2^2 eq. 1
Adapter initialisation (standard LoRA): A is drawn uniformly from (-d_in^-0.5, d_in^-0.5) and B is set to zero, so A B = 0 at the start app. B. LoRA scaling is alpha / r with alpha = 32 app. D.
Hyperparameters
Values are from the appendix tables. Table numbers are counted in LaTeX order because the source contains no PDF; please check against the printed numbering.
| Name | Value | Where | Notes |
|---|---|---|---|
| Optimiser (adapters) | AdamW, betas 0.9, 0.999 | tab. 4 tab. 5 tab. 6 tab. 7 tab. 8 | Modified AdamW per app. D; omega uses a plain gradient step optimizers.py |
| Learning rate for omega | 5 | tab. 4 to tab. 8 | Fixed in the paper; in code the warm-up schedule also scales it run_glue.py |
| Regularisation weight (lambda) | 0 (out-of-box); grid {0, 1, 0.1, ..., 1e-6} (DeBERTa, Llama, Qwen); 1e-4 (BART) | tab. 4 to tab. 8 | Grid is the paper's search space; scripts fix one value (see Paper vs code) |
| Model learning rate | Tuned (PEFT) (out-of-box, BART); grid {3, 5, 8, 10, 30, 50, 80, 100} x 1e-5 (DeBERTa, Llama, Qwen) | tab. 4 to tab. 8 | DeBERTa script uses 5e-5 only run_simplex_lora_glue.sh |
| LoRA alpha / dropout | alpha = 32 (scaling alpha / r); dropout 0.05 | app. D tab. 4 to tab. 8 | Scaling recomputed on resize layer.py |
| LR warm-up | 30 (DeBERTa); 50 (Llama, BART); 350 (Qwen); 30 (out-of-box) | tab. 4 to tab. 8 | Warm-up for omega itself: not stated |
| Gradient accumulation | 6 (out-of-box, DeBERTa); 1 (Llama, Qwen); 16 (BART) | tab. 4 to tab. 8 | See Paper vs code for BART |
| Batch size | 32 (out-of-box); task-specific (DeBERTa); 16 (Llama); 4 (Qwen); 8 (BART) | tab. 4 to tab. 8 | |
| Training budget | 512 iterations (out-of-box); task-specific epochs (DeBERTa); 1024 iterations (Llama); 3500 iterations (Qwen); 15 epochs (BART) | tab. 4 to tab. 8 | |
| Scheduler | Linear (out-of-box, DeBERTa, Qwen, BART); cosine (Llama) | tab. 4 to tab. 8 | Two-phase, see §3.1 |
| Target layers | Query, Key, Value (out-of-box table); all linear layers (DeBERTa, Llama, Qwen, BART) | tab. 4 to tab. 8 | See Paper vs code for the out-of-box text |
| K (repetitions) | 5 | §4.3 | Paper's stated setting for all experiments |
| T (warm-up steps per repetition) | 10 | §4.3 | Paper's stated setting for all experiments |
| Ablation budget | 2048 steps, MNLI, DeBERTa | §4.3 fig. 4 | Used for the robustness grid only |
| Initial rank r0 | Per table row (r = 1, 2, 4, 8) | tab. 2 tab. 3 | |
| Hardware | NVIDIA A100-SXM4-80GB | app. D | Software: PyTorch with Hugging Face |
| Seeds | 3 random seeds | app. D | Released DeBERTa script uses 2 (see Paper vs code) |
Results
Author-reported. Setup: GLUE (7 tasks, ALL = mean), XSum and CNN/DailyMail (ROUGE-1), GSM8K, BoolQ and HellaSwag (accuracy) §4 tab. 1 tab. 3. Differences are reported without standard deviations, and the caption says results are averaged over 3 runs tab. 2.
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| DeBERTaV3-base, out-of-box, r = 8, 0.24% params | GLUE ALL | LoRA 0.8696; full fine-tuning 0.8689; rsLoRA 0.8624 | SimplexLoRA 0.8788 | tab. 1 |
| DeBERTaV3-base, r = 1 / 2 / 4 / 8 | GLUE ALL | LoRA 0.8562 / 0.8629 / 0.8700 / 0.8804 | SimplexLoRA 0.8690 / 0.8717 / 0.8756 / 0.8826 | tab. 2 |
| Llama 3.1 8B, r = 1 / 2 / 4 / 8 | GLUE ALL | LoRA 0.8886 / 0.8900 / 0.8899 / 0.8892 | SimplexLoRA 0.8898 / 0.8933 / 0.8941 / 0.8952 | tab. 2 |
| Qwen3-8B, r = 1 / 2 / 4 / 8 | Avg of GSM8K, BoolQ, HellaSwag | LoRA 0.760 / 0.769 / 0.774 / 0.780 | SimplexLoRA 0.774 / 0.780 / 0.784 / 0.788 | tab. 3 |
| BART-large, r = 1 / 2 / 4 / 8 | XSum ROUGE-1 | LoRA 0.372 / 0.374 / 0.375 / 0.380 | SimplexLoRA 0.373 / 0.376 / 0.378 / 0.379 | tab. 3 |
| BART-large, r = 1 / 2 / 4 / 8 | CNN/DailyMail ROUGE-1 | LoRA 0.245 / 0.250 / 0.255 / 0.257 | SimplexLoRA 0.249 / 0.254 / 0.256 / 0.259 | tab. 3 |
The paper also includes DoRA, rsLoRA, AdaLoRA, VERA, LoHA, Krona, NormAL LoRA, ElaLoRA and FlexLoRA in the out-of-box comparison; SimplexLoRA has the best GLUE average at r = 8 tab. 1 §4.1. Qwen3-8B and BART-large use a separate table tab. 3 §4.2. Figure 3 shows the learned DeBERTa rank pattern on MRPC, where value projections get the highest ranks fig. 3 §3.2. The ablation fig. 4 reports that K * T below 100 is stable, that K = 3 with T = 5 underfits, and that K >= 10 or T >= 15 tends to overfit §4.3.
Paper vs code
Code: official repository at commit 8233d7ba77b4d34da52dfb973d88357beb96cbda.
Paths are relative to the repository root; long ones are shortened to a unique
suffix. I read the optimiser, the scheduler, the PEFT layer fork and the released run
scripts. I did not run anything.
- Rank rounding. Paper uses floor(r0 * omega_i) and says the budget is preserved §3.1. Code uses ceil optimizers.py. Since omega sums to n, ceil can push the total rank above n * r0.
- Projection. Paper projects onto n-scaled simplex §3.1. Code projects the raw weights onto the unit simplex and multiplies the result by n [code: src/optimizers.py#L372-L396] optimizers.py. These differ in general. Worked by hand for n = 2 and omega = (0.5, 0): the paper's projection gives (1.25, 0.75), the code gives (1.5, 0.5).
- Dropped adapters. Paper says low-weight adapters are reduced in magnitude, not eliminated §3.2. Code removes any adapter whose ceil-rank is 0 from the chosen set permanently optimizers.py. Its omega stays at 0 so it contributes nothing, but its A and B matrices stay allocated at rank r0.
- Optimiser for omega. Paper says a modified AdamW app. D. Code updates omega with a plain gradient step and lr_w = 5 optimizers.py. Adam is used only for A and B.
- Learning rate of omega. Paper says omega's learning rate is fixed §3.1. Code wraps the optimiser in a LinearLR warm-up from 0.001 times the base rate over K * T steps run_glue.py, and LinearLR scales every parameter group, including omega's [inferred from torch semantics, not run]. In the search phase omega's learning rate therefore ramps from about 0.005 to 5.
- K and T. Paper uses K = 5 and T = 10 for all experiments §4.3. Released scripts: DeBERTa fat_step 5, max_fat_steps 3 [code: deberta/run_simplex_lora_glue.sh#L46-L47]; Llama fat_step 5, max_fat_steps 1 run_simplex_lora_glue.sh; Qwen fat_step 5, max_fat_steps 1 qwen.sh; BART fat_step 10, max_fat_steps 2 run_fat_lora_nlg.sh. Mapping: fat_step = T, max_fat_steps = K.
- Llama training length. Table 6 says 1024 iterations tab. 6. The Llama GLUE script sets max_steps 261 run_simplex_lora_glue.sh.
- BART batch. Table 8 says batch 8 with gradient accumulation 16 tab. 8. The script uses per-device batch 32 with accumulation 6 [code: run_fat_lora_nlg.sh#L13-L15]. The script also caps training at 5000 examples run_fat_lora_nlg.sh; the paper does not state this.
- Qwen quantisation. The Qwen script loads the model in 4-bit [code: run_experiment/qwen.sh#L45]. The paper does not mention quantisation, and its hardware line names A100 80 GB only app. D.
- Seeds. Paper: 3 seeds app. D. The released DeBERTa script lists seeds 52 and 1917 run_simplex_lora_glue.sh. The Qwen script uses three seeds qwen.sh.
- RTE batch. Table 5 gives 64 for RTE tab. 5. The DeBERTa script uses 32 for RTE run_simplex_lora_glue.sh.
- Target layers, out-of-box. Section 4.1 says the out-of-box runs adapt self-attention only §4.1. The only DeBERTa target list in the repository includes the feed-forward dense layers utils.py. Table 2's setting (all linear layers) matches that list.
- Initialisation. Paper's formula for A app. B matches PEFT's Kaiming init with a = sqrt(5), whose bound is 1 / sqrt(fan-in) [code: tuners/lora/layer.py#L160-L172]. B is zeroed [code: tuners/lora/layer.py#L173]. No discrepancy.
- Omega initialisation and reset. omega is created at 1 [code: tuners/lora/layer.py#L150-L153] and reset to 1 on the final update layer.py. Consistent with §3.1.
- Resize. Code re-creates A and B at the new rank on every projection [code: tuners/lora/layer.py#L434-L436]. A commented-out QR-based carry-over exists and is inactive layer.py.
- Optimiser state. On each projection step, A and B skip their Adam update optimizers.py, and Adam moments are zeroed after resize optimizers.py. Not described in the paper.
- Code licence. The repository has no LICENSE at its root. The peft/ subfolder carries its own Apache-2.0 LICENSE. No code is quoted here.
Pitfalls
- Budget. Ceil plus omega summing to n can give a total rank above n * r0. If you need the exact budget of the paper, use floor and check the sum.
- Tiny weights. The projection bisection stops at tolerance 1e-4 [code: src/optimizers.py#L372]. Entries near zero may stay slightly positive, and ceil then gives them rank 1 where an exact projection would give 0. Count the ranks after each projection.
- Permanent drops. Once an adapter reaches rank 0 it cannot return. The remaining weights are projected over the surviving adapters only, so a survivor's rank can exceed r0.
- Softmax projection was rejected. The authors tried softmax and weighted softmax, saw severe degradation, and kept the Euclidean form app. C. Do not swap in softmax without re-checking.
- Omega learning rate. The value 5 is only meaningful under the code's plain gradient step and warm-up ramp. Copying the number into an Adam-based implementation will not reproduce the behaviour.
- Gains are small and unquantified. Several differences are about 0.001 (e.g. Llama r = 1: 0.8898 vs 0.8886; BART XSum r = 8: LoRA higher) and no spread is reported. The paper's statement that gains are largest at low ranks holds for DeBERTa and Qwen but not for Llama, where the largest gain is at r = 8 tab. 2.
- Theory is heuristic. The derivation assumes the loss is convex and only bounds the loss, not the optimal rank distribution app. A. The authors call it a design heuristic.
- Hidden costs. Each projection step re-creates the adapters, so the search phase costs a few extra optimiser passes and changes the optimiser state. Parameter counts in the tables are the reported budgets; check them against the code before comparing memory.
- Qwen and BART runs differ from the paper. Quantisation, batch size, K and T differ (see Paper vs code). Reproduce with the released scripts, not the table values alone.
- Ablation percentage. The paper states that K * T < 100 is under 5% of training. This depends on total training length, which is set per experiment.
Lineage
- Builds on: LoRA, arXiv:2106.09685. SimplexLoRA reduces to it when omega = 1 and keeps its alpha / r scaling §3 app. D.
- Builds on: AdaLoRA, arXiv:2303.10512, the importance-based rank budget it contrasts with, and whose SVD cost it avoids §2.
- Compared against: LoRA, LoHA, Krona, VERA, rsLoRA, DoRA, AdaLoRA, NormAL LoRA, ElaLoRA and FlexLoRA tab. 1.
- Superseded by: not stated.
Source and license
Adapted from “SimplexLoRA: Dynamic Rank Updating via Alternating Minimization with Simplex Projection” by Igor Shalygin, Grigorii Davydenko, Andrey Veprikov, Aleksey Antonov, Nikolay Bushkov, Stanislav Moiseev, Aleksandr Beznosikov (arXiv:2610.37206), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/brain-lab-research/SimplexLoRA at commit 8233d7ba77b4 (none (no LICENSE file at the repository root; peft/ subfolder is a vendored PEFT fork under Apache-2.0)). Any quoted code is shown under that license.