---
kind: concept
slug: adaptive-rank-allocation
title: Adaptive rank allocation for LoRA
summary: >-
  LoRA gives every adapted layer one fixed rank. Adaptive rank allocation lets the rank differ per layer, or grow during training. SimplexLoRA reallocates a fixed total budget with learned weights, then retrains standard LoRA. GEAR adds directions only after a validation test. Both report small gains over LoRA with fewer or equal parameters, with no spread reported.
tags: [fine-tuning, parameter-efficient, lora, rank-allocation]
papers: ["2610.37206", "2611.07349"]
related: [low-rank-adaptation, parameter-efficient-fine-tuning]
status: draft
updated_at: 2026-11-21
---

## Overview

Standard LoRA uses one rank for every adapted layer, whatever the layer's role
[arxiv:2610.37206 §1]. Adaptive rank allocation changes that choice. Two
questions are being answered, and the papers answer them differently:

- **Split the budget.** Given a fixed total rank, which adapters deserve more of
  it? SimplexLoRA answers this with per-adapter weights and integer ranks
  [arxiv:2610.37206 §3].
- **Find missing directions.** Can the current rank fail to contain the
  directions the task needs? GEAR argues that factor gradients can only observe
  the part of the gradient that lies inside the current fixed-rank space, so
  splitting the existing rank cannot reveal those missing directions
  [arxiv:2611.07349 §1] [§3.1] [app. C.1].

Both papers compare against the same LoRA baselines and against AdaLoRA, the
importance-based rank budget method [AdaLoRA](https://arxiv.org/abs/2303.10512)
[arxiv:2610.37206 tab. 1] [arxiv:2611.07349 §2]. The reference points on the
LoRA side are [low-rank adaptation](/concepts/low-rank-adaptation) and
[parameter-efficient fine-tuning](/concepts/parameter-efficient-fine-tuning).

## Variants

### Learned weights on a fixed budget (SimplexLoRA)

Each adapter i keeps its LoRA factors and gets one scalar weight omega_i,
starting at 1. The adapted layer is W0 + omega_i A_i B_i. The weights are kept
on the n-scaled simplex, so they sum to n and the total rank budget does not
change. A penalty pulls them toward 1 [arxiv:2610.37206 §3] [eq. 1].

The schedule has three parts, repeated K times [arxiv:2610.37206 §3.1]:

1. Train the adapters and weights for T steps.
2. Project the weights onto the n-scaled simplex.
3. Set each rank to floor(r0 * omega_i) and re-create the adapter at that rank.

After the last repetition, the weights are reset to 1, the adapters are
re-initialised at the chosen ranks, and ordinary LoRA training runs
[arxiv:2610.37206 §3.1]. The authors say the method adds no inference cost
[arxiv:2610.37206 §1]. Their theory is a convexity-based bound that they call a
design heuristic, not a guarantee [arxiv:2610.37206 app. A].

Reported results are author-reported. On DeBERTaV3-base, GLUE ALL at rank 8 with
0.24% trainable parameters is 0.8788 for SimplexLoRA and 0.8696 for LoRA
[arxiv:2610.37206 tab. 1]. Differences on some tasks are about 0.001 and no
spread is given [arxiv:2610.37206 tab. 2]. Card: [SimplexLoRA](/papers/2610.37206).

### Audited growth from a small rank (GEAR)

GEAR starts from a small rank, r0 = 4 in its settings, and never splits a fixed
budget. Between audits it trains on the fixed-rank manifold: a tangent gradient
step followed by a retraction back to rank r [arxiv:2611.07349 §3.2] [eq. 1]
[eq. 2]. The factors are not handed to an ordinary optimiser step
[arxiv:2611.07349 §3.2].

At scheduled audits, GEAR estimates the leading directions outside the current
space with two zero-output probes and randomized range finding. Each direction
is scored by estimated gain per added parameter [arxiv:2611.07349 §3.3] [eq. 3]
[eq. 4] [eq. 5]. A candidate expansion is accepted only if the relative
training-loss gain on separate microbatches exceeds a threshold ρ. Otherwise the
adapter is unchanged [arxiv:2611.07349 app. B.2] [eq. 6] [eq. 7]. The initial
subspace comes from a two-sided sketch of the gradient at zero, with the frozen
weight adjusted so the starting function is unchanged [arxiv:2611.07349 §3.4]
[eq. 8].

Reported results are author-reported and are means over three seeds with no
spread. LLaMA3-8B commonsense averages 88.45 with 14.4M trainable parameters,
against 86.61 for FlexLoRA with 42.0M [arxiv:2611.07349 tab. 1]. On GLUE with
DeBERTa-v3-base, the average is 89.63 with 1.5M parameters, against 88.85 with
1.9M for FlexLoRA [arxiv:2611.07349 tab. 2]. The one timing measurement, on DTD
with ViT-B/16, is slower than LoRA but uses less peak memory than FlexLoRA
[arxiv:2611.07349 tab. 4]. Card: [GEAR](/papers/2611.07349).

## When to use which

Choose by constraint. The papers report no measurement that would rank these
methods on latency or memory across settings.

| Situation | Use | Why |
|---|---|---|
| Fixed parameter budget, and you want the least change to a standard LoRA loop | SimplexLoRA | Adapters are rebuilt a few times, then training is ordinary LoRA [arxiv:2610.37206 §3.1] |
| Need to add directions the current rank cannot express | GEAR | Expansion is only possible in GEAR's audit design [arxiv:2611.07349 §3.3] |
| Little implementation budget | SimplexLoRA | Needs per-adapter weights, a projection and re-creation of adapters [arxiv:2610.37206 §3]; the card records no extra data beyond training batches |
| Calibration data or extra backward passes are a problem | Neither, without checking | GEAR needs a 128-example calibration set and probe passes [arxiv:2611.07349 app. D.1] [app. B.3]; the SimplexLoRA card records no separate calibration set |
| Memory during training is the binding limit | GEAR, after measuring | On DTD, GEAR's peak memory is between LoRA's and FlexLoRA's [arxiv:2611.07349 tab. 4] |
| Low ranks, where the paper reports its biggest gains | SimplexLoRA, check per model | Gains are largest at low ranks on DeBERTa and Qwen but not on Llama [arxiv:2610.37206 tab. 2] [tab. 3] |

## Implementation sketch

The shared core is a loop that adjusts ranks at scheduled points and then keeps
training. The steps marked (*) differ between the two papers.

```pseudocode
# Shared core of adaptive-rank LoRA. Steps marked (*) differ by method.
# Symbols are ours; the papers use omega, r0, T, K and different names.

def adaptive_rank_lora(modules, r0, schedule):
    # Every adapted matrix starts at the same rank  [arxiv:2610.37206 §3]
    ranks = {m: r0 for m in modules}
    adapters = build_lora(modules, ranks)
    for step in range(total_steps):
        # Ordinary training on the task loss  [arxiv:2610.37206 §3.1]
        adapters = optimiser_step(adapters, grad(task_loss, adapters))
        if step not in schedule:
            continue
        # (*) Score each module: a learned weight in SimplexLoRA, or
        # the estimated gain of new normal directions in GEAR
        # [arxiv:2610.37206 §3.1] [arxiv:2611.07349 §3.3]
        scores = score_modules(adapters, modules)
        # (*) Resize: SimplexLoRA keeps the total budget and may shrink a
        # module; GEAR only appends directions that pass a test
        # [arxiv:2610.37206 §3.1] [arxiv:2611.07349 app. B.2]
        ranks = resize(adapters, scores)
        # Re-create the adapters at the new ranks, then continue. SimplexLoRA
        # re-initialises; GEAR keeps the state and starts new directions at 0
        # [arxiv:2610.37206 §3.1] [arxiv:2611.07349 §3.3]
        adapters = rebuild(adapters, ranks)
    return adapters
```

## Contradictions

- **Whether splitting the rank is enough.** GEAR argues that methods which only
  reshape or split the rank already in use cannot test whether the task needs
  directions outside it [arxiv:2611.07349 §1]. SimplexLoRA keeps the total
  budget fixed and does not add directions [arxiv:2610.37206 §3]; the card
  records no claim about directions outside the current space. Neither paper
  tests the other's method, so this is not settled.
- **Reported levels are not comparable.** On DeBERTaV3-base GLUE, SimplexLoRA
  reports 0.8804 for LoRA at rank 8 on the GLUE ALL score [arxiv:2610.37206
  tab. 2]. GEAR reports 87.11 for LoRA on the mean of eight tasks at a reference
  rank of 8 [arxiv:2611.07349 tab. 2]. The aggregations and settings differ, so
  these figures cannot be set side by side.

## Papers

- [SimplexLoRA](/papers/2610.37206): fixed-budget rank reallocation with a learned weight per adapter, then standard LoRA training.
- [GEAR](/papers/2611.07349): rank growth at audits, tested by a validation gain, with a gradient-based start.
