Paper · 2026-11-05 · math.OC, cs.LG
From What Is Used to What Is Missing: Rethinking Dynamic Rank in LoRA
GEAR grows LoRA rank only when the normal part of the gradient shows task directions outside the current fixed-rank space. It trains the rank-r state by tangent-gradient steps with retraction, estimates normal directions by randomized probing, and starts from a gradient-based subspace. Author-reported, LLaMA3-8B commonsense: 88.45 with 14.4M parameters, versus 86.61 for FlexLoRA with 42.0M.
Low-rank adaptation (LoRA and variants)Parameter-efficient fine-tuningAdaptive rank allocation for LoRAparameter-efficient-fine-tuningloralow-rank-adaptationdynamic-rankriemannian-optimizationrandomized-linear-algebra
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
LoRA trains a small add-on to a frozen model, and the rank of that add-on sets how much it can express. Once the rank is fixed, the training signal only shows how to improve inside the directions the add-on can already represent. It cannot show whether the task would need directions the add-on lacks altogether. This paper says that blind spot is exact and proposes GEAR, which keeps training inside the current rank with a geometrically careful update. Every so often it checks whether a few new directions would help, using a cheap estimate of the part of the gradient that the current rank cannot reach. It adds them only when a short trial on training data shows a real loss improvement. The starting subspace is also taken from the gradient rather than chosen at random. The authors report higher scores with fewer trainable parameters, measured on a few language and vision benchmarks.
TL;DR
- With standard LoRA factor gradients, only the part of the full gradient that lies in the tangent space of the fixed-rank set is observable; the normal part is invisible §3.1 app. C.1.
- GEAR trains the rank-r state with a tangent-gradient step followed by a retraction back onto the fixed-rank set §3.2 eq. 1 eq. 2.
- At scheduled audits it estimates the leading normal singular directions with two zero-output probes, scores them by gain per added parameter, and adds them only if a validation test passes §3.3 eq. 3–eq. 7 app. B.2.
- The initial subspace comes from a two-sided sketch of the gradient, with the frozen weight adjusted so the starting function is unchanged §3.4 eq. 8.
- Author-reported averages: LLaMA3-8B commonsense 88.45 with 14.4M parameters versus 86.61 for FlexLoRA with 42.0M; Qwen2.5-7B 89.40 (14.0M) versus 87.65 (40.3M); GLUE on DeBERTa-v3-base 89.63 (1.5M) versus 88.85 (1.9M); VTAB-1K on ViT-B/16 73.48 (0.2M) versus 72.55 (0.3M) tab. 1 tab. 2 tab. 3.
- No official code was found. Results are means over three seeds with no spread reported. See Pitfalls.
Problem
A LoRA adapter can only move in directions its current rank can express. Dynamic-rank methods mostly decide how to split or reshape the rank already in use. AdaLoRA reallocates a fixed budget across singular components, DyLoRA trains nested ranks, and FlexLoRA and GeoLoRA let rank change through the spectral structure of the current adapter and gradient-driven basis augmentation §1 §2. None of these, the paper argues, tests whether task-relevant descent directions lie outside the current space §1.
The paper separates two situations that look alike from the outside: the current rank is enough but under-optimised, or the task needs directions the current space cannot reach at first order. Only the second justifies a higher rank §1. fig. 1 (panel a) makes the same point: one first-order signal may come from very different rank needs.
Method
Setting. The frozen weight W is adapted as W' = W + sX with X = BA, where s > 0 is a fixed scale §3.1. Let f(X) = L(W + sX) and G = ∇_X f. For a rank-r matrix X with compact SVD X = UΣV^T, the tangent space T_X and normal space N_X of the fixed-rank set split G into a tangent part G_T and a normal part G_N §3.1.
Observability result §3.1 app. C.1. Under the factorised parameterisation, the standard factor gradients are ∇_B f = GA^T and ∇A f = B^T G. Their map O{A,B}(G) = (GA^T, B^T G) has kernel exactly N_X. Two gradients give the same factor-gradient observation if and only if their tangent parts agree. The theorem assumes A and B have full rank. The paper reads this as: factor gradients see the tangent part, which is what is needed to optimise the current rank, and are blind to the normal part, which is what decides expansion §3.1.
Manifold update §3.2. GEAR writes X = UC with U^T U = I_r, where C has shape r × m, and takes V as an orthonormal basis of the column space of C^T §3.2. The tangent gradient is
G_T = U(U^\top G) + (I-UU^\top)\,G\,VV^\top eq. 1
To get its two contractions from backpropagation, GEAR uses a forward-equivalent split B_ext = [U, 0], A_ext = [C; V^T]. Its backward pass gives ∇_C f = U^T G and the gradient on the second block of B_ext equal to GV §3.2. The tangent step and retraction are
X^- = X - \eta\,G_T, \qquad X_{t+1} = \operatorname{Retr}_{\mathcal M_r}(X^-) eq. 2
The pre-retraction matrix can be written with at most 2r columns, so the retraction runs on small factors: thin QR of each, then an SVD of a core no larger than 2r × 2r, keeping the leading r components §3.2 app. B.1.
Normal spectrum and expansion §3.3. For the local quadratic model q_η(H) = ⟨G, H⟩ + ‖H‖²/(2η) over the tangent cone of the set of rank at most r + k, the paper proves (Theorem 2) that the optimal step is −η[G_T + (G_N)_k], where (G_N)_k is the rank-k matrix closest to G_N in Frobenius norm. The extra decrease over tangent-only optimisation is (η/2) times the sum of the first k squared normal singular values §3.3 app. C.2.
Theorem 3 bounds the loss of using an inexact normal estimate Ĝ_N with ‖Ĝ_N − G_N‖_F ≤ ε. The gap between the rank-k step from Ĝ_N and the exact one is between 0 and 2ηε(τ_k + ε), where τ_k = ‖G_N − (G_N)_k‖_F §3.3 app. C.3.
To estimate G_N without forming it, GEAR uses randomized range finding §3.3:
- Draw a probe Ω ∈ R^{m×h} with h much smaller than min(n, m) §3.3. Project it off the right subspace, Ω_⊥ = (I − VV^T)Ω.
- Add a zero-initialised auxiliary P with ΔX_probe = PΩ_⊥^T. The forward map is unchanged, and the backward pass gives ∇P f = GΩ⊥ §3.3.
- Project off the left subspace to get Y, then orthonormalise it to Q:
Y = (I-UU^\top)\,\nabla_P f = G_N\,\Omega, \qquad Q = \operatorname{orth}(Y) eq. 3
- A second zero-output probe, with Q held fixed, yields Q^T G. Then
Z_N = Q^\top G\,(I-VV^\top) = Q^\top G_N = \widetilde U\,\widehat\Sigma\,\widehat V^\top eq. 4
- The estimated normal triplets are û_i = Q ũ_i, v̂_i = V̂_{:,i} and σ̂_i = Σ̂_ii §3.3.
Each direction is scored by its estimated gain per added parameter, and a global budget is set from the weighted-average rank §3.3:
a_{\ell,i} = \frac{\widehat\sigma_{\ell,i}^{\,2}}{n_\ell+m_\ell}, \qquad \mathcal B_{\mathrm{target}} = \bar r_{\mathrm{budget}}\sum_{\ell}\,(n_\ell+m_\ell) eq. 5
Here n_ℓ + m_ℓ is the cost of one extra rank in module ℓ. The paper says candidates are selected subject to the budget and to a per-module rank limit, but does not spell out the selection rule §3.3.
Selected directions form a candidate, which is then tested:
X_{\ell,\mathrm{cand}} = X_\ell - \eta_b\,\widehat U_{\ell,k_\ell}\,\widehat\Sigma_{\ell,k_\ell}\,\widehat V_{\ell,k_\ell}^\top eq. 6
Several candidate prefixes with different birth-step scales η_b are evaluated on the same separate training microbatches, with matched stochasticity. The relative loss gain RI = (L̄_0 − L̄_cand)/|L̄_0| must exceed ρ. Among accepted prefixes, the one that adds the fewest ranks while staying near the best loss seen is kept; ties go to the smaller birth step. If none passes, the adapter is unchanged app. B.2.
An accepted expansion is committed as
U^+ = [U,\ \widehat U], \qquad C^+ = \begin{bmatrix} C \\ -\eta_b\,\widehat\Sigma\,\widehat V^\top \end{bmatrix}, \qquad X^+ = U^+C^+ eq. 7
Because the new directions are orthogonal to the old ones, the state moves from the rank-r set to the rank-(r + k) set. Existing optimiser states are kept and states of the new directions start at zero §3.3.
Audit schedule §4.1. Audits happen at t_start + kΔt for integers k ≥ 0, up to and including t_final. After t_final the rank is frozen and training continues on the fixed-rank manifold alone §4.1.
Pseudocode
The three blocks below restate the update, retraction and audit in our own words. Names follow the paper's symbols where they exist. The selection rule for the budget is not stated in the paper, so the code does not include one.
# Manifold step on the rank-r state X = U C [§3.2] [eq. 1] [eq. 2]
# grad_C = dL/dC = U^T G, shape (r, m); grad_B2 = G V, shape (n, r)
def manifold_step(U, C, V, grad_C, grad_B2, eta):
# Tangent pieces: D = U^T G and K = (I - U U^T) G V [eq. 1]
D = grad_C # (r, m)
K = grad_B2 - U @ (U.T @ grad_B2) # (n, r)
# Tangent step X^- = L R, L = [U, K], R = [C - eta D; -eta V^T] [eq. 2]
L = concat([U, K], axis=1) # (n, 2r)
R = concat([C - eta * D, -eta * V.T], axis=0) # (2r, m)
return retract(L, R, U.shape[1]) # [app. B.1]# Retraction of L R onto rank r, using only the small factors [app. B.1]
def retract(L, R, rank):
# Thin QR of both compact factors [app. B.1]
Q_left, R_left = qr(L) # (n, 2r), (2r, 2r)
Q_right, R_right = qr(R.T) # (m, 2r), (2r, 2r)
# SVD of the small core R_left R_right^T [app. B.1]
P, sigma, Q_core_T = svd(R_left @ R_right.T)
# Keep the leading rank components [app. B.1]
U_new = Q_left @ P[:, :rank] # (n, r)
V_new = Q_right @ Q_core_T.T[:, :rank] # (m, r)
C_new = diag(sigma[:rank]) @ V_new.T # (r, m)
return U_new, C_new, V_new# Randomized normal sketch at an audit [§3.3] [eq. 3] [eq. 4]
# grad_P: from the zero-output probe P, = G omega_perp [§3.3]
# grad_R: from the zero-output right probe with Q fixed, = G^T Q [§3.3]
def normal_sketch(grad_P, grad_R, U, V):
# Left range of the normal part, Y = G_N omega [eq. 3]
Y = grad_P - U @ (U.T @ grad_P) # (n, h)
Q, _ = qr(Y) # (n, h), orthonormal
# Sketch Z_N = Q^T G (I - V V^T) = Q^T G_N [eq. 4]
Z_N = grad_R.T - (grad_R.T @ V) @ V.T # (h, m)
U_tilde, sigma_hat, V_hat_T = svd(Z_N)
# Estimated normal triplets [§3.3]
u_hat = Q @ U_tilde # (n, h)
v_hat = V_hat_T.T # (m, h)
return u_hat, sigma_hat, v_hat# Candidate expansion with birth step eta_b, then acceptance test [eq. 6] [eq. 7] [app. B.2]
def propose_and_accept(X, U, C, u_hat, sigma_hat, v_hat, eta_b,
current_loss, candidate_loss, rho):
# Candidate state for the loss check [eq. 6]
# (the candidate is evaluated, not stored)
# Relative training-loss gain on separate microbatches [app. B.2]
relative_gain = (current_loss - candidate_loss) / abs(current_loss)
if relative_gain <= rho:
# Reject: the adapter is unchanged [app. B.2]
return U, C
# Accept: append directions orthogonal to U, V [eq. 7]
U_plus = concat([U, u_hat], axis=1) # (n, r + k)
coeff = sigma_hat[:, None] * v_hat.T # (k, m)
C_plus = concat([C, -eta_b * coeff], axis=0) # (r + k, m)
return U_plus, C_plusTask-induced initialisation (TGI) §3.4. Initialisation uses the same probing idea, applied to the gradient at X = 0:
- Left probe: X = PΩ^T with P = 0 and fixed Ω ∈ R^{m×h}. The backward pass gives Z_0 = G_0Ω ∈ R^{n×h}, where G_0 = ∇_X f(0) §3.4.
- Scale normalisation. For a sketch Z of shape d × h (d = n for the left probe, d = m for the right), with M = Z ⊙ Z and μ the mean of M, the row scales are (1/h)Σ_a M_ia + λμ and the column scales (1/d)Σ_i M_ia + λμ. The normalised sketch is D_row^{-1/2} Z D_col^{-1/2} §3.4.
- Thin QR of the normalised left sketch gives Q_L ∈ R^{n×h}. A right probe X = Q_L R with R = 0 then gives G_0^T Q_L ∈ R^{m×h}. The procedure runs p + 1 left–right rounds, that is 2(p + 1) stages. The paper uses p = 1 §3.4 app. B.3.
- At the last right stage, the normalised sketch is decomposed by SVD and the leading r_0 ≤ h components are kept:
U_0 = Q_L\,\widehat U_{:,1:r_0}, \qquad C_0 = \operatorname{diag}(\gamma_1,\ldots,\gamma_{r_0})\,V_{:,1:r_0}^\top, \qquad X_0 = U_0C_0 eq. 8
Here γ_i is a clipped value: a kept singular value divided by the average of the kept ones §3.4. The clipping range is not stated.
- Because X_0 ≠ 0, the frozen weight is replaced by W̃ = W − sX_0 so that the starting function equals the original W §3.4.
Hyperparameters
Commonsense reasoning, LLaMA3-8B and Qwen2.5-7B tab. 5 §4.2 app. D.1
| Name | Value | Where | Notes |
|---|---|---|---|
| Initial rank r_0 | 4 | tab. 5 | |
| Per-module maximum rank r_max | 32 | tab. 5 §4.2 | upper bound only; final ranks come from accepted expansions |
| Weighted-average budget r̄_budget | 16 | tab. 5 §4.2 | enters eq. 5 |
| Reference rank of the LoRA baseline | 16 | §4.2 | |
| Target modules | Q, K, V, O, Up, Down, Gate | tab. 5 | |
| Probe width h | 24 | tab. 5 app. D.1 | shared by TGI and audits |
| Calibration batch | 128 examples, 16 per task | app. D.1 | used for rank audits |
| Damping λ | 0.1 | app. D.1 | normalisation in TGI and audits |
| First audit t_start | 100 | tab. 5 | |
| Audit interval Δt | 50 | tab. 5 | |
| Last audit t_final | 500 | tab. 5 | |
| Max rank increase per module per audit | 4 | tab. 5 | |
| Acceptance threshold ρ | 0.5% | tab. 5 | relative training-loss gain |
| Learning rate | 2 × 10⁻² | tab. 5 | one value; adapter and head not split; optimiser not stated |
| Dropout | 0.05 | tab. 5 | |
| Scheduler / warmup | linear / 0.1 | tab. 5 | unit of 0.1 not stated |
| Batch configuration | 16 microbatch × 4 accumulation | tab. 5 | |
| Epochs | 2 | tab. 5 |
GLUE, DeBERTa-v3-base tab. 7 §4.3 app. D.2
Shared values: batch 32, r_0 = 4, r̄_budget = 8, r_max = 16, t_start = 0.1T, Δt = 0.075T, t_final = 0.7T, where T is the total number of optimisation steps. Each module may add at most one rank-one direction per audit. ρ = 0.1%. Probe width h = 24, "the same probing procedure as in commonsense" app. D.2. The paper does not restate λ for this setting. Reference rank 8 §4.3.
| Task | Adapter LR | Head LR | Epochs |
|---|---|---|---|
| CoLA | 1.6 × 10⁻³ | 8 × 10⁻⁴ | 20 |
| MNLI | 5 × 10⁻⁴ | 5 × 10⁻⁴ | 6 |
| MRPC | 3 × 10⁻³ | 1 × 10⁻³ | 30 |
| QNLI | 3 × 10⁻³ | 3 × 10⁻⁴ | 5 |
| QQP | 1.6 × 10⁻³ | 8 × 10⁻⁴ | 5 |
| RTE | 3.6 × 10⁻³ | 1.2 × 10⁻³ | 50 |
| SST-2 | 5 × 10⁻³ | 3 × 10⁻⁴ | 24 |
| STS-B | 4.4 × 10⁻³ | 2.2 × 10⁻³ | 25 |
VTAB-1K, ViT-B/16 tab. 9 §4.4 app. D.3
| Name | Value | Notes |
|---|---|---|
| r_0 / r̄_budget / r_max | 4 / 8 / 16 | "same rank configuration as NLU" §4.4 |
| Learning rate | 5 × 10⁻³ | one value |
| t_start / Δt / t_final | 160 / 50 / 1300 | |
| Acceptance threshold ρ | 0.03% | NLU uses 0.1% tab. 7 |
| Batch size / epochs | 64 / 100 |
Not stated for VTAB: probe width, damping λ, per-audit rank increase, head learning rate, and the procedure that aggregates dense gradient proxies for TGI app. D.3.
Not stated anywhere in the paper: optimiser type and its settings, weight decay, the LoRA scale s, the distribution of Ω, the birth-step scales η_b, the number of validation microbatches and prefixes, the clipping range of γ, baseline hyperparameters, and the hardware used for the main results.
Results
Author-reported. Each entry is the mean over three random seeds. Setup: commonsense reasoning on eight tasks, trained on Commonsense170K §4.2 app. D.1; GLUE, eight tasks §4.3; VTAB-1K, 19 tasks, ViT-B/16 §4.4 app. D.3. The paper reports no standard deviations tab. 1 tab. 2 tab. 3.
Commonsense, LLaMA3-8B tab. 1
| Approach (trainable) | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Mean |
|---|---|---|---|---|---|---|---|---|---|
| LoRA (41.9M) | 72.37 | 87.83 | 81.34 | 94.81 | 85.08 | 91.46 | 81.32 | 86.00 | 85.03 |
| AdaLoRA (31.6M) | 72.23 | 87.21 | 81.27 | 94.60 | 85.16 | 93.18 | 82.68 | 85.80 | 85.27 |
| Riemannion (41.9M) | 72.45 | 87.32 | 81.17 | 94.95 | 87.64 | 93.14 | 82.51 | 87.40 | 85.82 |
| GeoLoRA (41.3M) | 72.84 | 87.54 | 79.99 | 94.62 | 85.56 | 93.52 | 83.96 | 87.00 | 85.63 |
| FlexLoRA (42.0M) | 72.91 | 88.30 | 82.96 | 95.68 | 87.14 | 93.56 | 83.70 | 88.60 | 86.61 |
| GEAR (14.4M) | 75.19 | 90.81 | 83.78 | 96.82 | 89.82 | 93.77 | 85.58 | 91.80 | 88.45 |
Commonsense, Qwen2.5-7B tab. 1
| Approach (trainable) | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Mean |
|---|---|---|---|---|---|---|---|---|---|
| LoRA (40.4M) | 71.54 | 88.52 | 78.73 | 94.05 | 82.56 | 96.13 | 90.00 | 90.12 | 86.46 |
| AdaLoRA (35.2M) | 71.59 | 89.12 | 78.92 | 94.06 | 82.79 | 96.00 | 90.27 | 89.80 | 86.57 |
| Riemannion (40.4M) | 71.68 | 89.17 | 79.32 | 94.72 | 84.53 | 95.96 | 90.53 | 91.00 | 87.11 |
| GeoLoRA (40.1M) | 71.90 | 89.12 | 79.02 | 94.88 | 85.00 | 96.21 | 89.93 | 91.00 | 87.13 |
| FlexLoRA (40.3M) | 72.29 | 89.17 | 79.68 | 95.19 | 86.03 | 96.21 | 90.19 | 92.40 | 87.65 |
| GEAR (14.0M) | 75.51 | 90.86 | 81.58 | 96.34 | 90.53 | 96.80 | 89.76 | 93.80 | 89.40 |
GLUE, DeBERTa-v3-base tab. 2. Baselines use 1.9M parameters; GEAR uses 1.5M. The text says GEAR is strongest on seven of eight tasks §4.3. In the table, QNLI is a tie with AdaLoRA (both 94.27), and GEAR is not best on QQP tab. 2.
| Approach | CoLA (MCC) | MNLI (acc.) | MRPC (acc.) | RTE (acc.) | QNLI (acc.) | SST-2 (acc.) | STS-B (corr.) | QQP (acc.) | Mean |
|---|---|---|---|---|---|---|---|---|---|
| LoRA | 68.31 | 89.74 | 90.19 | 84.16 | 93.61 | 95.30 | 85.20 | 90.35 | 87.11 |
| AdaLoRA | 70.07 | 89.93 | 90.44 | 88.17 | 94.27 | 95.13 | 91.21 | 91.41 | 88.83 |
| Riemannion | 67.41 | 88.93 | 89.01 | 85.13 | 92.64 | 94.00 | 90.12 | 87.92 | 86.90 |
| GeoLoRA | 68.18 | 89.67 | 87.76 | 87.73 | 92.93 | 95.52 | 90.15 | 87.77 | 87.46 |
| FlexLoRA | 70.22 | 90.13 | 90.67 | 88.45 | 94.24 | 95.34 | 91.53 | 90.27 | 88.85 |
| GEAR (1.5M) | 70.85 | 90.97 | 92.46 | 89.53 | 94.27 | 96.22 | 91.83 | 90.88 | 89.63 |
VTAB-1K, ViT-B/16 tab. 3. Baselines use 0.3M parameters; GEAR uses 0.2M. GEAR is best on 14 of 19 tasks, including 7 of 8 Structured tasks §4.4.
| Task | LoRA | AdaLoRA | Riemannion | GeoLoRA | FlexLoRA | GEAR (0.2M) |
|---|---|---|---|---|---|---|
| CIFAR100 | 69.39 | 71.81 | 70.78 | 68.05 | 71.30 | 71.88 |
| Caltech101 | 90.60 | 89.96 | 91.03 | 88.87 | 91.72 | 91.78 |
| DTD | 71.01 | 71.76 | 71.28 | 68.88 | 70.18 | 71.60 |
| Flowers | 98.80 | 99.04 | 99.09 | 99.06 | 98.98 | 99.12 |
| Pets | 89.89 | 91.58 | 91.85 | 91.71 | 91.14 | 91.93 |
| SVHN | 90.53 | 90.59 | 89.08 | 91.39 | 90.25 | 91.89 |
| SUN397 | 53.78 | 55.61 | 55.12 | 54.63 | 54.14 | 55.03 |
| Camelyon | 83.26 | 81.92 | 81.60 | 82.16 | 83.98 | 83.93 |
| EuroSAT | 93.74 | 94.04 | 93.13 | 94.61 | 93.91 | 94.91 |
| RESISC45 | 85.40 | 85.37 | 84.30 | 84.52 | 85.41 | 85.98 |
| Retinopathy | 73.60 | 75.40 | 75.16 | 75.41 | 74.13 | 74.91 |
| CLEVR-C | 80.91 | 81.23 | 78.95 | 80.63 | 81.30 | 82.07 |
| CLEVR-D | 64.74 | 65.38 | 68.49 | 63.37 | 63.56 | 68.51 |
| DMLab | 50.04 | 50.23 | 49.95 | 50.08 | 52.02 | 52.18 |
| KITTI-D | 77.87 | 78.34 | 75.81 | 78.14 | 78.95 | 78.90 |
| dSpr-Loc | 75.76 | 73.38 | 72.40 | 73.62 | 75.49 | 76.78 |
| dSpr-Ori | 47.65 | 49.32 | 48.71 | 48.27 | 49.02 | 49.52 |
| sNORB-Azi | 29.49 | 28.00 | 27.76 | 30.25 | 29.84 | 31.32 |
| sNORB-Ele | 41.93 | 37.56 | 39.58 | 40.99 | 43.17 | 43.89 |
| Average | 72.02 | 72.13 | 71.79 | 71.82 | 72.55 | 73.48 |
Component ablation, LLaMA3-8B commonsense tab. 10 app. E. Checkmarks show which components are on. Averages are the only summary the paper gives per row.
| TGI | Manifold | Expansion | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | Mean |
|---|---|---|---|---|---|---|---|---|---|---|---|
| – | – | – | 72.37 | 87.83 | 81.34 | 94.81 | 85.08 | 91.46 | 81.32 | 86.00 | 85.03 |
| ✓ | – | – | 72.51 | 87.57 | 81.27 | 94.88 | 86.94 | 92.16 | 83.08 | 86.16 | 85.57 |
| – | ✓ | – | 72.40 | 87.03 | 81.12 | 94.46 | 87.38 | 92.51 | 83.12 | 87.21 | 85.65 |
| ✓ | ✓ | – | 73.15 | 89.13 | 82.01 | 94.90 | 87.27 | 92.88 | 83.30 | 87.17 | 86.23 |
| – | ✓ | ✓ | 73.95 | 89.73 | 83.13 | 95.84 | 88.66 | 93.04 | 84.71 | 89.51 | 87.32 |
| ✓ | ✓ | ✓ | 75.19 | 90.81 | 83.78 | 96.82 | 89.82 | 93.77 | 85.58 | 91.80 | 88.45 |
Cost on DTD, ViT-B/16 tab. 4 app. B.3. A single measurement; hardware and number of runs are not stated.
| Method | Training time (s) | Peak CUDA memory (MiB) |
|---|---|---|
| LoRA | 789.83 | 3825.15 |
| FlexLoRA | 1006.75 | 6553.87 |
| GEAR | 1060.83 | 4458.21 |
The paper's summary claim is that GEAR has the top average in every evaluated setting, with fewer parameters than the best adaptive-rank baseline §4.4. The three averages above bear this out. The paper also argues that the gains come from the adaptively chosen final ranks, not from a larger budget §4.4.
Paper vs code
No official code.
Searches, all done for this card. (1) Paper links: the arXiv abstract page and the TeX source (comments, bibliography, the reproducibility statement) contain no repository link. The bibliography URLs point to OpenReview only. (2) Project page: none. (3) GitHub repository search for "GEAR LoRA", "GEAR lora rank", "rank evolution lora tangent", "geometry-guided lora rank" and "rethinking dynamic rank lora". Results were lora-gear by chamodw and LoRA-Gear by yao-jiawei, neither owned by an author; a Gap-Init repository, a different method; and unrelated repositories. No organisation search was run, since no author affiliation maps to a GitHub organisation.
Pitfalls
- Optimiser and step size are unclear. Eq. 2 is a plain tangent step X − ηG_T, yet §3.3 says existing optimiser states are inherited and new ones start at zero §3.3. The paper never names the optimiser. The commonsense learning rate is one value, not split between adapter and head tab. 5, while GLUE splits them tab. 7 app. D.2. It is unclear whether η in eq. 2 is that learning rate.
- Do not hand the factors to an optimiser. The forward-equivalent split B_ext = [U, 0], A_ext = [C; V^T] reproduces X only while the zero block of B_ext is zero. An optimiser step on that block would change the forward map. Implement the update as the tangent step with retraction, and read the gradients from the probe pass §3.2. B_ext also has 2r columns of which r are zero, so it is not the full-rank factor that Theorem 1 assumes app. C.1.
- Notation clash in the retraction. The symbol Q is used for the left QR factor, the right QR factor, the right singular basis and its truncation Q_r. The formulas V_{t+1} = Q_R Q_r and U_{t+1} = Q_L P_r are only correct if these are kept distinct app. B.1.
- Expansion step size is separate from the theory's step size. Theorem 2 is stated for the quadratic model with step η. Expansion uses a different birth step η_b, and its values are not stated eq. 6 app. B.2. The optimality result does not carry over automatically.
- Randomized sketch error is not bounded. Theorem 3 is conditional on a normal-estimate error ε. The paper gives no bound on ε for the probe, and does not state how Ω is drawn §3.3. The guarantee therefore assumes a quality of sketch that is not shown.
- Audit scores are not normalised across modules. The score in eq. 5 uses raw squared normal singular values divided by n_ℓ + m_ℓ. Only the TGI initialisation uses scale normalisation §3.4. This is an inference from the equations and is not discussed in the paper: modules with large gradients would receive more of the budget.
- Budget and selection are underspecified. The paper defines B_target eq. 5 but not the selection rule, the role of k_ℓ, or the "close to best" test in the prefix choice §3.3 app. B.2. The current rank r_ℓ is introduced but does not appear in any formula.
- Acceptance threshold varies widely. ρ is 0.5% for commonsense tab. 5, 0.1% for GLUE tab. 7 and 0.03% for VTAB tab. 9. The text says VTAB uses "the same rank configuration" as NLU §4.4, but ρ differs. The paper does not ablate ρ.
- Expansion decisions use training microbatches. The validation batches are separate training microbatches, not a held-out dev set app. B.2. The paper does not say how many microbatches are used. Reproducing the acceptance decisions needs them.
- TGI details missing. The clipping range for γ is not stated §3.4. The damping λ = 0.1 is given only for commonsense app. D.1. The VTAB initialisation is a different procedure: it aggregates dense gradient proxies from three calibration batches, with O(nm) temporary memory and an aggregation rule that is not stated app. B.3 app. D.3.
- Probe cost is not counted. Each audit needs extra backward passes, and the paper gives no count for them app. B.3. TGI costs 2(p + 1) calibration backward passes app. B.3.
- Measured overhead is higher than FlexLoRA. On DTD, GEAR trains in 1060.83 s versus 1006.75 s for FlexLoRA, though with lower peak memory (4458.21 MiB versus 6553.87 MiB) tab. 4. This is one task, and the hardware is not stated.
- Parameter counts are not fully defined. The paper gives no counting rule. The LoRA baseline's 41.9M at reference rank 16 suggests the usual r(n + m) per adapted matrix, but this is inferred. Final per-module ranks are not reported. Probe variables are excluded from the counts app. B.3.
- Baseline settings are not stated. The paper says all methods use consistent settings §4. It gives no learning rates, schedules or re-run details for the baselines. Whether baseline numbers are re-run or copied from earlier reports is not stated.
- No spread is reported. Results are means over three seeds with no standard deviations and no significance test tab. 1 tab. 2 tab. 3. Margins such as 0.78 on GLUE and 0.93 on VTAB cannot be judged for noise from the paper alone.
- One average does not reproduce. Averaging the eight printed FlexLoRA GLUE scores gives 88.86, not the 88.85 printed in tab. 2. This is probably rounding of unrounded seed means. The other averages checked (LLaMA3-8B and Qwen2.5-7B for GEAR, GEAR on GLUE, VTAB, and the ablation rows) reproduce from the printed values.
- Ablation is narrow. Expansion is only tested on top of manifold optimisation. There are no rows for expansion alone or expansion with TGI but without manifold optimisation. The ablation covers LLaMA3-8B only tab. 10 app. E.
- The audit gradient comes from calibration data. Rank audits use 128-example calibration batches app. D.1, while candidate validation uses training microbatches app. B.2. The two sets are not related in the paper.
Lineage
- Builds on: LoRA's low-rank factor parameterisation §1; the geometry of fixed-rank matrices §3.2; randomized range finding for the normal sketch, cited as Halko et al. §3.3.
- Related methods discussed: AdaLoRA, DyLoRA, SoRA, FlexLoRA, GeoLoRA, GeLoRA, Riemannion, Manifold-LoRA, DoRA, PiSSA and LoRA-GA §2.
- Compared against in the experiments: LoRA, AdaLoRA, Riemannion, GeoLoRA and FlexLoRA §4.
- Superseded by: not known at the time of compiling.
Source and license
Adapted from “From What Is Used to What Is Missing: Rethinking Dynamic Rank in LoRA” by Peng Xue, Guodong Fan, Guang-Yong Chen, Xing Chen (arXiv:2611.07349), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.