wikillm

Paper · 2026-10-16 · cs.LG

How Finite-Rank Bottlenecks Shape the Low-Rank Adaptation Landscape

Theory of LoRA without NTK linearisation. With the backbone frozen, the loss sees a weight update only through a finite-rank map. Trace-norm regularisation then gives minimisers whose rank is capped by that map's rank. Attention gets sharper caps. Perturbed factorised LoRA has rank-deficient stationary points. Theory only, plus one small GLUE check.

Low-rank adaptation (LoRA and variants)Nuclear-norm regularizationNeural tangent kernelAttention mechanismparameter-efficient-fine-tuningtheorylow-rank

arXivPDFMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Fine-tuning a large pretrained model usually means adding a small trainable correction to its weight matrices. LoRA does this with a correction that has low rank, meaning it can only push the weights along a few directions. People pick that rank by hand, and it was not clear why such a small correction is enough.

The paper's answer is that, once the rest of the network is frozen, the training loss only "sees" the correction through what the frozen inputs can reveal. If the inputs span only a few directions, many different corrections give exactly the same outputs. Among those, a penalty that favours small nuclear norm picks one with low rank. The number of directions that matter is set by the frozen inputs.

The paper proves this for a single layer, for several stacked layers, and for attention, where two extra symmetries make the bound tighter. It also shows that, for the factorised LoRA training objective with a tiny random perturbation added, the points where training stalls do not use the full rank budget. These are theory results. The only experiment is a small check on three GLUE tasks, and the paper releases no code.

TL;DR

  • With a frozen backbone, the loss depends on a weight update only through a linear map determined by the frozen features; the rank of that map (the bottleneck rank) caps the rank needed.
  • Trace-norm (nuclear-norm) regularised fine-tuning has a global minimiser whose rank obeys r(r+1)/2 ≤ bottleneck rank §4.1.
  • Attention layers get sharper per-head caps for query, key and value updates §4.2.
  • After a small generic PSD perturbation, first-order stationary points of factorised LoRA are rank-deficient above a threshold §4.3. The paper does not show that they are global minima.
  • Relevant for theory readers; not a training recipe. Results are existence and bounds, not convergence guarantees §6.

Problem

LoRA writes the adapted weight as W = W0 + U V^T with a chosen rank r §1. The paper asks when the regularised fine-tuning objective itself admits a low-rank global minimiser, and what controls its rank §1. Existing analyses that answer nearby questions rely on the neural tangent kernel (linearisation around the pretrained weights) or on specific training dynamics §1, §2. The paper claims its arguments need no linearisation §1, §4.

Method

Setup. One adapted block has a frozen matrix W0 (m × n) and an update T (m × n) §3. Given N training samples, the frozen input features are stacked as H = [h_1, …, h_N] (n × N) §3. The network output on sample i is a function of T h_i alone, so across the training set the loss is a function of the single map Φ(T) = T H §3.

The bottleneck rank is d_Φ = rank(Φ) = dim range(Φ) §3, Def. 3.1. For Φ(T) = TH it equals m · rank(H) §3.

The trace-norm problem is min over T of L_vanilla(T) + λ‖T‖_* with λ > 0 eq. 3.1 §3. The rank-r LoRA objective adds weight decay to both factors:

\mathcal{L}^{\mathrm{lora}}_\lambda(U,V) = \mathcal{L}^{\mathrm{vanilla}}(UV^\top) + \tfrac{\lambda}{2}\left(\|U\|_F^2 + \|V\|_F^2\right) §3

By the nuclear-norm/balanced-weight-decay equivalence, its minimum equals the trace-norm minimum restricted to rank ≤ r §3. So if the unrestricted problem has a minimiser of rank ≤ r, rank-r LoRA loses nothing §3.

Steps.

  1. Freeze the backbone and compute H for the adapted block, with the forward pass deterministic per sample (no dropout, no batch-norm statistics) §3, Assumption 2.
  2. Compute the bottleneck rank d_Φ = m · rank(H) for a linear block §3, Def. 3.1.
  3. The single-block theorem: the trace-norm problem has a global minimiser. For any global minimiser T̃, there is another global minimiser T* with Φ(T*) = Φ(T̃) and r(r+1)/2 ≤ d_Φ, where r = rank(T*). Hence r ≤ min{m, n, ⌊(√(8 d_Φ + 1) − 1)/2⌋} §4.1, Thm. 4.1.
  4. Stacked layers: for L adapted stages, each block has a stage-wise bottleneck d_{t,S}(z) (taken at the reachable upstream state z) and an upper envelope d^max_{t,S}. A recursive budget β_{L+1} = 0 and β_t = β_{t+1} + Σ_S d^max_{t,S} is defined §3. The stage theorem states that a global minimiser exists with r_{t,S}(r_{t,S}+1)/2 ≤ d_{t,S}(z*{t+1}) ≤ d^max{t,S} for every stage and block §4.1, Thm. 4.3. Setting L = 1 recovers the single-block theorem §4.1.
  5. Attention. Each head a has adapted Q, K and V blocks; the output is O_i = Σ_a W_O^(a) Ṽ_i^(a) (A_i^(a))^T, with A_i^(a) = softmax_row(Q̃^T K̃ / √d_k + C_i^M) §3. Two invariances apply:
    • Score centring. Row-wise softmax ignores additive row constants, so Q and K updates are observed only through a centred subspace, which removes one key dimension per sample §4.2, Lemma in app. C.2.
    • Value compression. Value updates are seen only after attention-weighted pooling (rank of A) and after the frozen output projection W_O §4.2. The projector-compression principle says that if Φ(T) = Φ(P_L T P_R) for orthogonal projectors, a projected minimiser is also optimal and has rank at most min{rank P_L, rank P_R} §4.2, app. C.1.
  6. Attention rank caps (Thm. 4.4). At a rank-minimal global minimiser, with all features and attention matrices evaluated at that same minimiser:
    • query: rank ≤ min{d_k, rank(H_Q*), rank(H_K*), Σ_i (ϱ_K,i − 1)};
    • key: the same four terms as the query cap, in the same form §4.2;
    • value: rank ≤ min{d_v, rank(H_V*), Σ_i rank(A_i*)} §4.2, Thm. 4.4. ϱ_K,i is the number of key positions of sample i §3. Every attention row needs at least one unmasked key §4.2.
  7. Adapting the output projection too (Thm. 4.5). Query and key caps are unchanged. The value cap is as above. The output-projection update obeys rank ≤ min{d_out, |A| d_v, Σ_i ϱ_Q,i} §4.2, Thm. 4.5. The output-projection update is handled by conditioning on the other blocks, so the generic result (step 8) does not apply to it §4.2.
  8. Stationary points. The factorised objective with a PSD perturbation P ∈ S^{m+n}_+ is

\mathcal{L}^{\mathrm{lora}}_{\lambda,P}(U,V) = \mathcal{L}^{\mathrm{vanilla}}(UV^\top) + \tfrac{\lambda}{2}\left(\|U\|_F^2 + \|V\|_F^2\right) + \tfrac12 \langle P, R R^\top\rangle, \quad R = [U; V] §4.3, Def. 4.6

For almost every P, every R satisfying the stationarity equation with some G in a family of dimension at most p is rank-deficient whenever r(r+1)/2 > p §4.3, Thm. 4.7. For attention, the corollary requires r_{t,α}(r_{t,α}+1)/2 > β_{t+1} + m_{t,α} · r^max_{H,t,α} for every block α in stage t, and then, for almost every PSD perturbation, all first-order stationary points of the perturbed objective fall short of the rank budget §4.3, Cor. 4.8. The paper states the perturbation can be taken arbitrarily small §4.4. 9. Limit of step 8. For any rank r ≥ 2, λ > 0 and ε ∈ (0, 1), the paper builds a smooth nonconvex loss for which every perturbation with Frobenius norm at most ε leaves the origin as a sharp local optimum that is not the global one §4.3, Prop. 4.9. So first-order exclusion does not give the absence of spurious local minima §4.3. 10. Transfer. For a perturbed global minimiser, the regularised objective exceeds the trace-norm optimum by at most (1/2) Σ_j ⟨P_j, R_j* R_j*^T⟩ §4.4, Thm. 4.10. If ‖P_j‖2 ≤ ε_j, the gap is at most Σ_j ε_j ‖T_j*‖* §4.4, Thm. 4.10. 11. Generalisation. Over the trace-norm ball of radius D, with bounded loss B, Lipschitz constant L_ℓ and bounded features R_h, with probability at least 1 − η,

Pseudocode
sup_T (L(T) − L̂_S(T)) ≤ 2 𝔑_S(F_D) + 3B √(log(2/η) / (2N)),

and 𝔑_S(F_D) ≤ (√2 L_ℓ L_ψ(D, R_h) D / N) · E_σ‖E H^T‖_2 [app. B] [§4.4]. The matrix E holds independent Rademacher signs. The bound depends on the spectral norm of the random-sign-weighted features, not on the parameter count. The L-stage attention version is in app. B [§4.4].

Key equations.

\min_{T}\; \mathcal{L}^{\mathrm{vanilla}}(T) + \lambda\|T\|_\star \qquad [\text{eq. 3.1}]

\frac{\operatorname{rank}(T^\star)(\operatorname{rank}(T^\star)+1)}{2} \le d_\Phi \qquad [\text{§4.1, Thm. 4.1}]

Pseudocode, written from the paper's text:

Pseudocode
# Bottleneck rank of one frozen linear block  [§3, def. 3.1]
# Input:  H (n, N) frozen input features; m output dimension
# Output: d_phi, the dimension of the space the loss can observe

def bottleneck_rank(H, m):
    # Loss depends on T only through T @ H  [§3]
    r_H = numerical_rank(H)  # tolerance not stated in the paper
    # Range of T -> T @ H has dimension m * rank(H)  [§3]
    return m * r_H
Pseudocode
# Largest rank a global minimiser needs for one block  [§4.1, thm. 4.1]
# Input:  d_phi (bottleneck rank); m, n (matrix shape)
# Output: integer rank cap

def rank_cap(d_phi, m, n):
    # Barvinok-Pataki: r * (r + 1) / 2 <= d_phi  [§4.1]
    r_max = math.floor((math.sqrt(8 * d_phi + 1) - 1) / 2)
    return min(m, n, r_max)
Pseudocode
# Per-head rank caps for attention  [§4.2, thm. 4.4]
# All ranks are evaluated at the same minimiser; the caps are a posteriori.
# Input:  d_k, d_v (head dims); rank_HQ, rank_HK, rank_HV (ranks of the
#         frozen token matrices); rho_K (key count per sample);
#         attn_ranks (rank of each sample's attention matrix)
# Output: caps for the query, key and value updates

def attention_rank_caps(d_k, d_v, rank_HQ, rank_HK, rank_HV,
                        rho_K, attn_ranks):
    # Score centring removes one key dimension per sample  [§4.2]
    centred_keys = sum(rho - 1 for rho in rho_K)
    query_cap = min(d_k, rank_HQ, rank_HK, centred_keys)
    key_cap = query_cap
    # Value update seen only after attention pooling and W_O  [§4.2]
    value_cap = min(d_v, rank_HV, sum(attn_ranks))
    return query_cap, key_cap, value_cap
Pseudocode
# Perturbed factorised LoRA objective  [§4.3, def. 4.6]
# Input:  U (m, r), V (n, r) factors; P (m + n, m + n) PSD perturbation
# Output: scalar loss

def perturbed_lora_loss(U, V, P, lam, vanilla_loss):
    # Stack the two factors into R  [§4.3]
    R = concatenate([U, V], axis=0)              # (m + n, r)
    # Data loss, weight decay on both factors, perturbation term  [§4.3]
    data_term = vanilla_loss(U @ V.T)
    decay_term = (lam / 2) * (norm(U) ** 2 + norm(V) ** 2)
    perturb_term = 0.5 * trace(P @ (R @ R.T))
    return data_term + decay_term + perturb_term
Pseudocode
# Threshold proxy for one adapted block, as described in the text  [app. D]
# Input:  up_features (upstream frozen features); down_observed (the
#         downstream observable subspace after the trace-norm proxy fit)
# Output: numerical rank estimate for this block

def numerical_rank_gram(features, tol):
    # rank(H) equals rank(H H^T); count eigenvalues above tol  [app. D]
    gram = features @ features.T
    eigs = eigvalsh(gram)
    return count(eigs > tol)

def predict_threshold(blocks):
    # Threshold = max over blocks of min(upstream, downstream)  [app. D]
    best = 0
    for block in blocks:
        r_up = numerical_rank_gram(block.up_features, 1e-4)
        r_down = numerical_rank_gram(block.down_observed, 1e-4)
        best = max(best, min(r_up, r_down))
    return best

Hyperparameters

The paper gives no optimiser, epoch count, LoRA rank sweep, scaling factor or seed for the main LoRA runs. These are "not stated".

NameValueWhereNotes
Batch size (LoRA; all tasks)8tab. 4
Learning rate, "Head" (MNLI / RTE / SST-2)5e-4 / 1e-4 / 1e-4tab. 4Row labelled "Head"; read as the classification head from app. D text (inference)
Learning rate, "Attention" (MNLI / RTE / SST-2)8e-5 / 1e-3 / 5e-4tab. 4Row labelled "Attention"; read as the adapted block (inference)
LoRA regularisation λ1e-4 (all tasks)tab. 4Applied as λ/2 on each factor §3
Batch size (full fine-tuning; all tasks)8tab. 4
Learning rate, full fine-tuning (MNLI / RTE / SST-2)2.32e-5 / 3.375e-4 / 2e-4tab. 4
Trace regularisation λ (full fine-tuning)1e-4 (all tasks)tab. 4
ModelQwen2.5-1.5B§5Open checkpoint, as cited
Adapted blockone query projection, one head, layers 7 and 8§5
Head dimension d_k128§5Stated for Qwen2.5-1.5B
Classifier warm-up10 epochs, then frozenapp. DBackbone and classifier are frozen for the LoRA sweep §5
Samples, high-rank bound case100app. DTask assignment not stated
Samples, low-rank bound case4app. DTask assignment not stated
Proxy rank128app. DChosen to cover the query head
Proxy λ1e-4app. DSame objective as the LoRA sweep
Proxy optimiserAdamW, learning rate 5e-4app. D
Proxy checkpoint choicelowest training L_loraapp. D
Numerical-rank tolerance1e-4 on eigenvalues of H H^Tapp. DAbsolute threshold; see Pitfalls
Perturbation Pnot used in the experiments§5, app. DThe experiment text does not mention P

Results

The paper reports no numeric headline results. Author-reported findings, on a small GLUE check (SST-2, MNLI, RTE) using Qwen2.5-1.5B §5:

  • Loss curves for LoRA runs at several ranks show a cut-off. Runs below the estimated rank take separate paths and end at a worse loss. Runs at or above it almost lie on top of each other and end near the full fine-tuning loss fig. 1 §5.
  • The figure has one panel per task, each with single optimisation runs and no error bars fig. 1 [checklist].
  • The threshold is a proxy from the attention bound in §4.2. The paper gives no threshold values in the text, so none are quoted here.

Theory results (each stated in §4; app. C has the proofs):

ResultStatementSource
Single-block existenceGlobal minimiser of rank r with r(r+1)/2 ≤ d_Φ§4.1, Thm. 4.1
Stacked layersPer-stage caps with the β budget§4.1, Thm. 4.3
Attention capsQuery, key, value caps as in step 6§4.2, Thm. 4.4
Adapted output projectionOutput-projection cap§4.2, Thm. 4.5
Generic exclusionFull-rank stationary points excluded for almost every P§4.3, Thm. 4.7
Attention versionCorollary for L attention modules§4.3, Cor. 4.8
LimitationSpurious local minima can persist§4.3, Prop. 4.9
TransferNear-optimality of perturbed minimisers§4.4, Thm. 4.10
GeneralisationBound governed by E‖E H^T‖_2§4.4, app. B

Paper vs code

No official code. Searches (done for this card): the arXiv abstract page lists no code, project or demo link. The paper has no code link. The NeurIPS checklist states that the code is not yet released. GitHub repository searches for "bottleneck rank lora" and "trace-norm lora perturbation" returned third-party repositories owned by people who are not authors. "finite-rank lora" returned nothing. The author-name search found no repository owned by any author. No repository was cloned.

Pitfalls

  • Existence, not convergence. The rank theorems are about global minimisers of the trace-norm problem. The paper says gradient-based LoRA is not shown to reach them §6. A minimiser that exists is not a training outcome.
  • Bounds are a posteriori. The attention caps use ranks of H and of the attention matrices evaluated at the minimiser that is being bounded §4.2. You cannot compute them before training. The paper says so §6. The experiment uses a proxy instead app. D.
  • Exclusion is first-order only. Theorem 4.7 excludes full-rank stationary points for almost every P. It does not exclude spurious local minima. Proposition 4.9 gives a counterexample with rank r ≥ 2 §4.3.
  • The perturbation is not used in practice. The paper gives no recipe for sampling P or choosing its scale. The experiment's loss has no perturbation term app. D. Treat Theorem 4.10 as an analysis tool.
  • "Generic" means almost every P. The choice of P is a measure-zero issue in theory. No finite test can confirm it.
  • Related-work comparison uses a symbol not defined in the theorems. Section 2 compares an attention rank scale of min{K_out, N} with the earlier analysis §2. K_out does not appear in the theorems, and the main text does not derive that scale §2, §4.2. Remark 4.2 gives the counting scale only when the observed output dimension is small §4.1. Use the theorem caps, not that comparison.
  • Stationarity corollary condition. The text says the family dimension is at most β_{t+1} + d^max_{t,S} §4.3. Corollary 4.8 uses β_{t+1} + m · r^max_H instead §4.3. The corollary's version is a larger (stricter) quantity, so its condition is harder to meet. The paper does not reconcile the two.
  • Two invariances, one caveat. The output-projection update is not covered by the stationary-point result §4.2. Do not extend Corollary 4.8 to W_O without checking the proof.
  • Balanced weight decay. The factorised LoRA loss puts λ/2 on each factor, while the trace-norm objective uses λ on the nuclear norm §3. The equivalence needs the balanced factorisation. Implementers must not use λ on the product or on one factor only.
  • Attention needs unmasked keys. Every row of the attention matrix needs at least one unmasked key §4.2. Padding masks that leave no key for some row break the setting.
  • Deterministic features. The theory assumes no dropout and no batch-norm statistics, so H must come from a deterministic forward pass §3.
  • Threshold definition is inconsistent. Section 5 says the threshold is the smallest rank where both proxies saturate. The formula in app. D is the maximum over blocks of the minimum of upstream and downstream ranks. "Saturate" is not defined. Which one the paper used is unclear.
  • Numerical rank is scale-dependent. The rank of H is taken as the count of eigenvalues of H H^T above an absolute 1e-4 app. D. The paper does not say whether features are normalised. The result changes with feature scale.
  • Downstream observable subspace not specified. The text says it is estimated after the trace-norm proxy fit app. D. It does not say how the subspace is extracted.
  • Learning-rate rows are ambiguous. The table's "Head" and "Attention" rows are not defined in the table. The text suggests the classifier and the adapted block, but the mapping is inferred tab. 4 app. D.
  • Sample counts per task are missing. The 100-sample and 4-sample settings are not tied to tasks app. D.
  • Experiment is small and single-run. One adapted block, one model, three tasks, single runs, no error bars §5 fig. 1 [checklist]. The paper reports no GPU, memory or run-time figures [checklist].
  • Unstated for implementers: optimiser and schedule for the main LoRA sweep, training length, the LoRA rank values, the scaling factor, sequence length, seeds, and the exact definition of "saturation". Do not fill these from other papers.

Lineage

  • Builds on: linearised (NTK) analyses of LoRA, notably Jang et al. (2024) §2 [neural-tangent-kernel].
  • Relates to: LoRA training-dynamics and landscape work by Kim et al., Xu et al., Liu et al., Zhang et al. §2; linearisation analysis by Afzal et al. §2.
  • Extends: the nuclear-norm and low-rank matrix-recovery tools of Recht et al. and the Barvinok–Pataki rank bound §3, §4.1.
  • Superseded by: not known from this paper; no later version is recorded in this wiki.

Source and license

Adapted from “How Finite-Rank Bottlenecks Shape the Low-Rank Adaptation Landscape” by Long Nguyen-Chi, Quynh Nguyen, Thanh Nguyen-Cung, Binh T. Nguyen (arXiv:2610.23343), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.