wikillm

Paper · 2026-10-05 · cs.LG

A Fine-Grained Analysis of the LoRA Fine-Tuning Landscape with Implications for Data Selection

Ties LoRA's adapter rank to a data-dependent restricted-isometry constant, LoRA-RIP. When r/r* exceeds (2δ/(1−δ))², second-order stationary points that meet the curvature assumption are globally optimal, beyond the classic δ = 1/3 limit. Data covering weak directions and avoiding saturated examples lowers δ. MNLI and CIFAR-100 results are author-reported.

Low-rank adaptation (LoRA and variants)Over-parameterizationDataset curationloraparameter-efficient-fine-tuningdata-selectionlandscape-theory

arXivPDFMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

LoRA fine-tunes a large model by training a small add-on matrix instead of the whole model. A basic open question is how large that add-on should be, which is its rank. If the rank is too small, training can settle into poor solutions. If it is too large, you give up the savings that made LoRA worth using.

The authors argue that the right rank depends on the training data as well as on the model. They define one number, LoRA-RIP, that measures how well the data shapes the training problem so that it is easy to optimise. When the adapter is larger than the rank the best solution needs, by a factor set by that number, every stable solution training can reach is the best one, so there are no traps to get stuck in. Data that makes the problem better behaved lets you use a smaller adapter.

The same reasoning suggests a way to choose data. Spread the budget across groups of similar examples so that no group dominates, and avoid examples the model already predicts with near certainty, since they add little useful signal. The experiments cover one language task and one image task. They are consistent with the theory, but the central constant is never computed.

TL;DR

With enough rank over the target rank, LoRA's cross-entropy landscape has no bad second-order stationary points. The required ratio is set by the data-dependent LoRA-RIP constant δ, through r/r* > (2δ/(1−δ))² §4.2 eq. 6. Unlike the classic result, the guarantee also holds for δ above 1/3 §1. The paper derives data-side bounds on δ based on coverage of low-rank directions, softmax saturation and residual curvature §5.1 §5.2. Small MNLI and CIFAR-100 experiments are consistent with these predictions §6, but δ, α and β are not computed there app. E.1. No official code was found.

Problem

Choosing the adapter rank is a trade-off. An overly small rank may leave the optimisation poorly conditioned. An unnecessarily large rank costs the efficiency that motivates LoRA §1. The paper cites a report in which raising the rank from 16 to 32 lowered mean accuracy on eight commonsense tasks from 81.8% to 80.3% §1. Adaptive rank methods still need a prespecified budget §1.

Prior theory is limited. NTK-regime results guarantee no spurious minima only for a linearised model §1 §2. A RIP-type analysis under cross-entropy shows global optimality only when the RIP constant is below 1/3, and leaves the CE-specific form of the constant and the role of rank beyond that regime open §1. Classical RIP results assume a squared loss whose Hessian is constant. Under cross-entropy the Hessian changes from point to point, so the classical argument does not carry over directly §3.

Method

Setup. The frozen weight is W0, and the update is M = UV^T with U of size n1×r and V of size n2×r. The LoRA scale is absorbed into the factors §3. The empirical loss L̂(M) is averaged over N training examples §3. The standard regularised objective puts weight decay λ on both factors eq. 1. The balanced objective adds a term μ/4 ‖UᵀU − VᵀV‖²_F with μ > 0. The factorisation is not unique, and the balance term is a device for the analysis §3 eq. 2.

Target assumption. Let F_λ(M) = L̂(M) + λ‖M‖_* be the matrix-lasso form of the objective. Assumption 3.1 states that a global minimiser M*_λ of F_λ exists, has rank r*, and that r ≥ r* §3 eq. 3. The paper shows the infimum over factors equals the infimum over matrices of rank at most r §3 app. B.2.

Curvature assumption. Assumption 4.1 is taken for a rank order k ≥ r + r*. On the segment from M* to M = UV^T, the loss has curvature bounds 0 < α_k ≤ β_k in every direction of rank at most k §4.1 eq. 4. The LoRA-RIP constant is δ_k = (β_k − α_k)/(β_k + α_k) §4.1.

Main guarantee. Write X = [U; V], and let g(X) = 4h(U,V) with the augmented lifted error E = XXᵀ − ZZᵀ §4.2 eq. 5. Theorem 4.2 states the following. Take k ≥ r + r* and μ = α_k/2. Every second-order stationary point (SOSP) (U,V) of the balanced objective, whose product satisfies Assumption 4.1 with these constants, is globally optimal with UVᵀ = M*_λ, whenever r/r* > (2δ/(1−δ))² §4.2 eq. 6. The theorem also gives XXᵀ = ZZᵀ §4.2. The landscape-wide claim needs the same bounds for every candidate of rank at most r §4.2.

Rearranging eq. 6 gives other forms. For fixed rank, the condition is δ < √(r/r*)/(2 + √(r/r*)) §4.2. At δ = 1/3 the ratio threshold equals 1, so the δ > 1/3 regime requires r > r* [derived from eq. 6]. The paper's lemma states the same condition as (β − α)/β < 1/(1 + √(r*/r)) §4.2.

Proof idea. The proof builds a family of certificates, a combination of a gradient test and a Hessian test. At an SOSP each one is nonnegative, but the paper shows that some combination is strictly negative whenever E ≠ 0 §4.2. The error splits into a tangent part E_T and a normal part E_N §4.2. A direction D_0 tests the tangent part, and directions D_i, built from the orthogonal complement of the current span, test the normal part. Extra LoRA directions beyond r* make one certificate negative §4.2.

Transfer to standard LoRA. Any first-order stationary point of the standard objective eq. 1 satisfies UᵀU = VᵀV. So its SOSPs are SOSPs of the balanced objective §4.2 [prop. 4.3]. The same conclusion holds for standard LoRA SOSPs §4.2 [cor. 4.4].

General balancing weight. For 0 < μ < α_k, define m_μ = 2min{μ, α_k − μ}, L_μ = 2max{μ, β_k − μ} and κ_μ = L_μ/m_μ. The rank condition becomes r/r* > ((κ_μ − 1)/2)² app. B.3.2. At μ = α_k/2 this reduces to eq. 6 app. B.3.2.

Data side. Let ν be the empirical distribution over training instances. Reweighting, duplicating or changing the dataset changes ν, and so the loss L̂_ν(M) = E_{x∼ν}[−⟨q_x, log softmax(z_x(M))⟩] §5.1 eq. 7. Here z_x(M) is the logit vector and q_x the target distribution §5.1.

The first-order logit response is A_{x,M}[H] = Dz_x(M)[H]. Cross-entropy ignores a constant added to all logits, so only the centred response P⊥A matters, with P⊥ = I − 11ᵀ/K §5.1. Coverage of a direction is Q_ν(M,H) = E‖P⊥A_{x,M}[H]‖² §5.1.

The Hessian of the loss splits into a Gauss–Newton term G_ν and a residual term R_ν §5.1 eq. 8. Here G_ν = E⟨A, ΩA⟩ with the softmax covariance Ω = diag(p) − ppᵀ, and R_ν = E⟨p − q, B[H,H]⟩ with B the second-order logit response §5.1. A logit map that is linear in M would make the residual term vanish §5.1.

Over the admissible family 𝒯_k(ν,r) app. C.1, the data-side constants are as follows. c is the minimum coverage and C the maximum coverage. ω⁻ and ω⁺ bound the softmax weighting, and ρ bounds the residual §5.1. The restricted observability condition is c > 0, 0 < ω⁻ ≤ ω⁺ < ∞ and ρ < ω⁻c §5.1. This gives α^CE = ω⁻c − ρ and β^CE = ω⁺C + ρ §5.1, and the conservative certificate δ ≤ δ̄ = (ω⁺C − ω⁻c + 2ρ)/(ω⁺C + ω⁻c) eq. 9.

For a rank-one direction H = uvᵀ, coverage equals E[⟨v, a_x⟩² ‖P⊥J_x u‖²], with a_x the input hidden state and J_x the Jacobian to the logits §5.2 eq. 10. For a mixture over groups, Q is the weighted sum of the group coverages. The floor c and ceiling C do not decompose linearly across groups §5.2.

If the top probability is 1 − ε, the largest eigenvalue of the softmax covariance is at most 2ε − ε² §5.2. Confident examples therefore contribute little softmax-weighted curvature §5.2.

Selection rules §5.1 §5.2:

  • Coverage: prefer data that raises the weakest observed direction c, and avoid piling curvature on a few dominant directions, which raises C.
  • Saturation: favour examples with non-trivial softmax curvature. Very ambiguous examples can concentrate curvature or be label noise, so the rule is not simply "hardest first".
  • Duplicates and imbalance reinforce directions that are already visible, which hurts the certificate §5.1.
  • Experiments use gradient sketches as a practical proxy for the coverage operator, and the appendix argues they are aligned with it §5.2 app. C.5. This alignment is argued, not measured.

Pseudocode for the balanced objective §3 eq. 2:

Pseudocode
# Balanced LoRA objective  [§3] [eq. 2]
# Input:  factors U (n1, r), V (n2, r); balance weight mu; decay lam
# Output: scalar objective h(U, V)

def balanced_objective(U, V, mu, lam):
    # Adapter update M = U V^T, added to the frozen weight W0  [§3]
    update = U @ V.T                              # (n1, n2)
    # Average training loss of the adapted model  [§3]
    data_loss = mean_loss(W0 + update, examples)
    # Balance term pulls U^T U toward V^T V  [§3] [eq. 2]
    gap = U.T @ U - V.T @ V                       # (r, r)
    balance = (mu / 4) * sum_of_squares(gap)
    # Weight decay on both factors  [§3] [eq. 2]
    decay = (lam / 2) * (sum_of_squares(U) + sum_of_squares(V))
    return data_loss + balance + decay

Pseudocode for the probe-then-LoRA protocol of the experiments app. D.1 §6:

Pseudocode
# Probe-then-LoRA protocol used in the experiments  [app. D.1]
# Input:  frozen backbone, labelled training split, adapter rank
# Output: trained LoRA factors on one pre-specified query projection

def probe_then_lora(backbone, train_split, rank):
    # Train a linear head on the frozen backbone  [app. D.1] [app. D.2]
    head = train_linear_probe(backbone, train_split)
    # Keep the head fixed in the LoRA stage  [app. D.1]
    freeze(head)
    # Attach LoRA to the query projection of block 10  [§6] [app. D.1]
    adapter = attach_lora(backbone, block=10, rank=rank)
    # Factor A: Kaiming init with scale 0.01; factor B: zeros  [app. D.1]
    init_factors(adapter, scale=0.01)
    # Adam with explicit weight decay, learning rate 1e-4  [app. D.1]
    train(adapter, head, train_split)
    return adapter

Pseudocode for gradient-sketch groups in the MNLI coverage runs app. D.4:

Pseudocode
# Gradient-sketch groups for the MNLI coverage runs  [app. D.4]
# Input:  candidate pool; reference model (frozen backbone + probe head)
# Output: 8 groups per label (24 in total)

def gradient_sketches(pool, reference, proj_dim=16):
    # Margin gradient factors at the adapted matrix  [app. D.4 step 2]
    factors = margin_gradient_factors(reference, pool)
    # Random projections of each factor to proj_dim  [app. D.4 step 3]
    sketches = []
    for activation, grad_out in factors:
        sketch = outer(proj_a @ activation, proj_g @ grad_out)
        sketches.append(sketch.flatten())         # (256,)
    # Normalise the sketches  [app. D.4 step 4]
    return normalise(sketches)

Pseudocode for group-balanced subset sampling app. D.4:

Pseudocode
# Group-balanced subset sampling  [app. D.4 step 5]
# Input:  groups per label; group decay (None for balanced)
# Output: subset of 17,526 examples with label counts 5847/5788/5891

def sample_subset(groups_by_label, decay):
    # Mass across the 8 groups in each label  [app. D.4]
    # Balanced: uniform; long-tail: geometric with the decay factor
    mass = group_mass(groups_by_label, decay)
    # Draw without replacement under the fixed label counts  [app. D.4]
    return draw_without_replacement(groups_by_label, mass)

Pseudocode for saturation-band selection app. D.5:

Pseudocode
# Saturation-band selection inside one label-gradient group  [app. D.5]
# Input:  reference-model probabilities for the group's examples
# Output: a subset drawn from one quantile band of the score

def pairwise_score(p_gold, p_alt):
    # Gold probability against the strongest non-gold class  [app. D.5]
    total = p_gold + p_alt
    p_pair = p_gold / total
    # Softmax curvature weight of the pairwise view  [app. D.5]
    return p_pair * (1 - p_pair)

def select_band(scores, band, n):
    # Sort examples by score inside the group  [app. D.5] (direction inferred)
    ranked = argsort(scores)
    # Keep examples inside the quantile band  [app. D.5]
    # easy (0.10, 0.45); middle (0.33, 0.67); boundary (0.62, 0.95)
    in_band = quantile_slice(ranked, band)
    # Draw n examples from the band without replacement  [app. D.5]
    return sample_without_replacement(in_band, n)

Hyperparameters

Values below are the paper's experimental settings, not theory constants. Theory constants (α, β, δ, r*) are not computed app. E.1.

NameValueWhereNotes
Adapted matrixquery projection in block/layer 10§6 app. D.1Same block in MPNet-base and ViT-B/16
Ranks swept1, 4, 16, 32, 64, 128, 256, 512§6 app. D.1Rank sweep only
Weight decayλ_A = λ_B = 1e-4app. D.1Separate values for the two factors
Balance coefficient μ1e-4app. D.1Not the μ = α/2 of Theorem 4.2
Learning rate1e-4app. D.1Adam with explicit weight decay; schedule, warm-up not stated
Factor A initKaiming, "scale 0.01"app. D.1Meaning of scale not stated
Factor B initzerosapp. D.1
Scale γabsorbed into factors§3No numeric value given
Linear headtrained on frozen backbone, then fixedapp. D.1Not updated during LoRA stage
MNLI split353,432 / 39,270 / 9,815app. D.1train / validation / test
MNLI batch, max length32, 128app. D.1
MNLI epochs10 (rank sweep); 50 (interventions)app. D.1 app. D.2Probe trained for 10 epochs at batch 32 app. D.2
CIFAR-100 split45,000 / 5,000 / 10,000app. D.1train / validation / test
CIFAR-100 batch, epochs64; probe 20, LoRA 30app. D.2
Split seed, optimisation seed17, 0app. D.1Multi-seed runs use seeds 0, 1, 2 app. E.1
Coverage subset17,526 examples; labels 5,847 / 5,788 / 5,891app. D.4No replacement
Gradient sketchrandom projection, 256-dim bilinearapp. D.4"16×16" wording; see Pitfalls
Gradient groups8 clusters per label, 24 in totalapp. D.4Clustering method not stated
Long-tail decays (MPNet)0.75 light, 0.45 heavyapp. D.4Ordering of groups not stated
Long-tail decays (DeBERTa)0.60 light, 0.45 heavyapp. E.10Different backbone
Saturation bandseasy [0.10, 0.45]; middle [0.33, 0.67]; boundary [0.62, 0.95]app. D.5Quantile bands, drawn without replacement
Ambient reference λ1e-4app. E.3Numerical reference, not a certified target
HardwareNVIDIA RTX 5880 Ada, 48 GB; under 4 GB per workerapp. D.2One process per GPU, up to four GPUs

Results

All numbers are author-reported. Main tables use one optimisation seed app. D. Three-run results are in the diagnostics app. E. The paper compares ranks and data conditions only. It does not compare against other LoRA methods or published baselines §6.

Rank sweep (MNLI with MPNet-base, CIFAR-100 with ViT-B/16). Best-validation values tab. D.3 tab. D.4:

RankMNLI val accMNLI test accMNLI val CECIFAR val accCIFAR test accCIFAR val CE
10.60470.60360.87370.73840.72431.2393
40.61990.61670.85010.73920.72671.2299
160.63600.63240.83180.74000.72511.2291
320.64760.64560.81300.73980.72651.2323
640.65470.65060.80180.74140.72791.2254
1280.66300.65700.78830.74140.72961.2236
2560.66960.66340.77520.74220.73181.2244
5120.67280.66860.76770.74640.73071.2217

The main text reports clear validation-CE gains on MNLI and smaller gains on CIFAR-100 §6 fig. 1. Three-run means give MNLI validation accuracy of 0.6199 ± 0.0009 at rank 4 and 0.6695 ± 0.0009 at rank 256. CIFAR-100 gives 0.7389 ± 0.0009 and 0.7429 ± 0.0010 tab. E.5.

Gradient coverage, MNLI (balanced versus long-tail group mass at fixed subset size; single seed) tab. D.6 fig. 2:

RankBalanced val / testLight long-tail val / testHeavy long-tail val / test
40.5939 / 0.59540.5868 / 0.58750.5853 / 0.5838
160.6046 / 0.60330.5955 / 0.59400.5961 / 0.5922
320.6110 / 0.60840.6076 / 0.60160.6026 / 0.5946
640.6134 / 0.61330.6145 / 0.60670.6103 / 0.6023

Balanced has the higher validation accuracy at ranks 4, 16 and 32. At rank 64 Light is slightly higher on validation and lower on test tab. D.6. The paper's three-seed runs at six ranks find Balanced above Heavy on validation at each rank, and Heavy below Balanced in all 12 paired runs with re-drawn subsets app. E.4 (text only, no table). The data diagnostics put Balanced at the lowest Gauss–Newton condition number (34.19 ± 1.17) and the lowest residual, against 38.60 ± 1.32 for Heavy tab. E.6.

Saturation bands, MNLI (fixed gradient-group quotas; single seed) tab. D.8 fig. 3:

RankEasy val accMiddle val accBoundary val acc
40.58530.59620.5945
160.59670.60910.6056
320.60320.61490.6130
640.61830.62260.6218

Three-seed means at rank 4 are 58.83 ± 0.47 for Easy, 59.58 ± 0.04 for Middle and 59.43 ± 0.04 for Boundary (percent) tab. E.9. Middle and Boundary beat Easy at all four ranks in that table, and their relative order changes with rank app. E.7.

Other settings app. E.8 tab. E.10 tab. E.11 tab. E.12:

  • DeBERTa-v3-base on MNLI: Balanced versus Heavy test accuracy is 45.04 versus 42.86 at rank 16 (+2.18 points) and 45.16 versus 43.54 at rank 64 (+1.62 points).
  • MPNet-base on QNLI: Balanced versus Heavy test accuracy is 72.16 versus 66.66 at rank 16 (+5.50 points) and 76.06 versus 69.27 at rank 64 (+6.79 points).
  • Adapter location on MNLI: Balanced minus Heavy test gain is +0.890 (layer 5 query), +1.063 (layer 7), +1.939 (layer 8) and +1.124 (all-layer query and value).

Matrix-space reference tab. E.4: the gap between the best observed objective and the numerical reference falls from 0.06450 at rank 4 to 0.00261 at rank 256 on MNLI, and from 0.04666 to 0.00038 on CIFAR-100. These references are numerical, not certified global targets app. E.1.

Paper vs code

No official code. The paper says that relevant code is in its supplementary materials §6. The arXiv source contains no code and no link to a repository. Checks run: the arXiv abstract page has no code link, and no project page is linked. GitHub searches gave no candidate from the authors: "LoRA-RIP" returned one unrelated repository, and "lora rip landscape", "lora rank landscape data selection" and "lora restricted isometry" returned nothing. No code-level discrepancy check was possible.

Pitfalls

  • The main theorem is conditional. It applies to SOSPs whose product satisfies the curvature bounds with the given α and β. Only when the same bounds hold for every candidate of rank at most r does it describe the whole factorised landscape §4.2. Read it as a condition on the solution, not a training guarantee.
  • The threshold cannot be checked from the experiments. The authors state they do not compute α, β, δ, r* or the data-side constants c, C, ω, ρ app. E.1. The experimental support is qualitative §6. Stationary points found by local optimisation are not certified global targets app. E.1.
  • Theorem and experiments use different balance weights. Theorem 4.2 uses μ = α/2, which requires α. The experiments use μ = 1e-4 app. D.1. The experiments also add the balance term, although the transfer result concerns the standard objective eq. 1 §4.2. Decide which objective you implement before comparing numbers.
  • Weight decay differs. The theory uses one λ on both factors eq. 2. The experiments use λ_A = λ_B = 1e-4 app. D.1. The theorem needs only λ > 0 §4.2.
  • Unstated training details. Not stated: learning-rate schedule, warm-up, dropout, gradient clipping, Adam betas and epsilon, the exact meaning of "Kaiming scale 0.01", and any numerical LoRA scale γ app. D.1 §3. The factor scale is absorbed §3, so an implementation that uses a standard α/r scale has to map it onto the factors.
  • Linear head is frozen during the LoRA stage. This departs from usual practice, where the classifier is trained with the adapter. Absolute accuracies are likely to differ from a standard setup app. D.1.
  • Reference model for data scores. Gradient sketches and saturation scores are computed at the probe-head model with zero LoRA update, not at the trained adapter app. D.4 app. D.5.
  • Gradient sketch is ambiguous. The text says the activation and output-gradient factors are projected "to 16×16", yet the sketch is 256-dimensional. The reading in the pseudocode (16-dimensional projections whose outer product is 16×16) is inferred. The projection seeds, pool size and clustering method are not stated app. D.4.
  • Saturation band direction is inferred. The paper does not say whether quantiles are taken within each group, or in which direction the score is sorted. The score means in tab. D.7 (Easy lowest, Boundary highest) are consistent with ascending order within each group app. D.5.
  • Long-tail group order is unstated. The geometric decays need an ordering of the eight groups per label, which the paper does not give app. D.4. The decay values also differ across backbones (0.75 for MPNet, 0.60 for DeBERTa) app. D.4 app. E.10.
  • Small differences. Several gaps are about 0.1 points or less, such as Light versus Balanced at rank 64 tab. D.6. The paper reports no significance tests. Its multi-run spread is 0.05 to 0.14 points for rank runs app. E.4. Single-seed differences should not be read as ranking the conditions.
  • Non-monotone trends. CIFAR-100 test accuracy at rank 512 (0.7307) is below rank 256 (0.7318) tab. D.4. The main text calls Light and Balanced similar at rank 64 §6, and Light's validation accuracy is 0.0011 higher tab. D.6.
  • Sufficient, not necessary. The authors say most bounds are sufficient and that the gap to necessary bounds is open §7 app. F.
  • Cost. Each data-intervention run trains 50 epochs over 17,526 examples app. D.1. Curvature diagnostics use finite differences; a step-size check moved the projected matrices by no more than 1.9% app. E.1. The paper gives no total GPU-hours app. D.2.
  • Source comments. The arXiv source has three AUTHOR CHECK comments that do not appear in the rendered PDF. They note an incomplete compute inventory app. D.2, an undefined "±" (SD or SE) app. E.8, and missing QNLI long-tail, length and optimiser settings app. E.8. Treat those values as not stated.

Lineage

  • Builds on: Burer–Monteiro factorisation §2. LoRA itself is arXiv:2106.09685 §1. Landscape results for factorised problems are in §2. Rank over-parameterisation and RIP-type guarantees are arXiv:2207.01789 and arXiv:2104.10790 §2. The negative-certificate argument follows zhang2019sharp and zhangSharpGlobalGuarantees2025 §4.2. The restricted-curvature framing comes from classical RIP analysis §3.
  • Compared against (theory): the NTK-regime no-spurious-minima result for a linearised model §1; the ℓ2-loss rank-geometry analysis of LoRA §1; and arXiv:2502.09376, which gives the RIP-below-1/3 global-optimality result and the "fails loudly" alternative §1. The paper does not compare experimentally against these or against other data-selection methods §6.
  • Superseded by: not known.

Source and license

Adapted from “A Fine-Grained Analysis of the LoRA Fine-Tuning Landscape with Implications for Data Selection” by Bowen Zhang, Changrui Fang, Xinsong Ma, Jiaye Teng, Ziye Ma (arXiv:2610.06542), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.