wikillm

Paper · 2026-10-08 · cs.LG

A Categorical Atlas of Parameter-Efficient Fine-Tuning: Cones, Orbits, and Gauges

Theory paper. Models each PEFT method as a map from trainable settings into weight space, anchored at the pretrained weights, and sorts 382 methods by reach, training geometry and base-change behaviour. Proves that LoRA cannot exceed update rank 2r from any exact start, that LoRA+ matches LoRA at a rescaled scale under Adam (eps=0), and that exactly orthogonal methods keep singular values.

Parameter-efficient fine-tuningLow-rank adaptation (LoRA and variants)peftloratheoryoptimizationmodel-merging

arXivPDFCodeMarkdown9 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Fine-tuning a large pretrained model usually means adjusting its weights. Parameter-efficient methods such as LoRA change only a small set of extra parameters, and there are hundreds of them. They are usually ranked by how few parameters they use and by their benchmark scores. This paper asks different questions. It treats each method as a map: start from a small set of trainable settings, and each setting produces a set of weights, with the pretrained weights as the starting point.

Looking at the map shows three things. First, what the method can reach from its starting point, and whether that starting point is a smooth place or a sharp corner where some first steps are blocked. Second, how training moves through the settings, which depends on the optimiser and on the many settings that give the same weights. Third, what happens when the starting point changes, for example after quantising the base model, restarting, or merging an adapter back into the weights.

Several results follow from this view. No choice of starting values lets a LoRA adapter reach a weight change of rank above twice its own rank. Methods built from exactly orthogonal transformations keep the singular values of the weights, however often they are merged. Under Adam, when the epsilon term, decay and clipping are all switched off and every parameter group follows one schedule, LoRA+ retraces plain LoRA run at a larger scale, and the learning-rate ratio sets how much larger. The authors also catalogue 382 methods from 138 papers by these properties.

TL;DR

  • Represents each weight-space PEFT method as a map anchored at the pretrained weights, and compares methods by three invariants: the image near the pretrained point, the fibres together with the optimiser metric, and behaviour under base change § Thesis.
  • Proves that no exact initialisation lets LoRA of rank r produce a weight change of rank above 2r, and that zero-factor starts sit at an apex § II.1 § II.2.
  • Shows that LoRA+ at ratio λ and scale s reproduces plain LoRA at scale λs, for Adam with ε = 0 and with decay, clipping and per-group schedules all excluded § III.11.
  • Results are mathematical proofs, with no new training experiments. The authors report a catalogue of 382 methods; this card did not verify it.

Problem

Practitioners tend to read PEFT methods through two yardsticks, how many parameters they add and what benchmark score they reach. The paper argues that neither yardstick says how far a method can move the weights, how its training unfolds, or how it behaves after a different start, quantisation, restart or merge § Thesis. The paper treats these as precise mathematical questions, and says many have exact answers § Thesis.

Method

The object studied is a pointed smooth map

\rho : (Q, q_0) \to (\Theta, \theta_0), \qquad \rho(q_0) = \theta_0

where Q is a small parameter space, Θ is weight space and θ0 is the pretrained checkpoint § Thesis § I.2. A morphism h: M → N is a pointed smooth map with ρ_N ∘ h = ρ_M, read as "N simulates M". The frozen model is the initial object and full fine-tuning the terminal one § I.4. The "simulation arrows" in the catalogue are these morphisms, not numerical simulations § Abstract.

Three invariants, plus a further layer for architectures § Thesis:

  1. Image germ at θ0 (expressivity). Four data are read off it: the first-order image S1 (the tangent directions open to the first update), the expressive dimension d(M) = dim Im ρ, the cut-rank bound on the maximal rank of an update, and whether the base point is regular or an apex § II § I.5 § II.10.
  2. Fibres with the optimiser metric (training dynamics). Many parameter values give the same weight, so the fibres of ρ are gauge orbits. Pairing the parametrisation's derivative with the optimiser's learning rates Λ ≻ 0 defines an object on weight space, K = Dρ Λ Dρᵀ, which the paper calls a cometric § Thesis § III.1.
  3. Base change (travel). Moving the base point covers splitting initialisations, quantisation, restarts and merging § V.1 § V.3.

The paper gives the expressive dimension as d(M) = dim Im M, and the gauge waste as g(M) = |M| − d(M), where |M| is the parameter count § Thesis.

For LoRA with B0 = 0, the paper states the first-order image as mr-dimensional, and for PiSSA-style split starts as r(m+n−r)-dimensional § II.2(b) § II.3.

Closed-form LoRA dynamics under gradient flow with an isotropic charge Φ = −κ I_r and μ = s√(η_A η_B) are given in § III.9:

\dot W = -\mu^2 \big[ G\, V g_\kappa(\Sigma^2/\mu^2) V^\top + U f_\kappa(\Sigma^2/\mu^2) U^\top G \big]

with f_κ(x) = √(x + κ²/4) − κ/2 and g_κ = f_κ + κ. Here Z = W − W_res = UΣVᵀ is the compact SVD and G is the gradient § III.9.

Householder-reflection adaptation (HRA) of rank r, for even r with 2 ≤ r ≤ n−2 and an injective W0, is defined in § II.7:

Pseudocode
# HRA_r: adapt the frozen weight with r Householder reflections  [§ II.7]
# Input:  W0 (m, n), injective; vectors u_1 .. u_r, each nonzero, r even
# Output: adapted weight (m, n)
def hra(W0, reflection_vectors):
    # Build H_u = I - 2 u u^T / ||u||^2 for each vector  [§ II.7]
    reflections = [householder(u) for u in reflection_vectors]
    # Right-multiply W0 by the reflections in order  [§ II.7]
    weight = W0
    for H in reflections:
        weight = weight @ H
    return weight

Its image is the set of weights that are both reachable by rank-r LoRA and lie on the SO(n) orbit of W0, and its expressive dimension is rn − r(r+1)/2 § II.7(a).

Hyperparameters

The paper has no training recipe for a model. The table lists the conditions under which the stated theorems hold, and the settings of the numerical check in the code.

NameValueWhereNotes
LoRA rankr, with r < min(m, n)§ II.2full-rank pointings
LoRA+ ratioλ (learning rate of B = λ × that of A)§ III.11(b)proof uses the same A0 and shared schedule
LoRA scales§ III.11equivalence is to plain LoRA at scale λs
Adam epsilonε = 0§ III.11(b)ε > 0 needs per-block ε'_B = λε
Weight decaynone (AdamW: per-group decay λ·wd on B)§ III.11(b)
Gradient clippingnone§ III.10 § III.11global-norm clipping breaks the invariance
Learning-rate scheduleone schedule shared by all blocks§ III.10 § III.11warm-up is allowed
Adam betasβ1 ∈ [0, 1), β2 ∈ (0, 1)§ III.10bias correction allowed
HRA rankr even, 2 ≤ r ≤ n − 2§ II.7W0 injective
Numerical check: shapes, rank, stepsm = 7, n = 6, r = 2, 200 stepsframework_checks.pynot a paper setting
Numerical check: betas0.9, 0.999framework_checks.pynot a paper setting
Numerical check: LoRA+ valuesλ = 8.0, s = 0.5, η_A = 1e-3framework_checks.pynot a paper setting

Results

Author-reported. The results are mathematical theorems with proofs in app. C. The paper also reports numerical checks with float64 autograd app. E. There are no benchmark experiments on trained models in the source; the 382-method census is a classification of published formulas.

ResultClaimSettingSource
Strata of LoRA startsFull-rank LoRA starts form three strata; first-order dimensions are mr, nr and r(m+n−r)Single weight matrix, rank r§ II.2
Example dimensionFor 4096×4096 at rank 16: 65,536 for zero-factor starts, 130,816 for split startsSame§ II.3
Rank boundRank above 2r is out of reach for every exact start of LoRA_rAny exact pointing, including zero and split starts§ II.1
ApexZero-factor and paired HRA starts are apexes, not smooth pointsInjective W0§ II.2 § II.7(c)
Orthogonal methodsPreserve the singular spectrum of W0 in exact arithmetic under merge-and-restart cyclesHypothesis: image in W0·O(n)§ II.6
Torus invarianceTrajectories are invariant under rescaling of hyperparameters by c_i > 0Adam, no clipping, exact arithmetic§ III.10
LoRA+ equivalenceLoRA+ at ratio λ, scale s = plain LoRA at scale λsAdam with zero epsilon, decay and clipping off, one schedule for all groups§ III.11(b)
Optimiser symmetriesAdam: signed permutations of order 2^r r!; SGD: O(r)Full-rank LoRA locus, shared hyperparameters§ III.12
Catalogue counts382 methods, 138 papers, 403 arrows, 485 obstructions, 41 numbered resultsCatalogue of published formulas§ Abstract § Thesis

Numerical check, as reported in the paper: the gap between trajectories is 0.0 when the rescaling factors are powers of two, and roughly 1e-15 otherwise, including warm-up, and 1e-3 to O(1) when ε or decay is held fixed or with global-norm clipping § III.10 Num.

Paper vs code

The paper links the code repository https://github.com/xiaoyulics/peft-atlas app. F, which is the authors' own repository. The code is MIT-licensed and the content CC BY 4.0. This card uses commit 95a3321b193f28b2c6a6461f741b87ad527ac6d2, cloned at compile time. The repository contains the theory checks but no PEFT training code.

  • Checked: the LoRA+ check in theory/framework_checks.py compares LoRA+ with ratio λ against plain LoRA at scale λs, with the same A0 and learning rates, which matches the statement of § III.11(b) framework_checks.py.
  • Coverage gap: the Adam simulation takes an eps argument but all calls use the default 0.0, so the ε > 0 extension of § III.11(b) is not checked by that script framework_checks.py framework_checks.py.
  • Not verified: the script was read, not executed, so the numerical claims in § III.10 Num are not confirmed here.
  • Not verified: the Hugging Face PEFT claims refer to commit 85be3f3 of that library app. F, which is not in this repository.

Pitfalls

  • Exact arithmetic. The rank bound, the orthogonal invariance and the torus invariance are exact results. Floating point breaks them slightly. The torus invariance is bit-exact only when every c_i is a power of two § III.10.
  • Adam's equivalence is narrow. The stated equality assumes ε = 0 and the other conditions in the Hyperparameters table; nothing in the statement covers runs outside them without the adjustments below. With ε > 0 the paper requires a per-block epsilon ε'_B = λε, and with AdamW a decay of λ·wd on B § III.11(b). With SGD the equivalent scale is √λ·s instead § III.11(b).
  • The 2r bound concerns the update, not the stored model. Quantised residuals add a defect of generic rank at least min(m, n) − r, which is not covered by the 2r bound § II.1.
  • Orthogonal is a hypothesis, not a name. Corollary II.6 requires the image to lie in W0·O(n). The paper states that several methods that look orthogonal fail this hypothesis, and summarises which ones in a table § II.6 § II.6 remark. The HRA image lies on the SO(n) orbit. Taking the full O(n) orbit would add an odd component, which the HRA image does not contain § II.7.
  • Gauge and optimiser change the answer. Equivariance under the gauge group depends on the optimiser. SGD is equivariant under O(r), Adam only under signed permutations, and scaled GD under all of GL(r) without damping § III.12. A rule written for one optimiser may not transfer.
  • Dimension is a property of the method, not the data. d(M) does not depend on data, so it is not a measured intrinsic dimension § Thesis.
  • Many results are credited to others. Results marked "known" are due to the authors named in their headings, for example the LoRA/Adam torus is due to Schulman et al. § III.10. Cite the original authors along with the paper, as the paper asks § Thesis.
  • The census is agent-assembled. The 382-method catalogue was built by agent sweeps followed by independent fact-checking, with 905 corrections. A triage pass first saw a truncated list and a second pass was run app. F. The census is data from the paper, not verified by this card.
  • Notation traps. "Simulation" means a morphism between pointed maps, not a numerical simulation. The symbol μ in the LoRA dynamics is a rate, not the moment map used in § III.7 § III.9.

Lineage

  • Builds on: LoRA (Hu et al., 2021), the baseline whose zero start and rank structure the paper analyses § II.2; PiSSA (Meng et al., 2024), the split-start example, whose 2r conversion the paper explains § II.1; LoRA-GA (Wang et al.), CorDA, OLoRA, MiLoRA and EVA, the initialisation alternatives classified in § II.2; scaled GD (Tong et al., 2021) and Riemannian LoRA (Zhang et al., 2024), which the paper cites for descending cometrics § III.12; "LoRA Without Regret" (Schulman et al., 2025), credited by the paper for the torus case of LoRA under Adam and for the LoRA+ reading § III.10 § III.11.
  • Compared against: LoHa (FedPara), GraLoRA, OFT and HRA, and the other methods in the catalogue § II.6 § II.1.
  • Superseded by: not stated.

Source and license

Adapted from “A Categorical Atlas of Parameter-Efficient Fine-Tuning: Cones, Orbits, and Gauges” by Xiaoyu Li, Zhizhou Sha, Chiwun Yang, Dai Shi (arXiv:2610.12728), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/xiaoyulics/peft-atlas at commit 95a3321b193f (MIT). Any quoted code is shown under that license.