The Universal Weight Subspace Hypothesis

In plain words

Fine-tuning a neural network changes its weights, and it is easy to assume that every fine-tuned copy changes them in its own unrelated way. This paper checks that assumption on a large scale. It collects hundreds of models that were trained for different tasks, and looks at the weights of each layer across all of them at once. For most layers, a handful of directions capture nearly all of the variation between the models. In other words, the models seem to live in a small shared space inside the full weight space.

If that is true, you can store one shared set of directions plus a short list of numbers per model, instead of every full model. A new model can be described by its coordinates in the shared space, which is far less to store and train. The authors report this works well for several kinds of networks. They also say they do not yet know why it happens, and that the evidence is strongest where many models are available.

TL;DR

• Per layer, the weights of many models, stacked together, are dominated by a few principal directions §2 §4.
• Projecting a model onto those shared directions gives a close replacement for the original, without retraining. Author-reported numbers are close to full training in-distribution §4.
• The main uses the authors claim are storage (19x for 500 Mistral LoRAs, up to 100x for ViTs), model merging, and learning new tasks by training only a few coefficients §4 app. C.2.
• The paper is a large empirical study with a conditional theory section. It does not prove universality §3.

Problem

Storing, serving and merging many fine-tuned models costs memory, and each new task usually means training and keeping a full set of weights. The paper asks whether the weights of independently trained models are constrained to a low-dimensional shared subspace, which would make such a set of models much more compressible §1. The conditional theory assumes bounded task predictors and enough tasks and samples to estimate a shared top-k eigenspace. It states that the approximation error shrinks with the number of tasks at roughly the rate T^(-1/2), but the theory does not say that a low-rank shared space exists. That is left to the experiments §3.

Method

Overview. The method takes M models that share an architecture, so their weight tensors have the same shape. It removes the mean across models, unfolds the stacked tensor along one or more modes, and keeps the leading left singular vectors of each unfolding as a shared basis §2 alg. 1. Each model is then represented by its core, its coordinates in that basis. The paper calls the basis the Universal Subspace, and the model rebuilt from it a Universal variant §4.

Steps (the notation follows alg. 1):

1. Stack the M weight tensors along a new first axis alg. 1.
2. Compute the mean over the model axis and over every mode that is not decomposed. This mean is the feature-wise mean §2 alg. 1.
3. Subtract the mean from each model alg. 1.
4. For each decomposed mode p, unfold the centered stack along p and take a thin SVD of the unfolding alg. 1.
5. Pick the smallest rank whose cumulative explained variance reaches a threshold tau, and keep that many leading left singular vectors as the factor U(p) alg. 1. The paper does not define how the cumulative variance is normalised; see Pitfalls.
6. Project each centered model onto the factors to get its core C(m) alg. 1.
7. Rebuild a model as the mean plus the core multiplied by the factors on the decomposed modes. Modes that are not decomposed stay uncompressed §2.

# Shared subspace via truncated HOSVD  [alg. 1]
# Input:  models: M tensors of one shape; modes P; threshold tau
# Output: mean, factors U[p] for p in P, cores C[m]

def universal_subspace(models, modes, tau):
    # Stack the M models along a new leading axis  [alg. 1]
    stacked = stack(models)                    # (M, I_1, ..., I_P)
    # Feature-wise mean over models and over non-decomposed modes  [§2]
    mean = average_over_models_and_other_modes(stacked, modes)
    # Subtract the mean from every model  [alg. 1]
    centered = [model - mean for model in models]
    factors = {}
    for p in modes:
        # Unfold the centered stack along mode p  [alg. 1]
        unfolded = unfold(stack(centered), mode=p)
        # Thin SVD of the unfolding  [alg. 1]
        left, singular, _ = svd_thin(unfolded)
        # Smallest rank whose cumulative explained variance reaches tau
        # (normalisation assumed, not stated)  [alg. 1]
        rank = smallest_rank_reaching(singular, tau)
        # Keep the leading left singular vectors as the shared factor  [alg. 1]
        factors[p] = left[:, :rank]
    # Project every centered model onto the shared factors  [alg. 1]
    cores = [project(c, factors) for c in centered]
    return mean, factors, cores

Reconstruction. A model is rebuilt as the mean plus its core multiplied by each shared factor, on the decomposed modes §2. When only one mode is decomposed, this is ordinary PCA on the fibres of that mode §2 app. B.5.

LoRA adapters app. B.5. For each layer, the authors stack the A matrices of all adapters on top of each other, and do the same for the B matrices. The stacked matrix has shape (500r, d) for 500 adapters of rank r. It is centered by subtracting the feature-wise mean and decomposed with PCA. A and B are decomposed separately. The authors note that LoRA factors are not unique, since BA equals (BR)(R^-1 A) for any invertible R. So the analysis describes the stored factors, not the update BA itself app. B.5.

Full weights §4 app. B.6. The same procedure is applied to every layer except the first and last. These layers differ in shape and task across models, so they are excluded.

Coefficient-only adaptation to new tasks §4 app. C.2. The shared directions are frozen and only the per-task coefficients are trained by gradient descent. This is used for a ViT image-classification task and for six GLUE tasks with RoBERTa-base.

Suitability score (data-free) §2 app. B.1. For each layer, the authors fit a power-law tail exponent alpha to the spectrum of W^T W for each model, following the heavy-tailed self-regularisation analysis cited as [martin21]. Each layer gets a score q_l that combines three things: the number of valid alpha estimates, the fraction of them in [2, 6] (weighted by a decay in alpha), and the spread of alpha across models. The score is

q_l = \left(1 - e^{-N_l/\kappa}\right) \left( \frac{1}{N_l} \sum_{m=1}^{N_l} \mathbf{1}[2 \le \alpha_{l,m} \le 6]\, e^{-\eta(\alpha_{l,m}-2)} \right) e^{-\lambda \sigma_l} app. B.1

where N_l is the number of valid estimates, sigma_l their standard deviation, and kappa = 20, eta = 0.5, lambda = 1 unless stated otherwise app. B.1. Layer means must lie in [2, 6], the standard deviation must be at most 1, and at least half the estimates must lie in [2, 6]. Among layers passing these, the authors keep those with q_l >= 0.2 app. B.1. The family-level scores are geometric means of q_l over all layers (q_all) and over the kept layers (q_arch), with a floor of 1e-12 app. B.1. The paper calls these coarse heuristics, not measures of subspace error app. B.1.

Theory (conditional) §3. The paper models each task's predictor as an element of a Hilbert space and defines a task second-moment operator. It gives a concentration bound for the empirical operator, a Davis-Kahan bound on the eigenspace error, and a two-level bound that separates across-task error (decaying with T) from per-task estimation error. A projection-risk corollary adds the spectral tail as irreducible error. A separate proposition relates this to weights in a linearised regime around a shared initial point. The authors say the analysis is conditional motivation, not a proof, and that their algorithm departs from that regime in three ways §3.

Hyperparameters

Results

Author-reported. Each row names the setup from the paper. Numbers are the authors' own. Table and figure references follow the section where they appear.

Model merging against six gradient-free baselines is discussed in §4, with a table that is commented out in the source. The commented-out table is not used here.

Paper vs code

Official code: toshi2k2/unisub, linked from the authors' project page (https://toshi2k2.github.io/unisub/). The code is pinned at commit 9094a84a2ee26f81bfd264e43ad5630bd6460239 (fetched 2026-10-18).

• Centering in the GLUE / HOSVD path. Algorithm 1 subtracts the mean first alg. 1. The two basis functions in the HOSVD script do not subtract a mean; they take the SVD (mode-2) or Tucker decomposition (mode-3) of the raw matrices hosvd.py hosvd.py. The centering helper eigendecomposition in the same repository utils.py is not called by these functions. Discrepancy: the paper's Algorithm 1 is centered, and the code's HOSVD path is not. For the LoRA and classical-weight PCA scripts, the centering matches the paper: each script subtracts the element-wise mean over models run_pca.py run_pca.py run_pca.py.
• Random Gram-Schmidt components. The HOSVD script adds 32 random vectors orthogonalised against the shared basis by default hosvd.py hosvd.py utils.py. The paper does not describe this step. It is unclear whether the reported GLUE numbers include it, so do not assume they do.
• Default HOSVD mode and component count. The script defaults to mode-3 with 32 HOSVD components hosvd.py. The paper says order-1 and order-2 are used unless stated otherwise §2, and it reports order-3 only for GLUE §4.
• GLUE training hyperparameters. The paper gives learning rate 4e-4 and batch 64 for the GLUE coefficient-only runs app. C.2. The NLU README's example command uses learning rate 4e-3, batch size 16 and 30 epochs README.md README.md. The README example is for MRPC, and the paper's runs are per task, so it may not be the same setting. Discrepancy: the two are not reconciled in the repository.
• Number of components saved. The PCA scripts keep up to 300 components and, for the LLaMA and GPT-2 scripts, increase the count when the explained variance is below 0.90 run_pca.py run_pca.py. The ViT script caps components at 300 and does not apply the 0.90 rule run_pca.py. The paper's 16-dimension ViT subspace is not set anywhere in the scripts we read; it is the authors' reading of the scree plot §4.
• Storage numbers. The memory savings in §4 are computed in the paper's own accounting, which counts the shared bases once and excludes task-specific layers app. C.2. We did not re-derive them from the code.

Apart from the centering and component rules above, the LoRA PCA scripts follow the procedure in app. B.5.

Pitfalls

• Universality is an empirical claim. The theory is conditional. Theorem bounds depend on per-task error terms that the paper keeps abstract §3. Treat "universal" as "low-rank in these experiments", not as a proof.
• Cumulative variance is not defined. Algorithm 1 uses a cumulative explained-variance threshold tau alg. 1, but the paper does not give the normalisation. The code's PCA paths use scikit-learn's explained-variance ratios run_pca.py. Implementers should take that as the working definition, which is an inference from the code, not something the paper states.
• Choosing k is manual. The paper uses visual inspection of the scree plot, or a cut at explained variance below 0.01 app. C.1. No general rule is given for tau or k, and the per-experiment choices differ (see Hyperparameters). The paper gives no sweep over k or tau.
• Few models, weaker results. The authors say the approximation is worse when few models are available. ResNet-50 from scratch has q_all = 0.0183 and the StyleGAN2 set 0.0572 app. B.2 app. B.3. They frame these as lower bounds, not as validation of the subspace.
• Counts differ across the paper. The text says about 500 ViTs and 200 GPT-2 models in places §4 app. B.6. The suitability table lists 464 ViTs and 177 GPT-2 models app. B.6. The GPT-2 section also says about 200 models were used for the teaser figure app. B.6. Check the table for the exact set.
• Layer exclusions. The first and last layers are excluded from the analysis because their shapes depend on the task app. B.6. The reported memory savings exclude them too app. C.2. Full-model storage needs the excluded layers added back.
• LoRA gauge. The LoRA factors A and B are not unique, so the subspace describes the stored factors rather than BA app. B.5. A gauge-invariant analysis would decompose the product BA directly, which the paper names but does not run.
• Secondary subspace. The paper compares the top subspace with the leftover components on the Mistral LoRAs and reports a sharp drop §4 app. B.5. The numbers appear only in a figure, so no value is given here.
• Small quality gain is not explained. The SDXL CLIP scores are slightly higher for the projected LoRAs §4. The authors suggest denoising but do not test it app. B.5. Do not treat the gain as a result.
• Coefficient-only results are small-scale. The GLUE gain over LoRA is 0.34 points on average §4. The ViT results cover five datasets with one architecture and 4 to 5 source models app. C.2. The paper suggests that longer training or tuned hyperparameters might do better app. C.2.
• Out-of-distribution gap. The Universal ViT loses 3.5 points OOD §4. The paper does not analyse this gap.
• Code licence. The repository has no LICENSE file, and the project page states no licence. The code_license field records "none stated". Do not copy code from it verbatim. The repository also contains vendored copies of transformers, diffusers and peft, which carry their own licences and are not part of the authors' work.
• Hidden costs. Subspace extraction needs all source models in memory or on disk, plus an SVD or Tucker decomposition per layer app. B.6 app. C.2. The authors say the classical-weight analysis is much more expensive than the LoRA one app. B.6.

Lineage

• Builds on: LoRA [arXiv:2106.09685], used as the adapter format for most experiments §4. Earlier work on recurring eigendirections in CNNs, cited as [guth24] and [Krizhevsky2012ImageNet] §1.
• Heavy-tail spectral analysis of neural weights, cited as [martin21] and [martin2025setol] (arXiv:2507.17912) for the suitability score app. B.1.
• Mistral LoRA collection: "Compress then Serve" (arXiv:2407.00066), the source of the 500 Mistral adapters app. B.5.
• Closest prior work: EigenLoRAx (arXiv:2502.04700), which studies principal subspaces of adapters for efficient adaptation. The paper calls it an application of the same idea to LoRA app. C.1.
• Compared against: model-merging baselines RegMean, Task Arithmetic, TIES, DARE-TIES and KnOTS §4.
• Superseded by: not known from content in this wiki.