---
kind: concept
slug: tensor-decomposition
title: Tensor decomposition (for parameter-efficient adaptation)
summary: >-
  Stack the same-shaped weight matrices of a frozen network into one tensor, then train a low-rank factorisation of its increment, with factors shared across layers. FacT (Tensor-Train or Tucker) matches NOAH on VTAB-1K with about a fifth of its trainable parameters, author-reported. Trade-off: weights must share a shape, and results use test-set selection.
tags: [fine-tuning, parameter-efficient, compression]
papers: ["2212.03145"]
related: [low-rank-adaptation, parameter-efficient-fine-tuning, vision-transformer]
status: draft
updated_at: 2026-10-19
---

## Overview

Tensor decomposition here means writing the change to a group of weights as a
product of a few small factors, where the group is stacked into one tensor and
the factors are shared across its entries. The frozen backbone is kept as it
is. The problem it targets is the per-task cost of full fine-tuning, which
stores a dense updated copy of every weight, O(L d^2) per task for a ViT with
L blocks of width d [arxiv:2212.03145 §3.2].

The invariant the variants share: the weight tensor is stacked as
W in R^{12L x d x d} from the attention and feed-forward matrices of a ViT, the
trained increment Delta-W starts at zero, and only the factors and the
classification head receive gradients [arxiv:2212.03145 §3.1 eq. 4] [arxiv:2212.03145 §3.2].
The paper's premise is that the increments are redundant within a matrix and
across matrices [arxiv:2212.03145 §1] [arxiv:2212.03145 §3.2].

The only paper on this page so far is FacT. A tensor view of adapters is
treated more generally in [low-rank-adaptation](/concepts/low-rank-adaptation),
where the per-matrix factorisation is the special case the paper calls
Matrix-Batch [arxiv:2212.03145 §3.2.1].

## Variants

### Matrix-Batch (per-matrix low rank, LoRA case)

Treat the stacking axis as a batch dimension and factorise each d x d matrix on
its own. This is LoRA, and it removes only the redundancy within each matrix
[arxiv:2212.03145 §3.2.1 eq. 7]. The paper uses the restricted version that
tensorises only W_q and W_v, which has 4Ldr factors at rank r [arxiv:2212.03145 §3.2.1].
Card: [FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer](/papers/2212.03145).

### Tensor-Train (FacT-TT)

Two factors U and V are shared by all 12L matrices, each of shape d x r. Each
matrix i has its own r x r core slice Sigma_i, so the increment is
Delta-W_i = s U Sigma_i V^T [arxiv:2212.03145 §3.2.2 eq. 8 eq. 9]. The factor
count is 2dr + 12Lr^2 when both ranks are equal [arxiv:2212.03145 §3.2.2].
Reported on VTAB-1K with rank 4 (FacT-TT_4), it averages 73.5 with 0.008M
trainable parameters; with ranks searched per task over {1, 2, 4, 8, 16}
(FacT-TT_<=16) it averages 75.3 with 0.037M [arxiv:2212.03145 tab. 1]. Both
numbers are author-reported. Card: [FacT](/papers/2212.03145).

### Tucker (FacT-TK)

Shared U and V are as before, plus a per-matrix factor P in R^{12L x r} and one
shared core C in R^{r x r x r}. The increment is built from all three
[arxiv:2212.03145 §3.2.3 eq. 10 eq. 11]. The factor count is 2dr + 12Lr + r^3
at equal ranks [arxiv:2212.03145 §3.2.3]. With rank 8 (FacT-TK_8) it averages
74.0 with 0.014M parameters; with ranks searched over {2, 4, 8, 16, 32}
(FacT-TK_<=32) it averages 75.6 with 0.069M [arxiv:2212.03145 tab. 1].
Card: [FacT](/papers/2212.03145).

### Partitioned tensorisation (hierarchical backbones)

Swin-B has four stages with different widths, so one tensor cannot hold every
block. The paper tensorises each stage separately and fits a factor set per
stage [arxiv:2212.03145 §4.3]. On VTAB-1K, FacT-TT_16 averages 77.4 with
0.135M parameters on Swin-B, against 75.0 for full fine-tuning at 86.7M
[arxiv:2212.03145 tab. 2]. The parameter count grows with the number of stages
[arxiv:2212.03145 §4.3]. Card: [FacT](/papers/2212.03145).

## When to use which

Only the FacT paper is on this page, so the choices below rest on its numbers
and on its stated assumptions. The table is about constraints.

| Situation | Use | Why |
|---|---|---|
| Storage per task must be as small as possible, ViT backbone | FacT-TT, rank 4 | 0.008M trainable parameters, average 73.5 on VTAB-1K [arxiv:2212.03145 tab. 1] |
| Accuracy matters more than parameter count, ViT backbone | FacT-TT with searched rank, or FacT-TK with searched rank | Averages 75.3 (TT, 0.037M) and 75.6 (TK, 0.069M) [arxiv:2212.03145 tab. 1]; the search is per task, see Caveats |
| Need to match NOAH with fewer parameters | FacT-TK_<=32 | Averages 75.6 vs 75.5 with 0.069M vs 0.361M [arxiv:2212.03145 tab. 1]. The paper gives single runs, so this is parity, not a measured win [arxiv:2212.03145 tab. 1] |
| Backbone with several stages of different widths (e.g. Swin) | Partitioned FacT, one factor set per stage | Each stage has its own hidden size, so each is tensorised on its own [arxiv:2212.03145 §4.3] |
| Only a few matrices are adapted, or per-matrix factors are wanted | Matrix-Batch (LoRA) | Works per matrix and needs no stacking [arxiv:2212.03145 §3.2.1]. Stacking needs equal shapes, which is our reading of the setup [arxiv:2212.03145 §3.1] |
| Inference latency or memory is critical | Any variant, folded into the weights | The factors are absorbed before inference, so no added compute at inference [arxiv:2212.03145 §3.2] |

The paper tests ViT and Swin backbones on image classification only
[arxiv:2212.03145 §4]. The parameter savings for other models are not shown
by these results.

## Implementation sketch

Shared-factor adapter for one stacked tensor, Tensor-Train form. It follows
the paper's equations 8 and 9. Shapes are the paper's, with row-vector
tokens. Factor V is zero-initialised, so the increment starts at zero
[arxiv:2212.03145 §3.2.2].

```pseudocode
# Stack the frozen weights of the L blocks: 12L matrices of shape d x d
# [arxiv:2212.03145 §3.1 eq. 4]. Matrix index i runs over 0 .. 12L-1.

# Create the factors: U and V are shared, Sigma holds one r x r core per matrix
# [arxiv:2212.03145 §3.2.2 eq. 8]
U = random(d, r)                 # shared, shape (d, r)
V = zeros(d, r)                  # shared, zero-initialised so Delta-W = 0 at start
Sigma = random(12 * L, r, r)     # one core slice per matrix

# Forward pass for matrix i: frozen output plus the scaled increment
# [arxiv:2212.03145 §3.2.2 eq. 9]
def adapted_matmul(x, i):        # x: (B, N, d)
    base = x @ W0[i]             # frozen pretrained weight, (B, N, d)
    hidden = x @ U               # project into rank space, (B, N, r)
    hidden = hidden @ Sigma[i]   # this matrix's r x r core, (B, N, r)
    delta = s * (hidden @ V.T)   # map back to width d, (B, N, d)
    return base + delta

# Train only U, V, Sigma and the classification head; W0 stays frozen
# [arxiv:2212.03145 §3.2]
# After training, fold s * U Sigma_i V^T into W0[i] so inference cost is unchanged
# [arxiv:2212.03145 §3.2]
```

**Where variants differ.** Tucker replaces the per-matrix core with a
per-matrix row P_i times one shared core C [arxiv:2212.03145 §3.2.3 eq. 11].
Matrix-Batch drops the sharing entirely and keeps one U and V per matrix
[arxiv:2212.03145 §3.2.1 eq. 7]. The paper's ViT setup uses dropout and
drop-path that the paper's hyperparameter table does not list. The card's
Paper vs code section records these.

## Caveats

- **Selection on the test set.** The released code reports the best of several
  test-set evaluations rather than the final epoch. The paper describes no
  selection rule [arxiv:2212.03145 app. A]. Compare FacT numbers only with the
  same protocol, or re-run with a fixed final epoch. Details are on the card.
- **Searched ranks and scales.** Ranks and the scale s are searched per task.
  The paper does not state the search procedure, and the search is not in the
  released code [arxiv:2212.03145 §4.1]. Fixed-rank settings (TT_4, TK_8) are
  the ones that need no per-task search.
- **No variance on VTAB-1K.** Single numbers are reported [arxiv:2212.03145 tab. 1].
- **Few-shot learning rate.** The paper's table gives 5e-3, while its text says
  "other settings are the same as on VTAB-1K" [arxiv:2212.03145 tab. 5 §4.2].
  Few-shot code is not released.

## Contradictions

None known. Only one paper with a card covers this concept so far.

## Papers

- [FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer](/papers/2212.03145): stacks ViT weights into one tensor and tunes a Tensor-Train or Tucker factorisation of the increment, with a few shared factors across all layers.