Tensor decomposition (for parameter-efficient adaptation)

Overview

Tensor decomposition here means writing the change to a group of weights as a product of a few small factors, where the group is stacked into one tensor and the factors are shared across its entries. The frozen backbone is kept as it is. The problem it targets is the per-task cost of full fine-tuning, which stores a dense updated copy of every weight, O(L d^2) per task for a ViT with L blocks of width d 2212.03145 §3.2.

The invariant the variants share: the weight tensor is stacked as W in R^{12L x d x d} from the attention and feed-forward matrices of a ViT, the trained increment Delta-W starts at zero, and only the factors and the classification head receive gradients 2212.03145 §3.1 eq. 4 2212.03145 §3.2. The paper's premise is that the increments are redundant within a matrix and across matrices 2212.03145 §1 2212.03145 §3.2.

The only paper on this page so far is FacT. A tensor view of adapters is treated more generally in low-rank-adaptation, where the per-matrix factorisation is the special case the paper calls Matrix-Batch 2212.03145 §3.2.1.

Variants

Matrix-Batch (per-matrix low rank, LoRA case)

Treat the stacking axis as a batch dimension and factorise each d x d matrix on its own. This is LoRA, and it removes only the redundancy within each matrix 2212.03145 §3.2.1 eq. 7. The paper uses the restricted version that tensorises only W_q and W_v, which has 4Ldr factors at rank r 2212.03145 §3.2.1. Card: FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer.

Tensor-Train (FacT-TT)

Two factors U and V are shared by all 12L matrices, each of shape d x r. Each matrix i has its own r x r core slice Sigma_i, so the increment is Delta-W_i = s U Sigma_i V^T 2212.03145 §3.2.2 eq. 8 eq. 9. The factor count is 2dr + 12Lr^2 when both ranks are equal 2212.03145 §3.2.2. Reported on VTAB-1K with rank 4 (FacT-TT_4), it averages 73.5 with 0.008M trainable parameters; with ranks searched per task over {1, 2, 4, 8, 16} (FacT-TT_<=16) it averages 75.3 with 0.037M 2212.03145 tab. 1. Both numbers are author-reported. Card: FacT.

Tucker (FacT-TK)

Shared U and V are as before, plus a per-matrix factor P in R^{12L x r} and one shared core C in R^{r x r x r}. The increment is built from all three 2212.03145 §3.2.3 eq. 10 eq. 11. The factor count is 2dr + 12Lr + r^3 at equal ranks 2212.03145 §3.2.3. With rank 8 (FacT-TK_8) it averages 74.0 with 0.014M parameters; with ranks searched over {2, 4, 8, 16, 32} (FacT-TK_<=32) it averages 75.6 with 0.069M 2212.03145 tab. 1. Card: FacT.

Partitioned tensorisation (hierarchical backbones)

Swin-B has four stages with different widths, so one tensor cannot hold every block. The paper tensorises each stage separately and fits a factor set per stage 2212.03145 §4.3. On VTAB-1K, FacT-TT_16 averages 77.4 with 0.135M parameters on Swin-B, against 75.0 for full fine-tuning at 86.7M 2212.03145 tab. 2. The parameter count grows with the number of stages 2212.03145 §4.3. Card: FacT.

When to use which

Only the FacT paper is on this page, so the choices below rest on its numbers and on its stated assumptions. The table is about constraints.

The paper tests ViT and Swin backbones on image classification only 2212.03145 §4. The parameter savings for other models are not shown by these results.

Implementation sketch

Shared-factor adapter for one stacked tensor, Tensor-Train form. It follows the paper's equations 8 and 9. Shapes are the paper's, with row-vector tokens. Factor V is zero-initialised, so the increment starts at zero 2212.03145 §3.2.2.

# Stack the frozen weights of the L blocks: 12L matrices of shape d x d
# [arxiv:2212.03145 §3.1 eq. 4]. Matrix index i runs over 0 .. 12L-1.

# Create the factors: U and V are shared, Sigma holds one r x r core per matrix
# [arxiv:2212.03145 §3.2.2 eq. 8]
U = random(d, r)                 # shared, shape (d, r)
V = zeros(d, r)                  # shared, zero-initialised so Delta-W = 0 at start
Sigma = random(12 * L, r, r)     # one core slice per matrix

# Forward pass for matrix i: frozen output plus the scaled increment
# [arxiv:2212.03145 §3.2.2 eq. 9]
def adapted_matmul(x, i):        # x: (B, N, d)
    base = x @ W0[i]             # frozen pretrained weight, (B, N, d)
    hidden = x @ U               # project into rank space, (B, N, r)
    hidden = hidden @ Sigma[i]   # this matrix's r x r core, (B, N, r)
    delta = s * (hidden @ V.T)   # map back to width d, (B, N, d)
    return base + delta

# Train only U, V, Sigma and the classification head; W0 stays frozen
# [arxiv:2212.03145 §3.2]
# After training, fold s * U Sigma_i V^T into W0[i] so inference cost is unchanged
# [arxiv:2212.03145 §3.2]

Where variants differ. Tucker replaces the per-matrix core with a per-matrix row P_i times one shared core C 2212.03145 §3.2.3 eq. 11. Matrix-Batch drops the sharing entirely and keeps one U and V per matrix 2212.03145 §3.2.1 eq. 7. The paper's ViT setup uses dropout and drop-path that the paper's hyperparameter table does not list. The card's Paper vs code section records these.

Caveats

• Selection on the test set. The released code reports the best of several test-set evaluations rather than the final epoch. The paper describes no selection rule 2212.03145 app. A. Compare FacT numbers only with the same protocol, or re-run with a fixed final epoch. Details are on the card.
• Searched ranks and scales. Ranks and the scale s are searched per task. The paper does not state the search procedure, and the search is not in the released code 2212.03145 §4.1. Fixed-rank settings (TT_4, TK_8) are the ones that need no per-task search.
• No variance on VTAB-1K. Single numbers are reported 2212.03145 tab. 1.
• Few-shot learning rate. The paper's table gives 5e-3, while its text says "other settings are the same as on VTAB-1K" 2212.03145 tab. 5 §4.2. Few-shot code is not released.

Contradictions

None known. Only one paper with a card covers this concept so far.

Papers

• FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer: stacks ViT weights into one tensor and tunes a Tensor-Train or Tucker factorisation of the increment, with a few shared factors across all layers.