FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer

In plain words

Adapting a large pretrained vision model to a new task usually means changing all of its weights, so every task needs its own full copy of the model. Newer methods instead keep the pretrained model frozen and train a small add-on. This paper goes further. A Vision Transformer is built from many repeated blocks whose weight matrices look alike, so FacT stacks them into one large block of numbers. It then describes the change to all of them with a few small pieces that are shared across the layers, rather than storing a separate add-on for each layer.

Only these small pieces and the final classifier are trained and stored. The pieces are folded back into the model before inference, so the model runs as fast as before. On a 19-task benchmark the authors report that their main version matches the strongest earlier method with about a fifth of its trainable parameters, and that a tiny version with about 8K parameters beats full fine-tuning. The paper also tests a hierarchical Swin backbone and few-shot learning.

TL;DR

FacT treats the frozen weights of a ViT as one tensor and tunes a low-rank factorisation of the change to that tensor. Two decompositions are proposed: Tensor-Train (FacT-TT) and Tucker (FacT-TK). The factor that maps back to the weight space starts at zero, so training starts from the pretrained model. On VTAB-1K, the authors report parity with NOAH at about one fifth of its trainable parameters. The numbers are author-reported on the test split, and the released code picks its reported accuracy as the best of several test evaluations (see Pitfalls).

Problem

Full fine-tuning stores a dense updated copy of every weight for each downstream task, which costs O(L d^2) per task for a ViT with L blocks of width d §3.2. Earlier parameter-efficient methods, such as LoRA on each weight matrix, reduce this per matrix but do not exploit redundancy across layers §1. The paper's premise is that the weight increments of a pretrained ViT are redundant both within a matrix (intra-weight rank) and across matrices (inter-weight rank) §1 §3.2.

Method

Overview. The method has four parts: stack the transformer weights into one tensor §3.1; treat the increment of that tensor as the unit of adaptation §3.2; factorise the increment with a Tensor-Train or Tucker format §3.2.2 §3.2.3; and fine-tune only the factors end-to-end with the backbone frozen §3.2.

1. Tensorise the ViT §3.1 eq. 4. Each block has four d x d attention matrices (W_q, W_k, W_v, W_o) and eight d x d feed-forward pieces (four slices of W_up and four of W_down), giving 12 matrices per block. Stacking the L blocks gives one tensor of shape 12L x d x d. The patch embedding, classification head, normalisation layers and all bias parameters are not tensorised and stay frozen, except the head, which is trained §3.1.

2. Frame tuning as tuning an increment §3.2 eq. 5 eq. 6. Let W_0 be the pretrained tensor and Delta-W = W_ft - W_0 the increment. Gradients with respect to W and with respect to Delta-W are equal, and Delta-W is initialised to zero §3.2.

3. Decompose before training. The increment is decomposed into factors before fine-tuning, and the factors are then updated end-to-end. The paper argues that a zero increment can be given pre-defined factor initialisations without running an expensive decomposition, and that gradient descent keeps the useful information §3.2.

4. Matrix-Batch format (the LoRA case) §3.2.1 eq. 7. Treating the first axis as a batch dimension and decomposing each d x d matrix separately gives LoRA. Factor sizes are 4Ldr for the restricted case that tensorises only W_q and W_v §3.2.1. This format only removes intra-weight redundancy §3.2.1.

5. FacT-TT (Tensor-Train) §3.2.2 eq. 8 eq. 9. Delta-W is written with two shared matrices U, V in R^{d x r} and a per-matrix core Sigma in R^{12L x r x r}. The factor count is 2dr + 12Lr^2 when r = r_1 = r_2 §3.2.2.

6. FacT-TK (Tucker) §3.2.3 eq. 10 eq. 11. Delta-W uses shared U, V in R^{d x r} and P in R^{12L x r}, plus a shared core C in R^{r x r x r}. The factor count is 2dr + 12Lr + r^3 when r = r_1 = r_2 = r_3 §3.2.3.

7. Initialisation. In both formats the factor V is zero-initialised and all other factors are random, so Delta-W is zero at the start §3.2.2 §3.2.3.

8. Fine-tuning. The factors are trained with the backbone frozen; the gradient with respect to a factor such as U is given by eq. 12. The hyperparameter s scales the factor path and is said to adjust the learning rate of the factors §3.2.2.

9. Inference. The factors are absorbed into the weights before inference, so FacT adds no compute or latency at inference §3.2.

10. Partitioned tensorisation for hierarchical models §4.3. Swin-B has four stages with {2, 2, 18, 2} layers and hidden sizes {128, 256, 512, 1024}. Each stage is tensorised separately, giving tensors of size 24x128x128, 24x256x256, 216x512x512 and 24x1024x1024, each decomposed on its own §4.3.

Equations. Numbering follows the compiled equation order in the source.

\text{MHSA}(X) = \sum_{i=1}^{N_h} \text{softmax}\!\left(\frac{X W_q^{(i)} W_k^{(i)\intercal} X^\intercal}{\sqrt{d}}\right) X W_v^{(i)} W_o^{(i)\intercal} eq. 1

\text{FFN}(X) = \text{GELU}(X W_{up}) W_{down}\quad\text{(bias omitted)} eq. 2

\text{FFN}(X) = \sum_{i=1}^{4} \text{GELU}\!\left(X W_{up}^{(i)}\right) W_{down}^{(i)} eq. 3

\mathcal{W} = \Big\{\{W_q^j, W_k^j, W_v^j, W_o^j\} \cup \{W_{up}^{j,(i)}\}_{i=1}^{4} \cup \{W_{down}^{j,(i)}\}_{i=1}^{4}\Big\}_{j=1}^{L} \in \mathbb{R}^{12L \times d \times d} eq. 4

g_{\mathcal{W}} = \frac{\partial \mathcal{L}(\mathcal{D}; \mathcal{W})}{\partial \mathcal{W}}, \quad \mathcal{W} \text{ initialised to } \mathcal{W}_0 eq. 5

g_{\mathcal{W}} = g_{\Delta\mathcal{W}} = \frac{\partial \mathcal{L}(\mathcal{D}; \mathcal{W}_0 + \Delta\mathcal{W})}{\partial \Delta\mathcal{W}}, \quad \Delta\mathcal{W} \text{ initialised to } 0 eq. 6

\Delta\mathcal{W}_{i,:,:} = s\, U_{i,:,:} V_{i,:,:} \quad \forall i \in \{1, \dots, 12L\} eq. 7

\Delta\mathcal{W} = s\, \Sigma \times_2 U^\intercal \times_3 V^\intercal eq. 8

\Delta\mathcal{W}_{i,j,k} = s \sum_{t_1=1}^{r_1} \sum_{t_2=1}^{r_2} \Sigma_{i,t_1,t_2}\, U_{j,t_1}\, V_{k,t_2} eq. 9

\Delta\mathcal{W} = s\, C \times_1 P^\intercal \times_2 U^\intercal \times_3 V^\intercal eq. 10

\Delta\mathcal{W}_{i,j,k} = s \sum_{t_1=1}^{r_1} \sum_{t_2=1}^{r_2} \sum_{t_3=1}^{r_3} C_{t_1,t_2,t_3}\, P_{i,t_1}\, U_{j,t_2}\, V_{k,t_3} eq. 11

g_U = \frac{\partial \mathcal{L}(\mathcal{D}; \mathcal{W}_0 + \Delta\mathcal{W})}{\partial U} = s\, g_{\mathcal{W}} \frac{\partial(\Sigma \times_2 U^\intercal \times_3 V^\intercal)}{\partial U} eq. 12

\mathcal{W} = \{W_q^j, W_k^j, W_v^j, W_o^j\}_{j=1}^{L} \in \mathbb{R}^{4L \times d \times d} eq. 13

\mathcal{W} = \{\{W_{up}^{j,(i)}\}_{i=1}^{4} \cup \{W_{down}^{j,(i)}\}_{i=1}^{4}\}_{j=1}^{L} \in \mathbb{R}^{8L \times d \times d} eq. 14

Key algorithm: FacT-TT forward for one weight matrix (the paper gives the equations, not a listing; this is our restatement of eq. 8 and eq. 9, with row-vector inputs as in the code).

# FacT-TT adapter output for matrix i  [§3.2.2] [eq. 8] [eq. 9]
# Input: tokens (B, N, d); shared U, V (d, r); core slice Sigma_i (r, r)
# Output: increment added to the frozen matrix output (B, N, d)

def fact_tt_increment(tokens, shared_U, shared_V, core_slice, scale):
    # Project the tokens into the rank-r space with the shared U  [eq. 9]
    hidden = tokens @ shared_U                  # (B, N, r)
    # Apply this matrix's own r x r core  [eq. 8]
    hidden = hidden @ core_slice                # (B, N, r)
    # Map back to width d with the shared, zero-initialised V  [eq. 8]
    increment = scale * (hidden @ shared_V.T)   # (B, N, d)
    return increment

Key algorithm: FacT-TK forward for one weight matrix (our restatement of eq. 10 and eq. 11).

# FacT-TK adapter output for matrix i  [§3.2.3] [eq. 10] [eq. 11]
# Input: tokens (B, N, d); shared U, V (d, r); shared core C (r, r, r);
#        shared P (12L, r); row i of P is layer_factor
# Output: increment added to the frozen matrix output (B, N, d)

def fact_tk_increment(tokens, shared_U, shared_V, core, layer_factor, scale):
    # Mix the core with this matrix's row of P  [eq. 11]
    core_slice = contract_first_axis(core, layer_factor)  # (r, r)
    # Project the tokens into the rank-r space with the shared U  [eq. 10]
    hidden = tokens @ shared_U                  # (B, N, r)
    # Apply the per-matrix r x r slice  [eq. 10] [eq. 11]
    hidden = hidden @ core_slice                # (B, N, r)
    # Map back to width d with the shared, zero-initialised V  [eq. 10]
    increment = scale * (hidden @ shared_V.T)   # (B, N, d)
    return increment

Key algorithm: fine-tuning loop for VTAB-1K (the paper states the optimiser and schedule in prose and in tab. 5; the loop is our restatement).

# Fine-tuning on one VTAB-1K task  [§4.1] [tab. 5] [§3.2]
# Only the factors and the classification head receive gradients

def fine_tune(backbone, factors, head, train_set):
    # AdamW, learning rate 1e-3, weight decay 1e-4  [tab. 5]
    optimizer = adamw(factors + head, lr=1e-3, weight_decay=1e-4)
    for epoch in range(100):  # 100 epochs  [tab. 5]
        # Cosine schedule with 10 warm-up epochs  [tab. 5]
        set_learning_rate(optimizer, cosine_schedule(epoch))
        for images, labels in batches(train_set, size=64):  # [tab. 5]
            logits = head(backbone(images, factors))
            loss = cross_entropy(logits, labels)
            update(optimizer, loss)
    return factors, head

Hyperparameters

Values as stated in the paper. Where the paper says "not stated", the released code may set a value; that is listed under Paper vs code, not here.

Results

Author-reported. Setup: VTAB-1K, 19 datasets with 1,000 training images each, top-1 accuracy on the test sets, ViT-B/16 pretrained on ImageNet-21K §4.1 app. A. Results are in tab. 1. No standard deviations are reported for VTAB-1K. The "# params" column is the average over the group-wise averages, in millions, excluding the classification head tab. 1 caption.

Source: tab. 1. Headline claims, with the setup above:

• FacT-TK_<=32 averages 75.6 with 0.069M parameters; NOAH averages 75.5 with 0.361M tab. 1. The abstract's "5x more parameter-efficient" matches 0.361 / 0.069 = 5.2 from these rows.
• FacT-TT_4 averages 73.5 with 0.008M parameters and is above Full (68.9) and VPT-Deep (72.0) on average tab. 1. The abstract's "8K (0.01%)" is the same setting §4.1 (3).
• Text claim: FacT-TT_<=16 and FacT-TK_<=32 together beat NOAH on 11 of 19 tasks §4.1 (1). Table 1 agrees when a task counts if either setting beats NOAH; neither setting alone reaches 11.
• Swin-B, VTAB-1K, Table 2: FacT-TT_16 averages 77.4 with 0.135M parameters, against Full at 75.0 (86.7M), VPT-Deep at 71.6 (0.162M), BitFit at 65.6 (0.201M) and Linear at 62.6 tab. 2. Per-group numbers are in the table.

Table 2 (Swin-B, VTAB-1K). Source: tab. 2.

Few-shot (five fine-grained datasets, 1 to 16 shots). Results are only in fig. 6, with trainable-parameter averages in parentheses; the text gives no numbers. The text says FacT-TT_16 has the best average and is best on four of five datasets, with Food-101 the exception where it slightly trails NOAH at 8 and 16 shots §4.2 (1) §4.2 (2).

Ablation (VTAB-1K, Tensor-Train, ranks 4 to 32). Tensorising only the attention blocks is better than tensorising only the feed-forward blocks overall; tensorising all blocks is best on Specialized and Structured; performance saturates from r = 16 upward §4.4 fig. 7.

Paper vs code

Official code: github.com/JieShibo/PETL-ViT at 026e4f1, MIT (copyright Shibo Jie). The repository is linked from the paper's arXiv comments, and the owner is the first author. Its FacT/ directory holds the FacT scripts LICENSE. It contains VTAB-1K training only: no few-shot and no Swin code was found in the repository.

• Model selection on the test set. The paper says final results come from training on all 1,000 samples and testing app. A. It does not describe any selection rule. The code evaluates on the test loader every ten epochs and keeps the best value as the reported accuracy fact_tt.py fact_tt.py. The same pattern appears in FacT/fact_tk.py. Reported VTAB-1K numbers are therefore plausibly the best of ten test evaluations, not the final epoch.
• Dropout and stochastic depth. The released code applies dropout with p = 0.1 on the factor path fact_tt.py fact_tt.py and drop-path rate 0.1 on the backbone fact_tt.py. Neither appears in the paper's hyperparameters tab. 5.
• Warm-up and minimum learning rate. The code's cosine schedule starts warm-up at 1e-6 and has a minimum of 1e-5 fact_tt.py. The paper gives only the cosine schedule and 10 warm-up epochs tab. 5.
• Last partial batch dropped. The VTAB-1K training loader uses drop_last=True vtab.py. With 1,000 samples and batch size 64, 40 random samples are skipped each epoch. The paper does not state this.
• Rank grids. The paper searches FacT-TT over {1, 2, 4, 8, 16} and FacT-TK over {2, 4, 8, 16, 32} §4.1. The released per-task configs only use ranks 4, 8 and 16 for TT clevr_dist.yaml cifar.yaml and 16 and 32 for TK cifar.yaml. The search itself is not in the repository. run.sh runs the per-task config values, which is the searched setting; the fixed-rank settings FacT-TT_4 and FacT-TK_8 need the --dim argument fact_tt.py run.sh.
• Scale s. Per-task values in the configs are all in the stated set {0.01, 0.1, 1, 10, 100} dtd.yaml dtd.yaml. The search is not released.
• Tucker core axis order (our reading). Eq. 11 contracts P with the first index of C eq. 11. The code multiplies the core by the P slice along its last axis: FacTc @ FacTp[:, idx:idx+k] fact_tk.py. Because the core is free and initialised the same way along each axis, this reads as the same parameterisation with the axes relabelled. Keep one convention throughout an implementation.
• Factor initialisation. The paper says the V factor is zero and the others random §3.2.2. The code zero-initialises the shared up factor in both formats fact_tt.py fact_tk.py. For TK it uses Xavier uniform for the core and P fact_tk.py. The TT per-layer slices use PyTorch's default nn.Linear initialisation fact_tt.py.
• Input pipeline. The code resizes with bicubic interpolation vtab.py. The paper says only "resize to 224" app. B.3.
• Baseline numbers. The paper reuses some NOAH-reported baseline results rather than re-running them app. B.3. The repository has no baseline code for the FacT comparison.

No discrepancies were found in the core equations: the factor sizes match the formulas in code, and the shared U and V are the same across all 12L matrices in both code and paper.

Pitfalls

• Reporting set and selection. The released code picks its reported accuracy from the test set every ten epochs fact_tt.py. Compare FacT numbers only with the same protocol, or re-run with a fixed final epoch.
• No variance on VTAB-1K. tab. 1 gives single numbers. The gap between FacT-TK_<=32 (75.6) and NOAH (75.5) is 0.1 points, so "on par" is the only claim the data support. The few-shot results average over three seeds §4.2 but are only in a figure.
• Symbol reuse. U is both the per-matrix factor of Matrix-Batch eq. 7 and the shared factor of Tensor-Train eq. 8. Shapes differ: per-matrix U is (12L, d, r), shared U is (d, r).
• Few-shot learning rate conflict. tab. 5 gives 5e-3 for few-shot, but §4.2 says the settings are "the same as on VTAB-1K", where the rate is 1e-3 tab. 5 §4.2. Few-shot code is not released, so the intended value is ambiguous.
• Scale s with AdamW. The paper says s "adjust[s] the learning rate of factors" §3.2.2. Under AdamW the parameter update size is normalised, so s mainly changes the size of the function-space change (our reading, not stated in the paper). Tune s per task, as the paper does.
• Parameter counts exclude frozen and head parts. Patch embedding, normalisation and biases are frozen and not counted §3.1; the head is trained and counted separately. The "# params" column is averaged over groups tab. 1.
• Stated counts vs formula. The TT formula 2dr + 12Lr^2 §3.2.2 gives 8,448 for d = 768, L = 12, r = 4, which matches the "8K" claim §4.1 (3). The value d = 768 comes from the code fact_tt.py; the paper does not state it in the text.
• Hierarchical models. Partitioned tensorisation needs one factor set per stage, so the parameter count grows with the number of stages §4.3. Swin training hyperparameters are not stated, and no Swin code is released.
• Input normalisation. The paper reports that unnormalised inputs could add about 1 to 2 points to average VTAB-1K accuracy for PETL methods, including FacT, but keeps normalised inputs for comparability with NOAH app. B.3.
• Adapter baseline initialisation. The paper changes the Adapter baseline to Xavier weights and zero biases, following NOAH, and says this is why its Adapter beats VPT app. B.5. Reproducing baseline numbers requires the same change.

Lineage

• Builds on: LoRA (Hu et al., 2022; the bibliography entry gives no arXiv id), which the paper reinterprets as the Matrix-Batch special case of FacT §3.2.1.
• Builds on: tensor and rank compression of Transformers, and cross-layer weight sharing in ALBERT-style models §2 §3.2.
• Compared against: BitFit, VPT, Adapter, AdaptFormer, LoRA, NOAH (arXiv:2206.04673) §4.1. The paper names NOAH as the state of the art at submission.
• Superseded by: not recorded.