ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning

In plain words

Large vision models are trained on still images. Video models are usually built by adding a module that looks across frames and then retraining the whole network for every new video task. That is slow and each task ends up with its own full copy of the model.

This paper keeps the image model frozen and inserts a tiny add-on into every layer. The add-on squeezes the features down, lets neighbouring frames exchange information through a small filter, and expands them back, then adds the result to the original features. Think of it as a thin plug-in that teaches a picture-reader to notice motion. Only the plug-ins and the final classifier are trained, so each new video task costs a small amount of storage.

The authors report that this matches full fine-tuning on two standard action recognition benchmarks, and that it trains faster and uses less memory.

TL;DR

• Adapts a frozen image ViT to video action recognition by training only small per-block adapters and a linear classifier §3.2 §4.1.
• The adapter is a bottleneck with a depth-wise 3D convolution (kernel 3×1×1) in the middle, so it uses only common operators §3.2 app. A.
• Author-reported: Kinetics-400 82.0% top-1 with ViT-B/16 (CLIP) against 81.7% for full fine-tuning; Something-Something-v2 66.3% against 66.1% tab. 1.
• The official code ships only the ViT-B/16 models; some sentences in the text differ from the paper's own tables (see Pitfalls).

Problem

Full fine-tuning of a large pre-trained model stores one full copy of the model per downstream task, which is costly in training and storage §1. Video understanding needs temporal modelling that a pre-trained image model lacks, and the usual remedy is a dedicated video architecture trained from the image weights §1 fig. 1. Large video pre-trained models are harder to obtain than image ones, so the authors assume an image-pre-trained backbone as the starting point §1.

The paper studies whether an image model can be adapted to video with only a few task-specific parameters, and it benchmarks several efficient fine-tuning methods against video ViTs on this problem §1 §4.1.

Method

The paper first describes a simple baseline. Each frame goes through the ViT separately. Each frame's class token is kept, and the tokens are averaged over time before the classifier. This is the Space-Only TimeSformer setting, with no temporal modelling §3.

Video ViTs such as TimeSformer and XViT add temporal attention or temporal shift and are usually fully fine-tuned per task. The authors treat these as parameter-inefficient §3 §4.1.

The method builds on the NLP adapter. An adapter is a down-projection, a non-linearity and an up-projection, added to the layer input with a residual connection §3.1 eq. 1.

1. Split each frame into patches and run the ViT as usual; the adapter sits inside each block §3 §3.3.
2. Adapter input: the token sequence of one block, shape (T, 1+N, d). The class token is kept unchanged; the paper's equation applies to the patch tokens §3.2 models_adapter.py.
3. Down-project d to a bottleneck width r. The default is r = 384, and the study tests 64 to 768 app. A tab. 5.
4. Reshape the patch tokens from (T, N, r) to (T, h, w, r), where N = h × w, so that frames and the spatial grid are separate axes §3.2.
5. Apply a depth-wise 3D convolution with kernel k_T × k_H × k_W. The default is 3 × 1 × 1, which mixes each channel over three neighbouring frames only §3.2 app. A tab. 7.
6. Apply the activation f. The paper leaves f unnamed §3.2 eq. 2.
7. Reshape back to (T, N, r) and up-project r to d §3.2.
8. Add the result to the patch tokens with a residual connection §3.2 eq. 2.

The paper says the module adds about 2% parameters and about 0.3% computation through its spatio-temporal operator; the reference for these percentages is not stated §3.2.

Placement: the default is one adapter per block, put before the multi-head self-attention. Other positions are reported as similar on Kinetics-400, with a larger gain from two adapters per block on SSv2 §3.3 tab. 5(c) §4.3.

Training: the backbone stays frozen. Only the adapters and the linear classifier are updated. The classifier is always trained for every compared method §4.1 fig. 1.

# ST-Adapter on the tokens of one ViT block  [§3.2] [eq. 2]
# Input:  tokens (B*T, 1 + h*w, d); T frames per clip
# Output: tokens (B*T, 1 + h*w, d)
def st_adapter(tokens, num_frames):
    # Keep the class token unchanged and adapt the patch tokens only
    # [code: models_adapter.py#L43]
    class_token = tokens[:, :1, :]
    patches = tokens[:, 1:, :]                 # (B*T, h*w, d)
    # Down-project d to the bottleneck width r, default 384  [§3.2] [app. A]
    reduced = patches @ w_down + b_down        # (B*T, h*w, r)
    # Regroup the tokens into frames and a spatial grid  [§3.2]
    grid = reshape(reduced, (B, T, h, w, r))   # (B, T, h, w, r)
    # Depth-wise 3D convolution, kernel 3x1x1, zero padding in time
    # [§3.2] [app. A] [code: models_adapter.py#L23-L29]
    mixed = depthwise_conv3d(grid, kernel=(3, 1, 1))
    # Activation f; the paper does not name it  [§3.2] [eq. 2]
    mixed = f(mixed)
    # Flatten back to tokens and up-project r to d  [§3.2]
    update = reshape(mixed, (B * T, h * w, r)) @ w_up + b_up
    # Residual addition to the patch tokens  [§3.2] [eq. 2]
    return concat(class_token, patches + update)

# ViT-B/16 with ST-Adapters on one clip  [§3] [§3.3]
# Input:  video (B, T, 3, 224, 224) with T = 8 frames
# Output: class logits (B, num_classes)
def forward(video):
    # Patch-embed each frame, add class token and position  [§3]
    frames = patch_embed(video)                # (B*T, 1 + h*w, d)
    for block in vit_blocks:
        # Each block applies its ST-Adapter with the block  [§3.3]
        frames = block(frames, T)
    # Average the per-frame class tokens over time  [§3]
    # [code: models_adapter.py#L224]
    clip_feature = mean(class_token(frames), over=T)  # (B, d)
    # Layer norm, dropout, then the linear classifier  [§4.1]
    # [code: models_adapter.py#L226-L228]
    return classifier(dropout(layer_norm(clip_feature)))

Key equations:

\texttt{Adapter}(\mathbf{X}) = \mathbf{X} + f(\mathbf{X}\mathbf{W}_{down})\mathbf{W}_{up} eq. 1

\texttt{ST-Adapter}(\mathbf{X}) = \mathbf{X} + f\Big(\texttt{DWConv3D}(\mathbf{X}\mathbf{W}_{down})\Big)\mathbf{W}_{up} eq. 2

Hyperparameters

Values from the appendix tab. 8 unless stated. The paper's text and appendix give no learning-rate schedule.

Results

Author-reported. Setup: ViT-B/16 backbone, 8 frames, 3 test views unless stated. Full fine-tuning (FT) baselines are listed with their parameter counts in tab. 1. The comparison with prior video models is in tab. 2 and tab. 3, not reproduced here.

Other reported results: the ST-Adapter with a SWAG backbone is 80.9 on K400 and 67.2 on SSv2, against 80.1 and 45.2 for full FT tab. 9. With a Swin-B backbone, ST-Adapter gives 77.1 on K400 and 65.1 on SSv2, against 81.5 and 65.3 for joint-attention Swin full FT, so the authors say it underperforms full FT on ImageNet-21K pre-training tab. 10 app. A. Kernel shape 3×1×1 is best on SSv2 tab. 7. Adapters in deeper blocks give larger gains than adapters in shallower blocks, and the authors note this matters for deeper models §4.3 tab. 5(b).

Paper vs code

Official code: https://github.com/linziyi96/st-adapter, MIT license, at commit b914b388f45ddd1968348ae2b1590a17eca75f8f (shallow clone). The repository ships only ViT-B/16 models, training scripts for K400 and SSv2, and the adapter model.

• The paper's equation for the ST-Adapter includes a non-linearity f eq. 2. The code's adapter has no non-linearity between the down-projection, the depth-wise convolution and the up-projection models_adapter.py. Without f, the adapter is a linear map plus a residual.
• The text says a single adapter goes before the multi-head self-attention §3.3. The K400 model used by the script inserts it before the MLP instead, after the attention residual, with the pre-attention variant switched off models_adapter.py models_adapter.py train_k400_clip_vit_b16_adapter12x384.sh. The SSv2 model uses both positions models_adapter.py. On K400, the ablation rows for "before MHSA" and "after MHSA" differ by 0.1 tab. 5(c).
• The paper does not say the class token skips the adapter. The code leaves it unchanged models_adapter.py.
• Initialisation: the paper gives no initialisation. The code zeroes the depth-wise convolution weight and bias and both linear biases models_adapter.py. With the convolution output at zero, the adapter starts as an identity map.
• The paper's appendix gives lr 5e-4 at batch size 128 tab. 8. The code scales its default base rate of 1e-3 per 256 samples by batch size and world size main.py main.py. The K400 script uses 3 GPUs at batch 64, giving an effective lr of 7.5e-4 and batch 192. The SSv2 script uses 4 GPUs at batch 64, giving lr 1e-3 and batch 256. Neither script passes --lr, so these are the values used by the scripts.
• The appendix lists steps (K400 ViT-B 20k; SSv2 50k). The K400 script uses 12 epochs and the SSv2 script 50 epochs train_k400_clip_vit_b16_adapter12x384.sh train_ssv2_clip_vit_b16_adapter24x384.sh. The paper gives no dataset-size conversion, so the two cannot be compared directly.
• The code adds a dropout of 0.5 before the classifier models_adapter.py. The paper does not mention it.
• The code converts all frozen weights to half precision models_adapter.py and trains with autocast and a gradient scaler main.py main.py. The paper does not mention either.
• The paper lists the K400 training resize as ShortSideJitter 224–256 tab. 8. The code uses a short-side scale range of 1.0–1.15 train_k400_clip_vit_b16_adapter12x384.sh. Whether this gives exactly 224–256 depends on the transform, which was not checked.
• The paper reports ViT-L results for K400 and SSv2 tab. 2 tab. 3. The code defines only the ViT-B/16 constructors models_adapter.py, although the README lists CLIP ViT-L/14 as supported.
• The README says the code is reproduced and that checkpoint accuracy may differ slightly from the paper. No checkpoints are in the repository [README.md].
• The full fine-tuning and linear probing baselines are not in the repository.

Pitfalls

• No activation in the shipped adapter. The paper writes f(·) in its equation, but does not name it. The official code has no non-linearity at all models_adapter.py. Choose and state which one you use. A purely linear adapter has a different capacity from the equation as written.
• Start from an identity. The residual form only leaves the pre-trained features intact if the adapter output is near zero at the start. The code achieves this with zero initialisation models_adapter.py. The paper does not say this, so random initialisation of the convolution would disturb the frozen features.
• Zero padding in time. The 3-tap temporal convolution pads with zeros at the first and last frame. The paper does not state the padding models_adapter.py.
• Learning rate depends on batch and GPU count. The official scripts do not reproduce the appendix's lr 5e-4 at batch 128. Use the code's scaling rule if you want to match its results, and say so main.py.
• Trainable parameter count. Arithmetic from the code's dimensions (d = 768, r = 384) gives 592,512 parameters per adapter, or about 7.11M for 12 adapters, plus 0.31M for a 400-class head. Table 1 reports 7.20M. The paper does not explain the difference. This is an estimate from the code, not a measured count.
• Ratio versus the abstract. The abstract says about 20 times fewer updated parameters. Table 1 gives 121.57M against 7.20M, a ratio of about 17.
• Two percentages. Section 3.2 gives about 2% extra parameters for the spatio-temporal operator. The abstract gives about 8% per task. The paper does not say what each percentage is measured against.
• The state-of-the-art sentence. The text says the ViT-L model sets a new state of the art on K400 at 86.7%. Table 2 lists MTV-H at 89.9% and TokenLearner at 85.4% with different pre-training. Use the table, not the sentence, when comparing.
• Speed sentence. The text says the model is slightly slower than "TimeSformer space only". Table 11 shows 19 ms for the ST-Adapter, 17 ms for ViT-B and 28 ms for the TimeSformer row. The TimeSformer row uses 121.57M parameters, which is the SA+TA count in Table 1.
• Hyperparameter tuning. The appendix says hyperparameters were briefly tuned on a 20% held-out validation split. It does not say whether this split came from the training data. The reported numbers use the validation sets of K400 and SSv2 tab. 2 tab. 3.
• Full fine-tuning baseline. The paper scales the learning rate of CLIP layers by 1/100 for full fine-tuning, and says accuracy on K400 was below 50% otherwise app. A. The baseline code is not released, so this cannot be checked.
• Frozen backbone in half precision. The code's frozen weights are stored in fp16 models_adapter.py. Keep the adapters and the classifier in full precision if you port the training loop, or check the effect on the reported accuracy.

Lineage

• Builds on: the NLP adapter (cited as [adapter] in the paper, a parameter-efficient transfer method for NLP) §3.1 eq. 1. The ViT backbone [vit] §3. No card in this wiki yet.
• Compared against: full fine-tuning, partial fine-tuning, temporal fine-tuning, linear probing, adapter and prompt tuning §4.1 tab. 1; TimeSformer (arXiv:2102.05095, per the paper's bibliography, no card yet) and XViT §4.1 tab. 2.
• Superseded by: not known from content in this wiki.