Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

In plain words

CLIP is a model that matches pictures to text, so it can name objects it was never trained on by comparing an image with written class names. It gets better when shown a few labelled examples of the new classes, but the usual way to use those examples is to train a small add-on module, which takes time and GPU memory.

Tip-Adapter skips that training. It keeps a lookup table made of the CLIP features of the example pictures and their labels. When a new picture arrives, it compares the picture with the stored examples; examples that look alike vote for their labels. That vote is added to CLIP's own text-based guess. The table is built once and needs no gradient steps. An optional short fine-tuning step can then adjust the stored features a little, and it still needs far fewer training epochs than the original add-on method.

TL;DR

• Few-shot CLIP adapters (CLIP-Adapter, CoOp) need gradient training, which the authors call slow and memory-hungry §1.
• Tip-Adapter builds a key-value cache from CLIP features of the few-shot training images and sets an existing two-layer adapter's weights from it, with no backpropagation §3.2.
• Training-free, the paper's own ImageNet table puts Tip-Adapter below CLIP-Adapter at every shot count, although the authors call it comparable tab. 2. With 20 epochs of key fine-tuning, Tip-Adapter-F is reported above CLIP-Adapter's 200-epoch result on ImageNet and on the other datasets §4.2, §4.3, tab. 3 (author-reported).
• The official code differs from the paper in several places that change numbers: augmented cache features, a ×100 logit scale, an AdamW optimizer, and hyperparameter choice on the test set. See "Paper vs code".

Problem

Few-shot classification with CLIP means predicting new labels from a handful of labelled images per class. Zero-shot CLIP uses only class names §3.1. The two strongest baselines, CoOp (learned prompts) and CLIP-Adapter (a learned residual adapter), both fine-tune on the few-shot set with SGD, which the authors say converges slowly and uses a lot of GPU memory §1. The paper asks whether one can keep CLIP's training-free property and still reach CLIP-Adapter-level accuracy §1.

Method

CLIP-Adapter, the starting point §3.1. The L2-normalised CLIP image feature f_c goes through a two-layer MLP and a residual blend with f_c, and the result is scored with CLIP's text classifier W_c eq. 1 eq. 2. In CLIP-Adapter the MLP is randomly initialised and trained with cross-entropy and SGD §3.1.

Cache model, built once from the few-shot set §3.2. For N classes with K shots each, each training image gives a CLIP feature \mathbf{F}_{train} of size NK \times C, one row per image, and a one-hot label row in \mathbf{L}_{train} of size NK \times N eq. 3 eq. 4. The features are the keys and the one-hot labels are the values §3.2.

Retrieval at test time §3.2. The test feature f_{test} is the query. Its affinity to each key is A = \exp(-\beta(1 - f_{test}\mathbf{F}^T_{train})), which is a function of cosine similarity because both sides are L2-normalised eq. 5. The cache returns A\mathbf{L}_{train}, a weighted label vote. The final logits add this vote, scaled by \alpha, to CLIP's own zero-shot logits eq. 6. \beta sets how sharply nearby examples dominate, and \alpha balances cache against CLIP. The authors say \alpha should be large when the domain gap is large §3.2.

Why it is an adapter §3.2. Eqs. 7 and 8 set the adapter weights so that the CLIP-Adapter form of eq. 2 equals eq. 6: W_1 = \mathbf{F}_{train}, W_2 = \mathbf{L}^T_{train}, both biases zero, and the activation is \varphi(x) = \exp(-\beta(1-x)) eq. 7 eq. 8. Three differences from CLIP-Adapter are stated: the weights are set, not learned; the bottleneck is NK wide instead of small; and the activation is bounded in [0, 1] §3.2.

Optional key fine-tuning §3.2, §4.1. The cache can be used as the starting point for training. Only the keys (W_1) are unfrozen. Values, the text classifier and both CLIP encoders stay fixed. Cross-entropy on the few-shot training images is minimised for 20 epochs §4.1. This variant is Tip-Adapter-F.

Scaling beyond 16 shots §4.4. The paper keeps the cache at 16 entries per class by averaging random groups of examples into prototypes [§4.4, "Scaling Up to More Shots"].

Relation to earlier caches §3.3. The authors relate the cache to attention and kNN-style caches. They also read the zero-shot classifier W_c as a cache of text features, which makes the method a two-modality cache §3.3.

# Build the cache once from the few-shot set  [§3.2] [eq. 3] [eq. 4]
# Input:  K training images per class, N classes, CLIP image encoder
# Output: keys (N*K, C), values (N*K, N)

def build_cache(train_images, train_labels):
    # Encode each training image and L2-normalise its feature  [§3.2]
    features = [l2_normalise(clip_image(img)) for img in train_images]
    # Keys are the stacked features, values the one-hot labels  [eq. 3-4]
    keys = stack(features)                   # (N*K, C)
    values = one_hot(train_labels, N)        # (N*K, N)
    return keys, values

# Zero-shot class weights from prompt ensembling  [§4.1]
# Input:  class name, list of templates; Output: text_weights (C,)

def class_weight(name, templates):
    # Encode each filled-in template, normalise, then average  [§4.1]
    embeddings = [l2_normalise(clip_text(t(name))) for t in templates]
    return l2_normalise(mean(embeddings))

# Training-free inference  [§3.2] [eq. 5] [eq. 6]
# Input:  test image, keys (N*K, C), values (N*K, N), text_weights (C, N)
# Output: class logits (N,)

def tip_adapter_logits(image, keys, values, text_weights, alpha, beta):
    # Normalised test feature is the query  [§3.2]
    query = l2_normalise(clip_image(image))  # (C,)
    # Affinity is exp(-beta * (1 - cosine)) for every cached key  [eq. 5]
    affinity = exp(-beta * (1 - keys @ query))  # (N*K,)
    # Cache branch: affinity-weighted vote over one-hot labels  [eq. 6]
    cache_logits = alpha * (affinity @ values)  # (N,)
    # Zero-shot branch; eq. 6 has no scale, the official code uses 100  [eq. 6]
    zero_shot_logits = logit_scale * (query @ text_weights)  # (N,)
    return zero_shot_logits + cache_logits

# Optional fine-tuning: only the keys are learned  [§3.2] [§4.1]
# Batch size 256, learning rate 0.001, cosine schedule, 20 epochs

def finetune_keys(loader, keys, values, text_weights, alpha, beta):
    # Keys start from the cache and become trainable  [§3.2]
    learned_keys = make_trainable(keys)
    # Text says SGD; the official code uses AdamW (see Paper vs code)  [§4.1]
    optimizer = make_optimizer(learned_keys, lr=0.001)
    # Twenty epochs over the few-shot set  [§4.1]
    for epoch in range(20):
        for images, labels in loader:
            # CLIP encoders and values stay frozen  [§3.2]
            features = l2_normalise(clip_image(images))  # (B, C)
            logits = batch_logits(features, learned_keys, values, alpha, beta)
            # Cross-entropy on the few-shot labels  [§3.2] [§4.1]
            loss = cross_entropy(logits, labels)
            step(optimizer, loss)
    return learned_keys

Hyperparameters

Results

Author-reported. Setup: ImageNet, ResNet-50, CLIP-style pre-processing, 7-template prompt ensemble, tested on the full test set §4.1 §4.2. The code's test split is the ImageNet validation split imagenet.py.

Accuracy (%) on ImageNet tab. 2:

The zero-shot row is one value that the paper gives once tab. 2, repeated here for each column. Zero-shot CLIP with CoOp-style pre-processing is 57.81, and the CoOp-style rows of the same table are left out because this card uses the CLIP-style numbers only tab. 2.

Backbones, 16-shot on ImageNet tab. 1:

Cost, 16-shot ImageNet, one NVIDIA GeForce RTX 3090 §4.2 tab. 3:

Ablations on ImageNet, 16-shot, Tip-Adapter (training-free) unless stated tab. 4:

The 16-shot Tip-Adapter-F entry here is 65.47 while the main table gives 65.51 (see Pitfalls).

Fine-tuning which modules, 16-shot ImageNet tab. 5:

Other datasets, 16-shot fig. 4. The paper shows these only in a figure, and the repository's log gives the numbers exp.log. These values are author-reported, come from the repository log rather than the paper text, and may belong to the ECCV version (see Paper vs code):

The paper gives no separate timing for building the cache.

Paper vs code

Code: https://github.com/gaopengcuhk/Tip-Adapter at commit d0e2d6f8c5feb8b6ce937b757810761f7155d4d5. The repository has no LICENSE file, so the code is not licensed for reuse. Nothing is quoted from it; the differences below are described in our own words. The repository is linked from the paper's abstract, and its README lists two of the paper's authors as contributors, so it is treated as the authors' code.

1. Cache features are augmented and averaged, then normalised once. Paper: each training image gives an L2-normalised feature §3.2 eq. 3. Code: the training images are encoded under random crop and flip for 10 passes (augment_epoch: 10), the raw features are averaged over passes, and only the mean is normalised utils.py; config imagenet.yaml. The paper does not mention augmentation. The values use the first pass's labels utils.py.
2. CLIP logits are multiplied by 100. Paper: zero-shot term f_{test} W_c^T without a scale eq. 6. Code: 100 * features @ text_weights in every logit computation main_imagenet.py, main_imagenet.py, main_imagenet.py, main.py. The paper's α values only match the code's scale; see Pitfalls.
3. Prompt ensembling normalises each template first. Paper: average the 7 templates §4.1. Code: normalise each template embedding, average, normalise again utils.py.
4. Optimiser. Paper: "SGD" with learning rate 0.001 and a cosine schedule §4.1. Code: AdamW with eps=1e-4, learning rate 0.001, and a cosine schedule stepped per batch over all epochs main_imagenet.py, main_imagenet.py. Weight decay and betas are not set, so PyTorch defaults apply.
5. Fine-tuning length on EuroSAT. Paper: 20 epochs for all datasets §4.1. Code: 20 epochs for 10 datasets, 100 for EuroSAT eurosat.yaml.
6. Hyperparameters are per dataset and searched. Paper: β = 5.5 and α = 1.0 for ImageNet in the ablation §4.4; other datasets are not stated. Code: per-dataset initial values and search ranges imagenet.yaml, fgvc.yaml. The search grid is 200 values of β and 20 of α, in equal steps from 0.1 up to just below the configured bound utils.py.
7. Validation vs test for the search. The ImageNet script searches β and α on the test features main_imagenet.py, main_imagenet.py and discards the result (_ =), so its printed accuracy uses the initial values, not the searched ones. The other-datasets script searches on the validation split and evaluates the test split with the chosen values main.py, main.py, main.py, main.py.
8. Fine-tuned epoch is picked on the test set. Both scripts evaluate the test split after each epoch and save the epoch with the best test accuracy main_imagenet.py, main.py. The paper does not say how the epoch was chosen.
9. Few-shot sampling. ImageNet shots are drawn with random.sample per class from the training split imagenet.py. The paper does not describe the sampling; the code fixes seeds to 1 main_imagenet.py.
10. Test split for ImageNet. The "test" split is the ImageNet validation set, which is also the "val" split imagenet.py.
11. Training transform. Code: random resized crop with scale (0.5, 1), bicubic, and horizontal flip, for all datasets main.py. The paper names the operations but not the crop scale §4.1.
12. Matching parts. The affinity \exp(-\beta(1-x)) eq. 5 and the zero-bias, NK-wide keys eq. 7 match the code's adapter main_imagenet.py, main_imagenet.py.
13. Reported numbers vs code log. The repository log's 16-shot ImageNet numbers are close to the paper: Tip-Adapter 62.01 vs 62.03 and CLIP-Adapter 63.59 exp.log. The repository README points to arXiv:2207.09519, a different (ECCV 2022) paper version, for the numbers README.md.

Pitfalls

• Logit scale. Eq. 6 literally applied with the paper's α values weights the cache 100 times more than the code does with the same α. The α values in the tables (best 1.0 on ImageNet; code initial 1.17) match the code's ×100 scale. Implement the ×100 on the zero-shot term, as in the pseudocode, or the numbers will not match.
• Augmentation in the cache. The cache keys in the code average 10 augmented views per image. An implementation that encodes each image once will not reproduce the code's keys. The paper does not state this step.
• Test-set model selection. The ImageNet script runs its α and β search on the test split, and both scripts choose the fine-tuning epoch by test accuracy. Reported test numbers from these choices are optimistic. Use a validation split for every choice before reporting.
• Optimizer ambiguity. The text says SGD and cites Adam and AdamW papers in the same sentence §4.1. The code uses AdamW. Follow the code if you want to match the reported fine-tuning numbers, and say which one you used.
• Internal number mismatches in the paper. The ImageNet text says the 16-shot fine-tuning gain is "+3.44%" §4.2; the table row and the difference of its values give +3.48 tab. 2. The "more shots" table gives 65.47 for 16-shot Tip-Adapter-F tab. 4 vs 65.51 in the main table tab. 2.
• "Some datasets" understates the gap. The text says the training-free variant falls behind CLIP-Adapter "on some datasets" when there are more shots §4.3. In the repository log at 16 shots, Tip-Adapter is below CLIP-Adapter on 10 of 11 datasets; only OxfordPets is higher exp.log. On ImageNet the paper's own table shows it below CLIP-Adapter at every shot count tab. 2, so "comparable" in the abstract is the authors' wording.
• Hidden compute. The cache costs one encoder pass per training image per augmentation, and it is kept in memory at size NK × C. Training-free does not mean zero cost. The paper reports only the fine-tuning time tab. 3.
• Cache size. Going above 16 examples per class, the paper keeps 16 prototypes by averaging random groups, which the authors say still gains with more shots §4.4. The averaging groups are random, so results depend on the draw. The paper repeats the random division five times and averages the scores §4.4.
• Single draw, no variance. The paper reports one number per cell and does not give confidence intervals. The code fixes seed 1. Treat differences under a few tenths of a percent as within noise unless you repeat with new draws.
• Equation numbers. The numbers above were counted from the LaTeX source order because no compiled PDF was available. Check them against the PDF before relying on them.

Lineage

• Builds on CLIP (arXiv:2103.00020) for the image and text encoders, which the paper cites as Radford et al. §1.
• Builds on CLIP-Adapter (arXiv:2110.04544): same architecture, but its weights are set from a cache instead of learned §3.2.
• Compared against CoOp (arXiv:2109.01134) and linear-probe CLIP §4.1.
• Adapter idea from Houlsby et al. (parameter-efficient transfer learning, ICML 2019) §3.1.
• The repository README names CaFo (arXiv:2303.02151) as a later work built on Tip-Adapter. The paper does not say this, so it is not verified against the paper.