ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models

In plain words

Models trained to match pictures with their captions, such as CLIP, can often recognise classes they were never explicitly trained to label, as long as you describe those classes in words. This paper asks how to measure that kind of transfer fairly. It assembles 20 image-classification and 35 object-detection datasets, adds extra text about each class label (dictionary definitions, a WordNet path, and text generated by GPT-3), and provides a toolkit that searches the training settings automatically, so that no one has to tune them by hand.

The paper also changes how the new classifier is started. The usual choice is random numbers. The authors instead start it from the model's own text encoder, and report that this works better, especially when labelled examples are few.

The authors note that several of their findings differ from earlier reports, for example that few-shot training with the proposed initialisation beats zero-shot. Treat the numbers as the authors' own results.

TL;DR

ELEVATER is a benchmark (20 image-classification and 35 object-detection datasets, each with class-name knowledge) and a toolkit with automatic hyper-parameter search §3 §4. The headline method initialises the linear head of a language-augmented model from its text encoder, instead of random weights §4.2. The paper reports this as the better adaptation method across settings, and reports that external knowledge helps zero-shot transfer §5. All numbers are author-reported.

Problem

Language-augmented visual models are usually evaluated in ad hoc subsets, with adaptation details that are hard to reproduce §1. Existing zero-shot benchmarks are class-level, with disjoint train and test categories inside one domain, while task-level transfer across datasets had no public benchmark §2. Adapting such a model with a randomly initialised linear head is the default, but the paper finds that it makes full fine-tuning poor when data is scarce §5.1.

Method

Benchmark data. The image-classification (IC) suite has 20 datasets with 638,429 training and 192,677 test images, and 1,151 concepts in total app. A tab. 1. The object-detection (OD) suite has 35 Roboflow datasets with 132,314 training and 20,070 test images and 314 concepts app. A fig. 1.

Knowledge per class. Each class name is augmented with up to four text sources §3.1 fig. 1 (b):

• WordNet hierarchy: the words on the path from the class node up to the top parent.
• WordNet definition of the synset.
• Wiktionary definition.
• GPT-3 knowledge: five generated sequences per class, produced with an in-context prompt built from random pairs of class names and their Wiktionary definitions.

The paper says WordNet and Wiktionary coverage is incomplete, while GPT-3 covers every class but its quality is hard to judge §3.1.

Tracks. The industry track has no limit on pre-training data, except that benchmark images may not appear in pre-training when zero-shot or few-shot results are reported. The academic track restricts pre-training to named public datasets §3.2.

Baseline checkpoints. IC uses supervised ViT-B/32 (ImageNet-22K), CLIP ViT-B/32, UniCL Swin-Tiny and Knowledge UniCL Swin-Tiny. OD uses DyHead, GLIP, GLIP-A and Knowledge GLIP-A, all with Swin-Tiny backbones §3.2.

Evaluation settings. Sample efficiency covers zero-shot, few-shot and full data. IC uses N = 0, 5, 20, 50 and full. OD uses N = 0, 1, 3, 5, 10 and full. Few-shot subsets are drawn with three random seeds §3.3. Parameter efficiency compares linear probing (only the classifier is trained; for IC the image backbone is frozen) with full fine-tuning §3.3.

Language-augmented models. Such a model has an image encoder f and a text encoder g. Projection layers map both outputs into a joint space of dimension P. Zero-shot prediction takes the class whose averaged prompt embedding has the highest similarity to the image embedding §4.2 fig. 2 (b). Adaptation variants are compared as follows §4.2 fig. 2:

• Random head (original CLIP recipe): a randomly initialised linear layer on the projected features, P by K fig. 2 (c).
• Separate-Head (Language-S): the same P by K layer, initialised with the text-side class vectors V, with bias zero fig. 2 (d).
• Merge-Head (Language-M): a D by K layer on the backbone output, initialised with W_v V. This matches the trainable-parameter count of a language-free model fig. 2 (e) §5.1.

The paper recommends the language-initialised variants as the default. Its argument is that the initial classifier then reproduces the zero-shot classifier app. C, and that external knowledge can be concatenated into the prompt before encoding §4.2.

# Language-initialised linear head, Separate-Head  [§4.2] [fig. 2 (d)]
# Input:  class_names (K), prompt templates, text encoder g
# Output: weight W (P, K) and bias b (K); P = 512 for ViT-B/32 [§5.1]

def init_language_head(class_names, templates, text_encoder):
    # One column per class, built from its prompts  [§4.2]
    columns = []
    for name in class_names:
        # Encode each prompt that contains the class name and any
        # knowledge text  [§4.2] [§3.1]
        prompt_vectors = text_encoder(make_prompts(name, templates))
        # Normalise each prompt embedding, then average them
        # [code: vision_benchmark/evaluation/feature.py#L546-L547]
        prompt_vectors = normalise(prompt_vectors)  # (T, P)
        class_vector = mean(prompt_vectors, axis=0)  # (P,)
        # Renormalise the averaged vector  [code: ...feature.py#L548]
        columns.append(normalise(class_vector))
    W = stack(columns, axis=1)  # (P, K)
    # Bias starts at zero  [§4.2]
    b = zeros(K)
    return W, b

Object detection. GLIP is treated as a language-augmented detector: a CLIP-like dual encoder plus a cross-modal module. Linear probing updates the projection matrices W_v and W_t. Prompt tuning initialises the language input of the cross-modal module from V and trains only V §4.2. The paper says the boundary between linear probing and prompt tuning blurs when the cross-modal module is removed §4.2.

Toolkit. The IC protocol splits each class 80/20 into train and validation. It searches the learning rate and weight decay, trains each setting for 10 epochs, keeps the best validation score, and then runs a final training for 50 epochs §4.1. OD uses a validation set of 1 sample for N = 1, 3 and 5, 3 samples for N = 10, and the full split for the full-data setting §4.1. The learning rate drops on a validation plateau, and training stops after 9 epochs without improvement §4.1.

# Image-classification protocol: search, then final run  [§4.1]
# Input:  train_pool (N samples per class), test_set
# Output: test accuracy of the final run

def run_ic_protocol(train_pool, test_set):
    # 80% / 20% split of each class  [§4.1]
    # [code: vision_benchmark/evaluation/feature.py#L144-L145]
    train_split, val_split = split_each_class(train_pool, 0.2)
    best_lr = None
    best_wd = None
    best_val = -1.0
    # Six learning rates, 1e-6 to 1e-1, log-spaced
    # [code: vision_benchmark/evaluation/full_model_finetune.py#L425]
    for lr in log_grid(1e-6, 1e-1, count=6):
        # Coarse-to-fine weight-decay search; 10 epochs per run  [§4.1]
        wd, val_score = search_weight_decay(lr, train_split, val_split)
        if val_score > best_val:
            best_lr = lr
            best_wd = wd
            best_val = val_score
    # Final run on train plus validation, 50 epochs  [§4.1]
    # Returns the best per-epoch test accuracy, see Pitfalls
    # [code: vision_benchmark/evaluation/full_model_finetune.py#L270-L275]
    return train_final(train_pool, test_set, best_lr, best_wd, epochs=50)

Hyperparameters

Results

Author-reported. Averages over 20 IC datasets tab. 2 and 35 OD datasets tab. 3. Setup: FP32 §5; checkpoints and tracks as in §3.2. Entries are mean ± standard deviation over seeds. "–" means not reported.

Image classification, averaged over 20 datasets tab. 2

The zero-shot value is shared by all rows of the same checkpoint, because adaptation does not change zero-shot accuracy.

Object detection, averaged over 35 datasets tab. 3 (few-shot columns are N = 1, 3, 5, 10)

Knowledge added to CLIP tab. 4 (learned on top of CLIP, averaged over IC; wins are datasets out of 20)

Zero-shot with knowledge sources fig. 5 (bottom table) (zero-shot accuracy, averaged)

Comparison with self-supervised ViT-B/16 checkpoints tab. 5 (IC, averaged; starred row uses language initialisation)

The paper's headline findings §1 §5 are these:

• Language-initialised heads beat random heads in every setting shown, and with them few-shot results are above zero-shot results §5.1 fig. 3.
• With random heads, fine-tuning is far worse than linear probing at few shots tab. 2. With language initialisation, fine-tuning overtakes linear probing from about 20 shots §5.1 tab. 2.
• CLIP beats the supervised ViT in the few-shot settings, and the gap narrows with more data §5.2 fig. 4 (a).
• Knowledge helps zero-shot transfer fig. 5 §5.4, and selecting a knowledge source on validation data gives small gains tab. 4.

Paper vs code

The official toolkit is Computer-Vision-in-the-Wild/Elevater_Toolkit_IC at commit 00d0af78559d5f3d800ae4668210e6bd1f2f84b9, MIT licence. It is the repository of the Computer-Vision-in-the-Wild organisation, which the ELEVATER project page (named in the abstract) links through its issue tracker. The repository covers the image-classification toolkit only. I searched GitHub for other official repositories under the same organisation and found none for object detection. The OD results are therefore not checked against code.

What I checked in the IC code:

• Search protocol. The paper calls the hyper-parameter search a grid §4.1. The code uses six log-spaced learning rates [code: vision_benchmark/evaluation/full_model_finetune.py#L425]. For weight decay it runs a coarse-to-fine search over 97 log-spaced candidates [code: vision_benchmark/evaluation/full_model_finetune.py#L163-L211]. The paper's grid values are not stated.
• Final-run epochs. The paper states 10 search epochs and 50 final epochs §4.1. The code matches: 10 plus 40 extra [code: vision_benchmark/resources/model/vitb32_CLIP.yaml#L38-L39] [code: vision_benchmark/evaluation/full_model_finetune.py#L474].
• Final training data. The paper says the final run is for reporting on the test split §4.1. The code merges train and validation for the final run by default default.py [code: vision_benchmark/evaluation/full_model_finetune.py#L487-L489].
• Head initialisation. The paper says the Separate-Head weights are initialised with V and bias zero §4.2. The code does this with text features and a zero bias [code: vision_benchmark/evaluation/full_model_finetune.py#L106-L116]. The code also normalises each prompt embedding and then the mean [code: vision_benchmark/evaluation/feature.py#L546-L548], which the paper does not describe.
• Frozen parts. In adaptation, the text tower, token embedding, final layer norm, positional embedding and logit scale are all frozen [code: vision_benchmark/evaluation/full_model_finetune.py#L85-L86]. Only the visual side and the head are trained.
• Few-shot sampling. The paper's few-shot protocol draws three random seeds §3.3. Its OD text also says the train/validation split is re-sampled three times §4.1. The IC code draws the few-shot subset with the seed [code: vision_benchmark/evaluation/feature.py#L618-L621]. The validation split takes the first indices of each class, and a code comment says it is not random feature.py.
• Knowledge truncation. The paper describes the WordNet hierarchy as the full path up to the top parent §3.1. The code keeps at most three nodes [code: vision_benchmark/evaluation/feature.py#L477].
• Knowledge text format. The code joins each knowledge string to the class name with a fixed separator and word-tokenises it [code: vision_benchmark/evaluation/feature.py#L530]. The paper does not give the format.
• Learning-rate schedule. The paper does not state an IC schedule. The code uses a constant rate full_model_finetune.py. The yaml carries a WarmupCosine setting that this path does not read [code: vision_benchmark/resources/model/vitb32_CLIP.yaml#L46-L47].
• Logit scale. The paper's appendix contrasts CLIP's learned temperature with downstream adaptation app. C.2. In the code the logit scale is fixed at exp(0) = 1 full_model_finetune.py.

Pitfalls

• Test-set epoch selection in the IC final run. The final run evaluates on the test split after every epoch and keeps the best accuracy and logits [code: vision_benchmark/evaluation/full_model_finetune.py#L270-L275] [code: vision_benchmark/evaluation/full_model_finetune.py#L288]. The reported IC number is therefore the best epoch on the test split, not a fixed final epoch. If you compare against this paper, state whether you did the same. A fixed-epoch protocol may report a different number.
• N-shot is counted before the split. The sampled subset is split 80/20 into train and validation feature.py. During search, each class therefore trains on about 80% of N. The final run merges the two parts back default.py. The paper defines N as the training-set size §3.3, and the code applies it before the split. The sampler is in the external vision_datasets package, so I did not confirm that it returns exactly N per class.
• N = 1 has no training sample. Validation takes ceil(0.2 × 1) = 1 sample per class feature.py. The IC script therefore raises N = 1 to N = 2 and disables the final merge [code: vision_benchmark/commands/linear_probe.py#L71-L74]. The paper's IC settings do not include N = 1 §3.3.
• Silent failures score zero. Any exception during a search run is swallowed and the run is scored 0.0 [code: vision_benchmark/evaluation/full_model_finetune.py#L174-L176]. A broken configuration can therefore lose the search without any error.
• Coarse search may be shorter than intended. The seven coarse starting points are picked by exact floating-point equality with a second logspace full_model_finetune.py. Equality may fail for some points, so fewer than seven may run. I did not run this.
• Cost of the search. Counting the loops, each learning rate gets about 15 weight-decay runs (7 coarse plus 8 refinement), each of 10 epochs, for about 90 runs and 900 epochs in total, plus the 50-epoch final run. This is my count from the code, not a figure from the paper.
• Schedule in the yaml is inactive. Changing the WarmupCosine block in the model yaml will not change IC training [code: vision_benchmark/evaluation/full_model_finetune.py#L396-L403].
• Head initialisation depends on normalisation and scale. With the default configuration the features are not L2-normalised and the logit scale is 1, so the initial logits are not CLIP's zero-shot logits. The argmax at initialisation matches the zero-shot choice (unit-norm weights, same features), but the scale used for the loss does not [code: vision_benchmark/config/default.py#L143-L148]. This is my reading of the code, not a run.
• Template defaults. Prompts come from a per-dataset template list. If a dataset has none, the single template "a photo of a {}" is used [code: vision_benchmark/evaluation/feature.py#L499]. Results on new datasets depend on this choice.
• Figure 5 and text disagree. The text gives UniCL's best knowledge result as 33.93 §5.4. The table in Figure 5 gives 33.95 fig. 5. Use the table.
• OD protocol unchecked. The learning-rate and early-stop rules for OD are from the paper only §4.1 app. B. The code for them is not in the repository I cloned.
• Few-shot numbers are noisy. Several few-shot standard deviations exceed 5 points, for example ±11.9 for CLIP fine-tuning with a random head at 20 shots tab. 2. Single comparisons between rows within that range are not meaningful.

Lineage

• Builds on: the CLIP-style language-augmented visual models that the paper cites as CLIP and ALIGN §1, and the UniCL, GLIP and Knowledge UniCL checkpoints it evaluates §3.2.
• Compared against: ImageNet-1K zero-shot and class-level zero-shot benchmarks (AwA, CUB, SUN, aPY, ZS-ImageNet) tab. 1 §2, VTAB for task-level transfer without language §2, and self-supervised ViT, DeiT, MAE and MoCo-v3 tab. 5.
• Superseded by: nothing in this wiki yet.