---
kind: concept
slug: vision-language-pretraining
title: Vision-language pre-training
summary: >-
  Vision-language pre-training ties images to text. Papers here train encoders with contrastive or align-then-fuse losses, bootstrap captions, train one model to read images and write text, bridge frozen encoders to LLMs, or use a frozen CLIP as a loss, prior or weight-space target. Main trade-off: data and compute cost against per-sample cost and licences.
tags: [vision-language, pretraining, document-understanding, image-quality, contrastive-loss, zero-shot, instruction-tuning, segmentation, image-editing, reinforcement-learning, few-shot, video, robotics, 3d, composed-retrieval, benchmark, model-merging, web-data, data-curation, scaling-laws, distillation, unified-model, frozen-llm, multilingual, prompt-learning, image-captioning, text-to-3d, remote-sensing, compositionality, self-supervision, prompt-tuning, motion, multimodal-llm, nerf, vision-language-action, robot-policy, attention-pooling, style-transfer, visual-prompting, mixture-of-experts]
papers: ["2101.01169", "2102.08981", "2103.17249", "2104.00677", "2106.11539", "2107.07651", "2108.00946", "2110.02624", "2110.14378", "2111.02387", "2111.03930", "2111.09734", "2112.00374", "2112.01455", "2112.03221", "2112.04482", "2112.05744", "2112.09583", "2112.12143", "2112.12750", "2112.14757", "2201.12086", "2202.03052", "2202.11094", "2203.08063", "2203.17274", "2204.02491", "2204.03162", "2204.06125", "2204.08583", "2204.08790", "2205.14865", "2206.02770", "2206.08853", "2208.02816", "2208.05592", "2209.06794", "2210.03117", "2210.04150", "2210.08402", "2212.03191", "2212.05171", "2212.07143", "2212.07796", "2301.12597", "2302.03084", "2302.11713", "2302.12242", "2302.12766", "2303.14968", "2303.15343", "2303.17569", "2304.10592", "2304.14108", "2304.14178", "2305.06500", "2305.08275", "2305.14720", "2306.11029", "2307.06948", "2308.09126", "2310.11513", "2311.16445", "2401.06209", "2411.11394", "2412.16201", "2503.03734"]
related: [contrastive-learning, vision-transformer, multimodal-language-model, open-vocabulary-segmentation, zero-shot-learning, text-guided-image-editing, diffusion-models, few-shot-learning, normalizing-flows, policy-gradient, masked-image-modeling, systematic-generalization, benchmark-validity, linear-probing, model-merging, image-enhancement, soft-prompt-tuning, imitation-learning, retrieval-augmented-generation, transfer-learning, dataset-curation, image-captioning, knowledge-distillation, masked-language-modeling, neural-scaling-laws, instruction-tuning, visual-instruction-tuning, parameter-efficient-fine-tuning, text-to-image-generation, classifier-free-guidance, long-tailed-recognition, neural-radiance-field, image-to-3d-generation, differentiable-rendering, challenge-set-evaluation, foundation-model, semantic-segmentation, object-detection, instance-segmentation, multimodal-fusion, self-supervised-learning, gradient-projection, latent-space-editing, novel-view-synthesis, low-rank-adaptation, gan-inversion, vision-language-action-model, action-chunking, visual-prompting, mixture-of-experts, neural-style-transfer, internal-learning, domain-generalization]
status: draft
updated_at: 2026-11-01
---

## Overview

A vision-language model learns from pairs of images and text, such as photos
and their captions, so that it can later relate new images to new text
[arxiv:2303.15343 §1]. The papers here use such a model in several ways. Some
pre-train two encoders that share one embedding space with a contrastive loss
[arxiv:2303.15343 §3.1] [arxiv:2107.07651 §3.2]. One card adds a self-supervised
image loss to that contrastive objective [arxiv:2112.12750 §3.3]. Some pre-train one
encoder-decoder that reads an image and writes text, for many tasks at once
[arxiv:2202.03052 §3.1] [arxiv:2209.06794 §3.1]. Some keep a large image encoder
and a large language model frozen, and train only a small bridge between them
[arxiv:2301.12597 §3.1] [arxiv:2304.10592 §3.1]. Some use a frozen image-text
model to steer, score or reward a generator or agent that already exists
[arxiv:2112.12143 §3] [arxiv:2305.06500 §2.3] [arxiv:2204.08583 §2]
[arxiv:2206.08853 §4.2]. Some adapt a frozen CLIP with prompts, a gradient
constraint or weight blends [arxiv:2210.03117 §3.2] [arxiv:2205.14865 §3.2]
[arxiv:2208.05592 §2]. The same idea reaches beyond still images. Some cards
extend CLIP-style models to video [arxiv:2208.02816 §3.2] [arxiv:2212.03191 §3.1],
one maps human motion into CLIP's space [arxiv:2203.08063 §3], and some
pre-train robot or 3D encoders against captions, while another uses a frozen model
to map an image into a word token for a text query. One robot policy keeps CLIP
frozen and selects the image patches that match each instruction word
[arxiv:2503.03734 §3.1].

Five more cards carry the same parts into other settings. One edits StyleGAN
images from a text prompt, steering them with CLIP [arxiv:2103.17249 §4]. One
adds a CLIP-based semantic loss to few-photo NeRF training [arxiv:2104.00677
§4.1]. One trains a single transformer family on image, text and joint data
[arxiv:2112.04482 §3.1]. One links a CLIP encoder to LLaMA-7B through a
learnable abstractor [arxiv:2304.14178 §3.1]. One builds a benchmark that shows
where CLIP-based multimodal LLMs miss fine visual detail [arxiv:2401.06209 §1].

Several cards are about the data and how it scales. Two describe how web
image-text pairs are collected and filtered [arxiv:2102.08981 §2.2]
[arxiv:2210.08402 §3.1], and one fits how zero-shot accuracy grows with model
size, data and samples seen [arxiv:2212.07143 §4.1]. Six cards are benchmarks
rather than methods, namely EgoSchema for video question answering
[arxiv:2308.09126 §3.1], CREPE for compositional retrieval
[arxiv:2212.07796 §3] and InfoSeek for knowledge-based visual questions
[arxiv:2302.11713 §3], Winoground for word-order matching
[arxiv:2204.03162 §3.1], GenEval for text-to-image alignment
[arxiv:2310.11513 §3.2] and DataComp for filtering choices
[arxiv:2304.14108 §3.1]. One more card changes the data itself: RemoteCLIP
turns aerial annotations into captions [arxiv:2306.11029 §3.2].

The methods differ in how the two sides meet. Contrastive dual encoders keep
them apart until the loss, and the align-then-fuse designs
[arxiv:2107.07651 §3.1] add a fusion encoder after the contrastive step. Region
methods keep the image side at region level, so each caption word is matched to
a region instead of the whole image [arxiv:2112.12143 §3.2]. Bridge methods
train only a small query module or projection in front of a frozen LLM
[arxiv:2305.06500 §2.2] [arxiv:2304.10592 §3.1]. Frozen-scorer methods train
nothing new in the generator or agent, and use a frozen image-text model as a
signal [arxiv:2204.08583 §2] [arxiv:2112.05744 §3.1].

The cards differ in what they train. [SigLIP](/papers/2303.15343) is a
pre-training recipe for both towers. [ALBEF](/papers/2107.07651) aligns the
towers before fusing them, and uses a slowly updated copy as a teacher.
[BLIP](/papers/2201.12086) is one model that both matches and captions, trained
on bootstrapped captions. [OFA](/papers/2202.03052) and [PaLI](/papers/2209.06794)
are generative models with a single output format. [OpenSeg](/papers/2112.12143)
trains a segmentation model from class-agnostic masks and captions.
[BLIP-2](/papers/2301.12597) trains a Q-Former between frozen parts, and
[MiniGPT-4](/papers/2304.10592) trains one projection layer in its first stage.
[InstructBLIP](/papers/2305.06500) is an instruction-tuning stage on top of an
existing model. Its pre-training stage comes from BLIP-2
[arxiv:2301.12597 §3.1], so this page covers only what InstructBLIP reports
about its own training. [VQGAN-CLIP](/papers/2204.08583) does not pre-train anything; its
card lists this concept because it relies on CLIP's image-text alignment.
[MineCLIP](/papers/2206.08853) pre-trains a video-text encoder on Minecraft
footage. [BLIP-Diffusion](/papers/2305.14720) trains a subject encoder and
leaves the diffusion model's own pre-training alone. [CLIP-Forge](/papers/2110.02624)
trains a flow on top of frozen CLIP features, [unCLIP](/papers/2204.06125)
trains a prior from captions to CLIP image embeddings, and
[Tip-Adapter](/papers/2111.03930) is training-free in its main form, and
its fine-tuned form, Tip-Adapter-F, learns only its keys.
[MaPLe](/papers/2210.03117) trains prompts in both frozen CLIP encoders.

The later cards differ in the same way. [ALPRO](/papers/2112.09583) trains
a video-text encoder and adds an objective that predicts entity labels for
random crops [arxiv:2112.09583 §3.2]. [Voltron](/papers/2302.12766) pre-trains a
robot visual encoder by masking video frames while either reading or writing
the caption [arxiv:2302.12766 §3]. [ELEVATER](/papers/2204.08790) is a benchmark
that also proposes starting a new linear head from the text encoder
[arxiv:2204.08790 §4.2]. [PAINT](/papers/2208.05592) fine-tunes once and then
blends the fine-tuned and zero-shot weights [arxiv:2208.05592 §2].
[Pic2Word](/papers/2302.03084) trains only a map from an image to one word
token [arxiv:2302.03084 §3.2]. [CLIP-LIT](/papers/2303.17569) learns a pair of
text prompts and an illumination-map enhancer [arxiv:2303.17569 §3].
[ULIP-2](/papers/2305.08275) trains a 3D encoder against frozen OpenCLIP
features [arxiv:2305.08275 §3.3]. [CREPE](/papers/2212.07796) and
[InfoSeek](/papers/2302.11713) only evaluate.

The newest cards extend the same ideas beyond whole-image matching. ClipCap
maps a frozen CLIP embedding into GPT-2 inputs [arxiv:2111.09734 §3.1].
GroupViT and OVSeg turn image-text supervision into region-level labels
[arxiv:2202.11094 §3.2] [arxiv:2210.04150 §3.2], and Dream Fields uses a
frozen CLIP score to shape a 3D field [arxiv:2112.01455 §4.2]. CLIP-RLDrive
adds a CLIP-based term to a driving reward [arxiv:2412.16201 §2.4], and
InstruGen uses CLIP only to label rooms in house-tour video before a large
multimodal model writes instructions [arxiv:2411.11394 §3.1]. RemoteCLIP
adapts CLIP to aerial images [arxiv:2306.11029 §3.1], and GenEval scores
generated images with a detector and a CLIP colour check
[arxiv:2310.11513 §3.2]. One card is a survey rather than a method. It notes
that ViTs need large pre-training sets to beat CNNs [arxiv:2101.01169 §2.2]
[arxiv:2101.01169 §3.2.1].

Four more cards take the same parts in new directions. CLIPstyler uses a
frozen CLIP as the only judge of a style-transfer network trained for each
image [arxiv:2112.00374 §3.2]. The zero-shot segmentation baseline keeps CLIP
frozen and uses it to name masked regions [arxiv:2112.14757 §4.2.1]. Visual
prompting leaves a whole model frozen and learns only one additive input frame
[arxiv:2203.17274 §3.2.1]. LIMoE trains one shared sparse-expert transformer on
images and captions together [arxiv:2206.02770 §2.2].

## Variants

The variants below are grouped into ten families. The table gives the family,
what its members train, and when it is the right starting point. Each variant
has its own section further down, with its numbers and caveats.

| Family | Members (card links) | What is trained | Start here when |
|---|---|---|---|
| Web data and scaling | [CC12M](/papers/2102.08981), [LAION-5B](/papers/2210.08402), [OpenCLIP scaling laws](/papers/2212.07143), [DataComp](/papers/2304.14108) | Models for measurement; the pool or the recipe is the object | You choose or filter image-text data, or predict compute |
| Contrastive and align-then-fuse encoders | [ALBEF](/papers/2107.07651), [SigLIP and SigLiT](/papers/2303.15343), [SLIP](/papers/2112.12750), [LIMoE](/papers/2206.02770), [BriVL](/papers/2110.14378) | Two towers, plus a fusion encoder (ALBEF), a self-supervised loss (SLIP) or momentum queues of negatives (BriVL); LIMoE uses one sparse-expert tower | You need embeddings for retrieval or zero-shot classification |
| Single-model encoders, encoder-decoders and generalists | [METER](/papers/2111.02387), [BLIP](/papers/2201.12086), [OFA](/papers/2202.03052), [PaLI](/papers/2209.06794), [FLAVA](/papers/2112.04482) | One network for several tasks, trained from images and text | You want one model for many image-text tasks |
| Frozen image encoder bridged to a language model | [BLIP-2](/papers/2301.12597), [InstructBLIP](/papers/2305.06500), [MiniGPT-4](/papers/2304.10592), [ClipCap](/papers/2111.09734), [mPLUG-Owl](/papers/2304.14178) | A small query module or projection; sometimes LoRA on the LLM | You want chat or captions from a frozen LLM with a small training budget |
| Region-level open-vocabulary segmentation | [OpenSeg](/papers/2112.12143), [GroupViT](/papers/2202.11094), [OVSeg](/papers/2210.04150), [SAN](/papers/2302.12242), [zsseg baseline](/papers/2112.14757) | Mask or grouping models, or a side network on frozen CLIP; the zsseg baseline uses a frozen CLIP as a region classifier | You segment any label list from text |
| Adapting a frozen CLIP | [MaPLe](/papers/2210.03117), [ProGrad](/papers/2205.14865), [Tip-Adapter](/papers/2111.03930), [ELEVATER](/papers/2204.08790), [PAINT](/papers/2208.05592), [CLIP-LIT](/papers/2303.17569), [Pic2Word](/papers/2302.03084), [visual prompts](/papers/2203.17274), [CLAP](/papers/2311.16445) | Prompts, a cache, a head, weight blends, a token mapper, one additive input frame (visual prompts), or a residual MLP on frozen CLIP features (CLAP) | You adapt CLIP to a task with few labels or little compute |
| CLIP as a loss, score or guide for generation and editing | [VQGAN-CLIP](/papers/2204.08583), [StyleGAN-NADA](/papers/2108.00946), [StyleCLIP](/papers/2103.17249), [semantic diffusion guidance](/papers/2112.05744), [Dream Fields](/papers/2112.01455), [DietNeRF](/papers/2104.00677), [MotionCLIP](/papers/2203.08063), [unCLIP](/papers/2204.06125), [CLIP-Forge](/papers/2110.02624), [BLIP-Diffusion](/papers/2305.14720), [CLIPstyler](/papers/2112.00374) | Usually the generator, a latent or a small prior; CLIP stays frozen | You steer or condition a generator with text |
| Video | [X-CLIP](/papers/2208.02816), [InternVideo](/papers/2212.03191), [ALPRO](/papers/2112.09583), [MineCLIP](/papers/2206.08853) | Video adapters or video-text encoders, depending on the card | You work with clips rather than still images |
| Robots, 3D and remote sensing | [Voltron](/papers/2302.12766), [ULIP-2](/papers/2305.08275), [RemoteCLIP](/papers/2306.11029), [OTTER](/papers/2503.03734) | A robot or 3D encoder, a continued CLIP, or a policy head | Your inputs are robot video, point clouds or aerial images |
| Benchmarks and evaluation | [CREPE](/papers/2212.07796), [Winoground](/papers/2204.03162), [InfoSeek](/papers/2302.11713), [EgoSchema](/papers/2308.09126), [GenEval](/papers/2310.11513), [MMVP](/papers/2401.06209) | Nothing; they score models | You need to test a model, not train one |
| Survey | [Transformers in Vision](/papers/2101.01169) | Nothing | You want an overview of vision Transformers |

### Relaxed web filters for captioning data (CC12M)

Card: [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](/papers/2102.08981).

CC12M follows the CC3M pipeline but relaxes its unimodal filters. It has
12,423,374 pairs, against 3,318,333 in CC3M [arxiv:2102.08981 tab. 1]. The
captions are less precise: the authors estimate 76.6% precision against 90.3%
for CC3M, from a 100-caption check by two annotators, so the gap is an estimate
[arxiv:2102.08981 §2.3]. The benefit shows after fine-tuning. A captioner
pre-trained on CC12M and then fine-tuned on COCO reaches 87.4 overall CIDEr on
nocaps validation, against 54.7 with no pre-training. Without fine-tuning,
CC12M is below CC3M, at 27.1 against 29.7 [arxiv:2102.08981 tab. 3]. The
release is a list of image URLs and captions, not model code, and image URLs
disappear over time [/papers/2102.08981]. CC12M is part of the 14M-image
setting of ALBEF and BLIP [arxiv:2107.07651 tab. 8] [arxiv:2201.12086 §4.1].

### Open web-scale pairs filtered by CLIP (LAION-5B)

Card: [LAION-5B: An open large-scale dataset for training next generation image-text models](/papers/2210.08402).

LAION-5B releases 5.85 billion image-text pairs taken from web alt-text, of
which 2.32 billion are English [arxiv:2210.08402 §4]. The main quality filter is
a cosine similarity from OpenAI's CLIP ViT-B/32 of at least 0.28 for English and
0.26 for other languages, which removed about 90% of roughly 50 billion
candidates [arxiv:2210.08402 §3.1]. Potentially offensive pairs are tagged, not
removed, so users can choose [arxiv:2210.08402 §3.2]. Models trained on the
English subset roughly match the original CLIP. A ViT-L/14 trained on
LAION-2B-en reaches 75.2 zero-shot ImageNet top-1, against 75.6 for the
OpenAI model [arxiv:2210.08402 tab. 1]. The metadata are CC BY 4.0, but the
authors say they do not own the images or captions [arxiv:2210.08402 app. A].
Check each source's rights before any commercial use. The released worker code
has filter rules that the paper does not describe [/papers/2210.08402].

### Scaling laws for open CLIP (OpenCLIP on LAION)

Card: [Reproducible scaling laws for contrastive language-image learning](/papers/2212.07143).

The authors train OpenCLIP models from ViT-B/32 to ViT-g/14 on LAION subsets of
80M, 400M and 2B pairs, for 3B to 34B samples seen, and fit power laws to
error [arxiv:2212.07143 §3.2] [arxiv:2212.07143 §4.1]. The best measured
zero-shot ImageNet top-1 is 78.0, for ViT-H/14 on LAION-2B after 34B samples
[arxiv:2212.07143 tab. 1]. The fitted curve predicts 81.92 for ViT-G/14 at 68B
samples. That is an extrapolation, not a measurement [arxiv:2212.07143 tab. 4].
The exponents differ by data source. For ImageNet zero-shot the OpenCLIP fit is
−0.11 against −0.16 for OpenAI CLIP, while for Flickr30K retrieval R@5 it is
−0.19 against −0.10 [arxiv:2212.07143 §4.1] [arxiv:2212.07143 §4.2]. The full
study used 1,058,318 GPU-hours [arxiv:2212.07143 tab. 18]. Forecasts from a
fitted line are extrapolations, and the fit depends on which points are
binned (see Contradictions).

### Data choice under fixed compute (DataComp)

Card: [DataComp: In search of the next generation of multimodal datasets](/papers/2304.14108).

DataComp fixes the training recipe at each of four scales. Participants either
pick a subset of one candidate pool or bring their own data
[arxiv:2304.14108 §3.1] [arxiv:2304.14108 §3.3]. The pool is an index of 12.8B
image URLs and alt-text pairs, with CLIP scores and features published per
sample [arxiv:2304.14108 §3.2]. Models are scored zero-shot on 38 tasks
[arxiv:2304.14108 §3.5].

The best baseline, DataComp-1B, keeps pairs that pass an image-cluster filter
and a CLIP ViT-L/14 score in the top 30% [arxiv:2304.14108 §5.1]. Author-reported,
it reaches 79.2% ImageNet zero-shot, against 75.5% for OpenAI's CLIP at the
same training compute [arxiv:2304.14108 tab. 1]. At the xlarge scale the same
combination gives 0.792 ImageNet and 0.663 average over 38 tasks, against 0.723
and 0.621 with no filtering [arxiv:2304.14108 tab. 3]. Each result is a single
seed with no error bars, so gaps of about a point are not established
[arxiv:2304.14108 app. Checklist].

### Align before fuse with momentum distillation (ALBEF)

Card: [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](/papers/2107.07651).

ALBEF has three parts: a 12-layer ViT-B/16 image encoder, a text encoder made of
the first six BERT layers, and a fusion encoder made of the last six with
cross-attention to the image [arxiv:2107.07651 §3.1]. The image-text
contrastive loss is applied to the two unimodal encoders before fusion
[arxiv:2107.07651 §3.2] [arxiv:2107.07651 eq. 2]. A momentum copy of each
encoder, updated as a moving average with momentum 0.995, gives soft targets. The
targets mix the copy's distribution with the one-hot label, with a weight alpha
that ramps to 0.4 [arxiv:2107.07651 §3.3] [arxiv:2107.07651 eq. 6]
[arxiv:2107.07651 §3.5]. The paper argues this helps with noisy web captions
[arxiv:2107.07651 §1]. Results are author-reported. The 14M-image model reaches
94.1 zero-shot Flickr30K text retrieval R@1, against 88.6 for ALIGN, which uses
1.2B images [arxiv:2107.07651 tab. 3]. Its VQA test-std is 76.04 and its NLVR²
test-P is 83.14 [arxiv:2107.07651 tab. 4]. On 4M images, adding momentum
distillation to both the contrastive and the masked-language losses raises VQA
test-dev from 73.81 to 74.42 [arxiv:2107.07651 tab. 1]. The model needs no object detector or boxes
[arxiv:2107.07651 §1].

### Fully transformer model trained end to end (METER)

Card: [An Empirical Study of Training End-to-End Vision-and-Language Transformers](/papers/2111.02387).

METER pairs a text transformer, RoBERTa by default, with an image transformer,
CLIP-ViT by default. Six co-attention layers let each stream attend to the other
[arxiv:2111.02387 §3.1]. It has no object detector, and it is pre-trained with
masked language modeling and image-text matching only [arxiv:2111.02387 §3.3].
In the authors' ablation, adding masked image modeling lowered VQAv2 accuracy
[arxiv:2111.02387 tab. 7]. Results are author-reported. The CLIP-ViT base model
reaches 77.64 on VQAv2 test-std with about 4M pre-training images, against 76.60
for VinVL, which uses region features [arxiv:2111.02387 tab. 8]. A 14M-image
version with a larger backbone reaches 80.54 [arxiv:2111.02387 tab. 10]; the
released repository has no config for that run (see the card's Paper vs code).

### Bootstrapped captions with a filter (BLIP)

Card: [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](/papers/2201.12086).

BLIP uses one encoder-decoder transformer, the MED, that runs in three modes:
unimodal encoding, image-grounded matching and caption generation
[arxiv:2201.12086 §3.1]. It is trained with an image-text contrastive loss,
an image-text matching loss and a language-modelling loss [arxiv:2201.12086 §3.2].
Its data comes from CapFilt. A captioner writes one synthetic caption per web
image, and a filter removes any text, original or synthetic, whose matching
head says it does not match [arxiv:2201.12086 §3.3]. Decoding uses nucleus
sampling with p = 0.9 [arxiv:2201.12086 §4.3]. In the authors' ablation, with
COCO fine-tuning, removing both captioner and filter gives 127.8 CIDEr and
using both gives 129.7 [arxiv:2201.12086 tab. 1]. With 14M images, COCO 5K text
retrieval R@1 is 80.6, against 77.6 for ALBEF [arxiv:2201.12086 tab. 5]. With
129M images, NoCaps CIDEr reaches 106.3, and 109.6 with the CapFilt-L variant
[arxiv:2201.12086 tab. 7]. The released code has no captioner or filter
training step, only the bootstrapped data, so CapFilt cannot be rerun from the
repository alone [/papers/2201.12086].

### One sequence interface for all tasks (OFA)

Card: [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](/papers/2202.03052).

OFA casts every task, pre-training included, as generation from an instruction
and its inputs, using one Transformer encoder-decoder [arxiv:2202.03052 §3.1]
[arxiv:2202.03052 §3.2]. Words, discrete image codes and box coordinates share
one vocabulary. A 256×256 image becomes a 16×16 sequence of codes, and each box
corner is discretised into a location token [arxiv:2202.03052 §3.1]. The
released scripts use 1000 bins [/papers/2202.03052]. The five model sizes run
from 33M to 930M parameters [arxiv:2202.03052 tab. 1]. Reported results are
author-reported: VQA test-std 82.0 against 80.4 for the best listed prior
[arxiv:2202.03052 tab. 2], SNLI-VE test 91.2 against 86.3 [arxiv:2202.03052 tab. 2],
and COCO captioning with CIDEr optimisation 154.9 against 145.5 for LEMON
[arxiv:2202.03052 tab. 3]. The model is not best everywhere. On unimodal tasks
it is 96.6 on SST-2 against 97.0 for XLNet [arxiv:2202.03052 tab. 6], and its
zero-shot MNLI score is 37.3 against 49.6 for Uni-Perceiver, so instruction
wording matters [arxiv:2202.03052 tab. 9] [arxiv:2202.03052 §4.3].

### Multilingual generator with a scaled ViT (PaLI)

Card: [PaLI: A Jointly-Scaled Multilingual Language-Image Model](/papers/2209.06794).

PaLI reads an image and a text prompt and writes text, so captioning, question
answering and text reading share one interface [arxiv:2209.06794 §3.1]. A ViT
encodes the image, and an mT5 encoder-decoder, initialised from a pre-trained
checkpoint, produces the text [arxiv:2209.06794 §3.1]. The pre-training mix has
eight tasks, including span corruption, split captioning over alt-text, and
multilingual captioning and VQA [arxiv:2209.06794 §3.2] [arxiv:2209.06794 app. A.2].
The largest model, PaLI-17B, uses a 4B-parameter ViT-e [arxiv:2209.06794 app. A.1].
Author-reported results: COCO Karpathy test CIDEr 149.1 against 147.6 for
BEiT-3 [arxiv:2209.06794 tab. 1], VQAv2 test-dev 84.3 against 82.0 for Flamingo
under open-vocabulary generation [arxiv:2209.06794 tab. 3], and a 35-language
average of 53.6 CIDEr on Crossmodal-3600 [arxiv:2209.06794 tab. 2]. Scaling the
ViT from ViT-G to ViT-e adds 3.2 points on average over seven benchmarks, and
scaling the language model from mT5-Large to XXL adds 3.1 [arxiv:2209.06794 §4.5].
The 17B run used 1,024 TPUv4 chips for 7 days [arxiv:2209.06794 app. A.1]. The
pre-training data, WebLI, is not released [arxiv:2209.06794 app. B].

### Text-to-image through a CLIP image embedding (unCLIP)

Card: [Hierarchical Text-Conditional Image Generation with CLIP Latents](/papers/2204.06125).

The model has two stages. A prior maps a caption to a CLIP image embedding, and
a diffusion decoder inverts the CLIP image encoder to render an image
[arxiv:2204.06125 §2]. The diffusion prior is a causal decoder-only transformer
that predicts the clean embedding under a mean-squared error. Two samples are
drawn, and the one with the higher dot product to the text embedding is kept
[arxiv:2204.06125 §2.2]. The decoder is a 3.5B-parameter diffusion model, and
its CLIP embedding enters both the timestep embedding and four context tokens
[arxiv:2204.06125 §2.1] [arxiv:2204.06125 tab. 3]. Zero-shot MS-COCO FID is
10.39 with the diffusion prior, against 12.24 for GLIDE [arxiv:2204.06125 tab. 2].
Human raters preferred unCLIP to GLIDE for diversity at 70.5%, against 48.9% for
photorealism and 45.3% for caption similarity [arxiv:2204.06125 tab. 1]. Because
the image is encoded as a CLIP embedding, the same model can make variations and
text-direction edits [arxiv:2204.06125 §3]. No official code is known
[/papers/2204.06125].

### Text to 3D through a frozen CLIP score (Dream Fields)

Card: [Zero-Shot Text-Guided Object Generation with Dream Fields](/papers/2112.01455).

A NeRF starts from random weights and is optimised so that its renders, taken
from random azimuths, score high against the caption under a frozen CLIP
image-text model. No 3D data is used [arxiv:2112.01455 §4.2]
[arxiv:2112.01455 eq. 7]. Three priors keep the result a single object: a
transmittance penalty that rewards see-through density, random backgrounds
behind each training render, and a fixed bounding cube
[arxiv:2112.01455 §4.5] [arxiv:2112.01455 §4.6].

Author-reported, on object-centric COCO captions. R-Precision reaches 68.3 at
20K iterations and 252-pixel renders, against 31.37 for a simplified NeRF at
10K iterations [arxiv:2112.01455 tab. 1]. Removing the sparsity term lowers
R-Precision from 62.09 to 35.29 [arxiv:2112.01455 tab. 2]. Each object is an
optimisation of about 1 h 12 min for 10K iterations on eight TPU cores, at an
author-estimated $3 to $4 per object [arxiv:2112.01455 app. C]. The paper
anneals the transmittance target from 40% to 88%, but the code uses 50% to 90%
[arxiv:2112.01455 §4.5] [/papers/2112.01455].

### Pairwise sigmoid loss (SigLIP and SigLiT)

Card: [Sigmoid Loss for Language Image Pre-Training](/papers/2303.15343).

The usual CLIP-style loss normalises each image's similarities over the whole
batch with a softmax, in both directions [arxiv:2303.15343 §3.1]. The sigmoid
variant instead treats each of the |B|² image-text pairs as its own yes-or-no
problem: the true pairs on the diagonal are positives, all others are
negatives [arxiv:2303.15343 §3.2]. Each logit is a scaled dot product of
normalised embeddings plus a learned bias. The temperature is learned as an
exponential of a scalar and starts at 10, and the bias starts at −10 [arxiv:2303.15343 §3.2] [arxiv:2303.15343 §4.9].

Because no pair depends on the others, the sum can be computed in blocks, one
per device, while text embeddings are passed around the devices. Only one
block is held in memory at a time [arxiv:2303.15343 §3.3]. The card records
that the released trainer does not implement this chunked form
[/papers/2303.15343].

Variants reported in the paper:

- **SigLiT.** A frozen, pre-trained image tower with a text tower trained from
  scratch [arxiv:2303.15343 §4.1]. The best result is a g/14 image tower on
  four chips for about two days, 84.5% ImageNet zero-shot [arxiv:2303.15343 tab. 1].
  The cost of training that frozen tower is not counted [arxiv:2303.15343 §4.4].
- **SigLIP from scratch.** Both towers trained from scratch on English pairs.
  B/16 reaches 73.4% after five days on 32 chips [arxiv:2303.15343 tab. 1] [arxiv:2303.15343 §4.2].
- **SigLIP with a pre-trained image tower.** The image tower is unlocked and
  started from a pre-trained checkpoint. It gets a 0.1 learning-rate
  multiplier and no weight decay, while the text tower keeps weight decay
  [arxiv:2303.15343 §4.5]. The B/16 run on 16 chips reaches 71.0% [arxiv:2303.15343 tab. 1].
- **mSigLIP.** A multilingual variant with a 250k vocabulary. Its token
  embedding is bottlenecked to a small width K before projection. The authors
  report about a 0.5% ImageNet drop compared with the full embedding
  [arxiv:2303.15343 §4.3].

### Two towers with momentum queues on weakly correlated web pairs (BriVL)

Card: [Towards artificial general intelligence via a multimodal foundation model](/papers/2110.14378).

BriVL has an image encoder and a text encoder, trained with a symmetric
InfoNCE loss. Negatives come from two momentum-updated queues of recent keys,
so the batch does not have to hold every negative [arxiv:2110.14378 §Methods:
Contrastive Loss]. The image side is a CNN with multi-scale patch pooling and a
self-attention block, and it takes no object detector output [arxiv:2110.14378
§Methods: Image Encoder]. The authors pre-train on about 650M Chinese
image-text pairs that are only weakly related, and they argue this pushes the
model towards more abstract matching [arxiv:2110.14378 §Introduction]. Their
author-reported zero-shot results beat CLIP on remote-sensing scene
classification and Chinese news classification. For example, on UCM at a 5/16
unseen/seen split they report 82.41% against 76.02% for CLIP ResNet-50x4
[arxiv:2110.14378 fig. 4a] [arxiv:2110.14378 fig. 5a]. The main run uses a
batch of 2,688 with queues of 13,440 per side, a fixed temperature of 0.07, and
about 10 days on 112 A100 GPUs [arxiv:2110.14378 §Methods: Implementation
Details]. The paper does not say whether embeddings are normalised before the
dot product, and the card flags this as a choice an implementer must make
[/papers/2110.14378]. The paper states that its released model and inference
code are CC BY-NC-ND 4.0, which excludes commercial use and derivatives
[arxiv:2110.14378 §Code availability]. Treat BriVL as a research reference, not
a model to ship.

### Region-word grounding from captions (OpenSeg)

Card: [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](/papers/2112.12143).

The image is represented as a fixed set of N = 128 mask proposals, each with
its own feature vector, instead of one vector or a dense per-pixel map
[arxiv:2112.12143 §3]. The proposals are learned from class-agnostic
COCO masks with a Dice loss, and carry no class names [arxiv:2112.12143 §3.1]
[arxiv:2112.12143 eq. 1]. Caption words are kept only if they are nouns or
adjectives, and each kept word is aligned to the regions with a grounding loss
over the mini-batch, in both the image-to-caption and caption-to-image
directions [arxiv:2112.12143 §3.2] [arxiv:2112.12143 eq. 4]. Caption-only data
can be added with pseudo masks from a teacher model [arxiv:2112.12143 §3.3].
At inference, each class name is embedded with the same text encoder, and each
pixel takes the class with the highest score [arxiv:2112.12143 §3.4]
[arxiv:2112.12143 eq. 6].

Reported effects, author-reported:

- On PASCAL-20 with EfficientNet-B7 trained on COCO and Localized Narratives,
  the paper reports 70.2 mIoU, and 72.2 with prompt engineering. The LSeg
  ViT-L figure it compares with is 52.3 [arxiv:2112.12143 tab. 2]. Some
  baselines in that table are copied from other papers, which the card notes
  [/papers/2112.12143].
- Removing the mask loss, alpha = 0, gives poor performance [arxiv:2112.12143 §3.2].
- On PASCAL-Context-59 with ResNet-101, inference with predicted masks gives
  42.1 mIoU, against 32.1 for pixel-wise inference without masks
  [arxiv:2112.12143 tab. 4].

### Grouping patches from caption contrast alone (GroupViT)

Card: [GroupViT: Semantic Segmentation Emerges from Text Supervision](/papers/2202.11094).

GroupViT is a hierarchical ViT. Learned group tokens merge image patches into
larger segments in stages, and each merge is a hard assignment trained with a
straight-through gradient [arxiv:2202.11094 §3.1]. The only training signal is
image-text contrast. An extra multi-label term adds caption nouns, each put
into short templates, as further positives (K = 3) [arxiv:2202.11094 §3.2]
[arxiv:2202.11094 §4.1]. Zero-shot segmentation labels the final segments by
comparing them with label-name embeddings, with no fine-tuning
[arxiv:2202.11094 §3.3].

Author-reported, with a ViT-S trained on CC12M and filtered YFCC (12M and 14M
pairs) [arxiv:2202.11094 §4.1]. Zero-shot mIoU is 52.3 on PASCAL VOC 2012 and
22.4 on PASCAL Context. A fully supervised DeiT row reaches 53.0 and 35.9
[arxiv:2202.11094 tab. 5]. In the ablation, adding both the hard assignment and
the multi-label term raises VOC mIoU from 12.0 to 39.3
[arxiv:2202.11094 tab. 1]. Background classes such as ground and road are
often grouped correctly but labelled wrongly, so the approach is weakest there
[arxiv:2202.11094 app. C.3] [/papers/2202.11094]. The official code is licensed
for non-commercial use only [/papers/2202.11094], so a commercial build should
follow the paper alone.

### Open-vocabulary segmentation with masked CLIP (OVSeg)

Card: [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](/papers/2210.04150).

This is a two-stage segmenter. A class-agnostic mask model proposes about 100
masks per image [arxiv:2210.04150 §3.2], and CLIP classifies each masked crop
[arxiv:2210.04150 §3.1]. The CLIP image encoder is fine-tuned on masked crops
paired with caption nouns, which are mined with CLIP from COCO captions
[arxiv:2210.04150 §3.2] [arxiv:2210.04150 §4.1]. Mask prompt tuning then
replaces fully masked patch tokens with learned prompts, and it can run with
CLIP frozen [arxiv:2210.04150 §3.3]. The mask and CLIP scores are combined by
a weighted geometric mean [arxiv:2210.04150 §3.1].

Author-reported, with Swin-B and CLIP ViT-L/14 trained on COCO. The model
reaches 29.6 mIoU on ADE20K-150, against 21.1 for OpenSeg with EfficientNet-B7
[arxiv:2210.04150 tab. 1], and 9.0 on ADE20K-847 [arxiv:2210.04150 tab. 1].
Fine-tuning and prompt tuning together give 29.6 on ADE20K-150, against 28.8
with fine-tuning alone and 26.5 with prompts alone [arxiv:2210.04150 tab. 3].
Fine-tuning on fixed COCO-Stuff ground-truth classes, instead of mined
captions, lowers ADE20K-847 from the baseline's 7.3 to 5.3
[arxiv:2210.04150 tab. 2]. CLIP runs on each proposal, so inference takes about
0.6 s per image for ViT-L/14 on an A5000 [arxiv:2210.04150 app. A.9]. The
official code is CC BY-NC 4.0 [/papers/2210.04150], so use the paper for a
commercial build.

### Side network on a frozen CLIP (SAN)

Card: [Side Adapter Network for Open-Vocabulary Semantic Segmentation](/papers/2302.12242).

SAN keeps CLIP frozen, except for its positional embedding, and adds a light side
ViT with 8 blocks and width 240 [arxiv:2302.12242 §4.2]. Its mask branch proposes
masks. Its attention-bias branch adds one bias per CLIP head to CLIP's last
blocks, so CLIP classifies each proposal [arxiv:2302.12242 §3]. The side
network is the main trained part, with 8.4M trainable parameters
[arxiv:2302.12242 tab. 3]. The code also fine-tunes CLIP's positional embedding
by default [arxiv:2302.12242 §4.4].
Trained on COCO-Stuff, it reports 27.5 mIoU on ADE-150 with ViT-B/16 and 32.1
with ViT-L/14 [arxiv:2302.12242 tab. 2]. Table 2 marks OvSeg as ensembled and SAN
as not, so read the ensemble column next to any comparison with it. At 640² input,
SAN runs at 15.2 FPS on one GPU, against 0.8 for SimSeg [arxiv:2302.12242 tab. 3].
Results are author-reported. The "SAN ensemble" rows need a model that is not in
the public code, and the code's final map differs from the paper's formula (see
the card's Paper vs code).

### Bridge from a frozen image encoder to a frozen LLM (BLIP-2)

Card: [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](/papers/2301.12597).

BLIP-2 trains a Querying Transformer (Q-Former) of 188M parameters between a
frozen image encoder and a frozen LLM [arxiv:2301.12597 §3.1]. It has 32
learnable queries of width 768, which cross-attend to the image features in
every other block [arxiv:2301.12597 §3.1]. Stage 1 trains it with three losses
on a frozen ViT: image-text contrastive, image-text matching and image-grounded
generation, which differ only in their attention masks [arxiv:2301.12597 §3.2].
Stage 2 projects the query outputs into the LLM's embedding width and prepends
them as soft visual prompts. The LLM is OPT (decoder-only) or FlanT5
(encoder-decoder) [arxiv:2301.12597 §3.3]. Zero-shot VQAv2 test-dev reaches 65.0
with FlanT5-XXL, against 56.3 for Flamingo-80B, which has 10.2B trainable
parameters [arxiv:2301.12597 tab. 2]. Without stage 1, zero-shot VQA is much
lower for both LLM types, so the first stage should not be skipped
[arxiv:2301.12597 §4.1] [arxiv:2301.12597 fig. 5]. Stage 1 is reported at
under 6 days on 16 A100 (40 GB) GPUs for ViT-g with FlanT5-XXL
[arxiv:2301.12597 §3.4].

The InstructBLIP section below builds on this design.

### Instruction-tuned bridge (InstructBLIP)

Card: [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](/papers/2305.06500).

The image encoder (ViT-g/14) and the language model (FlanT5 or Vicuna) are
both frozen. A Query Transformer (Q-Former) holds a set of learnable queries.
The instruction is fed into the Q-Former alongside the queries, so the
queries cross-attend to the image features under the instruction's guidance
[arxiv:2305.06500 §2.3]. The query outputs are linearly projected and given to
the language model as soft prompts, followed by the instruction text
[arxiv:2305.06500 §2.3]. Only the Q-Former is trained, with the usual
language-modelling loss on the response [arxiv:2305.06500 §2.2].

The paper's training mix uses 26 public datasets across 11 task types,
converted to instructions with 10 to 15 hand-written templates per task.
13 are held in for training and 13 are held out for zero-shot tests
[arxiv:2305.06500 §2.1] [arxiv:2305.06500 §2.2]. Datasets are sampled with a
probability proportional to the square root of their size, then adjusted by
hand. The manual values are not stated [arxiv:2305.06500 §2.4].

Reported effects, on FlanT5-XL:

- Zero-shot on held-out data, Table 1 shows 119.9 CIDEr on NoCaps against
  104.5 for BLIP-2 with the same LLM [arxiv:2305.06500 tab. 1].
- Removing the instruction-aware visual features lowers the held-in average
  from 94.1 to 89.8, with the largest drop on VizWiz, from 32.7 to 25.1,
  then on ScienceQA, from 70.4 to 63.4 [arxiv:2305.06500 tab. 2].
- The finetuning results are reported as 90.7% on ScienceQA (image context)
  for FlanT5-XXL [arxiv:2305.06500 tab. 3].

### Single projection onto a frozen LLM (MiniGPT-4)

Card: [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](/papers/2304.10592).

MiniGPT-4 links a frozen EVA ViT and BLIP-2 Q-Former to a frozen Vicuna LLM
through one trainable linear layer [arxiv:2304.10592 §3]. Stage 1 trains that
layer on about 5M image-caption pairs from Conceptual Captions, SBU and LAION,
for 20,000 steps at batch 256 on four A100 GPUs [arxiv:2304.10592 §3.1]. Stage 1
output is often repetitive or fragmented, so stage 2 fine-tunes on about 3,500
detailed descriptions that the stage 1 model wrote and
ChatGPT then edited, for 400 steps at batch 12 [arxiv:2304.10592 §3.2]. Only the
stage 1 trainable part is stated; the stage 2 scope is not stated
[arxiv:2304.10592 §3.3]. The stage 2 step is the one the authors say fixes the
output. On 100 test images, the failure rate on detailed descriptions falls
from 35% before stage 2 to 2% after, and on poems from 32% to 1%
[arxiv:2304.10592 tab. 3]. Human raters judged 65 of 100 advanced-task requests
satisfied, against 5 of 100 for BLIP-2 [arxiv:2304.10592 tab. 1]. The comparison
is not balanced: BLIP-2 uses FlanT5-XXL and 129M pairs, and MiniGPT-4 uses
Vicuna and about 5M pairs [arxiv:2304.10592 §4.1] [arxiv:2304.10592 app. A.2].
Longer answers hallucinate more, with 9.6 CHAIR_i for long prompts against 7.2
for short ones [arxiv:2304.10592 tab. 5].

### CLIP prefix for a language model (ClipCap)

Card: [ClipCap: CLIP Prefix for Image Captioning](/papers/2111.09734).

ClipCap turns one frozen CLIP image embedding into a short run of prefix
vectors. The prefix sits in front of the caption embeddings of GPT-2, and
training uses the usual caption cross-entropy, with the loss taken over the
caption tokens only [arxiv:2111.09734 §3.1] [arxiv:2111.09734 eq. 5]. Two
mappers are reported. The MLP mapper trains together with GPT-2, giving 156M
parameters. The transformer mapper leaves GPT-2 frozen, so only the mapper
trains, giving 43M parameters [arxiv:2111.09734 §3.2] [arxiv:2111.09734 tab. 1].

Author-reported. On Conceptual Captions the MLP variant reaches 87.26 CIDEr,
against 77.57 for VLP [arxiv:2111.09734 tab. 1 (A)]. On nocaps the transformer
variant reaches 65.83 overall CIDEr, against 63.8 for Oscar, which uses object
tags [arxiv:2111.09734 tab. 1 (B)]. On COCO, Oscar stays ahead at 124.12 CIDEr,
against 113.08 for the transformer variant [arxiv:2111.09734 tab. 1 (C)]. The
timings are on different GPUs, so they are not directly comparable
[arxiv:2111.09734 tab. 1]. The paper names GPT-2 large, but the released code
loads the smaller `gpt2` model [arxiv:2111.09734 §3.1] [/papers/2111.09734].

### Prompt learning in both encoders (MaPLe)

Card: [MaPLe: Multi-modal Prompt Learning](/papers/2210.03117).

MaPLe learns prompt tokens in the text and image branches of a frozen CLIP
ViT-B/16. The vision prompts are not free parameters: a linear layer computes
each one from the language prompt of the same layer [arxiv:2210.03117 §3.2.3].
Prompts are placed in the first nine blocks for base-to-novel generalisation
[arxiv:2210.03117 §4.1]. Trained on 16 labelled images per class, the method
reaches 78.55 harmonic mean over base and novel classes, against 75.83 for
Co-CoOp, and 75.14 novel-class accuracy, against 71.69 [arxiv:2210.03117 tab. 3].
On cross-dataset transfer from ImageNet, the average over ten targets is 66.30,
against 65.74 for Co-CoOp [arxiv:2210.03117 tab. 4]. It adds about 3.55M
trainable parameters, 2.85% of CLIP [arxiv:2210.03117 tab. 6]. At batch size 1
it runs at 60.2 frames per second, against 64.6 for Co-CoOp, so the
no-overhead claim holds for larger batches [arxiv:2210.03117 tab. 6].

### Text-only residual correction on frozen CLIP (CLAP)

Card: [CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts](/papers/2311.16445).

CLAP trains a small residual MLP on top of frozen CLIP text features, using no
images. Each template prompt is paired with a copy in which style words are
dropped or reordered. An InfoNCE term pulls the prompt towards its augmented
copy, and a second term keeps the copy close to the bare class name
[arxiv:2311.16445 §4.2] [arxiv:2311.16445 eq. 4]. At inference the same network
is applied after CLIP's image encoder and text encoder, with a scalar weight α
on its residual branch [arxiv:2311.16445 §4.1.3] [arxiv:2311.16445 app. A.1].
Author-reported, the zero-shot average over PACS, VLCS, OfficeHome and
DomainNet rises from 77.4% for CLIP ViT-B/16 to 79.9%, and the noised-prompt
score rises from 70.4% to 76.1% [arxiv:2311.16445 tab. 3]. The paper also
reports 1-shot linear-probe gains over CLIP of 1.5% to 10% by dataset
[arxiv:2311.16445 §5.2.2]. Its image-augmentation comparison (Im.Aug) needs
generated images, which took about 3.3 hours on DomainNet against 47 minutes
for CLAP [arxiv:2311.16445 §5.1]. The α values were chosen by a search that the
code runs on the test environments, so the reported gains may be optimistic
[/papers/2311.16445]. The repository has no LICENSE file, so its terms are not
stated.

### Long-video question answering as an evaluation (EgoSchema)

Card: [EgoSchema](/papers/2308.09126).

EgoSchema is not a training method. It is a multiple-choice benchmark of
about 5,000 questions on three-minute Ego4D clips, with five options each
[arxiv:2308.09126 §3.1] [arxiv:2308.09126 app. Datasheet]. Published
video-language models score between 19.6% and 32.1% against 20% chance, while
humans reach 75.0% to 76.2% [arxiv:2308.09126 §4.2]. The authors estimate the
median temporal certificate, the span a human needs to verify an answer, at
about 100 seconds [arxiv:2308.09126 §4.1]. Use it to test long-video
understanding, not to train a model. Only 500 correct answers are public, so
full scores come from a submission server [/papers/2308.09126].

### Video-text pre-training with entity pseudo-labels (ALPRO)

Card: [Align and Prompt: Video-and-Language Pre-training with Entity Prompts](/papers/2112.09583).

ALPRO pre-trains a video encoder, a text encoder and a fusion encoder on 5.5M
pairs from WebVid-2M and CC-3M [arxiv:2112.09583 §3.1] [arxiv:2112.09583 §3.3].
Besides a video-text contrastive term over the [CLS] tokens
[arxiv:2112.09583 §3.2.1], it adds a term that predicts soft entity labels for
random video crops. A frozen prompter, itself trained with the contrastive
term, supplies those labels, so no detector or human annotation is needed
[arxiv:2112.09583 §3.2.2]. Pre-training uses 16 A100 GPUs for 100k iterations
at batch size 256 [arxiv:2112.09583 §3.4].

Reported effects, author-reported: after fine-tuning, R@1 on MSRVTT 1k is 33.9,
against 31.0 for FiT, which has the same pre-training data
[arxiv:2112.09583 tab. 2]. Zero-shot R@1 is 24.1 against 18.7
[arxiv:2112.09583 tab. 2]. The same fine-tuned model without pre-training
reaches 16.5 [arxiv:2112.09583 tab. 1]. The authors read the ablation as the
contrastive term helping retrieval most and the entity term helping QA most
[arxiv:2112.09583 §4.2].

### Language-supervised visual encoder for robots (Voltron)

Card: [Language-Driven Representation Learning for Robotics](/papers/2302.12766).

Voltron pre-trains a ViT on Something-Something-v2 video clips with their
captions. It masks most patches of two frames and reconstructs the pixels.
For each example it either conditions the reconstruction on the caption or
asks the decoder to write the caption, and a single probability alpha sets how
often each happens [arxiv:2302.12766 §3.1] [arxiv:2302.12766 §3.2]. The paper
names three models: VCond and VDual use alpha 0, VGen uses alpha 0.5
[arxiv:2302.12766 §3.2]. Captions enter through a frozen DistilBERT encoder
[arxiv:2302.12766 app. B.2].

Reported effects, with frozen features and a small head, against reimplemented
MVP and R3M on the same frames [arxiv:2302.12766 §4]. On grasp affordance,
top-1% precision is 80.71 for VCond ViT-S, against 61.47 for R-MVP and 40.55 for
R-R3M [arxiv:2302.12766 tab. 2]. On referring-expression grounding, total
accuracy is 89.38, against 63.30 for R-R3M and 49.58 for R-MVP
[arxiv:2302.12766 tab. 3]. Removing the language input drops referring accuracy
to 53.44 [arxiv:2302.12766 tab. 4]. The control results are low in absolute
terms, and the authors say few-demonstration control remains open
[arxiv:2302.12766 §5.3].

### Continual CLIP pre-training on remote sensing (RemoteCLIP)

Card: [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](/papers/2306.11029).

RemoteCLIP continues CLIP-style contrastive training on remote-sensing image
and text pairs [arxiv:2306.11029 §3.1]. The captions come from existing
annotations. Box-to-Caption writes five captions per image from object boxes,
and Mask-to-Box turns each class mask into boxes before that step
[arxiv:2306.11029 §3.2.1] [arxiv:2306.11029 §3.2.2]. The training set has
165,745 images with five captions each, 828,725 pairs in all
[arxiv:2306.11029 §4.1.2]. Three backbones are reported: a modified ResNet-50,
ViT-B-32 and ViT-L-14 [arxiv:2306.11029 §4.1.1].

Author-reported. The zero-shot average over 12 datasets rises from 56.02 to
62.41 for ViT-B-32, a gain of 6.39 [arxiv:2306.11029 tab. 3]. ViT-L-14 reaches
50.52 mean recall on RSITMD retrieval [arxiv:2306.11029 tab. 2]. The retrieval
claims need care: the text says RemoteCLIP is best on all three retrieval sets,
but a ResNet-18 baseline (FBCLM) reaches 61.60 on UCM, against 56.36 for the
best RemoteCLIP row [arxiv:2306.11029 §4.2.1] [arxiv:2306.11029 tab. 2]. The
zero-shot average is not uniformly higher, since RemoteCLIP falls below CLIP on
several datasets for every backbone [arxiv:2306.11029 tab. 3]. The largest
model took 233.4 hours on four GPUs [arxiv:2306.11029 §4.1.3]. The training
code is not in the public repository [/papers/2306.11029], and the source
datasets carry their own terms [/papers/2306.11029].

### Few-shot transfer with a language-initialised head (ELEVATER)

Card: [ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models](/papers/2204.08790).

ELEVATER is a benchmark of 20 classification and 35 detection datasets, with
optional class knowledge from WordNet, Wiktionary and GPT-3
[arxiv:2204.08790 §3.1]. Its method part concerns the new linear head. The
Separate-Head variant starts each class weight from the text embedding of that
class's prompts, with bias zero, instead of from random numbers
[arxiv:2204.08790 §4.2]. A Merge-Head variant keeps the parameter count of a
language-free model [arxiv:2204.08790 §4.2].

Reported effects, averaged over 20 classification datasets, CLIP ViT-B/32,
author-reported: at 5 shots, linear probing reaches 65.35 with the language
head against 58.09 with a random head, and at 20 shots fine-tuning reaches
72.19 against 46.76 [arxiv:2204.08790 tab. 2]. Zero-shot accuracy for the same
model is 56.64 [arxiv:2204.08790 tab. 2]. The authors report that random heads
make fine-tuning far worse than linear probing at few shots, while with the
language head, fine-tuning overtakes linear probing from about 20 shots
[arxiv:2204.08790 §5.1] [arxiv:2204.08790 tab. 2]. Adding WordNet, Wiktionary
or GPT-3 text to class names raises zero-shot accuracy for a UniCL model from
27.15 to between 29.92 and 33.95 [arxiv:2204.08790 fig. 5].

### Weight-space patching (PAINT)

Card: [Patching open-vocabulary models by interpolating weights](/papers/2208.05592).

Two steps. First, fine-tune the CLIP visual encoder on the weak task, with the
classifier head fixed to the output of CLIP's text tower
[arxiv:2208.05592 §2] [arxiv:2208.05592 §3]. Second, set each weight to a
blend of the zero-shot and fine-tuned weights, with a single coefficient alpha
between 0 and 1 chosen on held-out validation data [arxiv:2208.05592 §2]. No
parameters are added, so the model still accepts any class list
[arxiv:2208.05592 §3].

Reported effects, ViT-L/14, author-reported: on nine patch tasks, accuracy
rises by 15 to 60 points while ImageNet drops by less than one point
[arxiv:2208.05592 §4] [arxiv:2208.05592 §1]. Patching on one dataset raised
accuracy on a related dataset in 7 of 8 pairs, by 1.1 to 19.2 points, and
lowered it by 1.3 points for EuroSAT to RESISC45 [arxiv:2208.05592 tab. 2].
For sequential patching on ViT-B/32, interpolation changes the combined
accuracy by +11 points, against −4.6 points without it
[arxiv:2208.05592 §5].

### Composed image retrieval through a pseudo word (Pic2Word)

Card: [Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval](/papers/2302.03084).

A three-layer MLP maps a frozen CLIP ViT-L/14 image embedding to one token
that is inserted into the text encoder [arxiv:2302.03084 §3.2]
[arxiv:2302.03084 app. B]. It is trained on images alone, with a symmetric
contrastive loss against the prompt "a photo of" plus that token
[arxiv:2302.03084 §3.2] [arxiv:2302.03084 §4.1]. At query time the token fills
a placeholder in a template, such as "a [domain] of [*]" or "a photo of [*],
[text]" [arxiv:2302.03084 §3.3].

Reported effects, zero-shot, author-reported: CIRR test R@10 is 65.3, against
49.1 for CLIP image plus text and 56.7 for text only [arxiv:2302.03084 tab. 3].
Fashion-IQ validation averages 24.7 at R@10 [arxiv:2302.03084 tab. 4]. On
ImageNet domain conversion, R@10 averages 10.1 against 1.7 for the best
zero-shot baseline [arxiv:2302.03084 tab. 1]. The supervised Combiner trained
on CIRR still scores higher at R@10, 73.2 [arxiv:2302.03084 tab. 3]. The text
alone is a strong baseline on CIRR and Fashion-IQ [arxiv:2302.03084 §4.3].

### Prompt learning for backlit enhancement (CLIP-LIT)

Card: [Iterative Prompt Learning for Unsupervised Backlit Image Enhancement](/papers/2303.17569).

Training uses unpaired backlit and well-lit photos, with CLIP frozen
[arxiv:2303.17569 §3]. A learnable pair of positive and negative prompts, each
16 tokens of width 512, is fitted so that CLIP separates the two sets
[arxiv:2303.17569 §3.1]. The enhancer predicts a one-channel illumination map,
and its output is the input divided by that map [arxiv:2303.17569 §3.1]. The
CLIP term pushes the output away from the negative prompt, and an identity term
keeps the input's features [arxiv:2303.17569 eq. 3] [arxiv:2303.17569 eq. 4].
The prompts are then refined with a margin ranking loss, and the enhancer is
retuned, in turns [arxiv:2303.17569 §3.2].

Reported effects, BAID test set, author-reported: PSNR 21.579 and MUSIQ 55.682,
against 19.740 PSNR for Zero-DCE, the best released baseline in PSNR
[arxiv:2303.17569 tab. 1]. Replacing the CLIP term with an adversarial
discriminator drops PSNR to 17.407 [arxiv:2303.17569 tab. 5]. Fixed word
prompts give PSNR 14.748, against 21.579 for the learned prompts
[arxiv:2303.17569 tab. 3]. The reference images in BAID are retouched by
photographers, and the authors say they are sub-optimal targets
[arxiv:2303.17569 §2].

### 3D shape alignment with generated captions (ULIP-2)

Card: [ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding](/papers/2305.08275).

Each 3D shape is rendered from fixed viewpoints, 12 for Objaverse and 30 for
ShapeNet [arxiv:2305.08275 §4.1]. BLIP-2 writes ten captions per view, and the
one that CLIP ranks highest is kept [arxiv:2305.08275 §4.1]
[arxiv:2305.08275 §3.2]. A 3D encoder is trained so that each point cloud
matches its own image and caption embeddings in a frozen OpenCLIP ViT-G/14
space, with two symmetric contrastive losses [arxiv:2305.08275 §3.3].

Reported effects, zero-shot, author-reported: ULIP-2 with Point-BERT, trained on
Objaverse and ShapeNet, reaches 50.6 top-1 on Objaverse-LVIS and 84.7 top-1 on
ModelNet40 [arxiv:2305.08275 tab. 1]. The largest OpenShape row in the same
table reaches 46.8 on Objaverse-LVIS [arxiv:2305.08275 tab. 1]. On ModelNet40,
holistic BLIP-2 captions score 69.7 against 60.4 for manual captions in the
authors' ablation [arxiv:2305.08275 tab. 5].

### Compositionality test for retrieval models (CREPE)

Card: [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](/papers/2212.07796)

CREPE builds image-to-caption retrieval sets from Visual Genome scene graphs
[arxiv:2212.07796 §3.1]. The systematicity test asks whether a model handles
compounds that were unseen in the training captions of CC-12M, YFCC-15M or
LAION-400M [arxiv:2212.07796 §3.2]. The productivity test uses captions of 4 to
12 atoms, with hard negatives that swap or negate parts of the caption
[arxiv:2212.07796 §3.3].

Reported effects, zero-shot, author-reported: on the atom-foil subset, Recall@1
falls by 12.57 points from seen to unseen compounds for LAION-400M ViT-B/32,
and by 4.38 points for CC-12M RN50 [arxiv:2212.07796 app. B.1]. On the
compound-foil subset, the LAION-400M models gain 2.5 to 3.7 points instead
[arxiv:2212.07796 app. B.1]. On productivity, most models score between 8.35%
and 18.24% at 12 atoms on the atom-foil set, against 16.67% chance, with ALBEF
at 24.34% [arxiv:2212.07796 app. B.1].

### Word-order probe (Winoground)

Card: [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](/papers/2204.03162).

Winoground has 400 expert-written examples. Each has two images and two
captions made of the same words in a different order, so only word order and
visual grounding can tell the pairs apart [arxiv:2204.03162 §3.1]. A model is
scored on a text score, an image score and a group score. The group score needs
all four image-caption pairings to be right [arxiv:2204.03162 §3.2]
[arxiv:2204.03162 eq. 3].

Author-reported. Humans reach 85.50 on the group score. The best model, VinVL,
reaches 14.50, against 16.67 for chance [arxiv:2204.03162 tab. 3]. Section 5.1 says
six models exceed chance on the text score, while the card's Table 3 marks nine
(see Contradictions). Every model is below chance on the group score
[arxiv:2204.03162 §5.1]. The published text does not say which score
function each model uses, so the choice must be tested [arxiv:2204.03162 §3.2].
The images are Getty Images licensed for research, so check the terms before
redistributing them [arxiv:2204.03162 §3.1]. There is no official code
[/papers/2204.03162].

### Knowledge-seeking visual questions (InfoSeek)

Card: [Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?](/papers/2302.11713)

InfoSeek asks questions that need facts beyond the picture, such as when a
pictured building was built [arxiv:2302.11713 §1]. It has 8.9K human-written
questions and about 1.35M questions generated from Wikidata with templates
[arxiv:2302.11713 §3.1] [arxiv:2302.11713 §3.2]. In the No-KB set-up the model
answers from the image and question alone. In the With-KB set-up a CLIP-based
entity recognizer first links the image to a 100K-entry Wikipedia knowledge
base, and a reader then answers from the linked articles [arxiv:2302.11713 §4].

Reported effects, author-reported: fine-tuned PaLI-X reaches 22.1 overall on
the Wikidata split and 10.8 on the human split with No-KB
[arxiv:2302.11713 tab. 4]. A CLIP-to-FiD pipeline reaches 18.2 on the human
split [arxiv:2302.11713 tab. 5]. With the gold entity, the same pipeline reaches
45.6 on that split [arxiv:2302.11713 tab. 5]. Before fine-tuning, PaLI-X scores
2.0 on the Wikidata split [arxiv:2302.11713 fig. 3].

### Per-object checks for text-to-image output (GenEval)

Card: [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](/papers/2310.11513).

GenEval turns each prompt into metadata: object classes, counts, colours and
relative positions [arxiv:2310.11513 §3.1]. A Mask2Former detector trained on
the 80 COCO classes finds the objects, and a zero-shot CLIP ViT-L/14 classifier
picks colours from masked crops [arxiv:2310.11513 §3.2]
[arxiv:2310.11513 app. C.3]. Each image gets a pass or fail, and the six task
scores are averaged [arxiv:2310.11513 §3.2]. The prompt set has 553 prompts
[arxiv:2310.11513 §5.1].

Author-reported. IF-XL reaches an overall score of 0.61 and the best attribute
binding score, 0.35. The best position score is 0.15, for SD-XL
[arxiv:2310.11513 tab. 2]. Against annotators, GenEval agrees on 83% of images,
against 80% for thresholded CLIPScore [arxiv:2310.11513 §4]. The counting and
position thresholds were chosen on the same annotations used to report
agreement, so the held-out figures are the ones to quote
[arxiv:2310.11513 app. A.2]. The detector cannot check fine detail, such as the
number of fingers on a hand [arxiv:2310.11513 §6]. Overall scores move by 0.01
to 0.02 across seeds [arxiv:2310.11513 tab. 2].

### Self-supervised image loss added to CLIP (SLIP)

Card: [SLIP: Self-supervision meets Language-Image Pre-training](/papers/2112.12750).

SLIP trains one shared Vision Transformer with CLIP's image-text contrastive
loss plus a SimCLR loss on two augmented views of each image, summed with
weight 1.0 [arxiv:2112.12750 §3.3] [arxiv:2112.12750 alg. 1]. The CLIP baseline
is the authors' own re-implementation. On YFCC15M for 25 epochs with ViT-B/16,
SLIP reaches 42.8% zero-shot and 72.1% linear ImageNet accuracy, against 37.6%
and 66.5% for that baseline [arxiv:2112.12750 tab. 2]. The gain holds for
ViT-S, B and L, and at 100 epochs [arxiv:2112.12750 §5.1]
[arxiv:2112.12750 §5.2]. Fine-tuning gains are small, from +0.1 to +0.4 over
SimCLR alone [arxiv:2112.12750 tab. 2]. Fine-tuning CLIP from SimCLR-pretrained
weights gives 41.1 zero-shot, 68.2 linear and 82.3 fine-tuned, below SLIP's
42.8, 72.1 and 82.6 [arxiv:2112.12750 tab. 7]. The cost is higher: about three times the image
activations per step [arxiv:2112.12750 §3.3], and 30.5 hours against 22.3 hours
for ViT-B/16 on 64 V100 GPUs [arxiv:2112.12750 §7]. The paper's CC3M and CC12M
runs use 3.1M and 11.0M images, not the public totals [arxiv:2112.12750 app. A].

### Motion in CLIP space (MotionCLIP)

Card: [MotionCLIP: Exposing Human Motion Generation to CLIP Space](/papers/2203.08063).

MotionCLIP trains a transformer auto-encoder on SMPL body motion. Its latent
vector is trained to reconstruct the motion, and to sit close in cosine distance
to the frozen CLIP embedding of the motion's text label and of one rendered frame
[arxiv:2203.08063 §3] [arxiv:2203.08063 eq. 2] [arxiv:2203.08063 eq. 3]. At
inference a prompt goes through frozen CLIP and straight into the decoder, with
no text-motion training [arxiv:2203.08063 §4.2]. In author-reported user studies,
raters preferred MotionCLIP over JL2P 76.7% of the time on in-domain actions,
and 75.3% of the time on out-of-domain Olympic sports [arxiv:2203.08063 tab. 1]. For action
recognition on BABEL-60, top-1 is 40.9% against 41.14% for 2s-AGCN, but top-5
is lower, 57.71% against 73.18% [arxiv:2203.08063 tab. 3]. Without the text
loss, top-1 falls to 4.54% [arxiv:2203.08063 tab. 3]. The paper reports no
automated motion-quality metric [arxiv:2203.08063 §4.2]. The card does not
verify the licences of AMASS, BABEL, the rendered images or the SMPL models,
which the repository says apply [/papers/2203.08063].

### Gradient projection for few-shot prompts (ProGrad)

Card: [Prompt-aligned Gradient for Prompt Tuning](/papers/2205.14865).

ProGrad learns CoOp-style context vectors for a frozen CLIP. It keeps the
cross-entropy update unless that update points against the gradient of a KL term
toward zero-shot predictions. If it does, the conflicting part is removed
[arxiv:2205.14865 §3.2] [arxiv:2205.14865 eq. 4]. Setting the weight to zero
gives back CoOp [arxiv:2205.14865 §3.2]. Averaged over 11 datasets with
ResNet-50, ProGrad reaches 62.61% at one shot against 59.46% for CoOp, and
74.28% against 73.49% at 16 shots [arxiv:2205.14865 tab. 1]. In base-to-new
generalisation at 4 shots, its harmonic mean is 69.06 against 65.58 for CoOp
[arxiv:2205.14865 tab. 3]. Each step runs one extra no-gradient zero-shot pass
and two backward passes [/papers/2205.14865]. The projection can only protect
what the zero-shot prompt already gets right [arxiv:2205.14865 §4].

### Video from an image CLIP model (X-CLIP)

Card: [Expanding Language-Image Pretrained Models for General Video Recognition](/papers/2208.02816).

X-CLIP reuses a pre-trained CLIP image encoder on frames. In each block, one
message token per frame is exchanged across frames by cross-frame attention, and
a small transformer merges the frame features into one video vector
[arxiv:2208.02816 §3.2] [arxiv:2208.02816 §3.2.1]. Class names are turned into
video-specific text prompts by cross-attention over the video's patch tokens
[arxiv:2208.02816 §3.3.1]. On Kinetics-400, ViT-L/14 at 8 frames reaches 87.1
top-1, at 658 GFLOPs per view, against 84.8 for ViViT-H/16×2 at 8316 GFLOPs
[arxiv:2208.02816 tab. 1]. Trained on Kinetics-400 and tested zero-shot,
X-CLIP-B/16 reaches 72.0 on UCF-101 against 58.3 for ActionCLIP, and 65.2 on 220
unseen Kinetics-600 classes against 42.1 [arxiv:2208.02816 tab. 3]
[arxiv:2208.02816 tab. 4]. With two shots on HMDB-51, it reaches 53.0 against
20.9 for Swin-B [arxiv:2208.02816 tab. 5]. Those few-shot rows also compare
pretraining sources, not only methods [arxiv:2208.02816 §4.4]. Freezing the text
encoder costs 0.1 points fully supervised and 2.9 zero-shot, and cuts memory for
two 8-frame inputs from 22 GB to 6 GB [arxiv:2208.02816 tab. 7]. The released
code has no K-shot sampler, so the few-shot numbers cannot be rerun from it alone
[/papers/2208.02816].

### Video foundation model from two encoders (InternVideo)

Card: [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](/papers/2212.03191).

InternVideo trains two encoders separately. One is a masked video ViT-H, and the
other is a CLIP-initialised UniFormerV2-L/14 trained with video-text contrastive
and captioning losses [arxiv:2212.03191 §3.1]. Both are fine-tuned on
Kinetics-710 [arxiv:2212.03191 §3.2]. A cross-model attention module then fuses
the two frozen encoders with a learned linear combination of their scores
[arxiv:2212.03191 §3.3]. Author-reported: 91.1% top-1 on Kinetics-400 against
89.9% for MTV-H, and 77.2% on Something-Something V2 against 75.6% for
MaskFeat [arxiv:2212.03191 tab. 2] [arxiv:2212.03191 tab. 3]. Fine-tuned MSR-VTT
text-to-video R@1 is 55.2 against 49.4 [arxiv:2212.03191 tab. 6]. The masked
encoder was pre-trained on 64 A100 GPUs, and the video-text stage on 128 A100
GPUs for about two weeks [arxiv:2212.03191 §4.2]. The repository does not
contain the cross-model attention, and it does not contain the video-text
pre-training at the paper's settings [/papers/2212.03191].

### Survey of vision Transformers (Transformers in Vision)

Card: [Transformers in Vision: A Survey](/papers/2101.01169).

The survey has no method of its own. It sorts vision Transformer designs by
self-attention variant and by task. Its multi-modal row in
[arxiv:2101.01169 tab. 2] lists three models with numbers compiled from their
papers: ViLBERT, Oscar and UNITER. VQA accuracy is 70.6, 80.37 and 72.47. The
retrieval R@1 is 58.2 for ViLBERT (dataset not named), 57.5 for Oscar on COCO,
and 83.72 for UNITER on Flickr30K. The survey gives no setup for most rows of
that table, and it notes that Oscar relies on pre-trained object detectors
[arxiv:2101.01169 tab. 2]. Check the primary papers before quoting these numbers.

### Text-driven edits of a StyleGAN through CLIP (StyleCLIP)

Card: [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](/papers/2103.17249).

StyleCLIP steers a pretrained StyleGAN2 with CLIP's cosine distance to a text
prompt [arxiv:2103.17249 §4] [arxiv:2103.17249 eq. 1]. It has three routes.
Per-image latent optimisation is the most flexible and the slowest
[arxiv:2103.17249 §4]. A per-prompt latent mapper gives one step per input
after a training run [arxiv:2103.17249 §5] [arxiv:2103.17249 eq. 3]. A global
direction in style space S comes from per-channel CLIP relevance, so one edit
can be dialled up or down [arxiv:2103.17249 §6] [arxiv:2103.17249 eq. 6]
[arxiv:2103.17249 eq. 7].

Author-reported timings, on one GTX 1080Ti. Optimisation takes 98 s per
inference. The mapper needs 10 to 12 h of training and 75 ms per inference. The
global direction needs 4 h of pre-processing and 72 ms per inference
[arxiv:2103.17249 tab. 1]. The mapper's step directions agree fairly well across
inputs, with mean cosine similarity from 0.73 to 0.84 over eight prompts
[arxiv:2103.17249 tab. 2]. The "Surprised" standard deviation is printed as
0.893, against 0.081 to 0.145 for the other prompts, and the paper does not
flag it [arxiv:2103.17249 tab. 2]. The comparisons are visual only, with no
quantitative image-quality or identity metric [arxiv:2103.17249 §7].

### Semantic consistency for few-view NeRF (DietNeRF)

Card: [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](/papers/2104.00677).

DietNeRF trains a NeRF from a few posed photos with the usual pixel loss
[arxiv:2104.00677 eq. 3]. It adds a semantic term. A random unseen pose is
rendered, and the frozen CLIP ViT embedding of that render is pulled towards
the embedding of a random training image [arxiv:2104.00677 §4.1]
[arxiv:2104.00677 alg. 1] [arxiv:2104.00677 eq. 5].

Author-reported, with eight training views on Realistic Synthetic scenes. The
semantic model reaches 23.147 PSNR and 74.9 FID, against 14.934 and 228.1 for
NeRF [arxiv:2104.00677 tab. 1]. A 50k-iteration pixel-only fine-tune raises the
PSNR to 23.591 and lowers the FID to 72.0 [arxiv:2104.00677 tab. 1]
[arxiv:2104.00677 tab. 5]. In the encoder ablation on the Lego scene, a CLIP
ViT-B/32 gives 23.896 PSNR, against 22.059 for an ImageNet ViT-B/32
[arxiv:2104.00677 tab. 4]. The effect is not uniform. On single-view DTU, the
semantic fine-tune lowers PSNR from 15.550 to 14.242 but lowers FID from 266.1
to 190.7 [arxiv:2104.00677 tab. 2]. On the unseen half of Lego, DietNeRF's 20.753
PSNR is below 21.553 for Simplified NeRF [arxiv:2104.00677 tab. 3]. The
repository has no licence file, so do not reuse its code without asking the
authors [/papers/2104.00677].

### One recipe for vision, language and joint tasks (FLAVA)

Card: [FLAVA: A Foundational Language And Vision Alignment Model](/papers/2112.04482).

FLAVA has three transformer encoders: an image encoder, a text encoder with
separate weights, and a multimodal encoder over the projected image and text
states [arxiv:2112.04482 §3.1]. On paired data it uses a global contrastive
loss, masked multimodal modelling and image-text matching [arxiv:2112.04482
§3.2]. On unpaired images and text it uses masked image and masked language
modelling [arxiv:2112.04482 §3.3]. A downstream head sits on the image, text or
multimodal output, depending on the task [arxiv:2112.04482 §3.1].

Pre-training uses 70M public image-text pairs, plus unpaired ImageNet-1k images
and text [arxiv:2112.04482 §4] [arxiv:2112.04482 tab. 2]. Author-reported, over
35 tasks, the macro average is 75.85. The same table gives 66.78 for CLIP
ViT-B/16 trained on 400M pairs, and 61.52 for CLIP ViT-B/16 trained on the same
70M pairs [arxiv:2112.04482 tab. 3]. The card notes that four rows of that
table have their columns out of order, so read it with care
[/papers/2112.04482]. On vision alone the released CLIP is ahead, with a vision
average of 82.57 against 79.44 [arxiv:2112.04482 tab. 3]. On zero-shot
retrieval, FLAVA is below released CLIP on Flickr30K text retrieval R@1 (67.70
against 82.20) and above it on image retrieval (65.22 against 62.08)
[arxiv:2112.04482 tab. 4]. On VQAv2 test-dev it scores 72.8, against 55.3 for
released CLIP, and its linear ImageNet score is 75.5, against 80.2
[arxiv:2112.04482 tab. 5]. Without unimodal initialisation the macro average
falls to 73.72 [arxiv:2112.04482 tab. 3]. The code is BSD-3-Clause, and the card
notes that its configs are not the paper's 70M run [/papers/2112.04482].

### Modular vision encoder with two-stage LLM tuning (mPLUG-Owl)

Card: [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](/papers/2304.14178).

mPLUG-Owl gives LLaMA-7B image input through a CLIP ViT-L/14 encoder and a
learnable visual abstractor that compresses patch features into a short token
sequence [arxiv:2304.14178 §3.1]. Stage 1 aligns images with captions while the
LLM stays frozen. Stage 2 trains LoRA adapters on the LLM with text-only and
image-text instructions [arxiv:2304.14178 §3.2]. The model has 7.2B parameters
[arxiv:2304.14178 §4.1].

Author-reported, on OwlEval: 82 open-ended questions on 50 images, rated by hand.
66 answers are rated A or B, against 54 for MiniGPT-4 [arxiv:2304.14178 §4.2].
On text-only questions, with ChatGPT as judge, the full model scores 86.71%,
against 89.47% for Vicuna-7B [arxiv:2304.14178 tab. 3]. In the authors' ablation,
the full run is the highest of their five runs on all six ability codes, but it is
a single run with no variance reported [arxiv:2304.14178 tab. 2]. The stage-1
hyperparameters disagree within the paper, and the repository releases only the
stage-2 code [/papers/2304.14178]. The code licence is Apache-2.0 [/papers/2304.14178].

### Visual shortcomings of CLIP-based multimodal LLMs (MMVP)

Card: [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](/papers/2401.06209).

The authors look for image pairs that CLIP treats as near-identical while
DINOv2 separates them, and build the MMVP benchmark of simple visual questions on
those pairs [/papers/2401.06209]. They group the failures into nine visual
patterns, the MMVP-VLM benchmark [/papers/2401.06209].

Author-reported. Four human participants average 95.7% on MMVP [arxiv:2401.06209
§2.3] [arxiv:2401.06209 app. B.4]. GPT-4V scores 38.7%, or 40.3% with the
options swapped, and every model except GPT-4V and Gemini is below 25%, the
random level [arxiv:2401.06209 §2.3] [arxiv:2401.06209 app. B.3]. The best CLIP
model on the MMVP-VLM average is SigLIP ViT-SO-14 at 224², with 37.8
[arxiv:2401.06209 tab. 1]. LLaVA with its CLIP encoder scores 5.5 on MMVP.
Interleaved-MoF (I-MoF) adds DINOv2 tokens between the CLIP tokens and reaches
16.7 at 224², with 512 visual tokens [arxiv:2401.06209 tab. 2]
[arxiv:2401.06209 tab. 3]. For LLaVA-1.5, the 224² I-MoF reaches 28.0 with 512
tokens, against 24.7 for the 336² baseline with 576 tokens [arxiv:2401.06209
tab. 3]. The per-pattern correlation with CLIP is 0.87 for LLaVA-1.5 and 0.31
for GPT-4 [arxiv:2401.06209 app. C] [arxiv:2401.06209 tab. 5]. The I-MoF 224²
setting is lower than LLaVA-1.5 on MMBench, TextVQA and VQA-v2 [arxiv:2401.06209
tab. 7]. The released scripts cover only the LLaVA-1 setting. The README claims an MIT
licence, but the card found no LICENSE file at the repository root, so that
licence is not verified [/papers/2401.06209].

### Instruction-selected CLIP patches for a robot policy (OTTER)

Card: [OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction](/papers/2503.03734).

OTTER keeps a CLIP ViT frozen and uses its image features as the visual input
of a robot policy. It takes the last attention output of CLIP's vision
encoder, not the block output [arxiv:2503.03734 §3.1]. For each camera it
scores every patch against every instruction token, takes a softmax over
patches with a learnable temperature, and sums the patches with those weights
[arxiv:2503.03734 §3.1] [arxiv:2503.03734 eq. 4]. Each instruction word thus
gets one visual vector. A cross-attention pooling block turns each camera's
vectors into one token, and the words, the robot's proprioception and the
visual tokens are concatenated into one token per time step. A causal
transformer then predicts chunks of delta end-effector actions
[arxiv:2503.03734 §3.2]. Only the pooling blocks, proprioception encoder,
transformer and action head are trained [arxiv:2503.03734 §3.1]
[arxiv:2503.03734 §3.2].

Author-reported, on a Franka arm: 62% success on 70 unseen pick-and-place
trials, against 12% for fine-tuned Octo and 9% for fine-tuned OpenVLA. The
same table gives 61% for π0-Fast-Droid, evaluated directly and not
fine-tuned [arxiv:2503.03734 §5.1]. Fine-tuning CLIP on the robot data lowers
unseen success to 15% [arxiv:2503.03734 §5.3]. On LIBERO, unseen-task success
is 61%, against 48% for fine-tuned OpenVLA and 26% for fine-tuned Octo
[arxiv:2503.03734 §5.2]. Across four primitives on unseen tasks, the mean is
67%, against 4% for fine-tuned Octo [arxiv:2503.03734 §5.1]. Pre-training on
human hand videos raises unseen success from 62% to 70% and leaves training-task
success unchanged [arxiv:2503.03734 §5.4.1]. The trials are few, and the error
bars are standard errors across trials, not across training seeds
[arxiv:2503.03734 §4.1]. The released code is a PyTorch re-implementation
of the paper's method, not the authors' original JAX code. The card's reading
is that the cloned code fails to build the model, and that is untested. The
paper's numbers may therefore not reproduce from this code [/papers/2503.03734].

### Route-instruction pairs from house-tour video (InstruGen)

Card: [InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models](/papers/2411.11394).

InstruGen makes training data for navigation agents without a simulator. It
samples routes through 3879 YouTube house tours [arxiv:2411.11394 §3.1]
[arxiv:2411.11394 §4.1]. CLIP labels each room node with a room type and
objects, and an action inverse model estimates the action between rooms
[arxiv:2411.11394 §3.1]. ChatGPT-4V then writes an instruction from the
labelled triplets [arxiv:2411.11394 §3.2]. A consistency check extracts the
room and action pairs from the text and regenerates it on mismatch, and regex
rules clean the output [arxiv:2411.11394 §3.2]. A ViLBERT-style agent is
pre-trained on the pairs with masked modelling, path-instruction judgment and
path ranking [arxiv:2411.11394 §3.3].

Author-reported. On R2R val-unseen, the agent reaches 71.10 success with the
consistency check, regex rules and manual review, and 70.9 without the manual
review [arxiv:2411.11394 tab. 1]. On RxR zero-shot val-unseen it reaches 29.35
[arxiv:2411.11394 tab. 5]. The paper does not say how much manual review was
done, nor how it compares with the baseline rows [arxiv:2411.11394 tab. 3]. The
repository holds only a README, so the pipeline cannot be re-run from it
[/papers/2411.11394].

### CLIP score as a reward term (CLIP-RLDrive)

Card: [CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning](/papers/2412.16201).

Here CLIP does not generate data or train the agent's encoder. It scores each
camera frame against three action prompts (slow down, proceed, speed up), and
the score is added to the simulator's reward [arxiv:2412.16201 §2.4]
[eq. 14]. The authors use a ViT-B/32 image encoder, partly re-trained on 500
scene and instruction pairs [arxiv:2412.16201 app. A.2]. The reward is added
to DQN and PPO agents at an unsignalized intersection [arxiv:2412.16201 §2.1] [arxiv:2412.16201 §2.2].

Author-reported. With one other vehicle, CLIP-DQN reaches 96% success against
70% for DQN, and with six vehicles 72% against 50% [arxiv:2412.16201 tab. 2].
CLIP-PPO reaches 38% with one vehicle, below the 78% of plain PPO
[arxiv:2412.16201 tab. 2]. The paper reports 100 test episodes per agent and
no seeds or confidence intervals [arxiv:2412.16201 §3.2]. The reward rule in
the paper and the released code differ, so a re-implementation must pick one
rule [/papers/2412.16201].

### Document pre-training with OCR words, boxes and page image (DocFormer)

Card: [DocFormer: End-to-End Transformer for Document Understanding](/papers/2106.11539).

The image here is a scanned page, not a photo. The model reads OCR words, their
boxes and a ResNet50 grid of the page, and its attention adds spatial terms
whose query and key weights are shared between the text and visual branches
[arxiv:2106.11539 §3.1] [eq. 6] [eq. 7]. Pre-training on a 5M-page subset of
IIT-CDIP uses masked word prediction, page-image reconstruction and a
text-describes-image matching task, then drops those heads
[arxiv:2106.11539 §3] [arxiv:2106.11539 §3.2]. Author-reported: the base model
reaches 83.34 F1 on FUNSD [arxiv:2106.11539 tab. 1] and 96.17% on RVL-CDIP
[arxiv:2106.11539 tab. 2]. The ablations use 1M pages, so they are not
comparable with the headline numbers [arxiv:2106.11539 §4.5]. No official code
was found; implement from the card [/papers/2106.11539].

### Mesh stylisation through a frozen CLIP loss (Text2Mesh)

Card: [Text2Mesh: Text-Driven Neural Stylization for Meshes](/papers/2112.03221).

The input mesh's vertices and faces stay fixed. A small MLP gives each vertex a
colour and a bounded offset along its normal, and CLIP similarity between the
prompt and augmented multi-view renders trains it [arxiv:2112.03221 §3.1]
[arxiv:2112.03221 §3.2]. No 3D generator or dataset is used. Author-reported
user study: 3.90 for natural depiction against 2.83 for a VQGAN-CLIP mesh
baseline [arxiv:2112.03221 tab. 1]. The authors note that CLIP can score a
degenerate solution highly, so the geometry-only term and the augmentations are
needed [arxiv:2112.03221 §4.2]. The MIT-licensed release differs from the paper's
defaults in iterations, learning-rate decay and view averaging, so use its demo
flags for a paper-like run [/papers/2112.03221].

### Text edit of one image or video through a layer (Text2LIVE)

Card: [Text2LIVE: Text-Driven Layered Image and Video Editing](/papers/2204.02491).

A new generator is trained for each edit, on augmented copies of one input. It
outputs a colour layer and an opacity map, composited over the original
[arxiv:2204.02491 §3.1] [eq. 1]. CLIP losses drive it: a composition loss, a
green-screen loss on the layer alone, and a self-similarity structure loss
[arxiv:2204.02491 eq. 2]. For video, the same idea runs on a neural layered
atlas from a frozen pre-trained model [arxiv:2204.02491 §3.2]. Author-reported
human votes: 0.82 to 0.86 for the image method against mask-free image
baselines, and 0.73 to 0.74 against the two video baselines
[arxiv:2204.02491 tab. 1]. One image takes about nine minutes and one video
layer about an hour on the authors' GPUs [arxiv:2204.02491 app. A.3]. The
paper says the method is not suited to creating new objects
[arxiv:2204.02491 §4.5].

### Shared embedding for 3D point clouds (ULIP)

Card: [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](/papers/2212.05171).

The 3D encoder is the only trained part. Its output is pulled toward the frozen
image and text embeddings of SLIP with symmetric contrastive losses
[arxiv:2212.05171 §3.2] [eq. 4] [eq. 5]. Training triplets come from ShapeNet55:
each CAD model gives rendered views, a category caption and its point cloud
[arxiv:2212.05171 §3.1]. Author-reported: PointBERT gains 3.3 points on
ScanObjectNN classification [arxiv:2212.05171 tab. 1]. PointBERT+ULIP reaches
60.4% zero-shot top-1 on ModelNet40, all classes [arxiv:2212.05171 tab. 4]. The
code picks its checkpoint on that same test split, so expect optimistic
zero-shot numbers unless you hold out a validation split [/papers/2212.05171].
The release expects pre-rendered images, which the README puts at about 1 TB
[/papers/2212.05171].

### Quality scoring from joint text labels (LIQE)

Card: [Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective](/papers/2303.14968).

CLIP ViT-B/32 is fine-tuned so that each image is matched with 495 joint text
labels: 5 quality levels, 9 scene types and 11 distortion types
[arxiv:2303.14968 §3.1]. The quality score is the expected level under the
quality marginal, and the scene and distortion heads come from the same softmax
[arxiv:2303.14968 §3.2] [eq. 1] [eq. 3]. Training combines a pairwise fidelity
loss for quality with multi-label and multi-class losses for the other two
heads, weighted by dynamic weight averaging [arxiv:2303.14968 §3.2] [eq. 11].
Author-reported median SRCC: 0.970 on LIVE and 0.919 on KonIQ-10k
[arxiv:2303.14968 tab. 1]. Fine-tuning CLIP matters: a pretrained-CLIP variant
averages 0.693 SRCC against 0.890 for the fine-tuned model in the authors'
two-level ablation [arxiv:2303.14968 tab. 4]. The released MIT code freezes the
softmax temperature that the paper calls learnable [/papers/2303.14968].

### Self-regularised prompts that stay close to frozen CLIP (PromptSRC)

Card: [Self-regulating Prompts: Foundational Model Adaptation without Forgetting](/papers/2307.06948).

Deep prompts are trained on both CLIP encoders. Besides cross-entropy, three
agreement terms pull the prompted features and logits toward the frozen model:
an L1 term on image features, an L1 term on text features and a KL term on
logits [arxiv:2307.06948 §3.2.1] [eq. 2] [eq. 3]. The frozen text target
averages 60 templates per class [arxiv:2307.06948 §3.2.3]. Prompts are
combined across epochs with Gaussian weights [arxiv:2307.06948 §3.2.2]
[eq. 6]. Author-reported base-to-novel harmonic mean over 11 datasets: 79.97,
against 78.55 for MaPLe [arxiv:2307.06948 tab. 1]. The paper reports about
46K tunable parameters against 3.55M for MaPLe [arxiv:2307.06948 §4.3]. On
cross-dataset transfer the average over ten targets is 65.81, below the 66.30
for MaPLe [arxiv:2307.06948 tab. 3]. The MIT-licensed code computes the KL in
the other direction and uses fixed templates and a deterministic epoch weighting,
so check those before reproducing numbers [/papers/2307.06948].

### Per-image style transfer from a text prompt (CLIPstyler)

Card: [CLIPstyler: Image Style Transfer with a Single Text Condition](/papers/2112.00374).

A small U-Net is trained from scratch for each content image and text. Its
output is scored by CLIP with a whole-image directional loss and a patch-wise
directional loss over 64 random 128-pixel crops with perspective augmentation.
Crops whose loss is at or below 0.7 are set to zero [arxiv:2112.00374 §3.2]
[eq. 3] [eq. 4]. A content loss on VGG-19 features and a total-variation term
keep the layout [arxiv:2112.00374 §4.1]. The author-reported CLIP score is
0.2515, against 0.2487 for AdaIN + CLIP [arxiv:2112.00374 tab. 2]. In the
25-rater user study the style score is 3.78, against 3.32 for AdaIN + CLIP
[arxiv:2112.00374 tab. 1]. Training takes about 40 seconds per text on one
RTX 2080 Ti [arxiv:2112.00374 §4.1]. A fast variant trains only a decoder for
one text, and the paper reports under 0.5 seconds of inference per image
[arxiv:2112.00374 §5.1]. The prose on the threshold describes the opposite
direction to equation 4, and the card follows the equation.

### Two-stage open-vocabulary segmentation with a frozen CLIP labeller (zsseg baseline)

Card: [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](/papers/2112.14757).

A MaskFormer trained on seen classes proposes class-agnostic masks. Each masked
crop, with its background filled, is scored by a frozen CLIP ViT-B/16 against
text prompts. The score is a cosine similarity times 100, passed through a
softmax over the class names [arxiv:2112.14757 §4.1] [arxiv:2112.14757 §4.2.1]
[eq. 1]. The per-mask scores are summed into a pixel map [arxiv:2112.14757 §4.3]
[eq. 2]. The paper says the trained and CLIP classifiers are "ensembled" but
gives no formula, so the released code's weighting is a choice the card
describes separately [/papers/2112.14757]. The authors report hIoU 37.8 on
COCO-Stuff in the zero-shot setting, against 20.9 for FCN [arxiv:2112.14757
tab. 2], and 77.5 on Pascal VOC against 50.7 for FCN [arxiv:2112.14757 tab. 3].
Using CLIP alone on the crops gives 28.2 hIoU, and the retrained encoder alone
gives 8.7 [arxiv:2112.14757 tab. 7]. A learned prompt beats a hand-written one,
28.2 against 18.3 [arxiv:2112.14757 tab. 14]. The self-training numbers are
reported, but the paper does not say how its pseudo-labels are made
[arxiv:2112.14757 §6.4].

### Input-only visual prompts for a frozen model (visual prompting)

Card: [Exploring Visual Prompts for Adapting Large-Scale Models](/papers/2203.17274).

One additive prompt of the input's shape is learned by cross-entropy through
a frozen model, then added to every test image [arxiv:2203.17274 §3.2.1]
[eq. 1] [eq. 2]. The default is a padding frame of width 30 on 224 × 224
inputs, trained for 1,000 epochs with SGD at learning rate 40 [arxiv:2203.17274
§3.2.2] [arxiv:2203.17274 §3.4]. On CLIP ViT-B/32 across 12 datasets, the
author-reported average is 78.2 with the prompt and text labels, against 54.2
for the text prompt alone and 83.1 for a linear probe [arxiv:2203.17274 tab. 1].
On the three vision-only classifiers, the same method averages 25.7 to 32.6,
while a linear probe averages 71.7 to 72.6 [arxiv:2203.17274 tab. 4]. So the
method is strong with CLIP and weak with classifiers that have no text side.
The release covers CIFAR-100 only and fixes the CLIP text template, so the
12-dataset numbers cannot be rerun from it [/papers/2203.17274].

### One sparse mixture-of-experts tower for images and text (LIMoE)

Card: [Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts](/papers/2206.02770).

One transformer takes image patches and text tokens, with no modality flag. Its
MoE layers send each token to one of 32 experts [arxiv:2206.02770 §2.2]
[app. A]. The contrastive loss is the usual symmetric softmax over the batch
[arxiv:2206.02770 §2.1] [eq. 1]. Left alone, the router collapses on the
minority modality. The paper adds per-modality entropy losses, weighted by
0.04 in the final model, and batch priority routing [arxiv:2206.02770 §2.2.2]
[app. B] [arxiv:2206.02770 §2.2.3]. Against dense one-tower baselines at
matched compute, the authors report average gains of 10.1 points zero-shot and
12.2 points 10-shot on ImageNet [arxiv:2206.02770 §3.1]. The headline
84.1% zero-shot result for LIMoE-H/14 uses JFT-4B co-training and model souping
[arxiv:2206.02770 §3.2] [app. A.3]. The card records no official code, and the
auxiliary-loss combination is not fully specified [/papers/2206.02770].

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need image and text embeddings for retrieval or zero-shot classification | Pairwise sigmoid loss (SigLIP) | Produces a shared embedding space and trains without a batch-wide softmax [arxiv:2303.15343 §3.2] |
| Small batches, or few accelerators | Pairwise sigmoid loss | Sigmoid beats softmax at small batch in the authors' SigLiT sweep, e.g. 72.5% vs 69.5% at batch 512 on 450M examples [arxiv:2303.15343 tab. 8] |
| A strong, frozen image encoder already exists | SigLiT | Only the text tower is trained, on precomputed image embeddings [arxiv:2303.15343 §4.1] [arxiv:2303.15343 §4.4] |
| Batch size is the bottleneck | Keep batch near 32k | Gains saturate near 32k, and 307k hurt both losses in the SigLIP sweep [arxiv:2303.15343 §4.2] [arxiv:2303.15343 tab. 5] |
| Multilingual text, with a large vocabulary | mSigLIP | Bottlenecked token embedding keeps the vocabulary cost low, at about a 0.5% ImageNet 0-shot drop against the full embedding [arxiv:2303.15343 §4.3] |
| You need embeddings with fine-grained matching, noisy web captions, and no object detector | ALBEF | Aligns before fusing and uses momentum distillation; the paper reports 94.1 zero-shot Flickr30K text retrieval R@1 at 14M images [arxiv:2107.07651 tab. 3] |
| You need a fully transformer VQA or image-text model with no object detector, and can pre-train on about 4M images and about 9M image-caption pairs | METER, CLIP-ViT | Reports 77.64 VQAv2 test-std against 76.60 for region-feature VinVL [arxiv:2111.02387 tab. 8]; masked image modeling did not help in the authors' ablation [arxiv:2111.02387 tab. 7] |
| You need one model for both retrieval and captioning, and cheap web data | BLIP | CapFilt bootstraps captions and filters pairs; the paper reports COCO 5K text retrieval R@1 80.6 at 14M images [arxiv:2201.12086 tab. 5] |
| You need one generative model for captioning, VQA, detection and image-text tasks | OFA | One sequence interface with shared vocabulary; the paper reports VQA test-std 82.0 [arxiv:2202.03052 tab. 2] |
| You need multilingual captions or VQA, and can afford a very large model | PaLI | Reports 53.6 CIDEr on a 35-language captioning average at 17B [arxiv:2209.06794 tab. 2]; the training run used 1,024 TPUv4 for 7 days [arxiv:2209.06794 app. A.1] |
| You need text-to-image with diverse samples, and a CLIP-space model | unCLIP | Diffusion prior reports 70.5% diversity preference against GLIDE [arxiv:2204.06125 tab. 1]; no official code [/papers/2204.06125] |
| You need a pre-training corpus of image-text pairs, and can handle rights checks | LAION-5B | 2.32 billion English pairs filtered by CLIP similarity; the metadata are CC BY 4.0 but images are third-party [arxiv:2210.08402 app. A] |
| You need to know how much compute a CLIP-style model needs for a target accuracy | Scaling-law study | Gives power-law fits for OpenCLIP; predictions beyond the measured runs are extrapolations [arxiv:2212.07143 tab. 4] |
| You need a pre-training corpus that favours captioning and long-tail words, and will fine-tune | CC12M | Fine-tuned captioner reaches 87.4 nocaps CIDEr against 54.7 with no pre-training [arxiv:2102.08981 tab. 3] |
| You need images in many languages and one output format | Multilingual generator (PaLI) | One interface across tasks; the paper reports 53.6 on Crossmodal-3600 [arxiv:2209.06794 tab. 2] |
| You need free-form answers, captions or instructions about an image | Instruction-tuned bridge (InstructBLIP) | Produces generated text through a frozen LLM [arxiv:2305.06500 §2.5] |
| You need zero-shot visual questions, with a frozen LLM and a small trained module | BLIP-2 | Trains only the Q-Former; the paper reports 65.0 zero-shot VQAv2 with FlanT5-XXL [arxiv:2301.12597 tab. 2] |
| You need a chat model that describes images, and you can run a second training stage | MiniGPT-4 | One trained projection, and a second stage that cuts failed detailed descriptions from 35% to 2%; the authors report 65 of 100 advanced-task requests satisfied [arxiv:2304.10592 tab. 1] [arxiv:2304.10592 tab. 3] |
| You need to train only a small module | Instruction-tuned bridge | Only the Q-Former is trained; the paper reports 188M trainable parameters for finetuning [arxiv:2305.06500 §3.5] |
| Tasks are unseen at training time | Instruction-tuned bridge with instructions | Instruction-aware features help on held-out tasks, and the paper's multitask baseline without instructions matches BLIP-2 on unseen data [arxiv:2305.06500 §3.4] |
| You need generated or edited images, and no training budget | VQGAN-CLIP | Trains nothing, and the paper reports 0 V100-hours of training [arxiv:2204.08583 tab. 2] |
| You need images in seconds, for example in an interactive tool | A trained generator, not VQGAN-CLIP | VQGAN-CLIP takes 188.3 seconds per image on a V100 in the authors' runs, against 9.7 seconds for the filtered GLIDE baseline [arxiv:2204.08583 tab. 2] |
| You want to restyle a StyleGAN with text and have no example images | StyleGAN-NADA | Trains from two prompts only; check the checkpoint's licence first [arxiv:2108.00946 §4.1] [arxiv:2108.00946 tab. 4] |
| You want to steer an existing diffusion model at sampling time, with no retraining | Semantic diffusion guidance | Shifts each step along a CLIP gradient; the paper reports FFHQ FID 14.37 for image guidance [arxiv:2112.05744 tab. 1] |
| You need an agent to learn a task with no hand-written reward, and can run RL | MineCLIP reward | The reward is a video-text score; the paper reports Milk Cow 64.5% with it [arxiv:2206.08853 tab. 1] |
| You need a new subject in a diffusion model in under about 120 fine-tuning steps | BLIP-Diffusion | Subject tokens from a Q-Former; the paper reports 0.670 DINO after fine-tuning [arxiv:2305.14720 tab. 1] [arxiv:2305.14720 §3.3] |
| You need text-to-3D with no text-shape pairs, and ShapeNet categories are enough | CLIP-Forge | Trained on shapes and CLIP image features only; the paper reports FID 2425.25 [arxiv:2110.02624 tab. 1] |
| Few-shot classification with CLIP, and no GPU training budget | Tip-Adapter | Builds a key-value cache with no gradient steps; the paper reports 62.03% at 16 shots [arxiv:2111.03930 tab. 2] |
| Few-shot classification, and 20 epochs of training is affordable | Tip-Adapter-F | Learns the keys, reported 65.51% at 16 shots [arxiv:2111.03930 tab. 2] [arxiv:2111.03930 §4.1] |
| You need few-shot CLIP classification that generalises to unseen classes, and can train prompts in both encoders | MaPLe | Reports 78.55 harmonic mean over base and novel classes against 75.83 for Co-CoOp [arxiv:2210.03117 tab. 3] |
| You need to measure long-video understanding | EgoSchema | Multiple-choice benchmark with a reported certificate median of about 100 seconds [arxiv:2308.09126 §4.1] |
| You need a classifier over a new label set with only a few labelled images per class | Language-initialised head (ELEVATER) | Starts each class weight from its text embedding; the paper reports 65.35% at 5 shots against 58.09% with a random head [arxiv:2204.08790 tab. 2] |
| A fine-tuned model is good at a new task but has lost older skills, and you need both | Weight-space patching (PAINT) | One interpolation coefficient, chosen on validation data; the paper reports ImageNet drops of less than one point on nine tasks [arxiv:2208.05592 §4] |
| You need video-text retrieval or video QA, and can afford pre-training on about 5.5M pairs | ALPRO | Reports 33.9 R@1 on MSRVTT 1k against 31.0 for FiT with the same data [arxiv:2112.09583 tab. 2]; pre-training used 16 A100 GPUs [arxiv:2112.09583 §3.4] |
| You train a robot policy from frozen features, and only captioned human video is available | Voltron | Reports 89.38 referring accuracy against 49.58 for the reimplemented R-MVP [arxiv:2302.12766 tab. 3]; the baselines are the authors' own reimplementations |
| You need an image-plus-edit query, and have no labelled triples | Pic2Word | Zero-shot CIRR R@10 of 65.3 against 56.7 for text alone [arxiv:2302.03084 tab. 3]; a Combiner trained on CIRR still reaches 73.2 at R@10 |
| You need to enhance backlit photos, and have only unpaired images | CLIP-LIT | PSNR 21.579 against 19.740 for Zero-DCE on BAID [arxiv:2303.17569 tab. 1]; the BAID references are contested |
| You need to match 3D shapes to text with no 3D captions | ULIP-2 | Reports 50.6 top-1 on Objaverse-LVIS against 46.8 for OpenShape [arxiv:2305.08275 tab. 1]; the caption and ranking code is not in the release |
| You need to test whether a model reasons about attributes, relations or negation | CREPE | Shows drops on unseen compounds; up to 12.57 points on one subset [arxiv:2212.07796 app. B.1] |
| You need to test knowledge-based visual questions | InfoSeek | Fine-tuned PaLI-X reaches 22.1 on the Wikidata split; a pipeline with retrieval reaches 18.2 on the human split [arxiv:2302.11713 tab. 4] [arxiv:2302.11713 tab. 5] |
| You need a cheap captioner from frozen CLIP and GPT-2, and a few GPU hours | ClipCap, transformer variant | Trains a 43M-parameter mapper only; the paper reports 65.83 nocaps CIDEr against 63.8 for Oscar [arxiv:2111.09734 tab. 1] |
| You need text-to-3D from a caption, with no 3D training data, and can spend about an hour per object | Dream Fields | Reports R-Precision 68.3 at 20K iterations [arxiv:2112.01455 tab. 1]; about 1 h 12 min per 10K iterations [arxiv:2112.01455 app. C] |
| You need segmentation with no pixel labels, and can implement from the paper | GroupViT | Reports 52.3 zero-shot mIoU on VOC 2012 from caption-only training [arxiv:2202.11094 tab. 5]; the official code is non-commercial [/papers/2202.11094] |
| You need open-vocabulary segmentation for any label list, and can afford CLIP on about 100 crops per image | OVSeg | Reports 29.6 mIoU on ADE20K-150 [arxiv:2210.04150 tab. 1]; about 0.6 s of CLIP time per image on an A5000 [arxiv:2210.04150 app. A.9] |
| You need open-vocabulary segmentation with a frozen CLIP and few trainable parameters, at 640² input | SAN | Trains 8.4M parameters and runs at 15.2 FPS, against 0.8 for SimSeg [arxiv:2302.12242 tab. 3]; reports 27.5 mIoU on ADE-150 with ViT-B/16 [arxiv:2302.12242 tab. 2]; the ensemble rows are not in the public code |
| You need a remote-sensing image-text model, and can check the source datasets' terms | RemoteCLIP, ViT-B-32 | Reports a 6.39-point zero-shot average gain over CLIP on 12 datasets [arxiv:2306.11029 tab. 3]; a ResNet-18 baseline scores higher on UCM [arxiv:2306.11029 tab. 2] |
| You want to compare or filter image-text training data under fixed compute | DataComp, filtering track | Reports 79.2% ImageNet zero-shot for DataComp-1B against 75.5% for OpenAI's CLIP at the same compute [arxiv:2304.14108 tab. 1] |
| You need to test whether a model reads word order, and you accept a 400-example set | Winoground | Best group score 14.50 against 85.50 for humans [arxiv:2204.03162 tab. 3]; every model is below chance on the group score [arxiv:2204.03162 §5.1] |
| You need per-object checks of text-to-image outputs | GenEval | Reports IF-XL overall 0.61 and attribute binding 0.35 [arxiv:2310.11513 tab. 2]; agrees with annotators on 83% of images [arxiv:2310.11513 §4] |
| You want a better image encoder from captions, and can afford about 3× the image compute per step | SLIP | Reports 42.8% zero-shot on YFCC15M with ViT-B/16, against 37.6% for the CLIP baseline [arxiv:2112.12750 tab. 2]; the cost is 30.5 h against 22.3 h on 64 V100s [arxiv:2112.12750 §7] |
| You need text-to-motion from short phrases, with no paired text-motion data, and accept user-study evidence only | MotionCLIP | Reports 76.7% in-domain preference over JL2P [arxiv:2203.08063 tab. 1]; no automated motion metric [arxiv:2203.08063 §4.2]; check the AMASS and BABEL licences first [/papers/2203.08063] |
| You need to adapt CLIP to a few labelled images per class, and must not lose much zero-shot accuracy | ProGrad | At one shot, 62.61% against 59.46% for CoOp on the 11-dataset average with ResNet-50 [arxiv:2205.14865 tab. 1]; the released base-to-new scripts use 16 context tokens where the paper uses 4 [/papers/2205.14865] |
| You need a video classifier built on a CLIP image model, and cannot afford video-text pre-training | X-CLIP | Reports 87.1% top-1 on Kinetics-400 with ViT-L/14 at 8 frames [arxiv:2208.02816 tab. 1]; freezing the text encoder cuts memory from 22 GB to 6 GB [arxiv:2208.02816 tab. 7] |
| You have only a few labelled videos per class, and a CLIP backbone | X-CLIP, few-shot setting | Reports 53.0% at two shots on HMDB-51, against 20.9% for Swin-B; the comparison also changes the pretraining source [arxiv:2208.02816 tab. 5] [arxiv:2208.02816 §4.4] |
| You need a strong video backbone for action recognition or video-text retrieval, and accept that its fused model cannot be rebuilt from the release | InternVideo | Reports 91.1% top-1 on Kinetics-400 for its headline model [arxiv:2212.03191 tab. 2]; the cross-model fusion is not in the repository [/papers/2212.03191] |
| You want a map of vision Transformer designs before choosing a backbone, and will check numbers in the primary papers | Survey of vision Transformers | Sorts designs by attention type and task; its tables are compiled from the cited papers, often without setups [arxiv:2101.01169 tab. 2] [arxiv:2101.01169 tab. 3] |
| You need a text edit of a StyleGAN image, and can afford a per-prompt training run or a per-channel scan | StyleCLIP | Global direction takes 72 ms per edit after 4 h of pre-processing; the mapper takes 75 ms after 10 to 12 h of training [arxiv:2103.17249 tab. 1]; the paper gives no quantitative quality metric [arxiv:2103.17249 §7] |
| You need a scene from only a few photos, and can afford longer training with a frozen CLIP encoder | DietNeRF | Reports 23.147 PSNR on eight views against 14.934 for NeRF [arxiv:2104.00677 tab. 1]; the semantic term does not improve PSNR in every setting [arxiv:2104.00677 tab. 2] [arxiv:2104.00677 tab. 3] |
| You need one model for image-only, text-only and image-text tasks, and must use public pre-training data | FLAVA | Macro average 75.85 over 35 tasks on 70M public pairs, against 61.52 for CLIP trained on the same pairs [arxiv:2112.04482 tab. 3]; released CLIP is ahead on vision alone [arxiv:2112.04482 tab. 3] |
| You need image-grounded chat on a LLaMA-class model, and can run two training stages | mPLUG-Owl | Reports 66 of 82 OwlEval answers rated A or B against 54 for MiniGPT-4 [arxiv:2304.14178 §4.2]; the stage-1 hyperparameters disagree, and only stage-2 code is released [/papers/2304.14178] |
| You need to know whether a CLIP-based multimodal LLM misses fine visual detail before you deploy it | MMVP | Human participants average 95.7%, against 38.7% for GPT-4V; most models are at random level [arxiv:2401.06209 §2.3] [arxiv:2401.06209 app. B.3] |
| You want more visual detail in a CLIP-based LLM, and can afford extra visual tokens | Interleaved-MoF | The 224² setting raises LLaVA-1.5 to 28.0 MMVP with 512 tokens, against 24.7 at 336²; it is lower than LLaVA-1.5 on MMBench, TextVQA and VQA-v2 [arxiv:2401.06209 tab. 3] [arxiv:2401.06209 tab. 7] |
| You train a robot policy from text instructions, and want to keep the vision-language encoder's alignment | OTTER (frozen CLIP) | Keeps CLIP frozen and selects patches per instruction word; reports 62% unseen pick-and-place against 12% for fine-tuned Octo [arxiv:2503.03734 §5.1]. Fine-tuning CLIP on the same data drops unseen success to 15% [arxiv:2503.03734 §5.3] |
| You are about to fine-tune a CLIP-like encoder on a small robot dataset | Frozen encoder with a learned selection step | The card's ablations show frozen CLIP ahead of fine-tuned CLIP on unseen tasks, 62% against 15% [arxiv:2503.03734 §5.3]; those results come from one set of tasks, so test your own split |
| You need route-instruction pairs for navigation pretraining, and have a budget for a paid multimodal model and unlabelled house-tour video | InstruGen | Reports 71.10 R2R val-unseen success with manual review, and 70.9 without it [arxiv:2411.11394 tab. 1]; the pipeline cannot be re-run from the release [/papers/2411.11394] |
| You want a shaped reward for a driving decision, and can accept a CLIP prompt score as one term | CLIP-RLDrive, DQN variant | Reports 96% success with one other vehicle, against 70% for DQN without CLIP [arxiv:2412.16201 tab. 2]; the paper's PPO variant does worse than plain PPO on success [arxiv:2412.16201 tab. 2] |
| You want to reproduce OTTER's reported model size | The paper's 4-block, width-512 setting | The architecture table gives 11.8M parameters for the smaller setting and 25.5M for the larger one [/papers/2503.03734]; the code defaults to the larger 8-block, width-768 model [/papers/2503.03734] |
| You need to read scanned forms or receipts, with OCR words and boxes available, and can pre-train on millions of pages | DocFormer | Base model reaches 83.34 F1 on FUNSD and 96.17% on RVL-CDIP [arxiv:2106.11539 tab. 1] [arxiv:2106.11539 tab. 2]; the headline numbers use 5M pages, the ablations 1M [arxiv:2106.11539 §4.5] |
| You want to restyle an existing mesh from a text prompt, with no 3D training data, and can run one GPU for about 25 minutes per mesh | Text2Mesh | User-study score 3.90 against 2.83 for the VQGAN-CLIP baseline [arxiv:2112.03221 tab. 1]; about 25 minutes on one RTX 2080 Ti at 1500 iterations [arxiv:2112.03221 app. D.2]; installation needs a CUDA GPU [/papers/2112.03221] |
| You want to edit one image or video from a text prompt, with no mask and no training set, and can spend about nine minutes per image | Text2LIVE | Vote share of 0.82 to 0.86 against mask-free image baselines [arxiv:2204.02491 tab. 1]; video editing needs a pretrained layered atlas [arxiv:2204.02491 §3.2] |
| You need zero-shot 3D shape classification from a fixed class list, and can render images for every CAD model | ULIP | 60.4% zero-shot top-1 on ModelNet40, all classes, for PointBERT+ULIP [arxiv:2212.05171 tab. 4]; the test split also picks the checkpoint, so hold out a validation split [/papers/2212.05171] |
| You need a quality score for photos with no reference image, and have several quality-labelled datasets to train on | LIQE | Median SRCC 0.970 on LIVE and 0.919 on KonIQ-10k [arxiv:2303.14968 tab. 1]; on PLCC it is not the top method on LIVE [arxiv:2303.14968 tab. 1] |
| You adapt CLIP prompts to a task, and must keep accuracy on classes the task did not train on | PromptSRC | Base-to-novel harmonic mean 79.97 against 78.55 for MaPLe [arxiv:2307.06948 tab. 1]; cross-dataset average 65.81 against 66.30 for MaPLe [arxiv:2307.06948 tab. 3] |
| You want to restyle a photo from a text prompt, with no style image, and can spend about 40 seconds of GPU time per image and prompt | CLIPstyler | Style score 3.78 against 3.32 for AdaIN + CLIP in the authors' user study [arxiv:2112.00374 tab. 1]; the threshold direction in the prose contradicts equation 4 [arxiv:2112.00374 eq. 4] |
| You need open-vocabulary segmentation for new class names, and have segmentation labels only for seen classes | zsseg baseline | hIoU 37.8 on COCO-Stuff against 20.9 for FCN [arxiv:2112.14757 tab. 2]; the ensemble weight and the self-training method are not in the paper [arxiv:2112.14757 §4.2.1] [arxiv:2112.14757 §6.4] |
| You can change only the input of a frozen model, and the model is a CLIP-style image-text model | Visual prompts, CLIP | Reports 78.2 on 12 datasets against 54.2 for the text prompt [arxiv:2203.17274 tab. 1]; it is 83.1 for a linear probe, which needs access to features [arxiv:2203.17274 tab. 1] |
| You adapt an ImageNet-trained classifier, or a BiT-M or Instagram model, through its input alone | A linear probe or fine-tuning, not visual prompts | Visual prompts average 25.7 to 32.6 against 71.7 to 72.6 for linear probes on the vision-only models [arxiv:2203.17274 tab. 4] |
| You want one sparse encoder for images and text, and can reproduce the routing losses and the data mix | LIMoE | Reports 10.1-point average zero-shot gain over matched dense one-tower baselines [arxiv:2206.02770 §3.1]; the 84.1 headline uses JFT-4B co-training and souping [arxiv:2206.02770 §3.2] |
| You adapt a frozen CLIP to style or prompt-wording shifts, have class names and prompt templates, and no target images | CLAP | Trains on text only and reports a zero-shot average of 79.9 against 77.4 for CLIP ViT-B/16 [arxiv:2311.16445 tab. 3]; the image route took about 3.3 hours on DomainNet against 47 minutes [arxiv:2311.16445 §5.1] |
| You need a Chinese-language image-text encoder for research, and can accept that its released model is non-commercial | BriVL | Reports zero-shot gains over CLIP on remote sensing and Chinese news [arxiv:2110.14378 fig. 4a] [arxiv:2110.14378 fig. 5a]; its CC BY-NC-ND 4.0 terms exclude commercial use [arxiv:2110.14378 §Code availability] |

The families answer different questions. A sigmoid-trained model, or an
align-then-fuse model such as ALBEF, scores how well an image and a text match.
OpenSeg, OVSeg, SAN and GroupViT score regions against words. Generative models such as OFA, PaLI,
BLIP-2 and MiniGPT-4 produce text; their cards describe no embedding output for
retrieval [arxiv:2305.06500 §2.5] [arxiv:2304.10592 §3.1]. OTTER outputs robot
actions, so its card describes no text or embedding output either
[arxiv:2503.03734 §3.2]. The sampling-time and
few-shot cards use a match score as a loss, reward or cache key to change an
output or a classifier, and they output no embeddings for retrieval.

## Implementation sketch

The core of the sigmoid variant. Each step cites the source. The paper's
Algorithm 1 is the reference [arxiv:2303.15343 alg. 1].

```pseudocode
# Pairwise sigmoid loss over one batch of n image-text pairs
# [arxiv:2303.15343 alg. 1]
def sigmoid_loss(img_emb, txt_emb, t_prime, b):
    # Normalise both towers' outputs to unit length  [arxiv:2303.15343 §3.1]
    x = l2_normalize(img_emb)                   # (n, d)
    y = l2_normalize(txt_emb)                   # (n, d)
    # Temperature is exp of a learned scalar  [arxiv:2303.15343 §4.9]
    t = exp(t_prime)
    # One logit per image-text pair, with a learned bias  [arxiv:2303.15343 §3.2]
    logits = t * (x @ y.T) + b                  # (n, n)
    # +1 on the diagonal (true pairs), -1 everywhere else  [arxiv:2303.15343 §3.2]
    labels = 2 * eye(n) - 1
    # Sum over all pairs, divide by n not n²  [arxiv:2303.15343 §3.2]
    return -sum(log_sigmoid(labels * logits)) / n
```

Notes on the variants:

- The bias enters the logit with a plus sign, as in the algorithm. The printed
  equation in the paper uses a different sign. See the Pitfalls on the card
  [/papers/2303.15343].
- Across devices, the chunked form needs D−1 permutations of the text
  embeddings and a loop over devices. Every cross-device block is all
  negatives [arxiv:2303.15343 §3.3].
- OpenSeg scores a region against a caption word with a softmax over regions,
  using a learnable temperature, and a cosine similarity [arxiv:2112.12143 eq. 2]
  [arxiv:2112.12143 eq. 3]. The region features are pooled from the masks
  [arxiv:2112.12143 §3.2].
- ALBEF and BLIP both mix a momentum teacher's distribution into the
  contrastive target, so that some in-batch negatives count partly as positives
  [arxiv:2107.07651 eq. 6] [arxiv:2201.12086 §3.2]. The mixing weight is alpha in
  ALBEF and not stated in BLIP.
- OTTER's selection step is a softmax over patches, computed separately for
  each instruction word, with a learnable temperature. It runs before the
  pooling. The position embedding enters only the weighted sum, not the
  similarity scores [arxiv:2503.03734 §3.1] [arxiv:2503.03734 eq. 4]. The card's pseudocode
  follows the paper, with the temperature clamp as the only stated bound
  [/papers/2503.03734].

For the instruction-tuned bridge, the per-step flow is: frozen image encoder,
then the Q-Former with instruction tokens and queries, then a linear
projection, then the frozen language model. The loss covers only the response
[arxiv:2305.06500 §2.2] [arxiv:2305.06500 §2.3]. BLIP-2 and MiniGPT-4 share the
same frozen-bridge-LLM shape, but BLIP-2 trains a Q-Former and MiniGPT-4 trains
only one projection in its first stage [arxiv:2301.12597 §3.3] [arxiv:2304.10592 §3.1].

For VQGAN-CLIP, each step decodes the latent, takes augmented crops, embeds them
with CLIP, computes the distance to the prompt embedding, and takes one Adam
step on the latent [arxiv:2204.08583 §2]. The card's pseudocode follows the
text and the code [/papers/2204.08583].

For the sampling-time guidance cards, each reverse step adds a scaled gradient
of a CLIP-based score to the model's mean [arxiv:2112.05744 alg. 1]. For
Tip-Adapter, the training-free prediction adds a cache vote to the zero-shot
logits [arxiv:2111.03930 eq. 6]. Both cards publish pseudocode, and the
differences from the released code are in each card's Paper vs code section.

For the generative model in PaLI, the decoder writes the answer one token at a
time from the encoder states [arxiv:2209.06794 §3.1]. The paper does not say how
the visual tokens enter the encoder. The card's sketch assumes they are
concatenated with the prompt, which is an assumption [/papers/2209.06794].

For SLIP, each step adds a SimCLR term, weighted 1.0, to the CLIP term
[arxiv:2112.12750 §3.3]. For MotionCLIP, the latent is trained on a
reconstruction loss plus two cosine distances to frozen CLIP embeddings
[arxiv:2203.08063 eq. 4]. For ProGrad, one gradient step on the context vectors
uses the cross-entropy gradient, with its conflicting part removed against the
KL gradient [arxiv:2205.14865 eq. 4]. X-CLIP's per-frame message tokens are built
from each frame's [class] token and dropped before the feed-forward layer
[arxiv:2208.02816 §3.2.1].

## Contradictions

Numbers in different cards measure different things (ImageNet and retrieval for
SigLIP, PASCAL and ADE mIoU for OpenSeg, held-out multimodal benchmarks for
InstructBLIP, human ratings for VQGAN-CLIP, FFHQ FID for SDG, DreamBench DINO
for BLIP-Diffusion, FID on ShapeNet for CLIP-Forge, nocaps CIDEr for CC12M and
BLIP, COCO and VQAv2 test splits for OFA and PaLI, base-to-novel harmonic mean
for MaPLe, ADE20K and PASCAL mIoU for GroupViT and OVSeg, group score for
Winoground, ImageNet zero-shot for DataComp, mean recall for RemoteCLIP, the overall score for GenEval, and the MMVP and
OwlEval results, DocFormer's FUNSD and RVL-CDIP scores, Text2Mesh and Text2LIVE user-study votes, LIQE's SRCC, ULIP's zero-shot ModelNet40 accuracy, and the CLIP baselines in CLAP and BriVL, which use different backbones and datasets), so they should not be compared directly. Where two cards share a
measure, the numbers agree in the cases checked. For example, the ALBEF 14M
COCO text retrieval R@1 (77.6) appears in both the ALBEF and BLIP tables [arxiv:2107.07651 tab. 2] [arxiv:2201.12086 tab. 5], and so
does its zero-shot Flickr30K text retrieval R@1 (94.1) [arxiv:2107.07651 tab. 3]
[arxiv:2201.12086 tab. 6].

Cross-paper disagreements, each side cited:

- **COCO captioning with cross-entropy.** OFA reports 145.3 CIDEr against 143.3
  for SimVLM [arxiv:2202.03052 tab. 3]. PaLI reports 149.1 on the COCO Karpathy
  test and calls its result the best cross-entropy-trained result on COCO
  [arxiv:2209.06794 tab. 1] [arxiv:2209.06794 §4.1]. The two cards do not state
  the same split and baselines, so the numbers are not directly comparable.
- **BLIP-2 numbers in MiniGPT-4.** MiniGPT-4 reports BLIP-2 at GQA 42.4 with
  129M pairs [arxiv:2304.10592 app. A.2]. The BLIP-2 card reports GQA test-dev
  44.7 for its FlanT5-XXL model in zero-shot setting [arxiv:2301.12597 tab. 2].
  The configurations differ, and the two cards do not explain the gap.
- **Pre-training size in OFA.** The abstract says 20M image-text pairs, but
  the data table has 15.25M captioning and matching samples, 2.92M VQA samples
  and 3.20M grounding samples [arxiv:2202.03052 tab. 11]. The card records the
  text does not say how these relate [/papers/2202.03052].
- **Scaling-law fits.** The paper says both curves are fitted on the Pareto
  frontier [arxiv:2212.07143 fig. 1] [arxiv:2212.07143 §4.1]. The released
  notebook bins the OpenCLIP points into five compute bins, while it fits the
  OpenAI points by ordinary least squares over all points [/papers/2212.07143].
  The two exponents therefore come from different fits.
- **Batch sizes in the scaling study.** The text gives global batches of
  "86-88K", and the run table has 79K, 64K, 33K and 32K values for several runs
  [arxiv:2212.07143 §3.2] [arxiv:2212.07143 tab. 18].
- **LAION-2B in two papers.** The LAION-5B card reports 75.2 zero-shot ImageNet
  top-1 for ViT-L/14 trained on LAION-2B-en [arxiv:2210.08402 tab. 1]. DataComp
  reports 73.1 for LAION-2B under its own fixed recipe [arxiv:2304.14108 tab. 1]
  [arxiv:2304.14108 §3.4]. The LAION-5B card gives no matching recipe, so the two
  numbers are not directly comparable.

Inconsistencies inside a single paper, or between a paper and its code, recorded
so readers do not trust either side blindly:

- SigLiT with a B/8 image tower is 79.7% in the text and 79.8% in Table 1
  [arxiv:2303.15343 §4.4] [arxiv:2303.15343 tab. 1].
- The bias sign in the printed sigmoid-loss equation differs from Algorithm 1
  [arxiv:2303.15343 §3.2] [arxiv:2303.15343 alg. 1].
- FlanT5-XL is listed as 3B parameters in one section and 4B in another
  [arxiv:2305.06500 §2.6] [arxiv:2305.06500 §3.5].
- VQGAN-CLIP's paper gives a learning rate of 0.15 and 400 iterations
  [arxiv:2204.08583 §2]. The card records that the official code's defaults
  differ, so runs made with the code do not reproduce the paper settings
  [/papers/2204.08583].
- SDG's text guidance defaults to scale 120 in the text, but the FFHQ text row
  in Table 1 (FID 28.38) equals the scale-320 row of Table 3, not the 120 row
  [arxiv:2112.05744 §4.1] [arxiv:2112.05744 tab. 1] [arxiv:2112.05744 tab. 3].
- Tip-Adapter's abstract calls the training-free method comparable to
  CLIP-Adapter, while its ImageNet table puts it below CLIP-Adapter at every
  shot count [arxiv:2111.03930 tab. 2].
- CLIP-Forge's ablation reports FID 2591.87 for the RealNVP-D setting, against
  2425.25 for the same setting in the main table. The paper attributes the gap
  to seed selection [arxiv:2110.02624 tab. 4] [arxiv:2110.02624 tab. 1]
  [arxiv:2110.02624 app. A].
- The BLIP-Diffusion text gives a fine-tuning learning rate of 5e-5, and the
  appendix and code give 5e-6 [arxiv:2305.14720 §3.3] [arxiv:2305.14720 app. A.5].
- ALPRO's text sets the pseudo-label threshold at 0.2 on a normalised score.
  The released code compares an argmax index against it, so the threshold is
  never applied to a score [arxiv:2112.09583 §3.4] [/papers/2112.09583].
- CREPE's text calls the change on compound foils "little (1–3%) to no
  difference", but the LAION-400M rows gain 2.5 to 3.7 points
  [arxiv:2212.07796 §5.1] [arxiv:2212.07796 app. B.1]. Its abstract says
  Recall@1 drops "by up to 12%", which is 12.57 points on the atom-foil subset
  for one model, against 6.28 points on the combined set
  [arxiv:2212.07796 app. B.1].
- InfoSeek's written metric is VQA accuracy with a one-year tolerance for time
  answers. The released code scores these by exact match
  [arxiv:2302.11713 app. A.3] [/papers/2302.11713].
- CLIP-LIT's loss equation gives the identity term a weight of 0.9, while the
  code multiplies the two terms by 320 and 25 [arxiv:2303.17569 eq. 5]
  [/papers/2303.17569].
- ELEVATER's text gives UniCL's best knowledge result as 33.93, and its figure
  table gives 33.95 [arxiv:2204.08790 §5.4] [arxiv:2204.08790 fig. 5]. The
  table is the one to use.
- Pic2Word's text says the mapper has about 0.8M parameters. The layer widths
  in its appendix give about 0.92M [arxiv:2302.03084 §3.2] [/papers/2302.03084].
- ALBEF's footnote to §3.5 says the RandAugment list drops colour operations, because
  captions mention colours, while the released code still uses four colour
  operations [arxiv:2107.07651 §3.5] [/papers/2107.07651].
- ALBEF's warm-up is 1,000 iterations in the paper. By the card's reading of the
  released scheduler, it runs over about 2,000 iterations [arxiv:2107.07651 §3.5]
  [/papers/2107.07651].
- BLIP's text gives a linear learning-rate decay with rate 0.85, while the
  released code decays by 0.9 per epoch [arxiv:2201.12086 §4.1] [/papers/2201.12086].
- BLIP-2's stage 2 minimum learning rate is 5e-5 in the paper and 1e-5 in the
  code [arxiv:2301.12597 §3.4] [/papers/2301.12597]. Its text also gives beta1
  twice, and the second value is beta2 [arxiv:2301.12597 §3.4].
- MiniGPT-4's stage 2 is 400 steps in the paper and 1,000 steps in the released
  config [arxiv:2304.10592 §3.3] [/papers/2304.10592].
- LAION-5B's text sets the image cut-off at 5 KB, while its code counts bytes
  [arxiv:2210.08402 §3.1] [/papers/2210.08402].
- MaPLe's §4.1 and the code give a base-to-novel learning rate of 0.0035, while
  appendix A gives 0.0025 [arxiv:2210.03117 §4.1] [arxiv:2210.03117 app. A]
  [/papers/2210.03117]. Its §4.3 says MaPLe beats CLIP on novel accuracy on 6 of
  11 datasets, but table 3 shows 7 of 11 [arxiv:2210.03117 §4.3]
  [arxiv:2210.03117 tab. 3].
- unCLIP's text gives 10% for the CLIP-embedding dropout in decoder training,
  while §5.1 and the fig. 8 caption give 5% [arxiv:2204.06125 §2.1]
  [arxiv:2204.06125 §5.1].
- CC12M's table caption and text disagree on which model is best on nocaps
  [arxiv:2102.08981 tab. 4] [arxiv:2102.08981 §4.1].
- **ClipCap's model size.** The paper says GPT-2 large [arxiv:2111.09734 §3.1],
  but the released code loads the smaller `gpt2` model [/papers/2111.09734]. The
  transformer prefix length is 10 in the paper and 40 in the README
  [arxiv:2111.09734 §4] [/papers/2111.09734].
- **GroupViT's gains.** Section 4.2 pairs the 13.1-point gain with hard
  assignment and the 2.6-point gain with soft assignment. Table 1 shows the
  reverse [arxiv:2202.11094 §4.2] [arxiv:2202.11094 tab. 1].
- **OVSeg's PASCAL Context value.** Section 4.2 gives 55.5 for Swin-B on
  PC-59, while Table 1 gives 55.7, which matches the stated 13.6-point gap over
  OpenSeg. Use 55.7 from the table [arxiv:2210.04150 §4.2]
  [arxiv:2210.04150 tab. 1].
- **DataComp's loss.** The footnote in §3.4 gives the contrastive loss without
  its logarithm and minus sign. Implement the standard CLIP form
  [arxiv:2304.14108 §3.4].
- **RemoteCLIP's "best on all three".** The text says RemoteCLIP is best on all
  three retrieval benchmarks, but Table 2 gives FBCLM 61.60 on UCM
  [arxiv:2306.11029 §4.2.1] [arxiv:2306.11029 tab. 2].
- **Winoground's chance level.** The abstract says no model does much better
  than chance. §5.1 says six models exceed chance on the text score
  [arxiv:2204.03162 §5.1]. The card's Table 3 marks nine models above the 25.00
  text chance level: VinVL, UNITER large and base, ViLLA large and base, ViLT,
  FLAVA ITM, FLAVA contrastive and CLIP [arxiv:2204.03162 tab. 3]. Both counts
  are recorded here, and the page does not choose between them
  [/papers/2204.03162].
- **GenEval's position rule.** The appendix prints the "above" condition with a
  sign that overlaps the "below" condition and contradicts the rest of the rule
  [arxiv:2310.11513 app. C.3].
- **Dream Fields' transmittance schedule.** The paper uses 40% to 88%, and the
  code uses 50% to 90% [arxiv:2112.01455 §4.5] [/papers/2112.01455].
- **SLIP's YFCC15M linear values.** Table 5 prints the CLIP and SLIP linear
  values for YFCC15M the wrong way round, against tables 2, 6 and 7
  [arxiv:2112.12750 tab. 5] [arxiv:2112.12750 tab. 2]. The ViT-L zero-shot gain
  is printed as +4.8, and the text repeats a range that ends at +4.8, but 46.2 − 40.4 = 5.8
  [arxiv:2112.12750 tab. 2] [arxiv:2112.12750 §5.1]. Use the raw numbers.
- **InternVideo's Kinetics-710 size and benchmark count.** The text gives 650K
  videos for Kinetics-710, and Table 1 gives 680k [arxiv:2212.03191 §4.1]
  [arxiv:2212.03191 tab. 1]. The paper gives 39 datasets in §1 and 34 benchmarks
  in §3 [arxiv:2212.03191 §1] [arxiv:2212.03191 §3].
- **InternVideo's masked pre-training schedule, paper against code.** The paper
  states 1200 epochs at base learning rate 2.5e-4, and the released script runs
  301 epochs at 1e-3 [arxiv:2212.03191 §4.2] [/papers/2212.03191].
- **X-CLIP's similarity scale, paper against code.** The paper scores videos by
  plain cosine similarity, and the code multiplies the cosine by the exponential
  of a learned logit scale [arxiv:2208.02816 eq. 3] [/papers/2208.02816].
- **ProGrad's base-to-new context length, paper against code.** The paper uses 4
  context tokens for base-to-new, and the released scripts use 16
  [arxiv:2205.14865 §4] [/papers/2205.14865].
- **MotionCLIP's loss weights, paper against code.** The paper sets both
  text and image weights to 0.01 [arxiv:2203.08063 §4.1], while the code default
  is 1.0 and the README run keeps it [/papers/2203.08063].
- **StyleCLIP's standard deviation.** The "Surprised" entry in tab. 2 is printed
  as 0.893, while the other prompts are between 0.081 and 0.145
  [arxiv:2103.17249 tab. 2]. The paper does not say whether this is a typo.
- **FLAVA's table columns.** The card records four rows of the macro-average
  table with columns out of order [arxiv:2112.04482 tab. 3] [/papers/2112.04482].
- **InstructBLIP's ablation text against its table.** §3.1 says removing the
  instruction-aware features hurts most on ScienceQA and iVQA [arxiv:2305.06500
  §3.1]. Table 2 shows the largest drop on VizWiz (32.7 to 25.1), then on
  ScienceQA (70.4 to 63.4), with iVQA 53.1 to 47.5 [arxiv:2305.06500 tab. 2].
- **OTTER's action horizon.** The architecture table gives 12 for real robots
  and 1 for simulation, while §3.2 says 8 steps gave the best results
  [arxiv:2503.03734 §3.2] [/papers/2503.03734]. The code default is 12.
- **OTTER's trial counts.** The simulation table caption gives 50 trials per
  unseen task, and the simulation ablation table gives 100 trials on the same
  unseen tasks, with 61% in both [/papers/2503.03734].
- **OTTER's encoder size.** The text and the architecture table give ViT-L/14,
  and the scaling figure caption gives ViT-L/16 [arxiv:2503.03734 §5.4]
  [/papers/2503.03734].
- **OTTER's loss, paper against code.** The paper names no loss
  [arxiv:2503.03734 §3.2]. The code's forward pass uses L1 unless a flag is set,
  and its constructor reads a config field that the config class does not
  define. The card reads this as a construction error, which it did not run
  [/papers/2503.03734].
- **OTTER's 50 Hz claim, paper against code.** The paper says CLIP features are
  cached at inference, which allows 50 Hz on one GPU [arxiv:2503.03734 app. B.1].
  The code runs both CLIP encoders on every call [/papers/2503.03734].
- **OTTER's gradient clipping, paper against code.** Appendix B.2 lists a clip
  threshold of 1, but the training loop applies no clipping
  [arxiv:2503.03734 app. B.2] [/papers/2503.03734].
- **OTTER's mean column.** In the multi-primitive table the mean is not the
  average of the four task columns. For OTTER-OXE-L the columns average 80% and
  the mean is 77% [arxiv:2503.03734 §5.1].
- **CLIP-RLDrive's reward rule.** Eq. 14 always adds the CLIP term, while
  Algorithm 1 adds it only when the agent's action matches CLIP's suggestion
  [arxiv:2412.16201 eq. 14] [arxiv:2412.16201 alg. 1]. The released code uses a
  third rule with fixed penalties and bonuses [/papers/2412.16201].
- **CLIP-RLDrive's PPO claim.** The text says CLIP-PPO beats plain PPO on
  training reward [arxiv:2412.16201 §3.1]. The test table shows lower success
  for CLIP-PPO at every vehicle count, 38 against 78 with one vehicle
  [arxiv:2412.16201 tab. 2].
- **CLIP-RLDrive's paper against code, discount.** The paper gives DQN gamma
  0.95 and PPO gamma 0.99 [arxiv:2412.16201 tab. 1]. The code's DQN uses 0.99,
  and its commented-out PPO uses 0.95 [/papers/2412.16201].

## Papers

- [Conceptual 12M](/papers/2102.08981): relaxes the CC3M filters to build a 12.4M-pair web dataset for captioning pre-training.
- [Align before Fuse](/papers/2107.07651): aligns image and text features before fusion, with momentum distillation as a teacher.
- [An Empirical Study of Training End-to-End Vision-and-Language Transformers](/papers/2111.02387): METER, a fully transformer model with co-attention, trained end to end without an object detector.
- [Zero-Shot Text-Guided Object Generation with Dream Fields](/papers/2112.01455): optimises a NeRF against a frozen CLIP caption score, for text-to-3D without 3D training data.
- [More Control for Free! Image Synthesis with Semantic Diffusion Guidance](/papers/2112.05744): steers an unconditional DDPM at sampling time with CLIP-based language, image and style guidance.
- [StyleGAN-NADA](/papers/2108.00946): shifts a pretrained StyleGAN2 to a new domain from two text prompts, using a directional CLIP loss.
- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](/papers/2110.02624): generates 3D shapes from text using a conditional flow trained on frozen CLIP image features.
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](/papers/2111.03930): builds a training-free key-value cache from few-shot CLIP features, and can fine-tune the keys.
- [ClipCap: CLIP Prefix for Image Captioning](/papers/2111.09734): maps a frozen CLIP embedding to GPT-2 prefix vectors with a small mapper, for captioning.
- [Align and Prompt](/papers/2112.09583): pre-trains a video-text encoder with a contrastive term and an entity pseudo-label term, for retrieval and video QA.
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](/papers/2112.12143): learns mask proposals and aligns caption words to them with a grounding loss, so it can segment open vocabularies.
- [BLIP](/papers/2201.12086): one encoder-decoder model for matching and captioning, trained on bootstrapped web captions.
- [OFA](/papers/2202.03052): one sequence-to-sequence model for vision, language and cross-modal tasks, with a shared vocabulary.
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](/papers/2202.11094): groups image patches into segments with caption contrast alone, and labels them zero-shot.
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](/papers/2204.03162): a 400-example benchmark for matching captions to images through word order.
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](/papers/2204.06125): unCLIP, a prior from captions to CLIP image embeddings, followed by a diffusion decoder.
- [VQGAN-CLIP](/papers/2204.08583): generates and edits images by optimising a VQGAN latent against a frozen CLIP, with no training.
- [ELEVATER](/papers/2204.08790): a benchmark for transfer of language-augmented models, with a language-initialised linear head.
- [MineDojo](/papers/2206.08853): trains MineCLIP on Minecraft video and transcripts, and uses it as a dense reward for RL agents.
- [Patching open-vocabulary models by interpolating weights](/papers/2208.05592): patches a CLIP model by blending fine-tuned and zero-shot weights.
- [PaLI](/papers/2209.06794): a multilingual image-and-text-to-text model with a scaled ViT and mT5, trained on many tasks.
- [MaPLe](/papers/2210.03117): learns coupled prompts in both CLIP encoders for few-shot and base-to-novel classification.
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](/papers/2210.04150): fine-tunes CLIP on masked regions named by caption nouns, with mask prompt tuning, for open-vocabulary segmentation.
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](/papers/2302.12242): keeps CLIP frozen and adds a light side ViT that proposes masks and steers CLIP's last blocks to name them.
- [LAION-5B](/papers/2210.08402): a 5.85-billion-pair open image-text dataset filtered by CLIP similarity, with NSFW and watermark tags.
- [Reproducible scaling laws for contrastive language-image learning](/papers/2212.07143): fits power laws for OpenCLIP models trained on LAION data, with predictions flagged as extrapolations.
- [CREPE](/papers/2212.07796): a benchmark for compositional reasoning in vision-language models, with systematicity and productivity tests.
- [BLIP-2](/papers/2301.12597): bridges a frozen image encoder and a frozen LLM with a trained Q-Former, in two stages.
- [Pic2Word](/papers/2302.03084): maps an image to a word token for zero-shot composed image retrieval.
- [InfoSeek](/papers/2302.11713): a visual question answering benchmark that needs outside knowledge about the pictured entity.
- [Language-Driven Representation Learning for Robotics](/papers/2302.12766): pre-trains a robot visual encoder by reconstructing masked video frames, with the caption as a condition or a target.
- [Sigmoid Loss for Language Image Pre-Training](/papers/2303.15343): replaces the batch softmax with a per-pair sigmoid loss and reports SigLiT and SigLIP results.
- [CLIP-LIT](/papers/2303.17569): enhances backlit photos with learned CLIP prompts and an illumination-map network, trained on unpaired images.
- [MiniGPT-4](/papers/2304.10592): links a frozen BLIP-2 vision stack to a frozen Vicuna through one trained projection, with a curated second stage.
- [DataComp: In search of the next generation of multimodal datasets](/papers/2304.14108): a benchmark for choosing image-text training data under a fixed recipe, with a 12.8B-pair pool.
- [InstructBLIP](/papers/2305.06500): instruction-tunes a frozen image encoder and LLM through a trained Q-Former.
- [ULIP-2](/papers/2305.08275): aligns 3D point clouds with generated captions and frozen OpenCLIP features.
- [BLIP-Diffusion](/papers/2305.14720): turns a subject photo into prompt tokens with a BLIP-2 Q-Former, for subject-driven generation.
- [RemoteCLIP: A Vision Language Foundation Model for Remote Sensing](/papers/2306.11029): continues CLIP training on remote-sensing pairs built from detection and segmentation labels.
- [EgoSchema](/papers/2308.09126): a multiple-choice benchmark for long-form video question answering, for evaluation rather than training.
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](/papers/2310.11513): scores text-to-image outputs per object, count, colour and position with a detector and a CLIP colour check.
- [SLIP: Self-supervision meets Language-Image Pre-training](/papers/2112.12750): adds a SimCLR loss to CLIP's contrastive loss on one shared Vision Transformer.
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](/papers/2203.08063): aligns a motion auto-encoder's latent with frozen CLIP text and image embeddings, for text-to-motion.
- [Prompt-aligned Gradient for Prompt Tuning](/papers/2205.14865): ProGrad, which drops the parts of a prompt-tuning gradient that conflict with zero-shot CLIP predictions.
- [Expanding Language-Image Pretrained Models for General Video Recognition](/papers/2208.02816): X-CLIP, which adapts CLIP to video with cross-frame message tokens and video-specific text prompts.
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](/papers/2212.03191): trains a masked video encoder and a video-text encoder separately, then fuses them with cross-model attention.
- [Transformers in Vision: A Survey](/papers/2101.01169): surveys Transformer designs for vision by attention type and task, with tables compiled from other papers.
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](/papers/2103.17249): edits StyleGAN2 images from text prompts through CLIP, with an optimiser, a latent mapper and a global direction.
- [Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis](/papers/2104.00677): adds a frozen-CLIP semantic loss on unseen NeRF views, for reconstruction from few photos.
- [FLAVA: A Foundational Language And Vision Alignment Model](/papers/2112.04482): one transformer family for image, text and joint tasks, pre-trained on public image-text pairs and unpaired data.
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](/papers/2304.14178): connects a CLIP encoder to LLaMA-7B through a visual abstractor, trained in two stages.
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](/papers/2401.06209): builds MMVP, a benchmark of image pairs that CLIP confuses, and tests an interleaved CLIP and DINOv2 input.
- [InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models](/papers/2411.11394): labels house-tour video with CLIP and writes navigation instructions with a large multimodal model, with a consistency check.
- [CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning](/papers/2412.16201): adds a CLIP prompt score to the reward of DQN and PPO agents at an unsignalized intersection.
- [OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction](/papers/2503.03734): keeps CLIP frozen and pools its patch features by similarity to each instruction word, for a robot action policy.
- [DocFormer: End-to-End Transformer for Document Understanding](/papers/2106.11539): fuses OCR text, word boxes and a page image in one transformer, pre-trained on IIT-CDIP pages.
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](/papers/2112.03221): stylises a fixed mesh's colour and normal offset with a neural field trained on CLIP similarity.
- [Text2LIVE: Text-Driven Layered Image and Video Editing](/papers/2204.02491): trains a per-input generator that outputs a colour-plus-opacity layer, guided by CLIP losses.
- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](/papers/2212.05171): aligns a trainable 3D point-cloud encoder with frozen SLIP image and text embeddings.
- [Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective](/papers/2303.14968): fine-tunes CLIP to score quality, scene and distortion from 495 joint text labels.
- [Self-regulating Prompts: Foundational Model Adaptation without Forgetting](/papers/2307.06948): trains CLIP prompts with feature and logit agreement terms against the frozen model, and epoch-level prompt averaging.
- [CLIPstyler: Image Style Transfer with a Single Text Condition](/papers/2112.00374): trains a small U-Net per image against directional and patch-wise CLIP losses, for style transfer from text alone.
- [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](/papers/2112.14757): pairs a MaskFormer trained on seen classes with a frozen CLIP that labels each masked crop.
- [Exploring Visual Prompts for Adapting Large-Scale Models](/papers/2203.17274): learns one additive input prompt for a frozen model, trained with cross-entropy.
- [Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts](/papers/2206.02770): a sparse mixture-of-experts transformer on images and text, with a contrastive loss and routing losses.
- [Towards artificial general intelligence via a multimodal foundation model](/papers/2110.14378): BriVL, a two-tower image-text model with momentum queues, trained on 650M weakly correlated Chinese web pairs.
- [CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts](/papers/2311.16445): trains a residual MLP on frozen CLIP text features so that style words are ignored, and applies it to image and text features.