Vision-language pre-training

Overview

A vision-language model learns from pairs of images and text, such as photos and their captions, so that it can later relate new images to new text 2303.15343 §1. The papers here use such a model in several ways. Some pre-train two encoders that share one embedding space with a contrastive loss 2303.15343 §3.1 2107.07651 §3.2. One card adds a self-supervised image loss to that contrastive objective 2112.12750 §3.3. Some pre-train one encoder-decoder that reads an image and writes text, for many tasks at once 2202.03052 §3.1 2209.06794 §3.1. Some keep a large image encoder and a large language model frozen, and train only a small bridge between them 2301.12597 §3.1 2304.10592 §3.1. Some use a frozen image-text model to steer, score or reward a generator or agent that already exists 2112.12143 §3 2305.06500 §2.3 2204.08583 §2 2206.08853 §4.2. Some adapt a frozen CLIP with prompts, a gradient constraint or weight blends 2210.03117 §3.2 2205.14865 §3.2 2208.05592 §2. The same idea reaches beyond still images. Some cards extend CLIP-style models to video 2208.02816 §3.2 2212.03191 §3.1, one maps human motion into CLIP's space 2203.08063 §3, and some pre-train robot or 3D encoders against captions, while another uses a frozen model to map an image into a word token for a text query. One robot policy keeps CLIP frozen and selects the image patches that match each instruction word 2503.03734 §3.1.

Five more cards carry the same parts into other settings. One edits StyleGAN images from a text prompt, steering them with CLIP 2103.17249 §4. One adds a CLIP-based semantic loss to few-photo NeRF training [arxiv:2104.00677 §4.1]. One trains a single transformer family on image, text and joint data 2112.04482 §3.1. One links a CLIP encoder to LLaMA-7B through a learnable abstractor 2304.14178 §3.1. One builds a benchmark that shows where CLIP-based multimodal LLMs miss fine visual detail 2401.06209 §1.

Several cards are about the data and how it scales. Two describe how web image-text pairs are collected and filtered 2102.08981 §2.2 2210.08402 §3.1, and one fits how zero-shot accuracy grows with model size, data and samples seen 2212.07143 §4.1. Six cards are benchmarks rather than methods, namely EgoSchema for video question answering 2308.09126 §3.1, CREPE for compositional retrieval 2212.07796 §3 and InfoSeek for knowledge-based visual questions 2302.11713 §3, Winoground for word-order matching 2204.03162 §3.1, GenEval for text-to-image alignment 2310.11513 §3.2 and DataComp for filtering choices 2304.14108 §3.1. One more card changes the data itself: RemoteCLIP turns aerial annotations into captions 2306.11029 §3.2.

The methods differ in how the two sides meet. Contrastive dual encoders keep them apart until the loss, and the align-then-fuse designs 2107.07651 §3.1 add a fusion encoder after the contrastive step. Region methods keep the image side at region level, so each caption word is matched to a region instead of the whole image 2112.12143 §3.2. Bridge methods train only a small query module or projection in front of a frozen LLM 2305.06500 §2.2 2304.10592 §3.1. Frozen-scorer methods train nothing new in the generator or agent, and use a frozen image-text model as a signal 2204.08583 §2 2112.05744 §3.1.

The cards differ in what they train. SigLIP is a pre-training recipe for both towers. ALBEF aligns the towers before fusing them, and uses a slowly updated copy as a teacher. BLIP is one model that both matches and captions, trained on bootstrapped captions. OFA and PaLI are generative models with a single output format. OpenSeg trains a segmentation model from class-agnostic masks and captions. BLIP-2 trains a Q-Former between frozen parts, and MiniGPT-4 trains one projection layer in its first stage. InstructBLIP is an instruction-tuning stage on top of an existing model. Its pre-training stage comes from BLIP-2 2301.12597 §3.1, so this page covers only what InstructBLIP reports about its own training. VQGAN-CLIP does not pre-train anything; its card lists this concept because it relies on CLIP's image-text alignment. MineCLIP pre-trains a video-text encoder on Minecraft footage. BLIP-Diffusion trains a subject encoder and leaves the diffusion model's own pre-training alone. CLIP-Forge trains a flow on top of frozen CLIP features, unCLIP trains a prior from captions to CLIP image embeddings, and Tip-Adapter is training-free in its main form, and its fine-tuned form, Tip-Adapter-F, learns only its keys. MaPLe trains prompts in both frozen CLIP encoders.

The later cards differ in the same way. ALPRO trains a video-text encoder and adds an objective that predicts entity labels for random crops 2112.09583 §3.2. Voltron pre-trains a robot visual encoder by masking video frames while either reading or writing the caption 2302.12766 §3. ELEVATER is a benchmark that also proposes starting a new linear head from the text encoder 2204.08790 §4.2. PAINT fine-tunes once and then blends the fine-tuned and zero-shot weights 2208.05592 §2. Pic2Word trains only a map from an image to one word token 2302.03084 §3.2. CLIP-LIT learns a pair of text prompts and an illumination-map enhancer 2303.17569 §3. ULIP-2 trains a 3D encoder against frozen OpenCLIP features 2305.08275 §3.3. CREPE and InfoSeek only evaluate.

The newest cards extend the same ideas beyond whole-image matching. ClipCap maps a frozen CLIP embedding into GPT-2 inputs 2111.09734 §3.1. GroupViT and OVSeg turn image-text supervision into region-level labels 2202.11094 §3.2 2210.04150 §3.2, and Dream Fields uses a frozen CLIP score to shape a 3D field 2112.01455 §4.2. CLIP-RLDrive adds a CLIP-based term to a driving reward 2412.16201 §2.4, and InstruGen uses CLIP only to label rooms in house-tour video before a large multimodal model writes instructions 2411.11394 §3.1. RemoteCLIP adapts CLIP to aerial images 2306.11029 §3.1, and GenEval scores generated images with a detector and a CLIP colour check 2310.11513 §3.2. One card is a survey rather than a method. It notes that ViTs need large pre-training sets to beat CNNs 2101.01169 §2.2 2101.01169 §3.2.1.

Four more cards take the same parts in new directions. CLIPstyler uses a frozen CLIP as the only judge of a style-transfer network trained for each image 2112.00374 §3.2. The zero-shot segmentation baseline keeps CLIP frozen and uses it to name masked regions 2112.14757 §4.2.1. Visual prompting leaves a whole model frozen and learns only one additive input frame 2203.17274 §3.2.1. LIMoE trains one shared sparse-expert transformer on images and captions together 2206.02770 §2.2.

Variants

The variants below are grouped into ten families. The table gives the family, what its members train, and when it is the right starting point. Each variant has its own section further down, with its numbers and caveats.

Relaxed web filters for captioning data (CC12M)

Card: Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts.

CC12M follows the CC3M pipeline but relaxes its unimodal filters. It has 12,423,374 pairs, against 3,318,333 in CC3M 2102.08981 tab. 1. The captions are less precise: the authors estimate 76.6% precision against 90.3% for CC3M, from a 100-caption check by two annotators, so the gap is an estimate 2102.08981 §2.3. The benefit shows after fine-tuning. A captioner pre-trained on CC12M and then fine-tuned on COCO reaches 87.4 overall CIDEr on nocaps validation, against 54.7 with no pre-training. Without fine-tuning, CC12M is below CC3M, at 27.1 against 29.7 2102.08981 tab. 3. The release is a list of image URLs and captions, not model code, and image URLs disappear over time [/papers/2102.08981]. CC12M is part of the 14M-image setting of ALBEF and BLIP 2107.07651 tab. 8 2201.12086 §4.1.

Open web-scale pairs filtered by CLIP (LAION-5B)

Card: LAION-5B: An open large-scale dataset for training next generation image-text models.

LAION-5B releases 5.85 billion image-text pairs taken from web alt-text, of which 2.32 billion are English 2210.08402 §4. The main quality filter is a cosine similarity from OpenAI's CLIP ViT-B/32 of at least 0.28 for English and 0.26 for other languages, which removed about 90% of roughly 50 billion candidates 2210.08402 §3.1. Potentially offensive pairs are tagged, not removed, so users can choose 2210.08402 §3.2. Models trained on the English subset roughly match the original CLIP. A ViT-L/14 trained on LAION-2B-en reaches 75.2 zero-shot ImageNet top-1, against 75.6 for the OpenAI model 2210.08402 tab. 1. The metadata are CC BY 4.0, but the authors say they do not own the images or captions 2210.08402 app. A. Check each source's rights before any commercial use. The released worker code has filter rules that the paper does not describe [/papers/2210.08402].

Scaling laws for open CLIP (OpenCLIP on LAION)

Card: Reproducible scaling laws for contrastive language-image learning.

The authors train OpenCLIP models from ViT-B/32 to ViT-g/14 on LAION subsets of 80M, 400M and 2B pairs, for 3B to 34B samples seen, and fit power laws to error 2212.07143 §3.2 2212.07143 §4.1. The best measured zero-shot ImageNet top-1 is 78.0, for ViT-H/14 on LAION-2B after 34B samples 2212.07143 tab. 1. The fitted curve predicts 81.92 for ViT-G/14 at 68B samples. That is an extrapolation, not a measurement 2212.07143 tab. 4. The exponents differ by data source. For ImageNet zero-shot the OpenCLIP fit is −0.11 against −0.16 for OpenAI CLIP, while for Flickr30K retrieval R@5 it is −0.19 against −0.10 2212.07143 §4.1 2212.07143 §4.2. The full study used 1,058,318 GPU-hours 2212.07143 tab. 18. Forecasts from a fitted line are extrapolations, and the fit depends on which points are binned (see Contradictions).

Data choice under fixed compute (DataComp)

Card: DataComp: In search of the next generation of multimodal datasets.

DataComp fixes the training recipe at each of four scales. Participants either pick a subset of one candidate pool or bring their own data 2304.14108 §3.1 2304.14108 §3.3. The pool is an index of 12.8B image URLs and alt-text pairs, with CLIP scores and features published per sample 2304.14108 §3.2. Models are scored zero-shot on 38 tasks 2304.14108 §3.5.

The best baseline, DataComp-1B, keeps pairs that pass an image-cluster filter and a CLIP ViT-L/14 score in the top 30% 2304.14108 §5.1. Author-reported, it reaches 79.2% ImageNet zero-shot, against 75.5% for OpenAI's CLIP at the same training compute 2304.14108 tab. 1. At the xlarge scale the same combination gives 0.792 ImageNet and 0.663 average over 38 tasks, against 0.723 and 0.621 with no filtering 2304.14108 tab. 3. Each result is a single seed with no error bars, so gaps of about a point are not established 2304.14108 app. Checklist.

Align before fuse with momentum distillation (ALBEF)

Card: Align before Fuse: Vision and Language Representation Learning with Momentum Distillation.

ALBEF has three parts: a 12-layer ViT-B/16 image encoder, a text encoder made of the first six BERT layers, and a fusion encoder made of the last six with cross-attention to the image 2107.07651 §3.1. The image-text contrastive loss is applied to the two unimodal encoders before fusion 2107.07651 §3.2 2107.07651 eq. 2. A momentum copy of each encoder, updated as a moving average with momentum 0.995, gives soft targets. The targets mix the copy's distribution with the one-hot label, with a weight alpha that ramps to 0.4 2107.07651 §3.3 2107.07651 eq. 6 2107.07651 §3.5. The paper argues this helps with noisy web captions 2107.07651 §1. Results are author-reported. The 14M-image model reaches 94.1 zero-shot Flickr30K text retrieval R@1, against 88.6 for ALIGN, which uses 1.2B images 2107.07651 tab. 3. Its VQA test-std is 76.04 and its NLVR² test-P is 83.14 2107.07651 tab. 4. On 4M images, adding momentum distillation to both the contrastive and the masked-language losses raises VQA test-dev from 73.81 to 74.42 2107.07651 tab. 1. The model needs no object detector or boxes 2107.07651 §1.

Fully transformer model trained end to end (METER)

Card: An Empirical Study of Training End-to-End Vision-and-Language Transformers.

METER pairs a text transformer, RoBERTa by default, with an image transformer, CLIP-ViT by default. Six co-attention layers let each stream attend to the other 2111.02387 §3.1. It has no object detector, and it is pre-trained with masked language modeling and image-text matching only 2111.02387 §3.3. In the authors' ablation, adding masked image modeling lowered VQAv2 accuracy 2111.02387 tab. 7. Results are author-reported. The CLIP-ViT base model reaches 77.64 on VQAv2 test-std with about 4M pre-training images, against 76.60 for VinVL, which uses region features 2111.02387 tab. 8. A 14M-image version with a larger backbone reaches 80.54 2111.02387 tab. 10; the released repository has no config for that run (see the card's Paper vs code).

Bootstrapped captions with a filter (BLIP)

Card: BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.

BLIP uses one encoder-decoder transformer, the MED, that runs in three modes: unimodal encoding, image-grounded matching and caption generation 2201.12086 §3.1. It is trained with an image-text contrastive loss, an image-text matching loss and a language-modelling loss 2201.12086 §3.2. Its data comes from CapFilt. A captioner writes one synthetic caption per web image, and a filter removes any text, original or synthetic, whose matching head says it does not match 2201.12086 §3.3. Decoding uses nucleus sampling with p = 0.9 2201.12086 §4.3. In the authors' ablation, with COCO fine-tuning, removing both captioner and filter gives 127.8 CIDEr and using both gives 129.7 2201.12086 tab. 1. With 14M images, COCO 5K text retrieval R@1 is 80.6, against 77.6 for ALBEF 2201.12086 tab. 5. With 129M images, NoCaps CIDEr reaches 106.3, and 109.6 with the CapFilt-L variant 2201.12086 tab. 7. The released code has no captioner or filter training step, only the bootstrapped data, so CapFilt cannot be rerun from the repository alone [/papers/2201.12086].

One sequence interface for all tasks (OFA)

Card: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework.

OFA casts every task, pre-training included, as generation from an instruction and its inputs, using one Transformer encoder-decoder 2202.03052 §3.1 2202.03052 §3.2. Words, discrete image codes and box coordinates share one vocabulary. A 256×256 image becomes a 16×16 sequence of codes, and each box corner is discretised into a location token 2202.03052 §3.1. The released scripts use 1000 bins [/papers/2202.03052]. The five model sizes run from 33M to 930M parameters 2202.03052 tab. 1. Reported results are author-reported: VQA test-std 82.0 against 80.4 for the best listed prior 2202.03052 tab. 2, SNLI-VE test 91.2 against 86.3 2202.03052 tab. 2, and COCO captioning with CIDEr optimisation 154.9 against 145.5 for LEMON 2202.03052 tab. 3. The model is not best everywhere. On unimodal tasks it is 96.6 on SST-2 against 97.0 for XLNet 2202.03052 tab. 6, and its zero-shot MNLI score is 37.3 against 49.6 for Uni-Perceiver, so instruction wording matters 2202.03052 tab. 9 2202.03052 §4.3.

Multilingual generator with a scaled ViT (PaLI)

Card: PaLI: A Jointly-Scaled Multilingual Language-Image Model.

PaLI reads an image and a text prompt and writes text, so captioning, question answering and text reading share one interface 2209.06794 §3.1. A ViT encodes the image, and an mT5 encoder-decoder, initialised from a pre-trained checkpoint, produces the text 2209.06794 §3.1. The pre-training mix has eight tasks, including span corruption, split captioning over alt-text, and multilingual captioning and VQA 2209.06794 §3.2 2209.06794 app. A.2. The largest model, PaLI-17B, uses a 4B-parameter ViT-e 2209.06794 app. A.1. Author-reported results: COCO Karpathy test CIDEr 149.1 against 147.6 for BEiT-3 2209.06794 tab. 1, VQAv2 test-dev 84.3 against 82.0 for Flamingo under open-vocabulary generation 2209.06794 tab. 3, and a 35-language average of 53.6 CIDEr on Crossmodal-3600 2209.06794 tab. 2. Scaling the ViT from ViT-G to ViT-e adds 3.2 points on average over seven benchmarks, and scaling the language model from mT5-Large to XXL adds 3.1 2209.06794 §4.5. The 17B run used 1,024 TPUv4 chips for 7 days 2209.06794 app. A.1. The pre-training data, WebLI, is not released 2209.06794 app. B.

Text-to-image through a CLIP image embedding (unCLIP)

Card: Hierarchical Text-Conditional Image Generation with CLIP Latents.

The model has two stages. A prior maps a caption to a CLIP image embedding, and a diffusion decoder inverts the CLIP image encoder to render an image 2204.06125 §2. The diffusion prior is a causal decoder-only transformer that predicts the clean embedding under a mean-squared error. Two samples are drawn, and the one with the higher dot product to the text embedding is kept 2204.06125 §2.2. The decoder is a 3.5B-parameter diffusion model, and its CLIP embedding enters both the timestep embedding and four context tokens 2204.06125 §2.1 2204.06125 tab. 3. Zero-shot MS-COCO FID is 10.39 with the diffusion prior, against 12.24 for GLIDE 2204.06125 tab. 2. Human raters preferred unCLIP to GLIDE for diversity at 70.5%, against 48.9% for photorealism and 45.3% for caption similarity 2204.06125 tab. 1. Because the image is encoded as a CLIP embedding, the same model can make variations and text-direction edits 2204.06125 §3. No official code is known [/papers/2204.06125].

Text to 3D through a frozen CLIP score (Dream Fields)

Card: Zero-Shot Text-Guided Object Generation with Dream Fields.

A NeRF starts from random weights and is optimised so that its renders, taken from random azimuths, score high against the caption under a frozen CLIP image-text model. No 3D data is used 2112.01455 §4.2 2112.01455 eq. 7. Three priors keep the result a single object: a transmittance penalty that rewards see-through density, random backgrounds behind each training render, and a fixed bounding cube 2112.01455 §4.5 2112.01455 §4.6.

Author-reported, on object-centric COCO captions. R-Precision reaches 68.3 at 20K iterations and 252-pixel renders, against 31.37 for a simplified NeRF at 10K iterations 2112.01455 tab. 1. Removing the sparsity term lowers R-Precision from 62.09 to 35.29 2112.01455 tab. 2. Each object is an optimisation of about 1 h 12 min for 10K iterations on eight TPU cores, at an author-estimated $3 to $4 per object 2112.01455 app. C. The paper anneals the transmittance target from 40% to 88%, but the code uses 50% to 90% 2112.01455 §4.5 [/papers/2112.01455].

Pairwise sigmoid loss (SigLIP and SigLiT)

Card: Sigmoid Loss for Language Image Pre-Training.

The usual CLIP-style loss normalises each image's similarities over the whole batch with a softmax, in both directions 2303.15343 §3.1. The sigmoid variant instead treats each of the |B|² image-text pairs as its own yes-or-no problem: the true pairs on the diagonal are positives, all others are negatives 2303.15343 §3.2. Each logit is a scaled dot product of normalised embeddings plus a learned bias. The temperature is learned as an exponential of a scalar and starts at 10, and the bias starts at −10 2303.15343 §3.2 2303.15343 §4.9.

Because no pair depends on the others, the sum can be computed in blocks, one per device, while text embeddings are passed around the devices. Only one block is held in memory at a time 2303.15343 §3.3. The card records that the released trainer does not implement this chunked form [/papers/2303.15343].

Variants reported in the paper:

• SigLiT. A frozen, pre-trained image tower with a text tower trained from scratch 2303.15343 §4.1. The best result is a g/14 image tower on four chips for about two days, 84.5% ImageNet zero-shot 2303.15343 tab. 1. The cost of training that frozen tower is not counted 2303.15343 §4.4.
• SigLIP from scratch. Both towers trained from scratch on English pairs. B/16 reaches 73.4% after five days on 32 chips 2303.15343 tab. 1 2303.15343 §4.2.
• SigLIP with a pre-trained image tower. The image tower is unlocked and started from a pre-trained checkpoint. It gets a 0.1 learning-rate multiplier and no weight decay, while the text tower keeps weight decay 2303.15343 §4.5. The B/16 run on 16 chips reaches 71.0% 2303.15343 tab. 1.
• mSigLIP. A multilingual variant with a 250k vocabulary. Its token embedding is bottlenecked to a small width K before projection. The authors report about a 0.5% ImageNet drop compared with the full embedding 2303.15343 §4.3.

Two towers with momentum queues on weakly correlated web pairs (BriVL)

Card: Towards artificial general intelligence via a multimodal foundation model.

BriVL has an image encoder and a text encoder, trained with a symmetric InfoNCE loss. Negatives come from two momentum-updated queues of recent keys, so the batch does not have to hold every negative [arxiv:2110.14378 §Methods: Contrastive Loss]. The image side is a CNN with multi-scale patch pooling and a self-attention block, and it takes no object detector output [arxiv:2110.14378 §Methods: Image Encoder]. The authors pre-train on about 650M Chinese image-text pairs that are only weakly related, and they argue this pushes the model towards more abstract matching 2110.14378 §Introduction. Their author-reported zero-shot results beat CLIP on remote-sensing scene classification and Chinese news classification. For example, on UCM at a 5/16 unseen/seen split they report 82.41% against 76.02% for CLIP ResNet-50x4 2110.14378 fig. 4a 2110.14378 fig. 5a. The main run uses a batch of 2,688 with queues of 13,440 per side, a fixed temperature of 0.07, and about 10 days on 112 A100 GPUs [arxiv:2110.14378 §Methods: Implementation Details]. The paper does not say whether embeddings are normalised before the dot product, and the card flags this as a choice an implementer must make [/papers/2110.14378]. The paper states that its released model and inference code are CC BY-NC-ND 4.0, which excludes commercial use and derivatives 2110.14378 §Code availability. Treat BriVL as a research reference, not a model to ship.

Region-word grounding from captions (OpenSeg)

Card: Scaling Open-Vocabulary Image Segmentation with Image-Level Labels.

The image is represented as a fixed set of N = 128 mask proposals, each with its own feature vector, instead of one vector or a dense per-pixel map 2112.12143 §3. The proposals are learned from class-agnostic COCO masks with a Dice loss, and carry no class names 2112.12143 §3.1 2112.12143 eq. 1. Caption words are kept only if they are nouns or adjectives, and each kept word is aligned to the regions with a grounding loss over the mini-batch, in both the image-to-caption and caption-to-image directions 2112.12143 §3.2 2112.12143 eq. 4. Caption-only data can be added with pseudo masks from a teacher model 2112.12143 §3.3. At inference, each class name is embedded with the same text encoder, and each pixel takes the class with the highest score 2112.12143 §3.4 2112.12143 eq. 6.

Reported effects, author-reported:

• On PASCAL-20 with EfficientNet-B7 trained on COCO and Localized Narratives, the paper reports 70.2 mIoU, and 72.2 with prompt engineering. The LSeg ViT-L figure it compares with is 52.3 2112.12143 tab. 2. Some baselines in that table are copied from other papers, which the card notes [/papers/2112.12143].
• Removing the mask loss, alpha = 0, gives poor performance 2112.12143 §3.2.
• On PASCAL-Context-59 with ResNet-101, inference with predicted masks gives 42.1 mIoU, against 32.1 for pixel-wise inference without masks 2112.12143 tab. 4.

Grouping patches from caption contrast alone (GroupViT)

Card: GroupViT: Semantic Segmentation Emerges from Text Supervision.

GroupViT is a hierarchical ViT. Learned group tokens merge image patches into larger segments in stages, and each merge is a hard assignment trained with a straight-through gradient 2202.11094 §3.1. The only training signal is image-text contrast. An extra multi-label term adds caption nouns, each put into short templates, as further positives (K = 3) 2202.11094 §3.2 2202.11094 §4.1. Zero-shot segmentation labels the final segments by comparing them with label-name embeddings, with no fine-tuning 2202.11094 §3.3.

Author-reported, with a ViT-S trained on CC12M and filtered YFCC (12M and 14M pairs) 2202.11094 §4.1. Zero-shot mIoU is 52.3 on PASCAL VOC 2012 and 22.4 on PASCAL Context. A fully supervised DeiT row reaches 53.0 and 35.9 2202.11094 tab. 5. In the ablation, adding both the hard assignment and the multi-label term raises VOC mIoU from 12.0 to 39.3 2202.11094 tab. 1. Background classes such as ground and road are often grouped correctly but labelled wrongly, so the approach is weakest there 2202.11094 app. C.3 [/papers/2202.11094]. The official code is licensed for non-commercial use only [/papers/2202.11094], so a commercial build should follow the paper alone.

Open-vocabulary segmentation with masked CLIP (OVSeg)

Card: Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP.

This is a two-stage segmenter. A class-agnostic mask model proposes about 100 masks per image 2210.04150 §3.2, and CLIP classifies each masked crop 2210.04150 §3.1. The CLIP image encoder is fine-tuned on masked crops paired with caption nouns, which are mined with CLIP from COCO captions 2210.04150 §3.2 2210.04150 §4.1. Mask prompt tuning then replaces fully masked patch tokens with learned prompts, and it can run with CLIP frozen 2210.04150 §3.3. The mask and CLIP scores are combined by a weighted geometric mean 2210.04150 §3.1.

Author-reported, with Swin-B and CLIP ViT-L/14 trained on COCO. The model reaches 29.6 mIoU on ADE20K-150, against 21.1 for OpenSeg with EfficientNet-B7 2210.04150 tab. 1, and 9.0 on ADE20K-847 2210.04150 tab. 1. Fine-tuning and prompt tuning together give 29.6 on ADE20K-150, against 28.8 with fine-tuning alone and 26.5 with prompts alone 2210.04150 tab. 3. Fine-tuning on fixed COCO-Stuff ground-truth classes, instead of mined captions, lowers ADE20K-847 from the baseline's 7.3 to 5.3 2210.04150 tab. 2. CLIP runs on each proposal, so inference takes about 0.6 s per image for ViT-L/14 on an A5000 2210.04150 app. A.9. The official code is CC BY-NC 4.0 [/papers/2210.04150], so use the paper for a commercial build.

Side network on a frozen CLIP (SAN)

Card: Side Adapter Network for Open-Vocabulary Semantic Segmentation.

SAN keeps CLIP frozen, except for its positional embedding, and adds a light side ViT with 8 blocks and width 240 2302.12242 §4.2. Its mask branch proposes masks. Its attention-bias branch adds one bias per CLIP head to CLIP's last blocks, so CLIP classifies each proposal 2302.12242 §3. The side network is the main trained part, with 8.4M trainable parameters 2302.12242 tab. 3. The code also fine-tunes CLIP's positional embedding by default 2302.12242 §4.4. Trained on COCO-Stuff, it reports 27.5 mIoU on ADE-150 with ViT-B/16 and 32.1 with ViT-L/14 2302.12242 tab. 2. Table 2 marks OvSeg as ensembled and SAN as not, so read the ensemble column next to any comparison with it. At 640² input, SAN runs at 15.2 FPS on one GPU, against 0.8 for SimSeg 2302.12242 tab. 3. Results are author-reported. The "SAN ensemble" rows need a model that is not in the public code, and the code's final map differs from the paper's formula (see the card's Paper vs code).

Bridge from a frozen image encoder to a frozen LLM (BLIP-2)

Card: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.

BLIP-2 trains a Querying Transformer (Q-Former) of 188M parameters between a frozen image encoder and a frozen LLM 2301.12597 §3.1. It has 32 learnable queries of width 768, which cross-attend to the image features in every other block 2301.12597 §3.1. Stage 1 trains it with three losses on a frozen ViT: image-text contrastive, image-text matching and image-grounded generation, which differ only in their attention masks 2301.12597 §3.2. Stage 2 projects the query outputs into the LLM's embedding width and prepends them as soft visual prompts. The LLM is OPT (decoder-only) or FlanT5 (encoder-decoder) 2301.12597 §3.3. Zero-shot VQAv2 test-dev reaches 65.0 with FlanT5-XXL, against 56.3 for Flamingo-80B, which has 10.2B trainable parameters 2301.12597 tab. 2. Without stage 1, zero-shot VQA is much lower for both LLM types, so the first stage should not be skipped 2301.12597 §4.1 2301.12597 fig. 5. Stage 1 is reported at under 6 days on 16 A100 (40 GB) GPUs for ViT-g with FlanT5-XXL 2301.12597 §3.4.

The InstructBLIP section below builds on this design.

Instruction-tuned bridge (InstructBLIP)

Card: InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning.

The image encoder (ViT-g/14) and the language model (FlanT5 or Vicuna) are both frozen. A Query Transformer (Q-Former) holds a set of learnable queries. The instruction is fed into the Q-Former alongside the queries, so the queries cross-attend to the image features under the instruction's guidance 2305.06500 §2.3. The query outputs are linearly projected and given to the language model as soft prompts, followed by the instruction text 2305.06500 §2.3. Only the Q-Former is trained, with the usual language-modelling loss on the response 2305.06500 §2.2.

The paper's training mix uses 26 public datasets across 11 task types, converted to instructions with 10 to 15 hand-written templates per task. 13 are held in for training and 13 are held out for zero-shot tests 2305.06500 §2.1 2305.06500 §2.2. Datasets are sampled with a probability proportional to the square root of their size, then adjusted by hand. The manual values are not stated 2305.06500 §2.4.

Reported effects, on FlanT5-XL:

• Zero-shot on held-out data, Table 1 shows 119.9 CIDEr on NoCaps against 104.5 for BLIP-2 with the same LLM 2305.06500 tab. 1.
• Removing the instruction-aware visual features lowers the held-in average from 94.1 to 89.8, with the largest drop on VizWiz, from 32.7 to 25.1, then on ScienceQA, from 70.4 to 63.4 2305.06500 tab. 2.
• The finetuning results are reported as 90.7% on ScienceQA (image context) for FlanT5-XXL 2305.06500 tab. 3.

Single projection onto a frozen LLM (MiniGPT-4)

Card: MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models.

MiniGPT-4 links a frozen EVA ViT and BLIP-2 Q-Former to a frozen Vicuna LLM through one trainable linear layer 2304.10592 §3. Stage 1 trains that layer on about 5M image-caption pairs from Conceptual Captions, SBU and LAION, for 20,000 steps at batch 256 on four A100 GPUs 2304.10592 §3.1. Stage 1 output is often repetitive or fragmented, so stage 2 fine-tunes on about 3,500 detailed descriptions that the stage 1 model wrote and ChatGPT then edited, for 400 steps at batch 12 2304.10592 §3.2. Only the stage 1 trainable part is stated; the stage 2 scope is not stated 2304.10592 §3.3. The stage 2 step is the one the authors say fixes the output. On 100 test images, the failure rate on detailed descriptions falls from 35% before stage 2 to 2% after, and on poems from 32% to 1% 2304.10592 tab. 3. Human raters judged 65 of 100 advanced-task requests satisfied, against 5 of 100 for BLIP-2 2304.10592 tab. 1. The comparison is not balanced: BLIP-2 uses FlanT5-XXL and 129M pairs, and MiniGPT-4 uses Vicuna and about 5M pairs 2304.10592 §4.1 2304.10592 app. A.2. Longer answers hallucinate more, with 9.6 CHAIR_i for long prompts against 7.2 for short ones 2304.10592 tab. 5.

CLIP prefix for a language model (ClipCap)

Card: ClipCap: CLIP Prefix for Image Captioning.

ClipCap turns one frozen CLIP image embedding into a short run of prefix vectors. The prefix sits in front of the caption embeddings of GPT-2, and training uses the usual caption cross-entropy, with the loss taken over the caption tokens only 2111.09734 §3.1 2111.09734 eq. 5. Two mappers are reported. The MLP mapper trains together with GPT-2, giving 156M parameters. The transformer mapper leaves GPT-2 frozen, so only the mapper trains, giving 43M parameters 2111.09734 §3.2 2111.09734 tab. 1.

Author-reported. On Conceptual Captions the MLP variant reaches 87.26 CIDEr, against 77.57 for VLP 2111.09734 tab. 1 (A). On nocaps the transformer variant reaches 65.83 overall CIDEr, against 63.8 for Oscar, which uses object tags 2111.09734 tab. 1 (B). On COCO, Oscar stays ahead at 124.12 CIDEr, against 113.08 for the transformer variant 2111.09734 tab. 1 (C). The timings are on different GPUs, so they are not directly comparable 2111.09734 tab. 1. The paper names GPT-2 large, but the released code loads the smaller gpt2 model 2111.09734 §3.1 [/papers/2111.09734].

Prompt learning in both encoders (MaPLe)

Card: MaPLe: Multi-modal Prompt Learning.

MaPLe learns prompt tokens in the text and image branches of a frozen CLIP ViT-B/16. The vision prompts are not free parameters: a linear layer computes each one from the language prompt of the same layer 2210.03117 §3.2.3. Prompts are placed in the first nine blocks for base-to-novel generalisation 2210.03117 §4.1. Trained on 16 labelled images per class, the method reaches 78.55 harmonic mean over base and novel classes, against 75.83 for Co-CoOp, and 75.14 novel-class accuracy, against 71.69 2210.03117 tab. 3. On cross-dataset transfer from ImageNet, the average over ten targets is 66.30, against 65.74 for Co-CoOp 2210.03117 tab. 4. It adds about 3.55M trainable parameters, 2.85% of CLIP 2210.03117 tab. 6. At batch size 1 it runs at 60.2 frames per second, against 64.6 for Co-CoOp, so the no-overhead claim holds for larger batches 2210.03117 tab. 6.

Text-only residual correction on frozen CLIP (CLAP)

Card: CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts.

CLAP trains a small residual MLP on top of frozen CLIP text features, using no images. Each template prompt is paired with a copy in which style words are dropped or reordered. An InfoNCE term pulls the prompt towards its augmented copy, and a second term keeps the copy close to the bare class name 2311.16445 §4.2 2311.16445 eq. 4. At inference the same network is applied after CLIP's image encoder and text encoder, with a scalar weight α on its residual branch 2311.16445 §4.1.3 2311.16445 app. A.1. Author-reported, the zero-shot average over PACS, VLCS, OfficeHome and DomainNet rises from 77.4% for CLIP ViT-B/16 to 79.9%, and the noised-prompt score rises from 70.4% to 76.1% 2311.16445 tab. 3. The paper also reports 1-shot linear-probe gains over CLIP of 1.5% to 10% by dataset 2311.16445 §5.2.2. Its image-augmentation comparison (Im.Aug) needs generated images, which took about 3.3 hours on DomainNet against 47 minutes for CLAP 2311.16445 §5.1. The α values were chosen by a search that the code runs on the test environments, so the reported gains may be optimistic [/papers/2311.16445]. The repository has no LICENSE file, so its terms are not stated.

Long-video question answering as an evaluation (EgoSchema)

Card: EgoSchema.

EgoSchema is not a training method. It is a multiple-choice benchmark of about 5,000 questions on three-minute Ego4D clips, with five options each 2308.09126 §3.1 2308.09126 app. Datasheet. Published video-language models score between 19.6% and 32.1% against 20% chance, while humans reach 75.0% to 76.2% 2308.09126 §4.2. The authors estimate the median temporal certificate, the span a human needs to verify an answer, at about 100 seconds 2308.09126 §4.1. Use it to test long-video understanding, not to train a model. Only 500 correct answers are public, so full scores come from a submission server [/papers/2308.09126].

Video-text pre-training with entity pseudo-labels (ALPRO)

Card: Align and Prompt: Video-and-Language Pre-training with Entity Prompts.

ALPRO pre-trains a video encoder, a text encoder and a fusion encoder on 5.5M pairs from WebVid-2M and CC-3M 2112.09583 §3.1 2112.09583 §3.3. Besides a video-text contrastive term over the [CLS] tokens 2112.09583 §3.2.1, it adds a term that predicts soft entity labels for random video crops. A frozen prompter, itself trained with the contrastive term, supplies those labels, so no detector or human annotation is needed 2112.09583 §3.2.2. Pre-training uses 16 A100 GPUs for 100k iterations at batch size 256 2112.09583 §3.4.

Reported effects, author-reported: after fine-tuning, R@1 on MSRVTT 1k is 33.9, against 31.0 for FiT, which has the same pre-training data 2112.09583 tab. 2. Zero-shot R@1 is 24.1 against 18.7 2112.09583 tab. 2. The same fine-tuned model without pre-training reaches 16.5 2112.09583 tab. 1. The authors read the ablation as the contrastive term helping retrieval most and the entity term helping QA most 2112.09583 §4.2.

Language-supervised visual encoder for robots (Voltron)

Card: Language-Driven Representation Learning for Robotics.

Voltron pre-trains a ViT on Something-Something-v2 video clips with their captions. It masks most patches of two frames and reconstructs the pixels. For each example it either conditions the reconstruction on the caption or asks the decoder to write the caption, and a single probability alpha sets how often each happens 2302.12766 §3.1 2302.12766 §3.2. The paper names three models: VCond and VDual use alpha 0, VGen uses alpha 0.5 2302.12766 §3.2. Captions enter through a frozen DistilBERT encoder 2302.12766 app. B.2.

Reported effects, with frozen features and a small head, against reimplemented MVP and R3M on the same frames 2302.12766 §4. On grasp affordance, top-1% precision is 80.71 for VCond ViT-S, against 61.47 for R-MVP and 40.55 for R-R3M 2302.12766 tab. 2. On referring-expression grounding, total accuracy is 89.38, against 63.30 for R-R3M and 49.58 for R-MVP 2302.12766 tab. 3. Removing the language input drops referring accuracy to 53.44 2302.12766 tab. 4. The control results are low in absolute terms, and the authors say few-demonstration control remains open 2302.12766 §5.3.

Continual CLIP pre-training on remote sensing (RemoteCLIP)

Card: RemoteCLIP: A Vision Language Foundation Model for Remote Sensing.

RemoteCLIP continues CLIP-style contrastive training on remote-sensing image and text pairs 2306.11029 §3.1. The captions come from existing annotations. Box-to-Caption writes five captions per image from object boxes, and Mask-to-Box turns each class mask into boxes before that step 2306.11029 §3.2.1 2306.11029 §3.2.2. The training set has 165,745 images with five captions each, 828,725 pairs in all 2306.11029 §4.1.2. Three backbones are reported: a modified ResNet-50, ViT-B-32 and ViT-L-14 2306.11029 §4.1.1.

Author-reported. The zero-shot average over 12 datasets rises from 56.02 to 62.41 for ViT-B-32, a gain of 6.39 2306.11029 tab. 3. ViT-L-14 reaches 50.52 mean recall on RSITMD retrieval 2306.11029 tab. 2. The retrieval claims need care: the text says RemoteCLIP is best on all three retrieval sets, but a ResNet-18 baseline (FBCLM) reaches 61.60 on UCM, against 56.36 for the best RemoteCLIP row 2306.11029 §4.2.1 2306.11029 tab. 2. The zero-shot average is not uniformly higher, since RemoteCLIP falls below CLIP on several datasets for every backbone 2306.11029 tab. 3. The largest model took 233.4 hours on four GPUs 2306.11029 §4.1.3. The training code is not in the public repository [/papers/2306.11029], and the source datasets carry their own terms [/papers/2306.11029].

Few-shot transfer with a language-initialised head (ELEVATER)

Card: ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models.

ELEVATER is a benchmark of 20 classification and 35 detection datasets, with optional class knowledge from WordNet, Wiktionary and GPT-3 2204.08790 §3.1. Its method part concerns the new linear head. The Separate-Head variant starts each class weight from the text embedding of that class's prompts, with bias zero, instead of from random numbers 2204.08790 §4.2. A Merge-Head variant keeps the parameter count of a language-free model 2204.08790 §4.2.

Reported effects, averaged over 20 classification datasets, CLIP ViT-B/32, author-reported: at 5 shots, linear probing reaches 65.35 with the language head against 58.09 with a random head, and at 20 shots fine-tuning reaches 72.19 against 46.76 2204.08790 tab. 2. Zero-shot accuracy for the same model is 56.64 2204.08790 tab. 2. The authors report that random heads make fine-tuning far worse than linear probing at few shots, while with the language head, fine-tuning overtakes linear probing from about 20 shots 2204.08790 §5.1 2204.08790 tab. 2. Adding WordNet, Wiktionary or GPT-3 text to class names raises zero-shot accuracy for a UniCL model from 27.15 to between 29.92 and 33.95 2204.08790 fig. 5.

Weight-space patching (PAINT)

Card: Patching open-vocabulary models by interpolating weights.

Two steps. First, fine-tune the CLIP visual encoder on the weak task, with the classifier head fixed to the output of CLIP's text tower 2208.05592 §2 2208.05592 §3. Second, set each weight to a blend of the zero-shot and fine-tuned weights, with a single coefficient alpha between 0 and 1 chosen on held-out validation data 2208.05592 §2. No parameters are added, so the model still accepts any class list 2208.05592 §3.

Reported effects, ViT-L/14, author-reported: on nine patch tasks, accuracy rises by 15 to 60 points while ImageNet drops by less than one point 2208.05592 §4 2208.05592 §1. Patching on one dataset raised accuracy on a related dataset in 7 of 8 pairs, by 1.1 to 19.2 points, and lowered it by 1.3 points for EuroSAT to RESISC45 2208.05592 tab. 2. For sequential patching on ViT-B/32, interpolation changes the combined accuracy by +11 points, against −4.6 points without it 2208.05592 §5.

Composed image retrieval through a pseudo word (Pic2Word)

Card: Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval.

A three-layer MLP maps a frozen CLIP ViT-L/14 image embedding to one token that is inserted into the text encoder 2302.03084 §3.2 2302.03084 app. B. It is trained on images alone, with a symmetric contrastive loss against the prompt "a photo of" plus that token 2302.03084 §3.2 2302.03084 §4.1. At query time the token fills a placeholder in a template, such as "a [domain] of []" or "a photo of [], [text]" 2302.03084 §3.3.

Reported effects, zero-shot, author-reported: CIRR test R@10 is 65.3, against 49.1 for CLIP image plus text and 56.7 for text only 2302.03084 tab. 3. Fashion-IQ validation averages 24.7 at R@10 2302.03084 tab. 4. On ImageNet domain conversion, R@10 averages 10.1 against 1.7 for the best zero-shot baseline 2302.03084 tab. 1. The supervised Combiner trained on CIRR still scores higher at R@10, 73.2 2302.03084 tab. 3. The text alone is a strong baseline on CIRR and Fashion-IQ 2302.03084 §4.3.

Prompt learning for backlit enhancement (CLIP-LIT)

Card: Iterative Prompt Learning for Unsupervised Backlit Image Enhancement.

Training uses unpaired backlit and well-lit photos, with CLIP frozen 2303.17569 §3. A learnable pair of positive and negative prompts, each 16 tokens of width 512, is fitted so that CLIP separates the two sets 2303.17569 §3.1. The enhancer predicts a one-channel illumination map, and its output is the input divided by that map 2303.17569 §3.1. The CLIP term pushes the output away from the negative prompt, and an identity term keeps the input's features 2303.17569 eq. 3 2303.17569 eq. 4. The prompts are then refined with a margin ranking loss, and the enhancer is retuned, in turns 2303.17569 §3.2.

Reported effects, BAID test set, author-reported: PSNR 21.579 and MUSIQ 55.682, against 19.740 PSNR for Zero-DCE, the best released baseline in PSNR 2303.17569 tab. 1. Replacing the CLIP term with an adversarial discriminator drops PSNR to 17.407 2303.17569 tab. 5. Fixed word prompts give PSNR 14.748, against 21.579 for the learned prompts 2303.17569 tab. 3. The reference images in BAID are retouched by photographers, and the authors say they are sub-optimal targets 2303.17569 §2.

3D shape alignment with generated captions (ULIP-2)

Card: ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding.

Each 3D shape is rendered from fixed viewpoints, 12 for Objaverse and 30 for ShapeNet 2305.08275 §4.1. BLIP-2 writes ten captions per view, and the one that CLIP ranks highest is kept 2305.08275 §4.1 2305.08275 §3.2. A 3D encoder is trained so that each point cloud matches its own image and caption embeddings in a frozen OpenCLIP ViT-G/14 space, with two symmetric contrastive losses 2305.08275 §3.3.

Reported effects, zero-shot, author-reported: ULIP-2 with Point-BERT, trained on Objaverse and ShapeNet, reaches 50.6 top-1 on Objaverse-LVIS and 84.7 top-1 on ModelNet40 2305.08275 tab. 1. The largest OpenShape row in the same table reaches 46.8 on Objaverse-LVIS 2305.08275 tab. 1. On ModelNet40, holistic BLIP-2 captions score 69.7 against 60.4 for manual captions in the authors' ablation 2305.08275 tab. 5.

Compositionality test for retrieval models (CREPE)

Card: CREPE: Can Vision-Language Foundation Models Reason Compositionally?

CREPE builds image-to-caption retrieval sets from Visual Genome scene graphs 2212.07796 §3.1. The systematicity test asks whether a model handles compounds that were unseen in the training captions of CC-12M, YFCC-15M or LAION-400M 2212.07796 §3.2. The productivity test uses captions of 4 to 12 atoms, with hard negatives that swap or negate parts of the caption 2212.07796 §3.3.

Reported effects, zero-shot, author-reported: on the atom-foil subset, Recall@1 falls by 12.57 points from seen to unseen compounds for LAION-400M ViT-B/32, and by 4.38 points for CC-12M RN50 2212.07796 app. B.1. On the compound-foil subset, the LAION-400M models gain 2.5 to 3.7 points instead 2212.07796 app. B.1. On productivity, most models score between 8.35% and 18.24% at 12 atoms on the atom-foil set, against 16.67% chance, with ALBEF at 24.34% 2212.07796 app. B.1.

Word-order probe (Winoground)

Card: Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality.

Winoground has 400 expert-written examples. Each has two images and two captions made of the same words in a different order, so only word order and visual grounding can tell the pairs apart 2204.03162 §3.1. A model is scored on a text score, an image score and a group score. The group score needs all four image-caption pairings to be right 2204.03162 §3.2 2204.03162 eq. 3.

Author-reported. Humans reach 85.50 on the group score. The best model, VinVL, reaches 14.50, against 16.67 for chance 2204.03162 tab. 3. Section 5.1 says six models exceed chance on the text score, while the card's Table 3 marks nine (see Contradictions). Every model is below chance on the group score 2204.03162 §5.1. The published text does not say which score function each model uses, so the choice must be tested 2204.03162 §3.2. The images are Getty Images licensed for research, so check the terms before redistributing them 2204.03162 §3.1. There is no official code [/papers/2204.03162].

Knowledge-seeking visual questions (InfoSeek)

Card: Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

InfoSeek asks questions that need facts beyond the picture, such as when a pictured building was built 2302.11713 §1. It has 8.9K human-written questions and about 1.35M questions generated from Wikidata with templates 2302.11713 §3.1 2302.11713 §3.2. In the No-KB set-up the model answers from the image and question alone. In the With-KB set-up a CLIP-based entity recognizer first links the image to a 100K-entry Wikipedia knowledge base, and a reader then answers from the linked articles 2302.11713 §4.

Reported effects, author-reported: fine-tuned PaLI-X reaches 22.1 overall on the Wikidata split and 10.8 on the human split with No-KB 2302.11713 tab. 4. A CLIP-to-FiD pipeline reaches 18.2 on the human split 2302.11713 tab. 5. With the gold entity, the same pipeline reaches 45.6 on that split 2302.11713 tab. 5. Before fine-tuning, PaLI-X scores 2.0 on the Wikidata split 2302.11713 fig. 3.

Per-object checks for text-to-image output (GenEval)

Card: GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment.

GenEval turns each prompt into metadata: object classes, counts, colours and relative positions 2310.11513 §3.1. A Mask2Former detector trained on the 80 COCO classes finds the objects, and a zero-shot CLIP ViT-L/14 classifier picks colours from masked crops 2310.11513 §3.2 2310.11513 app. C.3. Each image gets a pass or fail, and the six task scores are averaged 2310.11513 §3.2. The prompt set has 553 prompts 2310.11513 §5.1.

Author-reported. IF-XL reaches an overall score of 0.61 and the best attribute binding score, 0.35. The best position score is 0.15, for SD-XL 2310.11513 tab. 2. Against annotators, GenEval agrees on 83% of images, against 80% for thresholded CLIPScore 2310.11513 §4. The counting and position thresholds were chosen on the same annotations used to report agreement, so the held-out figures are the ones to quote 2310.11513 app. A.2. The detector cannot check fine detail, such as the number of fingers on a hand 2310.11513 §6. Overall scores move by 0.01 to 0.02 across seeds 2310.11513 tab. 2.

Self-supervised image loss added to CLIP (SLIP)

Card: SLIP: Self-supervision meets Language-Image Pre-training.

SLIP trains one shared Vision Transformer with CLIP's image-text contrastive loss plus a SimCLR loss on two augmented views of each image, summed with weight 1.0 2112.12750 §3.3 2112.12750 alg. 1. The CLIP baseline is the authors' own re-implementation. On YFCC15M for 25 epochs with ViT-B/16, SLIP reaches 42.8% zero-shot and 72.1% linear ImageNet accuracy, against 37.6% and 66.5% for that baseline 2112.12750 tab. 2. The gain holds for ViT-S, B and L, and at 100 epochs 2112.12750 §5.1 2112.12750 §5.2. Fine-tuning gains are small, from +0.1 to +0.4 over SimCLR alone 2112.12750 tab. 2. Fine-tuning CLIP from SimCLR-pretrained weights gives 41.1 zero-shot, 68.2 linear and 82.3 fine-tuned, below SLIP's 42.8, 72.1 and 82.6 2112.12750 tab. 7. The cost is higher: about three times the image activations per step 2112.12750 §3.3, and 30.5 hours against 22.3 hours for ViT-B/16 on 64 V100 GPUs 2112.12750 §7. The paper's CC3M and CC12M runs use 3.1M and 11.0M images, not the public totals 2112.12750 app. A.

Motion in CLIP space (MotionCLIP)

Card: MotionCLIP: Exposing Human Motion Generation to CLIP Space.

MotionCLIP trains a transformer auto-encoder on SMPL body motion. Its latent vector is trained to reconstruct the motion, and to sit close in cosine distance to the frozen CLIP embedding of the motion's text label and of one rendered frame 2203.08063 §3 2203.08063 eq. 2 2203.08063 eq. 3. At inference a prompt goes through frozen CLIP and straight into the decoder, with no text-motion training 2203.08063 §4.2. In author-reported user studies, raters preferred MotionCLIP over JL2P 76.7% of the time on in-domain actions, and 75.3% of the time on out-of-domain Olympic sports 2203.08063 tab. 1. For action recognition on BABEL-60, top-1 is 40.9% against 41.14% for 2s-AGCN, but top-5 is lower, 57.71% against 73.18% 2203.08063 tab. 3. Without the text loss, top-1 falls to 4.54% 2203.08063 tab. 3. The paper reports no automated motion-quality metric 2203.08063 §4.2. The card does not verify the licences of AMASS, BABEL, the rendered images or the SMPL models, which the repository says apply [/papers/2203.08063].

Gradient projection for few-shot prompts (ProGrad)

Card: Prompt-aligned Gradient for Prompt Tuning.

ProGrad learns CoOp-style context vectors for a frozen CLIP. It keeps the cross-entropy update unless that update points against the gradient of a KL term toward zero-shot predictions. If it does, the conflicting part is removed 2205.14865 §3.2 2205.14865 eq. 4. Setting the weight to zero gives back CoOp 2205.14865 §3.2. Averaged over 11 datasets with ResNet-50, ProGrad reaches 62.61% at one shot against 59.46% for CoOp, and 74.28% against 73.49% at 16 shots 2205.14865 tab. 1. In base-to-new generalisation at 4 shots, its harmonic mean is 69.06 against 65.58 for CoOp 2205.14865 tab. 3. Each step runs one extra no-gradient zero-shot pass and two backward passes [/papers/2205.14865]. The projection can only protect what the zero-shot prompt already gets right 2205.14865 §4.

Video from an image CLIP model (X-CLIP)

Card: Expanding Language-Image Pretrained Models for General Video Recognition.

X-CLIP reuses a pre-trained CLIP image encoder on frames. In each block, one message token per frame is exchanged across frames by cross-frame attention, and a small transformer merges the frame features into one video vector 2208.02816 §3.2 2208.02816 §3.2.1. Class names are turned into video-specific text prompts by cross-attention over the video's patch tokens 2208.02816 §3.3.1. On Kinetics-400, ViT-L/14 at 8 frames reaches 87.1 top-1, at 658 GFLOPs per view, against 84.8 for ViViT-H/16×2 at 8316 GFLOPs 2208.02816 tab. 1. Trained on Kinetics-400 and tested zero-shot, X-CLIP-B/16 reaches 72.0 on UCF-101 against 58.3 for ActionCLIP, and 65.2 on 220 unseen Kinetics-600 classes against 42.1 2208.02816 tab. 3 2208.02816 tab. 4. With two shots on HMDB-51, it reaches 53.0 against 20.9 for Swin-B 2208.02816 tab. 5. Those few-shot rows also compare pretraining sources, not only methods 2208.02816 §4.4. Freezing the text encoder costs 0.1 points fully supervised and 2.9 zero-shot, and cuts memory for two 8-frame inputs from 22 GB to 6 GB 2208.02816 tab. 7. The released code has no K-shot sampler, so the few-shot numbers cannot be rerun from it alone [/papers/2208.02816].

Video foundation model from two encoders (InternVideo)

Card: InternVideo: General Video Foundation Models via Generative and Discriminative Learning.

InternVideo trains two encoders separately. One is a masked video ViT-H, and the other is a CLIP-initialised UniFormerV2-L/14 trained with video-text contrastive and captioning losses 2212.03191 §3.1. Both are fine-tuned on Kinetics-710 2212.03191 §3.2. A cross-model attention module then fuses the two frozen encoders with a learned linear combination of their scores 2212.03191 §3.3. Author-reported: 91.1% top-1 on Kinetics-400 against 89.9% for MTV-H, and 77.2% on Something-Something V2 against 75.6% for MaskFeat 2212.03191 tab. 2 2212.03191 tab. 3. Fine-tuned MSR-VTT text-to-video R@1 is 55.2 against 49.4 2212.03191 tab. 6. The masked encoder was pre-trained on 64 A100 GPUs, and the video-text stage on 128 A100 GPUs for about two weeks 2212.03191 §4.2. The repository does not contain the cross-model attention, and it does not contain the video-text pre-training at the paper's settings [/papers/2212.03191].

Survey of vision Transformers (Transformers in Vision)

Card: Transformers in Vision: A Survey.

The survey has no method of its own. It sorts vision Transformer designs by self-attention variant and by task. Its multi-modal row in 2101.01169 tab. 2 lists three models with numbers compiled from their papers: ViLBERT, Oscar and UNITER. VQA accuracy is 70.6, 80.37 and 72.47. The retrieval R@1 is 58.2 for ViLBERT (dataset not named), 57.5 for Oscar on COCO, and 83.72 for UNITER on Flickr30K. The survey gives no setup for most rows of that table, and it notes that Oscar relies on pre-trained object detectors 2101.01169 tab. 2. Check the primary papers before quoting these numbers.

Text-driven edits of a StyleGAN through CLIP (StyleCLIP)

Card: StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery.

StyleCLIP steers a pretrained StyleGAN2 with CLIP's cosine distance to a text prompt 2103.17249 §4 2103.17249 eq. 1. It has three routes. Per-image latent optimisation is the most flexible and the slowest 2103.17249 §4. A per-prompt latent mapper gives one step per input after a training run 2103.17249 §5 2103.17249 eq. 3. A global direction in style space S comes from per-channel CLIP relevance, so one edit can be dialled up or down 2103.17249 §6 2103.17249 eq. 6 2103.17249 eq. 7.

Author-reported timings, on one GTX 1080Ti. Optimisation takes 98 s per inference. The mapper needs 10 to 12 h of training and 75 ms per inference. The global direction needs 4 h of pre-processing and 72 ms per inference 2103.17249 tab. 1. The mapper's step directions agree fairly well across inputs, with mean cosine similarity from 0.73 to 0.84 over eight prompts 2103.17249 tab. 2. The "Surprised" standard deviation is printed as 0.893, against 0.081 to 0.145 for the other prompts, and the paper does not flag it 2103.17249 tab. 2. The comparisons are visual only, with no quantitative image-quality or identity metric 2103.17249 §7.

Semantic consistency for few-view NeRF (DietNeRF)

Card: Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis.

DietNeRF trains a NeRF from a few posed photos with the usual pixel loss 2104.00677 eq. 3. It adds a semantic term. A random unseen pose is rendered, and the frozen CLIP ViT embedding of that render is pulled towards the embedding of a random training image 2104.00677 §4.1 2104.00677 alg. 1 2104.00677 eq. 5.

Author-reported, with eight training views on Realistic Synthetic scenes. The semantic model reaches 23.147 PSNR and 74.9 FID, against 14.934 and 228.1 for NeRF 2104.00677 tab. 1. A 50k-iteration pixel-only fine-tune raises the PSNR to 23.591 and lowers the FID to 72.0 2104.00677 tab. 1 2104.00677 tab. 5. In the encoder ablation on the Lego scene, a CLIP ViT-B/32 gives 23.896 PSNR, against 22.059 for an ImageNet ViT-B/32 2104.00677 tab. 4. The effect is not uniform. On single-view DTU, the semantic fine-tune lowers PSNR from 15.550 to 14.242 but lowers FID from 266.1 to 190.7 2104.00677 tab. 2. On the unseen half of Lego, DietNeRF's 20.753 PSNR is below 21.553 for Simplified NeRF 2104.00677 tab. 3. The repository has no licence file, so do not reuse its code without asking the authors [/papers/2104.00677].

One recipe for vision, language and joint tasks (FLAVA)

Card: FLAVA: A Foundational Language And Vision Alignment Model.

FLAVA has three transformer encoders: an image encoder, a text encoder with separate weights, and a multimodal encoder over the projected image and text states 2112.04482 §3.1. On paired data it uses a global contrastive loss, masked multimodal modelling and image-text matching [arxiv:2112.04482 §3.2]. On unpaired images and text it uses masked image and masked language modelling 2112.04482 §3.3. A downstream head sits on the image, text or multimodal output, depending on the task 2112.04482 §3.1.

Pre-training uses 70M public image-text pairs, plus unpaired ImageNet-1k images and text 2112.04482 §4 2112.04482 tab. 2. Author-reported, over 35 tasks, the macro average is 75.85. The same table gives 66.78 for CLIP ViT-B/16 trained on 400M pairs, and 61.52 for CLIP ViT-B/16 trained on the same 70M pairs 2112.04482 tab. 3. The card notes that four rows of that table have their columns out of order, so read it with care [/papers/2112.04482]. On vision alone the released CLIP is ahead, with a vision average of 82.57 against 79.44 2112.04482 tab. 3. On zero-shot retrieval, FLAVA is below released CLIP on Flickr30K text retrieval R@1 (67.70 against 82.20) and above it on image retrieval (65.22 against 62.08) 2112.04482 tab. 4. On VQAv2 test-dev it scores 72.8, against 55.3 for released CLIP, and its linear ImageNet score is 75.5, against 80.2 2112.04482 tab. 5. Without unimodal initialisation the macro average falls to 73.72 2112.04482 tab. 3. The code is BSD-3-Clause, and the card notes that its configs are not the paper's 70M run [/papers/2112.04482].

Modular vision encoder with two-stage LLM tuning (mPLUG-Owl)

Card: mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality.

mPLUG-Owl gives LLaMA-7B image input through a CLIP ViT-L/14 encoder and a learnable visual abstractor that compresses patch features into a short token sequence 2304.14178 §3.1. Stage 1 aligns images with captions while the LLM stays frozen. Stage 2 trains LoRA adapters on the LLM with text-only and image-text instructions 2304.14178 §3.2. The model has 7.2B parameters 2304.14178 §4.1.

Author-reported, on OwlEval: 82 open-ended questions on 50 images, rated by hand. 66 answers are rated A or B, against 54 for MiniGPT-4 2304.14178 §4.2. On text-only questions, with ChatGPT as judge, the full model scores 86.71%, against 89.47% for Vicuna-7B 2304.14178 tab. 3. In the authors' ablation, the full run is the highest of their five runs on all six ability codes, but it is a single run with no variance reported 2304.14178 tab. 2. The stage-1 hyperparameters disagree within the paper, and the repository releases only the stage-2 code [/papers/2304.14178]. The code licence is Apache-2.0 [/papers/2304.14178].

Visual shortcomings of CLIP-based multimodal LLMs (MMVP)

Card: Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs.

The authors look for image pairs that CLIP treats as near-identical while DINOv2 separates them, and build the MMVP benchmark of simple visual questions on those pairs [/papers/2401.06209]. They group the failures into nine visual patterns, the MMVP-VLM benchmark [/papers/2401.06209].

Author-reported. Four human participants average 95.7% on MMVP [arxiv:2401.06209 §2.3] 2401.06209 app. B.4. GPT-4V scores 38.7%, or 40.3% with the options swapped, and every model except GPT-4V and Gemini is below 25%, the random level 2401.06209 §2.3 2401.06209 app. B.3. The best CLIP model on the MMVP-VLM average is SigLIP ViT-SO-14 at 224², with 37.8 2401.06209 tab. 1. LLaVA with its CLIP encoder scores 5.5 on MMVP. Interleaved-MoF (I-MoF) adds DINOv2 tokens between the CLIP tokens and reaches 16.7 at 224², with 512 visual tokens 2401.06209 tab. 2 2401.06209 tab. 3. For LLaVA-1.5, the 224² I-MoF reaches 28.0 with 512 tokens, against 24.7 for the 336² baseline with 576 tokens [arxiv:2401.06209 tab. 3]. The per-pattern correlation with CLIP is 0.87 for LLaVA-1.5 and 0.31 for GPT-4 2401.06209 app. C 2401.06209 tab. 5. The I-MoF 224² setting is lower than LLaVA-1.5 on MMBench, TextVQA and VQA-v2 [arxiv:2401.06209 tab. 7]. The released scripts cover only the LLaVA-1 setting. The README claims an MIT licence, but the card found no LICENSE file at the repository root, so that licence is not verified [/papers/2401.06209].

Instruction-selected CLIP patches for a robot policy (OTTER)

Card: OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction.

OTTER keeps a CLIP ViT frozen and uses its image features as the visual input of a robot policy. It takes the last attention output of CLIP's vision encoder, not the block output 2503.03734 §3.1. For each camera it scores every patch against every instruction token, takes a softmax over patches with a learnable temperature, and sums the patches with those weights 2503.03734 §3.1 2503.03734 eq. 4. Each instruction word thus gets one visual vector. A cross-attention pooling block turns each camera's vectors into one token, and the words, the robot's proprioception and the visual tokens are concatenated into one token per time step. A causal transformer then predicts chunks of delta end-effector actions 2503.03734 §3.2. Only the pooling blocks, proprioception encoder, transformer and action head are trained 2503.03734 §3.1 2503.03734 §3.2.

Author-reported, on a Franka arm: 62% success on 70 unseen pick-and-place trials, against 12% for fine-tuned Octo and 9% for fine-tuned OpenVLA. The same table gives 61% for π0-Fast-Droid, evaluated directly and not fine-tuned 2503.03734 §5.1. Fine-tuning CLIP on the robot data lowers unseen success to 15% 2503.03734 §5.3. On LIBERO, unseen-task success is 61%, against 48% for fine-tuned OpenVLA and 26% for fine-tuned Octo 2503.03734 §5.2. Across four primitives on unseen tasks, the mean is 67%, against 4% for fine-tuned Octo 2503.03734 §5.1. Pre-training on human hand videos raises unseen success from 62% to 70% and leaves training-task success unchanged 2503.03734 §5.4.1. The trials are few, and the error bars are standard errors across trials, not across training seeds 2503.03734 §4.1. The released code is a PyTorch re-implementation of the paper's method, not the authors' original JAX code. The card's reading is that the cloned code fails to build the model, and that is untested. The paper's numbers may therefore not reproduce from this code [/papers/2503.03734].

Route-instruction pairs from house-tour video (InstruGen)

Card: InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models.

InstruGen makes training data for navigation agents without a simulator. It samples routes through 3879 YouTube house tours 2411.11394 §3.1 2411.11394 §4.1. CLIP labels each room node with a room type and objects, and an action inverse model estimates the action between rooms 2411.11394 §3.1. ChatGPT-4V then writes an instruction from the labelled triplets 2411.11394 §3.2. A consistency check extracts the room and action pairs from the text and regenerates it on mismatch, and regex rules clean the output 2411.11394 §3.2. A ViLBERT-style agent is pre-trained on the pairs with masked modelling, path-instruction judgment and path ranking 2411.11394 §3.3.

Author-reported. On R2R val-unseen, the agent reaches 71.10 success with the consistency check, regex rules and manual review, and 70.9 without the manual review 2411.11394 tab. 1. On RxR zero-shot val-unseen it reaches 29.35 2411.11394 tab. 5. The paper does not say how much manual review was done, nor how it compares with the baseline rows 2411.11394 tab. 3. The repository holds only a README, so the pipeline cannot be re-run from it [/papers/2411.11394].

CLIP score as a reward term (CLIP-RLDrive)

Card: CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning.

Here CLIP does not generate data or train the agent's encoder. It scores each camera frame against three action prompts (slow down, proceed, speed up), and the score is added to the simulator's reward 2412.16201 §2.4 eq. 14. The authors use a ViT-B/32 image encoder, partly re-trained on 500 scene and instruction pairs 2412.16201 app. A.2. The reward is added to DQN and PPO agents at an unsignalized intersection 2412.16201 §2.1 2412.16201 §2.2.

Author-reported. With one other vehicle, CLIP-DQN reaches 96% success against 70% for DQN, and with six vehicles 72% against 50% 2412.16201 tab. 2. CLIP-PPO reaches 38% with one vehicle, below the 78% of plain PPO 2412.16201 tab. 2. The paper reports 100 test episodes per agent and no seeds or confidence intervals 2412.16201 §3.2. The reward rule in the paper and the released code differ, so a re-implementation must pick one rule [/papers/2412.16201].

Document pre-training with OCR words, boxes and page image (DocFormer)

Card: DocFormer: End-to-End Transformer for Document Understanding.

The image here is a scanned page, not a photo. The model reads OCR words, their boxes and a ResNet50 grid of the page, and its attention adds spatial terms whose query and key weights are shared between the text and visual branches 2106.11539 §3.1 eq. 6 eq. 7. Pre-training on a 5M-page subset of IIT-CDIP uses masked word prediction, page-image reconstruction and a text-describes-image matching task, then drops those heads 2106.11539 §3 2106.11539 §3.2. Author-reported: the base model reaches 83.34 F1 on FUNSD 2106.11539 tab. 1 and 96.17% on RVL-CDIP 2106.11539 tab. 2. The ablations use 1M pages, so they are not comparable with the headline numbers 2106.11539 §4.5. No official code was found; implement from the card [/papers/2106.11539].

Mesh stylisation through a frozen CLIP loss (Text2Mesh)

Card: Text2Mesh: Text-Driven Neural Stylization for Meshes.

The input mesh's vertices and faces stay fixed. A small MLP gives each vertex a colour and a bounded offset along its normal, and CLIP similarity between the prompt and augmented multi-view renders trains it 2112.03221 §3.1 2112.03221 §3.2. No 3D generator or dataset is used. Author-reported user study: 3.90 for natural depiction against 2.83 for a VQGAN-CLIP mesh baseline 2112.03221 tab. 1. The authors note that CLIP can score a degenerate solution highly, so the geometry-only term and the augmentations are needed 2112.03221 §4.2. The MIT-licensed release differs from the paper's defaults in iterations, learning-rate decay and view averaging, so use its demo flags for a paper-like run [/papers/2112.03221].

Text edit of one image or video through a layer (Text2LIVE)

Card: Text2LIVE: Text-Driven Layered Image and Video Editing.

A new generator is trained for each edit, on augmented copies of one input. It outputs a colour layer and an opacity map, composited over the original 2204.02491 §3.1 eq. 1. CLIP losses drive it: a composition loss, a green-screen loss on the layer alone, and a self-similarity structure loss 2204.02491 eq. 2. For video, the same idea runs on a neural layered atlas from a frozen pre-trained model 2204.02491 §3.2. Author-reported human votes: 0.82 to 0.86 for the image method against mask-free image baselines, and 0.73 to 0.74 against the two video baselines 2204.02491 tab. 1. One image takes about nine minutes and one video layer about an hour on the authors' GPUs 2204.02491 app. A.3. The paper says the method is not suited to creating new objects 2204.02491 §4.5.

Shared embedding for 3D point clouds (ULIP)

Card: ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding.

The 3D encoder is the only trained part. Its output is pulled toward the frozen image and text embeddings of SLIP with symmetric contrastive losses 2212.05171 §3.2 eq. 4 eq. 5. Training triplets come from ShapeNet55: each CAD model gives rendered views, a category caption and its point cloud 2212.05171 §3.1. Author-reported: PointBERT gains 3.3 points on ScanObjectNN classification 2212.05171 tab. 1. PointBERT+ULIP reaches 60.4% zero-shot top-1 on ModelNet40, all classes 2212.05171 tab. 4. The code picks its checkpoint on that same test split, so expect optimistic zero-shot numbers unless you hold out a validation split [/papers/2212.05171]. The release expects pre-rendered images, which the README puts at about 1 TB [/papers/2212.05171].

Quality scoring from joint text labels (LIQE)

Card: Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective.

CLIP ViT-B/32 is fine-tuned so that each image is matched with 495 joint text labels: 5 quality levels, 9 scene types and 11 distortion types 2303.14968 §3.1. The quality score is the expected level under the quality marginal, and the scene and distortion heads come from the same softmax 2303.14968 §3.2 eq. 1 eq. 3. Training combines a pairwise fidelity loss for quality with multi-label and multi-class losses for the other two heads, weighted by dynamic weight averaging 2303.14968 §3.2 eq. 11. Author-reported median SRCC: 0.970 on LIVE and 0.919 on KonIQ-10k 2303.14968 tab. 1. Fine-tuning CLIP matters: a pretrained-CLIP variant averages 0.693 SRCC against 0.890 for the fine-tuned model in the authors' two-level ablation 2303.14968 tab. 4. The released MIT code freezes the softmax temperature that the paper calls learnable [/papers/2303.14968].

Self-regularised prompts that stay close to frozen CLIP (PromptSRC)

Card: Self-regulating Prompts: Foundational Model Adaptation without Forgetting.

Deep prompts are trained on both CLIP encoders. Besides cross-entropy, three agreement terms pull the prompted features and logits toward the frozen model: an L1 term on image features, an L1 term on text features and a KL term on logits 2307.06948 §3.2.1 eq. 2 eq. 3. The frozen text target averages 60 templates per class 2307.06948 §3.2.3. Prompts are combined across epochs with Gaussian weights 2307.06948 §3.2.2 eq. 6. Author-reported base-to-novel harmonic mean over 11 datasets: 79.97, against 78.55 for MaPLe 2307.06948 tab. 1. The paper reports about 46K tunable parameters against 3.55M for MaPLe 2307.06948 §4.3. On cross-dataset transfer the average over ten targets is 65.81, below the 66.30 for MaPLe 2307.06948 tab. 3. The MIT-licensed code computes the KL in the other direction and uses fixed templates and a deterministic epoch weighting, so check those before reproducing numbers [/papers/2307.06948].

Per-image style transfer from a text prompt (CLIPstyler)

Card: CLIPstyler: Image Style Transfer with a Single Text Condition.

A small U-Net is trained from scratch for each content image and text. Its output is scored by CLIP with a whole-image directional loss and a patch-wise directional loss over 64 random 128-pixel crops with perspective augmentation. Crops whose loss is at or below 0.7 are set to zero 2112.00374 §3.2 eq. 3 eq. 4. A content loss on VGG-19 features and a total-variation term keep the layout 2112.00374 §4.1. The author-reported CLIP score is 0.2515, against 0.2487 for AdaIN + CLIP 2112.00374 tab. 2. In the 25-rater user study the style score is 3.78, against 3.32 for AdaIN + CLIP 2112.00374 tab. 1. Training takes about 40 seconds per text on one RTX 2080 Ti 2112.00374 §4.1. A fast variant trains only a decoder for one text, and the paper reports under 0.5 seconds of inference per image 2112.00374 §5.1. The prose on the threshold describes the opposite direction to equation 4, and the card follows the equation.

Two-stage open-vocabulary segmentation with a frozen CLIP labeller (zsseg baseline)

Card: A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model.

A MaskFormer trained on seen classes proposes class-agnostic masks. Each masked crop, with its background filled, is scored by a frozen CLIP ViT-B/16 against text prompts. The score is a cosine similarity times 100, passed through a softmax over the class names 2112.14757 §4.1 2112.14757 §4.2.1 eq. 1. The per-mask scores are summed into a pixel map 2112.14757 §4.3 eq. 2. The paper says the trained and CLIP classifiers are "ensembled" but gives no formula, so the released code's weighting is a choice the card describes separately [/papers/2112.14757]. The authors report hIoU 37.8 on COCO-Stuff in the zero-shot setting, against 20.9 for FCN [arxiv:2112.14757 tab. 2], and 77.5 on Pascal VOC against 50.7 for FCN 2112.14757 tab. 3. Using CLIP alone on the crops gives 28.2 hIoU, and the retrained encoder alone gives 8.7 2112.14757 tab. 7. A learned prompt beats a hand-written one, 28.2 against 18.3 2112.14757 tab. 14. The self-training numbers are reported, but the paper does not say how its pseudo-labels are made 2112.14757 §6.4.

Input-only visual prompts for a frozen model (visual prompting)

Card: Exploring Visual Prompts for Adapting Large-Scale Models.

One additive prompt of the input's shape is learned by cross-entropy through a frozen model, then added to every test image 2203.17274 §3.2.1 eq. 1 eq. 2. The default is a padding frame of width 30 on 224 × 224 inputs, trained for 1,000 epochs with SGD at learning rate 40 [arxiv:2203.17274 §3.2.2] 2203.17274 §3.4. On CLIP ViT-B/32 across 12 datasets, the author-reported average is 78.2 with the prompt and text labels, against 54.2 for the text prompt alone and 83.1 for a linear probe 2203.17274 tab. 1. On the three vision-only classifiers, the same method averages 25.7 to 32.6, while a linear probe averages 71.7 to 72.6 2203.17274 tab. 4. So the method is strong with CLIP and weak with classifiers that have no text side. The release covers CIFAR-100 only and fixes the CLIP text template, so the 12-dataset numbers cannot be rerun from it [/papers/2203.17274].

One sparse mixture-of-experts tower for images and text (LIMoE)

Card: Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts.

One transformer takes image patches and text tokens, with no modality flag. Its MoE layers send each token to one of 32 experts 2206.02770 §2.2 app. A. The contrastive loss is the usual symmetric softmax over the batch 2206.02770 §2.1 eq. 1. Left alone, the router collapses on the minority modality. The paper adds per-modality entropy losses, weighted by 0.04 in the final model, and batch priority routing 2206.02770 §2.2.2 app. B 2206.02770 §2.2.3. Against dense one-tower baselines at matched compute, the authors report average gains of 10.1 points zero-shot and 12.2 points 10-shot on ImageNet 2206.02770 §3.1. The headline 84.1% zero-shot result for LIMoE-H/14 uses JFT-4B co-training and model souping 2206.02770 §3.2 app. A.3. The card records no official code, and the auxiliary-loss combination is not fully specified [/papers/2206.02770].

When to use which

The families answer different questions. A sigmoid-trained model, or an align-then-fuse model such as ALBEF, scores how well an image and a text match. OpenSeg, OVSeg, SAN and GroupViT score regions against words. Generative models such as OFA, PaLI, BLIP-2 and MiniGPT-4 produce text; their cards describe no embedding output for retrieval 2305.06500 §2.5 2304.10592 §3.1. OTTER outputs robot actions, so its card describes no text or embedding output either 2503.03734 §3.2. The sampling-time and few-shot cards use a match score as a loss, reward or cache key to change an output or a classifier, and they output no embeddings for retrieval.

Implementation sketch

The core of the sigmoid variant. Each step cites the source. The paper's Algorithm 1 is the reference 2303.15343 alg. 1.

# Pairwise sigmoid loss over one batch of n image-text pairs
# [arxiv:2303.15343 alg. 1]
def sigmoid_loss(img_emb, txt_emb, t_prime, b):
    # Normalise both towers' outputs to unit length  [arxiv:2303.15343 §3.1]
    x = l2_normalize(img_emb)                   # (n, d)
    y = l2_normalize(txt_emb)                   # (n, d)
    # Temperature is exp of a learned scalar  [arxiv:2303.15343 §4.9]
    t = exp(t_prime)
    # One logit per image-text pair, with a learned bias  [arxiv:2303.15343 §3.2]
    logits = t * (x @ y.T) + b                  # (n, n)
    # +1 on the diagonal (true pairs), -1 everywhere else  [arxiv:2303.15343 §3.2]
    labels = 2 * eye(n) - 1
    # Sum over all pairs, divide by n not n²  [arxiv:2303.15343 §3.2]
    return -sum(log_sigmoid(labels * logits)) / n

Notes on the variants:

• The bias enters the logit with a plus sign, as in the algorithm. The printed equation in the paper uses a different sign. See the Pitfalls on the card [/papers/2303.15343].
• Across devices, the chunked form needs D−1 permutations of the text embeddings and a loop over devices. Every cross-device block is all negatives 2303.15343 §3.3.
• OpenSeg scores a region against a caption word with a softmax over regions, using a learnable temperature, and a cosine similarity 2112.12143 eq. 2 2112.12143 eq. 3. The region features are pooled from the masks 2112.12143 §3.2.
• ALBEF and BLIP both mix a momentum teacher's distribution into the contrastive target, so that some in-batch negatives count partly as positives 2107.07651 eq. 6 2201.12086 §3.2. The mixing weight is alpha in ALBEF and not stated in BLIP.
• OTTER's selection step is a softmax over patches, computed separately for each instruction word, with a learnable temperature. It runs before the pooling. The position embedding enters only the weighted sum, not the similarity scores 2503.03734 §3.1 2503.03734 eq. 4. The card's pseudocode follows the paper, with the temperature clamp as the only stated bound [/papers/2503.03734].

For the instruction-tuned bridge, the per-step flow is: frozen image encoder, then the Q-Former with instruction tokens and queries, then a linear projection, then the frozen language model. The loss covers only the response 2305.06500 §2.2 2305.06500 §2.3. BLIP-2 and MiniGPT-4 share the same frozen-bridge-LLM shape, but BLIP-2 trains a Q-Former and MiniGPT-4 trains only one projection in its first stage 2301.12597 §3.3 2304.10592 §3.1.

For VQGAN-CLIP, each step decodes the latent, takes augmented crops, embeds them with CLIP, computes the distance to the prompt embedding, and takes one Adam step on the latent 2204.08583 §2. The card's pseudocode follows the text and the code [/papers/2204.08583].

For the sampling-time guidance cards, each reverse step adds a scaled gradient of a CLIP-based score to the model's mean 2112.05744 alg. 1. For Tip-Adapter, the training-free prediction adds a cache vote to the zero-shot logits 2111.03930 eq. 6. Both cards publish pseudocode, and the differences from the released code are in each card's Paper vs code section.

For the generative model in PaLI, the decoder writes the answer one token at a time from the encoder states 2209.06794 §3.1. The paper does not say how the visual tokens enter the encoder. The card's sketch assumes they are concatenated with the prompt, which is an assumption [/papers/2209.06794].

For SLIP, each step adds a SimCLR term, weighted 1.0, to the CLIP term 2112.12750 §3.3. For MotionCLIP, the latent is trained on a reconstruction loss plus two cosine distances to frozen CLIP embeddings 2203.08063 eq. 4. For ProGrad, one gradient step on the context vectors uses the cross-entropy gradient, with its conflicting part removed against the KL gradient 2205.14865 eq. 4. X-CLIP's per-frame message tokens are built from each frame's [class] token and dropped before the feed-forward layer 2208.02816 §3.2.1.

Contradictions

Numbers in different cards measure different things (ImageNet and retrieval for SigLIP, PASCAL and ADE mIoU for OpenSeg, held-out multimodal benchmarks for InstructBLIP, human ratings for VQGAN-CLIP, FFHQ FID for SDG, DreamBench DINO for BLIP-Diffusion, FID on ShapeNet for CLIP-Forge, nocaps CIDEr for CC12M and BLIP, COCO and VQAv2 test splits for OFA and PaLI, base-to-novel harmonic mean for MaPLe, ADE20K and PASCAL mIoU for GroupViT and OVSeg, group score for Winoground, ImageNet zero-shot for DataComp, mean recall for RemoteCLIP, the overall score for GenEval, and the MMVP and OwlEval results, DocFormer's FUNSD and RVL-CDIP scores, Text2Mesh and Text2LIVE user-study votes, LIQE's SRCC, ULIP's zero-shot ModelNet40 accuracy, and the CLIP baselines in CLAP and BriVL, which use different backbones and datasets), so they should not be compared directly. Where two cards share a measure, the numbers agree in the cases checked. For example, the ALBEF 14M COCO text retrieval R@1 (77.6) appears in both the ALBEF and BLIP tables 2107.07651 tab. 2 2201.12086 tab. 5, and so does its zero-shot Flickr30K text retrieval R@1 (94.1) 2107.07651 tab. 3 2201.12086 tab. 6.

Cross-paper disagreements, each side cited:

• COCO captioning with cross-entropy. OFA reports 145.3 CIDEr against 143.3 for SimVLM 2202.03052 tab. 3. PaLI reports 149.1 on the COCO Karpathy test and calls its result the best cross-entropy-trained result on COCO 2209.06794 tab. 1 2209.06794 §4.1. The two cards do not state the same split and baselines, so the numbers are not directly comparable.
• BLIP-2 numbers in MiniGPT-4. MiniGPT-4 reports BLIP-2 at GQA 42.4 with 129M pairs 2304.10592 app. A.2. The BLIP-2 card reports GQA test-dev 44.7 for its FlanT5-XXL model in zero-shot setting 2301.12597 tab. 2. The configurations differ, and the two cards do not explain the gap.
• Pre-training size in OFA. The abstract says 20M image-text pairs, but the data table has 15.25M captioning and matching samples, 2.92M VQA samples and 3.20M grounding samples 2202.03052 tab. 11. The card records the text does not say how these relate [/papers/2202.03052].
• Scaling-law fits. The paper says both curves are fitted on the Pareto frontier 2212.07143 fig. 1 2212.07143 §4.1. The released notebook bins the OpenCLIP points into five compute bins, while it fits the OpenAI points by ordinary least squares over all points [/papers/2212.07143]. The two exponents therefore come from different fits.
• Batch sizes in the scaling study. The text gives global batches of "86-88K", and the run table has 79K, 64K, 33K and 32K values for several runs 2212.07143 §3.2 2212.07143 tab. 18.
• LAION-2B in two papers. The LAION-5B card reports 75.2 zero-shot ImageNet top-1 for ViT-L/14 trained on LAION-2B-en 2210.08402 tab. 1. DataComp reports 73.1 for LAION-2B under its own fixed recipe 2304.14108 tab. 1 2304.14108 §3.4. The LAION-5B card gives no matching recipe, so the two numbers are not directly comparable.

Inconsistencies inside a single paper, or between a paper and its code, recorded so readers do not trust either side blindly:

• SigLiT with a B/8 image tower is 79.7% in the text and 79.8% in Table 1 2303.15343 §4.4 2303.15343 tab. 1.
• The bias sign in the printed sigmoid-loss equation differs from Algorithm 1 2303.15343 §3.2 2303.15343 alg. 1.
• FlanT5-XL is listed as 3B parameters in one section and 4B in another 2305.06500 §2.6 2305.06500 §3.5.
• VQGAN-CLIP's paper gives a learning rate of 0.15 and 400 iterations 2204.08583 §2. The card records that the official code's defaults differ, so runs made with the code do not reproduce the paper settings [/papers/2204.08583].
• SDG's text guidance defaults to scale 120 in the text, but the FFHQ text row in Table 1 (FID 28.38) equals the scale-320 row of Table 3, not the 120 row 2112.05744 §4.1 2112.05744 tab. 1 2112.05744 tab. 3.
• Tip-Adapter's abstract calls the training-free method comparable to CLIP-Adapter, while its ImageNet table puts it below CLIP-Adapter at every shot count 2111.03930 tab. 2.
• CLIP-Forge's ablation reports FID 2591.87 for the RealNVP-D setting, against 2425.25 for the same setting in the main table. The paper attributes the gap to seed selection 2110.02624 tab. 4 2110.02624 tab. 1 2110.02624 app. A.
• The BLIP-Diffusion text gives a fine-tuning learning rate of 5e-5, and the appendix and code give 5e-6 2305.14720 §3.3 2305.14720 app. A.5.
• ALPRO's text sets the pseudo-label threshold at 0.2 on a normalised score. The released code compares an argmax index against it, so the threshold is never applied to a score 2112.09583 §3.4 [/papers/2112.09583].
• CREPE's text calls the change on compound foils "little (1–3%) to no difference", but the LAION-400M rows gain 2.5 to 3.7 points 2212.07796 §5.1 2212.07796 app. B.1. Its abstract says Recall@1 drops "by up to 12%", which is 12.57 points on the atom-foil subset for one model, against 6.28 points on the combined set 2212.07796 app. B.1.
• InfoSeek's written metric is VQA accuracy with a one-year tolerance for time answers. The released code scores these by exact match 2302.11713 app. A.3 [/papers/2302.11713].
• CLIP-LIT's loss equation gives the identity term a weight of 0.9, while the code multiplies the two terms by 320 and 25 2303.17569 eq. 5 [/papers/2303.17569].
• ELEVATER's text gives UniCL's best knowledge result as 33.93, and its figure table gives 33.95 2204.08790 §5.4 2204.08790 fig. 5. The table is the one to use.
• Pic2Word's text says the mapper has about 0.8M parameters. The layer widths in its appendix give about 0.92M 2302.03084 §3.2 [/papers/2302.03084].
• ALBEF's footnote to §3.5 says the RandAugment list drops colour operations, because captions mention colours, while the released code still uses four colour operations 2107.07651 §3.5 [/papers/2107.07651].
• ALBEF's warm-up is 1,000 iterations in the paper. By the card's reading of the released scheduler, it runs over about 2,000 iterations 2107.07651 §3.5 [/papers/2107.07651].
• BLIP's text gives a linear learning-rate decay with rate 0.85, while the released code decays by 0.9 per epoch 2201.12086 §4.1 [/papers/2201.12086].
• BLIP-2's stage 2 minimum learning rate is 5e-5 in the paper and 1e-5 in the code 2301.12597 §3.4 [/papers/2301.12597]. Its text also gives beta1 twice, and the second value is beta2 2301.12597 §3.4.
• MiniGPT-4's stage 2 is 400 steps in the paper and 1,000 steps in the released config 2304.10592 §3.3 [/papers/2304.10592].
• LAION-5B's text sets the image cut-off at 5 KB, while its code counts bytes 2210.08402 §3.1 [/papers/2210.08402].
• MaPLe's §4.1 and the code give a base-to-novel learning rate of 0.0035, while appendix A gives 0.0025 2210.03117 §4.1 2210.03117 app. A [/papers/2210.03117]. Its §4.3 says MaPLe beats CLIP on novel accuracy on 6 of 11 datasets, but table 3 shows 7 of 11 2210.03117 §4.3 2210.03117 tab. 3.
• unCLIP's text gives 10% for the CLIP-embedding dropout in decoder training, while §5.1 and the fig. 8 caption give 5% 2204.06125 §2.1 2204.06125 §5.1.
• CC12M's table caption and text disagree on which model is best on nocaps 2102.08981 tab. 4 2102.08981 §4.1.
• ClipCap's model size. The paper says GPT-2 large 2111.09734 §3.1, but the released code loads the smaller gpt2 model [/papers/2111.09734]. The transformer prefix length is 10 in the paper and 40 in the README 2111.09734 §4 [/papers/2111.09734].
• GroupViT's gains. Section 4.2 pairs the 13.1-point gain with hard assignment and the 2.6-point gain with soft assignment. Table 1 shows the reverse 2202.11094 §4.2 2202.11094 tab. 1.
• OVSeg's PASCAL Context value. Section 4.2 gives 55.5 for Swin-B on PC-59, while Table 1 gives 55.7, which matches the stated 13.6-point gap over OpenSeg. Use 55.7 from the table 2210.04150 §4.2 2210.04150 tab. 1.
• DataComp's loss. The footnote in §3.4 gives the contrastive loss without its logarithm and minus sign. Implement the standard CLIP form 2304.14108 §3.4.
• RemoteCLIP's "best on all three". The text says RemoteCLIP is best on all three retrieval benchmarks, but Table 2 gives FBCLM 61.60 on UCM 2306.11029 §4.2.1 2306.11029 tab. 2.
• Winoground's chance level. The abstract says no model does much better than chance. §5.1 says six models exceed chance on the text score 2204.03162 §5.1. The card's Table 3 marks nine models above the 25.00 text chance level: VinVL, UNITER large and base, ViLLA large and base, ViLT, FLAVA ITM, FLAVA contrastive and CLIP 2204.03162 tab. 3. Both counts are recorded here, and the page does not choose between them [/papers/2204.03162].
• GenEval's position rule. The appendix prints the "above" condition with a sign that overlaps the "below" condition and contradicts the rest of the rule 2310.11513 app. C.3.
• Dream Fields' transmittance schedule. The paper uses 40% to 88%, and the code uses 50% to 90% 2112.01455 §4.5 [/papers/2112.01455].
• SLIP's YFCC15M linear values. Table 5 prints the CLIP and SLIP linear values for YFCC15M the wrong way round, against tables 2, 6 and 7 2112.12750 tab. 5 2112.12750 tab. 2. The ViT-L zero-shot gain is printed as +4.8, and the text repeats a range that ends at +4.8, but 46.2 − 40.4 = 5.8 2112.12750 tab. 2 2112.12750 §5.1. Use the raw numbers.
• InternVideo's Kinetics-710 size and benchmark count. The text gives 650K videos for Kinetics-710, and Table 1 gives 680k 2212.03191 §4.1 2212.03191 tab. 1. The paper gives 39 datasets in §1 and 34 benchmarks in §3 2212.03191 §1 2212.03191 §3.
• InternVideo's masked pre-training schedule, paper against code. The paper states 1200 epochs at base learning rate 2.5e-4, and the released script runs 301 epochs at 1e-3 2212.03191 §4.2 [/papers/2212.03191].
• X-CLIP's similarity scale, paper against code. The paper scores videos by plain cosine similarity, and the code multiplies the cosine by the exponential of a learned logit scale 2208.02816 eq. 3 [/papers/2208.02816].
• ProGrad's base-to-new context length, paper against code. The paper uses 4 context tokens for base-to-new, and the released scripts use 16 2205.14865 §4 [/papers/2205.14865].
• MotionCLIP's loss weights, paper against code. The paper sets both text and image weights to 0.01 2203.08063 §4.1, while the code default is 1.0 and the README run keeps it [/papers/2203.08063].
• StyleCLIP's standard deviation. The "Surprised" entry in tab. 2 is printed as 0.893, while the other prompts are between 0.081 and 0.145 2103.17249 tab. 2. The paper does not say whether this is a typo.
• FLAVA's table columns. The card records four rows of the macro-average table with columns out of order 2112.04482 tab. 3 [/papers/2112.04482].
• InstructBLIP's ablation text against its table. §3.1 says removing the instruction-aware features hurts most on ScienceQA and iVQA [arxiv:2305.06500 §3.1]. Table 2 shows the largest drop on VizWiz (32.7 to 25.1), then on ScienceQA (70.4 to 63.4), with iVQA 53.1 to 47.5 2305.06500 tab. 2.
• OTTER's action horizon. The architecture table gives 12 for real robots and 1 for simulation, while §3.2 says 8 steps gave the best results 2503.03734 §3.2 [/papers/2503.03734]. The code default is 12.
• OTTER's trial counts. The simulation table caption gives 50 trials per unseen task, and the simulation ablation table gives 100 trials on the same unseen tasks, with 61% in both [/papers/2503.03734].
• OTTER's encoder size. The text and the architecture table give ViT-L/14, and the scaling figure caption gives ViT-L/16 2503.03734 §5.4 [/papers/2503.03734].
• OTTER's loss, paper against code. The paper names no loss 2503.03734 §3.2. The code's forward pass uses L1 unless a flag is set, and its constructor reads a config field that the config class does not define. The card reads this as a construction error, which it did not run [/papers/2503.03734].
• OTTER's 50 Hz claim, paper against code. The paper says CLIP features are cached at inference, which allows 50 Hz on one GPU 2503.03734 app. B.1. The code runs both CLIP encoders on every call [/papers/2503.03734].
• OTTER's gradient clipping, paper against code. Appendix B.2 lists a clip threshold of 1, but the training loop applies no clipping 2503.03734 app. B.2 [/papers/2503.03734].
• OTTER's mean column. In the multi-primitive table the mean is not the average of the four task columns. For OTTER-OXE-L the columns average 80% and the mean is 77% 2503.03734 §5.1.
• CLIP-RLDrive's reward rule. Eq. 14 always adds the CLIP term, while Algorithm 1 adds it only when the agent's action matches CLIP's suggestion 2412.16201 eq. 14 2412.16201 alg. 1. The released code uses a third rule with fixed penalties and bonuses [/papers/2412.16201].
• CLIP-RLDrive's PPO claim. The text says CLIP-PPO beats plain PPO on training reward 2412.16201 §3.1. The test table shows lower success for CLIP-PPO at every vehicle count, 38 against 78 with one vehicle 2412.16201 tab. 2.
• CLIP-RLDrive's paper against code, discount. The paper gives DQN gamma 0.95 and PPO gamma 0.99 2412.16201 tab. 1. The code's DQN uses 0.99, and its commented-out PPO uses 0.95 [/papers/2412.16201].

Papers

• Conceptual 12M: relaxes the CC3M filters to build a 12.4M-pair web dataset for captioning pre-training.
• Align before Fuse: aligns image and text features before fusion, with momentum distillation as a teacher.
• An Empirical Study of Training End-to-End Vision-and-Language Transformers: METER, a fully transformer model with co-attention, trained end to end without an object detector.
• Zero-Shot Text-Guided Object Generation with Dream Fields: optimises a NeRF against a frozen CLIP caption score, for text-to-3D without 3D training data.
• More Control for Free! Image Synthesis with Semantic Diffusion Guidance: steers an unconditional DDPM at sampling time with CLIP-based language, image and style guidance.
• StyleGAN-NADA: shifts a pretrained StyleGAN2 to a new domain from two text prompts, using a directional CLIP loss.
• CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation: generates 3D shapes from text using a conditional flow trained on frozen CLIP image features.
• Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling: builds a training-free key-value cache from few-shot CLIP features, and can fine-tune the keys.
• ClipCap: CLIP Prefix for Image Captioning: maps a frozen CLIP embedding to GPT-2 prefix vectors with a small mapper, for captioning.
• Align and Prompt: pre-trains a video-text encoder with a contrastive term and an entity pseudo-label term, for retrieval and video QA.
• Scaling Open-Vocabulary Image Segmentation with Image-Level Labels: learns mask proposals and aligns caption words to them with a grounding loss, so it can segment open vocabularies.
• BLIP: one encoder-decoder model for matching and captioning, trained on bootstrapped web captions.
• OFA: one sequence-to-sequence model for vision, language and cross-modal tasks, with a shared vocabulary.
• GroupViT: Semantic Segmentation Emerges from Text Supervision: groups image patches into segments with caption contrast alone, and labels them zero-shot.
• Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality: a 400-example benchmark for matching captions to images through word order.
• Hierarchical Text-Conditional Image Generation with CLIP Latents: unCLIP, a prior from captions to CLIP image embeddings, followed by a diffusion decoder.
• VQGAN-CLIP: generates and edits images by optimising a VQGAN latent against a frozen CLIP, with no training.
• ELEVATER: a benchmark for transfer of language-augmented models, with a language-initialised linear head.
• MineDojo: trains MineCLIP on Minecraft video and transcripts, and uses it as a dense reward for RL agents.
• Patching open-vocabulary models by interpolating weights: patches a CLIP model by blending fine-tuned and zero-shot weights.
• PaLI: a multilingual image-and-text-to-text model with a scaled ViT and mT5, trained on many tasks.
• MaPLe: learns coupled prompts in both CLIP encoders for few-shot and base-to-novel classification.
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP: fine-tunes CLIP on masked regions named by caption nouns, with mask prompt tuning, for open-vocabulary segmentation.
• Side Adapter Network for Open-Vocabulary Semantic Segmentation: keeps CLIP frozen and adds a light side ViT that proposes masks and steers CLIP's last blocks to name them.
• LAION-5B: a 5.85-billion-pair open image-text dataset filtered by CLIP similarity, with NSFW and watermark tags.
• Reproducible scaling laws for contrastive language-image learning: fits power laws for OpenCLIP models trained on LAION data, with predictions flagged as extrapolations.
• CREPE: a benchmark for compositional reasoning in vision-language models, with systematicity and productivity tests.
• BLIP-2: bridges a frozen image encoder and a frozen LLM with a trained Q-Former, in two stages.
• Pic2Word: maps an image to a word token for zero-shot composed image retrieval.
• InfoSeek: a visual question answering benchmark that needs outside knowledge about the pictured entity.
• Language-Driven Representation Learning for Robotics: pre-trains a robot visual encoder by reconstructing masked video frames, with the caption as a condition or a target.
• Sigmoid Loss for Language Image Pre-Training: replaces the batch softmax with a per-pair sigmoid loss and reports SigLiT and SigLIP results.
• CLIP-LIT: enhances backlit photos with learned CLIP prompts and an illumination-map network, trained on unpaired images.
• MiniGPT-4: links a frozen BLIP-2 vision stack to a frozen Vicuna through one trained projection, with a curated second stage.
• DataComp: In search of the next generation of multimodal datasets: a benchmark for choosing image-text training data under a fixed recipe, with a 12.8B-pair pool.
• InstructBLIP: instruction-tunes a frozen image encoder and LLM through a trained Q-Former.
• ULIP-2: aligns 3D point clouds with generated captions and frozen OpenCLIP features.
• BLIP-Diffusion: turns a subject photo into prompt tokens with a BLIP-2 Q-Former, for subject-driven generation.
• RemoteCLIP: A Vision Language Foundation Model for Remote Sensing: continues CLIP training on remote-sensing pairs built from detection and segmentation labels.
• EgoSchema: a multiple-choice benchmark for long-form video question answering, for evaluation rather than training.
• GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment: scores text-to-image outputs per object, count, colour and position with a detector and a CLIP colour check.
• SLIP: Self-supervision meets Language-Image Pre-training: adds a SimCLR loss to CLIP's contrastive loss on one shared Vision Transformer.
• MotionCLIP: Exposing Human Motion Generation to CLIP Space: aligns a motion auto-encoder's latent with frozen CLIP text and image embeddings, for text-to-motion.
• Prompt-aligned Gradient for Prompt Tuning: ProGrad, which drops the parts of a prompt-tuning gradient that conflict with zero-shot CLIP predictions.
• Expanding Language-Image Pretrained Models for General Video Recognition: X-CLIP, which adapts CLIP to video with cross-frame message tokens and video-specific text prompts.
• InternVideo: General Video Foundation Models via Generative and Discriminative Learning: trains a masked video encoder and a video-text encoder separately, then fuses them with cross-model attention.
• Transformers in Vision: A Survey: surveys Transformer designs for vision by attention type and task, with tables compiled from other papers.
• StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery: edits StyleGAN2 images from text prompts through CLIP, with an optimiser, a latent mapper and a global direction.
• Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis: adds a frozen-CLIP semantic loss on unseen NeRF views, for reconstruction from few photos.
• FLAVA: A Foundational Language And Vision Alignment Model: one transformer family for image, text and joint tasks, pre-trained on public image-text pairs and unpaired data.
• mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality: connects a CLIP encoder to LLaMA-7B through a visual abstractor, trained in two stages.
• Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs: builds MMVP, a benchmark of image pairs that CLIP confuses, and tests an interleaved CLIP and DINOv2 input.
• InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models: labels house-tour video with CLIP and writes navigation instructions with a large multimodal model, with a consistency check.
• CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning: adds a CLIP prompt score to the reward of DQN and PPO agents at an unsignalized intersection.
• OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction: keeps CLIP frozen and pools its patch features by similarity to each instruction word, for a robot action policy.
• DocFormer: End-to-End Transformer for Document Understanding: fuses OCR text, word boxes and a page image in one transformer, pre-trained on IIT-CDIP pages.
• Text2Mesh: Text-Driven Neural Stylization for Meshes: stylises a fixed mesh's colour and normal offset with a neural field trained on CLIP similarity.
• Text2LIVE: Text-Driven Layered Image and Video Editing: trains a per-input generator that outputs a colour-plus-opacity layer, guided by CLIP losses.
• ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding: aligns a trainable 3D point-cloud encoder with frozen SLIP image and text embeddings.
• Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective: fine-tunes CLIP to score quality, scene and distortion from 495 joint text labels.
• Self-regulating Prompts: Foundational Model Adaptation without Forgetting: trains CLIP prompts with feature and logit agreement terms against the frozen model, and epoch-level prompt averaging.
• CLIPstyler: Image Style Transfer with a Single Text Condition: trains a small U-Net per image against directional and patch-wise CLIP losses, for style transfer from text alone.
• A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model: pairs a MaskFormer trained on seen classes with a frozen CLIP that labels each masked crop.
• Exploring Visual Prompts for Adapting Large-Scale Models: learns one additive input prompt for a frozen model, trained with cross-entropy.
• Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts: a sparse mixture-of-experts transformer on images and text, with a contrastive loss and routing losses.
• Towards artificial general intelligence via a multimodal foundation model: BriVL, a two-tower image-text model with momentum queues, trained on 650M weakly correlated Chinese web pairs.
• CLAP: Isolating Content from Style through Contrastive Learning with Augmented Prompts: trains a residual MLP on frozen CLIP text features so that style words are ignored, and applies it to image and text features.