Image captioning

Overview

Image captioning produces text that describes an image. Most designs on this page share a core: an image encoder that yields features, a decoder that writes the text one token at a time, and a way for the decoder to read the image at each step, either attention or cross-attention 1711.08195 §3.3 2102.08981 §3.1.1 2201.12086 §3.1. ClipCap is the exception: it gives the image to a language model as input vectors placed before the text, so the language model reads the image as a prefix 2111.09734 §3.1.

The four cards on this page describe four uses of these ideas:

• a chest x-ray report generator, which writes several sentences per image 1711.08195 §1;
• a captioner trained as a pre-training objective on large web image-text data, then fine-tuned 2102.08981 §3.1.1;
• a captioner that writes new training captions for noisy web images, used together with a filter 2201.12086 §3.3;
• a small captioner built from pretrained CLIP and GPT-2 models, trained on image-caption pairs alone 2111.09734 §1 2111.09734 §3.2.

Evidence is author-reported throughout. The medical paper runs on two small datasets against re-implemented baselines, and its metrics count overlapping words, not whether the findings are correct 1711.08195 §4.1 §4.4. The other three papers report results on nocaps, a benchmark of objects that the training captions rarely name, and they use the same kind of n-gram metrics (CIDEr, SPICE) 2102.08981 §3.1.2 2201.12086 tab. 7 2111.09734 tab. 1. None of the four cards reports a check of the captions' factual accuracy beyond those metrics.

Variants

Two-level report decoder with co-attention

The encoder is a VGG-19 network. Its last convolutional map gives 196 region features, and its last fully connected layers predict a set of medical tags 1711.08195 §3.2. The ten most likely tags are embedded and attended to together with the regions, at every sentence step 1711.08195 §3.3.

A sentence LSTM emits one topic vector per sentence and a stop/continue signal. A word LSTM then writes each sentence from its topic vector [arxiv:1711.08195 §3.4] 1711.08195 §3.5. Co-attention on both regions and tags scored highest on IU X-Ray, with BLEU-4 0.247 and CIDEr 0.327 against 0.118 and 0.302 for a re-implemented soft-attention captioner 1711.08195 tab. 1. Card: Automatic Generation of Medical Imaging Reports.

The paper also compares attention variants. Using only the region attention gave BLEU-4 0.238 on IU X-Ray, and using only the tag attention gave 0.230 1711.08195 tab. 1. Both are below the combined model's 0.247, but the margins are small for single runs.

On PEIR Gross, one-sentence captions, the paper's hierarchical decoder without attention scores about the same as the plain CNN-RNN baseline. The gain on that dataset comes from attention 1711.08195 tab. 1.

Captioner pre-trained on relaxed web alt-text

The captioner is a six-layer transformer encoder-decoder with hidden size 1024 and eight heads. Its visual input is one global feature plus features for 16 detected boxes, with up to 16 cloud-API tags added as text [arxiv:2102.08981 §3.3] 2102.08981 app. E.1 2102.08981 app. E.2. It is trained with cross-entropy to predict the caption 2102.08981 §3.1.1.

The pre-training data is CC12M, about 12.4 million image and alt-text pairs. Its filters are looser than those of Conceptual Captions 3M, which gives about four times more pairs and more rare words. The price is precision: a 100-example check gave 76.6% for CC12M against 90.3% for CC3M, and the CC3M figure is taken from the CC3M paper 2102.08981 §2.2 [arxiv:2102.08981 §2.3] 2102.08981 tab. 1.

Results come after fine-tuning on COCO. Overall nocaps val CIDEr is 87.4, against 54.7 with no pre-training, and 90.2 when CC3M is added as well 2102.08981 tab. 3. Out-of-domain CIDEr is 91.3, against 65.3 for a CC3M-pre-trained model with the same fine-tuning 2102.08981 tab. 3. Without fine-tuning, CC12M is weaker than CC3M on nocaps (27.1 against 29.7 overall CIDEr) 2102.08981 tab. 3. Card: Conceptual 12M.

The repository holds the image URLs and captions only, with no model code 2102.08981 §3.3 README.md. The feature extractors are not released, so the exact architecture cannot be rebuilt from the repository.

Captioner that relabels web images (CapFilt)

BLIP uses one encoder-decoder transformer in three modes: a unimodal encoder, an image-grounded encoder for matching, and an image-grounded decoder for captioning 2201.12086 §3.1. In CapFilt, the decoder is fine-tuned on COCO and writes one synthetic caption per web image, sampled with nucleus sampling at p = 0.9 2201.12086 §3.3 2201.12086 §4.3. The matching head then drops any text it judges unmatched, applied to both the original web text and the synthetic caption. The surviving pairs are merged with human-annotated pairs for a new pre-training run 2201.12086 §3.3.

Results come from ablations and fine-tuned tables. Zero-shot NoCaps CIDEr is 102.2 with neither module and 105.1 with both 2201.12086 tab. 1. For the synthetic captions, nucleus sampling gave COCO fine-tuned CIDEr 129.7 at a 25% noise ratio, against 128.9 for beam search at 19% [arxiv:2201.12086 tab. 2]. Fine-tuned NoCaps CIDEr is 105.1 for the 14M-image setting, 109.6 for CapFilt-L on 129M images, and 113.2 with a ViT-L backbone [arxiv:2201.12086 tab. 7]. The paper describes one bootstrapping round 2201.12086 §3.3. Card: BLIP.

The captioner's own fine-tuning settings (learning rate, epochs, batch size) are not stated, and the repository has no captioner or filter training script 2201.12086 §3.3 README.md. The CapFilt numbers cannot be reproduced exactly from the paper or the repository.

Frozen CLIP embedding as a GPT-2 prefix (ClipCap)

A frozen CLIP image encoder gives one embedding per image. A mapping network turns that embedding into k vectors, each the size of a word embedding. These vectors are placed before the caption's word embeddings, and the language model is trained with cross-entropy on the caption tokens only [arxiv:2111.09734 §3.1] 2111.09734 eq. 5. No cross-attention is needed, because the image enters as input.

The paper gives two mappers 2111.09734 §3.2 2111.09734 §3.3:

• MLP mapper, GPT-2 fine-tuned. One hidden layer, trained together with the language model.
• Transformer mapper, GPT-2 frozen. Only the mapper trains. It is a transformer over the CLIP embedding and a set of learned constant inputs. The paper does not say which outputs form the prefix; the code takes the outputs at the constant positions train.py.

Author-reported results on Conceptual Captions, COCO and nocaps [arxiv:2111.09734 tab. 1]: on Conceptual Captions the MLP variant reaches CIDEr 87.26, against 77.57 for VLP and 71.82 for the transformer variant. On nocaps overall, the transformer variant reaches 65.83 and the MLP variant 65.7, against 63.8 for Oscar, which uses object tags 2111.09734 §4. On COCO, Oscar reaches 124.12 CIDEr, and the two ClipCap variants reach 113.08 (transformer) and 108.35 (MLP). Parameters are 156M for the MLP variant and 43M for the transformer variant. The paper's training times are 6 to 7 hours on one GTX 1080 in the COCO and nocaps rows, and 72 to 80 hours in the Conceptual Captions rows 2111.09734 tab. 1. Card: ClipCap.

When to use which

Implementation sketch

The sketch covers the attention-based variants: encode the image, attend at each step, decode. ClipCap follows a different pattern, noted after the sketch.

# Encode the image once into region features  [§3.2]
def encode(image):
    regions = cnn_last_conv(image)        # (196, 512)
    tag_scores = tag_head(image)          # (L,), softmax over tags
    return regions, tag_scores

# Attend over regions and tag embeddings at one decoding step  [§3.3]
def co_attend(regions, tag_feats, h_prev):
    alpha = softmax(score(regions, h_prev))   # over regions
    beta = softmax(score(tag_feats, h_prev))  # over tags
    return fuse(weighted(alpha, regions),
                weighted(beta, tag_feats))

# Each step: produce a topic, decide whether to stop, then write words  [§3.4] [§3.5]
def decode(regions, tag_feats, max_sentences):
    h, c = initial_state()    # not stated in the paper
    report = []
    for _ in range(max_sentences):
        ctx = co_attend(regions, tag_feats, h)
        h, c = sentence_lstm(ctx, h, c)
        if stop_probability(h) > 0.5:     # threshold per [§4.2]
            break
        report.append(write_sentence(topic_from(h, ctx)))
    return report

Where the other variants differ from this sketch:

• Report decoder: the sketch above. The paper trains with a sum of tag, sentence, word and attention-coverage losses, all weighted 1.0 [arxiv:1711.08195 §3.6] 1711.08195 §4.2. It does not state a maximum sentence count for inference, so set one when implementing 1711.08195 §3.4.
• CC12M captioner: the decoder cross-attends to the region features and tag embeddings in one encoder input, and is trained with per-token cross-entropy against the caption 2102.08981 §3.1.1 [arxiv:2102.08981 app. E.2]. The paper uses beam width 5 and a maximum of 36 output tokens 2102.08981 app. E.2.
• BLIP captioner: the decoder uses causal self-attention and cross-attends to the image. The language-modelling loss uses label smoothing of 0.1 2201.12086 §3.1 2201.12086 §3.2. For data generation it samples with nucleus sampling at p = 0.9. For captioning evaluation it uses beam 3 and a maximum length of 20 2201.12086 §4.3 [arxiv:2201.12086 app. A].
• ClipCap: no attention step. The image becomes k prefix vectors, placed before the caption embeddings, and the loss covers the caption tokens only 2111.09734 §3.1 2111.09734 §4. The paper's settings are a prefix length of 10, AdamW, a learning rate of 2e-5 with 5000 warm-up steps, batch size 40 and 10 epochs 2111.09734 §4. Decoding is greedy or beam search, and the paper gives no maximum length 2111.09734 §3.4. The paper says GPT-2 large, but the code loads the small gpt2 model, and the code decays the learning rate linearly over the run, which the paper does not state train.py train.py.

Contradictions

No direct numerical conflict between the cards. Four points need care:

• Data strategy. CC12M keeps relaxed web pairs and relies on fine-tuning. Its zero-shot results are below CC3M's 2102.08981 tab. 3. BLIP filters and relabels web pairs and reports gains already in zero-shot NoCaps CIDEr 2201.12086 tab. 1. The two use different models, data and evaluation settings, so these cards do not show which route is better.
• nocaps numbers across papers. CC12M reports 87.4 after COCO fine-tuning 2102.08981 tab. 3, BLIP reports 105.1 for its 14M-image setting 2201.12086 tab. 7, and ClipCap reports 65.83 overall [arxiv:2111.09734 tab. 1]. These are not comparable. The models differ, BLIP's 14M set includes about 10M pairs from Conceptual Captions 3M and Conceptual 12M combined 2201.12086 §4.1, and the beam width and maximum length differ (5 and 36 against 3 and 20) 2102.08981 app. E.2 2201.12086 app. A.
• COCO and nocaps rankings. In ClipCap's own table, the transformer variant is below Oscar on COCO (113.08 against 124.12) but above it on nocaps (65.83 against 63.8) 2111.09734 tab. 1. These cards do not explain the split, so one benchmark alone does not settle which design is better.
• Scale of the metrics. The medical paper reports BLEU-4 and CIDEr as fractions, such as CIDEr 0.327 1711.08195 tab. 1. The other three report CIDEr times 100, such as 87.4, 105.1 and 87.26 [arxiv:2102.08981 tab. 3] 2201.12086 tab. 7 2111.09734 tab. 1. Do not compare the numbers directly.

Papers

• Automatic Generation of Medical Imaging Reports: a hierarchical co-attention decoder for chest x-ray reports. Compared against re-implemented CNN-RNN, LRCN, soft-attention and ATT-RK baselines, which have no cards here.
• Conceptual 12M: a relaxed-filter web dataset for pre-training a transformer captioner, with nocaps results after COCO fine-tuning.
• BLIP: one encoder-decoder model in three modes, with CapFilt for relabelling and filtering web captions.
• ClipCap: a frozen CLIP embedding mapped to a GPT-2 prefix, with an MLP mapper (GPT-2 tuned) and a transformer mapper (GPT-2 frozen). Compared against VLP, Oscar and BUTD, which have no cards here.