--- kind: concept slug: image-captioning title: Image captioning summary: >- Four ways to generate captions: a chest x-ray report decoder with co-attention over regions and tags; a transformer captioner pre-trained on relaxed web alt-text; BLIP's captioner that relabels noisy web images; and ClipCap, which maps a CLIP embedding to a GPT-2 prefix. All numbers are author-reported and not comparable across papers. tags: [vision-language, captioning, attention, pre-training, web-data, medical-imaging, prefix-tuning] papers: ["1711.08195", "2102.08981", "2201.12086", "2111.09734"] related: [attention-mechanism, convolutional-recurrent-network, vision-language-pretraining, dataset-curation, nucleus-sampling, decoding-strategies, multimodal-language-model, parameter-efficient-fine-tuning] status: draft updated_at: 2026-10-19 --- ## Overview Image captioning produces text that describes an image. Most designs on this page share a core: an image encoder that yields features, a decoder that writes the text one token at a time, and a way for the decoder to read the image at each step, either attention or cross-attention [arxiv:1711.08195 §3.3] [arxiv:2102.08981 §3.1.1] [arxiv:2201.12086 §3.1]. ClipCap is the exception: it gives the image to a language model as input vectors placed before the text, so the language model reads the image as a prefix [arxiv:2111.09734 §3.1]. The four cards on this page describe four uses of these ideas: - a chest x-ray report generator, which writes several sentences per image [arxiv:1711.08195 §1]; - a captioner trained as a pre-training objective on large web image-text data, then fine-tuned [arxiv:2102.08981 §3.1.1]; - a captioner that writes new training captions for noisy web images, used together with a filter [arxiv:2201.12086 §3.3]; - a small captioner built from pretrained CLIP and GPT-2 models, trained on image-caption pairs alone [arxiv:2111.09734 §1] [arxiv:2111.09734 §3.2]. Evidence is author-reported throughout. The medical paper runs on two small datasets against re-implemented baselines, and its metrics count overlapping words, not whether the findings are correct [arxiv:1711.08195 §4.1] [§4.4]. The other three papers report results on nocaps, a benchmark of objects that the training captions rarely name, and they use the same kind of n-gram metrics (CIDEr, SPICE) [arxiv:2102.08981 §3.1.2] [arxiv:2201.12086 tab. 7] [arxiv:2111.09734 tab. 1]. None of the four cards reports a check of the captions' factual accuracy beyond those metrics. ## Variants ### Two-level report decoder with co-attention The encoder is a VGG-19 network. Its last convolutional map gives 196 region features, and its last fully connected layers predict a set of medical tags [arxiv:1711.08195 §3.2]. The ten most likely tags are embedded and attended to together with the regions, at every sentence step [arxiv:1711.08195 §3.3]. A sentence LSTM emits one topic vector per sentence and a stop/continue signal. A word LSTM then writes each sentence from its topic vector [arxiv:1711.08195 §3.4] [arxiv:1711.08195 §3.5]. Co-attention on both regions and tags scored highest on IU X-Ray, with BLEU-4 0.247 and CIDEr 0.327 against 0.118 and 0.302 for a re-implemented soft-attention captioner [arxiv:1711.08195 tab. 1]. Card: [Automatic Generation of Medical Imaging Reports](/papers/1711.08195). The paper also compares attention variants. Using only the region attention gave BLEU-4 0.238 on IU X-Ray, and using only the tag attention gave 0.230 [arxiv:1711.08195 tab. 1]. Both are below the combined model's 0.247, but the margins are small for single runs. On PEIR Gross, one-sentence captions, the paper's hierarchical decoder without attention scores about the same as the plain CNN-RNN baseline. The gain on that dataset comes from attention [arxiv:1711.08195 tab. 1]. ### Captioner pre-trained on relaxed web alt-text The captioner is a six-layer transformer encoder-decoder with hidden size 1024 and eight heads. Its visual input is one global feature plus features for 16 detected boxes, with up to 16 cloud-API tags added as text [arxiv:2102.08981 §3.3] [arxiv:2102.08981 app. E.1] [arxiv:2102.08981 app. E.2]. It is trained with cross-entropy to predict the caption [arxiv:2102.08981 §3.1.1]. The pre-training data is CC12M, about 12.4 million image and alt-text pairs. Its filters are looser than those of Conceptual Captions 3M, which gives about four times more pairs and more rare words. The price is precision: a 100-example check gave 76.6% for CC12M against 90.3% for CC3M, and the CC3M figure is taken from the CC3M paper [arxiv:2102.08981 §2.2] [arxiv:2102.08981 §2.3] [arxiv:2102.08981 tab. 1]. Results come after fine-tuning on COCO. Overall nocaps val CIDEr is 87.4, against 54.7 with no pre-training, and 90.2 when CC3M is added as well [arxiv:2102.08981 tab. 3]. Out-of-domain CIDEr is 91.3, against 65.3 for a CC3M-pre-trained model with the same fine-tuning [arxiv:2102.08981 tab. 3]. Without fine-tuning, CC12M is weaker than CC3M on nocaps (27.1 against 29.7 overall CIDEr) [arxiv:2102.08981 tab. 3]. Card: [Conceptual 12M](/papers/2102.08981). The repository holds the image URLs and captions only, with no model code [arxiv:2102.08981 §3.3] [code: README.md#L16-L23]. The feature extractors are not released, so the exact architecture cannot be rebuilt from the repository. ### Captioner that relabels web images (CapFilt) BLIP uses one encoder-decoder transformer in three modes: a unimodal encoder, an image-grounded encoder for matching, and an image-grounded decoder for captioning [arxiv:2201.12086 §3.1]. In CapFilt, the decoder is fine-tuned on COCO and writes one synthetic caption per web image, sampled with nucleus sampling at p = 0.9 [arxiv:2201.12086 §3.3] [arxiv:2201.12086 §4.3]. The matching head then drops any text it judges unmatched, applied to both the original web text and the synthetic caption. The surviving pairs are merged with human-annotated pairs for a new pre-training run [arxiv:2201.12086 §3.3]. Results come from ablations and fine-tuned tables. Zero-shot NoCaps CIDEr is 102.2 with neither module and 105.1 with both [arxiv:2201.12086 tab. 1]. For the synthetic captions, nucleus sampling gave COCO fine-tuned CIDEr 129.7 at a 25% noise ratio, against 128.9 for beam search at 19% [arxiv:2201.12086 tab. 2]. Fine-tuned NoCaps CIDEr is 105.1 for the 14M-image setting, 109.6 for CapFilt-L on 129M images, and 113.2 with a ViT-L backbone [arxiv:2201.12086 tab. 7]. The paper describes one bootstrapping round [arxiv:2201.12086 §3.3]. Card: [BLIP](/papers/2201.12086). The captioner's own fine-tuning settings (learning rate, epochs, batch size) are not stated, and the repository has no captioner or filter training script [arxiv:2201.12086 §3.3] [code: README.md#L100-L105]. The CapFilt numbers cannot be reproduced exactly from the paper or the repository. ### Frozen CLIP embedding as a GPT-2 prefix (ClipCap) A frozen CLIP image encoder gives one embedding per image. A mapping network turns that embedding into k vectors, each the size of a word embedding. These vectors are placed before the caption's word embeddings, and the language model is trained with cross-entropy on the caption tokens only [arxiv:2111.09734 §3.1] [arxiv:2111.09734 eq. 5]. No cross-attention is needed, because the image enters as input. The paper gives two mappers [arxiv:2111.09734 §3.2] [arxiv:2111.09734 §3.3]: - **MLP mapper, GPT-2 fine-tuned.** One hidden layer, trained together with the language model. - **Transformer mapper, GPT-2 frozen.** Only the mapper trains. It is a transformer over the CLIP embedding and a set of learned constant inputs. The paper does not say which outputs form the prefix; the code takes the outputs at the constant positions [code: train.py#L203-L210]. Author-reported results on Conceptual Captions, COCO and nocaps [arxiv:2111.09734 tab. 1]: on Conceptual Captions the MLP variant reaches CIDEr 87.26, against 77.57 for VLP and 71.82 for the transformer variant. On nocaps overall, the transformer variant reaches 65.83 and the MLP variant 65.7, against 63.8 for Oscar, which uses object tags [arxiv:2111.09734 §4]. On COCO, Oscar reaches 124.12 CIDEr, and the two ClipCap variants reach 113.08 (transformer) and 108.35 (MLP). Parameters are 156M for the MLP variant and 43M for the transformer variant. The paper's training times are 6 to 7 hours on one GTX 1080 in the COCO and nocaps rows, and 72 to 80 hours in the Conceptual Captions rows [arxiv:2111.09734 tab. 1]. Card: [ClipCap](/papers/2111.09734). ## When to use which | Situation | Use | Why | |---|---|---| | Output is several sentences per image, such as a report | Two-level decoder: sentence LSTM plus word LSTM, with a stop signal | The paper's multi-sentence setup, and its BLEU-4 and CIDEr gains over the re-implemented captioning baselines on IU X-Ray [arxiv:1711.08195 §1] [arxiv:1711.08195 tab. 1] | | Output is one sentence per image | Attention over regions; a two-level decoder alone is not enough | On PEIR Gross the no-attention variant matched the CNN-RNN baseline [arxiv:1711.08195 tab. 1] | | Labelled tags exist for each image | Co-attention over regions and tags | Co-attention had the higher BLEU-4 and CIDEr than either single-attention variant on both datasets [arxiv:1711.08195 tab. 1] | | No tag vocabulary can be built | Not covered by this paper | The tag branch needs tags. The paper derives them from MTI annotations or tf-idf and does not say whether the method works without them [arxiv:1711.08195 §4.1] | | Large web alt-text is available and a fine-tuning budget exists | Pre-train on relaxed web pairs, then fine-tune on curated captions | Fine-tuned nocaps CIDEr rose from 54.7 to 87.4 [arxiv:2102.08981 tab. 3]. Zero-shot results were lower than with CC3M, so budget for fine-tuning [arxiv:2102.08981 tab. 3] | | Web pairs are noisy and a captioner and matching filter can be trained on clean pairs | Caption relabelling plus matching filter (CapFilt) | Zero-shot NoCaps CIDEr rose from 102.2 to 105.1 in the authors' ablation [arxiv:2201.12086 tab. 1]. The captioner's training settings are not stated [arxiv:2201.12086 §3.3] | | One model must serve captioning, retrieval and VQA | BLIP's three-mode model | The same pre-trained weights are fine-tuned for each task [arxiv:2201.12086 §3.1] [arxiv:2201.12086 tab. 5] [arxiv:2201.12086 tab. 7] | | Only image-caption pairs are available, and the budget is one GPU and a few hours | ClipCap, transformer mapper with GPT-2 frozen | Only the mapper trains. The paper needs only images and captions [arxiv:2111.09734 §1], and reports nocaps overall CIDEr 65.83 and 6 hours on a GTX 1080 [arxiv:2111.09734 tab. 1] | | Same kind of data, and a higher Conceptual Captions score matters more than model size | ClipCap, MLP mapper with GPT-2 fine-tuned | CIDEr 87.26 against 71.82 for the transformer variant on Conceptual Captions, at 156M parameters and 80 hours [arxiv:2111.09734 tab. 1] | | COCO scores are the target | Not ClipCap alone; compare with tag-based captioners | On COCO, ClipCap trails the tag-based Oscar baseline (113.08 and 108.35 against 124.12) [arxiv:2111.09734 tab. 1] [arxiv:2111.09734 §4]. Oscar has no card here | | Latency or compute budget | Not stated for inference | No card reports inference speed. ClipCap reports training hours on one GTX 1080, and its baseline times were measured on other hardware, so they are not comparable [arxiv:2111.09734 tab. 1]. CC12M's pre-training cost is not stated [arxiv:2102.08981 §2] [arxiv:2102.08981 app. E.3]. BLIP's pre-training ran on two 16-GPU nodes [arxiv:2201.12086 §4.1] | ## Implementation sketch The sketch covers the attention-based variants: encode the image, attend at each step, decode. ClipCap follows a different pattern, noted after the sketch. ```pseudocode # Encode the image once into region features [§3.2] def encode(image): regions = cnn_last_conv(image) # (196, 512) tag_scores = tag_head(image) # (L,), softmax over tags return regions, tag_scores # Attend over regions and tag embeddings at one decoding step [§3.3] def co_attend(regions, tag_feats, h_prev): alpha = softmax(score(regions, h_prev)) # over regions beta = softmax(score(tag_feats, h_prev)) # over tags return fuse(weighted(alpha, regions), weighted(beta, tag_feats)) # Each step: produce a topic, decide whether to stop, then write words [§3.4] [§3.5] def decode(regions, tag_feats, max_sentences): h, c = initial_state() # not stated in the paper report = [] for _ in range(max_sentences): ctx = co_attend(regions, tag_feats, h) h, c = sentence_lstm(ctx, h, c) if stop_probability(h) > 0.5: # threshold per [§4.2] break report.append(write_sentence(topic_from(h, ctx))) return report ``` Where the other variants differ from this sketch: - **Report decoder:** the sketch above. The paper trains with a sum of tag, sentence, word and attention-coverage losses, all weighted 1.0 [arxiv:1711.08195 §3.6] [arxiv:1711.08195 §4.2]. It does not state a maximum sentence count for inference, so set one when implementing [arxiv:1711.08195 §3.4]. - **CC12M captioner:** the decoder cross-attends to the region features and tag embeddings in one encoder input, and is trained with per-token cross-entropy against the caption [arxiv:2102.08981 §3.1.1] [arxiv:2102.08981 app. E.2]. The paper uses beam width 5 and a maximum of 36 output tokens [arxiv:2102.08981 app. E.2]. - **BLIP captioner:** the decoder uses causal self-attention and cross-attends to the image. The language-modelling loss uses label smoothing of 0.1 [arxiv:2201.12086 §3.1] [arxiv:2201.12086 §3.2]. For data generation it samples with nucleus sampling at p = 0.9. For captioning evaluation it uses beam 3 and a maximum length of 20 [arxiv:2201.12086 §4.3] [arxiv:2201.12086 app. A]. - **ClipCap:** no attention step. The image becomes k prefix vectors, placed before the caption embeddings, and the loss covers the caption tokens only [arxiv:2111.09734 §3.1] [arxiv:2111.09734 §4]. The paper's settings are a prefix length of 10, AdamW, a learning rate of 2e-5 with 5000 warm-up steps, batch size 40 and 10 epochs [arxiv:2111.09734 §4]. Decoding is greedy or beam search, and the paper gives no maximum length [arxiv:2111.09734 §3.4]. The paper says GPT-2 large, but the code loads the small `gpt2` model, and the code decays the learning rate linearly over the run, which the paper does not state [code: train.py#L240] [code: train.py#L303-L305]. ## Contradictions No direct numerical conflict between the cards. Four points need care: - **Data strategy.** CC12M keeps relaxed web pairs and relies on fine-tuning. Its zero-shot results are below CC3M's [arxiv:2102.08981 tab. 3]. BLIP filters and relabels web pairs and reports gains already in zero-shot NoCaps CIDEr [arxiv:2201.12086 tab. 1]. The two use different models, data and evaluation settings, so these cards do not show which route is better. - **nocaps numbers across papers.** CC12M reports 87.4 after COCO fine-tuning [arxiv:2102.08981 tab. 3], BLIP reports 105.1 for its 14M-image setting [arxiv:2201.12086 tab. 7], and ClipCap reports 65.83 overall [arxiv:2111.09734 tab. 1]. These are not comparable. The models differ, BLIP's 14M set includes about 10M pairs from Conceptual Captions 3M and Conceptual 12M combined [arxiv:2201.12086 §4.1], and the beam width and maximum length differ (5 and 36 against 3 and 20) [arxiv:2102.08981 app. E.2] [arxiv:2201.12086 app. A]. - **COCO and nocaps rankings.** In ClipCap's own table, the transformer variant is below Oscar on COCO (113.08 against 124.12) but above it on nocaps (65.83 against 63.8) [arxiv:2111.09734 tab. 1]. These cards do not explain the split, so one benchmark alone does not settle which design is better. - **Scale of the metrics.** The medical paper reports BLEU-4 and CIDEr as fractions, such as CIDEr 0.327 [arxiv:1711.08195 tab. 1]. The other three report CIDEr times 100, such as 87.4, 105.1 and 87.26 [arxiv:2102.08981 tab. 3] [arxiv:2201.12086 tab. 7] [arxiv:2111.09734 tab. 1]. Do not compare the numbers directly. ## Papers - [Automatic Generation of Medical Imaging Reports](/papers/1711.08195): a hierarchical co-attention decoder for chest x-ray reports. Compared against re-implemented CNN-RNN, LRCN, soft-attention and ATT-RK baselines, which have no cards here. - [Conceptual 12M](/papers/2102.08981): a relaxed-filter web dataset for pre-training a transformer captioner, with nocaps results after COCO fine-tuning. - [BLIP](/papers/2201.12086): one encoder-decoder model in three modes, with CapFilt for relabelling and filtering web captions. - [ClipCap](/papers/2111.09734): a frozen CLIP embedding mapped to a GPT-2 prefix, with an MLP mapper (GPT-2 tuned) and a transformer mapper (GPT-2 frozen). Compared against VLP, Oscar and BUTD, which have no cards here.