Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

In plain words

Models that learn to link pictures with text need many pictures that come with a caption. The web has plenty: the text that describes an image on a page. The usual dataset for this, Conceptual Captions, is strict. It throws away many candidate pairs so that each caption reads like a clean description, which is what a caption-writing model needs. This paper asks what happens if the filters are loosened. The result, Conceptual 12M, keeps about four times as many pairs. The captions are messier, and the words are less generalised, so a specific place or a rare object keeps its own name.

The payoff is coverage of rare things. Words for uncommon objects, places and foods show up far more often in this data, so a model trained on it has seen more of them. A captioning model pre-trained this way and then fine-tuned does better on a benchmark built around objects that standard caption sets rarely name. Raw data is noisier, so the gain shows up after fine-tuning, not when the pre-trained model is used directly.

TL;DR

• CC12M has 12,423,374 image and alt-text pairs, about 4x Conceptual Captions 3M (CC3M). It relaxes CC3M's unimodal filters and keeps its image-text overlap filter §2.2 tab. 1.
• Caption precision is lower: 76.6% against 90.3% for CC3M, from a 100-example check by two annotators §2.3.
• A transformer encoder-decoder pre-trained on CC12M with image captioning, then fine-tuned on COCO, scores 87.4 overall CIDEr on nocaps val, against 54.7 with no pre-training. Adding CC3M gives 90.2 tab. 3.
• Used without fine-tuning, CC12M is weaker than CC3M on nocaps (27.1 against 29.7 overall CIDEr), so the benefit is in fine-tuned models tab. 3.
• The repository ships the data as image URLs and captions, not the model code. The model, the feature extractors and the training recipe are only partly specified §3.3 app. E.1–E.3 README.md.

Problem

Vision-and-language (V+L) pre-training helps downstream tasks such as visual question answering, retrieval and captioning §1. The image-text sets it uses are either small, from under a few hundred thousand images up to a few million, or drawn from the same COCO images §1. CC3M is large, about 3.3M pairs, but its filters keep only captions that suit captioning §1 §2.2. The authors argue that this cut is too strict for pre-training and loses long-tail concepts, meaning rare visual categories §1 §2.2.

The benchmark that tests this is nocaps. Its training images come from COCO, with 80 object classes, and its evaluation images come from Open Images, with 600 classes, so models must describe objects they never saw in captions §3.1.2. The paper does not state the cost of collecting or training on CC12M §2 app. E.3.

Method

Data collection. CC12M follows the CC3M pipeline §2.1 and changes the unimodal filters §2.2. The paper's changes are listed below, and the code block shows them as checks.

1. Image filter: JPEG only, with a larger-to-smaller side ratio up to 2.5 (CC3M: 2) and a size threshold of more than 400 pixels. Images that trigger a pornography detector are still excluded §2.2.
2. Text filter: 3 to 256 words. A caption still needs a noun and a determiner, but a preposition is no longer required. The unique-word, part-of-speech and capitalisation heuristics are dropped. Maximum word repetition is 0.2. The rare-word threshold rises from 5 to 20 §2.2.
3. Image-text filter: kept as in CC3M, the overlap between the Google Cloud Vision tags and the caption words §2.2 §2.1.
4. Text transform: no hypernymisation and no digit replacement. Only person names are replaced by the token <PERSON>, detected with the Google Cloud Natural Language API. About 25% of captions change this way §2.2.

Pre-training objectives. Two objectives are used, one at a time §3.

• Image captioning (IC): predict the caption from the image features with a standard cross-entropy loss against the ground-truth caption §3.1.1.
• Visual-linguistic matching (VLM): image and text features go into the same model. A contrastive softmax loss treats the original pair as positive and every other pair in the mini-batch as negative. The paper says mean pooling gives one vector per modality, and the product of the transformed image and text vectors goes into the softmax §3.2.1 §3.3.

The main comparison uses IC. The authors found it clearly better than masked objectives at low masking rates and competitive with the best masked setting they tried app. D.2.

Model. Visual input is one global Graph-RISE feature plus Graph-RISE features for the top 16 Faster R-CNN box proposals, giving 1 + 16 vectors. Up to 16 Google Cloud Vision tags are added as text inputs. Each box gets five geometric numbers: two corners and the relative area §3.3 app. E.1. The architecture is a six-layer transformer encoder-decoder with hidden size 1024, 8 heads and feed-forward size 4096 app. E.2. Its starting point is the captioner of Changpinyo et al. (2019), and the paper adds a deeper embedding layer, layer normalisation and the geometric features §3.3. The VLM model reuses the architecture without the decoder and adds a text encoder app. E.2.

Decoding. Beam search with width 5 and alpha 0.6, whose exact meaning is not stated. Maximum output length is 36 tokens, or 180 for Localized Narratives app. E.2.

# Relaxed unimodal filters that build CC12M  [§2.2]
# Input:  one candidate (image, alt_text) pair from the web
# Output: True if the pair is kept

def keep_pair(image, alt_text):
    # Image: JPEG only, size above 400 px (dimension not stated)  [§2.2]
    if not is_jpeg(image) or not larger_than(image, 400):
        return False
    # Image: long side at most 2.5 times the short side  [§2.2]
    if aspect_ratio(image) > 2.5:
        return False
    # Image: drop images that the pornography detector flags  [§2.2]
    if is_pornographic(image):
        return False
    words = tokenize(alt_text)
    # Text: between 3 and 256 words  [§2.2]
    if not 3 <= len(words) <= 256:
        return False
    # Text: needs a noun and a determiner, prepositions not required  [§2.2]
    if not (has_noun(words) and has_determiner(words)):
        return False
    # Text: word repetition fraction at most 0.2  [§2.2]
    if repetition_fraction(words) > 0.2:
        return False
    # Image-text: CC3M tag-overlap rule, kept unchanged  [§2.2] [§2.1]
    if not tags_overlap(image, alt_text):
        return False
    return True

# Person-name substitution for CC12M captions  [§2.2]
# Input:  alt_text (str)
# Output: caption with person names replaced

def hide_people(alt_text):
    # Named entities of type Person become one token  [§2.2]
    return replace_entities(alt_text, "Person", "<PERSON>")

# One image captioning (IC) training step  [§3.1.1] [§3.3] [app. E.2]
# Input:  image features (B, 17, 512), tag features (B, 16, 512),
#         caption token ids (B, T)
# Output: scalar cross-entropy loss

def ic_training_step(image_feats, tag_feats, caption_tokens):
    # Permute a set of input visual regions as augmentation; the
    # permuted set is not specified  [app. E.3]
    image_feats = permute_regions(image_feats)
    # Image vectors and tags are concatenated as one encoder input  [§3.3]
    encoder_input = concat(image_feats, tag_feats)
    memory = encoder(encoder_input)
    # Decoder predicts the caption; the input/target shift is not stated  [§3.1.1]
    logits = decoder(caption_tokens, memory)
    # Cross-entropy against the ground-truth caption  [§3.1.1]
    return cross_entropy(logits, caption_tokens)

# Visual-linguistic matching (VLM) loss for one mini-batch  [§3.2.1] [§3.3]
# Input:  image features (B, 17, 512), text features (B, T, 512)
# Output: scalar contrastive softmax loss

def vlm_loss(image_feats, text_feats):
    # Mean-pool each modality into one vector  [§3.3]
    image_vecs = mean_pool(image_encoder(image_feats))  # (B, H), H = 1024
    text_vecs = mean_pool(text_encoder(text_feats))     # (B, H)
    # Transform each side, then take all pairwise products  [§3.3]
    # The normalisation and temperature are not stated
    scores = transform_image(image_vecs) @ transform_text(text_vecs).T
    # Diagonal pairs are the positives; other batch pairs are negatives  [§3.3]
    return softmax_cross_entropy(scores, arange(B))

Hyperparameters

Results

Author-reported. Setup: pre-training on CC12M, CC3M or both with the IC objective, then fine-tuning on COCO for nocaps and CC3M for the CC benchmark §3.1.2 §4.1 app. E.5. Comparisons give similar resources to each setting app. E.5. nocaps test numbers come from the official evaluation server, and val numbers use the nocaps metric code app. E.4.

nocaps val, own runs tab. 3. Overall and out-of-domain scores.

nocaps, comparison with published models tab. 4. Overall CIDEr and SPICE.

The authors say their gain is mostly on out-of-domain objects, and CBS is not used §4.1. The "Human" row and the listed methods come from the paper's table and are not re-run here.

COCO and CC3M benchmarks tab. 5 tab. 7. CIDEr.

Localized Narratives tab. 6. CIDEr for COCO-trained captioning on Localized Narratives, with Open Images (OID) as the held-out set.

Image retrieval on Flickr30k tab. 8. Recall at 1, 5 and 10, test split.

On Localized Narratives Flickr, CC12M alone gives R@1 70.2 (CC3M 61.1, none 54.5) and CC3M + CC12M gives 71.0 tab. 8. The paper says its smaller model's zero-shot results beat ViLBERT's reported 31.9 R@1 on Flickr30k, with the CC3M-only model at 35.4 §4.2. ViLBERT has no card here.

Data statistics and precision tab. 1 §2.3. CC12M has 12,423,374 pairs against 3,318,333 in CC3M. Token-to-type ratio is 370.0 against 804.8, and caption length is 20.2 ± 16.3 against 10.3 ± 4.5. Estimated caption precision is 76.6% against 90.3% for CC3M, which is taken from the CC3M paper. Words that CC3M rarely keeps appear often in CC12M. Examples, counted in CC3M and then CC12M: mangosteen 0 → 212, durian 61 → 552, quiche 50 → 1109, and pokemon 1 → 8615 §2.3. Person names become the <PERSON> token, so the word "person" is much more frequent in CC12M §2.3.

Appendix results app. B app. D. Out-of-domain coverage, using nocaps' 394 out-of-domain classes as a proxy, is higher for CC12M than for any dataset compared tab. 9:

Pre-training on Localized Narratives OID gives overall CIDEr 63.9 on nocaps val, below CC3M at 73.2 and CC12M at 87.4 tab. 10. Masked pre-training reaches at most 88.5 at the highest rate, against 87.4 for IC tab. 11. Adding masked object classification to IC gives 89.9, and adding VLM gives 87.1 tab. 12. Pre-training on a 25% or 50% subsample of CC12M reaches CC3M-level results at 25%, shown as curves only fig. 6.

Paper vs code

The repository is the official one: the paper's abstract footnote links it, and it is owned by the Google Research dataset organisation [abs.]. It contains a README, a LICENSE and an images folder, and no model or training code, so the architecture and training steps above come from the paper only (checked at code_commit). The README gives the format: tab-separated image URL and caption README.md, plus a companion file of SHA256 and MD5 hashes for each image README.md. It states that the dataset is about 12 million pairs README.md. The paper gives 12,423,374 tab. 1.

The README says the authors do not provide image pixels, and some images will disappear over time README.md. The README says CC12M was designed to be disjoint from CC3M, with an intersection of about 63K URLs README.md. The paper says CC3M images are not included in its own experiments §3. No conflict on the numbers, but users who train on both must remove the overlap themselves.

Licence: the paper is CC BY 4.0. The repository's dataset licence is a Google terms file: free use for any purpose, attribution requested, no warranty LICENSE. It is not an SPDX licence, and it covers the pairs list, not the images, which belong to their owners.

Pitfalls

• No model code. The model is described in words and tables only. It starts from Changpinyo et al. (2019) §3.3, which has no card here, so do not borrow its details.
• Feature extractors are not reproducible from the paper. Graph-RISE, the Faster R-CNN trained on Visual Genome, and the Google Cloud Vision tags are not released or versioned. The paper gives no weights for them §3.3 §2.2.
• Dataset build depends on paid APIs. Person names and tags come from Google Cloud Natural Language and Vision APIs at extraction time §2.2. The API versions are not stated, so re-extraction will not match the released list.
• Image filter threshold. "Size greater than 400 pixels" does not say which dimension, and the image-text overlap rule is kept from CC3M without its parameters §2.2 §2.1.
• Text filters left open. CC3M's language and adult-content text checks are mentioned as part of CC3M §2.1. The paper does not say whether CC12M keeps them §2.2.
• Precision figures are rough. The 76.6% figure is from 100 captions judged by two annotators on a 1-to-5 scale. Precision means a score of 4 or more §2.3. Treat the gap to CC3M as an estimate, not a measurement.
• Caption and text disagree on the best model. The Table 4 caption says the italic best model is pre-trained on CC12M, but the text and bold scores point to CC3M + CC12M tab. 4 §4.1.
• Epochs versus steps. Warm-up and decay are given in epochs app. E.3, but the training budget is given in steps. The epoch length is not stated. At batch size 4096 (derived here, not stated), one pass over 12.4M pairs is about 3,034 steps, so 2M steps is about 660 passes if the same batch applies.
• Tuning is narrow. Only the learning rate is searched, and pre-training uses a fixed 3.2e-5 app. E.5. The fine-tuning step count for nocaps was tuned lightly, because the model overfits COCO app. E.5 §4.1.
• Matching loss details. The VLM temperature, normalisation and direction of the softmax are not stated §3.3. The number of VLM pre-training steps on CC12M is not stated app. E.3.
• Data augmentation is vague. "A set of input visual regions are permuted" app. E.3 does not say which regions, or how often.
• Zero-shot is weaker. On nocaps, zero-shot CC12M is below zero-shot CC3M tab. 3. Advantages show after fine-tuning.
• Leaderboard split. Models that use Localized Narratives for pre-training fall under a different nocaps leaderboard. CC-only models are on the default one app. E.4.
• Person-name token. About a quarter of captions contain <PERSON>, which changes word statistics, such as "person" becoming frequent §2.2 §2.3. The tokenizer must treat it as one unit.
• Bias check is preliminary. The authors found no large bias for sensitive terms or domains, and they call the check preliminary §2.3.
• Figure and table numbers are from source order. Numbers follow the order of LaTeX environments in the arXiv source, with appendix numbers continuing. They were not checked against the compiled PDF. Cite the section if a number does not match a reader's copy.
• Data decays. Image URLs disappear over time README.md. The paper's extraction is from May 2020 §2.3, so a present-day download is a different set.

Lineage

• Builds on: Conceptual Captions 3M (Sharma et al., 2018; cited as [cc3m] in the paper). CC12M reuses its pipeline and relaxes its unimodal filters §2.1 §2.2. No arXiv id is recorded on this card for it.
• Compared against: CC3M and Localized Narratives OID as pre-training sources tab. 6 tab. 10. Published nocaps models: UpDown, Oscar-L, VIVO tab. 4. None has a card here.
• Related card: LAION-400M, a later large web image-text dataset. This paper does not discuss it.
• Superseded by: not known from this wiki.