DataComp: In search of the next generation of multimodal datasets

In plain words

Models that link pictures to words learn from huge piles of image-caption pairs, yet almost nobody studies how to pick those pairs. Most research changes the model and keeps the data fixed. This paper flips that. Everyone gets the same pool of 12.8 billion image-caption pairs from Common Crawl, and the same training recipe. You only change which pairs you pick, or add your own. A trained model is then tested on 38 different tasks.

The main finding is that careful picking beats raw size. Keeping the pairs whose pictures and captions a reference model already finds to match, and dropping pictures unlike a clean image collection, gives better models than training on everything. Smaller, stricter subsets can outperform larger ones. The authors' best subset, DataComp-1B, trains a model that beats OpenAI's CLIP at the same training cost.

TL;DR

DataComp is a dataset-design benchmark: fix the training code and compute, let participants choose or add the training data, and score the result on 38 tasks. It ships a 12.8B-pair candidate pool from Common Crawl, four compute scales, and filtering baselines. The best baseline combines an image-similarity filter with a CLIP-score filter that keeps the top 30%. Its result (79.2% ImageNet zero-shot, ViT-L/14) is author-reported. Relevant if you curate image-text data or want a controlled way to test a data-filtering idea.

Problem

Image-text models such as CLIP depend on very large paired datasets, but these datasets are seldom studied in their own right. Many are proprietary, and for public ones such as LAION-2B it is unclear how source and filtering choices affect the model §1 §2. Model research has standard benchmarks; dataset research did not. Comparing datasets of equal size is also flawed, because the final size is a design choice. What is really limited is the candidate pool to filter and the training compute, so both must be fixed §3.1.

Method

Benchmark structure

1. Fix the training recipe per scale: a CLIP-style contrastive model, one architecture per scale, and fixed optimiser settings §3.4 tab. 2 tab. 14.
2. Participants submit either a list of pool uids (filtering track) or a dataset built from any sources (Bring Your Own Data, BYOD) §3.1 §3.3.
3. Four scales (small, medium, large, xlarge). Each step up multiplies the number of samples seen and the pool size by 10, from 12.8M to 12.8B §3.1 tab. 2. The authors give 4 to 40,000 A100 GPU hours as the cost range §1.
4. Score the trained model zero-shot on 38 tasks (classification and retrieval) and average the preferred metric per task §3.5 app. Evaluation details. The 38 tasks are listed in tab. 15.
5. Three further test sets are kept secret for a year to limit overfitting §3.

Filtering-track rules, from app. Benchmark rules: the training procedure is fixed; no code may depend on evaluation test images, but class names and prompt templates may be used; and participants are encouraged to publish their uid lists and code.

Candidate pool (CommonPool) §3.2 app. Pool creation pipeline

The pool is an index of image URLs and alt-text, not images. It is released under CC-BY-4.0 §3.2. Construction:

1. Extract image URL and non-empty alt-text pairs from Common Crawl snapshots from 2014 to 2022 with cc2dataset, deduplicate, and shuffle. This gives about 88B candidate pairs §3.2.
2. Attempt to download about 40B of them with img2dataset, which yields about 16.8B image-text pairs §3.2 app. Parsing Common Crawl.
3. Safety filtering: remove text flagged by Detoxify (threshold 0.1) and images flagged by a classifier trained on CLIP ViT-L/14 features, using the NSFW set from LAION-5B. About 19% of pairs are removed, leaving about 13.6B §3.2 app. NSFW filtering app. Detoxify.
4. Remove evaluation-set near-duplicates with an image-similarity model at cosine threshold 0.604169. About 2.8% of downloads are flagged; the pool falls to about 13.1B §3.2 app. Deduplication against evaluation sets.
5. Blur faces: an SCRFD detector, keeping boxes with score at least 0.3, is used to locate them §3.2 app. Face blurring.
6. Draw successive random subsets for the smaller pools: xlarge is 12.8B; large, medium and small are nested subsets of it §3.2.
7. Publish per-sample metadata (URL, alt-text, original resolution, CLIP features and CLIP similarity scores) and SHA-256 hashes of images §3.2 app. Pool creation pipeline.

Training procedure §3.4 app. Training details

The image encoder is a ViT and the text encoder is the CLIP text tower. Training uses a symmetric contrastive loss over image-caption pairs with a learnable temperature, following CLIP §3.4. The code base is OpenCLIP §3.4. The footnote in §3.4 gives the loss; the pseudocode below reproduces it literally (see Pitfalls for the missing logarithm).

# Symmetric contrastive loss as written in the §3.4 footnote
# Input:  image features (B, d), caption features (B, d)
# Output: scalar loss

def contrastive_loss(image_features, text_features):
    # Pairwise scores sigma_ij = exp(<g(x_i), h(y_j)>)  [§3.4]
    sigma = exp(image_features @ text_features.T)  # (B, B)
    # Matched-pair share of each row (image to text)  [§3.4]
    image_to_text = diag(sigma) / sum(sigma, axis=1)
    # Matched-pair share of each column (text to image)  [§3.4]
    text_to_image = diag(sigma) / sum(sigma, axis=0)
    # Average the two directions  [§3.4]
    return 0.5 * sum(image_to_text) + 0.5 * sum(text_to_image)

Per scale, the run lasts for a fixed number of samples, equal to the pool size tab. 2. The paper describes the data budget as "two passes" and "four passes" over subsets §3.1. The released code instead resamples shards with replacement (see Paper vs code).

# One training run for a scale  [§3.4] [tab. 14]
# Input:  scale name; participant-chosen subset of pool uids
# Output: trained image encoder and text encoder

def train_scale(scale, subset_uids):
    # Model, learning rate, warmup, batch size and beta2 per scale  [tab. 14]
    config = scale_config(scale)
    # Shards are resampled with replacement  [code: train.py#L282]
    loader = resampled_loader(subset_uids, config.batch_size)
    # Total samples equal the pool size for this scale  [tab. 2]
    for step in range(config.total_samples // config.batch_size):
        # Schedule shape is not stated; warmup length is  [tab. 14]
        learning_rate = schedule(step, config.learning_rate, config.warmup)
        images, captions = next(loader)
        loss = contrastive_loss(image_encoder(images), text_encoder(captions))
        update(loss, learning_rate, config.beta2)
    return image_encoder, text_encoder

Filtering baselines §4.1 app. Filtering track

The paper studies six simple filters, plus combinations. Each keeps a subset of the pool. Text-based filters use fastText for language; image-based filters use CLIP image embeddings.

1. No filtering: the whole pool, one pass §4.1.
2. Random subsets of 1%, 10%, 25%, 50% and 75% §4.1.
3. Basic filter: English caption, more than two words, more than five characters, smaller image side at least 200 px and aspect ratio at most 3 §4.1 app. Filtering track. Language uses fastText lid.176.
4. CLIP-score filter: keep pairs whose cosine similarity from an OpenAI CLIP model (ViT-B/32 or ViT-L/14) exceeds a threshold, or keep the top fraction (e.g. 30%) §4.1 app. Filtering track tab. 16 app. Extra figures and tables.
5. LAION-2B filter: English (cld3) and ViT-B/32 score at least 0.28 §4.1 app. Filtering track.
6. Text-based filter: English captions containing a word whose first WordNet synset is an ImageNet-21K class §4.1 app. Filtering track.
7. Image-based filter: cluster CLIP L/14 image embeddings into 100,000 clusters. Keep pool samples whose nearest cluster is also the nearest cluster of some ImageNet example §4.1 app. Filtering track.
8. Intersection: image-based and CLIP-score (L/14, top 30%) together. This gives DataComp-1B §5.1.

The image-based filter in pseudocode:

# Image-based filter  [§4.1] [app. Filtering track]
# Input:  pool image embeddings (N, d), centroids (K, d) with K = 100000,
#         ImageNet-1k embeddings (M, d), mask from the caption filter (N,)
# Output: indices of kept pool samples

def image_based_filter(pool_embeddings, centroids, imagenet, mask):
    # Nearest centroid by inner product, per ImageNet example
    # [app. Filtering track]
    imagenet_clusters = argmax(imagenet @ centroids.T, axis=1)
    target_clusters = unique(imagenet_clusters)
    # Nearest centroid for each pool sample  [app. Filtering track]
    pool_clusters = argmax(pool_embeddings @ centroids.T, axis=1)  # (N,)
    # Keep caption-passing samples in a target cluster
    # [app. Filtering track]
    return [
        i for i in range(N)
        if mask[i] and pool_clusters[i] in target_clusters
    ]

The CLIP-score filter, keeping a top fraction:

# CLIP-score filter, keep a top fraction  [§4.1] [app. Filtering track]
# Input:  cosine similarity per pool sample (N,), fraction f
# Output: indices of kept samples

def clip_score_filter(scores, fraction):
    # Count from the fraction, rounded down
    # [code: baselines/apply_filter.py#L391]
    keep_count = int(len(scores) * fraction)
    # Threshold: element keep_count of the descending order
    # [code: baselines/apply_filter.py#L392]
    threshold = sorted(scores, reverse=True)[keep_count]
    return [i for i in range(N) if scores[i] >= threshold]

The basic filter in pseudocode:

# Basic filter  [§4.1] [app. Filtering track]
# Input:  caption, width, height, fastText language label
# Output: True if the pair is kept

def keep_basic(caption, width, height, language):
    # English only  [code: baselines/apply_filter.py#L227]
    if language != "en":
        return False
    # More than two whitespace-separated words
    # [code: baselines/apply_filter.py#L219]
    if len(caption.split()) <= 2:
        return False
    # More than five characters  [code: baselines/apply_filter.py#L220]
    if len(caption) <= 5:
        return False
    # Short side at least 200 px, aspect ratio at most 3
    # [code: baselines/apply_filter.py#L229-L232]
    short_side = min(width, height)
    long_side = max(width, height)
    return short_side >= 200 and long_side / short_side <= 3

BYOD track §3.3 §4.2 app. BYOD track

Participants may add any data sources, as long as they do not contain evaluation test images app. Benchmark rules. The paper's baselines use CC12M, YFCC15M, RedCaps, Shutterstock (photo, illustration, vector subsets), WIT, COYO, LAION-COCO, LAION-A and LAION-2B app. BYOD track. The external sources are not re-released by the authors §3.3.

Hyperparameters

Scale-level values. The "Samples seen" row equals pool size.

Filtering-baseline parameters:

Results

Author-reported. Setup: CLIP trained on each submission at the stated scale, zero-shot ImageNet accuracy and average over 38 tasks §5.1 tab. 3. No error bars are reported app. Checklist.

Other reported results:

• Filtering helps at every scale. The best filters beat no filtering across all four scales §5.1 tab. 3.
• Random subsets are worse than CLIP-score filtering. Random subsets improve with size; CLIP-score filtering peaks at intermediate size, near the top 30% §5.1 fig. 2.
• Rankings are fairly stable. Rank correlation of filter performance between scales is 0.71 to 0.90 §5.2 fig. 3. Correlation between ImageNet accuracy and the 38-task average is 0.99 §5.3.
• Zero-shot and linear-probe rankings agree (Spearman 0.99 for the filtering track, 1.0 for BYOD) app. Evaluation details.
• Batch-size doubling at medium keeps rankings close (rank correlation 0.96 ImageNet, 0.98 average) tab. 10. A ConvNeXt at matched MACs gives 1.0 and 0.87 tab. 11. Ten times more steps at small changes some rankings tab. 12.
• Fairness: the filtered models show demographic biases; BYOD sources can raise misclassification risk §5.3 app. Fairness and biases.

Paper vs code

Code at mlfoundations/datacomp, commit 4a8df1992566ef8334773f7152e1855b1f716162. The owner is mlfoundations, the organisation the paper's bibliography uses for its own dataset2metadata repository §3.2 (main.bib). The LICENSE file names the paper's authors LICENSE. The learning rate, warmup, batch size and model for every scale match the paper. Differences:

• AdamW β2 for small–large. Paper: 0.98 tab. 14. Code: beta2 is None for those scales, so no --beta2 flag is passed scale_configs.py train.py. The effective value comes from the OpenCLIP library default, which this repository does not pin. Not determinable from the repository.
• Data order. Paper: "two passes over A" and "four passes over B" §3.1. Code: --dataset-resampled, which samples shards with replacement train.py. Repeats are random draws, not exact passes.
• Epochs. Code runs 5 "epochs" of total ÷ 5 samples each, a checkpoint schedule the paper does not describe train.py train.py.
• Precision and other flags. Code default is amp train.py. --local-loss, --gather-with-grad and --grad-checkpointing are passed train.py. The paper mentions none of these, and this card does not analyse their effect on the loss.
• ImageNet targets for image-based filtering. Paper: ImageNet-1k or -21k app. Filtering track. Code: loads only ImageNet-1k CLIP L/14 embeddings apply_filter.py. The 21k variant appears only in the ablation tab. 12.
• Text-based filtering. Paper: ImageNet-21K or ImageNet-1K synsets §4.1. Code: loads only the ImageNet-21K WordNet IDs apply_filter.py.
• Image-size rule. Paper: smaller side "above 200" and aspect ratio "below three" §4.1. Code: smaller side at least 200 and aspect ratio at most 3 apply_filter.py.
• Clustering sample. Paper: about 160M samples for large and xlarge app. Filtering track. Code: sample_ratio defaults to −1, meaning no subsample image_based_clustering.py. The released centroids are downloaded with SHA-256 checks from an external asset repository utils.py, so the clustering run itself is not in the repository.
• Count of kept items. Paper: "top 30%". Code keeps int(N × f) + 1 items when scores are untied, because the threshold is the element at index int(N × f) and the filter uses >= apply_filter.py apply_filter.py.
• Not in the released code. The text-based sampling and image-based sampling baselines in app. Filtering track do not appear in baselines.py or baselines/.

No discrepancy: batch sizes, learning rates, warmup, model per scale, and samples seen (all four scales) match tab. 14 scale_configs.py.

Pitfalls

• The loss in the §3.4 footnote is missing its logarithm and minus sign. As written, the expression is a ratio of exponentiated scores, not a negative log-likelihood. The standard CLIP form takes the negative log of these ratios. Implement the standard form and cite the paper as the source of the symmetric design §3.4. The paper gives no temperature initialisation and no normalisation convention.
• "Passes" are resampled draws. Use the same resampling (with replacement) if you want to match the released code train.py.
• β2 is not pinned for small–large. Setting 0.98 explicitly removes one ambiguity; the paper's table is the only source for the value.
• Caption word counts differ. The basic filter counts whitespace-separated words apply_filter.py. The image-based caption filter uses fastText tokenisation and checks for more than one word apply_filter.py apply_filter.py. The paper's text says "two words" for both. Expect small differences.
• Pool is an index. Only URLs and alt-text are released. About 40B URLs were attempted and about 16.8B yielded images §3.2, so link rot changes the usable pool. Use the hashes to check downloads §3.2.
• Test-set leakage is a rule, not a guarantee. Near-duplicate removal at cosine 0.604169 flags about 2.8% of downloads app. Deduplication against evaluation sets. Recall is 0.8 and precision 0.9 on the authors' check [app. Deduplication against evaluation sets].
• No error bars. The authors report none "due to computational constraints" app. Checklist. The appendix calls the accuracy variance "significant" even at a fixed training-set size app. Extra figures and tables. Gaps between filters of about a point are therefore not established by the paper.
• Compute cost is hidden in the headline. The paper cites 4 to 40,000 GPU hours §1. The xlarge scale used 81 hours on 512 GPUs §3.1. DataComp-1B needed the same training budget as OpenAI's CLIP run to beat it tab. 1.
• Safety measures are imperfect. Roughly 19% of pairs are removed as NSFW §3.2. Faces are blurred with a detector, which the authors found has little effect on accuracy §3.2. The paper still reports demographic biases; BYOD datasets can increase the risk of misclassification §5.3, and a significant share of people are classified as criminal in the harm test app. Fairness and biases.
• Eval class names and prompts are borrowed. Several datasets reuse prompts from a similar task, for example SVHN with MNIST prompts app. Evaluation details. This affects comparability with other zero-shot numbers.
• Filtering uses test-set class names, not test predictions. Class labels may be used; predictions or images from evaluation sets may not app. Benchmark rules.

Lineage

• Builds on: CLIP [radford2021learning] for the contrastive training recipe and the choice of ViT encoders §3.4; OpenCLIP as the code base §3.4. Neither has a card in this wiki.
• Builds on: LAION-400M, whose filtering ideas inspired the basic filters, and LAION-5B, the source of the LAION-2B subset and its CLIP-score filter §4.1. Both cards exist.
• Compared against: OpenAI's CLIP ViT-L/14 trained on WIT tab. 1; LAION-2B subsets and LAION-2B's filter tab. 1 §4.1.
• Related, not compiled: coreset and semantic-deduplication work (SemDeDup) §2.