LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

In plain words

Models that link pictures to words, such as CLIP and DALL-E, need hundreds of millions of image-caption pairs to learn from. Until this paper, no such collection was openly available, so training at that scale was limited to groups with private data. The authors built one from web pages: they took the short descriptions that web developers attach to images and kept only the pairs that a CLIP model judged to match.

The release is the data itself, not a new model. It comes as a list of image links with their metadata, a CLIP embedding for each pair, and search indexes over those embeddings, so people can find and sample subsets. The authors also tested the set by training a text-to-image model on a part of it. The samples looked reasonable, but the paper gives no score to measure that.

The paper mostly describes how the set was made and what it contains.

TL;DR

• Releases LAION-400M: about 400 million image-text pairs from Common Crawl, filtered with CLIP §2.1.1.
• Also releases the CLIP embeddings of the images and texts, and kNN indices for similarity search §2.
• The download and resize tool, img2dataset, is linked from the paper; the CLIP filter itself is not in that repository (see Paper vs code).
• Results are descriptive: image-size counts tab. 1 and a DALL-E sample comparison fig. 3, with no quantitative benchmark.
• The dataset's own license is not stated; per-image Creative Commons tags are included in the metadata §2.

Problem

Image-text models such as CLIP and DALL-E were trained on hundreds of millions of pairs that were not released, and the authors say the largest known image-text sets, between 400 million and around a billion pairs, have not been released §1. Without a public dataset of that scale, researchers cannot train such models from scratch §1. The collection pipeline uses CLIP similarity to remove mismatched alt-text from Common Crawl pages §2.1.1.

Method

The acquisition pipeline has two stages: a distributed pass over Common Crawl that produces candidate URL and caption pairs, and a lighter single-node post-processing pass that produces the final dataset §2. Stage one takes the WAT files of Common Crawl, extracts every HTML image tag that has an alt-text attribute, and downloads the raw images from those URLs §2.1. Stage two applies the filters and writes the output §2.1.1 §2.1.2.

The filters, in the order the paper lists them §2.1.1:

1. Drop a sample whose alt-text is shorter than 5 characters, or whose image is smaller than 5 KB.
2. Remove duplicates with a Bloom filter over the pair of URL and alt-text.
3. Compute a CLIP embedding of the image and of the alt-text, and drop the sample when the cosine similarity is below 0.3. The threshold was chosen by human inspection.
4. Use the CLIP embeddings to filter out illegal content. The paper gives no method or threshold for this step.

Image files are resized and stored in the webdataset format with img2dataset §2.1.2. The paper states that this tool can download 100 million images from a URL list in 20 hours on a single node with a 1 Gbps link, 32 GB of RAM and a 16-core CPU §2.1.2.

The release has three parts §2: parquet files with the image URL and metadata for each pair (sample ID, URL, the Creative Commons license type when it applies, an NSFW tag from CLIP, the cosine similarity between text and image embeddings, and width and height); the CLIP embedding of each image with its text; and several kNN indices for fast search §2.

Pseudocode of the acquisition as the paper describes it:

# LAION-400M acquisition  [§2] [§2.1] [§2.1.1]
# Input:  Common Crawl WAT files
# Output: kept (url, alt-text, similarity, nsfw_tag) records

def build_dataset(wat_files, clip):
    # Extract (url, alt-text) for every IMG tag with alt-text  [§2.1]
    pairs = parse_img_alt_text(wat_files)

    # Drop alt-text shorter than 5 characters  [§2.1.1]
    pairs = [p for p in pairs if len(p.alt_text) >= 5]

    # Drop duplicate (url, alt-text) pairs with a Bloom filter  [§2.1.1]
    pairs = bloom_deduplicate(pairs)

    kept = []
    for pair in pairs:
        # Download the image; drop it when the file is under 5 KB  [§2.1.1]
        image = download(pair.url)
        if image is None or size_bytes(image) < 5 * 1024:
            continue

        # Embed image and text with CLIP  [§2.1.1]
        image_emb = clip.encode_image(image)
        text_emb = clip.encode_text(pair.alt_text)

        # Keep pairs whose cosine similarity is at least 0.3  [§2.1.1]
        similarity = cosine(image_emb, text_emb)
        if similarity < 0.3:
            continue

        # Illegal-content filter: method and threshold not stated  [§2.1.1]
        if illegal_content(image_emb, text_emb):
            continue

        # NSFW tag is kept as metadata, not used to drop  [§2]
        nsfw_tag = nsfw_detect(image_emb)
        kept.append((pair.url, pair.alt_text, similarity, nsfw_tag))

    # Resize and write webdataset shards  [§2.1.2]
    write_webdataset(kept)
    return kept

The paper names no CLIP variant for the filter; the text names CLIP ViT-B/16 only for the DALL-E ranking step §3. The paper does not describe how the kNN indices are built §2.

DALL-E check

The authors ran DALLE-pytorch, an open replication of DALL-E, to test whether the data can train a text-to-image model §3. Image tokens come from a VQGAN pretrained on ImageNet §3. Training used a random subset of about 7.2 million LAION-400M samples for one epoch on one RTX 2070 Super with 8 GB of memory fig. 3. At generation time, CLIP ViT-B/16 ranks the top 8 of 128 samples per caption §3.

# DALL-E sample selection  [§3]
# Input:  caption text, 128 generated images
# Output: 8 images shown per caption

def select_samples(caption, generator, clip):
    # Generate 128 candidate images for one caption  [§3]
    candidates = [generator(caption) for _ in range(128)]

    # Score every candidate against the caption with CLIP ViT-B/16  [§3]
    scores = [clip.similarity(c, caption) for c in candidates]

    # Keep the 8 highest-scoring candidates  [§3]
    return top_k(candidates, scores, k=8)

Hyperparameters

Values stated in the paper, plus code defaults where they exist. The code defaults belong to img2dataset, not to the LAION pipeline.

Results

Author-reported. The paper has no baseline table. Setup: the dataset counts below are from the LAION-400M set itself; the DALL-E check uses a subset on one RTX 2070 Super §3 fig. 3.

The DALL-E comparison shows the three sets as curves in the figure, with no numbers in the text fig. 3. Image-size counts are only a description of the dataset; the paper says that high-resolution images are abundant and that users can pick subsets by size §3 tab. 1.

Paper vs code

The paper links img2dataset §2.1.2, and that is the only repository tied to this paper that we could verify. The repository is MIT-licensed, by Romain Beaumont, one of the paper's authors. We cloned commit 95523bc7579745a1749b8132d408b9fea89a338e, which is the current HEAD and not the 2021 release. The project page linked from the abstract could not be read, because WebFetch permission was not granted during compilation. A GitHub search for "LAION-400M" returned no LAION-owned pipeline repository with the CLIP filter.

• Paper: the filters and the CLIP threshold are part of the LAION-400M pipeline §2.1.1. Code: img2dataset contains none of them. The repo has no bloom filter, no NSFW classifier and no cosine threshold. Its only clip calls are np.clip for pixel values and blurring resizer.py blurrer.py. So the paper's filter is not reproducible from this repository.
• Paper: image size is 5 KB §2.1.1. Code: img2dataset checks the shorter image side in pixels, with a default of 0 resizer.py main.py. No 5 KB byte check was found.
• Paper: image download uses asynchronous requests with Trio and Asks §2.1. Code: downloads use urllib in a thread pool downloader.py. The paper's Trio and Asks stage is in the Common Crawl pipeline, which is not in this repository.
• Paper: images are stored in webdataset format §2.1.2. Code: the webdataset writer exists writer.py.
• Paper: download throughput of 100M images in 20 hours §2.1.2. Code: no benchmark or run configuration in the repo supports this figure; the thread count default is 256 main.py.
• Paper: DALL-E training is by DALLE-pytorch §3, a third-party library cited in the bibliography. It is not linked to this card as code.
• No official code for the CLIP filter, the NSFW tagger or the kNN index build was found.

Pitfalls

• Count mismatch. The abstract and the introduction say 400 million pairs §1; tab. 1 lists 413M unique samples. The card does not reconcile them, and the paper does not say which count is released.
• Unnamed CLIP model. The text does not name the CLIP variant used for the 0.3 cosine filter, so do not assume ViT-B/16 for the filter §2.1.1 §3.
• Hand-chosen threshold. The 0.3 threshold came from human inspection of samples, with no sample size, no precision or recall, and no sensitivity study §2.1.1. A different CLIP variant changes the score distribution.
• Undefined filters. The illegal-content filter has no method or threshold §2.1.1. The NSFW tag comes from CLIP, its threshold is not stated, and the under-1% share is not tied to a labelled check §2. The NSFW items are tagged, not removed; users filter by the tag §2.
• Bloom filter parameters. The filter size and false-positive rate are not stated. A false positive drops a valid pair. The paper also does not say whether URL and alt-text are normalised before comparison §2.1.1.
• Size definition. "5 KB image size" §2.1.1 refers to bytes in the paper; the code checks pixel side length. Use the code default with care, and set the check yourself if you need the paper's rule.
• Dataset license. The paper does not state the license of the released dataset. Its metadata carries a per-sample Creative Commons license type when one applies §2. This card does not establish that the released images or captions may be reused commercially. The license field covers the paper only.
• Compute and cost. The 20-hour, single-node claim assumes the listed 1 Gbps link, 32 GB RAM and 16-core CPU §2.1.2. The Common Crawl stage is "petabyte-scale" §2, and its cost is not stated.
• Evaluation. The only model result is a qualitative sample check on one DALL-E run fig. 3 §3. No FID, CLIP score or human study is reported, so the paper does not show the set is good for training.
• Unstated training details. The DALL-E model size, learning rate, batch size, optimizer, sampling seed and step count are not stated §3 fig. 3.
• Subset choice. The subset of 7.2M samples is "randomly picked" fig. 3. The paper gives no seed and no check that the subset matches the full distribution.

Lineage

• Builds on: CLIP, which supplies the filter and the embeddings §2.1.1 §2; arXiv:2103.00020 (the paper's bibliography entry [clip]). DALL-E, the model whose data gap this paper targets §1; arXiv:2102.12092 (the paper's [dalle] entry). VQGAN, used to tokenise images in the DALL-E check §3; arXiv:2012.09841 (the paper's [vqgan] entry).
• Compared against: Conceptual Captions 3M and 12M, and a 3M LAION subset, in the DALL-E check fig. 3. Neither is cited as a benchmark for the dataset itself.
• Related: ALIGN, cited in the introduction with CLIP and DALL-E as a model trained on large image-text data §1; arXiv:2102.05918 (the paper's [align] entry).
• Superseded by: not stated in this wiki.