LAION-5B: An open large-scale dataset for training next generation image-text models

In plain words

Training models that match pictures with text needs a huge pile of images with captions. Until this paper, no collection that large was openly available. The authors built one from a public web archive: they took the short descriptions that web pages attach to images (the text shown when a picture cannot load), downloaded the images, and kept only the pairs where a CLIP model judged the picture and the words to match.

The result is a dataset of about 5.85 billion pairs. It is noisy, because web captions are often spam or keyword lists. The authors also tag pictures that may be sexual or otherwise inappropriate, rather than deleting them, so users can choose what to include. To test the data, they train CLIP-style models on its subsets and fine-tune image generators on it. They report that the models roughly match the original CLIP on standard tests, and that more data helps when there is enough training compute.

TL;DR

LAION-5B is a dataset release, not a model. It provides 5.85 billion image-text pairs (about 2.32 billion English), each with a CLIP similarity score and NSFW and watermark scores. Use it if you need a large open pre-training corpus or want to reproduce CLIP-style training on public data. Read the pitfalls first: the filter is only a CLIP similarity threshold, the image rights stay with third parties, and the code differs from the paper in places.

Problem

Models such as CLIP, ALIGN, BASIC, GLIDE, Flamingo and Imagen were trained on billions of image-text pairs, but those datasets were not public, so the wider research community could not study or reproduce them [abstract]. The paper targets this gap by releasing a dataset of comparable scale [abstract].

Method

The dataset is built in three stages: (i) web-page filtering of image alt-text from Common Crawl, (ii) distributed download of the images, and (iii) content filtering with CLIP and tagging §3.1. The flowchart is fig. 2.

Stage 1, extraction. The authors parse image tags from Common Crawl's WAT metadata and keep only images that have alt-text §3.1. Language is detected with CLD3 into three groups: English, another language, or no confident language. A manual check of a random sample found that the "no language" group mostly holds short language-agnostic text such as product and place names §3.1.

Stage 2, download. Images are fetched with asynchronous requests on small cloud workers, about 300 in parallel, in chunks of 10,000 links §3.1. The authors built the img2dataset tool for this step; they report 100 million images in 20 hours on one node, and a distributed mode that handles the 5 billion samples in a week on 10 nodes app. C.1.

Stage 3, post-processing and filter. Pairs with fewer than 5 characters of text or less than 5 KB of image data are removed, as are unsafe, very large or redundant images §3.1. Each remaining pair gets a cosine similarity between its image and text embeddings from OpenAI's CLIP ViT-B/32. English pairs need a similarity of at least 0.28; all other pairs need at least 0.26, where the non-English score comes from a multilingual CLIP ViT-B/32 §3.1. This step removed about 90% of roughly 50 billion candidates, leaving just under 6 billion §3.1.

# Keep an image-text pair only if CLIP agrees with its caption  [§3.1]
def keep_pair(image, text, is_english):
    # Threshold depends on language: 0.28 English, 0.26 otherwise  [§3.1]
    threshold = 0.28 if is_english else 0.26
    # Embed both with ViT-B/32 (English) or the multilingual text tower  [§3.1]
    image_vector = image_encoder(image)
    text_vector = text_encoder(text, is_english)
    # Cosine similarity between the two embeddings  [§3.1]
    score = cosine(image_vector, text_vector)
    return score >= threshold

Tagging, not removal. The authors state that they keep potentially offensive pairs and tag them, so users can decide §3.2. Two tags are added: Q16 concept documentation and a dedicated pornographic/sexualised classifier (called NSFW). Both work on CLIP embeddings §3.2 app. C.5 app. C.6. A watermark detector is trained on manually annotated samples plus retrieved examples app. C.7. Each pair carries a cosine similarity, the NSFW and watermark scores, width, height, URL, caption and a 64-bit id §4.

Subsets. The release has three parts: 2.32 billion English pairs (LAION-2B-en), 2.26 billion pairs in over 100 other languages, and 1.27 billion pairs without a clear language. The five most frequent non-English languages are Russian (10.6%), French (7.4%), German (6.6%), Spanish (6.6%) and Chinese (6.3%) §4. About 3% of images are flagged NSFW §4.

Derived artefacts. CLIP ViT-L/14 embeddings of the images were computed with 32 A100 GPUs in about a week (312 samples/s per GPU, against 1800 for ViT-B/32) app. C.2. A PQ128 nearest-neighbour index was built on about 9 TB of embeddings app. C.3; the index and metadata are about 800 GB and are read through memory mapping app. C.4.

CLIP reproduction (validation). The authors train CLIP models with OpenCLIP on LAION-400M, and one on LAION-2B-en, following the original CLIP setup §5.2 app. E.1. The loss is InfoNCE with the global batch gathered across workers; the paper says this reduces the memory of the loss matrix from order N² to order nN, where n is the per-GPU batch and N the global batch app. E.2.

Generative validation. GLIDE is fine-tuned from OpenAI's filtered 64-pixel checkpoint on LAION-5B data; captions are replaced with the unconditional token 20% of the time app. F. Stable Diffusion checkpoints are trained on LAION-2B-en, laion-high-resolution and laion-aesthetics subsets app. F. These generative results are shown as samples fig. 12 fig. 13 fig. 14 and an external comparison report, not as measured scores §5.3.

Hyperparameters

CLIP reproduction, from tab. 3 (training) and tab. 2 (architecture), in app. E.1. All runs use AdamW with β1 = 0.9, β2 = 0.98 and weight decay 0.2, a linear warm-up followed by cosine decay, and the InfoNCE loss app. E.1.

Architectures tab. 2:

Filter and tagging settings (from the text, not a table):

Results

Author-reported. Setup: zero-shot ImageNet-1k and robustness sets plus the VTAB+ average over 35 tasks, top-1 accuracy, models trained with OpenCLIP §5.2.1 tab. 1 app. E.3.

Data versus compute, zero-shot ImageNet-1k top-1 tab. 5 fig. 11. Samples seen at fixed compute favour the larger dataset:

Headline claims: LAION-400M and CLIP WIT models match on ImageNet-1k, with larger differences on ObjectNet (CLIP better) and ImageNet-Sketch (LAION better) §5.2.1. LAION-2B-en improves over LAION-400M for B/32 and L/14 §5.2.1. Few-shot linear probes show similar transfer for LAION and CLIP models app. E.3 fig. 9 fig. 10.

Dataset numbers: about 50 billion candidates, about 90% removed, just under 6 billion kept §3.1; 5.85 billion pairs in total, split 2.32 / 2.26 / 1.27 billion §4.

Paper vs code

The official code is the GPU worker of the crawlingathome project, by Richard Vencu (an author), at commit 91e230f, MIT licence. The paper names it as the pipeline code §3.1. Checked: the filter thresholds, the language switch and the alt-text and image-size cut-offs. Not checked: img2dataset, clip-retrieval, autofaiss, OpenCLIP training code, the NSFW and watermark repositories, and whether this worker is the exact code that produced LAION-5B.

• Paper: similarity thresholds 0.28 for English and 0.26 otherwise §3.1. Code: same thresholds, chosen by the language job set [code: raw/code/2210.08402/gpu_inference.py#L111-L113] [code: raw/code/2210.08402/gpu_inference.py#L497-L501]. Matches.
• Paper: alt-text under 5 characters is removed §3.1. Code: same rule [code: raw/code/2210.08402/ccpp.py#L155]. Matches.
• Paper: image under 5 KB is removed §3.1. Code: drops responses under 5000 bytes dbdl.py. The paper says KB; the code counts bytes.
• Paper: potentially offensive pairs are kept and tagged, not removed §3.2. Code: additionally drops pairs when the top underage-related match index is below 4, when the caption contains "teen", "kid", "child" or "baby", or when the top animal-category match index is above 20 [code: raw/code/2210.08402/gpu_inference.py#L138-L145]. Not described in the paper.
• Paper: NSFW tags come from a trained classifier and the Q16 documentation app. C.5 app. C.6. Code: the only NSFW logic in the repository's .py files is the category heuristic in the filter [code: raw/code/2210.08402/gpu_inference.py#L127-L136]. The trained classifier, Q16 and the watermark model are not in these files (only grep-checked).
• Paper: duplicates are not discussed in the compiled text. Code: looks up pairs on a remote bloom-filter server ccpp.py.

Pitfalls

• Scoring is index-based in the code. prob() returns the indices of the two best-matching prompts, not probabilities [code: raw/code/2210.08402/gpu_inference.py#L99-L108]. The NSFW, underage and animal rules then compare those indices with fixed numbers 19, 4 and 20. The paper's descriptions do not make this clear, so reproduce the rules only from the code, or not at all.
• A missing comma merges two prompts. In the NSFW category list, the animal entry and the fashion entry are joined by no comma, so they become one prompt gpu_inference.py. Category positions after that point shift.
• Multilingual scoring mixes encoders. The code embeds the image with OpenAI's CLIP ViT-B/32 and the text with the sentence-transformers multilingual model gpu_inference.py [code: raw/code/2210.08402/gpu_inference.py#L77-L79]. The paper only says "multi-lingual CLIP ViT-B/32" §3.1. Follow the code if you need exact reproduction.
• The filter uses the smaller CLIP model. The authors filter with ViT-B/32 because larger CLIP models were not available when collection started §3.1. They expect a ViT-L/14 filter to be less noisy but did not test it §6.
• Costs the abstract hides. About 50 billion candidates were scanned to keep about 6 billion §3.1. The download used about 300 workers; ViT-L/14 embeddings took a week on 32 A100s; the search index is about 9 TB of embeddings and an 800 GB index §3.1 app. C.2 app. C.3 app. C.4.
• Text quality. Alt-text is often SEO spam or keyword lists §6. The "no language" group (1.27 billion pairs) is mostly product and place names §4.
• Test-set overlap. LAION-5B may contain images from downstream test sets if those are in Common Crawl. The authors think this is a small risk but have not measured it §6.
• Unexplained gap. On ImageNet-A, LAION-400M L/14 is 24.3 points below CLIP WIT L/14 app. E.3. The authors offer no explanation.
• NSFW error rates belong to another model. The reported 96.45% detection and 7.96% false positives are for the EfficientNet-V2 classifier. The authors then moved to a CLIP ViT-L/14 MLP and give no error rates for it app. C.5.
• Training details not stated. Not stated: temperature (logit scale) at initialisation, text token length, image augmentation, weight initialisation, and the LR schedule's end value. The L/14 (2B) row is commented out in the source app. E.1.
• GPU count conflicts with the table. The text says 128 to 400 GPUs §5.2, while tab. 3 lists 112 GPUs for B/32 on LAION-2B-en. Use the table, but check.
• Robustness and generative claims are weak evidence. Reported numbers are for classification on standard benchmarks. The generative section gives samples and a W&B report, with no measured quality numbers §5.3.
• Licensing of content. The paper's CC BY 4.0 licence covers the metadata only. The authors state they do not own the copyright of the images or text app. A. Do not copy images or captions into this wiki or a commercial product without checking each source's rights.
• Privacy. The data can link to images of people, including faces and medical images. The authors say specific samples can be removed on request app. G app. A.

Lineage

• Builds on: CLIP [radford2021learning] (cited in §5.2) and LAION-400M [schuhmann2021laion], the earlier open subset from the same group (cited in §4 and §5). LAION-400M is not compiled here.
• Compared against: OpenAI's CLIP WIT models on tab. 1.
• Used by: GLIDE and Stable Diffusion fine-tunes (generative validation) app. F; the paper also names BLIP, MAGMA, PyramidCLIP and latent diffusion as users of LAION subsets §5.1. None of these cards exist in this wiki yet.
• Superseded by: not stated in the paper.