--- kind: concept slug: dataset-curation title: Dataset curation summary: >- How training and evaluation data is sourced, filtered, deduplicated, labelled and documented. Covers web text, image-text pairs, speech, vision, 3D objects, parallel text, robot demonstrations, instruction and benchmark data, the filters each paper used and their reported effects, and the licence and documentation traps that come with reusing them. tags: [data, pretraining-data, data-quality, labelling, documentation, robotics, translation] papers: ["1610.02055", "1710.08092", "1802.07856", "1804.00247", "1804.03209", "1806.05258", "1810.03505", "1810.05726", "1812.01718", "1911.03894", "2006.14480", "2007.10937", "2011.01060", "2012.02110", "2012.15761", "2101.00027", "2101.02235", "2102.08981", "2103.04098", "2103.12028", "2104.08758", "2105.06762", "2106.15772", "2109.09607", "2110.00061", "2110.00530", "2110.04994", "2111.02114", "2111.08897", "2112.06409", "2201.06642", "2201.12086", "2203.10244", "2205.01068", "2205.12446", "2206.14858", "2207.04672", "2209.08538", "2210.08402", "2211.05100", "2212.03533", "2212.10560", "2304.01373", "2304.07327", "2304.14108", "2305.10403", "2305.11206", "2306.11644", "2307.05663", "2310.17389", "2403.12945", "2404.14219", "2501.14249", "2503.16219", "2512.10092", "2609.24410", "2609.34138", "2610.00223"] related: [language-model-pretraining, benchmark-validity, llm-ethics, algorithmic-fairness, data-poisoning, noisy-labels, instruction-tuning, contrastive-learning, vision-language-pretraining, llm-as-a-judge, data-augmentation, multilingual-evaluation, neural-machine-translation, mixture-of-experts, curriculum-learning, imitation-learning, neural-scaling-laws, novel-view-synthesis, functional-safety, anomaly-detection, adversarial-evaluation, face-recognition, self-training, toxicity-detection] status: draft updated_at: 2026-11-01 --- ## Overview Curation decides what a model or benchmark is trained or scored on: where examples come from, which ones are kept, how they are labelled, and what is written down about the result. The 55 cards on this page repeat the same steps: choose sources, filter by rules and by models, remove duplicates, label with human checks, split held-out data carefully, and document what was kept and what was dropped [arxiv:2104.08758 §1] [arxiv:2112.06409 §1]. The survey that maps this area describes the work as data-centric. It reports no experiments of its own, so its value is the taxonomy and the names of the techniques [arxiv:2112.06409 §2]–[§5]. Numbers below are author-reported unless a card says otherwise. Check the linked card before relying on one. **Licence and reuse.** Many releases here are not free for commercial use, even when the paper is CC BY. Examples: the DialogSum data repository declares non-commercial terms [arxiv:2105.06762 §2]; the Omnidata starter set is research-only [arxiv:2110.04994 §4.2]; ARKitScenes data and code fall under a non-commercial licence [arxiv:2111.08897 §1]. The MovieNet paper is CC BY 4.0, but that licence covers its text only. Its films, trailers and labels are reached through a user agreement that the project page describes, and the paper says the data will be published in line with regulations [arxiv:2007.10937 §1] ([card](/papers/2007.10937)). SMHD is shared under a data-use agreement [arxiv:1806.05258 §3.1]. CINIC-10 states no licence for its dataset, and its ImageNet images come from a source with separate terms [arxiv:1810.03505 §2] [§3]. DeepWeeds states CC BY 4.0 for its images and annotations [arxiv:1810.05726 § Data Availability]. Objaverse-XL keeps each object's own licence, and the dataset as a whole is ODC-By 1.0 [arxiv:2307.05663 app. C]. Its paper says savable Polycam objects are CC BY 4.0 [arxiv:2307.05663 §3.1], but the repository says Polycam data is for academic, non-commercial use on request [code: README.md]. Treat Polycam-derived objects as unresolved. The NLLB training bitext comes from many public corpora under their own terms [arxiv:2207.04672 §8.1.2]. Check each card's `code_license` and the dataset's own terms. WebFace260M and WebFace42M are marked "Public" in the paper's dataset table, but its text grants access only to applicants who sign a licence for research use [arxiv:2103.04098 tab. 1] [§6]. The dynamic hate-speech dataset's README states CC BY 4.0, but its repository has no LICENSE file [code: README.md#L7]. HLE asks users not to train on its questions [arxiv:2501.14249 §1]. Self-Instruct's released generated data has no licence stated in its paper, and it was produced with the GPT-3 API, whose terms the paper does not discuss [arxiv:2212.10560 §3] [app. A.2]. A dataset-analysis toolkit depends on a public sparse autoencoder and a Llama 3.3 reader model; its card records no licence for either [arxiv:2512.10092 §4]. ## Variants ### Web text for language-model pretraining These papers all start from web crawls or large document collections. The main decisions are the language filter, the rule filters, and the dedup method. - **C4** applies rule filters to one April 2019 Common Crawl snapshot and keeps English text, with a word blocklist. The blocklist removed 42% of documents classed as African American English against 6.2% for White-aligned English [arxiv:2104.08758 §5.3]. The authors advise against blocklists [arxiv:2104.08758 §6]. Card: [Documenting Large Webtext Corpora](/papers/2104.08758). - **The Pile** combines 22 weighted English sources with per-source processing and documentation [arxiv:2101.00027 §2]. Near-duplicate removal covers only OpenWebText2 and Pile-CC [arxiv:2101.00027 app. D]. Its 1.3B models beat CC-100 and raw Common Crawl on Pile components [arxiv:2101.00027 §3, §4]. Card: [The Pile](/papers/2101.00027). - **OSCAR 22.01** assigns one language per document rather than per line, trims short lines at the head and tail, and adds annotations such as "noisy" or "tiny" instead of deleting text [arxiv:2201.06642 §3] [§4] [§5]. The authors say they cannot yet state its quality [arxiv:2201.06642 §1]. Card: [Cleaner Document-Oriented Multilingual Crawled Corpus](/papers/2201.06642). - **OPT** deduplicates about 180B tokens from several sources with MinHash [arxiv:2205.01068 §2.3]. Card: [OPT](/papers/2205.01068). - **PaLM 2** uses a more multilingual, deduplicated mix, with de-identification and quality filtering. It also adds toxicity control tokens. Its mixture weights and sizes are not stated [arxiv:2305.10403 §3]. Card: [PaLM 2](/papers/2305.10403). - **BLOOM** trains on the ROOTS corpus of 46 natural and 13 programming languages [arxiv:2211.05100 §1] [§3.1]. Card: [BLOOM](/papers/2211.05100). - **Pythia** trains twice on the Pile, once as released and once deduplicated, in the same data order, and saves checkpoints for analysis [arxiv:2304.01373 §2.2]. Card: [Pythia](/papers/2304.01373). - **CamemBERT** uses French web text from OSCAR. A 4 GB sample performed about as well as the full 138 GB, and web text beat Wikipedia of the same size [arxiv:1911.03894 §6]. Card: [CamemBERT](/papers/1911.03894). - **GottBERT** compares German OSCAR with a filtered version that is 121 GB against 145 GB. Filtering made no clear difference, and the best filtered/unfiltered pairs differ by at most 0.85 points [arxiv:2012.02110 tab. 3]. Card: [GottBERT](/papers/2012.02110). - **Minerva** keeps web pages whose raw HTML contains math markup, so that TeX and AsciiMath survive, and drops pages with MathML only [arxiv:2206.14858 §2.1] [app. B.2]. Card: [Solving Quantitative Reasoning Problems](/papers/2206.14858). - **Phi-3** combines heavily filtered web data with LLM-written synthetic reasoning data [arxiv:2404.14219 §2]. Card: [Phi-3 Technical Report](/papers/2404.14219). - **phi-1** uses about 6B tokens of filtered Python code, selected by a classifier trained on about 100k GPT-4 ratings, plus synthetic textbooks [arxiv:2306.11644 §2] [§2.1]. At 350M parameters, HumanEval rose from 12.19% on unfiltered code to 17.68% on filtered code and 20.12% with the textbooks added [arxiv:2306.11644 §2.1]. Card: [Textbooks Are All You Need](/papers/2306.11644). - **E5** pretrains on CCPairs, about 270M text pairs from forums, encyclopedias, papers and web pages, kept by a consistency filter [arxiv:2212.03533 §3]. Card: [Text Embeddings by Weakly-Supervised Contrastive Pre-training](/papers/2212.03533). - **Word-similarity sets for six Indian languages** start from two English lists, RG-65 and WordSim-353. Three native speakers translate the pairs, and eight native speakers rate each pair from 0 to 10, averaged [arxiv:2609.34138 §4.1] [§4.2]. Inter-annotator agreement is reported as Fleiss' kappa per language, from 0.756 (Tamil) to 0.887 (Urdu) [arxiv:2609.34138 tab. 1]. The paper does not say how the 97 to 163 pairs per language were picked from the base lists [arxiv:2609.34138 §4]. Card: [Word Similarity Datasets for Indian Languages](/papers/2609.34138). - **Carbon accounting** for Noor counts the data stage as a cost: the 210 TB of raw dumps were downloaded and discarded, and about 2 TB of curated data was kept [arxiv:2610.00223 §5.1.1]. Card: [Carbon Footprint of Noor](/papers/2610.00223). ### Parallel text for translation - **NLLB-200** builds training data for 202 languages from three sources: public parallel corpora, bitext mined with LASER3 sentence encoders, and backtranslation from two generators [arxiv:2207.04672 §8.1.2] [§8.1.3]. Mined pairs are dropped below a LASER score of 1.06 or above a length ratio of 9.0. Pairs also fail on language identification and toxicity, and duplicates are removed after normalisation [arxiv:2207.04672 §8.1.4]. The final set covers about 18B sentence pairs in 2440 directions [arxiv:2207.04672 §8.1.6]. The authors report that filtering improved bilingual models and reduced hallucinated toxicity [arxiv:2207.04672 tab. 27]. Most of the 40,602 benchmark directions have no training data [arxiv:2207.04672 §8.3.2]. Card: [No Language Left Behind](/papers/2207.04672). ### Image-text pairs and vision data - **LAION-400M** keeps Common Crawl alt-text pairs whose CLIP similarity is at least 0.3. It then removes duplicates with a Bloom filter over URL and alt-text. The illegal-content filter has no stated method or threshold [arxiv:2111.02114 §2.1.1]. Card: [LAION-400M](/papers/2111.02114). - **LAION-5B** uses the same idea at larger scale, with a CLIP similarity threshold of 0.28 for English and 0.26 for other languages. It tags NSFW and watermark content and keeps it, rather than deleting it [arxiv:2210.08402 §3.1]. Card: [LAION-5B](/papers/2210.08402). - **CC12M** relaxes CC3M's unimodal filters for about four times the data. Caption precision falls to 76.6% from 90.3% on a 100-example check, and the gain shows up after fine-tuning [arxiv:2102.08981 §2.2] [§2.3] [tab. 3]. Card: [Conceptual 12M](/papers/2102.08981). - **BLIP (CapFilt)** trains a captioner to write new captions for web images, then a filter removes mismatched texts, both original and synthetic. The paper describes one bootstrapping round [arxiv:2201.12086 §3.3]. Card: [BLIP](/papers/2201.12086). - **DataComp** fixes the training recipe and lets participants choose the pairs. Its best baseline intersects an image-similarity filter with the top 30% by CLIP score [arxiv:2304.14108 §4.1]. With ViT-L/14 at the largest scale, ImageNet zero-shot accuracy goes from 0.723 unfiltered to 0.792 filtered [arxiv:2304.14108 tab. 3]. Card: [DataComp](/papers/2304.14108). - **Places** gathers candidates from web image search, has workers confirm labels, and uses a trained classifier to find more images for rare classes [arxiv:1610.02055 §2.2]. Card: [Places](/papers/1610.02055). - **VGGFace2** downloads about 1,400 images per name. A six-stage pipeline then detects faces, drops faces that a per-identity classifier scores below 0.5, removes near-duplicates, and has annotators clean doubtful cases [arxiv:1710.08092 §4] [§4.4] [tab. 2]. Card: [VGGFace2](/papers/1710.08092). - **Omnidata** renders labelled views from 3D scans, with camera and point sampling that can be steered. Its starter set has 14,601,449 images [arxiv:2110.04994 §3.1] [§4.2]. Card: [Omnidata](/papers/2110.04994). - **ARKitScenes** records homes with a tablet's LiDAR and aligns a laser-scanner depth map to each video by rendering scans from the camera pose [arxiv:2111.08897 §3.2]. Card: [ARKitScenes](/papers/2111.08897). - **RDD2022** collects road photos from six countries. Boxes are hand-drawn in Pascal VOC XML, and only the training split is labelled [arxiv:2209.08538 p. 10] [p. 11]. Card: [RDD2022](/papers/2209.08538). - **PubTables-1M** merges header cells that the source markup over-splits, then drops tables whose text does not match the page [arxiv:2110.00061 §3] [alg. 1]. Card: [PubTables-1M](/papers/2110.00061). - **xView** annotates satellite chips in stages: worker review, supervisory checks, and expert review against a gold set [arxiv:1802.07856 §3.2] [§3.3]. Card: [xView](/papers/1802.07856). - **ChartQA** pairs human-written questions with machine-generated ones on real charts [arxiv:2203.10244 §1] [tab. 1]. Card: [ChartQA](/papers/2203.10244). - **DeepWeeds** photographs eight weed species and a Negatives class in the field, with experts labelling whether each image contains a target species. Negatives are not stratified across folds, and five stratified 60/20/20 folds are released [arxiv:1810.05726 § Dataset Collection] [§ Deep Learning]. Balance is approximate: Charters Towers has 434 positives and 605 negatives [arxiv:1810.05726 tab. 1]. The paper says an image may show several species, but does not say how such images were labelled, and the released label file has one class per row [arxiv:1810.05726 § Deep Learning]. Card: [DeepWeeds](/papers/1810.05726). - **CINIC-10** fills CIFAR-10's classes with downsampled ImageNet images: 60,000 CIFAR-10 images plus 210,000 ImageNet images, drawn at 21,000 per class and split equally [arxiv:1810.03505 §2]. The repository README says the ImageNet part was not vetted, and the test set includes CIFAR-10 training images [arxiv:1810.03505 §2]. Card: [CINIC-10 is not ImageNet or CIFAR-10](/papers/1810.03505). - **Objaverse-XL** collects about 10.2M 3D objects from five web sources. It removes files with identical SHA-256 content hashes, which the authors say removes about 23M files, and does not remove near duplicates [arxiv:2307.05663 §3.1] [app. C]. An object is removed as NSFW when at least 3 of its 12 renders score above 0.9 on a CLIP classifier. This removes 815 objects [arxiv:2307.05663 §3.3]. A 2-layer MLP on CLIP features flags bad renders, trained on 1.2K manual labels, at a threshold of 0.5 [arxiv:2307.05663 §3.3]. The official repository does not include the classifiers or the deduplication code [code: README.md]. Card: [Objaverse-XL](/papers/2307.05663). - **MovieNet** gathers about 1,100 films with trailers, photos, scripts, synopses and subtitles. Scripts are aligned to the film timeline by matching dialogue to subtitles under a monotone constraint, solved by dynamic programming [arxiv:2007.10937 §3.1] [app. A]. The character boxes come from four steps: keyframe selection, a detector trained on those labels, detector proposals cleaned by hand, then identity labels. Only the top ten credited cast members are named; everyone else falls into one other class [arxiv:2007.10937 §3.2]. Tags are kept only at a minimum frequency of 25, which leaves 80 action and 90 place classes [arxiv:2007.10937 §3.2]. The paper's counts disagree in places: trailers are 60K in the text and 68K in the genre table [arxiv:2007.10937 §3.1] [tab. 3]. The public repository holds the toolbox, not the alignment or benchmark code [code: movienet/tools/]. Card: [MovieNet](/papers/2007.10937). ### Face data and self-training cleaning - **WebFace260M and WebFace42M** start from about 4M celebrity names, and an image search gives up to 200 images per name, in tiers whose ranking the paper does not state [arxiv:2103.04098 §2]. The CAST pipeline then cleans the faces in three rounds. Each round clusters each name's faces, keeps the largest cluster, and merges or drops folders by the cosine similarity of their centres. A new student then becomes the next teacher [arxiv:2103.04098 §3] [fig. 4]. Identities that overlap the benchmarks or the paper's own test set are removed [arxiv:2103.04098 §3]. The result is 2M identities and 42M faces [arxiv:2103.04098 tab. 2]. The paper reports that WebFace42M with ArcFace reaches 97.70 on IJB-C, against 96.03 for MS1MV2 [arxiv:2103.04098 tab. 5]. Not every benchmark improves: on MegaFace rank-1, the WebFace4M model scores below MS1MV2 [arxiv:2103.04098 tab. 5]. The cleaning steps are not fully specified. DBSCAN's parameters and the centre definition are not stated [arxiv:2103.04098 §3]. Card: [WebFace260M](/papers/2103.04098). ### Speech and speaker-held-out data - **Speech Commands** collects one-second word clips from volunteers, filters by file size, loudness and one worker's transcription, and assigns splits by hashing file names so they stay stable across releases [arxiv:1804.03209 §4] (card TL;DR). Card: [Speech Commands](/papers/1804.03209). - **FLEURS** is n-way parallel across 102 languages. Other workers check the recordings, and the splits hold speakers out so a model cannot recognise a voice it has already heard [arxiv:2205.12446 §2.1] [§4.2]. Card: [FLEURS](/papers/2205.12446). - **Lyft L5** assigns each vehicle's logs to one split only [arxiv:2006.14480 §3.1]. Card: [One Thousand and One Hours](/papers/2006.14480). - **Jordanian Arabic speech from public media** has about 97,000 recordings, 113 h in total, of which 52 h are transcribed [arxiv:2609.24410 §6.4]. Audio is split at silences, resampled to 16 kHz, and kept only between 1 s and 20 s. Initial transcripts come from a commercial speech API and are corrected by hand [arxiv:2609.24410 §6.2] [§6.3]. The paper gives 37,000 labelled records in one place [arxiv:2609.24410 §6.4.2] and 52 h elsewhere, and states no confidence filter for pseudo-labels [arxiv:2609.24410 §7.3]. Over five noisy-student rounds, word error rate on the authors' evaluation set fell from 0.568 to 0.515 [arxiv:2609.24410 tab. 4]. Each round is chosen on the same score it reports, and the paper gives no dataset licence or data link [arxiv:2609.24410 §9] [§ Data availability]. Card: [Jordanian speech-to-text](/papers/2609.24410). ### Robot demonstrations - **DROID** records 76k successful teleoperated Franka demonstrations across 564 scenes, on one shared hardware and recording setup [arxiv:2403.12945 §3.1] [§3.2]. About 16k unsuccessful episodes are released but not counted in the 76k [arxiv:2403.12945 §3.2]. A scene is counted as new only after a substantial change to the workspace. The paper merges 2,080 raw scene IDs into 564 with a heuristic it calls conservative [arxiv:2403.12945 app. D]. Its co-training batches are half the task's own demonstrations and half successful DROID episodes [arxiv:2403.12945 §5.1]. The experimental subset is the first 40K successful episodes that had language labels at training time, so a later rebuild gives a different set [arxiv:2403.12945 app. F.2]. Card: [DROID](/papers/2403.12945). ### Human-written, crowd-written and conversation data - **StrategyQA** has crowd workers write questions from Wikipedia terms. Five pretrained solvers reject a question when at least four answer it; three fine-tuned solvers reject it when all three succeed. Decompositions are written before evidence is matched, to avoid bias [arxiv:2101.02235 §3.1] [§3.2]. Card: [Did Aristotle Use a Laptop?](/papers/2101.02235). - **2WikiMultiHopQA** generates questions from templates and filters them so each needs two steps. Each sample carries its evidence triples [arxiv:2011.01060 §2.1] [§2.2]. Card: [2WikiMultiHopQA](/papers/2011.01060). - **DialogSum** cleans and normalises dialogues from several sources. Each summary is checked by a second annotator, and the authors re-check a 10% sample of each batch [arxiv:2105.06762 §2.1] [§2.2] [§2.3]. Card: [DialogSum](/papers/2105.06762). - **ToxicChat** labels real first-turn prompts with a human-AI pipeline. Prompts scored below a moderation threshold are labelled non-toxic without human review [arxiv:2310.17389 §2.2] [app. D]. Card: [ToxicChat](/papers/2310.17389). - **SMHD** finds diagnosed users with high-precision self-report patterns and a keyword within 40 characters, then matches controls on post count and subreddit overlap [arxiv:1806.05258 §3.1] [§3.2]. Card: [SMHD](/papers/1806.05258). - **Kuzushiji** crops characters from 35 printed books. Class counts are imbalanced and some classes are multi-modal [arxiv:1812.01718 §2]. Card: [Deep Learning for Classical Japanese Literature](/papers/1812.01718). - **OpenAssistant** collects volunteer conversation trees, with quality labels and rankings of replies. Moderation uses peer labels, points and human moderators [arxiv:2304.07327 §2] [§3] [§4]. Card: [OpenAssistant Conversations](/papers/2304.07327). - **LIMA** fine-tunes on 1,000 prompt-response pairs. Sources are filtered Stack Exchange answers, wikiHow articles, writing prompts and written examples [arxiv:2305.11206 §2.1] [§2.2]. Card: [LIMA](/papers/2305.11206). - **ASDiv** gives each problem a type and grade level, and scores corpus variety with a BLEU-based lexical diversity measure [arxiv:2106.15772 §3] [tab. 2]. Card: [A Diverse Corpus for Math Word Problem Solvers](/papers/2106.15772). - **Small-model RL on maths** curates 39,659 questions from two sources by dropping trivial and noisy items with small models [arxiv:2503.16219 §2]. Card: [Reinforcement Learning for Reasoning in Small LLMs](/papers/2503.16219). - **Humanity's Last Exam** drops questions that frontier models already answer, and keeps a private set to check for tuning on the public one [arxiv:2501.14249 §3.1] [§3.2] [app. B.1]. Card: [Humanity's Last Exam](/papers/2501.14249). ### Dynamic, model-in-the-loop collection - **Dynamically generated hate-speech data** runs four rounds. Annotators write posts that fool the current target model, and later rounds mix in label-flipping perturbations of earlier posts [arxiv:2012.15761 §5]. Each round is split 80/10/10, and about half of each test set comes from annotators unseen in training [arxiv:2012.15761 §5]. The in-loop error rate falls from 54.7% in round 1 to 27.7% in round 4 [arxiv:2012.15761 tab. 3], and HateCheck accuracy rises from 60% to 95% (author-reported) [arxiv:2012.15761 §6.3]. The authors say later rounds are harder, so the falling rate understates the change [arxiv:2012.15761 §6.1]. Pitfalls: 18 perturbations do not flip the label, which the authors found after publication [arxiv:2012.15761 §5]. The v0.2.3 release removes duplicates, all of them in round 1 [code: README.md#L10-L12]. Card: [Learning from the Worst](/papers/2012.15761). ### Model-generated instruction data - **Self-Instruct** starts from 175 hand-written seed tasks and prompts a base GPT-3 davinci to write new instructions, then inputs and outputs. The result is about 52K instructions and 82K instances [arxiv:2212.10560 §2] [tab. 1]. New instructions are kept only if their ROUGE-L against every existing task is below 0.7, and text filters remove malformed items [arxiv:2212.10560 §2.2]. Fine-tuning GPT-3 on the result raises zero-shot SuperNI ROUGE-L from 6.8 to 39.9, against 40.8 for InstructGPT-001 [arxiv:2212.10560 tab. 3]. Only 54% of 200 sampled instructions had all fields judged valid, by one author [arxiv:2212.10560 tab. 2]. Card: [Self-Instruct](/papers/2212.10560). ### Measuring and documenting the data - **Learning curves, not single scores.** The Transformer training notes argue for reporting full BLEU learning curves against wall-clock time, not one best score [arxiv:1804.00247 §2.1]. Card: [Training Tips for the Transformer](/papers/1804.00247). - **Documentation.** C4 is documented from three views: where text comes from, what it says, and what was removed. The authors recommend reporting domains, checking benchmark contamination, measuring social bias, and reporting excluded content [arxiv:2104.08758 §1] [§6]. Benchmark examples occur verbatim in C4 at rates from 1.87% to 24.88% of generation targets [arxiv:2104.08758 §4.2] [tab. 2]. - **Data documentation for the Pile** covers bias, profanity and consent issues [arxiv:2101.00027 §6]. - **Fairness benchmark datasets.** A survey of 15 tabular datasets admits a dataset only if it is public, sits in a known repository and was used in at least three fairness papers [arxiv:2110.00530 §2.1]. For each one it learns a Bayesian network to show links from the protected attribute to the label, then scores one logistic-regression fit on a single 70/30 split [arxiv:2110.00530 §2.2] [§4.1]. The network is read as a screen for possible bias, not as a causal result [arxiv:2110.00530 §2.2]. Several zero fairness gaps come from constant predictions: Credit card clients and Diabetes never predict the positive class, and OULAD always does [arxiv:2110.00530 tab. 15]. Card: [A survey on datasets for fairness-aware machine learning](/papers/2110.00530). - **Licence fields.** LAION-400M does not state its dataset licence, but keeps per-image Creative Commons tags in the metadata [arxiv:2111.02114 §2]. - **Auditing a corpus by its concepts.** A sparse autoencoder on one reader model's activations gives each document a vector of named concept latents, max-pooled over tokens. The authors use these vectors to diff two datasets, find correlated concepts, cluster by chosen properties and retrieve documents by property [arxiv:2512.10092 §3] [§4]. Author-reported: the SAE diff costs 2 to 8 times fewer tokens than LLM hypothesis baselines, but only about 2 times for a single two-dataset comparison [arxiv:2512.10092 §4.1] [app. D.4]. Its results depend on the SAE's labels, and texts must fit a 2048-token limit [arxiv:2512.10092 §6] [§4 fn. 1]. Card: [Interpretable Embeddings with Sparse Autoencoders](/papers/2512.10092). - **Reading samples from web-mined corpora.** A manual audit of 205 language-specific subsets from five public corpora (CCAligned, ParaCrawl v7.1, WikiMatrix, OSCAR and mC4) reads up to 100 lines per language, with one rater per line [arxiv:2103.12028 §4.1]. Macro `C` (correct) shares range from 23.74% (WikiMatrix) to 87.21% (OSCAR), and 15 subsets contain no correct sentence, 87 are under 50% correct [arxiv:2103.12028 tab. 3] [§4.2] [§7]. The authors also found 83 nonstandard language codes overall, and at least 48 of the affected corpora are entirely spurious [arxiv:2103.12028 §5] [§7]. Their recommendation is to read about 100 sentences and check the codes before use [arxiv:2103.12028 §7]. Card: [Quality at a Glance](/papers/2103.12028). - **Describing rare cases.** A position paper on automated driving argues that corner cases need a machine-readable description, so datasets can be checked for how many they contain. It builds on existing scenario languages and gives three conceptual examples: glare, unknown objects and occlusion. It defines no schema, labelling procedure or severity metric, and reports no experiments [arxiv:2109.09607 §1] [§3] [§4] [§5]. Card: [Description of Corner Cases in Automated Driving](/papers/2109.09607). ## When to use which Choose by constraint, not by which paper is best known. | Situation | Use | Why | Source | |---|---|---|---| | English web pretraining, need provenance | A documented multi-source corpus, or C4 with its blocklist effect reported | The Pile publishes per-source weights and docs; C4 documents domains and removed text | [arxiv:2101.00027 §2] [arxiv:2104.08758 §6] | | Non-English model, small web budget | Start from a few GB of diverse web text, and check whether filtering helps | A 4 GB sample matched 138 GB for French; GottBERT found no clear filter gain | [arxiv:1911.03894 §6] [arxiv:2012.02110 tab. 3] | | Per-document language labels | OSCAR 22.01 rules: document-level language ID with annotations | Keeps multilingual pages whole and lets users filter later | [arxiv:2201.06642 §4] [§5] | | Image-text data, no proprietary set | CLIP-filtered public sets, with the threshold reported | LAION sets were released with the threshold and tags | [arxiv:2111.02114 §2.1.1] [arxiv:2210.08402 §3.1] | | Fixed compute, want the best pairs | Compare filters under one recipe, as DataComp does | Shows that a filter can beat a larger unfiltered pool | [arxiv:2304.14108 §4.1] | | Noisy captions and a fine-tuning stage | Keep the relaxed pool (CC12M) and fine-tune | Gains appear after fine-tuning, not from raw use | [arxiv:2102.08981 §2.3] [tab. 3] | | Noisy captions and compute to spare | Caption and filter with a bootstrapping step (CapFilt) | Cleans web text with model output, at extra cost | [arxiv:2201.12086 §3.3] | | Code or maths, tight budget | Filtered web data plus synthetic exercises, with a strong teacher model | Ablations at small scale show gains from filtering and synthetic text | [arxiv:2306.11644 §2.1] [arxiv:2404.14219 §2] | | Style-level instruction tuning | A small, varied, hand-checked set on a strong base model | The authors argue style, not knowledge, is what tuning changes | [arxiv:2305.11206 §5] | | Human labels that must be trusted | A gold set, batch acceptance thresholds, and second-pass review | Batches are accepted only when they meet stated precision and recall | [arxiv:1802.07856 §3.3] [arxiv:2105.06762 §2.3] | | Splits for speech, vision or driving | Hold out by speaker, by vehicle or by hash | Stops a model from scoring well by recognising its own training groups | [arxiv:2205.12446 §2.1] [arxiv:2006.14480 §3.1] | | Dialect speech with few transcripts and much untranscribed audio | Pre-train on the unlabelled audio, then pseudo-label in rounds with augmentation, and check the pseudo-labels by hand | The authors' five rounds cut WER from 0.568 to 0.515 on their own set, in one run [arxiv:2609.24410 tab. 4] [§8.2] | | Fairness test on a tabular dataset | Choose a dataset that matches the task, and check the classifier is not constant before reading a zero gap | Zero gaps appeared with constant predictions, so a fair-looking score can mean a useless model [arxiv:2110.00530 tab. 15] | | No instruction data, a strong base model, and budget to check output | Bootstrap from a few hand-written seed tasks, filter by similarity, and check a sample by hand | Self-Instruct reached 39.9 against 40.8 on SuperNI, yet under half of a sample was fully valid [arxiv:2212.10560 tab. 3] [tab. 2] | | Benchmark needing variety | Measure diversity and check contamination before trusting scores | Low-variety sets can give over-optimistic results | [arxiv:2106.15772 §1] [arxiv:2104.08758 §4.2] | | Field images for a deployed robot or camera | Photograph in the deployment conditions, label each image, and release fold files | DeepWeeds targets rangeland conditions that lab and crop datasets miss, and ships its folds | [arxiv:1810.05726 § Introduction] [§ Deep Learning] | | Test set for a language with no English-style benchmark | Translate an existing English set with native speakers, rate each item with several raters, and report kappa | The six Indian-language sets used this route and reported per-language agreement | [arxiv:2609.34138 §4.1] [§4.2] [tab. 1] | | A larger pool that is not hand-checked | Check the licence and whether test images overlap training images before reuse | CINIC-10 adds unvetted ImageNet images and shares CIFAR-10 training images with its test set | [arxiv:1810.03505 §2] [§3] | | Parallel text for many low-resource languages | Mine bitext with a sentence encoder, filter it by score, length and language ID, and backtranslate only where a baseline is weak | NLLB used these steps and reports that most benchmark directions have no training data | [arxiv:2207.04672 §8.1.2] [§8.1.4] [§8.3.2] | | Large public 3D object collection | Deduplicate by file hash, render fixed views, and drop NSFW and bad renders with stated thresholds; check each object's licence | Objaverse-XL used these steps, but its classifiers are not in the public repository | [arxiv:2307.05663 §3.1] [§3.3] [app. C] | | Robot policy needs scene diversity but has few task demos | Mix the task's own demonstrations with a diverse set, half and half, and keep only successful episodes | DROID co-training beat no co-training and OXE co-training on the authors' six tasks | [arxiv:2403.12945 §5.1] [§5.2] | | Unknown properties in two corpora or model outputs | Diff the datasets on SAE concept latents, then verify the top hypotheses with a judge model | Authors report higher verified frequency gaps than LLM-hypothesis baselines on model outputs, at lower token cost [arxiv:2512.10092 §4.1] [fig. 2] [app. D.4] | | Retrieve documents by a steerable property, with a dense baseline available | Score documents on latents whose labels match the query, and combine with a dense ranking | SAE ranking is close to an OpenAI+LLM baseline, and the combined ranking scored best on the authors' six datasets [arxiv:2512.10092 §4.4] | | Safety testing for rare situations | Write each rare case down in a structured form before collecting or simulating it | The position paper sets the direction only; it gives no schema or metric | [arxiv:2109.09607 §3] [§5] | | Hard examples for a model that is already deployed or in a loop, with trained annotators | Collect in rounds: annotators try to fool the current model, validate, and retrain | The in-loop error rate fell across rounds in the hate-speech study; the rounds need a model served during collection [arxiv:2012.15761 §5] [§6.1] | | Large set of labelled faces from web search, no hand labels | Clean by self-training, with a fixed teacher per round and a check against benchmark identities | WebFace42M ArcFace beat MS1MV2 on IJB-C in the authors' runs; access is by signed research licence [arxiv:2103.04098 tab. 5] [§6] | | Web-mined multilingual corpus you did not build | Read about 100 sentences per language and check codes before use | 15 of 205 audited subsets had no correct sentence [arxiv:2103.12028 §4.2] [§7] | ## Implementation sketch The steps below combine the cards. They are not one paper's algorithm. Where a card gives a threshold or a rule, the step names it. ```pseudocode # Curation pipeline assembled from the cards (not one paper's algorithm) # Input: sources (each with domain, date, licence), scorer, gold set # Output: kept records, a held-out split, and a log of what was removed def curate(sources, scorer, threshold, gold): # 1. Keep provenance with every record [arxiv:2104.08758 §6] records = [with_provenance(r) for s in sources for r in s.fetch()] log = {"rule": [], "model": [], "duplicate": []} # 2. Rule filters: count what they remove [arxiv:2201.06642 §5] kept = [] for r in records: if rule_fails(r): log["rule"].append(r) else: kept.append(r) # 3. Model filter with a stated threshold [arxiv:2210.08402 §3.1] kept = [r for r in kept if scorer(r) >= threshold] # 4. Deduplicate; say whether matching is exact or near # [arxiv:2101.00027 app. D] kept, log["duplicate"] = dedup(kept) # 5. Label by people, and accept a batch only if it passes the gold set # [arxiv:1802.07856 §3.3] batches = label(kept) accepted = [b for b in batches if passes_qc(b, gold)] # 6. Split by group (speaker, vehicle, source), not by row # [arxiv:2205.12446 §2.1] train, test = group_split(accepted) # 7. Publish the counts and the removed-content log with the data # [arxiv:2104.08758 §6] return train, test, log ``` ## Contradictions - **Keep noisy pairs or filter them.** CC12M keeps a looser pool and reports lower caption precision (76.6% against 90.3%), yet its fine-tuned results improve [arxiv:2102.08981 §2.3] [tab. 3]. BLIP removes mismatched web text and its synthetic captions with a learned filter [arxiv:2201.12086 §3.3]. DataComp reports that filtering beats raw size at fixed compute [arxiv:2304.14108 §4.1]. The papers use different models and tasks, and the wiki does not resolve which holds in general. - **Size or curation for instruction data.** LIMA argues that about 1,000 curated examples are enough to teach style, and that quality and variety matter more than count [arxiv:2305.11206 §5]. OpenAssistant releases about 161,443 messages and reports gains from tuning on them [arxiv:2304.07327 tab. 1]. The two use different base models and evaluations. - **Whether filtering helps a language model at all.** GottBERT reports no clear effect from its filters [arxiv:2012.02110 tab. 3]. The phi-1 ablation reports a gain from a classifier-based code filter at 350M parameters [arxiv:2306.11644 §2.1]. These are different data and scales. - **Whether automatic filters fix quality.** On CCAligned, a Transformer language-ID filter applied to the corpora below 50% correct raised median precision from 13.8% to 43.9%, but cut recall by 77.5% [arxiv:2103.12028 §4.3]. The audit's conclusion is that automatic LangID filters are not a fix [arxiv:2103.12028 §7]. DataComp reports that filtering beats raw size at fixed compute [arxiv:2304.14108 §4.1]. The filters, languages and metrics differ, so neither result settles the question. ## Papers - [Places: An Image Database for Deep Scene Understanding](/papers/1610.02055): scene-centric photo collection built from web search with human confirmation. - [VGGFace2](/papers/1710.08092): face dataset built by a staged filtering pipeline for pose and age variety. - [xView](/papers/1802.07856): satellite object-detection dataset with a three-stage quality check. - [Training Tips for the Transformer Model](/papers/1804.00247): evaluation by learning curves rather than single scores. - [Speech Commands](/papers/1804.03209): one-second word clips with filtering and hash-based splits. - [SMHD](/papers/1806.05258): Reddit diagnosis dataset with matched control users. - [Deep Learning for Classical Japanese Literature](/papers/1812.01718): character-image datasets with imbalanced classes and domain transfer. - [CamemBERT](/papers/1911.03894): French pretraining, with data-size and data-source ablations. - [One Thousand and One Hours](/papers/2006.14480): self-driving motion dataset from one route. - [2WikiMultiHopQA](/papers/2011.01060): template-built multi-hop QA with evidence triples. - [GottBERT](/papers/2012.02110): German pretraining on OSCAR, with and without filtering. - [The Pile](/papers/2101.00027): 22-source English corpus with documentation and partial dedup. - [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](/papers/2012.15761): four rounds of model-in-the-loop hate-speech data with label-flipping perturbations. - [Did Aristotle Use a Laptop?](/papers/2101.02235): implicit-reasoning QA with model-in-the-loop filtering. - [Conceptual 12M](/papers/2102.08981): relaxed-filter web image-caption pairs. - [WebFace260M](/papers/2103.04098): web face dataset cleaned by CAST self-training, with the FRUITS time-limited test. - [Quality at a Glance](/papers/2103.12028): manual audit of 205 web-crawled multilingual subsets. - [Documenting Large Webtext Corpora](/papers/2104.08758): documentation and audit of C4. - [DialogSum](/papers/2105.06762): spoken dialogue summarisation dataset with annotation checks. - [A Diverse Corpus for Math Word Problem Solvers](/papers/2106.15772): ASDiv, with a lexical diversity measure. - [PubTables-1M](/papers/2110.00061): table dataset with canonicalised markup labels. - [Omnidata](/papers/2110.04994): steerable renderer for multi-task vision data from 3D scans. - [LAION-400M](/papers/2111.02114): CLIP-filtered 400M image-text pairs. - [ARKitScenes](/papers/2111.08897): mobile LiDAR RGB-D indoor dataset. - [MovieNet](/papers/2007.10937): film dataset with trailers, scripts and manual character, scene and action labels, plus nine tasks. - [A survey on datasets for fairness-aware machine learning](/papers/2110.00530): 15 tabular fairness datasets with per-dataset Bayesian-network and fairness scores. - [End-to-end Jordanian dialect speech-to-text self-supervised learning framework](/papers/2609.24410): Jordanian Arabic speech from public media, with noisy-student pseudo-labelling. - [Data Collection and Quality Challenges in Deep Learning](/papers/2112.06409): survey of data-centric collection, cleaning and fairness methods. - [Cleaner Document-Oriented Multilingual Crawled Corpus](/papers/2201.06642): OSCAR 22.01 with document-level language labels. - [BLIP](/papers/2201.12086): bootstrapped captions and filtering for image-text pretraining. - [ChartQA](/papers/2203.10244): chart QA benchmark with human and generated questions. - [OPT](/papers/2205.01068): open decoder-only LMs with a deduplicated web and book mix. - [FLEURS](/papers/2205.12446): 102-language parallel speech benchmark with held-out speakers. - [Minerva](/papers/2206.14858): maths-heavy continued pretraining with LaTeX kept intact. - [RDD2022](/papers/2209.08538): multi-country road damage image dataset. - [LAION-5B](/papers/2210.08402): 5.85B CLIP-filtered image-text pairs with safety tags. - [Text Embeddings by Weakly-Supervised Contrastive Pre-training](/papers/2212.03533): E5, trained on consistency-filtered text pairs. - [Pythia](/papers/2304.01373): training suite on the Pile and a deduplicated Pile. - [OpenAssistant Conversations](/papers/2304.07327): volunteer conversation trees with rankings. - [DataComp](/papers/2304.14108): benchmark for choosing image-text training data. - [PaLM 2 Technical Report](/papers/2305.10403): multilingual, deduplicated pretraining mixture with toxicity control. - [LIMA](/papers/2305.11206): 1,000 curated examples for instruction tuning. - [Self-Instruct](/papers/2212.10560): model-written instruction and instance data from 175 seed tasks, with ROUGE-L and heuristic filters. - [Textbooks Are All You Need](/papers/2306.11644): filtered code plus synthetic textbooks for a small code model. - [ToxicChat](/papers/2310.17389): real user prompts with a human-AI labelling pipeline. - [Word Similarity Datasets for Indian Languages: Annotation and Baseline Systems](/papers/2609.34138): six translated and human-rated word-similarity sets, with baselines for three languages. - [CINIC-10 is not ImageNet or CIFAR-10](/papers/1810.03505): CIFAR-sized benchmark with added ImageNet images and equal splits. - [DeepWeeds](/papers/1810.05726): field-collected weed photos with a Negatives class and five folds. - [Phi-3 Technical Report](/papers/2404.14219): small model from filtered web and synthetic data. - [Humanity's Last Exam](/papers/2501.14249): expert-written benchmark filtered against frontier models. - [Reinforcement Learning for Reasoning in Small LLMs](/papers/2503.16219): curated maths problem set for small-model RL. - [A Holistic Assessment of the Carbon Footprint of Noor](/papers/2610.00223): full project accounting, including data storage and processing. - [BLOOM](/papers/2211.05100): 176B multilingual model trained on the ROOTS corpus. - [Description of Corner Cases in Automated Driving](/papers/2109.09607): position paper on machine-readable corner-case descriptions. - [No Language Left Behind](/papers/2207.04672): 202-language translation data from mined and backtranslated bitext. - [Objaverse-XL](/papers/2307.05663): 10.2M web-crawled 3D objects with hash dedup and render filters. - [DROID](/papers/2403.12945): in-the-wild robot demonstrations with co-training and calibration. - [Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit](/papers/2512.10092): SAE concept latents for dataset diffing, correlations, clustering and retrieval.