--- kind: concept slug: noisy-labels title: Learning with noisy labels summary: >- Web-scraped, crowd-tagged or production labels are often partly wrong. Options: clean before training (score filtering, relabelling model disagreements, or audited cross-split checks), isolate noise in the loss (sub-center ArcFace) then drop and retrain, reweight or select during training, or model uncertain labels as a third state. Trade-off: annotation time versus automatic, model-based cleanup. tags: [noisy-labels, data-cleaning, face-recognition, loss-function, partial-labels, uncertain-labels, relabelling, content-moderation] papers: ["1701.08118", "1710.08092", "1801.07698", "1901.07031", "2102.00225", "2103.04098", "2112.06409", "2208.03274", "2610.17635"] related: [dataset-curation, face-recognition, angular-margin-loss, multi-label-classification, calibration, text-classification, toxicity-detection, active-learning, inter-annotator-agreement, self-training] status: draft updated_at: 2026-11-01 --- ## Overview Large labeled sets are often built by taking search-engine results for a query (a person's name, a class name) as the label. Some of those results show something else, so a share of the labels is wrong. For face identity data the share can be large: the authors of ArcFace estimate that 47.1% to 54.4% of MS1MV0 is noisy [arxiv:1801.07698 §1], and VGGFace2 points to label noise in MS-Celeb-1M, whose images were not manually filtered [arxiv:1710.08092 §2]. Labels can also be unreliable because annotators disagree about what the label means. In two surveys of lay participants labelling German tweets as hate speech, Krippendorff's α stayed between .18 and .29, and showing the platform's definition did not consistently raise it [arxiv:1701.08118 §5]. Measuring agreement first tells you whether cleaning is the right fix at all. Wrong labels matter most for margin-based classification losses, which assume each sample belongs to its labeled class and pull it toward that class's center [arxiv:1801.07698 §1, §3.2]. Crowd-tagged multi-label data has a related problem: each image carries candidate tags, and some of them are not true [arxiv:2610.17635 §1]. The wiki's cards cover four ways to deal with this: 1. **Clean before training.** Score every sample with a classifier, then drop or send to annotators the ones that don't fit their label [arxiv:1710.08092 §4.4, §4.6]. A variant relabels only the samples where a first model disagrees with the human label, and feeds that model's prediction to the second model [arxiv:2102.00225 §3]. Another splits the training set in two, trains one model per half, and flags samples that the other half's model disagrees with [arxiv:2208.03274 §3.2]. For a web-scale set, the cleaning can repeat over several rounds, each with a new teacher trained on the previous round's output [arxiv:2103.04098 §3]. 2. **Isolate during training, then clean.** Change the loss so that noisy samples gather apart from clean ones, use that to drop them, and retrain on what is left [arxiv:1801.07698 §3.2]. 3. **Reweight or select during training.** Skip the samples with the highest loss in each mini-batch and update on the rest [arxiv:2112.06409 §4.3], or weight each candidate label by a calibrated estimate that it is correct [arxiv:2610.17635 §3, §4.1]. 4. **Model uncertain labels as their own state.** Labels mined from reports can say "uncertain" rather than yes or no. The CheXpert card treats this as clinical ambiguity to model, not as a wrong label [arxiv:1901.07031 §Problem, §Uncertainty Approaches]. Families 1 to 3 use a model trained on the noisy data itself to decide what is noise. The exception is the first teacher in the iterated CAST variant, which is trained on a different public set (MS1MV2) before it cleans the target data [arxiv:2103.04098 §3]. Families 1 and 2 also end with a final model trained on the cleaned set, not on the raw one. In family 4 the uncertain label is a deliberate output of the labeler, so no cleaning step is implied. For the wider pipeline (collection, deduplication, contamination) see [dataset curation](/concepts/dataset-curation). ## Variants ### Measure annotator agreement before cleaning (hate speech surveys) Before choosing a cleaning method, check whether the labellers agree with each other. The paper's measure is Krippendorff's α, computed per question [arxiv:1701.08118 §4, §5]. The authors cite .80 as the reference for content analysis and .66 as the lowest value they accept when some uncertainty is tolerable [arxiv:1701.08118 §5]. A pre-study of six researchers labelling 541 tweets reached α = .38 [arxiv:1701.08118 §3]. In the surveys, lay participants saw 20 tweets each, and the hate speech, ban and offensiveness questions were analysed separately [arxiv:1701.08118 §4, §5, fig. 1]. One group was shown Twitter's hateful-conduct definition and the other was not. Agreement stayed low in both, so the definition did not reliably raise it [arxiv:1701.08118 §5]. The definition group marked more tweets for banning (32.6% against 17.6%, p = .01), but the paper reports no agreement gain for that question [arxiv:1701.08118 tab. 1, §5]. These are author-reported figures from samples of 25 and 31 participants, with no correction for the three tests [arxiv:1701.08118 tab. 1]. The authors' stated next step is a coding scheme with clearer criteria [arxiv:1701.08118 §6]. Card: [Measuring the Reliability of Hate Speech Annotations](/papers/1701.08118). ### Classifier-score filtering with score-banded human review (VGGFace2) VGGFace2 controls label noise at several stages of its collection pipeline [arxiv:1710.08092 §4]: - **Vet labels first.** Annotators look at 100 images per candidate name and keep the name only if about 90% or more of them show one person [arxiv:1710.08092 §4.1]. Ambiguous labels are removed before any data is downloaded at scale. - **Automatic filter.** For each identity, faces from its top-100 search results are positives and faces from all other identities' top-100 results are negatives. 1-vs-rest classifiers on VGGFace descriptors score every face in [0, 1], and faces below 0.5 are dropped. The threshold was set by inspecting 500 random identities [arxiv:1710.08092 §4.4]. The classifier type is not stated. - **Merge or drop confused classes.** Each class is split in half; a ResNet-50 is trained on one half and its top-1 confusion matrix on the other half reveals overlapping identities. 20 confused subjects were found and 19 noisy classes removed [arxiv:1710.08092 §4.6]. - **Banded manual cleaning.** A retrained classifier sorts each identity's images into H [1, 0.95], I (0.95, 0.8] and L (0.8, 0.5]. Annotators clean only L, unless H itself contains several people, in which case they clean all three bands. Band I is then cleaned using predictions of a model trained on the cleaned H and L. Target purity is above 96% [arxiv:1710.08092 §4.6]. The final filtering stage took 21 days of human time and cut 5.45M images to 3.31M [arxiv:1710.08092 tab. 2]. The paper has no ablation that isolates the effect of noise removal; its comparisons swap the whole training set. A ResNet-50 trained from scratch on VGGFace2 reaches 3.9% top-1 error on the VGGFace2 test set, versus 5.6% when trained on MS-Celeb-1M [arxiv:1710.08092 tab. 3], and 0.895 versus 0.851 TAR@FAR=0.001 on IJB-A [arxiv:1710.08092 tab. 6]. Card: [VGGFace2: A dataset for recognising faces across pose and age](/papers/1710.08092). ### Disagreement relabelling with the first model's prediction as input (Guo, 2021) The author fine-tunes BERT on 2,000,000 hand-labelled items (Model-A), runs it on the full labelled pool, and marks every item whose predicted label differs from the human label. That flags 300,000 items, about 14.7% of each split [arxiv:2102.00225 §3, tab. 1]. Annotators relabel the flagged items. They see both the old label and Model-A's guess, but decide for themselves [arxiv:2102.00225 §3]. A second BERT model (Model-C) is trained on the corrected labels, with Model-A's one-hot prediction fed in as an extra input [arxiv:2102.00225 §3]. At inference, Model-A runs first, so each query needs two forward passes [arxiv:2102.00225 §6]. Reported results on one private industrial dataset, BERT-Base, one run per model [arxiv:2102.00225 tab. 2]: Model-A 83.3% test accuracy, a model trained on the corrected labels without the prediction input 91.7%, and Model-C 92.5%. The text never defines the middle model; its meaning is inferred from the figure and abstract [arxiv:2102.00225 §3]. No variance is reported, and the step from the middle model to Model-C is 0.8 points. Two caveats from the card affect how these numbers read. The 40,000-item test split goes through the same relabelling, and the paper does not say which labels its accuracies are scored against. The flagging step also runs Model-A on its own training data, so the flagged set reflects in-sample disagreement [arxiv:2102.00225 §3, tab. 1]. Card: [Learning From How Humans Correct](/papers/2102.00225). ### Cross-split mislabel detection with an audit gate (undesired-content classifier) The OpenAI moderation work checks its training set periodically. The set is split in two, one model is trained on each half, and each model scores the other half. A sample is flagged when the prediction disagrees with its label [arxiv:2208.03274 §3.2]. A random part of the flagged samples is audited by people. If more than 30% of the audited ones are mislabeled, every flagged sample is labeled a second time [arxiv:2208.03274 §3.2]. Labels come from written rules, and regular calibration sessions review disagreements between external annotators and internal auditors [arxiv:2208.03274 §3.2]. The same card reports label-quality audits that sample 10 undesired items and 10 items above 50% model probability per category, and compute F1 with auditor labels as ground truth [arxiv:2208.03274 §3.2]. The card records no measured effect of the mislabel-detection step on its own; the paper reports results for the whole pipeline, not for this step. Card: [A Holistic Approach to Undesired Content Detection in the Real World](/papers/2208.03274). ### Iterated self-training cleaning for web-scale identities (CAST) CAST cleans a web-scraped face set in rounds. Every round starts from the same pre-processed faces, not from the previous round's output. A teacher network embeds each face, and each name's folder is clustered with DBSCAN. Only the largest cluster is kept, and only when it has more than two faces [arxiv:2103.04098 §3]. The kept faces give each folder a centre. Two folders are merged when their centres have cosine above 0.7, and the smaller one is dropped when the cosine is between 0.5 and 0.7 [arxiv:2103.04098 §3]. A new student is trained on the cleaned folders, and it becomes the teacher for the next round. The first teacher is an ArcFace ResNet-100 trained on MS1MV2 [arxiv:2103.04098 §3]. After the last round, the paper removes duplicates within each identity (cosine above 0.95) and drops identities that overlap the benchmark sets and its own test set (cosine above 0.7) [arxiv:2103.04098 §3]. The authors use three rounds; a fourth gave saturated results [arxiv:2103.04098 §5.4]. With the same network throughout, the paper reports IJB-C rising from 93.45 on raw MS1M to 96.55 after three rounds of CAST, with 4.9M faces left from 10M. For that run the first teacher is trained on IMDB-Face, not MS1MV2 [arxiv:2103.04098 tab. 7, §5.4]. On MS1M, DBSCAN gives 96.55 IJB-C against 96.03 for K-Means and 96.48 and 96.42 for two graph-based cleaners [arxiv:2103.04098 tab. 8]. The WebFace42M set, cleaned this way, reaches 97.70 on IJB-C [arxiv:2103.04098 tab. 7]. Caveats from the card. No official code is linked, and the paper does not state the DBSCAN eps and min_samples, how a folder's centre is formed, or what happens below cosine 0.5 [arxiv:2103.04098 §3]. The cleaned data is licence-gated for research use [arxiv:2103.04098 §6], so the dataset is not a drop-in for commercial training. The card has the full list of gaps. Card: [WebFace260M](/papers/2103.04098). ### Sub-center ArcFace: let the loss separate noise, then drop it Sub-center ArcFace gives each class $K$ L2-normalized weight vectors instead of one. The class score is the maximum cosine over its sub-centers, and the usual ArcFace margin loss is applied to these class scores [arxiv:1801.07698 eq. 7]: $$ L_7 = -\log\frac{e^{s\cos(\theta_{y_i}+m)}}{e^{s\cos(\theta_{y_i}+m)}+\sum_{j=1,j\neq y_i}^{N}e^{s\cos\theta_{j}}},\quad \theta_{j} = \arccos\Big(\max_k W^T_{j_k} x_i\Big),\ k\in\{1,\dots,K\} $$ [arxiv:1801.07698 eq. 7] A mislabeled sample only has to get close to one of its class's $K$ sub-centers, so it tends to form a separate, non-dominant sub-class instead of dragging the main center toward it [arxiv:1801.07698 §1, §3.2]. On MS1MV0 with ResNet50 and $K=3$, the authors measure 57.24% easy clean and 12.40% noisy samples in dominant sub-classes and 4.28% hard clean and 26.08% noisy samples in non-dominant ones; with $K=1$, 38.47% noisy samples sit near the class center [arxiv:1801.07698 §3.2, fig. 7]. Cleaning then runs offline: drop each class's non-dominant sub-centers, drop samples more than 75° from the remaining dominant one, and retrain from scratch with $K=1$ [arxiv:1801.07698 §3.2, §4.1]. How the dominant sub-center is picked is not stated explicitly; the paper describes it as the sub-class holding most of the clean samples [arxiv:1801.07698 §1, §3.2]. Author-reported ablation, ResNet50, TPR@FPR=1e-4 [arxiv:1801.07698 tab. 3]: | Setting | IJB-B | IJB-C | |---|---|---| | MS1MV0, K=1 | 87.87 | 90.27 | | MS1MV0, K=3 | 91.70 | 93.72 | | MS1MV0, K=3, softmax pooling | 91.53 | 93.55 | | MS1MV0, K=5 | 91.47 | 93.62 | | MS1MV0, K=10 | 63.84 | 67.94 | | MS1MV0, K=3↓1, drop >70° | 94.44 | 95.91 | | MS1MV0, K=3↓1, drop >75° | 94.56 | 95.92 | | MS1MV0, K=3↓1, drop >80° | 94.04 | 95.74 | | MS1MV0, K=3↓1, drop >85° | 93.33 | 95.10 | | MS1MV0, K=3, SoftTriple regularizer | 91.53 | 93.64 | | MS1MV0, Co-mining | 91.80 | 93.82 | | MS1MV0, NT | 91.57 | 93.65 | | MS1MV0, NR | 91.58 | 93.60 | | MS1MV3, K=1 | 95.13 | 96.50 | | MS1MV3, K=3 | 94.84 | 96.35 | | MS1MV3, K=3↓1 | 94.87 | 96.43 | | Celeb500K, K=1 | 90.96 | 92.15 | | Celeb500K, K=3 | 93.76 | 94.90 | | Celeb500K, K=3↓1 | 95.65 | 96.91 | [arxiv:1801.07698 tab. 3] Reading the table: training with $K=3$ alone already helps on noisy data, roughly on par with the noise-robust baselines Co-mining, NT and NR as run by the authors; the drop-and-retrain step adds most of the remaining gain. Thresholds of 70°–80° perform alike. On the already cleaned MS1MV3, sub-centers bring no gain. Too many sub-centers ($K=10$) collapse training [arxiv:1801.07698 tab. 3, §4.3]. The authors use this procedure to clean MS1MV0 and Celeb500K and merge them into IBUG-500K (11.96M images, 493K identities) [arxiv:1801.07698 §4.1]. Card: [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](/papers/1801.07698). ### Small-loss selection during training (survey taxonomy) The survey sorts noisy-label training methods into loss correction, sample selection, hybrid, early-stopping and two-network families. It runs no experiments and gives no ranking [arxiv:2112.06409 §1, §4.3, tab. 1]. Its most detailed example is small-loss selection (Co-teaching): in each mini-batch, run a forward pass, keep the fraction $1-\tau$ with the lowest loss as presumed clean, and update only on those [arxiv:2112.06409 §4.3]. The noise rate $\tau$ is taken as given, and the survey does not say how to estimate it [arxiv:2112.06409 §4.3]. Other named members, each described only in words: - **SELFIE** keeps low-loss samples as clean and relabels consistent samples that disagree with their label to the most frequent prediction, discarding the rest [arxiv:2112.06409 §4.3]. - **MORPH** estimates when to stop early by fitting a two-component Gaussian mixture to the loss distribution, without a clean validation set [arxiv:2112.06409 §4.3]. - **DivideMix** uses two networks, each fitting a Gaussian mixture to per-sample losses to split data into a likely-clean and a likely-noisy set, then trains with semi-supervised learning [arxiv:2112.06409 §4.3]. The survey cites label-error rates of 8–38% across several real datasets, which it does not verify [arxiv:2112.06409 §4.3]. Card: [Whang et al., data-centric AI survey](/papers/2112.06409). ### Calibrated per-tag weights for partial multi-label data (PML-CD) This variant targets partial multi-label learning. Each image has candidate positive tags, some of them wrong, and the method keeps all candidates but weights each one. The weight for a candidate is the probability that it is correct, given the model's confidence $t$ in it [arxiv:2610.17635 §3, eq. 3–4]. Estimating that probability works as follows [arxiv:2610.17635 §4.1]: - For each class, count the confidences on candidate positives in 20 bins, and split each bin into true and false positives. - The reliability of a bin is its share of true positives. A learned two-layer MLP maps the histogram to these reliabilities, and a confidence gets its weight by interpolating between neighbouring bins [arxiv:2610.17635 eq. 5–7]. The calibrator is pre-trained once on VOC 2007, using an identity-weighted baseline run at noise rates of 10% to 90%, and then reused unchanged on other datasets and noise rates [arxiv:2610.17635 §4.1, §5.1]. Training has two phases: warm-up with plain BCE on the raw candidates, then a weighted classification loss plus a prototype cosine loss, with the weights refreshed each epoch [arxiv:2610.17635 §4.2, eq. 12–13, alg. 1]. Author-reported results, ResNet-101, mAP in % [arxiv:2610.17635 tab. 1]: - Best in 15 of 16 settings: four datasets (VOC, COCO, NUS-WIDE, CUB) at four noise rates (20% to 80%), against nine baselines. The exception is VOC at 20%, where LL-R is higher. - On COCO at 80% noise, the pre-trained calibrator gives 61.48 mAP in one epoch-1 comparison; weights from the true labels give 66.42. The true-label weighting is an upper bound, not a method one can deploy [arxiv:2610.17635 tab. 2]. - The prototype term helps mainly at high noise. On VOC at 80% it moves mAP from 66.91 to 71.70 [arxiv:2610.17635 tab. 3]. Caveats from the card. The paper does not state the number of runs or seeds, and gives no variance [arxiv:2610.17635 §5.1]. Its Table 1 and Table 3 disagree on several VOC values, with no explanation [arxiv:2610.17635 tab. 1, tab. 3]. The authors' released code differs from the paper in several documented ways, including a rescaling of every weight that the paper does not describe. The released code also selects checkpoints on the test split, which the paper does not state; that could bias the reported numbers upward. The card lists these differences in full. Card: [Calibrated Disambiguation for Partial Multi-label Learning](/papers/2610.17635). ### Uncertain labels as a third state (CheXpert) The CheXpert labeler reads radiology reports and gives each of 14 observations one of three values: positive, negative or uncertain. The uncertain share is large for some observations. For Consolidation it is 12.78% of training studies, against 6.78% positive [arxiv:1901.07031 tab. 1]. The paper compares five ways to train on the uncertain value [arxiv:1901.07031 §Uncertainty Approaches]: - **U-Ignore.** Mask the uncertain observations out of the loss. The card notes that this is biased when the uncertain rate is high, as for Consolidation [arxiv:1901.07031 §Ignoring]. - **U-Zeros and U-Ones.** Map every uncertain value to 0 or to 1, then train with ordinary BCE [arxiv:1901.07031 §Binary Mapping]. - **U-SelfTrained.** Train U-Ignore to convergence, then use its probabilities as soft targets for the uncertain entries [arxiv:1901.07031 §Self-Training]. The number of rounds and the stopping rule are not stated. - **U-MultiClass.** Give each observation a three-way softmax over negative, positive and uncertain [arxiv:1901.07031 §3-Class Classification]. On a 200-study validation set, measured by AUROC with DeLong intervals, U-Ones is best for Atelectasis (0.858) and U-MultiClass for Cardiomegaly (0.854). The paper reports significant differences only for those two observations, and no significant difference for Consolidation, Edema or Pleural Effusion [arxiv:1901.07031 tab. 3, §Results]. For the test set, the card says each task used the approach that scored best on the validation ensemble [arxiv:1901.07031 §Test Results]. Card: [CheXpert](/papers/1901.07031). The labeler's code is MIT-licensed; no training code for the image models is linked, so the paper's description is the only source for them. ## When to use which | Situation | Use | Why | |---|---|---| | Subjective labels from lay annotators (hate speech, offensiveness), agreement not yet measured | Measure α per question before cleaning; revise the guidelines if α is low | Surveys gave α between .18 and .29, and a definition alone did not reliably raise it [arxiv:1701.08118 §5, tab. 1]. Small samples, no multiple-comparison correction [arxiv:1701.08118 tab. 1] | | Building a new labeled set from search results, with annotation budget | VGGFace2-style label vetting, score filter and banded review | Annotators see only the uncertain band unless the confident band is mixed; purity target above 96% [arxiv:1710.08092 §4.1, §4.6] | | Very large web-scraped identity set, no manual budget, compute for several full training runs | Iterated self-training cleaning (CAST): cluster, merge, retrain, three rounds | Fully automatic at millions of identities; same ResNet-100 on MS1M goes from 93.45 to 96.55 IJB-C after three rounds [arxiv:2103.04098 tab. 7]. Each round is a full training run, and the DBSCAN parameters are not stated [arxiv:2103.04098 §3, §5.4] | | Existing large set with heavy noise, no annotation budget | Sub-center ArcFace, $K=3$, drop >75°, retrain with $K=1$ | Fully automatic; +5.65 IJB-C points on MS1MV0 and +4.76 on Celeb500K over plain $K=1$ training [arxiv:1801.07698 tab. 3] | | Trained model in place, human relabelling budget for the disputed items | Relabel where the model disagrees, then train with its prediction as input | Single private dataset, one run per model: 83.3% to 92.5% test accuracy [arxiv:2102.00225 tab. 2]. Costs a second forward pass per query [arxiv:2102.00225 §6] | | No time for a second training run | Train once with $K=3$ | Gets part of the gain on noisy data (90.27 → 93.72 IJB-C on MS1MV0), but well below drop-and-retrain [arxiv:1801.07698 tab. 3] | | Production labels from rules and annotators, with a team that can audit a sample each round | Cross-split mislabel detection; relabel all flagged items only if the audit finds more than 30% wrong | Catches disagreements without a second labeled set; the threshold is the authors' choice for their own labels [arxiv:2208.03274 §3.2]. The card records no measured effect for this step alone [arxiv:2208.03274 §3.2] | | Data already cleaned | Plain $K=1$ training | $K=3$ and $K=3$↓1 are slightly worse than $K=1$ on MS1MV3 [arxiv:1801.07698 tab. 3] | | Each tag is a candidate that may be wrong (crowd over-tagging), multi-label | Calibrated per-tag weights (PML-CD) | Weights come from the model's own confidences, not a hand-set threshold; best mAP in 15 of 16 author-reported settings [arxiv:2610.17635 §1, tab. 1]. The calibrator recipe must be rebuilt from the paper's description, which gives no seeds or epoch counts [arxiv:2610.17635 §5.1] | | Labels are hedged by a rule-based extractor ("possible", "cannot exclude") | U-Ones or U-MultiClass, chosen per observation on validation data | No single approach wins for every observation; U-Ignore is biased when the uncertain rate is high [arxiv:1901.07031 tab. 3, §Ignoring] | | Classifier memory is tight (many classes) | Keep $K$ small | The classifier grows by a factor of $K$; $K=3$ was chosen partly for GPU memory, and $K=10$ collapsed [arxiv:1801.07698 §4.3, tab. 3] | | Labels themselves may be ambiguous (two people sharing a name, merged classes) | Confusion-matrix check on a held-out half of each class | Per-sample filters don't catch overlapping classes; VGGFace2 found 20 confused subjects this way [arxiv:1710.08092 §4.6] | | Training a classifier end to end, no separate cleaning step, and you can estimate the noise rate | Small-loss selection per mini-batch | Needs $\tau$ as an input; the survey gives no estimation method [arxiv:2112.06409 §4.3] | | Noise rate unknown | Sub-center ArcFace or a cleaning pass | Sub-center ArcFace uses a fixed angle (75°), not a rate, though that value was tuned on the authors' data [arxiv:1801.07698 §4.1, tab. 3] | ## Implementation sketch ```text # Step 0: agreement per question (Krippendorff's alpha) [arxiv:1701.08118 §5] for each question q: alpha_q = krippendorff_alpha(labels_by_annotator[q]) # low alpha: revise the guidelines or split the question before cleaning # Family 1: filter, then review (VGGFace2-style) for each class c: seed = most_confident_samples(c) # e.g. top-100 search hits clf_c = train_one_vs_rest(seed, seeds_of_other_classes) drop samples with clf_label(x) < t # t = 0.5 detect overlapping classes via confusion on held-out halves; merge or drop retrain classifier; band each class by score: H, I, L human_review(L); if H is mixed: human_review(H, I, L) clean I with a model trained on cleaned H and L # Family 1b: relabel model disagreements, prediction as input (Guo, 2021) model_a = fine_tune(bert, train_split) flagged = [x for x in pool if model_a(x) != x.human_label] corrected = relabel_by_hand(flagged, hints = model_a(x)) model_c = fine_tune(bert, merged, extra = one_hot(model_a(x)), target = corrected) # inference: model_c(one_hot(model_a(text)), text) -- two forward passes # Family 1c: cross-split mislabel check with audit gate (arxiv:2208.03274 §3.2) halves = split(train_set, 2) model_0 = train(halves[0]) model_1 = train(halves[1]) # each half is scored by the model trained on the other half flagged = [x for x in halves[1] if model_0(x) != x.label] flagged += [x for x in halves[0] if model_1(x) != x.label] audited = random_part(flagged) # human audit if fraction_mislabeled(audited) > 0.30: relabel_all(flagged) # second labeling pass # Family 1d: iterated self-training cleaning (CAST, arxiv:2103.04098 §3) teacher = arcface_r100_trained_on(ms1mv2) # round-0 teacher, not on the target set for round in 1..3: emb = teacher.embed(all_faces) # 512-d, same raw pool every round for each name folder: keep largest DBSCAN cluster if size > 2 # eps, min_samples not stated merge folders whose centres have cosine > 0.7 drop smaller folder if cosine in [0.5, 0.7] # below 0.5: not stated teacher = train_arcface_r100(cleaned) # student becomes next teacher remove within-identity duplicates (cosine > 0.95) drop identities overlapping benchmarks (cosine > 0.7) # Family 2: sub-center loss, then drop and retrain (sub-center ArcFace) model = train(data, head = ArcFace with K = 3 sub-centers, class score = max over sub-centers) for each class c: keep only the dominant sub-center of c # selection rule not stated in the paper for each sample x with label c: if angle(embed(x), center_c) > 75 deg: drop x model = train_from_scratch(remaining, head = ArcFace with K = 1) # Family 3: small-loss selection inside training (survey taxonomy) for each mini-batch (x, y_noisy): loss_i = per_sample_loss(model, x, y_noisy) keep = lowest (1 - tau) fraction of loss_i # tau assumed given update model on mean(loss_i[keep]) # Family 3b: calibrated per-tag weights (PML-CD) warm up with BCE on raw candidate tags for each epoch: conf = model_clf(images) # probabilities, no grad hist = 20-bin histogram of conf on candidate positives, per class rel = calibrator(hist) # reliability per bin w = interpolate(conf, rel) on candidate positives; 1 elsewhere loss = weighted_bce(clf, tags, w) + weighted_bce(proto, tags, w) # Family 4: uncertain as a third state (CheXpert) # target per observation: 1, 0, or 'u'; choose one of: U-Ignore: mask 'u' out of the BCE sum U-Zeros/Ones: map 'u' to 0 or 1, then BCE U-SelfTrained: teacher = U-Ignore model; soft target = teacher prob for 'u' U-MultiClass: 3-way softmax (neg, pos, uncertain) per observation ``` Except for the uncertainty family, the signal in every family comes from a model trained on the same labels: the classifier itself, the first model in relabelling, the teacher in U-SelfTrained. The one exception is CAST's first teacher, which is trained on MS1MV2 [arxiv:2103.04098 §3]. Parameters the cards report as not stated must be chosen by the implementer: the stage-4 classifier type in VGGFace2 [arxiv:1710.08092 §4.4]; the DBSCAN eps and min_samples in CAST [arxiv:2103.04098 §3]; the dominant-sub-center rule in ArcFace [arxiv:1801.07698 §3.2]; the noise rate $\tau$ in small-loss selection [arxiv:2112.06409 §4.3]; the number of relabelling rounds in U-SelfTrained [arxiv:1901.07031 §Self-Training]; and the PML-CD warm-up and disambiguation lengths, which the paper leaves unstated [arxiv:2610.17635 §4]. ## Contradictions The papers don't compare their methods on a common dataset. Scores use different metrics (top-1 error, TPR@FPR, mAP, accuracy, AUROC, AUPRC) on different data, so they cannot be ranked against each other. The survey's 8–38% label-error range [arxiv:2112.06409 §4.3] and ArcFace's 47.1–54.4% estimate for MS1MV0 [arxiv:1801.07698 §1] come from different datasets and measurement methods, so they are not directly comparable. Within the wiki, one paper's own claim conflicts with earlier work. The moderation paper reports that adding zero-shot and few-shot synthetic data, at twice the size of its labeled training set, reduced performance. It attributes this to label noise and says it contradicts earlier work in low-data settings. The card does not name that earlier work [arxiv:2208.03274 §3.3]. ## Papers - [Measuring the Reliability of Hate Speech Annotations](/papers/1701.08118): two surveys find that showing a hate speech definition does not reliably raise annotator agreement (α .18 to .29). - [VGGFace2](/papers/1710.08092): builds a face dataset with name vetting, classifier-score filtering, confusion-based class removal and score-banded manual cleaning. - [CheXpert](/papers/1901.07031): chest radiograph labels from reports with a third "uncertain" value, and five ways to train on it. - [Learning From How Humans Correct](/papers/2102.00225): relabels items where a first BERT model disagrees with the human label, then trains a second model with the first model's prediction as input. - [A Holistic Approach to Undesired Content Detection in the Real World](/papers/2208.03274): flags training samples that a model trained on the other half disagrees with, with an audit gate before relabelling. - [WebFace260M](/papers/2103.04098): builds a web-scale face set and cleans it with CAST, three rounds of cluster, merge and retrain; the cleaned set is licence-gated. - [ArcFace](/papers/1801.07698): sub-center ArcFace isolates noisy samples in non-dominant sub-centers so they can be dropped automatically before retraining. - [Data Collection and Quality Challenges in Deep Learning](/papers/2112.06409): survey of data-centric methods; its noisy-label section covers small-loss selection, SELFIE, MORPH and DivideMix without experiments of its own. - [Calibrated Disambiguation for Partial Multi-label Learning](/papers/2610.17635): weights each candidate tag by a calibrated estimate of its probability of being correct.