Toxicity detection

Overview

The task is to decide whether a piece of text is toxic, usually as a binary label. The shared definition in ToxicChat follows the SemEval-2019 undesired-content guidelines 2310.17389 §2.1. A second, separate label marks jailbreaking prompts, which try to get a chatbot to produce toxic output through prompt tricks 2310.17389 §2.

The main problem is domain shift. Most detectors were built on social-media or LLM-generated text. Chatbot users write questions and instructions, and they can hide intent behind harmless wording, so detectors trained elsewhere may miss toxic chatbot traffic 2310.17389 §1 fig. 1.

Toxic language is one of the 21 risks in a taxonomy of language-model harms, under discrimination and exclusion 2112.04359 §2.1. That paper reports no experiments and no detector, so it frames the problem but gives no numbers.

Variants

Eleven papers cover this page. ToxicChat is the only one that compares detectors on one chatbot test set. KUISAIL and Vidgen et al. also compare models, on the OffensEval 2020 test sets and on HateCheck. The others cover other routes: a byte-level multilingual model for production, crowd labels on a large corpus, machine-generated text, several annotators per item, rounds of annotators writing against a live model, and a production pipeline that mixes public data, active learning and audits. Two more papers study the labels themselves and a generation-time filter that does not classify input at all. Unless noted, numbers are author-reported.

Off-the-shelf moderation tools

Pretrained tools are used as they are, with no training on the target data. Those tested are OpenAI Moderation (binary output), the Perspective API (score threshold 0.7), HateBERT and ToxDectRoberta 2310.17389 §3 2310.17389 app. A. On ToxicChat, F1 ranges from 5.3 (Perspective) to 34.6 (ToxDectRoberta). OpenAI Moderation has precision 84.3 but recall 11.7. HateBERT has the highest jailbreaking recall of this group (60.5) with precision 6.3 2310.17389 tab. 1.

In-domain fine-tuning

A roberta-base classifier is trained on the ToxicChat training half and tested on the test half. It reaches precision 75.9, recall 68.7, F1 72.1 and jailbreaking recall 83.5 2310.17389 tab. 2. Card: ToxicChat. The paper's own test set is the only evaluation, so this does not show how the model performs on other chatbots 2310.17389 Limitations.

Out-of-domain fine-tuning

The same roberta-base setup is trained on one public toxicity dataset at a time, then tested on ToxicChat. Six datasets were tried: HSTA, Movie Reviews, Jigsaw, ToxiGen, RealToxicPrompts and ConvAbuse 2310.17389 §4.1. The best single source was ConvAbuse (F1 51.6). Movie Reviews gave F1 0.0. Training on the combination of all six gave F1 42.7, below the best single source 2310.17389 tab. 2. Subsample sizes other than HSTA's 8,000 rows are not stated 2310.17389 app. A.

Input or output features

Three feature sets were tried with roberta-base, all trained and tested on ToxicChat: the user prompt, the Vicuna response, and both together. The user prompt alone gives F1 72.1. The response alone gives 63.3. Both together give 72.6, which the authors describe as not significant 2310.17389 tab. 3 2310.17389 §4.2. Using the response needs a response to exist, so pre-generation screening can use the prompt only.

Language-specific BERT with a CNN head

KUISAIL at SemEval-2020 Task 12 fine-tunes one BERT-base model per language and puts a CNN on top. The CNN takes the outputs of the last four encoder layers as input channels. It has 160 filters in five heights, from one to five tokens, with max pooling and a sigmoid output 2007.13184 §4.4. It labels tweets as offensive or not in Arabic, Greek and Turkish, on the OffensEval 2020 subtask A data 2007.13184 §3.

• Author-reported macro-F1 on the test sets is 0.897 for Arabic, 0.843 for Greek and 0.814 for Turkish, an average of 0.851. Language-specific BERT alone averages 0.841, and multilingual BERT 0.796 2007.13184 tab. 2.
• The gain over BERT alone is +0.013 in Arabic and +0.021 in Greek. In Turkish the CNN head is slightly lower, 0.814 against 0.816 2007.13184 tab. 2. The average gain of 0.010 therefore comes mostly from two languages.
• The paper reports single numbers with no repeated runs, so the size of the gain is uncertain 2007.13184 §5.
• Arabic uses ArabicBERT, which the authors trained and released themselves 2007.13184 §4.3.
• The paper leaves several training settings unstated, such as the loss, optimiser, dropout and decision threshold. The released code sets them, and it differs from the paper on epochs and checkpoint selection. The card's Paper vs code section lists the differences, so check it before reuse 2007.13184 §4.4.

Crowd-labelled classifier for corpus-scale labelling

This variant targets personal attacks in English Wikipedia talk comments, not general toxicity. Ex Machina collects at least 10 crowd labels per comment, trains a bag-of-n-grams classifier on them, and labels a corpus of 63M comments with it 1610.08914 §3 1610.08914 app. A.

• The best model is an MLP on character 1–5-grams, trained on the fraction of annotators who said "attack". It reaches AUC 96.59 on the dev split. Logistic regression on the same features reaches 96.24 1610.08914 tab. 2. The dev split both selects the model and reports these numbers, so they are optimistic 1610.08914 §4.2.
• Against crowd ensembles, the model beats an ensemble of three labels on AUC (97.19 against 95.49) and on Spearman. On AUC it sits level with five labels (97.13) 1610.08914 tab. 3. The paper's "as good as three crowd workers" claim is one reading of this table.
• The decision threshold of 0.425 balances false positives and false negatives. It gives precision 0.63 and recall 0.63 on dev and test together 1610.08914 §5.1. The paper ties the score to this model and label type 1610.08914 §4.1, so a different model needs its own threshold.
• Crowd agreement is modest, with Krippendorff's alpha 0.45 1610.08914 §3.
• The corpus findings show the use case. Anonymous comments carry 3.1% attacks against 0.5% for registered comments 1610.08914 §5.2. A bootstrap check places the machine-estimated attack rate inside the crowd-labelled interval 1610.08914 §5.1.

Per-annotator heads for disagreeing labels

When each text has labels from several annotators, this variant keeps them apart instead of taking a majority label first. One shared encoder feeds one output head per annotator, and each head trains only on the texts that annotator labelled. The prediction is the majority over all heads 2110.05719 §3.4. The test set is the Gab Hate Corpus, a hate-speech set with at least three annotators per post and 18 annotators in total 2110.05719 §4.1.1. Hate speech is close to toxicity but not the same label, so these numbers do not transfer directly.

• On this corpus, majority-vote F1 is 59.49 for the multi-task model and 57.32 for majority-label training. The ensemble of per-annotator models reaches 53.54 2110.05719 tab. 1. The paper calls the gap significant, but it reports no statistical test 2110.05719 §4.3.1.
• Scored against each annotator's own labels, the multi-task model reaches F1 63.20, against 60.94 for the ensemble 2110.05719 tab. 1.
• On six GoEmotions emotions, the multi-task model is below the baseline on four of six with all 82 annotators, and above it on four of six in a subset of 53 annotators with more than 1,000 labels each 2110.05719 tab. 3. This is emotion data, not toxicity, and the subset was chosen after the full-data results 2110.05719 §4.4.
• Spread across the heads is the uncertainty signal. On the Gab Hate Corpus it tracks annotator disagreement better than softmax or MC dropout, and its mean is 0.170 on wrong predictions against 0.049 on correct ones 2110.05719 §4.3.2 2110.05719 §5.2.
• Training one fold took 22.3 minutes for the multi-task model, against 158.4 for the ensemble 2110.05719 tab. 2. Only BERT-base was tested, and the approach needs labels that name the annotator 2110.05719 §3 2110.05719 §6.2.

Synthetic training data from a language model

ToxiGen generates 274,186 statements about 13 minority groups with GPT-3. The set is balanced between toxic and benign text within each group, and 98.2% of statements contain no profanity or slur by a word-list test 2203.09509 tab. 2. Its target is implicit toxicity, which detectors often miss 2203.09509 §1.

• Generation uses demonstration-based prompting, with five examples drawn at random from a group's set for each statement 2203.09509 §3.1 2203.09509 app. A.
• ALICE is a classifier-guided constrained beam search. It pushes output away from the label of its prompt. It produced 14,174 of the statements, about 5% 2203.09509 §3.2 2203.09509 tab. 2.
• Author-reported gains: HateBERT fine-tuned on ALICE and top-k output together reaches AUC 0.71, 0.67 and 0.66 on three human-written implicit-hate sets (SBF, ImplicitHateCorpus, DynaHate), against 0.60, 0.60 and 0.47 without fine-tuning 2203.09509 tab. 4. RoBERTa gains are smaller, from 0.65, 0.57 and 0.49 to 0.70, 0.66 and 0.54 2203.09509 tab. 4.
• ToxiGen-HumanVal, 792 machine-generated statements rated by humans, reaches AUC 0.96 for HateBERT 2203.09509 tab. 4 2203.09509 §4.2. It is the same kind of text as the training data, so it overstates transfer.
• The training label is the label of the prompt, which is a proxy 2203.09509 §5. Raters agreed with the prompt label for 67.7% of top-k toxic prompts and 40.3% of ALICE toxic prompts 2203.09509 §4.3. Fleiss' kappa for the human ratings is 0.46 2203.09509 §4.2.
• The fine-tuning learning rate, epochs, batch size and sequence length are not stated 2203.09509 §5.
• The released code applies the classifier term twice, so its effective weights differ from the single weighted sum in the paper's eq. 1. See the card's Paper vs code section before reusing the code 2203.09509 eq. 1.
• The arXiv LaTeX source of this paper has no table text. The numbers above were read from the PDF, so check them against it before use. See the card's Pitfalls section.

Model-in-the-loop data collection

Learning from the Worst by Vidgen et al. builds an English online hate dataset in four rounds. In each round, annotators write posts that the current target model, a RoBERTa-base classifier, gets wrong. Checked entries join the training data, and the next target is trained on the earlier rounds plus the new one 2012.15761 §1 2012.15761 §5. From round 2 on, half of each round is perturbations of earlier posts that flip their label. These were written without model feedback 2012.15761 §5.2. The release has 41,255 entries, 54% of them hateful 2012.15761 tab. 1.

• The in-loop error rate is the share of annotator entries that fooled the target. It falls from 54.7% in round 1 to 27.7% in round 4 2012.15761 tab. 3. The authors warn that a falling rate does not show how much harder later rounds became, so the rate is not comparable across rounds 2012.15761 §6.1.
• Author-reported macro-F1 on the round-4 test set rises from 60.91 for the round-1 target to 75.97 for the round-4 target, over five training seeds 2012.15761 tab. 4.
• On HateCheck, the round-4 target reaches 95% accuracy. The round-1 target reaches 60%, and Perspective's toxicity classifier, the best of four external models, reaches 77% 2012.15761 §6.3. The authors say HateCheck can show only that some weaknesses are absent 2012.15761 §6.3.
• Duplication is part of the recipe. Each target trains on earlier data with one round repeated, and the repeat factors (5, 100 and 1 across the three targets) were chosen on dev sets 2012.15761 §5 2012.15761 app. D.
• The paper releases the data and annotation guidelines, but no training code. Batch size, maximum length, warm-up and loss are not stated 2012.15761 app. D.
• Twenty annotators took part, and the authors describe their setting as one with expert-overseen annotators and a backend that can run the target with low latency 2012.15761 §4 2012.15761 §7.

Production pipeline with active learning and audits

A Holistic Approach to Undesired Content Detection in the Real World trains an eight-category classifier for a chatbot-style production system. It covers five top-level categories and three severe subcategories, with one head per category 2208.03274 §2 2208.03274 §4.1. The method combines written labeling rules, label audits, active learning on live traffic, synthetic data and domain-adversarial training 2208.03274 §3.

• Live traffic is the bottleneck. Public data is shifted from production, and some categories are rare: self-harm appears in 0.04% of sampled user prompts 2208.03274 §1. Active learning raised the share of undesired samples about 10 to 22 times per category against random sampling 2208.03274 tab. 4. Overall, about 40% of active-learning samples carry an undesired label, against 3.4% of random samples 2208.03274 §4.3.
• Label quality is checked in three ways: calibration sessions, a per-category audit of 10 undesired and 10 high-probability items, and a mislabel loop that re-labels a flagged set when more than 30% of its audited samples are wrong 2208.03274 §3.2.
• Domain-adversarial training with a Wasserstein critic is the paper's answer to public-to-production shift. Trained on public data alone, it raises average AUPRC for self-harm from .063 to .281 and for V2 from .165 to .453. With synthetic or production data in the mix, the gains are smaller. For hateful content the adversarial model is lower on the synthetic-data set, .476 against .551 2208.03274 tab. 5 2208.03274 §4.4. The paper reports no spread over seeds for the main model.
• Against Perspective API on the paper's public test set, the classifier has higher AUPRC on four categories: sexual .9703 against .8709, hateful .7968 against .6914, violence .7371 against .5201 and harassment .6191 against .3902. Some Perspective cells use approximated taxonomy mappings, and the two taxonomies differ 2208.03274 tab. 3.
• Synthetic data is used in two ways. A curated set of 69k counterfactual examples is added to the training data 2208.03274 §4.4. Large sets of GPT-3 zero-shot and few-shot examples, at twice the labelled size, reduced performance, which the authors attribute to label noise 2208.03274 §3.3. The paper's linked repository holds only an evaluation set, not the training code. See the card's Paper vs code section for its label semantics.
• The taxonomy is US-centric, and the classifier was evaluated only in English, with about 5% of its training samples non-English 2208.03274 §7.1 2208.03274 §6.
• The learning rate of 0.05 is stated as written. Optimizer, schedule and warmup are not stated, so test this value before reuse 2208.03274 §4.1.

Written definition shown to annotators

This variant changes the labelling step, not the model. Two survey groups each labelled 20 German tweets about the 2016 refugee debate. One group saw Twitter's hateful-conduct definition before each tweet and answered by that definition. The other answered from personal opinion 1701.08118 §4. Card: Ross et al. (2017), a reliability study of hate-speech labels.

• The definition did not raise agreement. Krippendorff's alpha was .18 to .29 across questions and both groups, below the .66 the authors cite as the minimum for uncertain applications 1701.08118 §5 fig. 1.
• Six researchers who knew the definitions reached alpha .38 on 541 tweets 1701.08118 §3. The survey and pre-study numbers are not directly comparable, and the paper does not claim they are.
• The definition group more often said a tweet should be banned, 32.6% against 17.6%, with p = .01 and no correction for the three group tests 1701.08118 tab. 1.
• The survey has 25 and 31 participants and 20 items each, and the paper does not state which 20 tweets were shown 1701.08118 §4 tab. 1. Treat the alpha values as a rough signal.

Moral direction as a decoding filter

This variant does not classify input. It filters next-token candidates while a language model generates. Schramowski et al. (2021), a moral direction in BERT takes a sentence embedding from BERT-large, places verbs in several question frames, and uses the first principal component of the averaged embeddings as a direction 2103.11790 § Identifying the Moral Direction of Language Models [suppl. B.1]. Projecting a phrase on this direction gives a score.

• The score tracks human yes/no judgements, with Pearson r = 0.78 and 0.79 on two panels [arxiv:2103.11790 § BERT's MoralDirection Strongly Correlates with Human Moral Norms]. The result depends on the verbs. Three random verb sets gave r = 0.64, 0.60 and -0.01 2103.11790 suppl. B.2.
• In GPT-2 decoding, top-k and top-p sampling produce the candidates. A candidate is kept if the score of the text with it appended is above a threshold t, and at least m candidates are kept [arxiv:2103.11790 § Reducing Neural Toxic Degeneration in Language Models]. The paper's runs use t = -0.5, 0 and 0.5, and m = 5 2103.11790 § Experimental setup.
• Author-reported, on the RealToxicityPrompts testbed scored with Perspective: the share of prompts with a generation above toxicity 0.9 falls to 1.17% at t = 0, against 9.82% for GPT-2 [arxiv:2103.11790 § Reducing Neural Toxic Degeneration in Language Models]. In the full table, t = 0.5 is marked best in all six toxicity columns 2103.11790 tab. 3.
• The threshold only means something on the scale the authors used. The paper gives no tuning procedure, so a different encoder, verb set or PCA fit needs its own threshold [arxiv:2103.11790 § Identifying the Moral Direction of Language Models]. The paper reports no runtime or memory cost 2103.11790. Each step scores every candidate with a BERT-large pass, so cost grows with top-k 2103.11790. See the card's Pitfalls section for details. The card's Paper vs code section covers the differences from the released code, so check it before reuse.

Byte-level multilingual encoder for production scoring

Perspective API, next generation replaces per-language word pieces with a token-free model. UTF-8 bytes pass through a learnable tokenizer (GBST, from Charformer), then a T5-style encoder-decoder 2202.11176 §3.1 2202.11176 §3.2. Pretraining is span denoising, with spans averaging 20 bytes and covering 15% of the input 2202.11176 §3.4. For fine-tuning the decoder is dropped, the first position of the encoder output is pooled, and a linear layer with a sigmoid per label gives multi-label outputs 2202.11176 §3.3.

• Production-Multilingual AUC-ROC averages .910 for the 102M model and .919 for the 268M model. mT5-base averages .906, and custom mBERT .840. Perspective API is reported only for English (.974) and Russian (.907) 2202.11176 tab. 1. The 102M model is only slightly above mT5-base on this set.
• On JMTCC, the 268M model reaches AUC-ROC .9367 and the 102M model .9194. mT5-base reaches .9239 and Perspective API .8770. The best entry is an ensemble with an estimated size above 5B parameters 2202.11176 tab. 2.
• On code-switched sets, the 102M model is below mT5-base on JMTCC-CS (.9191 against .9393). The 268M model is best on both sets, at .9446 and .9023 2202.11176 tab. 3.
• Zero-shot on TweetEval hate classification, the 102M model reaches macro F1 53.3 and mT5-base 57.7. Fine-tuned English-only models reach 52.3 to 56.4 2202.11176 tab. 4.
• On obfuscated English text, the authors report that the model beats every baseline at each obfuscation rate above 0%. After fine-tuning on Civil Comments with 30% obfuscation it reaches AUC-ROC 86.0, against 84.5 for mT5-small 2202.11176 §6.2. On identity-term templates, some Korean and Japanese terms have BPSN AUC of 0.5 or below, which the authors attribute to false positives on non-toxic templates 2202.11176 §6.4.
• On CivilComments-WILDS, the 102M model's worst-group accuracy is 82.6, and its average-to-worst gap is 11.6. mT5-small gets 81.8 and 12.2. The DistilBERT rows come from prior work trained differently, so they are not a like-for-like comparison 2202.11176 tab. 5 2202.11176 §6.3.2.
• Half of the pretraining mix is a proprietary corpus of about 4.6B texts, and the other half is mC4 2202.11176 §3.4. Pretraining cannot be repeated from public data alone, and the paper links no code or weights 2202.11176 §3.4.
• The authors' serving figures are a 45 ms median at 1K QPS on one TPUv2 chip, and about 200 ms median in production for ten new languages 2202.11176 §8. The fine-tuning speeds in 2202.11176 tab. 8 are training throughput, not serving latency.

When to use which

Caveats: the ToxicChat results come from one chatbot demo, English prompts only, and first turns only 2310.17389 §2 2310.17389 Limitations. Do not assume the ranking carries over to multi-turn or non-English traffic. The Wikipedia, ToxiGen and Vidgen et al. results are in English and cover other kinds of text. The KUISAIL results are for Arabic, Greek and Turkish tweets, on one shared task's test sets 2007.13184 §3. Vidgen et al. used twenty annotators, and their target models were trained on their own rounds 2012.15761 §4.

Implementation sketch

Common core for the in-domain fine-tuning variant. The paper does not state the learning rate, batch size, epochs, maximum length or optimizer 2310.17389 §4.1. Choose them and report them.

# Labelled prompts: text, toxic (0/1), jailbreak (0/1)  [arxiv:2310.17389 §2.2]
# Split into random halves for train and test  [arxiv:2310.17389 §3]
train, test = split(prompts)

# Fine-tune on the user prompt only  [arxiv:2310.17389 tab. 3]
model = load_pretrained("roberta-base")  # [arxiv:2310.17389 tab. 2]
# Learning rate, batch size, epochs: not stated, choose and report
model.fit(train.text, train.toxic)

# Score the human-labelled part of the test half  [arxiv:2310.17389 §3]
human = test.where(human_labelled)
pred = model.predict(human.text)
precision, recall, f1 = prf(human.toxic, pred)

# JR: share of jailbreaking prompts flagged as toxic  [arxiv:2310.17389 §3]
jr = mean(pred[human.jailbreak == 1])

Where the variants differ: the off-the-shelf variant replaces the fit step with a call to the moderation tool and a threshold choice. The out-of-domain variant swaps train for another public dataset and keeps the ToxicChat test half. The crowd-labelled variant trains on the fraction of annotators who said "toxic" rather than a binary label, and tunes its threshold on held-out data 1610.08914 §4.1 1610.08914 §5.1. The per-annotator variant keeps one head per annotator in place of the single head, sums the losses over the labels each annotator gave, and takes the majority of head outputs at prediction time 2110.05719 §3.4. The synthetic-data variant replaces train with generated statements and keeps the human-rated set as test data 2203.09509 §4.2. The language-specific CNN variant keeps the encoder, feeds the last four layers into CNN filters, and sends the pooled features to a sigmoid output 2007.13184 §4.4. The model-in-the-loop variant has no fixed training set. Each round adds posts that fool the current target, and the next target trains on the accumulated data with one round repeated 2012.15761 §5. The production-pipeline variant adds a second, unlabelled production batch and a critic term to the loss, following the WDAT objective in the paper 2208.03274 §3.4. The moral-direction variant has no fit step on labelled text. Instead, each generation step drops candidates whose direction score is at or below t, keeping at least m 2103.11790 § Reducing Neural Toxic Degeneration in Language Models. The byte-level variant changes only the input side. It learns a byte-to-subword step in place of a fixed vocabulary, keeps the encoder, and drops the decoder for classification. Pooling and the sigmoid head follow the paper's regression head 2202.11176 §3.1 2202.11176 §3.3. Its optimiser is Adafactor with a fixed learning rate of 1e-3, dropout 0.1 and batch size 512, but the paper says fine-tuning runs "until convergence" without a stated criterion 2202.11176 app. A 2202.11176 §5.

Contradictions

No numeric conflict across papers: the taxonomy reports no results 2112.04359 Results. The other papers measure different things, so their numbers are not directly comparable. ToxicChat scores toxicity under the SemEval-2019 guideline 2310.17389 §2.1. Ex Machina scores personal attacks on Wikipedia comments 1610.08914 §3. ToxiGen scores implicit hate on its own test sets, with labels taken from prompts 2203.09509 §4.2. The per-annotator paper scores majority-vote F1 on hate speech and on emotions 2110.05719 tab. 1 tab. 3. The production-pipeline paper reports AUPRC on its own public test set and on a production set it does not release 2208.03274 §4.2 tab. 3. KUISAIL reports macro-F1 on three OffensEval test sets 2007.13184 §5, and Vidgen et al. report macro-F1 on their own round test sets and accuracy on HateCheck 2012.15761 tab. 4 §6.3. Do not rank the F1, AUC, AUPRC and accuracy values above against each other.

Two more sources use different measures. The moral-direction paper reports Pearson correlations with human yes/no ratings [arxiv:2103.11790 § BERT's MoralDirection Strongly Correlates with Human Moral Norms]. Its toxicity labels come from Perspective API at a score of 0.5 or more [arxiv:2103.11790 § Testbed for evaluating the toxicity of text generations]. ToxicChat's Perspective row uses 0.7 2310.17389 §3. The two Perspective results therefore use different thresholds on different text, and they should not be compared. Vidgen et al. also report Perspective's toxicity classifier, at 77% accuracy on HateCheck 2012.15761 §6.3. That is a third test, with its own accuracy measure, so it cannot be set against either result above. KUISAIL reports macro-F1 on OffensEval 2020 test sets, and the card gives no Perspective comparison 2007.13184 §5. The hate-speech survey reports Krippendorff's alpha, which measures agreement and not accuracy 1701.08118 §5.

On synthetic data, the papers point in different directions. ToxiGen reports gains for HateBERT and RoBERTa after fine-tuning on its generated statements 2203.09509 tab. 4. The production-pipeline paper reports that large GPT-3 zero-shot and few-shot sets reduced performance. It says this contradicts earlier work in low-data settings 2208.03274 §3.3. The two studies differ in model, data size and label method, and the card does not resolve the conflict.

Perspective API appears in five papers, and each uses its own data, threshold or measure. ToxicChat reports F1 at a score of 0.7 2310.17389 §3. The moral-direction paper uses 0.5 [arxiv:2103.11790 § Testbed for evaluating the toxicity of text generations]. The byte-level paper reports AUC-ROC on JMTCC (.8770) and on its production set, for English and Russian only [arxiv:2202.11176 tab. 1] tab. 2. The production-pipeline paper reports AUPRC on its own set 2208.03274 tab. 3. Vidgen et al. report accuracy on HateCheck 2012.15761 §6.3. These numbers are not comparable across papers.

The byte-level paper's ranking against mT5-base also shifts by test set. On Production-Multilingual the 102M model is ahead, at .910 against .906. On JMTCC and TweetEval it is behind, at .9194 against .9239 and 53.3 against 57.7 2202.11176 tab. 1 tab. 2 tab. 4. The paper's summary says it matches or beats mT5-base on average, and the table supports this only on the production set.

Within the ToxicChat paper, some figures conflict and the card does not resolve them:

• The trial toxicity rate is 7.22% 2310.17389 §2.1, but 48 of the 720 trial prompts are toxic 2310.17389 §2.2, which is 6.67%.
• Perspective filtered 4,668 of 7,599 prompts in stage one, about 61% 2310.17389 §2.2. Appendix D's table gives 60.71% of Perspective-scored data still needing review at a 1% safe rate 2310.17389 app. D.

Papers

• ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation: a 10,166-prompt benchmark from real first-turn user prompts, with toxicity and jailbreaking labels. It shows public detectors transfer poorly to chatbot traffic.
• Ethical and social risks of harm from Language Models: a taxonomy of 21 risks in six areas, with toxic language listed as one of them. It gives no detector, dataset or measurement.
• Ex Machina: Personal Attacks Seen at Scale: crowd labels plus a bag-of-n-grams classifier, used to label 63M Wikipedia talk comments for personal attacks and to measure attack prevalence.
• ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection: 274,186 GPT-3 statements about 13 groups, with ALICE, a classifier-guided decoder. Fine-tuning on the data raises AUC on three human-written implicit-hate sets.
• Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations: one BERT encoder with one head per annotator, each trained on that annotator's labels and voting at prediction time. On hate speech it edges out majority-label training, and its head spread tracks annotator disagreement.
• A Holistic Approach to Undesired Content Detection in the Real World: an eight-category production classifier built with labeling audits, active learning on live traffic, synthetic data and Wasserstein domain-adversarial training. It reports gains over Perspective API on its own public test set.
• Ross et al., Measuring the Reliability of Hate Speech Annotations: two surveys of 56 people on 20 German tweets each. A written hate-speech definition did not raise Krippendorff's alpha above .29, though it changed how often tweets were marked for banning.
• Schramowski et al., Large Pre-trained Language Models and moral norms: a moral direction in BERT-large sentence embeddings, used to score phrases and to filter GPT-2 candidates during decoding. It reports lower toxicity on RealToxicityPrompts, with a fragile dependence on the verbs used.
• KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media: a language-specific BERT-base with a CNN over its last four layers, for offensive-tweet detection in Arabic, Greek and Turkish. Its gain over BERT alone is small and the Turkish result is slightly lower.
• Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection: four rounds of annotators trying to fool the current RoBERTa-base target, with label-flipping perturbations from round 2. Later targets fool annotators less and score higher on HateCheck. The release has data and guidelines but no training code.
• A New Generation of Perspective API: Efficient Multilingual Character-level Transformers: a token-free byte-level encoder with a learned tokenizer, fine-tuned for multi-label toxicity scoring across many languages. It reports gains over custom mBERT and mT5 on its production set, but a smaller lead on public sets, and it is not reproducible from public data.