--- kind: concept slug: toxicity-detection title: Toxicity detection summary: >- Labelling a text as toxic or not. Detectors trained on social-media or other public text transfer poorly to real chatbot prompts: on ToxicChat, an in-domain RoBERTa-base reaches F1 72.1 (author-reported), while four off-the-shelf moderation tools reach at most F1 34.6. Other routes are crowd-labelled corpora, LLM-generated training data and model-in-the-loop data collection. tags: [safety, toxicity, classification, benchmark, dataset, annotator-disagreement, decoding, multilingual] papers: ["2310.17389", "2112.04359", "1610.08914", "2203.09509", "2110.05719", "2208.03274", "1701.08118", "2103.11790", "2007.13184", "2012.15761", "2202.11176"] related: [red-teaming, text-classification, domain-adaptation, dataset-curation, benchmark-validity, llm-ethics, prompt-injection, algorithmic-fairness, moderation-analysis, controllable-text-generation, constrained-decoding, prompt-engineering, multi-task-learning, uncertainty-estimation, active-learning, noisy-labels, inter-annotator-agreement, text-embedding, decoding-strategies, language-model-pretraining, adversarial-evaluation, subword-tokenization, multilingual-evaluation] status: draft updated_at: 2026-11-01 --- ## Overview The task is to decide whether a piece of text is toxic, usually as a binary label. The shared definition in ToxicChat follows the SemEval-2019 undesired-content guidelines [arxiv:2310.17389 §2.1]. A second, separate label marks jailbreaking prompts, which try to get a chatbot to produce toxic output through prompt tricks [arxiv:2310.17389 §2]. The main problem is domain shift. Most detectors were built on social-media or LLM-generated text. Chatbot users write questions and instructions, and they can hide intent behind harmless wording, so detectors trained elsewhere may miss toxic chatbot traffic [arxiv:2310.17389 §1] [fig. 1]. Toxic language is one of the 21 risks in a taxonomy of language-model harms, under discrimination and exclusion [arxiv:2112.04359 §2.1]. That paper reports no experiments and no detector, so it frames the problem but gives no numbers. ## Variants Eleven papers cover this page. ToxicChat is the only one that compares detectors on one chatbot test set. KUISAIL and Vidgen et al. also compare models, on the OffensEval 2020 test sets and on HateCheck. The others cover other routes: a byte-level multilingual model for production, crowd labels on a large corpus, machine-generated text, several annotators per item, rounds of annotators writing against a live model, and a production pipeline that mixes public data, active learning and audits. Two more papers study the labels themselves and a generation-time filter that does not classify input at all. Unless noted, numbers are author-reported. ### Off-the-shelf moderation tools Pretrained tools are used as they are, with no training on the target data. Those tested are OpenAI Moderation (binary output), the Perspective API (score threshold 0.7), HateBERT and ToxDectRoberta [arxiv:2310.17389 §3] [arxiv:2310.17389 app. A]. On ToxicChat, F1 ranges from 5.3 (Perspective) to 34.6 (ToxDectRoberta). OpenAI Moderation has precision 84.3 but recall 11.7. HateBERT has the highest jailbreaking recall of this group (60.5) with precision 6.3 [arxiv:2310.17389 tab. 1]. ### In-domain fine-tuning A `roberta-base` classifier is trained on the ToxicChat training half and tested on the test half. It reaches precision 75.9, recall 68.7, F1 72.1 and jailbreaking recall 83.5 [arxiv:2310.17389 tab. 2]. Card: [ToxicChat](/papers/2310.17389). The paper's own test set is the only evaluation, so this does not show how the model performs on other chatbots [arxiv:2310.17389 Limitations]. ### Out-of-domain fine-tuning The same `roberta-base` setup is trained on one public toxicity dataset at a time, then tested on ToxicChat. Six datasets were tried: HSTA, Movie Reviews, Jigsaw, ToxiGen, RealToxicPrompts and ConvAbuse [arxiv:2310.17389 §4.1]. The best single source was ConvAbuse (F1 51.6). Movie Reviews gave F1 0.0. Training on the combination of all six gave F1 42.7, below the best single source [arxiv:2310.17389 tab. 2]. Subsample sizes other than HSTA's 8,000 rows are not stated [arxiv:2310.17389 app. A]. ### Input or output features Three feature sets were tried with `roberta-base`, all trained and tested on ToxicChat: the user prompt, the Vicuna response, and both together. The user prompt alone gives F1 72.1. The response alone gives 63.3. Both together give 72.6, which the authors describe as not significant [arxiv:2310.17389 tab. 3] [arxiv:2310.17389 §4.2]. Using the response needs a response to exist, so pre-generation screening can use the prompt only. ### Language-specific BERT with a CNN head [KUISAIL at SemEval-2020 Task 12](/papers/2007.13184) fine-tunes one BERT-base model per language and puts a CNN on top. The CNN takes the outputs of the last four encoder layers as input channels. It has 160 filters in five heights, from one to five tokens, with max pooling and a sigmoid output [arxiv:2007.13184 §4.4]. It labels tweets as offensive or not in Arabic, Greek and Turkish, on the OffensEval 2020 subtask A data [arxiv:2007.13184 §3]. - Author-reported macro-F1 on the test sets is 0.897 for Arabic, 0.843 for Greek and 0.814 for Turkish, an average of 0.851. Language-specific BERT alone averages 0.841, and multilingual BERT 0.796 [arxiv:2007.13184 tab. 2]. - The gain over BERT alone is +0.013 in Arabic and +0.021 in Greek. In Turkish the CNN head is slightly lower, 0.814 against 0.816 [arxiv:2007.13184 tab. 2]. The average gain of 0.010 therefore comes mostly from two languages. - The paper reports single numbers with no repeated runs, so the size of the gain is uncertain [arxiv:2007.13184 §5]. - Arabic uses ArabicBERT, which the authors trained and released themselves [arxiv:2007.13184 §4.3]. - The paper leaves several training settings unstated, such as the loss, optimiser, dropout and decision threshold. The released code sets them, and it differs from the paper on epochs and checkpoint selection. The card's Paper vs code section lists the differences, so check it before reuse [arxiv:2007.13184 §4.4]. ### Crowd-labelled classifier for corpus-scale labelling This variant targets personal attacks in English Wikipedia talk comments, not general toxicity. [Ex Machina](/papers/1610.08914) collects at least 10 crowd labels per comment, trains a bag-of-n-grams classifier on them, and labels a corpus of 63M comments with it [arxiv:1610.08914 §3] [arxiv:1610.08914 app. A]. - The best model is an MLP on character 1–5-grams, trained on the fraction of annotators who said "attack". It reaches AUC 96.59 on the dev split. Logistic regression on the same features reaches 96.24 [arxiv:1610.08914 tab. 2]. The dev split both selects the model and reports these numbers, so they are optimistic [arxiv:1610.08914 §4.2]. - Against crowd ensembles, the model beats an ensemble of three labels on AUC (97.19 against 95.49) and on Spearman. On AUC it sits level with five labels (97.13) [arxiv:1610.08914 tab. 3]. The paper's "as good as three crowd workers" claim is one reading of this table. - The decision threshold of 0.425 balances false positives and false negatives. It gives precision 0.63 and recall 0.63 on dev and test together [arxiv:1610.08914 §5.1]. The paper ties the score to this model and label type [arxiv:1610.08914 §4.1], so a different model needs its own threshold. - Crowd agreement is modest, with Krippendorff's alpha 0.45 [arxiv:1610.08914 §3]. - The corpus findings show the use case. Anonymous comments carry 3.1% attacks against 0.5% for registered comments [arxiv:1610.08914 §5.2]. A bootstrap check places the machine-estimated attack rate inside the crowd-labelled interval [arxiv:1610.08914 §5.1]. ### Per-annotator heads for disagreeing labels When each text has labels from several annotators, this variant keeps them apart instead of taking a majority label first. One shared encoder feeds one output head per annotator, and each head trains only on the texts that annotator labelled. The prediction is the majority over all heads [arxiv:2110.05719 §3.4]. The test set is the Gab Hate Corpus, a hate-speech set with at least three annotators per post and 18 annotators in total [arxiv:2110.05719 §4.1.1]. Hate speech is close to toxicity but not the same label, so these numbers do not transfer directly. - On this corpus, majority-vote F1 is 59.49 for the multi-task model and 57.32 for majority-label training. The ensemble of per-annotator models reaches 53.54 [arxiv:2110.05719 tab. 1]. The paper calls the gap significant, but it reports no statistical test [arxiv:2110.05719 §4.3.1]. - Scored against each annotator's own labels, the multi-task model reaches F1 63.20, against 60.94 for the ensemble [arxiv:2110.05719 tab. 1]. - On six GoEmotions emotions, the multi-task model is below the baseline on four of six with all 82 annotators, and above it on four of six in a subset of 53 annotators with more than 1,000 labels each [arxiv:2110.05719 tab. 3]. This is emotion data, not toxicity, and the subset was chosen after the full-data results [arxiv:2110.05719 §4.4]. - Spread across the heads is the uncertainty signal. On the Gab Hate Corpus it tracks annotator disagreement better than softmax or MC dropout, and its mean is 0.170 on wrong predictions against 0.049 on correct ones [arxiv:2110.05719 §4.3.2] [arxiv:2110.05719 §5.2]. - Training one fold took 22.3 minutes for the multi-task model, against 158.4 for the ensemble [arxiv:2110.05719 tab. 2]. Only BERT-base was tested, and the approach needs labels that name the annotator [arxiv:2110.05719 §3] [arxiv:2110.05719 §6.2]. ### Synthetic training data from a language model [ToxiGen](/papers/2203.09509) generates 274,186 statements about 13 minority groups with GPT-3. The set is balanced between toxic and benign text within each group, and 98.2% of statements contain no profanity or slur by a word-list test [arxiv:2203.09509 tab. 2]. Its target is implicit toxicity, which detectors often miss [arxiv:2203.09509 §1]. - Generation uses demonstration-based prompting, with five examples drawn at random from a group's set for each statement [arxiv:2203.09509 §3.1] [arxiv:2203.09509 app. A]. - ALICE is a classifier-guided constrained beam search. It pushes output away from the label of its prompt. It produced 14,174 of the statements, about 5% [arxiv:2203.09509 §3.2] [arxiv:2203.09509 tab. 2]. - Author-reported gains: HateBERT fine-tuned on ALICE and top-k output together reaches AUC 0.71, 0.67 and 0.66 on three human-written implicit-hate sets (SBF, ImplicitHateCorpus, DynaHate), against 0.60, 0.60 and 0.47 without fine-tuning [arxiv:2203.09509 tab. 4]. RoBERTa gains are smaller, from 0.65, 0.57 and 0.49 to 0.70, 0.66 and 0.54 [arxiv:2203.09509 tab. 4]. - ToxiGen-HumanVal, 792 machine-generated statements rated by humans, reaches AUC 0.96 for HateBERT [arxiv:2203.09509 tab. 4] [arxiv:2203.09509 §4.2]. It is the same kind of text as the training data, so it overstates transfer. - The training label is the label of the prompt, which is a proxy [arxiv:2203.09509 §5]. Raters agreed with the prompt label for 67.7% of top-k toxic prompts and 40.3% of ALICE toxic prompts [arxiv:2203.09509 §4.3]. Fleiss' kappa for the human ratings is 0.46 [arxiv:2203.09509 §4.2]. - The fine-tuning learning rate, epochs, batch size and sequence length are not stated [arxiv:2203.09509 §5]. - The released code applies the classifier term twice, so its effective weights differ from the single weighted sum in the paper's eq. 1. See the card's Paper vs code section before reusing the code [arxiv:2203.09509 eq. 1]. - The arXiv LaTeX source of this paper has no table text. The numbers above were read from the PDF, so check them against it before use. See the card's Pitfalls section. ### Model-in-the-loop data collection [Learning from the Worst](/papers/2012.15761) by Vidgen et al. builds an English online hate dataset in four rounds. In each round, annotators write posts that the current target model, a RoBERTa-base classifier, gets wrong. Checked entries join the training data, and the next target is trained on the earlier rounds plus the new one [arxiv:2012.15761 §1] [arxiv:2012.15761 §5]. From round 2 on, half of each round is perturbations of earlier posts that flip their label. These were written without model feedback [arxiv:2012.15761 §5.2]. The release has 41,255 entries, 54% of them hateful [arxiv:2012.15761 tab. 1]. - The in-loop error rate is the share of annotator entries that fooled the target. It falls from 54.7% in round 1 to 27.7% in round 4 [arxiv:2012.15761 tab. 3]. The authors warn that a falling rate does not show how much harder later rounds became, so the rate is not comparable across rounds [arxiv:2012.15761 §6.1]. - Author-reported macro-F1 on the round-4 test set rises from 60.91 for the round-1 target to 75.97 for the round-4 target, over five training seeds [arxiv:2012.15761 tab. 4]. - On HateCheck, the round-4 target reaches 95% accuracy. The round-1 target reaches 60%, and Perspective's toxicity classifier, the best of four external models, reaches 77% [arxiv:2012.15761 §6.3]. The authors say HateCheck can show only that some weaknesses are absent [arxiv:2012.15761 §6.3]. - Duplication is part of the recipe. Each target trains on earlier data with one round repeated, and the repeat factors (5, 100 and 1 across the three targets) were chosen on dev sets [arxiv:2012.15761 §5] [arxiv:2012.15761 app. D]. - The paper releases the data and annotation guidelines, but no training code. Batch size, maximum length, warm-up and loss are not stated [arxiv:2012.15761 app. D]. - Twenty annotators took part, and the authors describe their setting as one with expert-overseen annotators and a backend that can run the target with low latency [arxiv:2012.15761 §4] [arxiv:2012.15761 §7]. ### Production pipeline with active learning and audits [A Holistic Approach to Undesired Content Detection in the Real World](/papers/2208.03274) trains an eight-category classifier for a chatbot-style production system. It covers five top-level categories and three severe subcategories, with one head per category [arxiv:2208.03274 §2] [arxiv:2208.03274 §4.1]. The method combines written labeling rules, label audits, active learning on live traffic, synthetic data and domain-adversarial training [arxiv:2208.03274 §3]. - Live traffic is the bottleneck. Public data is shifted from production, and some categories are rare: self-harm appears in 0.04% of sampled user prompts [arxiv:2208.03274 §1]. Active learning raised the share of undesired samples about 10 to 22 times per category against random sampling [arxiv:2208.03274 tab. 4]. Overall, about 40% of active-learning samples carry an undesired label, against 3.4% of random samples [arxiv:2208.03274 §4.3]. - Label quality is checked in three ways: calibration sessions, a per-category audit of 10 undesired and 10 high-probability items, and a mislabel loop that re-labels a flagged set when more than 30% of its audited samples are wrong [arxiv:2208.03274 §3.2]. - Domain-adversarial training with a Wasserstein critic is the paper's answer to public-to-production shift. Trained on public data alone, it raises average AUPRC for self-harm from .063 to .281 and for V2 from .165 to .453. With synthetic or production data in the mix, the gains are smaller. For hateful content the adversarial model is lower on the synthetic-data set, .476 against .551 [arxiv:2208.03274 tab. 5] [arxiv:2208.03274 §4.4]. The paper reports no spread over seeds for the main model. - Against Perspective API on the paper's public test set, the classifier has higher AUPRC on four categories: sexual .9703 against .8709, hateful .7968 against .6914, violence .7371 against .5201 and harassment .6191 against .3902. Some Perspective cells use approximated taxonomy mappings, and the two taxonomies differ [arxiv:2208.03274 tab. 3]. - Synthetic data is used in two ways. A curated set of 69k counterfactual examples is added to the training data [arxiv:2208.03274 §4.4]. Large sets of GPT-3 zero-shot and few-shot examples, at twice the labelled size, reduced performance, which the authors attribute to label noise [arxiv:2208.03274 §3.3]. The paper's linked repository holds only an evaluation set, not the training code. See the card's Paper vs code section for its label semantics. - The taxonomy is US-centric, and the classifier was evaluated only in English, with about 5% of its training samples non-English [arxiv:2208.03274 §7.1] [arxiv:2208.03274 §6]. - The learning rate of 0.05 is stated as written. Optimizer, schedule and warmup are not stated, so test this value before reuse [arxiv:2208.03274 §4.1]. ### Written definition shown to annotators This variant changes the labelling step, not the model. Two survey groups each labelled 20 German tweets about the 2016 refugee debate. One group saw Twitter's hateful-conduct definition before each tweet and answered by that definition. The other answered from personal opinion [arxiv:1701.08118 §4]. Card: [Ross et al. (2017), a reliability study of hate-speech labels](/papers/1701.08118). - The definition did not raise agreement. Krippendorff's alpha was .18 to .29 across questions and both groups, below the .66 the authors cite as the minimum for uncertain applications [arxiv:1701.08118 §5] [fig. 1]. - Six researchers who knew the definitions reached alpha .38 on 541 tweets [arxiv:1701.08118 §3]. The survey and pre-study numbers are not directly comparable, and the paper does not claim they are. - The definition group more often said a tweet should be banned, 32.6% against 17.6%, with p = .01 and no correction for the three group tests [arxiv:1701.08118 tab. 1]. - The survey has 25 and 31 participants and 20 items each, and the paper does not state which 20 tweets were shown [arxiv:1701.08118 §4] [tab. 1]. Treat the alpha values as a rough signal. ### Moral direction as a decoding filter This variant does not classify input. It filters next-token candidates while a language model generates. [Schramowski et al. (2021), a moral direction in BERT](/papers/2103.11790) takes a sentence embedding from BERT-large, places verbs in several question frames, and uses the first principal component of the averaged embeddings as a direction [arxiv:2103.11790 § Identifying the Moral Direction of Language Models] [suppl. B.1]. Projecting a phrase on this direction gives a score. - The score tracks human yes/no judgements, with Pearson r = 0.78 and 0.79 on two panels [arxiv:2103.11790 § BERT's MoralDirection Strongly Correlates with Human Moral Norms]. The result depends on the verbs. Three random verb sets gave r = 0.64, 0.60 and -0.01 [arxiv:2103.11790 suppl. B.2]. - In GPT-2 decoding, top-k and top-p sampling produce the candidates. A candidate is kept if the score of the text with it appended is above a threshold t, and at least m candidates are kept [arxiv:2103.11790 § Reducing Neural Toxic Degeneration in Language Models]. The paper's runs use t = -0.5, 0 and 0.5, and m = 5 [arxiv:2103.11790 § Experimental setup]. - Author-reported, on the RealToxicityPrompts testbed scored with Perspective: the share of prompts with a generation above toxicity 0.9 falls to 1.17% at t = 0, against 9.82% for GPT-2 [arxiv:2103.11790 § Reducing Neural Toxic Degeneration in Language Models]. In the full table, t = 0.5 is marked best in all six toxicity columns [arxiv:2103.11790 tab. 3]. - The threshold only means something on the scale the authors used. The paper gives no tuning procedure, so a different encoder, verb set or PCA fit needs its own threshold [arxiv:2103.11790 § Identifying the Moral Direction of Language Models]. The paper reports no runtime or memory cost [arxiv:2103.11790]. Each step scores every candidate with a BERT-large pass, so cost grows with top-k [arxiv:2103.11790]. See the card's Pitfalls section for details. The card's Paper vs code section covers the differences from the released code, so check it before reuse. ### Byte-level multilingual encoder for production scoring [Perspective API, next generation](/papers/2202.11176) replaces per-language word pieces with a token-free model. UTF-8 bytes pass through a learnable tokenizer (GBST, from Charformer), then a T5-style encoder-decoder [arxiv:2202.11176 §3.1] [arxiv:2202.11176 §3.2]. Pretraining is span denoising, with spans averaging 20 bytes and covering 15% of the input [arxiv:2202.11176 §3.4]. For fine-tuning the decoder is dropped, the first position of the encoder output is pooled, and a linear layer with a sigmoid per label gives multi-label outputs [arxiv:2202.11176 §3.3]. - Production-Multilingual AUC-ROC averages .910 for the 102M model and .919 for the 268M model. mT5-base averages .906, and custom mBERT .840. Perspective API is reported only for English (.974) and Russian (.907) [arxiv:2202.11176 tab. 1]. The 102M model is only slightly above mT5-base on this set. - On JMTCC, the 268M model reaches AUC-ROC .9367 and the 102M model .9194. mT5-base reaches .9239 and Perspective API .8770. The best entry is an ensemble with an estimated size above 5B parameters [arxiv:2202.11176 tab. 2]. - On code-switched sets, the 102M model is below mT5-base on JMTCC-CS (.9191 against .9393). The 268M model is best on both sets, at .9446 and .9023 [arxiv:2202.11176 tab. 3]. - Zero-shot on TweetEval hate classification, the 102M model reaches macro F1 53.3 and mT5-base 57.7. Fine-tuned English-only models reach 52.3 to 56.4 [arxiv:2202.11176 tab. 4]. - On obfuscated English text, the authors report that the model beats every baseline at each obfuscation rate above 0%. After fine-tuning on Civil Comments with 30% obfuscation it reaches AUC-ROC 86.0, against 84.5 for mT5-small [arxiv:2202.11176 §6.2]. On identity-term templates, some Korean and Japanese terms have BPSN AUC of 0.5 or below, which the authors attribute to false positives on non-toxic templates [arxiv:2202.11176 §6.4]. - On CivilComments-WILDS, the 102M model's worst-group accuracy is 82.6, and its average-to-worst gap is 11.6. mT5-small gets 81.8 and 12.2. The DistilBERT rows come from prior work trained differently, so they are not a like-for-like comparison [arxiv:2202.11176 tab. 5] [arxiv:2202.11176 §6.3.2]. - Half of the pretraining mix is a proprietary corpus of about 4.6B texts, and the other half is mC4 [arxiv:2202.11176 §3.4]. Pretraining cannot be repeated from public data alone, and the paper links no code or weights [arxiv:2202.11176 §3.4]. - The authors' serving figures are a 45 ms median at 1K QPS on one TPUv2 chip, and about 200 ms median in production for ten new languages [arxiv:2202.11176 §8]. The fine-tuning speeds in [arxiv:2202.11176 tab. 8] are training throughput, not serving latency. ## When to use which | Situation | Use | Why | |---|---|---| | No labelled data and no training budget | An off-the-shelf tool, with a measured recall check on your own traffic | On ToxicChat the best tool reaches F1 34.6. Perspective at 0.7 reaches recall 2.7, and OpenAI Moderation 11.7 [arxiv:2310.17389 tab. 1] | | Labelled prompts from your own chatbot | Fine-tune a classifier on the user prompt | In-domain data gave F1 72.1 against at most 51.6 from out-of-domain data [arxiv:2310.17389 tab. 2] | | Only public, out-of-domain labelled data | Expect weak transfer, and test on in-domain data before use | Best single source F1 51.6, and the six-set combination 42.7 [arxiv:2310.17389 tab. 2] | | Jailbreaks matter more than general toxicity | Track jailbreaking recall as its own metric | HateBERT's recall is 60.5 but its precision is 6.3. In-domain training gives 83.5 [arxiv:2310.17389 tab. 1] [tab. 2] | | Response text is available and cheap to use | Prompt-only input is enough on this evidence | Adding the response did not give a significant gain [arxiv:2310.17389 §4.2] | | Detector must treat all user groups and dialects alike | Measure error per dialect or group before deployment, not only overall F1 | The taxonomy warns that toxicity filters can disadvantage some dialects [arxiv:2112.04359 § Discussion]. ToxicChat reports no per-dialect results | | Need a rate over a large corpus, and a crowd budget for a sample | Crowd-label a sample, train a bag-of-n-grams model, and check it against crowd ensembles | The model matched about five crowd labels on AUC for personal attacks [arxiv:1610.08914 tab. 3]. The evidence covers English Wikipedia comments only | | Several labels per text from named annotators, and their disagreement matters | One shared encoder with one head per annotator, and the spread of the heads as uncertainty | On the Gab Hate Corpus, majority-vote F1 was 59.49 against 57.32 for majority labels [arxiv:2110.05719 tab. 1]. GoEmotions results were mixed [arxiv:2110.05719 tab. 3] | | Need implicit-hate examples across many groups, and can accept label noise | Generate statements per group, and keep a human-rated test set | Fine-tuning gave 7 to 19 AUC points on three implicit sets for HateBERT [arxiv:2203.09509 tab. 4]. Prompt labels matched rater labels for only 40.3% of ALICE toxic prompts [arxiv:2203.09509 §4.3] | | Live traffic is available and rare categories matter, with a budget for labeling and audits | Sample traffic with active learning, audit labels, and check the model's mislabel rate | Active learning raised the undesired share 10 to 22 times per category [arxiv:2208.03274 tab. 4]. The paper's gains come from its own production data, which is not shared [arxiv:2208.03274 §4.2] | | Public labelled data only, and a production target with no labels | Domain-adversarial training with a Wasserstein critic, checked on a labelled production validation set | Public-only AUPRC for self-harm rose from .063 to .281 [arxiv:2208.03274 tab. 5]. Gains shrink when in-distribution data is added | | Raters must label toxicity, and you need the labels to agree | Measure agreement per question before training, and do not expect a written definition to fix it | With a definition, alpha was .18 to .29 across questions, against .38 for expert researchers [arxiv:1701.08118 §5] [§3]. The definition moved ban answers, not agreement [arxiv:1701.08118 tab. 1] | | Generated text must avoid toxic output, and the generator cannot be retrained | Filter candidates with a direction score during decoding, and re-tune its threshold for your encoder | Author-reported toxicity fell in the GPT-2 testbed [arxiv:2103.11790 § Reducing Neural Toxic Degeneration in Language Models]. Random verb sets gave r = -0.01 to 0.64 [arxiv:2103.11790 suppl. B.2], so the score is fragile | | Labelled data for each language, and a language-specific encoder for each | Fine-tune a language-specific BERT-base. A CNN head over its last four layers is an option to test | Macro-F1 averaged 0.851 against 0.841 for language-specific BERT alone. Turkish was slightly lower than BERT alone [arxiv:2007.13184 tab. 2] | | The current model is easy to fool, and annotators can work against a live model | Model-in-the-loop rounds: annotators try to fool the current target, and the next target trains on all earlier rounds | The round-4 target reached 95% on HateCheck, against 60% for the round-1 target. The in-loop error rate fell from 54.7% to 27.7% [arxiv:2012.15761 §6.3] [tab. 3] | | Many languages in one model, with mixed-language or obfuscated text, and you can pretrain | A byte-level encoder with a multi-label sigmoid head, fine-tuned on labelled data per language | Production-Multilingual AUC-ROC .910 (102M) and .919 (268M), against .840 for custom mBERT [arxiv:2202.11176 tab. 1]. The pretraining corpus is proprietary and no weights are linked [arxiv:2202.11176 §3.4] | Caveats: the ToxicChat results come from one chatbot demo, English prompts only, and first turns only [arxiv:2310.17389 §2] [arxiv:2310.17389 Limitations]. Do not assume the ranking carries over to multi-turn or non-English traffic. The Wikipedia, ToxiGen and Vidgen et al. results are in English and cover other kinds of text. The KUISAIL results are for Arabic, Greek and Turkish tweets, on one shared task's test sets [arxiv:2007.13184 §3]. Vidgen et al. used twenty annotators, and their target models were trained on their own rounds [arxiv:2012.15761 §4]. ## Implementation sketch Common core for the in-domain fine-tuning variant. The paper does not state the learning rate, batch size, epochs, maximum length or optimizer [arxiv:2310.17389 §4.1]. Choose them and report them. ```pseudocode # Labelled prompts: text, toxic (0/1), jailbreak (0/1) [arxiv:2310.17389 §2.2] # Split into random halves for train and test [arxiv:2310.17389 §3] train, test = split(prompts) # Fine-tune on the user prompt only [arxiv:2310.17389 tab. 3] model = load_pretrained("roberta-base") # [arxiv:2310.17389 tab. 2] # Learning rate, batch size, epochs: not stated, choose and report model.fit(train.text, train.toxic) # Score the human-labelled part of the test half [arxiv:2310.17389 §3] human = test.where(human_labelled) pred = model.predict(human.text) precision, recall, f1 = prf(human.toxic, pred) # JR: share of jailbreaking prompts flagged as toxic [arxiv:2310.17389 §3] jr = mean(pred[human.jailbreak == 1]) ``` Where the variants differ: the off-the-shelf variant replaces the fit step with a call to the moderation tool and a threshold choice. The out-of-domain variant swaps `train` for another public dataset and keeps the ToxicChat test half. The crowd-labelled variant trains on the fraction of annotators who said "toxic" rather than a binary label, and tunes its threshold on held-out data [arxiv:1610.08914 §4.1] [arxiv:1610.08914 §5.1]. The per-annotator variant keeps one head per annotator in place of the single head, sums the losses over the labels each annotator gave, and takes the majority of head outputs at prediction time [arxiv:2110.05719 §3.4]. The synthetic-data variant replaces `train` with generated statements and keeps the human-rated set as test data [arxiv:2203.09509 §4.2]. The language-specific CNN variant keeps the encoder, feeds the last four layers into CNN filters, and sends the pooled features to a sigmoid output [arxiv:2007.13184 §4.4]. The model-in-the-loop variant has no fixed training set. Each round adds posts that fool the current target, and the next target trains on the accumulated data with one round repeated [arxiv:2012.15761 §5]. The production-pipeline variant adds a second, unlabelled production batch and a critic term to the loss, following the WDAT objective in the paper [arxiv:2208.03274 §3.4]. The moral-direction variant has no `fit` step on labelled text. Instead, each generation step drops candidates whose direction score is at or below `t`, keeping at least `m` [arxiv:2103.11790 § Reducing Neural Toxic Degeneration in Language Models]. The byte-level variant changes only the input side. It learns a byte-to-subword step in place of a fixed vocabulary, keeps the encoder, and drops the decoder for classification. Pooling and the sigmoid head follow the paper's regression head [arxiv:2202.11176 §3.1] [arxiv:2202.11176 §3.3]. Its optimiser is Adafactor with a fixed learning rate of 1e-3, dropout 0.1 and batch size 512, but the paper says fine-tuning runs "until convergence" without a stated criterion [arxiv:2202.11176 app. A] [arxiv:2202.11176 §5]. ## Contradictions No numeric conflict across papers: the taxonomy reports no results [arxiv:2112.04359 Results]. The other papers measure different things, so their numbers are not directly comparable. ToxicChat scores toxicity under the SemEval-2019 guideline [arxiv:2310.17389 §2.1]. Ex Machina scores personal attacks on Wikipedia comments [arxiv:1610.08914 §3]. ToxiGen scores implicit hate on its own test sets, with labels taken from prompts [arxiv:2203.09509 §4.2]. The per-annotator paper scores majority-vote F1 on hate speech and on emotions [arxiv:2110.05719 tab. 1] [tab. 3]. The production-pipeline paper reports AUPRC on its own public test set and on a production set it does not release [arxiv:2208.03274 §4.2] [tab. 3]. KUISAIL reports macro-F1 on three OffensEval test sets [arxiv:2007.13184 §5], and Vidgen et al. report macro-F1 on their own round test sets and accuracy on HateCheck [arxiv:2012.15761 tab. 4] [§6.3]. Do not rank the F1, AUC, AUPRC and accuracy values above against each other. Two more sources use different measures. The moral-direction paper reports Pearson correlations with human yes/no ratings [arxiv:2103.11790 § BERT's MoralDirection Strongly Correlates with Human Moral Norms]. Its toxicity labels come from Perspective API at a score of 0.5 or more [arxiv:2103.11790 § Testbed for evaluating the toxicity of text generations]. ToxicChat's Perspective row uses 0.7 [arxiv:2310.17389 §3]. The two Perspective results therefore use different thresholds on different text, and they should not be compared. Vidgen et al. also report Perspective's toxicity classifier, at 77% accuracy on HateCheck [arxiv:2012.15761 §6.3]. That is a third test, with its own accuracy measure, so it cannot be set against either result above. KUISAIL reports macro-F1 on OffensEval 2020 test sets, and the card gives no Perspective comparison [arxiv:2007.13184 §5]. The hate-speech survey reports Krippendorff's alpha, which measures agreement and not accuracy [arxiv:1701.08118 §5]. On synthetic data, the papers point in different directions. ToxiGen reports gains for HateBERT and RoBERTa after fine-tuning on its generated statements [arxiv:2203.09509 tab. 4]. The production-pipeline paper reports that large GPT-3 zero-shot and few-shot sets reduced performance. It says this contradicts earlier work in low-data settings [arxiv:2208.03274 §3.3]. The two studies differ in model, data size and label method, and the card does not resolve the conflict. Perspective API appears in five papers, and each uses its own data, threshold or measure. ToxicChat reports F1 at a score of 0.7 [arxiv:2310.17389 §3]. The moral-direction paper uses 0.5 [arxiv:2103.11790 § Testbed for evaluating the toxicity of text generations]. The byte-level paper reports AUC-ROC on JMTCC (.8770) and on its production set, for English and Russian only [arxiv:2202.11176 tab. 1] [tab. 2]. The production-pipeline paper reports AUPRC on its own set [arxiv:2208.03274 tab. 3]. Vidgen et al. report accuracy on HateCheck [arxiv:2012.15761 §6.3]. These numbers are not comparable across papers. The byte-level paper's ranking against mT5-base also shifts by test set. On Production-Multilingual the 102M model is ahead, at .910 against .906. On JMTCC and TweetEval it is behind, at .9194 against .9239 and 53.3 against 57.7 [arxiv:2202.11176 tab. 1] [tab. 2] [tab. 4]. The paper's summary says it matches or beats mT5-base on average, and the table supports this only on the production set. Within the ToxicChat paper, some figures conflict and the card does not resolve them: - The trial toxicity rate is 7.22% [arxiv:2310.17389 §2.1], but 48 of the 720 trial prompts are toxic [arxiv:2310.17389 §2.2], which is 6.67%. - Perspective filtered 4,668 of 7,599 prompts in stage one, about 61% [arxiv:2310.17389 §2.2]. Appendix D's table gives 60.71% of Perspective-scored data still needing review at a 1% safe rate [arxiv:2310.17389 app. D]. ## Papers - [ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation](/papers/2310.17389): a 10,166-prompt benchmark from real first-turn user prompts, with toxicity and jailbreaking labels. It shows public detectors transfer poorly to chatbot traffic. - [Ethical and social risks of harm from Language Models](/papers/2112.04359): a taxonomy of 21 risks in six areas, with toxic language listed as one of them. It gives no detector, dataset or measurement. - [Ex Machina: Personal Attacks Seen at Scale](/papers/1610.08914): crowd labels plus a bag-of-n-grams classifier, used to label 63M Wikipedia talk comments for personal attacks and to measure attack prevalence. - [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](/papers/2203.09509): 274,186 GPT-3 statements about 13 groups, with ALICE, a classifier-guided decoder. Fine-tuning on the data raises AUC on three human-written implicit-hate sets. - [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](/papers/2110.05719): one BERT encoder with one head per annotator, each trained on that annotator's labels and voting at prediction time. On hate speech it edges out majority-label training, and its head spread tracks annotator disagreement. - [A Holistic Approach to Undesired Content Detection in the Real World](/papers/2208.03274): an eight-category production classifier built with labeling audits, active learning on live traffic, synthetic data and Wasserstein domain-adversarial training. It reports gains over Perspective API on its own public test set. - [Ross et al., Measuring the Reliability of Hate Speech Annotations](/papers/1701.08118): two surveys of 56 people on 20 German tweets each. A written hate-speech definition did not raise Krippendorff's alpha above .29, though it changed how often tweets were marked for banning. - [Schramowski et al., Large Pre-trained Language Models and moral norms](/papers/2103.11790): a moral direction in BERT-large sentence embeddings, used to score phrases and to filter GPT-2 candidates during decoding. It reports lower toxicity on RealToxicityPrompts, with a fragile dependence on the verbs used. - [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](/papers/2007.13184): a language-specific BERT-base with a CNN over its last four layers, for offensive-tweet detection in Arabic, Greek and Turkish. Its gain over BERT alone is small and the Turkish result is slightly lower. - [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](/papers/2012.15761): four rounds of annotators trying to fool the current RoBERTa-base target, with label-flipping perturbations from round 2. Later targets fool annotators less and score higher on HateCheck. The release has data and guidelines but no training code. - [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](/papers/2202.11176): a token-free byte-level encoder with a learned tokenizer, fine-tuned for multi-label toxicity scoring across many languages. It reports gains over custom mBERT and mT5 on its production set, but a smaller lead on public sets, and it is not reproducible from public data.