ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection

In plain words

Hate-speech detectors often learn shortcuts. They flag any sentence that mentions a minority group, and they miss hateful sentences that use no swear words. Human-labelled data that covers this well is scarce and expensive to collect.

The authors ask a large language model to write the data. They show it a few example sentences about one group and let it continue the list. Its outputs are a balanced mix of hostile and neutral sentences that name the group but avoid slurs. A second technique, ALICE, runs a toxicity classifier during the writing itself, so the text is pushed towards sentences the classifier gets wrong. Models trained on this text do better on human-written examples of implicit hate. Annotators often could not tell the generated sentences from human ones.

TL;DR

• Dataset: 274,186 machine-generated statements, toxic and benign in equal measure, about 13 minority groups; 98.2% contain no profanity or slur tab. 1 tab. 2.
• Generation: GPT-3 with demonstration-based prompts and top-k sampling, plus ALICE, a classifier-guided constrained beam search that is used for only about 5% of the statements §3.1 §3.2 tab. 2.
• Author-reported gains: fine-tuning HateBERT lifts AUC on three human-written implicit-hate test sets by 7 to 19 points §1 tab. 4.
• Caveats: the paper's code does not match the paper in several places (see Paper vs code and Pitfalls); the labels are proxies taken from the prompt, and their agreement with what the text actually says is only moderate.

Problem

Toxicity classifiers often flag any text that mentions a minority group, and they miss toxicity that is implicit, meaning it has no slurs or profanity §1 §2. Existing implicit-hate corpora are small, cover few groups and come from social media, and they are often unbalanced between toxic and benign text tab. 1 §2. The authors want a large, balanced, mostly implicit corpus with equal coverage of many groups, and a way to make it hard for existing classifiers §1 §3.

Method

1. Demonstration-based prompting §3.1 §3.1.1 app. A

• A prompt is a list of example sentences about one group, each prefixed with a hyphen and ending with a newline, followed by an open hyphen. The language model writes the next sentence in the same style app. A app. F.
• For each generated sentence, 5 random examples are drawn from the group's demonstration set, and a new draw is used every time app. A.
• Generation stops at the first newline app. A.
• The authors build 26 prompt sets: one toxic and one benign set for each of 13 groups §3.1.2. Benign examples come from blog posts and news articles that mention a group. Toxic examples come from hate forums [de Gibert et al., cited in §3.1.2] and Reddit [Breitfeller et al., cited in §3.1.2].
• Demonstration sets were expanded by a human-in-the-loop process: sample prompts, read the outputs, keep the best ones, repeat §3.1.2. The authors report 20 to 50 demonstration sentences per group §3.1.2.

2. Generation without ALICE §3.3

• Each prompt is decoded with top-k sampling (Fan et al., cited in §3.3) using GPT-3 as the language model.

3. ALICE: classifier-in-the-loop constrained beam search §3.2 eq. 1 fig. 2

• The decoding score of a candidate next token combines the language-model log-probability and the classifier's log-probability for the class it is pushed towards eq. 1.
• Two adversarial setups are used §3.2:
  • False negatives: toxic prompts. The classifier's benign-class probability is maximised, so the output tries to look benign while staying toxic.
  • False positives: benign prompts. The classifier's toxic-class probability is maximised, so the output tries to look toxic while staying benign.
• The authors note that the first setup can also detoxify output when the classifier wins §3.2.
• The classifier is a HateBERT model fine-tuned on OffensEval, as described by Caselli et al. (cited in §3.3). The paper states that it covers direct and veiled offence types §3.3.

4. Decoding settings §3.3

• Language-model weight and classifier weight are both 0.5.
• Maximum generation length is 30 tokens; beam size is 10; temperature is 0.9.
• The GPT-3 API only returns log-probabilities for a subset of the vocabulary, so the authors keep the top 100 tokens and then resample among tokens that do not appear in the prompt, using top-k §3.3.
• Tokens copied from the prompt are blocked to prevent direct copying. Punctuation from the prompt may still appear §3.3, fn. 6.

5. Dataset assembly §3.4 tab. 2

• 274,186 statements in total: 260,012 from top-k and 14,174 from ALICE.
• The mix is balanced between toxic and benign statements within each group (per-group counts are in the Results section).
• Generation length varies a lot. Almost all statements are implicit, by a word-list test (see Results) §3.4.
• The released data includes the prompts, the generation method, the prompt label, the group, and predictions from a RoBERTa model app. H.

6. Human validation §4 §4.1 §4.2 §4.3

• The human-validated test set ToxiGen-HumanVal has 792 statements. Each was rated by 3 Amazon Mechanical Turk workers, drawn from a pool of 156 prequalified workers §4.2. Training statements with cosine similarity above 0.7 to any test statement were removed §4.2.
• Raters answered questions on whether the text was written by a human or an AI, whether it would be harmful, the writer's intent, the targeted group, lewdness and fact versus opinion §4.1.
• A separate 125-prompt comparison measures ALICE against top-k §4.3.

7. Improving classifiers §5

• HateBERT and ToxDectRoBERTa are fine-tuned on the training portion, with the prompt label as the training label §5.
• They are tested on three human-written sets and on ToxiGen-HumanVal §5 tab. 4.

# ALICE decoding, paper's description  [§3.2] [eq. 1] [§3.3]
# Input:  prompt (str), lm (top-k log-probs), clf (toxicity classifier)
# Output: one statement (str)

def alice(prompt, lm, clf, prompt_is_toxic):
    # Beam size and length limit  [§3.3]
    beam_size = 10
    max_tokens = 30
    # Weights lambda_L and lambda_C in eq. 1  [eq. 1] [§3.3]
    lm_weight = 0.5
    clf_weight = 0.5
    # Push the classifier away from the prompt label  [§3.2]
    if prompt_is_toxic:
        target_class = BENIGN
    else:
        target_class = TOXIC
    beams = [(prompt, 0.0)]
    for step in range(max_tokens):
        # Top-100 next-token log-probs for every beam  [§3.3]
        candidates = lm_top_logprobs(lm, beams, vocab_size=100)
        # Drop prompt tokens, but keep stop words and punctuation  [§3.3]
        candidates = drop_prompt_tokens(candidates, prompt)
        # Classifier log-prob of the target class for each candidate  [§3.2]
        clf_scores = clf_logprob(clf, beams, candidates, target_class)
        # Combine the two scores per eq. 1  [eq. 1]
        scores = lm_weight * candidates + clf_weight * clf_scores
        # Keep the best beam_size hypotheses; end ones that emit newline  [§3.3]
        beams = keep_best(beams, candidates, scores, beam_size)
    return best_hypothesis(beams)

Hyperparameters

Results

Author-reported. Setup: GPT-3 generation; HateBERT and ToxDectRoBERTa (called RoBERTa in the table) fine-tuned on the training split; tested on three human-written datasets and ToxiGen-HumanVal §5 tab. 4.

Fine-tuning, AUC tab. 4. "None" means the off-the-shelf model with no fine-tuning. The three fine-tuning columns use equal-sized data: ALICE only, top-k only, or both. The "ALICE + top-k" column is the combined set.

Values transcribed from the arXiv v4 PDF; see Pitfalls.

Dataset statistics tab. 2. Avg. chars is the mean sentence length in characters. % implicit is the share with no profanity or slur, using a word list §3.4 [fn. 7].

Dataset comparison tab. 1. ToxiGen: 274,186 statements, 98.2% implicit, 50.1% hate class. The other ten datasets in the table range from 2,934 to 80,000 items, with 26.1% to 99.4% implicit and 5.0% to 100.0% hate class.

Human validation §4.2 fig. 3 fig. 4 fig. 5

• 90.5% of machine-generated examples were judged human-written by a majority of raters. Toxic examples were judged human-written 92.9% of the time, against 90.2% for benign ones §4.2.
• 94.56% of toxic examples were labelled as hate speech §4.2.
• 30.2% of machine-generated examples were judged harmful (score above 3 on the 1-to-5 scale), and 4% were ambiguous §4.2.
• Agreement: Fleiss' kappa 0.46 and Krippendorff's alpha 0.64, described as moderate. All three raters agreed on 55.17% of items; at least two agreed on 93.4% §4.2.

ALICE against top-k, 125 prompts §4.3

The toxicity difference is reported as significant at p below 0.001 §4.3.

Larger human study, about 9,000 training samples app. E

• 8,960 training statements were rated, split 50.9% top-k and 49.1% ALICE.
• Toxic prompt labels matched human toxicity for 66.86% of ALICE statements and 57.91% of top-k statements. Benign prompt labels matched for 93.21% and 90.01%.
• Fooling HateBERT: 58.97% of toxic ALICE statements against 26.88% of toxic top-k statements; 15.51% against 11.35% for benign statements.
• At least one rater named the target group for 78.3% of ALICE statements and 70.4% of top-k statements.

Other analyses

• Perplexity under GPT-2, with sentences above 500 dropped: ALICE total 23.54, top-k total 105.31. ALICE is lower for every group app. C.
• Target-group mention rate, all groups: ALICE 0.84, top-k 0.81 app. D.

Paper vs code

The official code is at microsoft/ToxiGen, commit 01517e814639e0334c920a3c502036c8158aac87, under the MIT licence LICENSE.txt. Its ALICE file carries an Apache-2.0 header for beam-search code adapted from Hugging Face. None of it is copied here.

• Vocabulary. The paper says the top 100 tokens are kept §3.3. The code sets vocab_size = min(5, vocab_size), so only 5 candidates per beam are used by default, with a comment saying the limit depends on the OpenAI API alice.py.
• Classifier term applied twice. The paper has one weighted sum eq. 1. The code computes the weighted sum twice in one loop, once in lines 152-153 and again in lines 159-160 alice.py. With both weights at 0.5, the net effect is 0.25 for the language model and 0.75 for the classifier, not 0.5 and 0.5. Both passes compute the same classifier logits, so the second pass repeats the first rather than refining it.
• Scored text. Eq. 1 scores the classifier on the sequence w_0 to w_{i+1}. The code scores only the generated words after the prompt, by splitting off the first len(prompt.split(' ')) words alice.py, alice.py.
• Prompt output. beam_search starts every hypothesis from the full prompt string and returns the best hypothesis as a whole alice.py, alice.py. The returned text therefore starts with the prompt. The paper does not say whether the prompt is removed before release.
• Prompt-token blocking. The paper blocks tokens that do not appear in the prompt §3.3. The code blocks a candidate if it is absent from the stop-word list and is a substring of the prompt text alice.py. Substring matching can block more tokens than exact-token matching.
• Classifier weights. The paper names the HateBERT OffensEval fine-tune §3.3. The code loads the tokenizer from the base GroNLP/hateBERT, and the weights from a path the user supplies pretrained_classifiers.py, with the path left as <path_to_hatebert_files> generate.py. The checkpoint used in the paper is not identified in the code.
• Quote stripping. The GPT-3 client removes single and double quotes from every prompt before sending it language_models.py. The paper does not mention this.
• Prompt sets. The paper describes 26 sets over 13 groups §3.1.2 and 26,000 prompts released app. F. The repo has 31 prompt files of 1,000 lines each (16 toxic, 15 benign), including bisexual, immigrant and trans sets that are not among the 13 groups . The README says the immigrant and bisexual demonstrations were community contributions in March 2024 README.md.
• Demonstration counts. The paper reports 20 to 50 demonstrations per group §3.1.2. Four demonstration files sampled from the repo have 7 to 14 lines each . The repo may be a subset of the full set, which cannot be checked here.
• Top-k path. generate.py without --ALICE writes the raw API response dictionary to the output file, which the line f.write(response + "\n") cannot do, since a dictionary cannot be concatenated with a string generate.py, language_models.py. The paper's top-k generation therefore cannot be reproduced from this entry point as written.
• Flag parsing. --ALICE is declared type=bool, so any non-empty string, including False, is read as true generate.py.
• Fine-tuning. The paper describes fine-tuning HateBERT and ToxDectRoBERTa §5. The repo has no training script; a text search for learning rate, epochs, batch size and fine-tuning found only the generation code and the README.
• Helper script name. The README names make_prompts.py, but the file is demonstrations_to_prompts.py README.md, demonstrations_to_prompts.py.

Pitfalls

• Missing tables in the source archive. The LaTeX tables/ directory is empty, so Tables 1, 2 and 4 are not in the source. I transcribed them from the arXiv v4 PDF as a rendered page image, with no text layer. Check every digit against the PDF before relying on it. Cells in the rendered page are small, so a misread is possible.
• Vocabulary limit. The paper's 100-token vocabulary is cut to 5 in the released code. Reproducing the paper's ALICE data needs an API that returns log-probabilities for 100 tokens, and the code has to be changed.
• Double weighting. The classifier term is applied twice, which changes the balance between language model and classifier (see Paper vs code). If you implement from the paper, use one weighted sum. If you match the released data, the code's behaviour is the reference.
• Prompt in the output. Strip the prompt prefix from ALICE output before using it as a statement. The paper does not say whether the released generation field has it removed.
• Gains are AUC points, not percent. The introduction says fine-tuning improves performance by 7 to 19% on three human-written sets §1. In Table 4, that range matches HateBERT with ALICE + top-k: +11 SBF, +7 IHC, +19 DynaHate AUC points. The RoBERTa gains on the same sets are +5, +9 and +5 AUC points. The paper does not state the metric unit in the text tab. 4.
• Labels are prompt proxies. The training label is the prompt's label §5. Human-checked agreement with the prompt label is 67.7% for top-k toxic prompts and 40.3% for ALICE toxic prompts §4.3. Toxic-prompt data is noisy, especially for ALICE. The released RoBERTa predictions are meant to help correct some labels app. G.
• ALICE is a small share. ALICE contributes 14,174 of 274,186 statements, or about 5.2% tab. 2. Results that mix the two methods mostly describe top-k data.
• Fine-tuning recipe is absent. Learning rate, epochs, batch size, sequence length and seed are not stated. The LaTeX source has a commented-out draft with a learning rate and epoch count. It is not in the published paper, so it is not used here.
• Stop words and punctuation. Stop words and punctuation are exempt from the prompt-token block, so the generated text can repeat function words that appear in the prompt §3.3 alice.py.
• Retry loops with no backoff. The API calls in beam_search retry indefinitely on any exception, with no delay and no error limit [code: toxigen/alice.py#L109-L116], alice.py. A persistent error will spin without stopping.
• Human-validation labels. Toxic and benign labels are derived from the maximum of two harm scores, mapped to three classes where exactly 3 is ambiguous §4.2. Agreement is moderate (kappa 0.46) §4.2. Treat the per-statement labels as noisy.
• Intro versus results. The introduction says 90% of 792 sentences are mistaken for human-written text §1, while §4.2 says 90.5%. Use §4.2, which gives the exact figure.
• Perplexity filter. The perplexity analysis drops sentences above 500 under GPT-2 app. C. The filter is for that analysis only, and the released dataset is not filtered by it.
• Ethics. The authors name misuse risk: the method makes the generator more toxic on purpose, and the code and data are public §7. The paper also states its dataset covers only implicit toxicity for 13 groups and can be noisy §7.

Lineage

• Builds on: GPT-3 as the generator (Brown et al., cited in §1); demonstration-based prompting (Gao et al.; Mishra et al., both cited in §3.1.1); top-k sampling (Fan et al., cited in §3.3); constrained beam search (Anderson et al.; Hokamp and Liu; Lu et al., cited in §3.2); HateBERT (Caselli et al., cited in §3.3); and ToxDectRoBERTa (Zhou et al., cited in §5).
• Compared against: Table 1 compares ten existing hate-speech datasets, including ImplicitHateCorpus, SocialBiasFrames and DynaHate tab. 1.
• Related concepts: toxicity-detection, controllable-text-generation, constrained-decoding, prompt-engineering (see concepts: above).