ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversation

In plain words

Chatbots get asked rude, hateful and tricky questions by real people. Most tools for spotting toxic text were built on social-media posts, and people talk to chatbots in a different way: they ask questions and give instructions. The authors collected real first questions sent to an open chatbot demo and had them labelled as toxic or not.

Labelling everything by hand is slow, so the team first let a moderation service screen out the clearly harmless questions and only sent the rest to people. They also found a special group of "jailbreak" prompts, which dress up a harmful request so the bot will ignore its rules. Those get their own label.

The result is a benchmark that shows that toxicity detectors trained on other text fail on chatbot traffic, and that a model trained on the new data does much better on it.

TL;DR

• ToxicChat has 10,166 real user prompts from the Vicuna demo, with a binary toxicity label and a separate jailbreaking label §2 §2.2.
• Public toxicity models and APIs (OpenAI Moderation, Perspective, HateBERT, ToxDectRoberta) score poorly on it §3 tab. 1.
• A RoBERTa-base trained on the ToxicChat training half beats RoBERTa models trained on six other toxicity datasets (and their combination) on the same test set, with F1 72.1 against at most 51.6 for the strongest alternative tab. 2.
• All numbers are author-reported. No official code was found.

Problem

Toxicity detection work has mostly used social-media or LLM-generated text §1. Chatbot users write questions and instructions, and they may hide intent behind harmless-looking wording, so detectors trained on other domains may miss toxic chatbot traffic §1 fig. 1. The paper asks whether existing detectors and public moderation tools work on real user-AI conversations §1 §3.

Method

Data collection. Prompts were sampled at random from user inputs to the Vicuna online demo between 30 March and 12 April 2023. Only the first user turn is used §2. Preprocessing removes uninformative content, very short prompts (fewer than three words) and non-ASCII-heavy text, keeps only English prompts using a fastText language identifier, and masks emails, phone numbers and addresses app. B.

Labels. Each prompt gets a binary toxicity label (1 = toxic). The definitions follow the SemEval-2019 undesired-content guidelines §2.1. A separate jailbreaking label marks prompts that try to make the chatbot produce toxic content through prompt tricks §1 §2 app. C tab. 4.

Annotation. Four researchers annotated a common trial set of 720 prompts. The inter-annotator agreement was 96.11% and the toxicity rate 7.22% §2.1. For the full set, the authors used a human-AI pipeline §2.2 app. D:

# Human-AI labelling pipeline  [§2.2] [app. D]
# Input:  prompts (first-turn user texts)
# Output: labels (dict: prompt -> 0 or 1)

def annotate(prompts, perspective_score):
    # Prompts below the threshold are labelled non-toxic without review
    # (threshold 10**-1.43, about 0.037)  [§2.2] [app. D]
    labels = {}
    to_review = []
    for prompt in prompts:
        if perspective_score(prompt) < 10 ** -1.43:
            labels[prompt] = 0
        else:
            to_review.append(prompt)
    # Every reviewed prompt gets two independent human labels  [§2.2]
    for prompt in to_review:
        first, second = two_annotators_label(prompt)
        # Disagreements are discussed until the annotators agree  [§2.2]
        if first == second:
            labels[prompt] = first
        else:
            labels[prompt] = discuss_until_agreed(prompt)
    return labels

After annotation, the authors used GPT-4 to re-check the non-toxic labels for toxic prompts that annotators missed. They also labelled jailbreaking text with the same process §2.2.

Benchmark size. The first stage collected 7,599 prompts. Perspective filtered out 4,668 of them, and 2,931 were annotated. A second stage added 2,756 manually labelled prompts to enrich the test set. After a manual check, 10,166 prompts remain. The toxicity rate is 7.10% and the jailbreaking rate 1.75% §2.2.

Baselines. OpenAI Moderation (its binary output), Perspective API (score threshold 0.7), HateBERT and ToxDectRoberta are scored on the human-annotated part of the test set. Precision, recall, F1 and jailbreaking recall (JR) are reported §3 tab. 1. JR is the share of jailbreaking prompts the model flags as toxic §3.

Domain study. Six public toxicity datasets (HSTA, Movie Reviews, Jigsaw, ToxiGen, RealToxicPrompts, ConvAbuse) are subsampled and used to train roberta-base separately, plus one model on their combination and one on the ToxicChat training half. All seven are tested on the ToxicChat test half, with five random seeds §4.1 app. A tab. 2.

Chatbot output study. Three feature sets are tried with roberta-base: the user input (default), the Vicuna response, and both concatenated §4.2 tab. 3.

Hyperparameters

Results

Author-reported. Metrics are computed on the human-annotated part of the test half §3.

Off-the-shelf moderation tools and models tab. 1

RoBERTa-base trained on each dataset, tested on ToxicChat (mean (std) over 5 seeds) tab. 2

RoBERTa-base with different inputs, trained and tested on ToxicChat tab. 3

The paper reports the Input + Output gain as not significant §4.2. A figure in the appendix plots how the OpenAI Moderation and Perspective scores are distributed over the trial set app. D fig. 2; it is described here, not reproduced.

Paper vs code

No official code. Searched: the paper and arXiv abstract page list no repository; a GitHub repository search for "toxicchat" returned only third-party projects; a search restricted to the lm-sys organisation (which hosts the dataset on Hugging Face §1) returned none; a search with title words was blocked (HTTP 403). The dataset link is the only resource named §1.

Pitfalls

• Trial toxicity rate conflicts. The text gives 7.22% for the 720 trial prompts §2.1, but also says 48 of them were toxic §2.2, which is 6.67%.
• Filter share conflicts. Perspective filtered 4,668 of 7,599 prompts in stage one (about 61%) §2.2. Appendix D's table gives 60.71% of Perspective-scored data still needing human review at a 1% safe rate app. D. Those two figures point in opposite directions. The card does not resolve this.
• Human-labelled count is not stated for the final set. Stage one annotated 2,931 and stage two 2,756, so 5,687 before the manual removal step §2.2. The paper does not say how many of the 10,166 final prompts were human-labelled, or whether the Perspective-filtered prompts remain in the release.
• Evaluation excludes the auto-labelled part. Reported metrics use only the human-annotated test portion §3. The authors argue that the auto-labelled part matters little §2.2, but this is an assumption, not a measurement. The authors estimate that the Perspective filter missed 1 of 48 toxic trial prompts §2.2.
• Two different Perspective thresholds. The pre-filter uses 10^-1.43 §2.2 app. D. The baseline uses 0.7 §3. Do not swap them.
• Two RoBERTa sizes appear. The domain and feature studies train roberta-base §4.1 tab. 2 tab. 3. ToxDectRoberta is a RoBERTa-large base app. A. Do not treat the two as the same setup.
• Subsample sizes are not stated. The introduction says prior datasets are "10 times larger" than ToxicChat's training data §1. Section 4.1 only says ToxicChat's training data is "not particularly larger" §4.1. No sizes are given for the subsamples, so the claim cannot be checked from the card.
• Training hyperparameters are not stated (learning rate, batch size, epochs, max length, optimizer). The paper points to each source dataset's suggested settings §4.1. An implementation has to choose its own values and say so.
• Jailbreaking and toxicity labels overlap unclearly. The paper says jailbreaking gets "an special label" §1 and reports JR as the share of jailbreaking prompts flagged as toxic §3. It does not say whether jailbreaking prompts also carry toxic = 1 in the release. Check the dataset before training.
• Access and license. The introduction gives a Hugging Face link §1. The ethics section says access will be gated by a request form [Ethical Considerations]. The paper does not state the dataset license. Confirm both before using the data.
• Scope. Only first turns, only English (over 99%), only one chatbot demo §2 [Ethical Considerations] [Limitations]. The paper names higher-traffic chatbots as the better source but had no access to them [Limitations].
• Model-in-the-loop checks. GPT-4 was used to re-check non-toxic labels §2.2. The paper does not report how many GPT-4 flags were accepted.
• Appendix and figure numbers are inferred. Appendix letters A to D follow the order of the LaTeX \appendix sections app. A-D. Figure and table numbers follow LaTeX order. Check them against the PDF before citing.
• The safe-rate table in appendix D is unlabelled and is cited here as app. D.

Lineage

• Builds on: the idea of combining a moderation API with human annotation to cut workload, inspired by Kivlichan et al. (2021), cited §2.2. No arXiv card.
• Compared against: OpenAI Moderation, Perspective API §3, HateBERT and ToxDectRoberta app. A, and six public toxicity datasets §4.1.
• Superseded by: not stated.