A Holistic Approach to Undesired Content Detection in the Real World

In plain words

Anyone who lets people type into a language model needs a filter that flags text which is sexual, hateful, violent, self-harming or harassing. This paper describes how the authors built one that works on real user traffic, not only on the public datasets researchers usually use.

The core idea is a sequence of disciplined steps. Write labeling rules that are concrete enough that annotators agree. Audit the labels and look for mistakes. Keep searching the real traffic for the rare bad examples, because random sampling almost never finds them. Add synthetic examples where the data has gaps. Use a training trick that pushes the model to ignore whether a text came from public data or from production. Each step is simple on its own; the paper's claim is that the chain as a whole is what makes the classifier work.

The authors report that the resulting model beats an existing off-the-shelf moderation API on several of their test categories. They also say the method is meant as a blueprint that can be adapted to other category sets.

TL;DR

• A transformer classifier predicts eight undesired-content categories, with one MLP head per category §4.1.
• Production data is the bottleneck: public data is shifted, and rare categories need active learning on live traffic (about 10 to 22 times more undesired samples than random sampling, author-reported) §4.3 tab. 4.
• Labeling quality is managed with concrete rules, calibration sessions, per-category audits and a mislabel-detection loop §3.2.
• Wasserstein domain-adversarial training (WDAT) helps most when only public data is available; its gain shrinks as in-distribution data grows §4.4 tab. 5.
• Author-reported AUPRC beats Perspective API on the public test set for four categories, but the comparison is approximate because the taxonomies differ tab. 3.

Problem

Undesired-content detection fails in several ways that the paper names explicitly:

• No widely agreed taxonomy exists, and labeling depends on subjective, culturally shaped judgments §1 §2.
• Public and academic datasets are shifted from production traffic: production prompts are longer, often contain few-shot prompts, and their categories do not match academic taxonomies §1 §3.4.
• Some categories are rare in real traffic. The authors saw self-harm in 0.04% of sampled user prompts and hateful content with threats in 0.017% §1.
• Prior work covers narrower sets, such as toxicity, hate speech and abuse, or targets one use case §1 §5.
• For some categories (self-harm, graphic violence) no usable public data exists to start with, which is the cold-start problem §3.3.

Method

Taxonomy §2. Five top-level categories: sexual (S), hateful (H), violence (V), self-harm (SH) and harassment (HR). Each has subcategories with a severity scale. The model predicts eight labels: the five top-level categories plus three severe subcategories, sexual content involving minors (S3), hateful content with threats or calls to violence (H2) and extremely graphic violence (V2) §2. The other subcategories (S1, S2, H1, V1) are defined in the taxonomy but are not model outputs §2.

Model §4.1. A lightweight transformer decoder whose final output layer is replaced by eight MLP heads, one per category. Each head is written as one matrix of shape [d_model, 256, 1]. The authors say this avoids interference between categories and needs fewer parameters than one deep MLP with an eight-dimensional output §4.1. The network is initialised from a pretrained GPT model and fine-tuned §4.1. The paper does not state the GPT model size or d_model.

Data selection and active learning §3.1 fig. 1. Repeated in rounds:

1. Draw a large random sample of production traffic. Mask personally identifiable information. Score it with the latest model and see which samples trigger any category §3.1.
2. Select a subset with three parallel strategies: random sampling (keeps the data distribution), threshold sampling (samples whose score for a category is above a threshold, to find likely undesired items) and uncertainty sampling (samples whose score for a category is closest to 0.5) §3.1.
3. Pool the selected samples and re-weight them using metadata counts. The sampling weight is proportional to the square root of the sample count §3.1. The paper does not define which metadata or which count.
4. Adjust the mixture of sub-strategies over time, as the data and the categories that need work change §3.1.

The loop as run in the active-learning experiment §4.3:

# Iterative active learning  [§4.3] [§3.1]
# Input:  labeled set (6000 public examples), validation set (~5500
#         production samples), production pool (unlabeled)
# Output: model after N = 3 rounds

def run_active_learning(labeled, validation, pool):
    for round in range(3):
        # Train a new model on all labels collected so far  [§4.3]
        model = train(labeled)
        # Measure on the fixed production validation set  [§4.3]
        report = evaluate(model, validation)
        # Score about 5e5 production samples with this model  [§4.3]
        scored = score(model, pool)
        # Three sub-strategies: random, above-threshold, uncertain  [§3.1]
        candidates = (
            random_part(scored)
            + above_threshold_part(scored)
            + uncertain_part(scored)  # score for a category nearest 0.5
        )
        # Re-weight candidates by metadata counts, then keep ~2000  [§3.1]
        picked = weighted_subsample(candidates, count=2000)
        # Label the picks and add them to the training set  [§4.3]
        labeled = labeled + label(picked)
    return train(labeled)

Labeling and quality control §3.2.

• Instructions are made well-defined (mutually exclusive categories and subcategories) and concrete (examples and clearer rules for borderline cases). Regular calibration sessions review disagreements between external annotators and internal auditors §3.2.
• Audit: per category, sample 10 items labeled undesired and 10 items whose model probability is above 50%. The first set measures false positives; the second estimates recall. Compute F1 per category, with annotator labels as predictions and auditor labels as ground truth §3.2.
• Mislabel detection: periodically split the training set in two, train one model on each half, and score the other half. A sample is flagged when the prediction disagrees with its label. A random part of the flagged samples is audited. If more than 30% of those are mislabeled, all flagged samples are labeled a second time §3.2.

Synthetic data §3.3 tab. 1 tab. 2.

• Zero-shot cold start: prompts built from human-written templates with random ingredients for diversity are sent to a GPT-3 model; the outputs are labeled as the first training set. The template example in tab. 1 is for self-harm §3.3 tab. 1.
• Few-shot for rare categories: prompts built from existing undesired examples generate more text, which is labeled and manually inspected to limit bias amplification §3.3.
• Curated counterfactual data: templates such as "[subject] is selfish / foolish / narrow-minded", where the subject is either a demographic term (hateful) or a random object (safe). These target a bias the authors observed, where a phrase naming a group was flagged as hateful §3.3.
• Large noisy synthetic sets hurt: adding zero-shot and few-shot data at twice the size of the labeled training set reduced performance. The authors suspect the label noise §3.3.

Domain adversarial training (WDAT) §3.4. The feature extractor is the transformer encoder. A classification head predicts labels; a critic head maps embeddings to a real number. The critic loss is the absolute difference between the mean critic output on source (labeled public) and target (unlabeled production) embeddings. The total objective is the classification loss plus λ times the maximum of the critic loss over the critic parameters, minimised over the encoder and classifier parameters. The equation in the paper is unnumbered §3.4. The Lipschitz constraint is enforced by clipping the critic weights to [-0.01, 0.01], and λ = 0.01 §3.4. The authors chose this Wasserstein variant over a classifier-based domain adversary because it was more stable and performed better on production traffic §3.4.

# One WDAT training step  [§3.4]
# Source: labeled public batch. Target: unlabeled production batch.
# lambda = 0.01; critic weights clipped to [-0.01, 0.01]

def wdat_loss(source_x, source_y, target_x):
    # Shared transformer encoder embeds both domains  [§3.4]
    source_h = encoder(source_x)
    target_h = encoder(target_x)
    # Classification loss on labeled source data  [§3.4]
    class_loss = classification_loss(head(source_h), source_y)
    # Critic gap between domains, absolute value  [§3.4]
    gap = abs(mean(critic(source_h)) - mean(critic(target_h)))
    # Encoder and head minimise this. The critic is trained to
    # maximise gap, with its weights clipped after each update  [§3.4]
    return class_loss + 0.01 * gap

Model probing and red-teaming §3.5. Input reduction removes tokens greedily until the prediction drops below 0.8. On average this cut the characters per sample from 722.3 to 15.9. Cross-checking the remaining key tokens against third-party APIs and keyword lists judged about 97% as unsafe. Problems found: irrelevant symbols carry score, some key tokens are long enough to suggest memorised training samples, and typos and adversarial input are handled poorly §3.5. Red-teaming before each release found hate scores triggered by the "#" token and by racial terms such as "black" §3.5. Both were patched with synthetic data and mix adjustments §3.5.

Figure 1 shows the framework in words: public crawled, academic and generated data form the cold-start set; training data combines it with production data; overfitting-phrase detection and mislabel detection feed the training data; the model is trained, evaluated, and used to select unlabeled production samples for labeling; DAT is drawn as a method on the training path. Augmentation and fairness work are drawn as future work fig. 1. The figure also draws "user feedback" as a method, but the text does not discuss it fig. 1.

Hyperparameters

Not stated anywhere in the paper: optimizer, learning-rate schedule, warmup, the sizes of the GPT base model, the number of runs behind the main model's results, and the critic head architecture.

Results

Author-reported. Setup: a classifier trained on public and production data, tested on the paper's public test set and on four public datasets §4.2. The production test set is not shared §4.2. The baseline is Perspective API, with approximated taxonomy mappings marked by an asterisk tab. 3 app. A, tab. 6.

Share of samples carrying each label, random sampling versus active learning. The paper does not state the number of samples behind each column. Multipliers are the paper's (active-learning share divided by random share) tab. 4.

Overall, about 40% of active-learning samples carry at least one undesired label, against 3.4% of random samples §4.3. The paper's active-learning AUPRC curves over three sampling iterations are shown for eight categories fig. 2; the body gives no numbers for them, and the section says active learning gives a greater gain in every category §4.3.

WDAT, average AUPRC on a production validation set, by training set tab. 5 §4.4:

PUB = public data only; SYN = PUB + 69k curated synthetic; MIX = SYN + 60k labeled production. Bold is the better of each pair tab. 5. The paper gives no spread across the three seeds.

Paper vs code

The linked repository at code_commit is the evaluation dataset release, not the model or its training code. It contains a README, an MIT LICENSE and data/samples-1680.jsonl.gz, and the README says that file is the test set used in the paper README.md. It holds no labeling, active learning, WDAT or model code, so none of the method can be checked against code. The repository is archived on GitHub.

Differences and checks:

• Label semantics: the README says the category label is a binary flag, and a missing label means the label is unknown, not safe README.md. An implementer who treats a missing key as negative will mis-score the data.
• Category wording: the README defines violence as content that promotes or glorifies violence or celebrates suffering or humiliation [code: README.md#L20-L20]. The paper defines it as content that depicts or shows support for physical violence §2. The README also gives its own H2 and V2 wordings README.md. Use the paper's definitions for the classifier and treat the README as the release's description.
• Sample count: the file name says 1680 samples. The paper does not state the size of its public test set §4.2, so this number is not confirmed.
• Model access: the model is served through the OpenAI Moderation endpoint §1 footnote. The paper does not say that weights are released.

No discrepancies can be checked beyond these, because the repository does not contain the method's code.

Pitfalls

• Equations are unnumbered. The WDAT objective (§3.4) and the critic loss are unnumbered in the source, so there is no eq. N to cite. Cite §3.4.
• Learning rate 0.05 is as written. It is far above typical fine-tuning values. Optimizer, schedule and warmup are not stated. Do not change it silently; test it first.
• Intro claim without numbers. The introduction says active learning improves rare categories by up to 10× §1. The body's per-category AUPRC numbers that would support this were commented out of the source; only the trend in fig. 2 remains. The 22× figure matches the table maximum tab. 4.
• Two different architectures. The main model is a transformer with eight heads §4.1. The DAT experiments use two hidden layers of 300 units §4.4. The paper does not say which of these produced the production results in tab. 5 or which model the active-learning experiment uses.
• Multiclass versus per-category. WDAT describes the classifier head f_c as multiclass §3.4, while the production model has eight independent per-category heads §4.1. The paper does not reconcile the two. The loss function for the heads is not stated.
• Critic details. WDAT takes the absolute value of the critic loss because the initial loss can be negative, and enforces the Lipschitz constraint by weight clipping to [-0.01, 0.01] §3.4. The critic architecture is not stated beyond "a domain critic head".
• "Slightly hurt" understates a drop. The text says WDAT may slightly hurt H and V with sufficient in-distribution data §4.4. In tab. 5 the SYN H baseline is .551 and the DAT value is .476, a drop of 0.075.
• Uncertainty and threshold sampling are underspecified. The threshold value, the random-sampling fraction and the metadata used for the square-root weighting are not stated §3.1. The earlier appendix with those numbers is commented out in the source.
• Synthetic labels are noisy by design. Zero-shot labels are set from what the prompt asked for, and few-shot labels from whether all examples were undesired §3.3. The paper reports that mixing these in hurt performance, and says this contradicts earlier work in low-data settings §3.3.
• Input reduction and key tokens. The 0.8 threshold is not tied to a named category score, and the 97% figure rests on cross-checks against third-party APIs and keyword lists rather than human review §3.5.
• Perspective comparison is approximate. Several cells come from approximated taxonomy mappings and are marked with asterisks tab. 3 app. A, tab. 6. Perspective is called with language "en" for all text, although about 4% of the evaluation set is non-English app. A.
• Variance is missing. Only the DAT experiment reports averages over three seeds, and it gives no spread §4.4. The main classifier's results come without error bars.
• Taxonomy fit. The authors say the taxonomy is US-centric and that categories and labels must be checked for each deployment §7.1. The classifier is English-focused: about 5% of the training samples are non-English and non-English performance was not evaluated §6.
• Harassment availability. A footnote says the harassment category was under improvement and not yet on the public endpoint at the time §1 footnote. The paper does not say whether it has since been released.
• Data handling. No production data is released §7.4. Customers' consent and opt-out are described §7.4, but these are the authors' statements and are not verifiable from the paper.

Lineage

• Builds on: Wasserstein domain adversarial training, following Shen et al. 2018 and the Wasserstein GAN critic of Arjovsky et al. 2017, with weight clipping §3.4. Input reduction follows Feng et al. 2018 §3.5. Active learning uses uncertainty sampling in the style of Lewis and Gale 1994 §3.1.
• Compared against: Perspective API on the public test set, TweetEval, the Stormfront hate dataset, the Reddit sexual-content subset and Jigsaw tab. 3 §4.2.
• Superseded by: none known in this wiki.