Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection

In plain words

Hate-speech detectors trained on a fixed collection of posts tend to miss new tricks, such as misspellings, or wrongly flag posts that quote hate to condemn it. This paper changes how the training data is made. A model is trained, then trained people try to write posts that fool it. Those posts are checked, labelled and added to the training data, and a new model is trained on the bigger set. The loop repeats.

Half of the later-round posts are small edits of an earlier post that flip its label, so the model has to learn what actually changes the meaning. The result is a dataset of about 41,000 synthetic posts, labelled as hateful or not, with a type and a target for the hateful ones. Each later model in the loop is harder for annotators to trick and does better on a separate test suite of functional checks.

TL;DR

• Four rounds of human-and-model-in-the-loop data collection on English online hate, with trained annotators.
• Dataset of 41,255 entries, 54% hateful, with type and target labels; the release has an annotator ID for each entry §4 README.md.
• Target models are RoBERTa-base; the in-loop error rate falls from 54.7% (round 1) to 27.7% (round 4).
• Author-reported gains: test macro F1 and HateCheck accuracy both rise across rounds.
• The public repository has the data and guidelines but no training code, so model details come from the paper.

Problem

Static hate-speech benchmarks lack variety, are often biased by how they were sampled, and go out of date as language changes. Models trained on them make false positives (identity terms, reclaimed slurs, counter speech) and false negatives (misspellings, obfuscation, subtle or implicit hate) §1 §2. The authors argue that a fixed dataset cannot show these weaknesses in time, so the data should be collected adaptively against the model that will be trained on it §1 §2. Their setting assumes a team of expert-overseen annotators and a serving backend that can run the target model with low latency §7.

Method

The loop has one target model per round. Annotators try to make the current target err, the entries are validated, and the next target is trained on the accumulated data §1 §5.

1. Labels. Each entry is Hate or Not Hate. Hate entries also carry a type (Animosity, Dehumanization, Derogation, Support for hateful entities, Threatening) and one or more targets §3 §3.1 §3.2. Round 1 did not record type or target §5.1.
2. Round 0 base data. The first target, M1, is trained on R0: eleven public English hate and toxicity datasets, 468,928 entries, of which 22% are hateful or toxic. Usernames are replaced and URLs are replaced with a token §5.1.
3. Round 1. Annotators write synthetic content to trick M1, using their own creativity and the real-time feedback from the model §5.1. Every entry is checked by one other annotator; disputed entries go to an expert §5.1.
4. Round 2. Annotators get adversarial pivots (10 hateful and 12 not-hateful categories) §5.2 app. B. Half the round is original content; the other half is perturbations of originals, written offline without model feedback §5.2. Perturbation guidance has four points: keep the text realistic, meet the flipped label and type, maximise diversity, and change as little as possible §5.2. Templated entries that differ only in the identity term were banned after this round §5.2.
5. Round 3. Annotators search real online hate for inspiration. Every real-world item gets at least one substantial adjustment before it is shown to the model; half the round is perturbations §5.3.
6. Round 4. As round 3, and each annotator is also assigned one target identity to focus on §5.4.
7. Validation. Originals that fooled the previous target are checked by three to five other annotators; perturbations by one §5.2. Each entry is marked correct, incorrect or flagged. In round 1, entries marked incorrect went to an expert: 1,011 entries, 385 of them excluded §5.1. In round 2, originals flagged by at least two validators and perturbations flagged by one went to an expert: 760 entries §5.2. Expert review covered 981 entries in round 3 §5.3 and 967 in round 4 §5.4.
8. Splits. Each round is split 80/10/10 into train, dev and test §5. About half of each test set comes from annotators who do not appear in that round's train or dev data, 1 to 4 per round §5. The authors use this to limit annotator bias in evaluation §5 §4.
9. Target training. The target for round k is trained on the training data of target k-1 plus earlier-round training data, duplicated by a factor chosen on the dev set §5 app. D. The factors reported are: M2 uses round 1 duplicated five times; M3 uses round 2 duplicated 100 times; M4 uses one copy of round 3 §5 app. D. The grids searched were 1, 5, 10, 20, 40, 100 (plus 200 for round 3) app. D.
10. Baselines. Non-target models M(X) are trained from scratch on the listed data, with 5 seeds and 4 epochs, using early stopping on dev loss. The cumulative models M(R0+...) add data without duplication §6.2 app. D.
11. Evaluation. The in-loop model error rate is the share of annotator entries that tricked the target §6.1. Each model is scored by macro F1 on each round's test set §6.2. Robustness is scored on the HateCheck functional suite §6.3.

# Dynamic data collection loop  [§1] [§5]
# Input:  previous target model, earlier-round data
# Output: new round data and the next target model

def run_round(target, earlier_data):
    # Annotators write posts that the current target gets wrong  [§5.1]
    entries = annotators_try_to_fool(target)
    # Originals and perturbations are checked and reviewed  [§5.2]
    labelled = validate_and_review(entries)
    # Split 80/10/10 for this round  [§5]
    train, dev, test = split_80_10_10(labelled)
    # Next target is built from earlier data plus this round  [§5] [app. D]
    next_target = train_target(earlier_data, train, dev)
    return train, dev, test, next_target

# Target model training  [§5] [app. D]
# Input:  earlier-round train data, last round's train data, dev data
# Output: fine-tuned RoBERTa-base classifier

def train_target(earlier_data, last_round_train, dev, factor):
    # Duplicate the last round by a factor chosen on dev  [§5] [app. D]
    data = earlier_data + repeat(last_round_train, factor)
    # Fine-tune with lr 2e-5; early stop on dev loss  [app. D]
    model = roberta_base_sequence_classifier()
    model.fit(data, learning_rate=2e-5, epochs=epochs)
    return model

Hyperparameters

Results

Author-reported. Setup: 41,255 entries over four rounds; RoBERTa-base; per-round test sets; M(X) baselines trained from scratch; HateCheck (29 functional tests, 3,728 labelled entries) §6.3 §5.

Error rate by round tab. 3: the in-loop error rate is the share of entries that tricked the target.

Macro F1 on each round's test set, mean ± standard deviation over 5 training runs tab. 4. Early stopping on the latest dev set of each round.

Dev-set macro F1 app. C, tab. 6, same layout, mean ± standard deviation over 5 runs.

Dataset composition tab. 1 tab. 2:

The most frequent targets are Black people, women and Jewish people, with ten groups listed in the paper tab. 2.

HateCheck §6.3 fig. 1: M1 is about 60% accurate overall, M2 is 91% on hateful and 67% on not-hateful entries, and M4 is 95% on hateful and 93% on not-hateful entries. The paper's comparison is with four models from Röttger et al.; the best of them, Perspective's toxicity classifier, reaches 77% overall, with 90% on hateful and 48% on not-hateful entries §6.3.

Paper vs code

• The paper says the code is provided [abstract] §1. The repository at code_commit has the two CSV releases, the annotation guidelines PDF and a README. It has no training, model or evaluation code. The README's only code statement is that the code was implemented with the Hugging Face transformers library README.md. Every model setting on this card therefore comes from the paper only app. D.
• The repository is the one the paper links. Its owner, bvidgen, matches the first author, Bertie Vidgen, whom the README names as the contact README.md.
• The licence is stated in the README as CC-BY 4.0 for the dataset README.md. The repository has no LICENSE file, so code_license is recorded as not stated.
• The released CSV has two versions. v0.2.2 is the full dataset the paper used. v0.2.3 removes duplicates, all of which are in round 1 README.md.
• The CSV target column uses shorthand codes for identities, with a mapping given in the README README.md. The paper lists 29 identities plus five intersectional variants app. A tab. 5. Some README codes, such as nazi and hitler, are for the Support type, which the paper's identity list does not contain.
• The README's description of the label scheme matches the paper's. The CSV contents were not checked against the paper's tables. The annotation guidelines PDF is in the repository but was not read for this card.

Pitfalls

• No training code is released. Batch size, maximum sequence length, warm-up, weight decay, loss, optimiser name and early-stopping patience are not stated. An implementation has to choose them and report the choice.
• Per-type counts differ between text and table. Section 5 gives round-2 Derogation as 3,577 and Support as 39, while Table 1 gives 3,574 and 41. Across rounds 2 to 4, the per-type counts in the text differ from Table 1 by one to six §5.2–§5.4 tab. 1. The round totals match, so the differences are in the per-type split. Use the released CSV for counts, not the text.
• Dataset size is rounded. The abstract says about 40,000 entries and about 15,000 perturbations. The table total is 41,255. The round-2 perturbation count is not given separately; the paper says half the round §5.2.
• 18 perturbations do not flip the label. The authors found this after publication and kept them. Check each original and perturbation pair before training on them §5.
• Perturbations carry no in-loop signal. They were written offline without model feedback §5.2, and annotators were not told whether their perturbations fooled the model tab. 3 caption. Their error rate is not part of Table 3.
• Duplication is heavy. Round 2 is duplicated 100 times in M3's data. The factors are selected on the dev sets; the selection metric is not stated, and the paper does not analyse overfitting to duplicates §5 app. D.
• Naming differs across sections. The main text calls the targets M1 to M4; Appendix D calls them the round 1 to round 4 targets and describes the round-4 target as trained on the round-3 target's data app. D. The table caption also uses the round names. Check which one a reproduction is using.
• Error rate is not comparable across rounds. Later rounds contain harder content and the authors say the falling error rate underestimates how much harder the rounds got §6.1.
• Annotator effects. Twenty annotators, a small pool. Half of each test set is from unseen annotators, but the other half is from annotators seen in training §4 §5. Round-2 analysis found that held-out annotators' performance dropped substantially §5.2.
• Agreement is moderate after round 2. Krippendorff's alpha is 0.815 for round-2 originals and 0.55 and 0.52 for rounds 3 and 4 when flagged entries count as incorrect §5.2–§5.4. Round 1 alpha is not given.
• HateCheck is a weak test. It only shows the absence of some weaknesses §6.3. Its design overlaps with the round-2 pivots: after lower-casing and removing punctuation, 21 entries match HateCheck, which the authors say is 0.05% of the dataset §6.3.
• Template statements. Early round-2 entries were templates that differ only in the identity term, which made them easy for models. They were banned after round 2 §5.2. Filter for such duplicates if you rebuild the data.
• Provenance. Entries are described as synthetic, and no content was copied from online sites. Round-4 entries inspired by real content had substantial adjustments [impact statement] §5.3.

Lineage

• Builds on: dynamic adversarial data collection, with annotators trying to fool a model in rounds (Nie et al., 2020, and Dinan et al., 2019, as cited in §2) §2. Contrast sets, label-flipping edits (Kaushik et al., 2019, and Gardner et al., 2020) §1 §2. No arXiv id or card is linked in this wiki.
• Compared against: the four models of Röttger et al. on HateCheck, including the Perspective toxicity classifier §6.3. Contrast-set hate detection by Samory et al., 2020 §2. None has a card in this wiki.
• Superseded by: not known from wiki content.