Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis

In plain words

Teaching a computer to spot hate speech needs many messages that people have labelled. The labels are only useful if different people give the same label to the same message. This paper checks whether they do. The authors collected tweets about the 2016 refugee debate in Germany and asked internet users whether each one was hate speech, whether it should be banned, and how offensive it was.

One group was first shown a definition of hate speech, and the other was not. The definition made people's answers line up more with the definition, and the definition group more often said a tweet should be banned. It did not make people agree with each other more. Agreement stayed low in both groups. The authors conclude that hate speech is not a simple yes-or-no matter, and that annotators need more detailed instructions.

TL;DR

• Two between-subjects online surveys, 56 participants in total, each shown 20 German tweets §4 tab. 1.
• One group saw Twitter's "hateful conduct" definition; the other did not §4.
• Agreement is very low in both groups: Krippendorff's α between .18 and .29, below the .66 the authors cite as the minimum for uncertain applications §5 fig. 1.
• The definition group more often said a tweet should be banned (32.6% vs 17.6%, p = .01) but did not agree more with each other §5 tab. 1.
• Results are author-reported and rest on a small sample; the analysis code is not released.

Problem

A hate speech classifier needs a large set of texts that human annotators have labelled as hate speech or not. Its quality is bounded by how reliably people label the data §1. The authors found that six researchers, who knew the definitions, agreed only weakly (α = .38) on 541 tweets §3. Group discussions suggested that existing definitions leave much room for personal judgement and that wording matters §1.

The paper asks two questions: can hate speech be annotated reliably, and does showing a definition improve reliability or agreement with the definition §1 §4.

Method

Corpus §3.

1. Collect tweets that contain one of 10 hashtags that are used offensively, listed in the paper's footnote. The collection ran from about February to March 2016 and returned 13,766 tweets §3.
2. Drop tweets that consist only of links or images, and keep only original tweets (no retweets or replies) §3.
3. Remove near-duplicates. The paper says a tweet is discarded when its normalised Levenshtein distance to an earlier tweet is smaller than .85 §3.
4. Drop one search term, #Pack, because it mostly found hate speech unrelated to the refugee crisis §3.
5. Read the remaining tweets by hand and remove those that are hard to understand §3.
6. The result is 541 tweets §3.

Pre-study §3. Six annotators split the 541 tweets into six parts. Each part was labelled by two of the six, and the pairs were rotated so that each pair shared one part. Each annotator marked hate speech yes or no and rated offensiveness on a 6-point scale §3. The authors report α = .38 for this group §3.

Surveys §4.

1. Build two online surveys with a between-subjects design. Participants are randomly assigned to one of them §4.
2. Group 1 (definition): Twitter's hateful-conduct definition is shown at the start and again above every tweet §4.
3. Group 2 (no definition): no definition is shown §4.
4. Each survey presents 20 tweets in random order §4.
5. For each tweet, participants answer three questions §4:
   • Q1: is this hate speech? Group 1 answers according to the definition, group 2 according to personal opinion (yes/no) §4.
   • Q2: should this tweet be banned from Twitter? (yes/no) §4.
   • Q3: how offensive is it, on a 6-point scale from "not offensive at all" to "very offensive" §4?
   • If Q3 is 4 or higher, the participant may name the words they found offensive §4.
6. After the tweets, group 1 is asked whether Twitter's definition is sufficient. Group 2 is asked to propose a definition. Both groups answer sociodemographic questions §4.
7. The surveys were approved by the ethics committee of the authors' department §4.

The paper has no algorithm block. The survey protocol, as described above, is restated here for implementers §4:

# Between-subjects survey protocol  [§4]
# Input:  tweets (20 per survey, see Pitfalls); participants
# Output: answers[participant][tweet] = (hate, ban, offensiveness)

def run_surveys(tweets, participants):
    # Random assignment to one of two groups  [§4]
    for participant in participants:
        participant.group = random_choice(["definition", "no_definition"])
    for participant in participants:
        # Tweets in random order; definition shown above each tweet  [§4]
        for tweet in shuffled(tweets):
            if participant.group == "definition":
                show(TWITTER_HATEFUL_CONDUCT_DEFINITION)
            hate = ask_yes_no("Is this hate speech?")          # Q1  [§4]
            ban = ask_yes_no("Should it be banned?")           # Q2  [§4]
            offensiveness = ask_likert(1, 6)                   # Q3  [§4]
            if offensiveness >= 4:
                # Optional free-text marking of offensive words  [§4]
                words = ask_optional_text("Which words?")
            answers[participant][tweet] = (hate, ban, offensiveness)
    return answers

Analysis §5.

1. For each participant, compute the share of tweets marked hate speech, the share marked for banning, and the mean offensiveness rating §5.
2. Compare the two groups on each of the three measures with the Wilcoxon-Mann-Whitney test. The authors chose it because a Shapiro-Wilk check showed that some data were not normally distributed §5 tab. 1.
3. Compute Krippendorff's α per group and per question §5 fig. 1.
4. Compute Pearson's correlation between two per-tweet series: the share of group 1 that marks each tweet as hate speech (Q1, with definition), and the share of group 2 that does (Q1, opinion) §5.

Hyperparameters

Results

Author-reported. Setup: 56 participants, two groups, 20 tweets each, 1,120 annotations in total §5. The corpus is German tweets about refugees §3. Participants' language and recruitment are not stated.

Group comparison tab. 1. "Hate speech" and "ban" are the share of tweets marked yes, per participant. "r" is the effect size from the test, as labelled in the table.

Age and gender were not reported by every participant tab. 1.

Consistency within the definition group §5. Group 1 gave different answers to Q1 and Q2 in 18 of 500 instances (3.6%). The authors read this as evidence that the group aligned its own view with the definition §5.

Agreement §5 fig. 1. The text gives a range of α = .18 to .29 across the groups and questions. The bar heights below are read by eye from the figure, so they are approximate. The question numbers follow the order in §4.

The authors conclude that reliability did not consistently rise when a definition was shown §5.

Agreement with the definition §5. Across tweets, the share of group 1 that marks a tweet as hate speech correlates strongly with the share of group 2 that does (Pearson r = .895, p < .0001) §5.

Paper vs code

No official code for the method or the analysis. The paper's footnote links a repository of the authors' organisation (UCSM-DUE/IWG_hatespeech_public, commit e14501bcada5f2d2c5dea7a6d4bca3a873ac4a81) §3. It contains a README and one CSV file of tweets with expert labels, but no survey code and no analysis scripts. The README names a different venue (NLP4CMC III, Bochumer Linguistische Arbeitsberichte) than the LaTeX source's conference style (KONVENS 2016) README.md [src: konvens2016.tex#L16].

Differences between paper and repository:

• The CSV has a header and 469 data rows, with two expert annotators per tweet and an offensiveness rating from the second expert german hatespeech refugees.csv. The paper gives 541 tweets and six annotators §3. The number 469 appears in the introduction, where six annotators rated 469 tweets §1. The repository therefore matches the preliminary count, not the final 541-tweet corpus. The paper does not say this.
• The repository does not contain the survey responses from the 56 participants, so the Results tables above cannot be reproduced from it §4 §5.
• The README declares CC BY-SA 3.0 for the repository and has no LICENSE file README.md. The paper itself is CC BY 4.0. This card does not use the repository's contents.

Pitfalls

• Which 20 tweets? Each survey shows 20 tweets §4, but the paper does not say which tweets these are, whether they come from the 541-tweet corpus, or whether all participants saw the same 20 §4. The "random order" statement implies that they did.
• Rater counts per tweet. Agreement is computed within each group, so each tweet has about 25 raters (definition group) or about 31 (no-definition group). The paper does not state this directly §4 §5.
• Level of measurement for α. The paper says Krippendorff's α applies to nominal and interval data §4, but it does not say which level it used for the yes/no questions and which for the 6-point offensiveness scale §5. Reproduce the three questions separately and state the level you chose.
• Normalised Levenshtein threshold. The paper says a tweet is discarded when its normalised edit distance to an earlier tweet is smaller than .85 §3. The normalising base is not stated. Read literally, a distance threshold of .85 removes most pairs of tweets, so the value may refer to a similarity. Check the original code before reusing this filter.
• Identical figures in Table 1. The definition group has 32.6% for both "hate speech" and "ban" tab. 1. The paper does not comment on this. Verify it against the raw responses before quoting either number.
• Multiple comparisons. Three group tests are reported with no correction. The ban result (p = .01) would be weaker under a correction; the paper does not discuss this tab. 1.
• Small samples. Each test compares 25 and 31 participants with 20 items each tab. 1 §4. The authors do not report confidence intervals.
• Pre-study versus survey. The pre-study agreement (α = .38) comes from researchers on 541 tweets §3. The survey agreement (α = .18 to .29) comes from lay participants on 20 tweets each §5. The two are not directly comparable, and the paper does not claim they are.
• Survey wording. Group 1 is asked to answer based on the definition and group 2 based on opinion for the same question §4. The two groups therefore answered different questions, and the correlation (r = .895) compares a definition-based answer with an opinion-based one §5.
• Language and recruitment. The tweets are German. The paper does not state the participants' language, recruitment method or compensation §4.
• Dataset licence. The linked repository declares CC BY-SA 3.0 and has no LICENSE file README.md. Do not copy its tweets into a page without checking that licence and that the tweets may be reused.

Lineage

• Builds on: Krippendorff's α as the agreement measure §4 §5, cited from Krippendorff (2004). The paper's hate speech definitions cite two earlier works by bibliography key only (warner2012, waseem2016hateful) §2. Their full references are in the paper's bibliography, which is not part of the source archive used here.
• Compared against: Twitter's hateful-conduct policy as of June 2016, which the paper uses as its definition §2 §4.
• Superseded by: not known from this wiki. The authors planned a new coding scheme with clearer criteria §6.