Inter-annotator agreement

Overview

Inter-annotator agreement asks whether several people who label the same items give the same labels. Labelled data is only as reliable as this agreement allows. The authors of the hate speech study say a classifier's quality is bounded by how reliably people label its training data 1701.08118 §1.

Three papers in the wiki report agreement. Two are about hate speech and both report Krippendorff's α. The first computes it within each group of annotators, per question, and not across groups 1701.08118 §4 1701.08118 §5. The second reports it per round of a dynamic collection, where annotators write posts that fool the current model 2012.15761 §5.2. The third measures agreement between a language model and expert coders with Cohen's κ, for deductive coding of children's questions 2304.10548 §4. The first paper's own discussion suggests that existing hate speech definitions leave much room for personal judgement, and that wording matters 1701.08118 §1.

Variants

Three papers in the wiki report agreement. The first three variants below are the annotation settings the two hate-speech papers compare. The fourth is a language model labelling against expert codes, which the third paper studies. They differ in who labels, what they are told, and whether labelling happens inside a loop where annotators try to beat a model.

Expert pre-study on a shared corpus

Six researchers who knew the definitions labelled 541 German tweets. Each tweet was labelled by two of them, with the pairs rotated so that every pair shared one part of the corpus. Each annotator answered whether the tweet was hate speech (yes or no) and rated its offensiveness on a 6-point scale 1701.08118 §3. Agreement on this set is α = .38 1701.08118 §3. The sizes of the parts and the exact pairing are not stated, which limits how closely the design can be copied.

Card: Hate speech annotation reliability study.

Lay online survey, with or without a shown definition

Two between-subjects surveys were run with 56 participants in total. 25 saw Twitter's hateful-conduct definition at the start and above each tweet, and 31 saw no definition 1701.08118 §4 1701.08118 tab. 1. Each participant rated 20 tweets in random order. The definition group answered "hate speech" according to the definition, and the other group according to their own opinion 1701.08118 §4.

Agreement across the groups and questions is α = .18 to .29 1701.08118 §5. The definition group more often said that a tweet should be banned (32.6% against 17.6%, p = .01). The shares that called a tweet hate speech did not differ significantly (32.6% against 40.3%, p = .26), and neither did offensiveness (mean 3.49 against 3.42) 1701.08118 tab. 1. Across tweets, the definition group's hate-speech shares correlate with the opinion group's shares at Pearson r = .895 1701.08118 §5. The authors conclude that agreement did not consistently rise when a definition was shown 1701.08118 §5.

Things to check before reusing this design:

• The paper does not say which 20 tweets each survey showed, or whether every participant saw the same ones 1701.08118 §4.
• It does not state the level of measurement used for α on the yes/no questions versus the 6-point scale 1701.08118 §4 1701.08118 §5.
• Three group comparisons are reported without a correction for multiple tests 1701.08118 tab. 1.
• The two groups answered the same question in different ways, so the correlation compares a definition-based answer with an opinion-based one 1701.08118 §4 1701.08118 §5.

Card: Hate speech annotation reliability study.

Dynamic rounds with validator checks

Annotators write posts that fool the current target model, in four rounds. Round 1 checks each entry with one other annotator. From round 2, originals that fooled the previous target are checked by three to five other annotators, and perturbations by one. Each entry is marked correct, incorrect or flagged, and incorrect or flagged entries go to an expert 2012.15761 §5.1 2012.15761 §5.2.

Krippendorff's α is 0.815 for round-2 originals 2012.15761 §5.2. For rounds 3 and 4 it is 0.55 and 0.52 when flagged entries count as incorrect 2012.15761 §5.3 2012.15761 §5.4. The paper gives no α for round 1 2012.15761 §5. The annotator pool is small, about twenty people 2012.15761 §4 2012.15761 §5, and the setting assumes expert oversight 2012.15761 §7.

Card: Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection.

Language model coder against expert codes

GPT-3 (davinci-text-002, temperature 0.0) labels 668 French children's questions with the expert codebook in the prompt, and no training 2304.10548 §2.1 2304.10548 §3. Two dimensions are coded: question complexity, which is binary, and syntactic structure, with four categories 2304.10548 §2.1. Complexity was coded on GPT-3's English translations, not on the original questions (footnote 1) 2304.10548 §2.1.

Agreement with the experts' final codes is Cohen's κ = 0.61 for complexity and 0.38 for syntax 2304.10548 §4. The two experts who coded with the same codebook reached κ = 0.88 and 0.90 on the two dimensions 2304.10548 §4.

Prompt design matters in this study. Codebook-centred prompts, which list each code with its description and examples, beat example-centred prompts 2304.10548 §4 2304.10548 fig. 1. The largest gain came from moving from zero to one example per code. The paper says one and five examples "did not differ much" 2304.10548 §4. The per-variant κ values appear only in the figure 2304.10548 fig. 1.

Things to check before reusing this design:

• The dataset, the codebook text and the prompts are not published, so the design cannot be copied from the paper 2304.10548 §2.1.
• Only the experts' final codes serve as the reference. How many coders labelled each item, and how the final codes were reached, is not stated 2304.10548 §2.1 2304.10548 §4.
• Temperature 0.0 is called reproducible, but no repeat runs are reported 2304.10548 §3.
• The model is a closed API tied to a time period, and the paper pins only the model name 2304.10548 §3.
• Question placement, maximum output length and the choice of few-shot examples are not stated 2304.10548 §3.

Card: Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding.

When to use which

The three papers compare these settings and do not test other agreement measures or annotation protocols. The table is therefore limited to what their results support.

Whatever the setting, name the agreement statistic: Krippendorff's α, which needs a stated level of measurement for each question, or Cohen's κ. Report agreement per question, and per group where the design has groups. The papers leave some of these details unstated, and the reported values cannot be reproduced without them 1701.08118 §4 1701.08118 §5 2304.10548 §4.

Implementation sketch

The first paper gives no algorithm block. The steps below restate its analysis, with the level of measurement left as an explicit input, since the paper does not state which one it used.

# Agreement per group and per question  [arxiv:1701.08118 §5]
# Input:  labels[group][annotator][item][question]; a level per question
# Output: alpha[group][question]

def agreement_report(labels, levels):
    alpha = {}
    for group in labels:
        # Compute within each group only, not across groups  [§5]
        for question in ["hate", "ban", "offensiveness"]:
            units = labels[group].items_by_question(question)
            # Level must be chosen and reported; not stated in the paper  [§4]
            alpha[group][question] = krippendorff_alpha(
                units, level=levels[question])
    return alpha

To compare a model with expert codes, the third paper reports Cohen's κ for each dimension separately 2304.10548 §4:

# Model vs expert agreement, one dimension at a time  [arxiv:2304.10548 §4]
# Input:  model_codes[item], expert_codes[item] for one dimension
# Output: kappa for that dimension

def model_expert_kappa(model_codes, expert_codes):
    # Final expert codes are the reference; coder count not stated  [§2.1]
    # Compare with expert-to-expert kappa on the same items, if available
    return cohen_kappa(model_codes, expert_codes)

Contradictions

The three papers give very different magnitudes, and none of them explains the gaps. The first reports α = .38 for six expert raters on German tweets, who gave a yes or no hate label and a 6-point offensiveness rating 1701.08118 §3. It reports .18 to .29 for lay survey raters 1701.08118 §5. The second reports 0.815 for round-2 originals and 0.52 to 0.55 for rounds 3 and 4 2012.15761 §5.2 2012.15761 §5.4. The third reports expert-to-expert κ of 0.88 and 0.90 on children's questions, which is well above the expert α of .38 in the first paper 2304.10548 §4 1701.08118 §3.

The first paper's expert raters were researchers who knew the definitions, and its lay raters were survey participants. The second paper's raters were annotators under expert oversight. The third paper's reference is the experts' final codes, and its labeller is a model. The items, languages (German, English and French), label schemes, statistics (α against κ), numbers of raters and settings all differ. The numbers are not comparable, and this page does not resolve the differences.

Papers

• Hate speech annotation reliability study: shows a hate speech definition to lay annotators does not raise their agreement, while it changes their ban votes.
• Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection: reports α per round for adversarially written hate-speech posts, with validator checks on each entry.
• Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding: reports Cohen's κ between GPT-3 and expert codes for two dimensions of children's questions, below expert-to-expert agreement.