---
kind: concept
slug: inter-annotator-agreement
title: Inter-annotator agreement
summary: >-
  Agreement between annotators, or between a model and experts, shows whether labels are reliable enough to train or judge on. Hate-speech studies report Krippendorff's alpha of .18 to .38, and 0.815 for round-2 originals in a dynamic collection. A language model coding against expert codebooks reaches Cohen's kappa of .61 and .38, against .88 and .90 for experts. These measures are not comparable.
tags: [annotation, inter-annotator-agreement, krippendorff-alpha, cohen-kappa, label-quality, hate-speech, dynamic-data-collection, llm-annotation]
papers: ["1701.08118", "2012.15761", "2304.10548"]
related: [noisy-labels, toxicity-detection, dataset-curation, adversarial-evaluation, llm-as-a-judge, prompt-engineering]
status: draft
updated_at: 2026-11-01
---

## Overview

Inter-annotator agreement asks whether several people who label the same items
give the same labels. Labelled data is only as reliable as this agreement
allows. The authors of the hate speech study say a classifier's quality is
bounded by how reliably people label its training data [arxiv:1701.08118 §1].

Three papers in the wiki report agreement. Two are about hate speech and both
report Krippendorff's α. The first computes it within each group of annotators,
per question, and not across groups [arxiv:1701.08118 §4] [arxiv:1701.08118 §5].
The second reports it per round of a dynamic collection, where annotators write
posts that fool the current model [arxiv:2012.15761 §5.2]. The third measures
agreement between a language model and expert coders with Cohen's κ, for
deductive coding of children's questions [arxiv:2304.10548 §4]. The first
paper's own discussion suggests that existing hate speech definitions leave
much room for personal judgement, and that wording matters [arxiv:1701.08118 §1].

## Variants

Three papers in the wiki report agreement. The first three variants below are
the annotation settings the two hate-speech papers compare. The fourth is a
language model labelling against expert codes, which the third paper studies.
They differ in who labels, what they are told, and whether labelling happens
inside a loop where annotators try to beat a model.

### Expert pre-study on a shared corpus

Six researchers who knew the definitions labelled 541 German tweets. Each tweet
was labelled by two of them, with the pairs rotated so that every pair shared
one part of the corpus. Each annotator answered whether the tweet was hate
speech (yes or no) and rated its offensiveness on a 6-point scale
[arxiv:1701.08118 §3]. Agreement on this set is α = .38 [arxiv:1701.08118 §3].
The sizes of the parts and the exact pairing are not stated, which limits how
closely the design can be copied.

Card: [Hate speech annotation reliability study](/papers/1701.08118).

### Lay online survey, with or without a shown definition

Two between-subjects surveys were run with 56 participants in total. 25 saw
Twitter's hateful-conduct definition at the start and above each tweet, and
31 saw no definition [arxiv:1701.08118 §4] [arxiv:1701.08118 tab. 1]. Each
participant rated 20 tweets in random order. The definition group answered
"hate speech" according to the definition, and the other group according to
their own opinion [arxiv:1701.08118 §4].

Agreement across the groups and questions is α = .18 to .29
[arxiv:1701.08118 §5]. The definition group more often said that a tweet
should be banned (32.6% against 17.6%, p = .01). The shares that called a
tweet hate speech did not differ significantly (32.6% against 40.3%, p = .26),
and neither did offensiveness (mean 3.49 against 3.42) [arxiv:1701.08118 tab. 1].
Across tweets, the definition group's hate-speech shares correlate with the
opinion group's shares at Pearson r = .895 [arxiv:1701.08118 §5]. The authors
conclude that agreement did not consistently rise when a definition was shown
[arxiv:1701.08118 §5].

Things to check before reusing this design:

- The paper does not say which 20 tweets each survey showed, or whether every
  participant saw the same ones [arxiv:1701.08118 §4].
- It does not state the level of measurement used for α on the yes/no
  questions versus the 6-point scale [arxiv:1701.08118 §4] [arxiv:1701.08118 §5].
- Three group comparisons are reported without a correction for multiple
  tests [arxiv:1701.08118 tab. 1].
- The two groups answered the same question in different ways, so the
  correlation compares a definition-based answer with an opinion-based one
  [arxiv:1701.08118 §4] [arxiv:1701.08118 §5].

Card: [Hate speech annotation reliability study](/papers/1701.08118).

### Dynamic rounds with validator checks

Annotators write posts that fool the current target model, in four rounds.
Round 1 checks each entry with one other annotator. From round 2, originals
that fooled the previous target are checked by three to five other annotators,
and perturbations by one. Each entry is marked correct, incorrect or flagged,
and incorrect or flagged entries go to an expert [arxiv:2012.15761 §5.1]
[arxiv:2012.15761 §5.2].

Krippendorff's α is 0.815 for round-2 originals [arxiv:2012.15761 §5.2]. For
rounds 3 and 4 it is 0.55 and 0.52 when flagged entries count as incorrect
[arxiv:2012.15761 §5.3] [arxiv:2012.15761 §5.4]. The paper gives no α for
round 1 [arxiv:2012.15761 §5]. The annotator pool is small, about twenty
people [arxiv:2012.15761 §4] [arxiv:2012.15761 §5], and the setting assumes
expert oversight [arxiv:2012.15761 §7].

Card: [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](/papers/2012.15761).

### Language model coder against expert codes

GPT-3 (davinci-text-002, temperature 0.0) labels 668 French children's questions
with the expert codebook in the prompt, and no training [arxiv:2304.10548 §2.1]
[arxiv:2304.10548 §3]. Two dimensions are coded: question complexity, which is
binary, and syntactic structure, with four categories [arxiv:2304.10548 §2.1].
Complexity was coded on GPT-3's English translations, not on the original
questions (footnote 1) [arxiv:2304.10548 §2.1].

Agreement with the experts' final codes is Cohen's κ = 0.61 for complexity and
0.38 for syntax [arxiv:2304.10548 §4]. The two experts who coded with the same
codebook reached κ = 0.88 and 0.90 on the two dimensions [arxiv:2304.10548 §4].

Prompt design matters in this study. Codebook-centred prompts, which list each
code with its description and examples, beat example-centred prompts
[arxiv:2304.10548 §4] [arxiv:2304.10548 fig. 1]. The largest gain came from
moving from zero to one example per code. The paper says one and five examples
"did not differ much" [arxiv:2304.10548 §4]. The per-variant κ values appear
only in the figure [arxiv:2304.10548 fig. 1].

Things to check before reusing this design:

- The dataset, the codebook text and the prompts are not published, so the
  design cannot be copied from the paper [arxiv:2304.10548 §2.1].
- Only the experts' final codes serve as the reference. How many coders
  labelled each item, and how the final codes were reached, is not stated
  [arxiv:2304.10548 §2.1] [arxiv:2304.10548 §4].
- Temperature 0.0 is called reproducible, but no repeat runs are reported
  [arxiv:2304.10548 §3].
- The model is a closed API tied to a time period, and the paper pins only the
  model name [arxiv:2304.10548 §3].
- Question placement, maximum output length and the choice of few-shot
  examples are not stated [arxiv:2304.10548 §3].

Card: [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](/papers/2304.10548).

## When to use which

The three papers compare these settings and do not test other agreement
measures or annotation protocols. The table is therefore limited to what their
results support.

| Situation | Use | Why |
|---|---|---|
| Expert annotators label a shared corpus, and you need an agreement figure for that set | Expert pre-study design | Reported α is .38 on this design [arxiv:1701.08118 §3] |
| Lay annotators, and you want to know whether a written definition improves agreement | Survey with and without the definition | The paper finds no consistent gain in agreement [arxiv:1701.08118 §5] |
| You want to change how often annotators vote to ban content | Show the definition, and measure the ban rate | In this study the ban share rose, while agreement did not [arxiv:1701.08118 tab. 1] [arxiv:1701.08118 §5] |
| Annotators write adversarial examples against a model in rounds, and you need a label check for each round | Validator pass, with α reported per round | Reported α is 0.815 for round-2 originals and 0.52 to 0.55 for rounds 3 and 4, so a single figure would not describe every round [arxiv:2012.15761 §5.2] [arxiv:2012.15761 §5.4] |
| You have a written codebook, no labelled training data, and many items to code | Codebook-centred prompt, with zero or one example per code | The study used the codebook alone in the prompt [arxiv:2304.10548 §3]. Codebook-centred prompts beat example-centred ones, and one example per code gave most of the gain [arxiv:2304.10548 §4] |
| You want to know whether a model can replace one human coder | Model-to-expert agreement per dimension, against expert-to-expert agreement on the same codebook | On this evidence the model does not match the experts: κ 0.61 against 0.88 for complexity and 0.38 against 0.90 for syntax [arxiv:2304.10548 §4] |
| Labels mix a semantic dimension and a structural one | Report agreement per dimension | Complexity reached substantial κ (0.61) and syntax only fair κ (0.38) [arxiv:2304.10548 §4] |
| You need a threshold for "reliable enough" | State the threshold and its source before reporting α | The authors cite .80 for content analysis and .66 for uncertain applications, from Krippendorff (2004). Their survey α values all fall below .66 [arxiv:1701.08118 §5] |

Whatever the setting, name the agreement statistic: Krippendorff's α, which
needs a stated level of measurement for each question, or Cohen's κ. Report
agreement per question, and per group where the design has groups. The papers
leave some of these details unstated, and the reported values cannot be
reproduced without them [arxiv:1701.08118 §4] [arxiv:1701.08118 §5]
[arxiv:2304.10548 §4].

## Implementation sketch

The first paper gives no algorithm block. The steps below restate its analysis,
with the level of measurement left as an explicit input, since the paper does
not state which one it used.

```pseudocode
# Agreement per group and per question  [arxiv:1701.08118 §5]
# Input:  labels[group][annotator][item][question]; a level per question
# Output: alpha[group][question]

def agreement_report(labels, levels):
    alpha = {}
    for group in labels:
        # Compute within each group only, not across groups  [§5]
        for question in ["hate", "ban", "offensiveness"]:
            units = labels[group].items_by_question(question)
            # Level must be chosen and reported; not stated in the paper  [§4]
            alpha[group][question] = krippendorff_alpha(
                units, level=levels[question])
    return alpha
```

To compare a model with expert codes, the third paper reports Cohen's κ for
each dimension separately [arxiv:2304.10548 §4]:

```pseudocode
# Model vs expert agreement, one dimension at a time  [arxiv:2304.10548 §4]
# Input:  model_codes[item], expert_codes[item] for one dimension
# Output: kappa for that dimension

def model_expert_kappa(model_codes, expert_codes):
    # Final expert codes are the reference; coder count not stated  [§2.1]
    # Compare with expert-to-expert kappa on the same items, if available
    return cohen_kappa(model_codes, expert_codes)
```

## Contradictions

The three papers give very different magnitudes, and none of them explains the
gaps. The first reports α = .38 for six expert raters on German tweets, who
gave a yes or no hate label and a 6-point offensiveness rating
[arxiv:1701.08118 §3]. It reports .18 to .29 for lay survey raters
[arxiv:1701.08118 §5]. The second reports 0.815 for round-2 originals and 0.52
to 0.55 for rounds 3 and 4 [arxiv:2012.15761 §5.2] [arxiv:2012.15761 §5.4].
The third reports expert-to-expert κ of 0.88 and 0.90 on children's questions,
which is well above the expert α of .38 in the first paper
[arxiv:2304.10548 §4] [arxiv:1701.08118 §3].

The first paper's expert raters were researchers who knew the definitions, and
its lay raters were survey participants. The second paper's raters were
annotators under expert oversight. The third paper's reference is the experts'
final codes, and its labeller is a model. The items, languages (German, English
and French), label schemes, statistics (α against κ), numbers of raters and
settings all differ. The numbers are not comparable, and this page does not
resolve the differences.

## Papers

- [Hate speech annotation reliability study](/papers/1701.08118): shows a hate speech definition to lay annotators does not raise their agreement, while it changes their ban votes.
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](/papers/2012.15761): reports α per round for adversarially written hate-speech posts, with validator checks on each entry.
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](/papers/2304.10548): reports Cohen's κ between GPT-3 and expert codes for two dimensions of children's questions, below expert-to-expert agreement.