ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks

In plain words

Social scientists and NLP teams often need people to read texts and tag them, for example "is this tweet about content moderation?" Hiring trained research assistants is accurate but expensive, and paid crowd-workers are cheap but often inconsistent. This paper asks whether ChatGPT can do the tagging instead.

The authors give ChatGPT the same written instructions they gave their human coders, with no examples, and ask it to label one tweet or article at a time. They then compare its labels with a set of labels made by trained coders and with labels from crowd-workers. ChatGPT is more accurate than the crowd-workers for most tasks, agrees with itself more often than people agree with each other, and costs a small fraction of a cent per label. The authors present this as evidence that language models could make large annotation jobs much cheaper, while noting that the test set and task choice limit how far the result goes.

TL;DR

• Zero-shot ChatGPT (gpt-3.5-turbo, the same codebooks as human coders) was compared with MTurk crowd-workers and trained research assistants on five annotation tasks over four samples §4.1 §4.2.
• ChatGPT's accuracy against trained annotators was about 25 percentage points above MTurk on average across the four datasets §2.
• Intercoder agreement was about 91% at the default temperature and 97% at temperature 0.2, against 56% for MTurk and 79% for trained annotators §2.
• Cost is stated as below $0.003 per annotation, which the authors put at about thirty times cheaper than MTurk §1.
• Evidence is author-reported on four samples of US political and content-moderation texts; the language mix of the three non-2023 samples is not stated, and no code is published (see Paper vs code).

Problem

Text annotation (labelling documents with categories such as relevance, stance, topic or frame) is needed to train classifiers and to evaluate unsupervised models §1. Researchers have two usual options: trained annotators, who give high quality at high cost, or crowd-workers on platforms such as MTurk, who are cheaper but may be inconsistent, particularly for complex tasks and for languages other than English §1. The paper asks whether a zero-shot large language model can match or beat crowd-workers at a lower cost §1.

Method

The study re-runs an existing annotation project. Texts and codebooks come from a previous study with trained annotators §1 §2. The same codebook text is used for trained annotators, MTurk workers and ChatGPT, so the three groups receive identical instructions §4.2.

1. Samples. Four datasets: a random sample of 2,382 content-moderation tweets (January 2020 to April 2021); 1,856 tweets by US Congress members (2017 to 2022); 1,606 news articles on content moderation (January 2020 to April 2021); and 500 tweets from January 2023, of which 339 were in English §4.1.
2. Five tasks. Relevance to content moderation (relevant/irrelevant); relevance to political content (relevant/irrelevant); stance on Section 230 (keep/repeal/neutral); topic (six classes: Section 230, Trump Ban, Complaint, Platform Policies, Twitter Support, Other); and frames, with a problem/solution frame task and a policy frame task that uses fourteen topics §4.2 app. S1.3–app. S1.9.
3. Gold standard. Two trained research assistants annotate each text independently §4.3.
4. Crowd baseline. Each text is annotated by two MTurk workers, restricted to "MTurk Masters", HIT approval above 90%, at least 50 approved HITs and US location; no worker may annotate more than 20% of a task's tweets §4.4.
5. ChatGPT. Each text is sent to the ChatGPT API in a new chat session. The task instruction is embedded in the prompt "Here's the tweet I picked, please label it as [task instruction]." §4.5. Each text is labelled at temperature 1 (the default) and at temperature 0.2, with two responses per setting, so four responses per text §4.5.
6. Scoring. Accuracy is measured against the trained annotators' label, using only texts on which both trained annotators agreed. Intercoder agreement is the share of texts on which both annotators in a group give the same label §4.6.

# ChatGPT annotation loop  [§4.5]
# Input:  texts (list of str); instruction (codebook text for one task)
# Output: labels[text][temperature] = [label_1, label_2]

def annotate_with_chatgpt(texts, instruction):
    # Fresh chat per text, so earlier answers do not sway later ones  [§4.5]
    labels = {}
    for text in texts:
        labels[text] = {}
        # Default temperature 1, plus the less random 0.2  [§4.5]
        for temperature in [1.0, 0.2]:
            # Two responses per setting, for intercoder agreement  [§4.5]
            labels[text][temperature] = []
            for run in range(2):
                reply = chat_once(
                    temperature=temperature,
                    prompt=f"Here's the tweet I picked, please label it as "
                           f"{instruction}. {text}",
                )
                labels[text][temperature].append(reply)
    return labels

# Scoring  [§4.6]
# Input:  predicted (dict text -> label); gold (dict text -> (label_a, label_b))
# Output: accuracy (float), agreement (float)

def accuracy(predicted, gold):
    # Keep only texts where the two trained annotators agreed  [§4.6]
    kept = [t for t in gold if gold[t][0] == gold[t][1]]
    correct = sum(predicted[t] == gold[t][0] for t in kept)
    return correct / len(kept)

def intercoder_agreement(labels_a, labels_b):
    # Share of texts where both raters in a group give the same class  [§4.6]
    same = sum(labels_a[t] == labels_b[t] for t in labels_a)
    return same / len(labels_a)

Note on the scoring code: the paper describes both metrics in words; the pseudocode above is our reading of that description, not code from the authors §4.6.

Hyperparameters

Results

Author-reported. Setup: four samples (n as in Hyperparameters), zero-shot ChatGPT (gpt-3.5-turbo), baselines are MTurk crowd-workers and trained research assistants §2 §4.1.

Per-task accuracy for the other tasks and the MTurk and trained-annotator accuracies are shown only in fig. 1; they are not given numerically in the text.

Paper vs code

No official code.

Search performed: (1) links in the paper: none to code; the only external resource is the replication package on Harvard Dataverse (doi 10.7910/DVN/PQYF6M) §4.7. (2) Project page: the authors' site (fabriziogilardi.org) lists no code for this paper on its homepage; its publications page was not read. (3) GitHub repository search via the API: chatgpt outperforms crowd-workers text-annotation and gilardi chatgpt both returned zero repositories. The Dataverse page could not be read in this session (empty response), so it is unverified whether the replication package contains scripts; if it does, those scripts are not covered by this card and their license is unknown. Web search was not available for further queries.

Pitfalls

• Sample total does not match the stated n. The text states n = 6,183 §2 §4.1, but the four sample sizes sum to 6,344 (2,382 + 1,856 + 1,606 + 500) §4.1. The 2023 sample has 500 tweets of which 339 were English §4.1; which subset enters the accuracy and agreement figures is not stated.
• Two versions of the abstract disagree. The arXiv metadata abstract (version 2 listing) reports 2,382 tweets, "about twenty times cheaper", and accuracy above crowd-workers "for four out of five tasks". The LaTeX source used for this card reports four samples with n = 6,183, "about thirty times cheaper", and "about 25 percentage points" on average §2. The body text of this card follows the LaTeX source.
• Frame-task class count differs. The Methods describe the general frame task as two opposing frames, problem and solution §4.2. The Results and the codebook use three classes, adding neutral §2 app. S1.5.
• Policy frames: fourteen topics or fifteen labels. The text says fourteen policy frames §4.2. The codebook lists fourteen topics plus OTHER app. S1.6 app. S1.7.
• Task-to-dataset mapping is not stated in the text. The appendix says stance, problem/solution and topic were unsuitable for Congress tweets, and that human annotation for stance, topic and policy frames on news articles was unsuccessful app. S1. The Results list the tasks without mapping them to datasets §2. The mapping can only be read from fig. 1.
• Accuracy uses only agreed gold labels. Texts on which the two trained annotators disagreed are dropped from accuracy §4.6. How many were dropped is not stated, so the accuracy numbers are on the subset where humans agreed.
• Agreement is raw percent agreement. Intercoder agreement is the share of matching labels §4.6. No chance correction is mentioned, so high agreement on skewed classes can look better than it is.
• Correlations are reported without a sample size. The Pearson values in §2 do not state the number of observations or whether they are per task or per dataset.
• Temperature choice is not separated from the prompt. Temperature 0.2 gives the highest agreement §2, and the authors suggest it "may be preferable" §2. Top_p and other sampling settings were not reported §4.5, so a re-run may not reproduce the numbers.
• Model snapshot. Only gpt-3.5-turbo and the annotation window are stated §4.5. Hosted models change, so later API calls may give different labels.
• Memorisation check is weak evidence. The 2023 sample is included to test whether ChatGPT relies on memorised training text §1 §2. The authors say this does not suggest memorisation is a major issue §2, but ChatGPT struggled to classify tweets about specific user suspensions in the relevance task, which the authors attribute to a lack of examples in the prompt §2.
• Cost figures are not derived in the text. The per-annotation cost is stated as about $0.003 §1, but the token counts, prices and the MTurk per-annotation payment behind "thirty times cheaper" are not stated.
• Figure-only numbers. Per-task accuracies, MTurk and trained-annotator accuracies are only in fig. 1 (a plot). They are not extracted here and must not be read off the figure without checking the source.
• Section numbers are derived. The source uses the article class without a numbered PDF; the § numbers above count sections in order (Introduction = §1, Results = §2, Discussion = §3, Materials and Methods = §4 with §4.1–§4.6; appendix sections = S1 onwards) and may differ from the published version.
• Generalisation limits. The authors note that tasks were hard, with many classes and low trained-annotator agreement on some §2; they also note that performance in other languages and few-shot prompting were not tested §3.

Lineage

• Builds on: the trained-annotator codebooks and tweet samples of Alizadeh et al. (2022) §1 §4.1; not carded.
• Related ChatGPT annotation studies named as preliminary: Kuzman et al. (2023) and Huang et al. (2023) §1; not carded.
• Compared against: MTurk crowd-workers and trained research assistants §4.3 §4.4.
• Superseded by: none known from the wiki.