ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning

In plain words

Social scientists often want to label large piles of text, such as posts, by hand. Trained people are slow and costly, and crowd workers on online labour platforms are cheap but inconsistent. This paper asks whether a large language model can make the same kind of label with only a plain-language instruction and no training examples.

The test case is a set of tweets from US senators written in the two months before the 2020 election. Each tweet is labelled with the party of the person who wrote it. The model gets one instruction, to guess Democrat or Republican, and is asked the same question many times. Its answers are then compared with two groups of human labellers and with the true parties.

The main finding is that the model is more accurate and more consistent than the human labellers, and leans toward Democrat about as much as the experts do. The author stresses that the results depend on careful checking of each task and that the model's reasons may not match what happens inside it.

TL;DR

A zero-shot ChatGPT-4 prompt labels US senators' 2020 election tweets by party. Across 500 tweets, the model beats individual human coders and the crowd plurality on accuracy, and it is more consistent (higher Krippendorff's alpha) than the human coders §Results fig. 1 fig. 2. All coders lean toward Democrat, and the model's lean is similar to the experts' §Results fig. 3. Author-reported, single model, single dataset.

Problem

Interpretive text labelling, where the right label needs context about politics or the author's intent, has been done by expert coders or crowd workers. Both are slow, costly and biased, and they limit studies to small samples [intro]. Earlier LLM studies lacked ground truth or compared only with crowd workers, which the author regards as unreliable [intro]. The paper therefore asks how accuracy, reliability and bias of ChatGPT-4 compare with both experts and crowd workers on a task with known ground truth [intro] §Method.

Method

The task is a binary label: the party (Democrat or Republican) of the author of a tweet, judged from the tweet text alone §Method.

1. Data. Use the dataset of van Vliet et al. §Method. Keep all tweets from US senators posted between 3 Sep and 3 Nov 2020. Drop retweets, replies, tweets containing a URL and tweets shorter than 100 characters §Method. Draw 250 tweets from Republican and 250 from Democratic senators, giving 500 tweets §Method.
2. Ground truth. The party of the account holder is the label §Method.
3. LLM labelling. Send each tweet to ChatGPT-4 through the API with the instruction quoted below §Method. Tweets are presented in random order §Method. Run every tweet 5 times at temperature 0.2 and 5 times at temperature 1.0, for 5000 calls in total §Method. Each run is scored on its own fig. 1.
4. Crowd labelling. Ten Master-qualified MTurk workers in the US label each tweet with the same instruction. Their answers are grouped into batches of 50 questions, each batch with 4 control questions whose answer is evident §Method. The paper reports both single-worker accuracy and the plurality of the ten answers per tweet §Results fig. 1.
5. Expert labelling. Two political science researchers, who do not specialise in US politics, label all 500 tweets with the same instruction §Method.
6. Scoring. Accuracy is the share of correct labels §Results fig. 1. Reliability is Krippendorff's alpha with a 95% bootstrap interval fig. 2. Bias is the share of answers that name Democrat §Results fig. 3.

The instruction given to the model and the crowd, quoted exactly §Method:

"You will be given a set of Twitter posts from different US politicians, sent during the two months preceding the 2020 US presidential election, that is, between September 3rd, 2020, and November 3rd, 2020. Your task is to use your knowledge of US politics to make an educated guess on whether the poster is a Democrat or Republican. Respond either Democrat' or Republican'. If the message does not have enough information for an educated guess, just make your best guess."

The method has no training, fitting or model weights. The steps below restate the procedure; they are not code from the paper.

# Zero-shot labelling of tweets by party  [§Method]
# Input:  tweets: list of (text, gold) with gold in {"Democrat", "Republican"}
# Output: answers: list of (tweet index, temperature, run, label)

def label_tweets(tweets, instruction):
    # Present tweets in random order  [§Method]
    answers = []
    for index in shuffled(range(len(tweets))):
        # Two temperatures, five independent runs each  [§Method]
        for temperature in [0.2, 1.0]:
            for run in range(5):
                # One API call per tweet and run  [§Method]
                reply = llm(instruction, tweets[index].text, temperature)
                # Parsing of replies is not stated in the paper
                label = parse_label(reply)
                answers.append((index, temperature, run, label))
    return answers

# Scores for one coder  [§Results] [fig. 1] [fig. 3]
# Input:  labels: list of answers for this coder; gold: matching party labels
# Output: accuracy and share of "Democrat" answers

def score_coder(labels, gold):
    # Accuracy: share of answers equal to the poster's party  [fig. 1]
    accuracy = mean(label == g for label, g in zip(labels, gold))
    # Bias: share of answers that name Democrat  [fig. 3]
    democrat_share = mean(label == "Democrat" for label in labels)
    return accuracy, democrat_share

# Crowd label for one tweet  [§Method] [§Results]
# Input:  votes: the 10 worker answers for this tweet
# Output: the plurality label

def crowd_label(votes):
    # Plurality of the ten answers; tie-breaking is not stated
    return most_common(votes)

Hyperparameters

Results

Author-reported. Setup: 500 US senators' tweets from 2020, ChatGPT-4 zero-shot with 5 runs at each of two temperatures, 10 MTurk workers per tweet, 2 expert coders §Method §Results.

The text gives no numeric accuracy, alpha or bias values. These are shown only in the figures and are not reproduced here.

Two tweets were also followed up with a request for the model to explain its answer §Results. Both explanations were judged plausible by the author, who notes that their stringency can be questioned §Results.

Paper vs code

No official code. The paper gives no code, data or prompt-run link, and the abstract and arXiv metadata name none. I searched GitHub with tornberg chatgpt and chatgpt political twitter annotation; both returned no repositories. The author's GitHub profile could not be located from the paper.

Pitfalls

• Unnumbered paper. Sections, figures and the opening text carry no numbers in the source. Citations use section titles ([§Method], [§Results]), [intro] for the unnumbered opening, and [fig. 1] to [fig. 3] in order of appearance. The paper has no equations, tables or algorithms.
• Model version is not pinned. The paper says "ChatGPT-4 through the API" and gives no snapshot, date or decoding settings beyond temperature. Results from one time point may not reproduce on later model versions.
• Reply parsing is not stated. The paper does not say how a reply that is neither label, or that adds text, is scored.
• Forced guess. The instruction tells the model to "just make your best guess" when unsure. There is no abstain option, so accuracy counts forced guesses only.
• Ground truth is the party of the account. The task asks the model to infer the poster's party, and the label is the account's party. Tweets from politicians may be easier than tweets from ordinary users, which the paper notes but does not test.
• Narrow test. One chamber of US Congress, one election period, one model, 500 tweets, one balanced sample. The paper says the method "could be similarly applied" to ordinary users, but does not test that.
• Balanced classes. The sample is 50/50, so accuracy is not comparable to a skewed real-world base rate.
• Per-run scoring. Figure 1 shows accuracy per model run and per worker, so each run counts as one coder. Whether the LLM is also aggregated across its 5 runs (majority vote or otherwise) is not stated.
• Crowd baseline depends on quality control. In a pre-study without strict quality control, MTurk accuracy was not significantly different from random. The final baseline uses Master qualifications, controls and a rejection rule. The rejection test for near-random accuracy is not stated.
• Expert baseline is weak. The two experts do not specialise in US politics. The paper states this itself §Method.
• Reliability units. The paper computes Krippendorff's alpha but does not say which units are the coders (runs, workers, or experts), or how many bootstrap resamples were drawn fig. 2.
• Bias test not described. The text says the biases are "significantly" different, but names no test or correction §Results. Abstract wording "equal or lower bias" should be read with the text: the LLM and experts are not significantly different, and only MTurk is significantly more biased §Results.
• Reasoning is anecdotal. The two explanations are selected cases. The paper concedes that it is unknown what drives the model's answers.
• No cost figures. The claim that LLMs are "orders of magnitude less costly and time-consuming" §Discussion is not backed by any cost or time data in the paper.
• Known LLM caveats from the paper. Sensitivity to prompt design, emergent zero-shot ability that is not well understood, and bias inherited from training data §Discussion. The paper recommends validating every task.

Lineage

• Builds on: the dataset of van Vliet et al. (2020), used for the tweets §Method. The paper cites Gilardi et al. (2023) and Wu et al. (2023) as related LLM annotation and scaling studies [intro]; no cards exist for them, so nothing from them is used here.
• Compared against: MTurk crowd workers and two expert coders, both from this paper's own study §Method.
• Superseded by: not known from content in this wiki.