Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding

In plain words

Researchers who study open-ended text, such as survey answers or what children say, often sort each item into a fixed set of labels using a written codebook. Doing this by hand takes a lot of time, and the work gets slow with large datasets. This paper asks whether a general-purpose language model can do that sorting if it is simply given the codebook in its prompt, with no training of a special model.

The authors tried it on one task: labelling children's curiosity questions by how hard the answer is and by how the question is worded. The model did reasonably well on one of the two labelling dimensions and less well on the other. Its agreement with the human experts stayed clearly below the agreement between the experts themselves. The paper is a small first study, and the authors present it as a starting point for further work.

TL;DR

• Uses GPT-3 with the expert's codebook (codes, descriptions, examples) in the prompt, for deductive coding of 668 French children's questions §2.1 §3.
• Agreement with the experts' final codes: Cohen's kappa 0.61 for question complexity and 0.38 for syntactic structure §4.
• Codebook-centred prompts beat example-centred prompts; the largest gain comes from moving from zero to one example per code §4.
• Preliminary, single-dataset study; results are author-reported and no code is released.

Problem

Deductive coding labels each data item with a fixed set of codes from a codebook that researchers have already written §2. Manual coding is slow and needs several coders to reach reliable agreement §1. Trained classifiers need large labelled datasets and compute, and unsupervised topic models are hard to steer toward a specific research question §1. The paper's premise is that a pre-trained LLM, prompted with the codebook and without fine-tuning, could do the coding §1.

The assumptions are: the codebook is written in a form a model can read; the data is not in the model's training set (the authors say their dataset and codebook were never published online §2.1); and agreement with expert coders is an acceptable measure of quality §4.

Method

The task is to assign one code to each child's question, choosing from a fixed code set §3. The pipeline has these parts:

1. Dataset and codebooks. A team of psychologists coded 668 French questions on two dimensions §2.1:
   • Question complexity, a binary code (the example prompt uses the code HIGH; the other code's label is not written out in the text) §3.
   • Syntactic structure, with four categories: closed or declarative questions; question words in the middle of the sentence; questions without an interrogative form; and questions with a question word at the start and interrogative syntax §2.1.
2. Language handling. For question complexity, GPT-3 first translated the questions into English. For syntactic structure, the questions were kept in French so that their syntax was preserved §2.1, footnote 1.
3. Prompt construction. Every prompt includes an identity line that asks the model to act as a developmental psychologist with expertise in linguistics, and an instruction to choose from a given list of candidate codes §3.
4. Prompt style. Two designs are compared §3:
   • Codebook-centred: each code is laid out as Code, Description, Examples, similar to how a researcher reads a codebook.
   • Example-centred: each example comes with a sentence explaining why it has that code. The code, descriptions and examples are the same in both designs §3.
5. Number of examples. Zero-shot gives only code and description, one-shot gives one example per code, and few-shot gives five examples per code. The example-centred style has no zero-shot variant, because it needs at least one example §3.
6. Decoding. The model is GPT-3 (davinci-text-002) at temperature 0.0, which the authors chose because it is greedy and reproducible §3.
7. Scoring. Agreement is Cohen's kappa between the model's labels and the experts' final labels. The paper also reports kappa between the two experts who coded the dataset with the same codebook §4.

# Codebook-centred deductive labelling  [§3]
# Input:  codebook (per code: name, description, examples); question text
# Output: one code from the code set

def label_question(codebook, question):
    # 1. Identity line, used in every prompt variant  [§3]
    prompt = identity_line()
    # 2. Codebook block: code, description, then examples  [§3]
    for code in codebook:
        prompt += format_code_block(code.name, code.description)
        # zero-shot: no examples; one-shot: 1 per code; few-shot: 5 per code
        prompt += format_examples(code.examples)
    # 3. Constrain the output space to the candidate codes  [§3]
    prompt += "Choose from the following candidates: " + codebook.names
    # 4. Place the question in the prompt: placement not stated  [§3]
    # 5. Greedy decoding, temperature 0.0  [§3]
    label = llm_complete(prompt, temperature=0.0)
    return label

Other parameters, such as the maximum output length, top-p and how the question is inserted into the prompt, are not stated.

Hyperparameters

Results

Author-reported. Setup: 668 French children's questions §2.1; GPT-3 davinci-text-002 at temperature 0.0 §3; expert agreement as the reference §4. There are no baseline models beyond the experts themselves.

fig. 1 shows, for each dimension, the kappa of each prompt variant against the experts §4. Its caption says codebook-centred prompts with examples reach the highest agreement fig. 1. The per-variant kappa values are only in the figure and are not restated in the text.

Paper vs code

No official code. Paper links: the only URL is a footnote to a product page §1, and the arXiv metadata has no comment or project link. Searches: GitHub repository searches for "deductive coding", "codebook GPT-3" and "qualitative coding LLM" returned no repository from the authors; a GitHub user search for the first author's name found a login whose repositories contain nothing on coding or codebooks.

Pitfalls

• No code and no data release. The paper does not say the dataset or the prompts will be shared. Re-running the study requires the authors' data or a new labelled set, and the codebook text itself, which is what gives the method its meaning, is not reproduced in the paper.
• Complexity codes were applied to GPT-3's English translations, not the originals §2.1 fn. 1. Any errors in the translation enter the labels, and the paper does not measure them.
• Only the final expert labels are used as reference. How the experts reached those final codes, and how many coders there were per item, are not stated §2.1 §4.
• The model is a closed, time-specific API. The paper says it was the most advanced publicly available GPT-3 when the experiments ran §3. Output may not repeat with later API versions; the paper gives no version pinning beyond the model name.
• Temperature 0.0 does not guarantee identical outputs. The authors call greedy decoding reproducible §3, but no repeat runs are reported, so run-to-run variance is unknown.
• No confidence intervals or sample sizes per kappa are given §4. The kappa differences between prompt variants are described only qualitatively.
• Interpretation thresholds. "Fair" and "substantial" follow McHugh's scale, cited in the paper §4; the paper does not restate the numeric cut-offs.
• Two dimensions, one dataset, one prompt set. The authors call the study preliminary §5. The binary complexity and four-way syntax labels are specific to this codebook and may not transfer.
• Prompt details left out. Question placement, output length, and which examples were selected for few-shot are not stated §3. An implementer will have to choose these and should record the choice.
• The example-centred style has no zero-shot variant §3, so the zero-shot comparison covers only the codebook-centred style.
• The paper's own challenges §5: error analysis of disagreements was not done, the model produced incorrect labels, and the approach has not been tested for inductive coding.

Lineage

• Builds on: in-context and prompt-based learning with a frozen LLM (cited as prior work §1 §3) and the deductive-coding literature (cited §2). None of those works has a card on this site.
• Compared against: expert-to-expert agreement with the same codebook §4; prompt variants within the paper §3.
• Superseded by: not stated in this card.