---
kind: concept
slug: qualitative-coding
title: Qualitative coding with LLMs
summary: >-
  Using a language model to assign fixed codes from a researcher's codebook to open-ended text (deductive coding). A codebook-style prompt with about one example per code gave the most agreement with experts in one small study, at kappa 0.61 for one dimension and 0.38 for another, both below expert-to-expert agreement. One study so far, on one dataset.
tags: [qualitative-analysis, deductive-coding, prompting, annotation]
papers: ["2304.10548"]
related: [prompt-engineering, in-context-learning, inter-annotator-agreement, text-classification, llm-evaluation]
status: draft
updated_at: 2026-11-01
---

## Overview

Qualitative coding means reading open-ended material, such as survey answers,
interview transcripts or children's questions, and tagging each item with a
code from a set. In deductive coding the set is fixed in advance in a
codebook: each code has a name, a description and examples [arxiv:2304.10548 §2].
The concept here is to let a language model apply that codebook, with no
training of a new model, and to measure how well its labels match those of
human experts [arxiv:2304.10548 §1] [arxiv:2304.10548 §4].

The shared assumptions are that the codebook can be written in a form a model
reads, that the data was not seen during the model's training, and that
agreement with expert coders is an acceptable quality measure
[arxiv:2304.10548 §1] [arxiv:2304.10548 §4]. The one study so far tests these
on a single dataset, so they are not established beyond it.

## Variants

### Codebook-style prompt

Each code is laid out as a block with its name, its description and its
examples, the way a researcher reads a codebook. The prompt also states a
persona line for a developmental psychologist and lists the candidate codes
[arxiv:2304.10548 §3]. This was the stronger of the two prompt styles tested
[arxiv:2304.10548 §4]. Card: [Xiao et al., GPT-3 codebook prompting for deductive coding](/papers/2304.10548).

### Example-style prompt

Each example is given with a sentence that explains why it has its code. The
code, description and examples are the same as in the codebook-style prompt;
only the layout differs [arxiv:2304.10548 §3]. The paper reports it as weaker
than the codebook-style prompt on its dataset [arxiv:2304.10548 §4]. This
style needs at least one example, so it has no zero-shot form
[arxiv:2304.10548 §3].

### Number of examples per code

The paper compares three settings [arxiv:2304.10548 §3]:

- **Zero-shot:** code name and description only.
- **One-shot:** one example per code.
- **Few-shot:** five examples per code.

Moving from zero-shot to one-shot gave the largest gain. The authors describe
the difference between one-shot and five-shot as small [arxiv:2304.10548 §4].
The paper does not say how the examples were selected for few-shot prompts
[arxiv:2304.10548 §3].

### Language of the input

The paper coded two dimensions differently [arxiv:2304.10548 §2.1]. For one
dimension (question complexity) the model first translated the items into
English. For the other (syntactic structure) the items stayed in French so
that their syntax was kept [arxiv:2304.10548 §2.1]. The translation step is
part of the labelling pipeline, and the paper does not measure its effect.

## When to use which

The only evidence is one study, so these are the conditions under which the
study's results apply, not general rules.

| Situation | Use | Why |
|---|---|---|
| A finished codebook with examples for each code | Codebook-style prompt, one example per code | Codebook-style beat the example-style prompt, and one example per code gave most of the gain over zero-shot [arxiv:2304.10548 §4] |
| No examples yet | Zero-shot codebook-style prompt | Only the codes and descriptions are needed; expect lower agreement than one-shot [arxiv:2304.10548 §4] |
| Many codes with long examples, tight context | One example per code rather than five | Five shots did not differ much from one, so the extra context was not repaid in this study [arxiv:2304.10548 §4] |
| Coding a dimension that depends on surface syntax | Keep the original language | The paper coded syntax in French and got kappa 0.38 against experts [arxiv:2304.10548 §2.1] [arxiv:2304.10548 §4] |
| Coding a dimension about meaning or difficulty | Translation to English is possible | The paper translated for complexity and got kappa 0.61 [arxiv:2304.10548 §2.1] [arxiv:2304.10548 §4] |
| Results must match expert coders closely | Keep a human coder in the loop | Agreement stayed below expert-to-expert agreement on both dimensions [arxiv:2304.10548 §4]; the paper does not test a human-in-the-loop workflow |

Two cautions follow from the numbers. The complexity result is labelled
"substantial" and the syntax result "fair" on the scale the paper uses
[arxiv:2304.10548 §4], so the two dimensions differ a lot in quality. The
paper also reports only the final expert labels as the reference, so the
experts' own disagreement is not reflected in the model's score
[arxiv:2304.10548 §2.1] [arxiv:2304.10548 §4].

## Implementation sketch

The common core is one prompt per item, built from the codebook, with the
model's output restricted to the candidate codes. Placement of the item in the
prompt, output length and sampling settings other than temperature are not
stated in the paper [arxiv:2304.10548 §3].

```pseudocode
# Codebook-style deductive labelling  [arxiv:2304.10548 §3]
# Input:  codebook (per code: name, description, examples); one item
# Output: one code from the candidate set

def label_item(codebook, item, shots_per_code):
    # 1. Role line for the model; the paper used a developmental psychologist
    prompt = role_line()
    # 2. One block per code: name, description, then examples
    #    shots_per_code = 0 (zero-shot), 1 (one-shot) or 5 (few-shot)
    for code in codebook:
        prompt += format_code_block(code.name, code.description)
        prompt += format_examples(code.examples[:shots_per_code])
    # 3. Tell the model to choose only from the candidate codes
    prompt += "Choose from the following candidates: " + codebook.names
    # 4. Put the item in the prompt; placement is not stated in the paper
    prompt += format_item(item)
    # 5. Greedy decoding, temperature 0.0, as in the paper [arxiv:2304.10548 §3]
    return llm_complete(prompt, temperature=0.0)
```

If the dimension depends on syntax, skip the translation step. If it
depends on meaning, the paper's pipeline translated the item first
[arxiv:2304.10548 §2.1]. Record which choice was made, because the paper
does not measure the effect of translation.

## Contradictions

None known. Only one paper covers this concept so far.

## Papers

- [Xiao et al., GPT-3 codebook prompting for deductive coding](/papers/2304.10548): GPT-3 with codebook-style prompts for deductive coding of 668 French children's questions; kappa 0.61 for complexity and 0.38 for syntax against expert labels; a preliminary, author-reported study.