Qualitative coding with LLMs

Overview

Qualitative coding means reading open-ended material, such as survey answers, interview transcripts or children's questions, and tagging each item with a code from a set. In deductive coding the set is fixed in advance in a codebook: each code has a name, a description and examples 2304.10548 §2. The concept here is to let a language model apply that codebook, with no training of a new model, and to measure how well its labels match those of human experts 2304.10548 §1 2304.10548 §4.

The shared assumptions are that the codebook can be written in a form a model reads, that the data was not seen during the model's training, and that agreement with expert coders is an acceptable quality measure 2304.10548 §1 2304.10548 §4. The one study so far tests these on a single dataset, so they are not established beyond it.

Variants

Codebook-style prompt

Each code is laid out as a block with its name, its description and its examples, the way a researcher reads a codebook. The prompt also states a persona line for a developmental psychologist and lists the candidate codes 2304.10548 §3. This was the stronger of the two prompt styles tested 2304.10548 §4. Card: Xiao et al., GPT-3 codebook prompting for deductive coding.

Example-style prompt

Each example is given with a sentence that explains why it has its code. The code, description and examples are the same as in the codebook-style prompt; only the layout differs 2304.10548 §3. The paper reports it as weaker than the codebook-style prompt on its dataset 2304.10548 §4. This style needs at least one example, so it has no zero-shot form 2304.10548 §3.

Number of examples per code

The paper compares three settings 2304.10548 §3:

• Zero-shot: code name and description only.
• One-shot: one example per code.
• Few-shot: five examples per code.

Moving from zero-shot to one-shot gave the largest gain. The authors describe the difference between one-shot and five-shot as small 2304.10548 §4. The paper does not say how the examples were selected for few-shot prompts 2304.10548 §3.

Language of the input

The paper coded two dimensions differently 2304.10548 §2.1. For one dimension (question complexity) the model first translated the items into English. For the other (syntactic structure) the items stayed in French so that their syntax was kept 2304.10548 §2.1. The translation step is part of the labelling pipeline, and the paper does not measure its effect.

When to use which

The only evidence is one study, so these are the conditions under which the study's results apply, not general rules.

Two cautions follow from the numbers. The complexity result is labelled "substantial" and the syntax result "fair" on the scale the paper uses 2304.10548 §4, so the two dimensions differ a lot in quality. The paper also reports only the final expert labels as the reference, so the experts' own disagreement is not reflected in the model's score 2304.10548 §2.1 2304.10548 §4.

Implementation sketch

The common core is one prompt per item, built from the codebook, with the model's output restricted to the candidate codes. Placement of the item in the prompt, output length and sampling settings other than temperature are not stated in the paper 2304.10548 §3.

# Codebook-style deductive labelling  [arxiv:2304.10548 §3]
# Input:  codebook (per code: name, description, examples); one item
# Output: one code from the candidate set

def label_item(codebook, item, shots_per_code):
    # 1. Role line for the model; the paper used a developmental psychologist
    prompt = role_line()
    # 2. One block per code: name, description, then examples
    #    shots_per_code = 0 (zero-shot), 1 (one-shot) or 5 (few-shot)
    for code in codebook:
        prompt += format_code_block(code.name, code.description)
        prompt += format_examples(code.examples[:shots_per_code])
    # 3. Tell the model to choose only from the candidate codes
    prompt += "Choose from the following candidates: " + codebook.names
    # 4. Put the item in the prompt; placement is not stated in the paper
    prompt += format_item(item)
    # 5. Greedy decoding, temperature 0.0, as in the paper [arxiv:2304.10548 §3]
    return llm_complete(prompt, temperature=0.0)

If the dimension depends on syntax, skip the translation step. If it depends on meaning, the paper's pipeline translated the item first 2304.10548 §2.1. Record which choice was made, because the paper does not measure the effect of translation.

Contradictions

None known. Only one paper covers this concept so far.

Papers

• Xiao et al., GPT-3 codebook prompting for deductive coding: GPT-3 with codebook-style prompts for deductive coding of 668 French children's questions; kappa 0.61 for complexity and 0.38 for syntax against expert labels; a preliminary, author-reported study.