LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis

In plain words

Thematic analysis is a common way to make sense of open-ended survey answers: people read the answers, tag them with short codes, group the codes into themes, and check that two readers would tag things the same way. Doing this well takes at least two human readers and several rounds of argument, so it is slow.

This paper tries a different split of the work. One human reads a few answers and writes a few worked examples that show how to turn an answer into codes. A chat model then reads the answers, proposes codes, and groups them. The human and the model go back and forth: the human edits the codes and says why, and the model says where it agrees or disagrees, until both are satisfied. The result is a codebook, which the model then uses to tag every answer.

The authors tested this on two public surveys, about how people experience music on their devices and how people manage passwords. The authors report that the results are comparable to what two human coders produce. They note that their prompts may not be the best possible, and that institutional review board rules may restrict sending the data to a third-party model.

TL;DR

• Human and LLM build a thematic codebook together, through a prompt-driven discussion loop.
• Uses GPT-3.5 (gpt-3.5-turbo-16k) at temperature 0 app. B.
• On two survey datasets, human-plus-LLM Cohen's kappa is 0.87 and 0.81, against 0.66 and 0.77 for the original studies' codes tab. 1 §5.
• Results are author-reported, with one human coder per dataset and a third coder for comparison §4.2.
• Tested only on these two survey datasets and only with GPT-3.5 §4.2 app. B. The language of the data is not stated.

Problem

Thematic analysis needs at least two human coders who become familiar with the data, build a codebook, and then code the data over several rounds until their agreement is acceptable app. A. The authors describe this as labor-intensive and time-consuming §1 §2.

Input size is a second constraint. The GPT-3.5 context limit meant the Password Manager responses could not all be sent at once, so the authors built the codebook from a sample §4.2. They also note that the data may not be allowed into a third-party service under some ethics rules § Limitations § Ethics Statement.

Method

The framework has four steps, shown in fig. 1. The human coder (HC) and the machine coder (MC, the LLM) work through the same prompts §3.

1. Data familiarization and exemplar generation §3. The HC reads the data and writes four to eight exemplars. Each exemplar is one response plus worked "actions" in the form: a quoted sentence refers to a definition, therefore a code is given. The format is chain-of-thought-like: "step by step" and "code by code" §3 fig. 2.
2. Initial code generation §3. The prompt contains the exemplars, the study goal, and the open-ended question. The MC applies the same format to each response and extracts initial codes app. D (Prompt 1).
3. Code refinement and initial codebook generation §3. The MC groups similar initial codes into codes app. D (Prompt 2). Then comes a discussion cycle. The HC reviews the codes, edits them if needed, and states each change with its reason. The MC says whether it agrees or disagrees and why. The HC revises again if needed and starts the next cycle. The cycle ends when both are satisfied. The refined codes then become the initial codebook §3 app. D (Prompt 3).
4. Theme identification and evaluation §3. The MC and HC identify themes over the codes, which gives the final codebook. Both coders then code all responses with that codebook. Agreement is measured with Cohen's kappa, and the codes are compared with the original authors' codes by cosine similarity of embeddings §3 §4.1.

The study uses a single human coder per dataset and a third coder (Coder 3) who codes with both codebooks, so the comparisons are made on the same data §4.2.

# LLM-in-the-loop thematic analysis  [§3] [fig. 1] [app. D]
# Input:  responses (list of str); question (str); exemplars (4 to 8)
# Output: final codebook (themes with codes); labels for every response

def llm_in_the_loop(responses, question, exemplars):
    # The HC writes the exemplars; the MC reuses them as examples  [§3]
    exemplar_prompt = build_prompt(exemplars, question)
    # The MC extracts initial codes for each response  [§3] [app. D] Prompt 1
    initial_codes = llm_chat(exemplar_prompt, responses)
    # The MC groups similar initial codes into codes  [§3] [app. D] Prompt 2
    codes = llm_chat(group_prompt(question, initial_codes))
    # HC edits codes with reasons; MC agrees or disagrees; repeat  [§3]
    # [app. D] Prompt 3
    while not (hc_satisfied and mc_satisfied):
        hc_edits = human_review(codes)
        codes = llm_chat(discussion_prompt(codes, hc_edits))
    # Refined codes become the initial codebook  [§3]
    initial_codebook = codes
    # Themes are identified; the final codebook is generated  [§3]
    final_codebook = llm_chat(themes_prompt(initial_codebook))
    # Both coders label all responses with the final codebook  [§3]
    labels = [code_response(r, final_codebook) for r in responses]
    return final_codebook, labels

The paper gives only the prompts in app. D (Prompts 1, 2 and 3). The prompt that assigns labels to responses and the theme-generation prompt are not in the paper; see § Paper vs code below.

Hyperparameters

Results

Author-reported. Setup: two survey datasets, GPT-3.5 at temperature 0 app. B, one human coder, a third coder (Coder 3), and the original study codes as the reference ("Gold") §4.2 §4.3. Hardware is not stated. Cohen's kappa for Gold is the original study's own coder agreement app. C §4.3.

Cohen's kappa tab. 1:

Seen = responses used to develop the codebook; Unseen = not used; All = both tab. 1 caption. Music Shuffle has no Seen/Unseen split in the table.

Agreement with the original authors' codes tab. 2:

Accuracy and recall are not defined in the paper.

Code-level versus theme-level kappa tab. 4:

Each cell is kappa at code level / kappa at theme level tab. 4.

Mismatched responses by cause tab. 3:

Headline claims §4.3 §5: human-plus-LLM agreement is "almost perfect" in both datasets, and the authors conclude the framework performs as well as two human coders, with one human instead of two. A codebook built from the partial (unseen) data reached kappa 0.82 §4.3. Results come from two datasets; the number of runs is not stated.

Paper vs code

The repository is linked from the paper's introduction §1 and belongs to the first author's GitHub account. It contains chat_mods.py, a Jupyter notebook (thematic_analysis.ipynb), and a README. There is no LICENSE file.

• Summary step is not in the paper. The code asks the model to summarise the initial codes as JSON (response number, then code and definition), and then flattens the code names into a set chat_mods.py. The paper describes the MC extracting codes and then grouping them, with no summary step §3.
• Codes passed as assistant role. The grouping prompt sends the codes as an assistant-role message, and the survey question is prepended to the instruction chat_mods.py. Prompt 2 app. D shows the instruction as a user prompt with the question and codes in it.
• Codebook numbering is not in the paper. The final codebook is numbered sequentially across all themes chat_mods.py. The notebook's labelling cell passes that codebook to label() thematic_analysis.ipynb.
• Labelling prompt is not in the paper. label() sends the codebook and all responses in one new user message with no prior conversation chat_mods.py. The paper says both coders code all responses but gives no prompt or context handling §3 app. D.
• Refinement is manual. Each discussion round stops when the human saves the edited revised_theme.json and chooses to continue or complete; the notebook says the loop runs "until saturation" thematic_analysis.ipynb thematic_analysis.ipynb. The paper states the end condition as both coders being satisfied §3.
• Evaluation is not in the repository. Cohen's kappa, the embedding cosine similarity, accuracy, recall, the 100-response sample and the pool split are not implemented in the code. The notebook refers to NLTK's agreement module for IAA thematic_analysis.ipynb.
• No discrepancies found in model and temperature: both match the paper (gpt-3.5-turbo-16k, temperature 0) app. B chat_mods.py.

Pitfalls

• Old SDK interface. The code calls openai.ChatCompletion.create chat_mods.py, the interface of the pre-1.0 OpenAI Python SDK. The README says pip install openai with no version pin README.md. A current install will not run this code without changes.
• Malformed output crashes the loop. json.loads runs on every model reply with no retry chat_mods.py chat_mods.py chat_mods.py. The notebook tells users to re-run the cell thematic_analysis.ipynb.
• Blank lines become responses. load_data treats every line of the input file as a response, including empty ones, and numbers them chat_mods.py. The paper does not say how it cleaned its data.
• Gold is not a like-for-like reference. The original-study kappa values (0.66 for Music Shuffle, 0.77 for Password Manager) come from the original studies' own coders and full data app. C §4.3. The Music Shuffle study here uses only the first question §4.2, so the reference may not match the subset.
• Human coder not described. The paper does not say who the HC was, what their expertise is, or how long they spent. Their edits shape the codes, and the paper does not measure their influence §3 §4.2.
• Runs not repeated. Temperature 0 is said to aid reproducibility, but the paper reports no repeat runs or variance app. B §4.
• Partial codebook. Using 100 of 280 Password Manager responses for the codebook is described as feasible, with the Unseen kappa at 0.82 §4.3. Only one dataset is tested this way; pool sizes are not stated §1 §4.2.
• Coder 3 drops sharply on Unseen data. Coder 3 + MC falls to 0.34 kappa on the Unseen Password Manager responses tab. 1. The authors say a discussion step with re-coding may be needed and leave it as future work §4.3.
• Band labels undefined. "Almost perfect" and "substantial" are used without a cited scale §4.1 §4.3.
• Data leaves the institution. Sending responses to a third-party model may conflict with IRB rules. The paper advises checking before use § Limitations § Ethics Statement.
• Language and other models not stated. The paper does not state the data's language; other LLMs are not tested § Limitations. Transfer to other languages or models is not shown.
• Prompts may not be optimal. The authors say better prompts may exist § Limitations. The prompts used are in app. D.

Lineage

• Builds on: in-context learning and chain-of-thought prompting, used to design the exemplar format §3; thematic analysis as described by Braun and Clarke §2 (not in this wiki).
• Related LLM coding work named by the authors: Xiao (2023), a deductive coding study with GPT-3 and an expert codebook §2; Gao et al. (2023) CollabCoder §2; Depaoli et al. (2023), ChatGPT for thematic analysis §2.
• Compared against: the original study codes ("Gold") and Coder 3 §4.2 §4.3.
• Superseded by: not known.