Chain-of-thought prompting

Overview

Chain-of-thought (CoT) prompting asks a model to write intermediate steps before its answer, and the answer is read from the end of that text. The original recipe puts worked exemplars in the prompt, each with a written chain that ends in the answer 2201.11903 §3.1. A later recipe drops the exemplars and appends a trigger such as "Let's think step by step" 2302.06476 §3. The gain is largest on arithmetic and multi-step word problems, and it appeared mainly in the largest models tested 2201.11903 app. A 2206.07682 §4. For PaLM 540B on GSM8K, 8-shot CoT moves accuracy from 17% to 54%, and 58% with an external calculator 2204.02311 tab. 10.

Most variants share one core: the prompt or the training data contains the reasoning text, and the answer is taken from it. They differ in what happens around that text. The main axes are:

• how the steps are obtained (exemplars, a trigger phrase, sampling, or training on rationales),
• whether the steps are executed by a tool (a Python interpreter, a solver, retrieval, or an environment), and
• how many chains are produced and how they are combined.

This page covers prompting and the closely related training uses. Work that trains a model to write its own chains at inference time is on the chain-of-thought and chain-of-thought-compression pages.

Variants

Few-shot CoT with worked exemplars

The original form. Eight hand-written exemplars with chains were used for arithmetic, and the paper also reports CoT on commonsense and symbolic tasks with fewer exemplars 2201.11903 §3.1. On PaLM 540B, the card reports GSM8K at 17.9 with standard prompting and 56.9 with CoT 2201.11903 app. B. The same family of method also appears in PaLM's own 8-shot evaluation 2204.02311 tab. 10, and in Minerva, which uses a fixed 4-shot prompt for MATH 2206.14858 app. D.2. Card: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.

Exemplar choice matters. Complexity-based selection picks the exemplars with the most reasoning steps, and reports GSM8K 66.6 with greedy decoding against 61.0 for handcrafted exemplars on code-davinci-002 2210.00720 tab. 1. On MathQA, retrieved exemplars score 69.5 against 42.5 for complexity-based selection 2210.00720 tab. 4. Card: Complexity-Based Prompting.

Exemplars can also be made without labels. Auto-CoT clusters test questions, generates one zero-shot chain per cluster and uses those as demonstrations. On MultiArith it scores 92.0 against 91.7 for manual CoT, and on GSM8K 47.9 against 46.9 2210.03493 tab. 3. Card: Automatic Chain of Thought.

Zero-shot trigger

No exemplars: each item gets the instruction and the input, and for reasoning tasks a two-stage prompt, the trigger phrase followed by an answer cue 2302.06476 §3. For GPT-3.5 this gives GSM8K 59.5, against 78.9 for ChatGPT in the same study 2302.06476 tab. 11. The same study reports CoT lowering accuracy on CommonsenseQA (73.7 to 71.5) and StrategyQA (61.1 to 55.5) 2302.06476 tab. 3. Card: Is ChatGPT a General-Purpose NLP Task Solver?.

Zero-shot prompts are also used in clinical and driving settings. A frozen multimodal model answers a fixed sequence of sub-questions in a zero-shot chain for driving 2411.14256 §2.1, and VLMnav uses a visual cue rather than text chains 2411.05755 §3.4. A position paper proposes a step-splitting metaprompt but reports no accuracy for it 2102.07350 §4.6.

Sampling and majority voting (self-consistency)

Sample m chains at a non-zero temperature, extract each final answer, and return the most frequent one. No training is needed. On PaLM 540B GSM8K goes from 56.5 to 74.4, and on code-davinci-002 from 60.1 to 78.0 2203.11171 tab. 2. Weighting votes by the chain's probability collapses to 22.1 on GSM8K 2203.11171 tab. 1. Card: Self-Consistency Improves Chain of Thought Reasoning.

Minerva used the same idea with a different count: maj1@k at 540B reaches 50.3 on MATH against 33.6 greedy 2206.14858 tab. 3. Medprompt combines retrieved exemplars, GPT-4-written chains that are kept only when correct, and a vote over option-shuffled calls. The card reports 90.2% on MedQA with 5 exemplars and 5 calls, and it names the cost in calls as the price 2311.16452 tab. 1 2311.16452 §4.4. Ensemble refinement, used in the medical QA work, adds a second stage that conditions on the first 11 chains 2305.09617 §3.3. Card: Towards Expert-Level Medical Question Answering.

CoT-decoding removes the prompt. It branches on the top-k first tokens, decodes each branch greedily, and scores each answer by a probability margin over its answer span. On Mistral-7B, GSM8K goes from 9.9% greedy to 25.1% 2402.10200 tab. 4. The card notes that its "Q: / A:" format is kept, so the method is not strictly prompt-free 2402.10200 §2.1. Card: Chain-of-Thought Reasoning Without Prompting.

Gemini Ultra routes between two modes. It takes 32 CoT samples and returns the majority answer when agreement is high enough, and otherwise falls back to greedy decoding without CoT. The card reports 90.04% on MMLU, against 83.7% for 5-shot 2312.11805 tab. 2. Routing alone gives 90.0, against 85.0 for 32 samples without routing 2312.11805 §10.2. Card: Gemini.

Program-aided and tool-executed steps

The steps are written as code or a formal language, and an interpreter or solver produces the answer. PAL has Codex write Python with comment steps and run it. It reports GSM8K 72.0 against 65.6 for CoT on code-davinci-002 2211.10435 tab. 1. Card: PAL.

Logic-LM translates a problem into logic, runs a deterministic solver, and on error has the model revise using the error message. On GPT-4 LogicalDeduction it scores 87.63 against 75.25 for CoT, but on PrOntoQA CoT is higher (98.79 against 83.20) 2305.12295 tab. 2. Card: Logic-LM.

Decomposed prompting routes each sub-question to a handler, which can be a CoT prompt, a symbolic function, retrieval or a recursive call 2210.02406 §3.2. Chameleon is a planner that calls a fixed set of modules, including an LLM solution generator, and reports GPT-4 at 86.54 on ScienceQA against 83.99 for CoT 2304.09842 tab. 3. DIN-SQL splits text-to-SQL into four prompted modules and reaches 85.3 execution accuracy on the Spider holdout with GPT-4 2304.11015 tab. 2. Cards: DecomP, Chameleon, DIN-SQL.

Search over thoughts

Tree of Thoughts proposes several candidate steps, scores partial states and searches with breadth-first or depth-first moves. On Game of 24 it reports 74% with b=5, against 4.0% for CoT and 9.0% for CoT with 100 samples 2305.10601 tab. 2. The cost is in calls: $0.74 per case, against $0.47 for best-of-100 CoT 2305.10601 tab. 7. Card: Tree of Thoughts.

Interleaving reasoning with retrieval or actions

IRCoT writes one chain sentence at a time, uses its last sentence as a retrieval query and feeds the retrieved paragraphs back in 2212.10509 §3. With GPT-3 it reports HotpotQA 60.7 against 53.6 for one-step retrieval 2212.10509 tab. 4. Card: IRCoT.

ReAct alternates thought, action and observation lines. On HotpotQA it is below CoT with EM 27.4 against 29.4, but CoT with self-consistency followed by ReAct reaches 34.2 2210.03629 tab. 1. On WebShop it reaches 40.0 success against 30.1 for action-only prompting 2210.03629 tab. 4. Card: ReAct.

Selection-Inference alternates a selection step and an inference step, each run by a frozen model. The 7B model reaches 58.75% on its generative average against 41.32% for CoT 2205.09712 §5. Card: Selection-Inference.

Multi-agent and debate

Several copies of one chat model answer, read each other's answers and revise over rounds. On GSM8K, three agents reach 85.0 against 77.0 for one 2305.14325 tab. 1. MedAgents is zero-shot, with role-played experts who vote and revise. Its GPT-4 MedQA score is 83.7 2311.10537 tab. 2. The card reports a temperature of 1.0 in the paper against 0 in the code. SurgRAW routes questions to a visual or knowledge stream and uses CoT prompts for each 2503.10265 §3.2. Cards: Multiagent Debate, MedAgents, SurgRAW.

Answer-first, rationale-after and multimodal chains

Some work puts the answer before the reasoning. ScienceQA's format writes "The answer is X" and then the explanation, and it reports GPT-3 at 75.17 with explanation against 73.97 for answer-only 2209.09513 tab. 3. Its dataset carries a non-commercial licence, so only aggregate numbers from it are used here 2209.09513 app. A.1.

Multimodal CoT first writes a rationale from text and image, then answers from text plus the rationale 2302.00923 §4.1. This is fine-tuning, not prompting, and the card reports a rationale-first pilot below the answer-first pilot in one setting 2302.00923 tab. 2. Card: Multimodal CoT.

Training on chains instead of prompting

Chains can be produced by a teacher and used to train a small model. Specialization samples 40 solutions per GSM8K question from code-davinci-002 and keeps the correct ones 2301.12726 §4.1. The student then gains on GSM8K (FlanT5-XL 3B from 13.5 to 22.4) but loses generic reasoning, with BBH-CoT at 0.0 for every specialized model 2301.12726 tab. 1. Card: Specializing Smaller Language Models.

Instruction tuning on CoT data has a similar aim. The Flan Collection mixes zero-shot and few-shot templates, and without its CoT tasks the CoT score falls from 35.8 to 28.8 2301.13688 tab. 1. Flan-PaLM 540B reaches MMLU 75.2 with CoT and self-consistency, against 69.3 for PaLM 2210.11416 tab. 4. Cards: The Flan Collection, Flan.

Distilled reasoning pruning labels each step of a teacher-split trace as keep, delete or compress, then fine-tunes a 7B student on the result. On GSM8K it reports 94.1% at 328 tokens, against 91.7% at 917 tokens for the unpruned 7B model 2505.13975 tab. 1. Card: DRP.

GPT-3 writes explanations that a Flan-T5 Large model is then fine-tuned on, reaching 80.76 on CoNLL04 2305.05003 tab. 2. Galactica does not use prompts for its chains; its work tokens are learned in pre-training 2211.09085 §3.2. Card: Galactica.

TG-LLM bootstraps chains from sampled GPT-3.5 answers that are correct, and it fine-tunes Llama-2 13B with LoRA 2401.06853 §3.2.1. Think Then Embed writes a reasoning trace before an embedding 2510.05014 §4. Cards: TG-LLM, Think Then Embed.

Decoding and sampling choices

Sampling temperature is a separate knob. A study of nine models across temperatures 0.0 to 1.0 recommends 0.0 for problem solving 2402.05201 §4.1. Self-consistency needs sampling, and the papers that use it report temperatures of 0.5 to 0.7 2203.11171 §3.1. Card: The Effect of Sampling Temperature.

Clinical and other domain use

Flan-PaLM 540B with CoT demonstrations and self-consistency over 11 paths reaches 67.6 on MedQA 2212.13138 tab. 4 2212.13138 §4.4. Its soft-prompt variant adds 100 tokens to every prompt 2212.13138 §3.3.3. Card: Med-PaLM.

Ensemble refinement is one of several medical QA routes; the MedQA-only variant reaches 86.5 2305.09617 tab. 4. ManipLLM writes a three-step chain at inference, and removing it gives 0.56 against 0.59 for the full model 2312.16217 tab. 2.

Benchmarks and analyses

Several cards measure CoT rather than propose a method.

• Irrelevant context: one irrelevant sentence added to GSM8K problems lowers CoT, and self-consistency recovers some accuracy 2302.00093 tab. 3. Card: Large Language Models Can Be Easily Distracted.
• Faithfulness: when a biasing feature is in the prompt, CoT explanations often omit it. On BBH the Suggested Answer drop is -36.3 for GPT-3.5 2305.04388 tab. 8. Card: Unfaithful Explanations.
• Benchmarks with CoT baselines: MMLU-Pro reports GPT-4o at 72.6 with 5-shot CoT 2406.01574 tab. 2. VSI-Bench reports that text-only reasoning prompts lower average scores by about 4% 2412.14171 §5.2.
• Negotiation: CoT and few-shot are tested only on subsets of the 35 tasks 2402.13550 §5. Card: Negotiation evaluation.
• Prompt search: an LLM proposes instructions and scores them on a training subset. On GSM8K its best instruction scores 80.2 against 71.8 for "Let's think step by step." 2309.03409 tab. 4. Card: LLMs as Optimizers.
• Probes and surveys: Sparks of AGI uses a pre-release GPT-4 and gives no decoding settings 2303.12712 §3.1.1. Multi-step reasoning survey and Agentic LLMs survey list methods without new experiments. Prompt engineering survey does the same for prompt engineering 2310.14735 §3.1.1. The agentic survey's tables list 21 reasoning rows 2503.23037 tab. 1.
• Limits of reasoning: The Illusion of Thinking finds that reasoning models fail on rule-based puzzles above a certain size 2506.06941 app. A.1.

When to use which

Implementation sketch

Self-consistency over a few-shot CoT prompt. Greedy decoding is the m = 1 special case. Lines are in Python style.

# Build the prompt: exemplars whose chains end in "The answer is N."
# [arxiv:2201.11903 §3.1]
prompt = join(exemplars) + "\nQ: " + question + "\nA:"

# Greedy CoT: one decode, then parse the final answer
# [arxiv:2201.11903 §3.1]
answer = parse_final(generate(prompt, temperature=0))

# Self-consistency: sample m chains at a non-zero temperature
# [arxiv:2203.11171 §3.1] (T = 0.5 to 0.7; m = 40 in §3.2)
chains = [generate(prompt, temperature=0.7) for _ in range(m)]

# Vote over parsed answers, unweighted; tie rule not stated
# [arxiv:2203.11171 §2]
answers = [parse_final(c) for c in chains]
answer = most_common(answers)

Tool-executed variants replace the parse step with execution. PAL runs the generated Python and reads its result 2211.10435 §5.1. Retrieval variants add a search call between chain sentences 2212.10509 §3.

Contradictions

• CoT on commonsense and strategy tasks. One card reports CoT gains of several points on the same benchmark families 2201.11903 §4, while another reports CoT losses on CommonsenseQA and StrategyQA with different models and prompts 2302.06476 tab. 3. The cards do not say which setting explains the gap.
• PaLM 540B on GSM8K. One card reports 54% with 8-shot CoT 2204.02311 tab. 10. Another reports 56.9 with CoT for the same model and benchmark 2201.11903 app. B. The cards do not state the exemplars or decoding that differ.
• Temperature. One card recommends temperature 0.0 for problem solving 2402.05201 §4.1. Self-consistency relies on sampling at 0.5 to 0.7 2203.11171 §3.1.
• Size of the self-consistency gain. The survey's text says "10–20 points" while its table gives +4 to +18 2407.11511 §3.2.1 2407.11511 tab. 2. One study finds its own claim of more than 11 points is not reproducible from its table 2302.00093 tab. 3.
• Scale threshold. The emergence survey places CoT gains near 1.3e23 FLOPs 2206.07682 §4 2206.07682 tab. 1. The same survey notes that thresholds do not transfer across model families 2206.07682 §2. Treat any single size as model-specific.
• Baselines in medical CoT. Diagnostic prompts for GPT-3.5 and GPT-4 are compared with baselines that differ: GPT-3.5 used a framework and self-consistency, GPT-4 did not 2308.06834 p. 4. For GPT-4, the CoT result (76%) is not clearly different from the intuitive prompt (77%) 2308.06834 p. 3.

Papers

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models: the original few-shot CoT method and its scale results.
• Self-Consistency Improves Chain of Thought Reasoning: sample m chains and vote by unweighted majority.
• PaLM: a model paper whose 8-shot GSM8K result with CoT is a main reference point.
• Selection-Inference: alternating selection and inference steps with a frozen model.
• Minerva: maj1@k sampling for quantitative reasoning.
• Emergent Abilities: survey of scale thresholds for CoT and related gains.
• ScienceQA CoT: answer-first explanations on science questions.
• Automatic Chain of Thought: exemplars chosen by clustering, with no labels.
• ReAct: thought, action and observation lines.
• Flan: instruction tuning on CoT tasks at scale.
• Galactica: a science model where reasoning lives in trained work tokens.
• Complexity-Based Prompting: exemplars chosen by number of steps.
• DecomP: decomposed prompting with handlers per sub-question.
• PAL: code steps executed by an interpreter.
• The Flan Collection: instruction mixture design with CoT tasks.
• Irrelevant context study: GSM-IC benchmark and robustness of CoT.
• Specializing Smaller Language Models: distilling teacher chains into smaller students.
• Is ChatGPT a General-Purpose NLP Task Solver?: zero-shot CoT evaluation.
• Multimodal CoT: rationale generation before answering, with vision features.
• Sparks of AGI: probing study of an early GPT-4.
• Chameleon: planner that calls compositional modules.
• DIN-SQL: decomposed prompting for text-to-SQL.
• Revisiting relation extraction: explanations for relation extraction with a small model.
• Towards Expert-Level Medical QA: ensemble refinement for medical QA.
• Unfaithful explanations: CoT explanations that omit biasing features.
• Multiagent Debate: several copies of one model debate over rounds.
• Logic-LM: logic translation with a solver and revision.
• Diagnostic reasoning prompts: clinician-styled few-shot prompts for diagnosis.
• LLMs as Optimizers: an LLM proposes and scores prompt instructions.
• Prompt engineering survey: survey of prompting methods.
• MedAgents: zero-shot multi-agent role-play for medical QA.
• Medprompt: retrieved exemplars with CoT and voting on MedQA.
• Gemini: uncertainty-routed CoT with majority voting on MMLU.
• ManipLLM: embodied model with chain-style manipulation steps.
• TG-LLM: LoRA tuning on bootstrapped temporal chains.
• Temperature study: temperature effects on problem solving.
• CoT-decoding: branching on first tokens instead of prompting.
• Negotiation evaluation: zero-shot and CoT evaluation of negotiation.
• MMLU-Pro: benchmark with a 5-shot CoT protocol.
• Multi-step reasoning survey: survey of generation, evaluation and control.
• Vision-language navigation: zero-shot visual prompting for navigation.
• Autonomous driving with zero-shot LLMs: zero-shot CoT sub-questions for steering.
• VSI-Bench: spatial benchmark with text-only reasoning probes.
• SurgRAW: multi-agent surgical QA with CoT prompts.
• Agentic LLMs survey: survey of reasoning, acting and interacting.
• DRP: pruning distilled reasoning traces.
• The Illusion of Thinking: reasoning models on rule-based puzzles.
• Think Then Embed: reasoning before embedding.