Self-Consistency Improves Chain of Thought Reasoning in Language Models

In plain words

When a language model is asked a maths word problem and told to "show its work", it usually writes one line of reasoning and one answer. If a single step in that line is wrong, the answer is wrong, and there is no second opinion.

Self-consistency asks the same model the same question many times, letting it pick slightly different words each time so that the reasoning takes different routes. Each attempt ends with an answer. Wrong reasoning tends to go wrong in many different ways and land on scattered answers, while correct reasoning, however it is phrased, tends to land on the same answer. So the method simply counts the answers and returns the most common one, much like asking several people to solve a problem independently and trusting the result most of them agree on.

Nothing is trained and no extra model is needed; it only changes how answers are generated from an existing model. In the paper this lifted accuracy on a grade-school maths benchmark from about 57% to 74% for one large model. The cost is running the model many times per question (the paper uses 40 runs, and suggests 5 to 10 as a cheaper start), and it only works when answers can be compared directly, such as numbers or multiple-choice letters.

TL;DR

Self-consistency is a decoding strategy for few-shot chain-of-thought (CoT) prompting: instead of one greedy decode, sample m reasoning paths (temperature/top-k/nucleus sampling), parse a final answer from each, and return the plurality answer. Author-reported, with m=40 and averaged over 10 runs, it beats greedy CoT on every model tested (UL2-20B, LaMDA-137B, PaLM-540B, GPT-3 code-davinci-001/002), e.g. GSM8K +17.9 points on both PaLM-540B and code-davinci-002, AQuA +12.2 to +12.5, SVAMP +7.6 to +11.0. It beats sample-and-rank, beam search, prompt-order and multi-prompt ensembles. Relevant if you need more accuracy from a frozen LLM on tasks with a short, comparable final answer and can pay roughly m\times decoding cost.

Problem

Chain-of-thought prompting decodes one reasoning path greedily, which inherits greedy decoding's repetitiveness and local optimality, while a single sampled path is noisy §1. Earlier fixes for reasoning quality train an extra verifier or a re-ranker on human annotations §1 §4. The paper assumes a problem admits several valid reasoning routes to one correct answer, and that correct routes agree on the answer more often than incorrect ones §2. It also assumes the answer lies in a fixed answer set, so that two answers can be checked for equality §2.

Method

Prompt the model with hand-written CoT exemplars, sample many outputs from the decoder, split each into a reasoning path \mathbf{r}_i and an answer \mathbf{a}_i, then marginalise over the paths by voting on the answers §2 fig. 1.

1. Prompt. Build the standard few-shot CoT prompt §2. The paper reuses the prompts of Wei et al. (arXiv:2201.11903): the same 8 hand-written exemplars for all arithmetic tasks, and 4-7 training-set exemplars with hand-written chains for each commonsense task §3.1. Exemplars follow the format "{reasoning}. The answer is X." §2. Exact prompts are listed in app. A.3 tab. 14 tab. 15 tab. 17.
2. Sample. Draw m candidates (\mathbf{r}_i, \mathbf{a}_i), i=1..m, independently from the decoder with any sampling scheme (temperature, top-k, nucleus) §2. The reasoning path is a latent variable that only serves to reach the answer §2.
3. Truncate and parse. Keep generated text up to the start of the next "Q:" app. A.2. The parser is task-dependent: for arithmetic, take the first number after "The answer is "; for commonsense, take the full string after "The answer is " §2.
4. Aggregate. Return \arg\max_a \sum_{i=1}^m \mathbb{1}(\mathbf{a}_i = a), the majority (plurality) vote, which the paper calls the most "consistent" answer §2.

Alternative aggregations compared in tab. 1: weight each vote by P(\mathbf{r}_i, \mathbf{a}_i \mid \text{prompt}, \text{question}), either raw or length-normalised:

P(\mathbf{r}_i, \mathbf{a}_i \mid \text{prompt}, \text{question}) = \exp\Big(\frac{1}{K}\sum_{k=1}^{K} \log P(t_k \mid \text{prompt}, \text{question}, t_1,\ldots,t_{k-1})\Big) eq. 1

where t_k is the k-th token of (\mathbf{r}_i, \mathbf{a}_i) and K its token count §2. The "weighted sum" gives each answer the sum of these weights over its samples; the "weighted average" divides that sum by the answer's vote count §2. Normalised weighted sum and plain majority vote are nearly identical in accuracy, because the normalised probabilities of different samples are very close to one another; unnormalised weights and the weighted average are clearly worse §2 tab. 1. The PaLM-540B main results equal the majority-vote row of tab. 1 (e.g. GSM8K 74.4) tab. 2.

# Self-consistency over chain-of-thought samples  [§2] [fig. 1]
# Input:  question string; cot_exemplars (few-shot prompt text)
# Output: final answer and its agreement ratio

def self_consistency(question, cot_exemplars, num_samples=40):
    # Few-shot CoT prompt in "Q: ... A: ... The answer is X." format  [§2]
    prompt = cot_exemplars + "Q: " + question + "\nA:"
    answers = []
    for sample_index in range(num_samples):
        # Independent sample; T and top-k depend on the model  [§3.1]
        completion = sample(prompt, temperature=0.7, top_k=40)
        # Cut the generation where the next "Q:" begins  [app. A.2]
        completion = completion.split("Q:")[0]
        # Task-specific parser after "The answer is "  [§2]
        answer = parse_answer(completion)
        answers.append(answer)
    # Majority vote over answers; reasoning paths are marginalised out  [§2]
    counts = count_occurrences(answers)
    best_answer = argmax(counts)
    # Share of samples agreeing with the vote tracks accuracy  [§3.5] [fig. 5]
    agreement = counts[best_answer] / num_samples
    return best_answer, agreement

def parse_answer(completion):
    # Text after the answer marker  [§2]
    tail = text_after(completion, "The answer is ")
    # Arithmetic: first number; commonsense: the whole string  [§2]
    if task_is_arithmetic:
        return first_number(tail)
    return tail

Hyperparameters

Datasets: test splits where labels exist; CommonsenseQA dev split; StrategyQA question-only set from BIG-bench §3.1.

Results

Author-reported accuracy (%), self-consistency with 40 sampled paths averaged over 10 runs, against greedy CoT ("CoT-prompting") with the same prompts §3.2.

Arithmetic reasoning tab. 2:

Previous-SoTA sources per tab. 2 caption: AddSub/MultiArith from a relevance and LCA operation classifier; ASDiv, AQuA, SVAMP from three separate prior works; GSM8K 35 is GPT-3 175B fine-tuned on 7.5k examples and 55 adds a 175B verifier (Cobbe et al.).

Commonsense and symbolic reasoning tab. 3:

Previous-SoTA sources per tab. 3 caption: CSQA from DeBERTaV3-large + KEAR; StrategyQA from PaLM; ARC from fine-tuned UnifiedQA.

Aggregation strategies on PaLM-540B tab. 1:

Common NLP tasks on PaLM-540B, where CoT can hurt §3.3 tab. 5:

Beam search vs self-consistency on UL2-20B, by beam size / number of paths §3.4 tab. 6:

Prompt ensembles on LaMDA-137B §3.4 tab. 7:

Robustness on GSM8K §3.5 tab. 8:

"Imperfect" prompts replace every number in the exemplar reasoning except the final answer with random numbers §3.5. "Equations" prompts reduce the reasoning to bare equations such as "3 + 2 = 5" §3.5.

Appendix results, all GSM8K on PaLM-540B unless stated:

Other findings, described from figures: accuracy increases with the number of sampled paths (1, 5, 10, 20, 40) on LaMDA-137B and PaLM-540B fig. 2 fig. 7 fig. 8; sample-and-rank (pick the sample with the highest log probability) on code-davinci-001 improves with more samples but far less than self-consistency §3.4 fig. 3; gains hold across temperature, top-k and nucleus settings on PaLM-540B and LaMDA-137B, and across LaMDA model sizes, with smaller gains for smaller models §3.5 fig. 4 fig. 6; the share of samples agreeing with the voted answer correlates with accuracy on GSM8K §3.5 fig. 5.

Paper vs code

No official code.

Pitfalls

• One sample is worse than greedy. With m=1, sampling scores below greedy decoding on UL2-20B (AQuA 19.7 vs 23.6, MultiArith 9.5 vs 10.7) tab. 6. Gains only appear once several samples are voted; do not judge the sampling settings from a single draw.
• The parser carries the method. Only "first number after 'The answer is '" (arithmetic) and "full string after 'The answer is '" (commonsense) are stated §2. Normalising answers before comparing them (e.g. "$18" vs "18", "26" vs "26.0", "(b)" vs "(b) forest", case, trailing periods) is not stated, and neither is how samples without a parsable answer are handled. These choices decide which samples count as agreeing.
• Tie-breaking is not stated. With small m (the paper suggests 5 or 10 to save cost §5) ties are common; pick and document a rule.
• Do not divide by vote counts. The "weighted average" (sum of probabilities divided by the answer's count) collapses to 22.1% on GSM8K with normalised weights, far below greedy tab. 1. If you weight at all, use the length-normalised weighted sum eq. 1; raw sequence probabilities are worse than plain voting tab. 1. Which tokens K covers (e.g. whether the "The answer is" suffix counts) is not stated beyond "the tokens in (\mathbf{r}_i, \mathbf{a}_i)" §2.
• Sampling settings are per model. T=0.5 (UL2, LaMDA) vs T=0.7 (PaLM, GPT-3), with or without top-k=40 §3.1. The paper reports robustness to these settings fig. 4, but the exact grid is only shown in plots.
• Cost scales with m. The headline numbers use 40 full CoT decodes per question §3.2, and the paper names extra computation as the main limitation §5. Max output length is stated only for GPT-3 (128 tokens) app. A.2.
• Fixed answer sets only. Voting needs answers that can be tested for equality; open-ended generation would need a consistency metric between generations, which the paper only proposes as future work §2.
• Short reasoning gives little diversity. Equation-only reasoning paths gain only 1.5 points on GSM8K (5.0 to 6.5) tab. 8, and beam search, being less diverse, is worse than sampling as the path generator tab. 6.
• Small models gain little. UL2-20B gets +3 to +7 points on arithmetic and nothing on 4-letter concatenation (0.0) tab. 2 tab. 3; the paper ties this to abilities that need scale §3.5.
• Reproducibility. LaMDA-137B and PaLM-540B were not public; the paper provides prompts instead of checkpoints (Reproducibility Statement). StrategyQA is the BIG-bench question-only version and CommonsenseQA uses the dev split §3.1, so numbers are not directly comparable to papers using other splits.
• Majority answers can rest on wrong reasoning. The paper notes sampled rationales can be incorrect or nonsensical even when the voted answer is right (e.g. inexact population figures in a StrategyQA example) §5 tab. 4. Do not treat the agreeing rationales as verified explanations.

Lineage

• Builds on: chain-of-thought prompting (arXiv:2201.11903), replacing its greedy decode with sample-and-vote and reusing its prompts §1 §3.1; sampling schemes from open-ended generation (temperature, top-k, nucleus) §2.
• Compared against: greedy CoT on UL2-20B, LaMDA-137B, PaLM-540B and GPT-3 Codex §3.2; sample-and-rank, beam search, prompt-order and multi-prompt ensembles §3.4; multi-model ensembles app. A.1.3; zero-shot CoT (arXiv:2205.11916) as a base prompt §3.5; trained verifiers and re-rankers (Cobbe et al., arXiv:2110.14168; LaMDA) are discussed as heavier alternatives §1 §4.
• Later work in this wiki: Tree of Thoughts uses CoT with self-consistency as a baseline.