Self-consistency

Overview

Greedy decoding of a chain-of-thought (CoT) prompt gives one reasoning path, and a single sampled path is noisy. Self-consistency asks for several sampled paths from the same prompt, reads a final answer from each, and keeps the answer that appears most often 2203.11171 §1 2203.11171 §2. The shared invariant across the variants is the assumption behind the vote. The paper assumes a problem admits several valid reasoning routes to one correct answer, and that correct routes agree on the answer more often than incorrect ones 2203.11171 §2. Voting also needs the answer to come from a fixed set, so two answers can be checked for equality 2203.11171 §2.

No training, verifier or extra model is involved. The method only changes how the answers are generated and combined 2203.11171 §1.

Nineteen cards on wikillm list this concept, and they use it in different ways:

• Defines and measures it: the original paper 2203.11171.
• Tests it under another condition: one measures it on inputs with an irrelevant sentence 2302.00093, one votes over executed programs instead of text chains 2210.02875, and one uses a sampled vote as a text-only baseline on spatial questions about video 2412.14171 §5.2.
• Reports it as a headline number or as a fallback in a larger system: a math-pretrained model that uses it as its main metric 2206.14858, a clinical prompt study with a sampled-vote setting 2212.13138, an agent that switches between a tool loop and a vote 2210.03629, and a medical model whose best inference mode is a vote over five sampled chains 2311.16079 tab. 5.
• Describes it in a survey, without new runs: a survey of prompt-based multi-step reasoning files it under evaluation and under ensemble control 2407.11511 §3.2.1 2407.11511 §3.3.2. Its numbers are copied from the cited papers 2407.11511 tab. 2.
• Uses it as a component or a baseline for a new method: a reward inside tree search 2507.07426, a decoding rule that needs no prompt 2402.10200, a multi-agent consensus 2311.10537, a multi-agent debate that compares against a vote 2305.14325, and a refinement stage that conditions new answers on sampled chains 2305.09617.
• Uses the vote as an evaluation measure, not as an answer to return: a paper that argues for reporting sampled results beside greedy ones, and measures the gain of a vote over k samples on classification, summarisation, reasoning and safety tests 2601.07239 §1 2601.07239 §4.
• Mentions it only in a prompt study: the clinical study that applies it to its baseline prompts without isolating its effect 2308.06834.
• Changes what is voted over, or how the paths are drawn: a vote over only the longest chains 2210.00720, a vote over calls that differ in option order with retrieved examples 2311.16452, and a consensus over features of executed solutions 2505.11792.

The exemplars in the original paper come from few-shot CoT prompting, which has its own page: Chain-of-thought prompting.

Variants

Majority vote over sampled chains

The method as described in the paper. Draw m chains with temperature or nucleus sampling, parse each answer, and return the unweighted plurality 2203.11171 §2. The paper uses m = 40 and averages 10 runs 2203.11171 §3.2. Against greedy CoT with the same prompts on PaLM-540B, GSM8K goes from 56.5% to 74.4% and SVAMP from 79.0% to 86.6% 2203.11171 tab. 2. On code-davinci-002, GSM8K goes from 60.1% to 78.0% 2203.11171 tab. 2. These are author-reported figures. Card: Self-Consistency Improves Chain of Thought Reasoning in Language Models.

Vote over the most complex chains

Complexity-based consistency samples N chains, counts the steps in each (one per line of the chain), and votes only over the K with the most steps. K = N gives ordinary self-consistency. The paper uses N = 50 and says the best K is typically 30 to 40; it shows the K sweep as a curve rather than one value per dataset 2210.00720 §3.2 2210.00720 fig. 6. The same step-count rule also picks the few-shot prompt, and that prompt choice is the paper's main contribution 2210.00720 §3.1. On code-davinci-002 for GSM8K, the handcrafted prompt with a vote gives 74.6%, the complex prompt with a vote gives 82.6%, and the complex prompt with a vote over only the complex chains gives 82.9% 2210.00720 tab. 1. So on the test set, the step-count vote adds 0.3 points on top of the complex prompt. On a validation split, the same vote adds 4.0 points over the complex prompt 2210.00720 tab. 3. The authors report these as their own numbers, and the repository holds prompt text only, so the voting rule cannot be checked against code prompt_complex.txt. Card: Complexity-Based Prompting for Multi-Step Reasoning.

Majority vote over sampled solutions in a math-pretrained model

Minerva continues training PaLM on technical text and reports maj1@k, a vote over k sampled solutions with nucleus sampling at temperature 0.6 and p = 0.95 2206.14858 §2.3 2206.14858 §2.5. The sample count is not fixed across the table: 256 for MATH at 8B and 62B and 64 at 540B, and 100 for GSM8k at 8B and 62B and 40 at 540B 2206.14858 tab. 3. At 540B, GSM8k goes from 58.8% with greedy decoding to 78.5% with maj1@k 2206.14858 tab. 3. The paper finds that voting saturates faster than pass@k, reaching 97% of its large-k accuracy at k = 64 on MATH and k = 16 on GSM8k 2206.14858 §2.5. The paper does not say whether predictions are grouped by normalised string or by symbolic equivalence, or how ties break 2206.14858 §2.5. Card: Solving Quantitative Reasoning Problems with Language Models.

Zero-shot cue and a vote on a finetuned medical model

MEDITRON continues Llama-2 pretraining on medical text, then finetunes one model per benchmark 2311.16079 §4.1 2311.16079 §4.2. Its self-consistency mode adds a zero-shot step-by-step cue to the question and draws five generations at temperature 0.8. It then takes the majority over the extracted options 2311.16079 §4.3. The cue is zero-shot, so this mode uses no exemplars 2311.16079 §4.3. On the 70B model, the average over five result columns rises from 69.7 with a single CoT answer to 72.0 with the vote 2311.16079 tab. 5. The per-column gains run from 0.6 on PubMedQA to 2.9 on MedQA, computed from the same table. For Llama-2-70B under the same finetuning, the vote moves the average from 68.7 to 69.2 2311.16079 tab. 5. The table compares only these two base models, so the size of the effect on other models is not shown. The authors report these as their own numbers. Card: MEDITRON-70B: Scaling Medical Pretraining for Large Language Models.

Probability-weighted vote

Each sample can be weighted by its own probability instead of counting it as one vote. The paper scores a chain by the geometric mean of its token probabilities, which is the exponential of the mean log-probability 2203.11171 eq. 1. Two ways of summing these weights are compared against the plain vote 2203.11171 §2. The normalised weighted sum is nearly as accurate as the plain vote, while the unnormalised sum and the weighted average are clearly worse 2203.11171 tab. 1. Dividing by the vote count collapses accuracy: on GSM8K, the normalised weighted average reaches 22.1%, below greedy decoding at 56.5% 2203.11171 tab. 1. Card: Self-Consistency Improves Chain of Thought Reasoning in Language Models.

Sample-and-rank

A different way to use the same samples. Instead of voting, return the single sample with the highest log-probability. On code-davinci-001 this improves as more samples are drawn, but far less than voting 2203.11171 §3.4 2203.11171 fig. 3. Card: Self-Consistency Improves Chain of Thought Reasoning in Language Models.

Beam search as the path generator

Beam search can replace sampling for producing the candidate chains. Beam search is less diverse, and the paper finds it trails sampling at the same number of paths. On UL2-20B at 40 paths, AQuA reaches 24.2% with beam-generated paths and 26.9% with sampled paths. MultiArith reaches 10.8% with beam and 14.7% with sampling 2203.11171 tab. 6. Use beam only when the decoder cannot sample. Card: Self-Consistency Improves Chain of Thought Reasoning in Language Models.

Combined with prompt-set variation

Self-consistency can also be run with several prompt sets or prompt orderings. On PaLM-540B for GSM8K, sampling with 40 paths gives 74.4%. Adding a different prompt set gives 75.4%, and adding a prompt permutation gives 73.8% 2203.11171 tab. 11. The gain from changing prompts is about one point or less in these runs. A prompt ensemble without sampling is weaker: on LaMDA-137B, 40 prompt permutations with greedy decoding reach 19.2% on GSM8K, against 27.7% for self-consistency 2203.11171 tab. 7. Card: Self-Consistency Improves Chain of Thought Reasoning in Language Models.

Option-shuffled calls with retrieved exemplars (Medprompt)

Medprompt does not vary the prompt once per question. For each test question it retrieves the k most similar items from a pool of solved questions, places them with their stored reasoning chains in the prompt, and makes n calls. Before each call the answer options are shuffled. The chosen position is mapped back to the original option, and the most frequent answer wins 2311.16452 alg. 1 2311.16452 §4.4. The pool keeps only items whose generated chain reaches the gold answer 2311.16452 §4.2. The standard setting is k = 5 and n = 5, and the best MedQA run uses 20 and 11 2311.16452 §4.4 2311.16452 §5.2. With GPT-4 on MedQA, the standard setting reaches 90.2% and the larger one about 90.6%, against 81.4% for 5-shot prompting 2311.16452 tab. 1. In the paper's stepwise ablation, choice shuffling adds 2.2 points over the 5-shot baseline. The order of the steps is the authors' choice, and they do not compute Shapley values 2311.16452 §5.2. The paper does not state a sampling temperature 2311.16452 §4.3. If the decoder were deterministic, the calls would differ only by option order, which makes the order the only source of variation. This is an inference from the method, not a claim by the paper. Card: Medprompt.

Voting over sampled chains with irrelevant context

A distractibility study applies the same vote to GSM8K problems with one unrelated sentence added (GSM-IC). With 20 samples at temperature 0.7, micro accuracy on the 4,000-example evaluation set rises for every prompt tested. For code-davinci-002, CoT goes from 72.4% to 88.1%, least-to-most from 77.5% to 93.4%, and program prompts from 54.4% to 74.6% 2302.00093 tab. 3 2302.00093 §5.1. Macro accuracy, which counts a base problem only if all its variants are right, stays low: least-to-most with voting reaches 45.0%, and the paper gives 45% as the best macro figure for any prompting method it tested 2302.00093 tab. 3 2302.00093 §5.1. The paper also reports that an instruction to ignore irrelevant information helps, on its own and alongside voting 2302.00093 §4.3 2302.00093 §5.1. Card: Large Language Models Can Be Easily Distracted by Irrelevant Context.

Majority vote over executed programs

Binder does not vote over text chains. It samples programs (SQL or Python) that can call the language model for sub-steps, runs each program, and returns the most frequent answer 2210.02875 §2.2 2210.02875 app. A.3. The paper samples 20 programs per question for WikiTQ and 50 for TabFact, and weights votes by program type or by answer 2210.02875 app. A.2 2210.02875 app. A.3. With 14 annotated exemplars and Codex, it reports 64.6% execution accuracy on WikiTQ test and 85.1% on the TabFact small test 2210.02875 §3.2. The card does not report a single-program versus voted-answer comparison, so the vote's own gain is not isolated. The public code's default engine is a chat model with no logprobs, which makes tie-breaking depend on answer order rather than probability 2210.02875 app. A.3. Card: Binding Language Models in Symbolic Languages.

Vote over multiple-choice paths in a clinical prompt set

Flan-PaLM 540B is run with 11 chain-of-thought paths per multiple-choice question, and the final answers are voted 2212.13138 §4.4 2212.13138 §3.3.2. A second setting uses 41 decodes for the deferral study 2212.13138 §4.4. The sampling temperature is not stated; the paper says only that it used temperature sampling 2212.13138 app. A.2. On MedQA, self-consistency raises accuracy from 60.3% (few-shot) to 67.6%, and on MedMCQA from 53.6% (CoT) to 57.6% 2212.13138 tab. 6. On PubMedQA it lowers accuracy, from 79.0% (few-shot) and 77.2% (CoT) to 75.2% 2212.13138 tab. 6. The paper does not give a reason. Card: Large Language Models Encode Clinical Knowledge.

Vote after a refinement stage (ensemble refinement)

Med-PaLM 2 lists self-consistency as one of its prompting options, with 11 CoT samples and a plurality vote 2305.09617 §3.3. Its ensemble refinement (ER) first draws 11 CoT generations. It then samples 33 refined generations, each conditioned on the question and all 11 first-stage generations, and takes a plurality vote over the 33 answers 2305.09617 §3.3 fig. 2. The authors describe ER as a generalisation of self-consistency, because the model can weigh the explanations it has already written 2305.09617 §3.3. On the unified model, MedQA goes from 79.7% (5-shot) to 83.7% (11-sample vote) and 85.4% (ER), and MMLU anatomy from 77.8% to 80.0% and 84.4% 2305.09617 tab. 5. On PubMedQA both voting variants are below 5-shot, at 74.0% and 75.0% against 79.2% 2305.09617 tab. 5, the same direction as the clinical result above. The 86.5% headline MedQA figure comes from a MedQA-only tuned variant, not from the ER run 2305.09617 §3.2 §4.1. ER is applied only to multiple-choice questions, and the sampling temperatures for both stages are not stated 2305.09617 §3.3. Card: Towards Expert-Level Medical Question Answering with Large Language Models.

Vote as a fallback in an agent loop

ReAct interleaves reasoning and tool actions. For question answering, the paper combines it with CoT self-consistency using 21 sampled chains at temperature 0.7 and two switching rules 2210.03629 §3.2. ReAct falls back to CoT-SC after 7 steps (HotpotQA) or 5 steps (FEVER), and CoT-SC falls back to ReAct when the majority answer appears fewer than n/2 times 2210.03629 §3.2. On HotpotQA exact match, CoT-SC scores 33.4, ReAct 27.4, CoT-SC→ReAct 34.2 and ReAct→CoT-SC 35.1 2210.03629 tab. 1. On FEVER accuracy, the figures are 60.4, 60.9, 64.6 and 62.0 2210.03629 tab. 1. The paper also reports that 3 to 5 CoT-SC samples reach the 21-sample result when combined with ReAct 2210.03629 fig. 2. The released notebooks do not include the 21-sample baseline or the switching rules 2210.03629 §3.2. Card: ReAct: Synergizing Reasoning and Acting in Language Models.

Consensus among role-play experts (not sample voting)

MedAgents uses self-consistency only as a baseline. Its own method asks several role-play experts to write analyses, merges them into one report, and revises the report until every expert votes YES. It then picks an option from the agreed report 2311.10537 §2 2311.10537 eq. 4. The baseline is few-shot CoT with 5 sampled chains at temperature 0.7 2311.10537 §3.1. On GPT-4, the average over nine medical datasets is 85.4% for the baseline and 86.7% for MedAgents, a gap of 1.3 points 2311.10537 tab. 2. The authors describe this as comparable, not better 2311.10537 §3.2. Each question takes at least 18 model calls when experts agree at once, by the card's count from the code 2311.10537 §2. This is a consensus rule over experts, not a vote over sampled answers, so it should not be read as a self-consistency result. Card: MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning.

Consensus over features of executed solutions (instance-enhanced)

SIRL asks ten roles to write an optimisation model and solver code for one problem. Each role's program is run, and four features are read from the model file: the objective value, the optimisation direction, the number of binary variables and the number of general integer variables. A role's score is a weighted sum of the square roots of how many roles share each feature value. All four weights are 1 in the implementation, and the highest-scoring answer is kept 2505.11792 §3.1 2505.11792 eq. 2 2505.11792 eq. 3. Compared with a vote over the objective value alone, this rule raises the average over five benchmarks from 42.2% to 45.0% for the 7B model and from 49.1% to 51.2% for the 32B model, at 5 samples 2505.11792 tab. 2. At 10 samples the figures are 44.4% to 46.3% and 51.9% to 53.5% 2505.11792 tab. 2. The paper's pass@1 for 7B is 37.6% and for 32B 43.5% 2505.11792 tab. 2. The main subject of the paper is reinforcement learning with solver-based rewards, and this consensus is one part of its data and evaluation pipeline. Card: Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling.

Debate between instances of one model, compared with a vote

In the debate method, several instances of one chat model answer, read each other's latest answers, and revise over a few rounds 2305.14325 §2.1. The paper reads self-consistency as acting on one instance at a time, and proposes debate as a complement 2305.14325 §1. Its final answer is the consensus the agents reach, and the code takes a majority vote over the agents' final answers for arithmetic and GSM8K 2305.14325 §2.1. With gpt-3.5-turbo, the single agent scores 67.0% on arithmetic and 77.0% on GSM8K, a multi-agent majority vote 69.0% and 81.0%, and debate with 3 agents and 2 rounds 81.8% and 85.0% 2305.14325 tab. 1. The paper's factuality table leaves out the majority vote, because the authors say individual biographies are not easy to compare 2305.14325 tab. 2 2305.14325 §3.2. The debate is not a vote over sampled chains of one model, so its numbers are not a self-consistency result. Card: Improving Factuality and Reasoning in Language Models through Multiagent Debate.

Agreement share as a reward inside tree search

DrugMCTS does not use the vote as a final answer. Inside a Monte Carlo tree search, each rollout's chosen target is asked for k times, and the most frequent answer p* sets the relative reward: the share of picks equal to p* 2507.07426 §2.4 2507.07426 eq. 4. The paper uses k = 4 at temperature 0.8 2507.07426 §3.2. The search adds an absolute reward, the share of yes answers to a direct question, and averages the two 2507.07426 eq. 5 2507.07426 eq. 6. The paper credits the self-consistency idea to the original method 2507.07426 §2.4.1. The vote runs over a fixed set of protein identifiers, which fits the equality condition above. With k = 4, the relative term takes only five values, so ties are common 2507.07426 §2.4. Dropping the absolute term costs 1.46 points on DrugBank at top-k and 0.31 on KIBA at top-k, so the agreement term alone comes close to the full reward 2507.07426 tab. 2. These are author-reported ablations on a benchmark of 788 and 626 entries 2507.07426 §3.1. Card: DrugMCTS: a drug repurposing framework combining multi-agent, RAG and Monte Carlo Tree Search.

Branching on the first token, scored by answer confidence

CoT-decoding does not sample. It takes the top-k tokens at the first decoding step (k = 10 in the paper), continues each branch greedily, and keeps the branch whose answer tokens have the widest top-1 versus top-2 probability margin 2402.10200 §2.2 2402.10200 §3. The aggregated variant sums these scores per distinct answer 2402.10200 §2.2 2402.10200 tab. 7. On GSM8K, greedy decoding gives 9.9% for Mistral-7B and 34.8% for PaLM-2 L; the max-path rule gives 25.1% and 63.2% 2402.10200 tab. 7. Sampled self-consistency without a CoT prompt gives 12.9% and 40.6% 2402.10200 tab. 7. CoT-decoding does not beat zero-shot CoT prompting on PaLM-2 L, which gives 75.1% 2402.10200 tab. 7. The paper names the answer-span rule and the output filters as part of the method, and the code is not public 2402.10200 §9 2402.10200 §2.2. Card: Chain-of-Thought Reasoning Without Prompting.

Sampled CoT inside a larger prompt study

A clinical study applies self-consistency to its GPT-3.5 chain-of-thought prompts, to free-response diagnosis questions from a modified MedQA set 2308.06834 p. 4. The sample count and sampling settings are not stated, and the paper does not say whether its four clinical prompts were sampled the same way 2308.06834 p. 4. The study has no greedy-versus-vote pair, so it gives no measure of the method's gain. Its GPT-3.5 chain-of-thought baseline is 46% 2308.06834 p. 3, and the GPT-4 runs do not use self-consistency 2308.06834 p. 4. Card: Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine.

Sampled vote as a text-only baseline on spatial video

VSI-Bench asks video models about the layout of indoor rooms, such as object counts, distances and routes 2412.14171 §3.1. One of its text-only reasoning baselines is a majority over five CoT runs, sampled at a temperature the paper gives as 0.7 in the appendix 2412.14171 app. B.3. The paper reports that this vote lowers the average score relative to no prompting, by about 1.1%, and gives that figure only in a chart 2412.14171 fig. 8. The paper's cognitive-map prompt is not a vote. It asks the model to place objects on a 10x10 grid before answering, and it raises relative-distance accuracy for Gemini-1.5 Pro from 46.0 to 56.0 2412.14171 tab. 3. Card: Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces.

Best-of-k vote as an evaluation measure, next to greedy

This variant keeps the vote but changes its purpose. The vote is not returned to a user; it is scored against gold labels beside the greedy result, to show how much a single greedy run hides 2601.07239 §1. Each input gets k sampled labels at temperature 0.7 and top-p 0.9 in the ICL and reasoning sections, and the paper calls the majority over k labels "best-of-k" 2601.07239 §4. The authors define the exploration gain as the best-of-k accuracy minus the greedy accuracy, and they tie the vote to a binomial tail: if one draw is right with probability p, a majority of k draws is right with a probability fixed by p and k 2601.07239 §4. The paper does not state its tie rule for even k 2601.07239 §4. On few-shot sentiment and sarcasm tests with 4 or 8 demonstrations, the authors report a typical gain of 10 to 20 points at k = 16 and a further 2 to 5 points at k = 64 2601.07239 §4. The paper names one example that moves LLaMA-3 8B from about 55% to about 75% 2601.07239 §4. The ICL accuracy-against-k figure is a placeholder in the source, so the curve itself is not reported 2601.07239 §4.

On GLUE-style paraphrase and perturbation tests, the same kind of vote over K = 10 labels is the stochastic arm of a robustness ratio. The ratio divides the mean accuracy on three perturbed splits by the accuracy on the original split 2601.07239 §3. The paper's Claude caption gives stochastic ratios of 0.85 to 0.91 and deterministic ratios of 0.79 to 0.82 across four tasks 2601.07239 §3. The per-model numbers for the other models are in a heatmap image and are not transcribed here 2601.07239 §3. Card: Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition.

When to use which

Implementation sketch

Python-flavoured pseudocode of the common core, with notes on where the variants differ.

# Self-consistency over sampled chain-of-thought answers  [§2] [fig. 1]
# Input:  question string; cot_prompt (few-shot exemplars, as text)
# Output: most common final answer and its vote share

def self_consistency(question, cot_prompt, m=40):
    # Exemplars first, then the test question  [§2]
    prompt = cot_prompt + "Q: " + question + "\nA:"
    parsed = []
    for _ in range(m):
        # Independent sample; temperature and top-k depend on the model  [§3.1]
        chain = sample(prompt, temperature=0.7, top_k=40)
        # Cut the output where the next question starts  [app. A.2]
        chain = chain.split("Q:")[0]
        # Task parser: first number or full string after the answer marker  [§2]
        # Unparsable samples: handling not stated in the paper
        parsed.append(parse_answer(chain))
    # Unweighted plurality vote  [§2] [§3.2]; tie-breaking not stated
    best, votes = most_common(parsed)
    return best, votes / m

Where variants differ:

• Probability-weighted vote replaces the count in most_common with a length-normalised chain probability 2203.11171 eq. 1. Do not divide by the number of votes 2203.11171 tab. 1.
• Sample-and-rank returns the sample with the highest log-probability instead of voting 2203.11171 §3.4.
• Beam search replaces sample(...) with beam decoding. This gives less diverse paths 2203.11171 tab. 6.
• Irrelevant-context robustness keeps the same loop, and adds an instruction before the exemplars 2302.00093 §4.3 2302.00093 §5.
• Math-pretrained maj1@k keeps the loop, but the sample count m is set per dataset and model size, and the grouping rule for equal answers is not stated 2206.14858 tab. 3 2206.14858 §2.5.
• Program voting replaces sample(...) with sampling of a program, executes it, and votes over the executed answers 2210.02875 app. A.3. Votes may be weighted by program type or answer.
• Agent fallback runs the agent loop first and switches to the vote when the step limit is reached, or the reverse. The switch rule is the variant 2210.03629 §3.2.
• Zero-shot cue on a finetuned model keeps the loop, but replaces few-shot exemplars with a step-by-step cue appended to the question, and draws five samples at temperature 0.8 2311.16079 §4.3.
• Ensemble refinement keeps the first loop as stage-1 samples. A second loop of 33 calls gets the question and all stage-1 generations in its prompt, and the vote runs over the stage-2 answers 2305.09617 §3.3 fig. 2. Whether stage-1 answers enter the vote is not stated.
• Debate replaces the single sample(...) loop with rounds in which each agent sees the other agents' previous answers. The vote is taken over the final answers only 2305.14325 §2.1.
• CoT-decoding replaces sample(...) with a greedy continuation from each of the top-k first tokens, and replaces the count with the answer-span margin 2402.10200 §2.2. The answer-span rule decides which tokens are scored, so it must be stated in any reimplementation 2402.10200 §9.
• Complexity-based consistency keeps the K chains with the most steps before the vote, where one step is one line of the chain 2210.00720 §3.2. Setting K = N gives the ordinary vote. The paper does not say how ties in step count are broken 2210.00720 §3.1.
• Medprompt replaces sample(...) with a call that has shuffled options, and maps the chosen position back to the original option before the vote 2311.16452 alg. 1. The paper does not state a tie rule 2311.16452 alg. 1.
• Instance-enhanced consensus replaces the answer string with a score per role. The score is a weighted sum of square roots of consensus counts over the objective, the direction and two variable counts 2505.11792 eq. 3.
• Evaluation measure keeps the loop and the vote, but scores the vote against gold labels next to the greedy result on the same items, rather than returning it 2601.07239 §4. Its stochastic arm uses a fixed temperature and top-p in the ICL and reasoning sections 2601.07239 §4 §5.
• The paper does not state the answer-normalisation rule (for example, "18" vs "$18"), so the parser and any normalisation step decide which samples agree 2203.11171 §2.

Contradictions

No direct conflict on the same measure. The papers do not report the same comparison, and the numbers should not be set side by side without that caveat:

• The clinical study has no greedy-versus-vote pair for self-consistency, so its 46% baseline cannot be set against the arithmetic gains above 2308.06834 p. 3 2203.11171 tab. 2. Its tasks are free-response diagnoses, and the arithmetic results should not be read as applying to them.
• CoT-decoding reports self-consistency without a CoT prompt (12.9% on Mistral-7B, 40.6% on PaLM-2 L), while the original paper reports self-consistency with a CoT prompt. These are different baselines 2402.10200 tab. 7 2203.11171 §2.
• Sample counts differ across these tables. MedAgents uses 5 sampled chains for its baseline 2311.10537 §3.1, the original paper uses 40 2203.11171 §3.2, Minerva uses 16 to 256 depending on dataset and size 2206.14858 tab. 3, and Med-PaLM uses 11 for multiple choice 2212.13138 §4.4. Med-PaLM 2 also uses 11 for its vote, and 11 plus 33 for ER 2305.09617 §3.3. Complexity-based consistency uses 50 chains 2210.00720 §3.2, Medprompt uses 5 calls in its standard setting 2311.16452 §4.4, and the instance-enhanced consensus uses 10 roles 2505.11792 §3.1. MEDITRON uses 5 draws 2311.16079 §4.3. The survey does not state a count 2407.11511 §3.2.1. Their gains are not comparable as measured.
• Med-PaLM 2 reports 81.8% on PubMedQA with 11-sample self-consistency in its best-per-row table 2305.09617 tab. 4, but 74.0% for the same vote on its unified model 2305.09617 tab. 5. The best-per-row table mixes model variants and prompts, and the paper's PubMedQA prompt was tuned on the development set 2305.09617 tab. 4 app. A.3.2, so the two figures are not like for like.
• The ReAct paper gives its own CoT-SC HotpotQA score as 33.4 2210.03629 tab. 1, but its footnote quotes 33.8 for the same method as reported by Wang et al. 2210.03629 tab. 1. The paper does not explain the gap.
• The distractibility paper's text and table disagree. Its text says self-consistency raises micro accuracy by "more than 11 percentage points" 2302.00093 §5.1, but the table gives gains of 15.7 for CoT and 15.9 for least-to-most 2302.00093 tab. 3. The card's computation from the table is the reference used here; the paper's stated figure cannot be reproduced 2302.00093 tab. 3.
• The same model and benchmark give different handcrafted-prompt vote results across two papers. Code-davinci-002 on GSM8K reaches 78.0% with self-consistency in the original paper 2203.11171 tab. 2, and 74.6% with a handcrafted prompt and a vote in the complexity paper 2210.00720 tab. 1. The prompts and sampling settings differ, and the two papers do not report the same setup, so neither figure is a correction of the other.
• The survey disagrees with its own table on the size of the gain. Its text gives 10 to 20 points for self-consistency over chain-of-thought 2407.11511 §3.2.1, while its Table 2 gives +4 to +18 points across five benchmarks 2407.11511 tab. 2. The survey does not explain the gap.
• The VSI-Bench paper gives the temperature for its self-consistency runs as 1.0 in the main text 2412.14171 §5.2 and 0.7 in the appendix 2412.14171 app. B.3. The card cannot settle which one the reported runs used.
• The evaluation paper's reasoning and safety tables are labelled as simulations, not measurements 2601.07239 §5 §6. Its text says multi-sample decoding recovers 5 to 15 points of CoT accuracy 2601.07239 §5, and its reasoning table gives gains of +0.08 to +0.15 for the same three benchmarks 2601.07239 §5. Neither is a measured run, so they should not be set against the measured gains above.

Papers

• Self-Consistency Improves Chain of Thought Reasoning in Language Models: samples many chain-of-thought paths and returns the plurality answer, adding 3 to 24 points over greedy CoT on arithmetic tasks across the models tested, at 40 paths.
• Solving Quantitative Reasoning Problems with Language Models: Minerva, a PaLM model trained further on technical text, reports majority voting over sampled solutions (maj1@k) as its headline metric on maths benchmarks.
• Binding Language Models in Symbolic Languages: samples programs that call the language model for sub-steps, runs them, and votes over the answers; training-free, with 14 exemplars.
• ReAct: Synergizing Reasoning and Acting in Language Models: combines a tool-using agent loop with CoT self-consistency and switching rules in both directions for question answering.
• Large Language Models Encode Clinical Knowledge: votes over 11 chain-of-thought paths for Flan-PaLM 540B on multiple-choice medical questions, with mixed results across datasets.
• Large Language Models Can Be Easily Distracted by Irrelevant Context: measures self-consistency on GSM8K problems with one irrelevant sentence added, where voting gives the largest gain but macro accuracy stays at 45% or below.
• Towards Expert-Level Medical Question Answering with Large Language Models: uses an 11-sample vote and ensemble refinement, which conditions 33 answers on the first 11 samples, on medical multiple-choice questions, where ER beats the 11-sample vote on MedQA.
• Improving Factuality and Reasoning in Language Models through Multiagent Debate: several instances of one model debate over rounds; compared against a majority vote over agents' final answers, and against single answers.
• Chain-of-Thought Reasoning Without Prompting: uses self-consistency without a CoT prompt as a baseline, and proposes branching on the first token with an answer-confidence score as an alternative.
• MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning: uses few-shot CoT with self-consistency as its strongest baseline; its own method is a multi-agent consensus over a summary report.
• Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine: applies self-consistency only to GPT-3.5 chain-of-thought baselines in a clinical prompt study, with no isolated effect reported.
• DrugMCTS: a drug repurposing framework combining multi-agent, RAG and Monte Carlo Tree Search: uses the agreement share of k sampled answers as the relative reward in a tree search over drug-target candidates, with a yes/no check added.
• Complexity-Based Prompting for Multi-Step Reasoning: votes only over the K most complex of N sampled chains, and picks few-shot prompts by step count; author-reported gains on math word problems with GPT-3 and Codex.
• Medprompt: a GPT-4 prompting recipe for medicine, which votes over option-shuffled GPT-4 calls that use retrieved exemplars and filtered, self-written rationales; author-reported on MedQA.
• Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling: scores ten roles' executed solutions by consensus over four solver features, compared with a vote over the objective value alone; the main subject is RL with solver-based rewards.
• Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces: uses a five-run CoT vote as a text-only baseline on the VSI-Bench spatial video benchmark, where it does not help on average.
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models: continued pretraining of Llama-2 on medical text, then per-benchmark finetuning; its best inference mode is a five-sample vote with a zero-shot step-by-step cue, author-reported.
• Multi-Step Reasoning with Large Language Models, a Survey: a survey that places self-consistency among evaluation and ensemble methods for prompt-based reasoning; it reports the cited papers' numbers and runs no experiments.
• Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition: argues for reporting sampled results beside greedy ones; uses majority votes over k samples as the stochastic arm in four stress tests, several of whose tables are labelled simulated.