Decoding strategies

Overview

At each step a language model gives a probability to every token in its vocabulary. A decoding strategy is the rule that turns those probabilities into the text that is written. The simplest rule is greedy decoding: take the most likely token at every step 2402.10200 §2.2. Other rules sample from the distribution or search over several continuations. The paper Chain-of-Thought Reasoning Without Prompting compares such rules as baselines 2402.10200 tab. 4. A second paper, DoctorGLM, tests temperature and top-p sampling on a fine-tuned chat model 2304.01097 §4.2. A third, INSPIRED, samples whole dialogue replies with temperature, top-k and top-p and then picks one 2009.14306 §6.2 2009.14306 app. D.

The idea the first paper adds is that the first decoding step matters most. Only that step branches. Every later step is greedy, and the paths are then ranked by how confident the model is in the answer 2402.10200 §2.2 2402.10200 §3. The authors also tried branching at later steps and found that early branching gives more diverse paths, while later branches are strongly shaped by the tokens before them 2402.10200 §2.2 2402.10200 fig. 2.

A further paper, Solving Quantitative Reasoning Problems with Language Models, uses greedy decoding for single answers and nucleus sampling for majority voting over many answers 2206.14858 §2.3. Its headline maths numbers come from the voting, not from a single answer 2206.14858 §2.5.

A fourth paper, Multi-Decoder Attention Model, uses the word "decoder" in another sense. Its decoders build a vehicle-routing tour one stop at a time, and its beam search keeps partial tours instead of tokens 2012.10638 §3.1.2. The same ideas (greedy, sampling, beam) apply, but the scores are costs of partial routes, not token probabilities.

Three more cards add rules. SimCTG proposes contrastive search, which picks each next token from the top-k candidates by probability minus a penalty for similarity to the context 2202.06417 §3.2. Its name is close to contrastive decoding, below, but the two methods are unrelated. BRIO is mainly a training method, but its summarisers are decoded with length-normalised beam search, and one of its two variants ranks candidate summaries 2203.16804 §3 2203.16804 tab. 5. The controllable text survey files guided decoding under post-processing, where a control signal reshapes the next-token distribution while the language model's weights stay fixed 2201.05337 §3.4.

Four further cards change what the scores are or how candidates are chosen. Visual Contrastive Decoding contrasts the next-token logits from the real image with those from a noised copy, so a token wins only if the clean image supports it more 2311.16922 §3.3.1. Self-RAG keeps a segment-level beam in which each candidate is scored by reflection tokens the model emits itself 2310.11511 §3.3. CycleVLA applies minimum Bayes risk (MBR) selection to sampled robot action chunks, not to tokens 2601.02295 §4.2. The survey defines the basic rules in one place 2402.06196 §3.9, and the reproducibility study shows that greedy outputs can change with the hardware they run on 2506.09501 §3.2.

Two more cards change which candidates survive or how the loop runs. MoRT filters the candidates left by top-k and top-p sampling, scoring each one with a direction found in BERT's sentence embeddings 2103.11790 § Guiding Generative Language Models using the MoralDirection. SpecInfer keeps the sampling rule and changes the loop instead: a small draft model proposes a token tree, and the large model checks all of its nodes in one pass 2305.09781 §4.2.

A study of temperature, The Effect of Sampling Temperature on Problem Solving in Large Language Models, sweeps temperature from 0.0 to 1.0 on multiple-choice questions and finds no significant change in mean accuracy over that range 2402.05201 §3.1. It recommends 0.0 for problem solving 2402.05201 §4.1. The opposite view appears in Stochastic CHAOS, which argues that one greedy run understates what a model can do, and reports that majority votes over sampled outputs score higher 2601.07239 §2 2601.07239 §3. The two are set side by side under Contradictions.

Three further cards are less about accuracy. The hallucination survey lists decoding-time methods that contrast or edit internal states, and as a survey it runs no experiments of its own 2401.01313 §3.1 2401.01313 tab. 1. The ad headline paper uses greedy decoding only as a training baseline and decodes with length-normalised beam search 2607.06818 §3.3 2607.06818 §3.4. The energy benchmark fixes temperature 0.8 and top-p 0.95 and does not compare decoding rules 2310.03003 §3.1.

Variants

Greedy decoding

One path, and the most likely token at each step. It is the reference in every comparison in the paper, and its compute cost is O(1) against O(k) for the multi-path methods 2402.10200 tab. 7. On GSM8K it reaches 9.9% on pre-trained Mistral-7B 2402.10200 tab. 4 and 34.8% on PaLM-2 L 2402.10200 tab. 7.

Greedy is not stable across hardware. Under BF16, the same greedy run changes with GPU count, GPU type and batch size, because the order of floating-point sums changes. On DeepSeek-R1-Distill-Qwen-7B, AIME'24 accuracy has a standard deviation of 9.15% across 12 configurations, and FP32 gives 0 2506.09501 tab. 3. Output length varies by up to 9,189 tokens under BF16 2506.09501 tab. 4. The authors report that over 90% of MATH500 examples diverge under BF16 on that model 2506.09501 §3.2. The greedy numbers elsewhere on this page are single reported values. This paper suggests that a single greedy score can sit anywhere in a spread of this size, depending on the configuration that produced it.

For multiple-choice problems, the temperature study also recommends 0.0, on the same ground of reproducibility and with no measured accuracy cost 2402.05201 §4.1. The sweep is described below.

Sampling baselines

The paper's GSM8K comparison on pre-trained Mistral-7B uses four sampling and search rules 2402.10200 tab. 4:

• Top-k sampling (k = 10): 4.9%.
• Nucleus sampling (p = 0.9): 6.4%.
• Beam search (b = 10): 6.7%.
• Temperature sampling (T = 0.7): 7.5%.

In this one setup all four score below greedy decoding (9.9%). The card does not describe how each rule works; the paper uses them only as baselines.

Temperature and top-p sampling

Temperature rescales the next-token distribution, and top-p (nucleus) keeps the smallest set of most likely tokens whose probability reaches the threshold, then samples from that set 2304.01097 §4.2. The paper does not state which of the two is applied first 2304.01097 §4.2. The survey writes the temperature softmax as the logits divided by T before the softmax 2402.06196 eq. 3, and leaves the top-p value unstated 2402.06196 §3.9.

The paper grades a grid of temperature 0.90, 0.95 and 0.99 against top-p 0.6, 0.7 and 0.8. A doctor gave each cell one letter grade, A (best) to F (worst), for one child-fever question 2304.01097 tab. 6. Three cells got A: (0.95, 0.7), (0.95, 0.8) and (0.99, 0.6). Two got F: (0.90, 0.6) and (0.95, 0.6). With one question and one grader, the grid shows a direction, not a validated setting. The repository's notebook uses top-p 0.7 and temperature 0.99 chat_lora.ipynb, and the paper grades that pair B 2304.01097 tab. 6.

Temperature sweep on multiple-choice problems

Renze and Guven answer each question ten times at each temperature from 0.0 to 1.0 in steps of 0.1, with one worked example and a chain-of-thought prompt for GPT-3.5 2402.05201 §2.1 2402.05201 §2.3. Accuracy is the share of the ten answers that match the gold choice, averaged over questions 2402.05201 §2.4.

For GPT-3.5 on the 1,000-question exam, the test finds no significant effect across 0.0 to 1.0 (H(10) = 10.439, p = 0.403) 2402.05201 §3.1. On the 100-question exam none of the nine models is significant. The lowest p-value is 0.072, for Llama 2 7B, whose answers were badly formatted 39% of the time 2402.05201 tab. 3 2402.05201 §3.1. Two of the ten exams are significant, LSAT-AR and SAT-Math, and the authors attribute this to the all-or-nothing answers at 0.0 2402.05201 tab. 5 2402.05201 §3.1. Above 1.0 the text turns incoherent, and GPT-3.5 accuracy falls to about zero near 1.6 2402.05201 §3.1 fig. 2.

The authors recommend 0.0 for problem-solving tasks, citing reproducibility and no measured accuracy cost 2402.05201 §4.1. The study covers only multiple-choice questions, and only one configuration was tested above 1.0 2402.05201 §4.2. Tokens per response rise with temperature, which adds cost above 1.0 2402.05201 §4.4. Greedy output can still change with hardware under BF16 2506.09501 tab. 3, so temperature 0.0 does not by itself guarantee repeatable runs.

Stochastic evaluation with a reported spread

Stochastic CHAOS runs one protocol on four stress tests. It compares greedy decoding (temperature 0) with sampling at temperature 0.7 and top-p 0.9, using temperature 0.7 or 1.0 for the safety runs, with k samples per input and a majority vote over them where a metric needs one 2601.07239 §3 2601.07239 §4 2601.07239 §5 2601.07239 §6. The measures are a robustness ratio over paraphrased, perturbed and adversarial GLUE variants, an exploration gain (best-of-k minus greedy accuracy), and a risk gap for attack success between sampled and greedy decoding 2601.07239 §3 2601.07239 §4 2601.07239 §6.

The reported values are author-reported. In a figure caption, the stochastic robustness ratio for Claude is 0.85 to 0.91 across MNLI, QQP, QNLI and SST-2, against 0.79 to 0.82 for greedy decoding 2601.07239 §3. On in-context learning with BESSTIE, greedy accuracy for many model-task pairs lies between 40% and 65%. Best-of-16 typically adds 10 to 20 points, and best-of-64 adds 2 to 5 more 2601.07239 §4. The accuracy-versus-k figure is a placeholder in the source 2601.07239 §4. The chain-of-thought and safety tables are labelled simulated, so use them as a template for reporting, not as numbers 2601.07239 §5 2601.07239 §6.

Each stochastic measure needs k generations per input, and the reasoning and safety sections use up to 32 samples 2601.07239 §5 2601.07239 §6.

Sample, filter and pick (dialogue replies)

INSPIRED draws each recommender reply by sampling tokens with temperature 0.8, then keeps the 400 most likely tokens, then keeps the top-p 0.9 nucleus 2009.14306 app. D interactive.py. The paper gives 0.82 for the baseline model and 0.8 for the strategy model 2009.14306 app. D. The sampling stops at a turn-end token or after 200 tokens interactive.py.

The paper draws five candidate replies and keeps those that contain encouragement or are longest, then picks one at random 2009.14306 §6.2. Candidates that repeat a bigram are removed first interactive.py. The released script draws three candidates and picks uniformly among those that pass the bigram filter. Its length rule is disabled interactive.py interactive.py.

The reported comparison is between two models, not between decoding settings. Author-reported: the strategy model has test perplexity 8.93 against 9.28 and BLEU-4 6.63 against 5.11 for the baseline 2009.14306 tab. 7. The card records no test of the decoding rule itself.

Self-consistency without a CoT prompt

The paper also runs self-consistency with 10 paths, with and without a chain-of-thought prompt 2402.10200 tab. 3 2402.10200 tab. 7. On pre-trained Mistral-7B it reaches 12.9% on GSM8K, and on PaLM-2 L 40.6% 2402.10200 tab. 7. It costs O(k) 2402.10200 tab. 7.

Majority voting over samples (maj1@k)

Draw k solutions with nucleus sampling at temperature 0.6 and p = 0.95, then return the most common final answer 2206.14858 §2.3 2206.14858 §2.5. The answer is read from a fixed sentence in the output, and only that answer is compared, not the reasoning 2206.14858 app. D.1. This is the same family as the self-consistency row above, applied to a model trained on technical text.

Author-reported, Minerva 540B, MATH: greedy single answer 33.6%, maj1@k 50.3% (k = 64) 2206.14858 tab. 3. On GSM8k the same model goes from 58.8% to 78.5% (k = 40) 2206.14858 tab. 3. The authors say 97% of the large-k accuracy on MATH is reached at k = 64 2206.14858 §2.5.

Sampling is not free on single answers. On the 62B model, pass@1 is 26.5% with greedy decoding and 21.8% at T = 0.6 2206.14858 tab. 8. Majority voting recovers this loss only because it aggregates over many samples. Log-likelihood reranking of the same samples does much worse on MATH with 62B: 23.8% against 42.0% for majority voting 2206.14858 tab. 8.

Majority voting also needs a grouping rule and a tie-break. The paper says predictions are grouped by final answer but does not say whether grouping uses the normalized string or SymPy equivalence, and tie-breaking is not stated 2206.14858 §2.5. An implementation has to choose both.

CoT-decoding

The method in 2402.10200 §2.2:

1. Take the top-k tokens at the first decoding step. The default is k = 10 2402.10200 §3.
2. Continue each branch greedily.
3. Score each path by the mean over its answer tokens of the top-1 probability minus the top-2 probability at that step.
4. Return the answer of the highest-scoring path (the "max" rule), or sum the scores of paths that give the same answer and return the largest total (the "agg" rule) 2402.10200 tab. 7.

The prompt is only the bare "Q: [question] / A:" format. No chain-of-thought prompt and no tuning are used 2402.10200 §2.1.

Reported gains, author-reported:

• Pre-trained Mistral-7B, GSM8K: 9.9% greedy to 25.1% with CoT-decoding 2402.10200 tab. 4 2402.10200 tab. 7. The agg rule gives 25.3% 2402.10200 tab. 7.
• PaLM-2 L, GSM8K: 34.8% greedy to 63.2% (max rule) or 64.1% (agg rule) 2402.10200 tab. 7.
• Instruction-tuned Mistral-7B, GSM8K: 31.2% greedy to 38.2% 2402.10200 tab. 5.
• Combined with zero-shot CoT prompting, the agg rule gives 48.4% on Mistral-7B and 87.0% on PaLM-2 L 2402.10200 tab. 7.

Beam search over routing tours

In the routing paper, a learned decoder adds one node to a partial tour per step. Each of the five decoders has its own beam, so beams do not mix 2012.10638 §3.1.2 2012.10638 §4.1. When two partial tours share the same remaining subproblem and one is not longer and has at least as much vehicle capacity left, the worse one is merged away and the survivor keeps the larger probability 2012.10638 §3.1.2 2012.10638 fig. 2.

Author-reported on CVRP100 2012.10638 tab. 1:

• Greedy decoding of the same model: gap 4.86%, 45 s for 10,000 instances.
• Beam 30 per decoder: gap 2.49%, 31 min.
• Beam 50 per decoder: gap 2.23%, 53 min.
• Attention Model with 1,280 samples: gap 3.72%, 2 h.

The paper also reports that the multi-decoder model with greedy decoding beats a single decoder 2012.10638 tab. 2. Its routing setup differs from token decoding, so the numbers are not comparable with the rows above.

Length-normalised beam search (ad headlines)

The ad headline paper decodes from a BERT-Large masked language model. Its mask lets each headline position see only earlier headline tokens 2607.06818 §3.3 eq. 3. Inference uses beam search, which stops at a maximum length or when every beam emits the end token [arxiv:2607.06818 §3.4]. The beam ranks candidates by a length-normalised word score rather than plain log-likelihood, scaling each word's log-likelihood by a factor that depends on its position 2607.06818 §3.4 eq. 10. The normalisation coefficient, the beam width and the maximum length are not stated 2607.06818 §3.4.

The ablation removes beam normalisation and post-processing together. ROUGE-L falls from 6.33 to 5.37, in the paper's table of differences from its bi-LSTM baseline 2607.06818 tab. 1. Because both steps change at once, the ablation cannot isolate normalisation.

Decoding also appears in training. One sampled headline and one greedy headline are both scored with ROUGE-L F1 against approved headlines. The sampled headline's probability goes up when it scores above the greedy one, and down when below 2607.06818 §3.3 eq. 9 2607.06818 §4.1. The reward is the metric the table reports, so part of the gain may come from optimising that metric 2607.06818 §4.1 2607.06818 tab. 1.

Contrastive search for repetition (SimCTG)

SimCTG pairs a training loss with a decoder, and the decoder is contrastive search. At each step it takes the top-k next tokens, scores each one by (1 − α) times its probability minus α times the largest cosine similarity between its last-layer state and the states of all earlier tokens, and emits the best one 2202.06417 §3.2 eq. 5. With α = 0 it is greedy search 2202.06417 §3.2. The paper says k is typically 3 to 10 2202.06417 §3.2. The document setting uses k = 8 and α = 0.6 2202.06417 §4 inference.sh. The dialogue setting uses k = 5 and α = 0.6 2202.06417 §5. The appendix recommends α from 0.5 to 0.8 and k from 5 to 10 2202.06417 app. E.

Author-reported on WikiText-103 with GPT-2 small 2202.06417 tab. 1. With the SimCTG model, contrastive search has a 2-gram repeat rate of 3.93% against 4.05% for nucleus sampling, and a higher MAUVE (0.94 against 0.92) and coherence (0.610 against 0.584). With the MLE model instead of SimCTG, the same decoder repeats 44.20% of 2-grams and has MAUVE 0.18 2202.06417 tab. 1. On 200 prefixes rated by five graders, SimCTG with contrastive search scores higher than nucleus sampling on coherence (3.25 against 2.96) and fluency (3.57 against 3.34), and the paper marks both as significant. On informativeness the two tie at 3.96 2202.06417 tab. 2.

Cost: each step scores k candidates, and the paper's latency comparison, at batch size 1, finds contrastive search and beam search nearly equal when k and the beam width are both at most 6, with contrastive search faster above that 2202.06417 §6.4 fig. 5. The penalty compares each candidate with every earlier token, so the states of the whole output must be kept 2202.06417 §3.2.

Contrastive decoding for vision-language models

Visual Contrastive Decoding targets object hallucination in large vision-language models, with no training. For each token, the model produces logits for the real image and for a copy with Gaussian noise added. The sampled logits are (1 + α) times the first minus α times the second, with α = 1 2311.16922 §4.1 eq. 3. Only tokens whose probability under the clean image is at least β times the top probability are kept, with β = 0.1 2311.16922 §3.3.2 eq. 4 §4.1. Setting α = 0 gives ordinary decoding 2311.16922 §3.3.1.

Author-reported on POPE, mean over 5 runs 2311.16922 tab. 1: for Qwen-VL on GQA with popular negatives, accuracy goes from 75.99 to 81.83 and F1 from 74.84 to 82.23, the paper's headline gain 2311.16922 §4.2. On the MME hallucination subset, LLaVA-1.5 goes from 565.33 to 604.66 2311.16922 tab. 2. Regular decoding still wins on the position items for Qwen-VL and InstructBLIP 2311.16922 tab. 2.

The noise has a cost. Each generated token needs a second forward pass on the noised image 2311.16922 §3.3.1, and the paper gives no timing or memory figures. The released code uses a one-shot noise schedule instead of the step-wise rule in eq. 2 vcd_add_noise.py, and the released code only samples, so beam search with this method is not implemented vcd_sample.py.

Context and layer contrast (hallucination survey)

The hallucination survey files three methods under decoding 2401.01313 §3.1. CAD contrasts the output distribution with and without the context, which helps when the context conflicts with what the model already knows. DoLa contrasts logits from later and earlier layers, because it treats factual knowledge as stored in certain layers. ITI is filed here too, but it does not choose tokens by a contrast. It selects a sparse set of attention heads and moves activations along truth-related directions at each generated token 2401.01313 §3.1.

The survey's table gives tasks and models, not comparable results 2401.01313 tab. 1. DoLa is listed on LLaMA 7B to 65B and GPT-4, and CAD on OPT, GPT-Neo, LLaMA and FLAN-T5 2401.01313 tab. 1. The primary papers for these methods are not cards on this site, so this page gives none of their numbers.

Segment-level beam with self-critique (Self-RAG)

Self-RAG trains one language model to emit reflection tokens. These cover whether to retrieve, whether a passage is relevant, whether a segment is supported by it, and how useful the whole answer is 2310.11511 §3.1 tab. 1. At inference, for each segment the model first predicts a retrieve token. If retrieval is needed, it writes one continuation per retrieved passage and keeps the best B by LM probability plus a weighted critique score 2310.11511 §3.3 eq. 3 eq. 4 alg. 1. The paper uses B = 2, greedy token decoding, and critique weights of 1.0, 1.0 and 0.5 for relevance, support and usefulness 2310.11511 §4.3. The retrieval threshold is 0.2 for most tasks 2310.11511 §4.3. Hard constraints can also drop candidates that contain an undesired critique token 2310.11511 §3.3.

Author-reported, 13B 2310.11511 tab. 2: PubHealth 74.5 against 70.1 for ChatGPT, and PopQA 55.8 against 50.8 for ChatGPT with retrieval. An ablation at 7B with 50k training instances gives ASQA exact match 32.1 for the full model and 18.1 without the critic 2310.11511 §5.2 fig. 3.

The method needs a model trained with reflection tokens. Those labels come from GPT-4 prompts, 4k to 20k per token type 2310.11511 §3.2.1 app. A.1. Cost grows with passages times beam width 2310.11511 §3.3.

MBR selection over sampled action chunks (CycleVLA)

This variant is not for token output. A robot policy draws N = 8 action chunks from the same observation, and the chunk with the smallest average L2 distance to the others is run 2601.02295 §4.2 eq. 4 §5.1. Each chunk is turned into a path by adding up its translation and rotation deltas, giving a feature vector of length 6H 2601.02295 §4.2 app. C. The rule is used only after a backtrack to an earlier subtask 2601.02295 §4.2. The policy has to be stochastic for this to work 2601.02295 §3.

Author-reported. Table 4 is estimated offline from recorded rollouts, not from closed-loop runs 2601.02295 §5.3. At the 350K checkpoint with failure correction, N = 8 random draws give 80.3% and MBR gives 90.2%, against 83.2% without correction 2601.02295 tab. 4. L2 is the best of the distance choices tested at every checkpoint 2601.02295 tab. 5. Running MBR at every step reaches 96.9% against 95.3% for the main method, at about 2.2 times the inference time 2601.02295 tab. 6 §5.4. The MBR step itself takes under 0.1% of runtime on the A100 2601.02295 tab. 9.

Length-normalised beam search and candidate ranking (BRIO)

BRIO trains a summariser with an extra ranking loss, and its decoding uses two existing tools. The first is beam search with a length-normalised score: the sum of a candidate's token log-probabilities divided by its length raised to a penalty α 2203.16804 eq. 9. The paper sets α = 2.0 for CNN/DailyMail and 0.6 for XSum, equal to the beam search's own length penalty 2203.16804 tab. 13 app. B. The 16 training candidates per document come from diverse beam search, run offline before training 2203.16804 app. B §3. The paper states the candidate count but not the beam width for them 2203.16804 app. B.

The second is the ranker. BRIO-Ctr is trained with the ranking loss alone, can select among candidates, and cannot generate 2203.16804 §3. BRIO-Mul keeps the cross-entropy term and generates 2203.16804 §5.1. On CNN/DailyMail, BRIO-Ctr scores 47.28 ROUGE-1 against 47.78 for BRIO-Mul 2203.16804 tab. 2.

Beam width matters differently for the two models. Author-reported ROUGE-1 on CNN/DailyMail 2203.16804 tab. 5: BART falls from 44.29 with 4 beams to 42.79 with 100 beams, while BRIO-Mul rises from 47.78 to 48.23. The paper reports this on one dataset and does not test other causes. The default beam width for final generation is 4 2203.16804 tab. 5.

Decoding-time guidance (controllable text survey)

The controllable text survey sorts control methods by where the control signal enters. The post-processing family keeps the language model's weights fixed and reshapes the next-token distribution at decoding time. It covers guided methods that use a separate guide model at inference (PPLM, GeDi, DExperts, FUDGE, COLD and others) and trainable modules that are fitted jointly with the frozen model 2201.05337 §3.4. In GeDi, a class-conditional model's posterior is combined with the language model's distribution by Bayes' rule 2201.05337 eq. 9.

The survey says that post-processing can be slow at inference and can lower fluency. PPLM loses some fluency, and Mix and Match and COLD need many iterations 2201.05337 §3.4 §5.1. The survey runs no experiments of its own, so it gives no comparative numbers for these methods.

Toxicity filter after top-k and top-p (MoRT)

Ordinary top-k and then top-p sampling give a set of candidate tokens. Each candidate is appended to the prefix, and the whole text is scored on the moral direction. Candidates scored at or below a threshold t are removed, but at least m are always kept 2103.11790 § Guiding Generative Language Models using the MoralDirection. The paper uses BERT-large with SBERT pooling for these sentence embeddings 2103.11790 § Guiding Generative Language Models using the MoralDirection. It tests t = -0.5, 0 and 0.5 with m = 5 2103.11790 § Experimental setup. The paper does not state top-p; the released wrapper uses 0.9 generation.py.

Author-reported on RealToxicityPrompts, where toxicity is scored by an external API: the share of 100,000 prompts with a generation above toxicity 0.9 falls from 9.82% for GPT-2 to 1.17% with the filter at t = 0 2103.11790 § Reducing Neural Toxic Degeneration in Language Models. The authors say that such scoring tools are imperfect and biased 2103.11790 § Testbed for evaluating the toxicity of text generations.

The filter has a per-step cost. Each candidate is decoded as a full text and embedded with BERT-large, and the whole prefix is embedded again each time mcm_models.py. The paper reports no runtime or memory figures.

Tree-based speculative verification (SpecInfer)

A speculator, here a small draft model, builds a token tree rather than one continuation. The large model runs once over every node, with a tree attention mask that lets each node see its own ancestors and the verified prefix, not its siblings' branches 2305.09781 §4.2. Greedy verification walks down the tree and takes the child whose token matches the large model's prediction at that node, so its output matches ordinary greedy decoding 2305.09781 §4.3 alg. 2. The paper also gives a multi-step sampling rule, stated as leaving the output distribution equal to the large model's, and points to a separate reference for the proof 2305.09781 §4.3.

Author-reported speed-ups over incremental-decoding serving systems: 1.5-2.5x on one node and 2.4-2.8x on two nodes with eight GPUs 2305.09781 §6.2. Against FlexGen with offloading, OPT-13B and OPT-30B on one A10 go 2.6-3.5x faster per token 2305.09781 §6.3. Multi-step sampling verifies 1.26 to 1.28 times more tokens per stochastic step than naive sampling 2305.09781 tab. 3. The paper estimates that one draft model adds under 1% memory per GPU when it is 100 to 1000 times smaller than the large model 2305.09781 §5.3. For the speculate-then-verify idea itself, see the speculative-decoding concept page.

When to use which

Compute is the main limit for CoT-decoding. Each branch is a full decode, and the paper calls the extra cost a limitation 2402.10200 §5. The paper's experiments use Mistral-7B and Gemma-7B on A100 GPUs and PaLM-2 on TPU v4 2402.10200 §9. PaLM-2 access is not described in the card, so open-weight models are the practical choice for reproduction 2402.10200 §9.

Implementation sketch

Language-agnostic pseudocode of the common cores, from the steps above. Each block is one variant; the paper's answer-span and filter rules are given for its tasks, and a new task must choose its own 2402.10200 §9.

Sampling one token, the core of the temperature and top-k/top-p variants:

# Temperature, top-k, then top-p sampling for one token  [app. D]
# [code: code/interactive.py#L1590-L1591]
# Input:  logits for the next token, temperature T, k, p
# Output: one sampled token id

def sample_token(logits, T=0.8, k=400, p=0.9):
    # Sharpen or flatten the distribution  [app. D]
    logits = logits / T
    # Keep only the k highest-scoring tokens  [app. D]
    logits = top_k_filter(logits, k)
    # Keep the smallest set whose probability reaches p  [app. D]
    logits = top_p_filter(logits, p)
    return sample_from(softmax(logits))

Notes from the dialogue setup: the order of the three steps follows the released code, and the paper gives 0.82 as the baseline temperature app. D interactive.py.

CoT-decoding:

# CoT-decoding  [arxiv:2402.10200 §2.2] [arxiv:2402.10200 §9]
# Input:  model, prompt "Q: [question]\nA:", k (default 10)
# Output: the answer with the highest score

def cot_decoding(model, prompt, k=10):
    # Take the k most likely tokens at the first step  [§2.2] [§3]
    first_tokens = top_k_tokens(model, prompt, k)

    # Continue each branch greedily; 128 steps is the PaLM-2 budget  [§9]
    paths = [greedy_decode(model, prompt, first=t, max_new=128)
             for t in first_tokens]

    # Drop empty, truncated and question-ending responses  [§9]
    paths = [p for p in paths if is_well_formed(p)]

    # Find the answer span in each path  [§2.2] [§9]
    for p in paths:
        p.answer = find_answer_span(p)

    # Mean top-1 minus top-2 probability over the answer tokens  [§2.2]
    for p in paths:
        p.score = mean_margin(model, p, p.answer)

    # Max rule: the best-scoring path's answer  [§2.2] [tab. 7]
    best = max(paths, key=lambda p: p.score)
    return best.answer

Notes from the paper's setup:

• Math tasks take the last number as the answer for Mistral; PaLM-2 appends "So the answer is" and aligns that span with the path 2402.10200 §9.
• Yes/no tasks compare the probability mass on "yes/true" with "no/false" instead of using the margin 2402.10200 §9.
• For Mistral, outputs that look like training data are dropped 2402.10200 §9.

Majority voting over samples, with final-answer comparison:

# maj1@k decoding  [arxiv:2206.14858 §2.5] [arxiv:2206.14858 §2.3]
# Input:  model, problem text, k (samples)
# Output: the most common final answer

def maj1_at_k(model, problem, k):
    # Draw k independent solutions at T = 0.6, nucleus p = 0.95  [§2.3]
    responses = [sample(model, problem, T=0.6, p=0.95) for _ in range(k)]
    # Read the answer from the fixed "Final Answer" sentence  [app. D.1]
    answers = [extract_final_answer(r) for r in responses]
    # Normalize answers before grouping; the paper's normalizer is in
    # app. D.1  [app. D.1]. Grouping rule and tie-breaking are not stated,
    # so the implementer must choose both  [§2.5]
    answers = [normalize(a) for a in answers]
    return most_common(answers)

Beam search over routing tours, one beam per decoder:

# Customised beam search, one beam per decoder  [arxiv:2012.10638 §3.1.2]
# Input:  M decoders; per-decoder beam size B
# Output: the shortest complete tour found by any beam

def beam_search(decoders, B):
    for each decoder m in decoders:
        # Beams stay separate per decoder, so tours stay diverse  [§3.1.2]
        beam = [empty partial tour]
        while some partial tour is unfinished:
            # Extend every partial tour with its valid next nodes
            candidates = expand(beam, decoder_m)
            # Merge candidates that share the remaining subproblem and
            # are inferior  [§3.1.2] [fig. 2]
            candidates = merge_inferior(candidates)
            # Keep the B best by accumulated log-probability  [§3.1.2]
            beam = top_k(candidates, B)
    return shortest(all finished tours)

Notes from the routing setup:

• The paper does not state how many successors each beam entry expands. The released search code keeps 3 per entry, so a 30-wide beam sees at most 90 candidates per step model_search.py. Copy this constant if you want to reproduce the reported beam results.
• Per-decoder beams of size B give M times B partial tours in total; the paper reports bs30 and bs50 as B per decoder 2012.10638 §4.1.

Visual contrastive decoding, one token, as the paper defines it 2311.16922 §3.3 eq. 3 eq. 4:

# One token of visual contrastive decoding  [arxiv:2311.16922 §3.3]
# Input:  image, prompt, prefix, alpha, beta, noised image
# Output: next token id

def vcd_next_token(model, image, noised, prompt, prefix, alpha, beta):
    # Logits from the clean image and from the noised copy  [§3.3.1]
    logits = model.next_logits(image, prompt, prefix)
    logits_noised = model.next_logits(noised, prompt, prefix)
    # Keep tokens that are plausible under the clean image  [§3.3.2] [eq. 4]
    probs = softmax(logits)
    keep = probs >= beta * max(probs)
    # Favour what the clean image supports over the noised one  [eq. 3]
    contrast = (1 + alpha) * logits - alpha * logits_noised
    contrast = where(keep, contrast, -inf)
    return sample(softmax(contrast))

MBR selection over N sampled action chunks, as in the robot paper 2601.02295 §4.2 eq. 4:

# MBR selection over N sampled action chunks  [arxiv:2601.02295 §4.2] [eq. 4]
# Input:  N chunks, each of shape (H, 9)
# Output: the chunk to execute

def mbr_select(chunks):
    # Turn each chunk into a path by summing its translation and rotation
    # deltas, so the feature vector has length 6H  [§4.2] [app. C]
    paths = [to_path(c) for c in chunks]
    # Risk of a candidate: mean L2 distance to all candidates  [eq. 4]
    risk = [mean([l2(p, q) for q in paths]) for p in paths]
    return chunks[argmin(risk)]

Notes from the robot setup: the released code does not use the plain rule by default. It uses a density-based variant run_libero_eval_decomposed_progress_mbr.py. The paper's text describes the plain rule above 2601.02295 app. C, but it does not say which rule produced its tables 2601.02295 §5.3.

Two more pseudocode blocks follow the same pattern. The first is the MoRT filter, which sits after top-k and top-p sampling:

# Toxicity filter after top-k and top-p sampling
# [arxiv:2103.11790 § Guiding Generative Language Models using the MoralDirection]
# Input:  prefix text, LM logits for the next token, moral score, threshold t, minimum m
# Output: one sampled token id

def filtered_sample(lm, prefix, t, m, k=50, p=0.9):
    # Ordinary top-k, then top-p sampling gives the candidate set  [§ Guiding ...]
    candidates = top_p_filter(top_k_filter(lm.next_logits(prefix), k), p)
    # Score each candidate by the whole text, prefix plus candidate  [§ Guiding ...]
    scores = {c: moral_score(prefix + c) for c in candidates}
    # Remove candidates scored at or below t  [§ Guiding ...]
    kept = [c for c in candidates if scores[c] > t]
    # Keep at least m candidates, the best-scoring ones  [§ Guiding ...]
    if len(kept) < m:
        kept = sorted(candidates, key=lambda c: scores[c])[-m:]
    return sample(kept)

The second is greedy verification in SpecInfer, which the large model checks after one tree-parallel pass:

# Greedy verification over a token tree  [arxiv:2305.09781 §4.3] [alg. 2]
# Input:  token tree from the draft model; large-model prediction at each node
# Output: verified tokens, in order

def verify_greedy(tree, llm_pred):
    verified = []
    node = tree.root
    # Walk down while a child's token matches the large model's prediction here  [§4.3]
    child = matching_child(tree, node, llm_pred[node])
    while child is not None:
        verified.append(tree.token[child])
        node = child
        child = matching_child(tree, node, llm_pred[node])
    return verified

Contrastive search, as SimCTG defines it:

# Contrastive search  [arxiv:2202.06417 eq. 5] [§3.2]
# Input:  context tokens, k (candidates), alpha (penalty weight), length L
# Output: L new token ids

def contrastive_search(model, context, k=8, alpha=0.6, L=128):
    for step in range(L):
        # Top-k next tokens and their model probabilities  [§3.2]
        probs, cands = top_k_next_tokens(model, context, k)
        # Last-layer states of the context, reused for every candidate  [§3.2]
        past = model.last_layer_states(context)
        scores = []
        for p, v in zip(probs, cands):
            # Last-layer state of v when appended to the context  [§3.2]
            h_v = model.last_layer_state(context + [v])
            # Penalty: largest cosine to any earlier token state  [eq. 5]
            penalty = max(cosine(h_v, h) for h in past)
            # Probability term minus weighted penalty  [eq. 5]
            scores.append((1 - alpha) * p - alpha * penalty)
        # Emit the best-scoring candidate  [eq. 5]
        context = context + [cands[argmax(scores)]]
    return context

Notes: the released code batches the k candidates with a key-value cache utlisgpt.py. The default k = 8 and α = 0.6 are the document setting §4, not values the paper gives for every task.

Contradictions

Two within-paper gaps in the reproducibility study matter for implementation:

• The reproducibility paper gives 2.2% of FP32 MATH500 examples diverging in its text, and 5.80% in its table for the same model 2506.09501 §3.2 app. C.
• Its random-sampling appendix repeats one model's values in another model's row 2506.09501 app. E, which differs from the main table 2506.09501 tab. 5.

Papers disagree on whether temperature changes accuracy:

• Renze and Guven find no significant change in mean accuracy from temperature 0.0 to 1.0 on multiple-choice questions, and recommend 0.0 2402.05201 §3.1 2402.05201 §4.1. Stochastic CHAOS argues that a single greedy run understates capability and risk. It reports that majority votes over sampled outputs beat greedy decoding on its GLUE, in-context learning and summarisation tests 2601.07239 §2 2601.07239 §3 2601.07239 §4. Minerva reports pass@1 of 26.5% with greedy decoding and 21.8% at temperature 0.6 on the 62B model 2206.14858 tab. 8. The tasks, metrics and models differ, so these sources do not settle which rule is better.

Paper-to-code gaps, each with both sides cited:

• Self-RAG's equation 3 adds the LM probability and the critique score. The long-form runner multiplies each child's score by its parent's, so the score is a product over segments 2310.11511 eq. 3 run_long_form_static.py.
• VCD's equation 2 applies a fixed per-step noise rate. The released code uses a one-shot schedule and never reads the rate 2311.16922 eq. 2 vcd_add_noise.py.
• CycleVLA's equation 4 is the plain average-distance rule, and the released code defaults to a density-based variant 2601.02295 eq. 4 run_libero_eval_decomposed_progress_mbr.py.
• MoRT's paper sets m = 5 2103.11790 § Experimental setup, but the GPT-2 class defaults to 30 gpt2MCM_generation.py. The shipped run scripts pass 5, so the reported runs match the paper. The paper gives no top-p value, and the code uses 0.9 generation.py.
• SpecInfer's tables use tree widths up to 5 2305.09781 tab. 2 2305.09781 tab. 3. The released code sets the maximum beam width to 3 batch_config.h. The paper describes multi-step stochastic verification 2305.09781 §4.3, but the released code's only verifier walks the tree by exact token match request_manager.cc. Its merge-based tree construction is also disabled in code request_manager.cc.
• SimCTG's paper says the penalty uses the "last layer". The code takes the last entry of the model's hidden states, which in the Hugging Face GPT-2 code is after the final layer norm utlisgpt.py utlisgpt.py. The paper does not say this 2202.06417 §3.2.
• BRIO's paper gives 16 candidates and no beam width for them 2203.16804 app. B. The released CNN/DailyMail script uses 16 beams in 16 groups with length penalty 2.0, and the XSum script uses 128 beams and keeps every 8th output gen_candidate.py gen_candidate.py. Its generation settings use 4 beams for CNN/DailyMail and 8 for XSum, with no-repeat 3-grams config.py config.py main.py. The paper gives a default of 4 beams only for its CNN/DailyMail beam experiment 2203.16804 tab. 5, and no XSum beam width.
• Renze and Guven say the other sampling parameters were fixed 2402.05201 §2.3, but the code sets only temperature, so provider defaults apply openai_model.py. A temperature result from this code therefore depends on each provider's default top-p.

The papers test different setups, so their numbers are not set against each other either, apart from the temperature bullet above. Within the first paper, numbers for the same model differ across tables because the subsets and setups differ. For example, PaLM-2 L greedy decoding is 44.0% on a top-100 GSM8K subset 2402.10200 tab. 2 and 34.8% on the full test set 2402.10200 tab. 7.

Two paper-to-code gaps are listed in the variants rather than here, because they are not disagreements between papers: the INSPIRED baseline temperature (0.82 in the paper, not in the code) 2009.14306 app. D interactive.py, and the number of candidates (five in the paper, three in the code) 2009.14306 §6.2 interactive.py.

Papers

• Survey of controllable generation with pre-trained transformers: a survey that files guided and trainable decoding-time methods under post-processing. It runs no experiments.
• A Contrastive Framework for Neural Text Generation: SimCTG, a contrastive training loss plus contrastive search, which picks among the top-k tokens with a similarity penalty.
• BRIO: Bringing Order to Abstractive Summarization: a summariser trained with a ranking loss over beam candidates, decoded by length-normalised beam search, with a scorer variant that ranks candidates.
• Chain-of-Thought Reasoning Without Prompting: CoT-decoding, which branches on the top-k first tokens and picks the answer with the widest top-1 vs top-2 margin. Compares against greedy, sampling baselines and self-consistency.
• DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task: LoRA fine-tuning of ChatGLM-6B for Chinese medical dialogue. Its decoding content is one temperature and top-p grid, graded by a doctor on one question.
• INSPIRED: Toward Sociable Recommendation Dialog Systems: a GPT-2 recommender that samples candidate replies with temperature, top-k and top-p, then picks one at random after a bigram filter.
• Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems: beam search over routing tours with one beam per decoder, compared with greedy and sampling for the same learned policy.
• Solving Quantitative Reasoning Problems with Language Models: Minerva, which answers maths with greedy decoding for single answers and majority voting over T = 0.6 samples (maj1@k) for headline numbers.
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection: one model that emits reflection tokens, with a segment-level beam scored by those tokens at inference.
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding: training-free contrast between the clean image and a noised copy, with a plausibility cutoff on each token.
• Large Language Models: A Survey: a broad survey whose section on decoding gives temperature, top-k and nucleus sampling. It has no experiments of its own, and its tables are compiled from other papers.
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference: shows that BF16 greedy outputs vary with GPU count, GPU type and batch size, and proposes LayerCast, which stores weights in BF16 and computes in FP32.
• CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decoding: MBR selection over sampled action chunks, used after a backtrack in a robot policy.
• Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do: a moral direction in BERT embeddings, used as a candidate filter after top-k and top-p sampling in GPT-2.
• SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification: a draft model proposes a token tree, and the large model verifies it in one pass with tree attention.
• The Effect of Sampling Temperature on Problem Solving in Large Language Models: temperature sweeps from 0.0 to 1.0 on multiple-choice questions, with no significant accuracy change in that range and a recommendation of 0.0.
• Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition: argues for reporting sampled outputs next to a greedy score, with stress tests on GLUE, in-context learning, summarisation, chain-of-thought and safety.
• A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models: a survey that files CAD, DoLa and ITI under decoding, with a table of 32 techniques and no experiments.
• Energy benchmark for LLaMA inference on V100 and A100 GPUs: energy and GPU use for LLaMA inference at 7B to 65B on V100 and A100 GPUs, with temperature 0.8 and top-p 0.95 fixed.
• Ad Headline Generation using Self-Critical Masked Language Model: a BERT-Large masked language model fine-tuned with self-critical policy gradient, decoding by length-normalised beam search.