--- kind: concept slug: decoding-strategies title: Decoding strategies summary: >- How a model picks each next step from its scores: greedy, temperature, top-k and top-p sampling, CoT-decoding, majority voting, beam search, contrastive search, contrastive decoding, MBR over robot action chunks, toxicity filtering and tree-based speculative verification. Twenty paper cards; numbers are author-reported. tags: [inference, decoding, sampling, beam-search, reasoning, prompt-free, majority-voting, contrastive-search, contrastive-decoding, minimum-bayes-risk, speculative-decoding, toxicity, reproducibility, temperature, energy, summarization, length-normalisation] papers: ["2402.10200", "2304.01097", "2009.14306", "2012.10638", "2206.14858", "2310.11511", "2311.16922", "2402.06196", "2506.09501", "2601.02295", "2103.11790", "2305.09781", "2310.03003", "2401.01313", "2402.05201", "2601.07239", "2607.06818", "2201.05337", "2202.06417", "2203.16804"] related: [chain-of-thought-prompting, self-consistency, constrained-decoding, speculative-decoding, dialogue-generation, policy-gradient, hallucination-mitigation, retrieval-augmented-generation, low-precision-arithmetic, vision-language-action-model, controllable-text-generation, toxicity-detection, llm-serving, prompt-engineering, emergent-abilities, red-teaming, masked-language-modeling, self-refinement, contrastive-learning, text-summarization] status: draft updated_at: 2026-11-01 --- ## Overview At each step a language model gives a probability to every token in its vocabulary. A decoding strategy is the rule that turns those probabilities into the text that is written. The simplest rule is greedy decoding: take the most likely token at every step [arxiv:2402.10200 §2.2]. Other rules sample from the distribution or search over several continuations. The paper [Chain-of-Thought Reasoning Without Prompting](/papers/2402.10200) compares such rules as baselines [arxiv:2402.10200 tab. 4]. A second paper, [DoctorGLM](/papers/2304.01097), tests temperature and top-p sampling on a fine-tuned chat model [arxiv:2304.01097 §4.2]. A third, [INSPIRED](/papers/2009.14306), samples whole dialogue replies with temperature, top-k and top-p and then picks one [arxiv:2009.14306 §6.2] [arxiv:2009.14306 app. D]. The idea the first paper adds is that the first decoding step matters most. Only that step branches. Every later step is greedy, and the paths are then ranked by how confident the model is in the answer [arxiv:2402.10200 §2.2] [arxiv:2402.10200 §3]. The authors also tried branching at later steps and found that early branching gives more diverse paths, while later branches are strongly shaped by the tokens before them [arxiv:2402.10200 §2.2] [arxiv:2402.10200 fig. 2]. A further paper, [Solving Quantitative Reasoning Problems with Language Models](/papers/2206.14858), uses greedy decoding for single answers and nucleus sampling for majority voting over many answers [arxiv:2206.14858 §2.3]. Its headline maths numbers come from the voting, not from a single answer [arxiv:2206.14858 §2.5]. A fourth paper, [Multi-Decoder Attention Model](/papers/2012.10638), uses the word "decoder" in another sense. Its decoders build a vehicle-routing tour one stop at a time, and its beam search keeps partial tours instead of tokens [arxiv:2012.10638 §3.1.2]. The same ideas (greedy, sampling, beam) apply, but the scores are costs of partial routes, not token probabilities. Three more cards add rules. [SimCTG](/papers/2202.06417) proposes contrastive search, which picks each next token from the top-k candidates by probability minus a penalty for similarity to the context [arxiv:2202.06417 §3.2]. Its name is close to contrastive decoding, below, but the two methods are unrelated. [BRIO](/papers/2203.16804) is mainly a training method, but its summarisers are decoded with length-normalised beam search, and one of its two variants ranks candidate summaries [arxiv:2203.16804 §3] [arxiv:2203.16804 tab. 5]. The [controllable text survey](/papers/2201.05337) files guided decoding under post-processing, where a control signal reshapes the next-token distribution while the language model's weights stay fixed [arxiv:2201.05337 §3.4]. Four further cards change what the scores are or how candidates are chosen. [Visual Contrastive Decoding](/papers/2311.16922) contrasts the next-token logits from the real image with those from a noised copy, so a token wins only if the clean image supports it more [arxiv:2311.16922 §3.3.1]. [Self-RAG](/papers/2310.11511) keeps a segment-level beam in which each candidate is scored by reflection tokens the model emits itself [arxiv:2310.11511 §3.3]. [CycleVLA](/papers/2601.02295) applies minimum Bayes risk (MBR) selection to sampled robot action chunks, not to tokens [arxiv:2601.02295 §4.2]. The [survey](/papers/2402.06196) defines the basic rules in one place [arxiv:2402.06196 §3.9], and [the reproducibility study](/papers/2506.09501) shows that greedy outputs can change with the hardware they run on [arxiv:2506.09501 §3.2]. Two more cards change which candidates survive or how the loop runs. [MoRT](/papers/2103.11790) filters the candidates left by top-k and top-p sampling, scoring each one with a direction found in BERT's sentence embeddings [arxiv:2103.11790 § Guiding Generative Language Models using the MoralDirection]. [SpecInfer](/papers/2305.09781) keeps the sampling rule and changes the loop instead: a small draft model proposes a token tree, and the large model checks all of its nodes in one pass [arxiv:2305.09781 §4.2]. A study of temperature, [The Effect of Sampling Temperature on Problem Solving in Large Language Models](/papers/2402.05201), sweeps temperature from 0.0 to 1.0 on multiple-choice questions and finds no significant change in mean accuracy over that range [arxiv:2402.05201 §3.1]. It recommends 0.0 for problem solving [arxiv:2402.05201 §4.1]. The opposite view appears in [Stochastic CHAOS](/papers/2601.07239), which argues that one greedy run understates what a model can do, and reports that majority votes over sampled outputs score higher [arxiv:2601.07239 §2] [arxiv:2601.07239 §3]. The two are set side by side under Contradictions. Three further cards are less about accuracy. The [hallucination survey](/papers/2401.01313) lists decoding-time methods that contrast or edit internal states, and as a survey it runs no experiments of its own [arxiv:2401.01313 §3.1] [arxiv:2401.01313 tab. 1]. The [ad headline paper](/papers/2607.06818) uses greedy decoding only as a training baseline and decodes with length-normalised beam search [arxiv:2607.06818 §3.3] [arxiv:2607.06818 §3.4]. The [energy benchmark](/papers/2310.03003) fixes temperature 0.8 and top-p 0.95 and does not compare decoding rules [arxiv:2310.03003 §3.1]. ## Variants ### Greedy decoding One path, and the most likely token at each step. It is the reference in every comparison in the paper, and its compute cost is O(1) against O(k) for the multi-path methods [arxiv:2402.10200 tab. 7]. On GSM8K it reaches 9.9% on pre-trained Mistral-7B [arxiv:2402.10200 tab. 4] and 34.8% on PaLM-2 L [arxiv:2402.10200 tab. 7]. Greedy is not stable across hardware. Under BF16, the same greedy run changes with GPU count, GPU type and batch size, because the order of floating-point sums changes. On DeepSeek-R1-Distill-Qwen-7B, AIME'24 accuracy has a standard deviation of 9.15% across 12 configurations, and FP32 gives 0 [arxiv:2506.09501 tab. 3]. Output length varies by up to 9,189 tokens under BF16 [arxiv:2506.09501 tab. 4]. The authors report that over 90% of MATH500 examples diverge under BF16 on that model [arxiv:2506.09501 §3.2]. The greedy numbers elsewhere on this page are single reported values. This paper suggests that a single greedy score can sit anywhere in a spread of this size, depending on the configuration that produced it. For multiple-choice problems, the temperature study also recommends 0.0, on the same ground of reproducibility and with no measured accuracy cost [arxiv:2402.05201 §4.1]. The sweep is described below. ### Sampling baselines The paper's GSM8K comparison on pre-trained Mistral-7B uses four sampling and search rules [arxiv:2402.10200 tab. 4]: - Top-k sampling (k = 10): 4.9%. - Nucleus sampling (p = 0.9): 6.4%. - Beam search (b = 10): 6.7%. - Temperature sampling (T = 0.7): 7.5%. In this one setup all four score below greedy decoding (9.9%). The card does not describe how each rule works; the paper uses them only as baselines. ### Temperature and top-p sampling Temperature rescales the next-token distribution, and top-p (nucleus) keeps the smallest set of most likely tokens whose probability reaches the threshold, then samples from that set [arxiv:2304.01097 §4.2]. The paper does not state which of the two is applied first [arxiv:2304.01097 §4.2]. The survey writes the temperature softmax as the logits divided by T before the softmax [arxiv:2402.06196 eq. 3], and leaves the top-p value unstated [arxiv:2402.06196 §3.9]. The paper grades a grid of temperature 0.90, 0.95 and 0.99 against top-p 0.6, 0.7 and 0.8. A doctor gave each cell one letter grade, A (best) to F (worst), for one child-fever question [arxiv:2304.01097 tab. 6]. Three cells got A: (0.95, 0.7), (0.95, 0.8) and (0.99, 0.6). Two got F: (0.90, 0.6) and (0.95, 0.6). With one question and one grader, the grid shows a direction, not a validated setting. The repository's notebook uses top-p 0.7 and temperature 0.99 [code: Doctor_GLM/chat_lora.ipynb#L62], and the paper grades that pair B [arxiv:2304.01097 tab. 6]. ### Temperature sweep on multiple-choice problems Renze and Guven answer each question ten times at each temperature from 0.0 to 1.0 in steps of 0.1, with one worked example and a chain-of-thought prompt for GPT-3.5 [arxiv:2402.05201 §2.1] [arxiv:2402.05201 §2.3]. Accuracy is the share of the ten answers that match the gold choice, averaged over questions [arxiv:2402.05201 §2.4]. For GPT-3.5 on the 1,000-question exam, the test finds no significant effect across 0.0 to 1.0 (H(10) = 10.439, p = 0.403) [arxiv:2402.05201 §3.1]. On the 100-question exam none of the nine models is significant. The lowest p-value is 0.072, for Llama 2 7B, whose answers were badly formatted 39% of the time [arxiv:2402.05201 tab. 3] [arxiv:2402.05201 §3.1]. Two of the ten exams are significant, LSAT-AR and SAT-Math, and the authors attribute this to the all-or-nothing answers at 0.0 [arxiv:2402.05201 tab. 5] [arxiv:2402.05201 §3.1]. Above 1.0 the text turns incoherent, and GPT-3.5 accuracy falls to about zero near 1.6 [arxiv:2402.05201 §3.1] [fig. 2]. The authors recommend 0.0 for problem-solving tasks, citing reproducibility and no measured accuracy cost [arxiv:2402.05201 §4.1]. The study covers only multiple-choice questions, and only one configuration was tested above 1.0 [arxiv:2402.05201 §4.2]. Tokens per response rise with temperature, which adds cost above 1.0 [arxiv:2402.05201 §4.4]. Greedy output can still change with hardware under BF16 [arxiv:2506.09501 tab. 3], so temperature 0.0 does not by itself guarantee repeatable runs. ### Stochastic evaluation with a reported spread [Stochastic CHAOS](/papers/2601.07239) runs one protocol on four stress tests. It compares greedy decoding (temperature 0) with sampling at temperature 0.7 and top-p 0.9, using temperature 0.7 or 1.0 for the safety runs, with k samples per input and a majority vote over them where a metric needs one [arxiv:2601.07239 §3] [arxiv:2601.07239 §4] [arxiv:2601.07239 §5] [arxiv:2601.07239 §6]. The measures are a robustness ratio over paraphrased, perturbed and adversarial GLUE variants, an exploration gain (best-of-k minus greedy accuracy), and a risk gap for attack success between sampled and greedy decoding [arxiv:2601.07239 §3] [arxiv:2601.07239 §4] [arxiv:2601.07239 §6]. The reported values are author-reported. In a figure caption, the stochastic robustness ratio for Claude is 0.85 to 0.91 across MNLI, QQP, QNLI and SST-2, against 0.79 to 0.82 for greedy decoding [arxiv:2601.07239 §3]. On in-context learning with BESSTIE, greedy accuracy for many model-task pairs lies between 40% and 65%. Best-of-16 typically adds 10 to 20 points, and best-of-64 adds 2 to 5 more [arxiv:2601.07239 §4]. The accuracy-versus-k figure is a placeholder in the source [arxiv:2601.07239 §4]. The chain-of-thought and safety tables are labelled simulated, so use them as a template for reporting, not as numbers [arxiv:2601.07239 §5] [arxiv:2601.07239 §6]. Each stochastic measure needs k generations per input, and the reasoning and safety sections use up to 32 samples [arxiv:2601.07239 §5] [arxiv:2601.07239 §6]. ### Sample, filter and pick (dialogue replies) INSPIRED draws each recommender reply by sampling tokens with temperature 0.8, then keeps the 400 most likely tokens, then keeps the top-p 0.9 nucleus [arxiv:2009.14306 app. D] [code: code/interactive.py#L1590-L1591]. The paper gives 0.82 for the baseline model and 0.8 for the strategy model [arxiv:2009.14306 app. D]. The sampling stops at a turn-end token or after 200 tokens [code: code/interactive.py#L1588-L1597]. The paper draws five candidate replies and keeps those that contain encouragement or are longest, then picks one at random [arxiv:2009.14306 §6.2]. Candidates that repeat a bigram are removed first [code: code/interactive.py#L1635-L1643]. The released script draws three candidates and picks uniformly among those that pass the bigram filter. Its length rule is disabled [code: code/interactive.py#L1539] [code: code/interactive.py#L1646-L1674]. The reported comparison is between two models, not between decoding settings. Author-reported: the strategy model has test perplexity 8.93 against 9.28 and BLEU-4 6.63 against 5.11 for the baseline [arxiv:2009.14306 tab. 7]. The card records no test of the decoding rule itself. ### Self-consistency without a CoT prompt The paper also runs self-consistency with 10 paths, with and without a chain-of-thought prompt [arxiv:2402.10200 tab. 3] [arxiv:2402.10200 tab. 7]. On pre-trained Mistral-7B it reaches 12.9% on GSM8K, and on PaLM-2 L 40.6% [arxiv:2402.10200 tab. 7]. It costs O(k) [arxiv:2402.10200 tab. 7]. ### Majority voting over samples (maj1@k) Draw k solutions with nucleus sampling at temperature 0.6 and p = 0.95, then return the most common final answer [arxiv:2206.14858 §2.3] [arxiv:2206.14858 §2.5]. The answer is read from a fixed sentence in the output, and only that answer is compared, not the reasoning [arxiv:2206.14858 app. D.1]. This is the same family as the self-consistency row above, applied to a model trained on technical text. Author-reported, Minerva 540B, MATH: greedy single answer 33.6%, maj1@k 50.3% (k = 64) [arxiv:2206.14858 tab. 3]. On GSM8k the same model goes from 58.8% to 78.5% (k = 40) [arxiv:2206.14858 tab. 3]. The authors say 97% of the large-k accuracy on MATH is reached at k = 64 [arxiv:2206.14858 §2.5]. Sampling is not free on single answers. On the 62B model, pass@1 is 26.5% with greedy decoding and 21.8% at T = 0.6 [arxiv:2206.14858 tab. 8]. Majority voting recovers this loss only because it aggregates over many samples. Log-likelihood reranking of the same samples does much worse on MATH with 62B: 23.8% against 42.0% for majority voting [arxiv:2206.14858 tab. 8]. Majority voting also needs a grouping rule and a tie-break. The paper says predictions are grouped by final answer but does not say whether grouping uses the normalized string or SymPy equivalence, and tie-breaking is not stated [arxiv:2206.14858 §2.5]. An implementation has to choose both. ### CoT-decoding The method in [arxiv:2402.10200 §2.2]: 1. Take the top-k tokens at the first decoding step. The default is k = 10 [arxiv:2402.10200 §3]. 2. Continue each branch greedily. 3. Score each path by the mean over its answer tokens of the top-1 probability minus the top-2 probability at that step. 4. Return the answer of the highest-scoring path (the "max" rule), or sum the scores of paths that give the same answer and return the largest total (the "agg" rule) [arxiv:2402.10200 tab. 7]. The prompt is only the bare "Q: [question] / A:" format. No chain-of-thought prompt and no tuning are used [arxiv:2402.10200 §2.1]. Reported gains, author-reported: - Pre-trained Mistral-7B, GSM8K: 9.9% greedy to 25.1% with CoT-decoding [arxiv:2402.10200 tab. 4] [arxiv:2402.10200 tab. 7]. The agg rule gives 25.3% [arxiv:2402.10200 tab. 7]. - PaLM-2 L, GSM8K: 34.8% greedy to 63.2% (max rule) or 64.1% (agg rule) [arxiv:2402.10200 tab. 7]. - Instruction-tuned Mistral-7B, GSM8K: 31.2% greedy to 38.2% [arxiv:2402.10200 tab. 5]. - Combined with zero-shot CoT prompting, the agg rule gives 48.4% on Mistral-7B and 87.0% on PaLM-2 L [arxiv:2402.10200 tab. 7]. ### Beam search over routing tours In the routing paper, a learned decoder adds one node to a partial tour per step. Each of the five decoders has its own beam, so beams do not mix [arxiv:2012.10638 §3.1.2] [arxiv:2012.10638 §4.1]. When two partial tours share the same remaining subproblem and one is not longer and has at least as much vehicle capacity left, the worse one is merged away and the survivor keeps the larger probability [arxiv:2012.10638 §3.1.2] [arxiv:2012.10638 fig. 2]. Author-reported on CVRP100 [arxiv:2012.10638 tab. 1]: - Greedy decoding of the same model: gap 4.86%, 45 s for 10,000 instances. - Beam 30 per decoder: gap 2.49%, 31 min. - Beam 50 per decoder: gap 2.23%, 53 min. - Attention Model with 1,280 samples: gap 3.72%, 2 h. The paper also reports that the multi-decoder model with greedy decoding beats a single decoder [arxiv:2012.10638 tab. 2]. Its routing setup differs from token decoding, so the numbers are not comparable with the rows above. ### Length-normalised beam search (ad headlines) The [ad headline paper](/papers/2607.06818) decodes from a BERT-Large masked language model. Its mask lets each headline position see only earlier headline tokens [arxiv:2607.06818 §3.3] [eq. 3]. Inference uses beam search, which stops at a maximum length or when every beam emits the end token [arxiv:2607.06818 §3.4]. The beam ranks candidates by a length-normalised word score rather than plain log-likelihood, scaling each word's log-likelihood by a factor that depends on its position [arxiv:2607.06818 §3.4] [eq. 10]. The normalisation coefficient, the beam width and the maximum length are not stated [arxiv:2607.06818 §3.4]. The ablation removes beam normalisation and post-processing together. ROUGE-L falls from 6.33 to 5.37, in the paper's table of differences from its bi-LSTM baseline [arxiv:2607.06818 tab. 1]. Because both steps change at once, the ablation cannot isolate normalisation. Decoding also appears in training. One sampled headline and one greedy headline are both scored with ROUGE-L F1 against approved headlines. The sampled headline's probability goes up when it scores above the greedy one, and down when below [arxiv:2607.06818 §3.3] [eq. 9] [arxiv:2607.06818 §4.1]. The reward is the metric the table reports, so part of the gain may come from optimising that metric [arxiv:2607.06818 §4.1] [arxiv:2607.06818 tab. 1]. ### Contrastive search for repetition (SimCTG) [SimCTG](/papers/2202.06417) pairs a training loss with a decoder, and the decoder is contrastive search. At each step it takes the top-k next tokens, scores each one by (1 − α) times its probability minus α times the largest cosine similarity between its last-layer state and the states of all earlier tokens, and emits the best one [arxiv:2202.06417 §3.2] [eq. 5]. With α = 0 it is greedy search [arxiv:2202.06417 §3.2]. The paper says k is typically 3 to 10 [arxiv:2202.06417 §3.2]. The document setting uses k = 8 and α = 0.6 [arxiv:2202.06417 §4] [code: document_generation/inference.sh#L8-L9]. The dialogue setting uses k = 5 and α = 0.6 [arxiv:2202.06417 §5]. The appendix recommends α from 0.5 to 0.8 and k from 5 to 10 [arxiv:2202.06417 app. E]. Author-reported on WikiText-103 with GPT-2 small [arxiv:2202.06417 tab. 1]. With the SimCTG model, contrastive search has a 2-gram repeat rate of 3.93% against 4.05% for nucleus sampling, and a higher MAUVE (0.94 against 0.92) and coherence (0.610 against 0.584). With the MLE model instead of SimCTG, the same decoder repeats 44.20% of 2-grams and has MAUVE 0.18 [arxiv:2202.06417 tab. 1]. On 200 prefixes rated by five graders, SimCTG with contrastive search scores higher than nucleus sampling on coherence (3.25 against 2.96) and fluency (3.57 against 3.34), and the paper marks both as significant. On informativeness the two tie at 3.96 [arxiv:2202.06417 tab. 2]. Cost: each step scores k candidates, and the paper's latency comparison, at batch size 1, finds contrastive search and beam search nearly equal when k and the beam width are both at most 6, with contrastive search faster above that [arxiv:2202.06417 §6.4] [fig. 5]. The penalty compares each candidate with every earlier token, so the states of the whole output must be kept [arxiv:2202.06417 §3.2]. ### Contrastive decoding for vision-language models [Visual Contrastive Decoding](/papers/2311.16922) targets object hallucination in large vision-language models, with no training. For each token, the model produces logits for the real image and for a copy with Gaussian noise added. The sampled logits are (1 + α) times the first minus α times the second, with α = 1 [arxiv:2311.16922 §4.1] [eq. 3]. Only tokens whose probability under the clean image is at least β times the top probability are kept, with β = 0.1 [arxiv:2311.16922 §3.3.2] [eq. 4] [§4.1]. Setting α = 0 gives ordinary decoding [arxiv:2311.16922 §3.3.1]. Author-reported on POPE, mean over 5 runs [arxiv:2311.16922 tab. 1]: for Qwen-VL on GQA with popular negatives, accuracy goes from 75.99 to 81.83 and F1 from 74.84 to 82.23, the paper's headline gain [arxiv:2311.16922 §4.2]. On the MME hallucination subset, LLaVA-1.5 goes from 565.33 to 604.66 [arxiv:2311.16922 tab. 2]. Regular decoding still wins on the position items for Qwen-VL and InstructBLIP [arxiv:2311.16922 tab. 2]. The noise has a cost. Each generated token needs a second forward pass on the noised image [arxiv:2311.16922 §3.3.1], and the paper gives no timing or memory figures. The released code uses a one-shot noise schedule instead of the step-wise rule in [eq. 2] [code: vcd_utils/vcd_add_noise.py#L7-L8], and the released code only samples, so beam search with this method is not implemented [code: vcd_utils/vcd_sample.py#L160-L161]. ### Context and layer contrast (hallucination survey) The [hallucination survey](/papers/2401.01313) files three methods under decoding [arxiv:2401.01313 §3.1]. CAD contrasts the output distribution with and without the context, which helps when the context conflicts with what the model already knows. DoLa contrasts logits from later and earlier layers, because it treats factual knowledge as stored in certain layers. ITI is filed here too, but it does not choose tokens by a contrast. It selects a sparse set of attention heads and moves activations along truth-related directions at each generated token [arxiv:2401.01313 §3.1]. The survey's table gives tasks and models, not comparable results [arxiv:2401.01313 tab. 1]. DoLa is listed on LLaMA 7B to 65B and GPT-4, and CAD on OPT, GPT-Neo, LLaMA and FLAN-T5 [arxiv:2401.01313 tab. 1]. The primary papers for these methods are not cards on this site, so this page gives none of their numbers. ### Segment-level beam with self-critique (Self-RAG) [Self-RAG](/papers/2310.11511) trains one language model to emit reflection tokens. These cover whether to retrieve, whether a passage is relevant, whether a segment is supported by it, and how useful the whole answer is [arxiv:2310.11511 §3.1] [tab. 1]. At inference, for each segment the model first predicts a retrieve token. If retrieval is needed, it writes one continuation per retrieved passage and keeps the best B by LM probability plus a weighted critique score [arxiv:2310.11511 §3.3] [eq. 3] [eq. 4] [alg. 1]. The paper uses B = 2, greedy token decoding, and critique weights of 1.0, 1.0 and 0.5 for relevance, support and usefulness [arxiv:2310.11511 §4.3]. The retrieval threshold is 0.2 for most tasks [arxiv:2310.11511 §4.3]. Hard constraints can also drop candidates that contain an undesired critique token [arxiv:2310.11511 §3.3]. Author-reported, 13B [arxiv:2310.11511 tab. 2]: PubHealth 74.5 against 70.1 for ChatGPT, and PopQA 55.8 against 50.8 for ChatGPT with retrieval. An ablation at 7B with 50k training instances gives ASQA exact match 32.1 for the full model and 18.1 without the critic [arxiv:2310.11511 §5.2] [fig. 3]. The method needs a model trained with reflection tokens. Those labels come from GPT-4 prompts, 4k to 20k per token type [arxiv:2310.11511 §3.2.1] [app. A.1]. Cost grows with passages times beam width [arxiv:2310.11511 §3.3]. ### MBR selection over sampled action chunks (CycleVLA) This variant is not for token output. A robot policy draws N = 8 action chunks from the same observation, and the chunk with the smallest average L2 distance to the others is run [arxiv:2601.02295 §4.2] [eq. 4] [§5.1]. Each chunk is turned into a path by adding up its translation and rotation deltas, giving a feature vector of length 6H [arxiv:2601.02295 §4.2] [app. C]. The rule is used only after a backtrack to an earlier subtask [arxiv:2601.02295 §4.2]. The policy has to be stochastic for this to work [arxiv:2601.02295 §3]. Author-reported. Table 4 is estimated offline from recorded rollouts, not from closed-loop runs [arxiv:2601.02295 §5.3]. At the 350K checkpoint with failure correction, N = 8 random draws give 80.3% and MBR gives 90.2%, against 83.2% without correction [arxiv:2601.02295 tab. 4]. L2 is the best of the distance choices tested at every checkpoint [arxiv:2601.02295 tab. 5]. Running MBR at every step reaches 96.9% against 95.3% for the main method, at about 2.2 times the inference time [arxiv:2601.02295 tab. 6] [§5.4]. The MBR step itself takes under 0.1% of runtime on the A100 [arxiv:2601.02295 tab. 9]. ### Length-normalised beam search and candidate ranking (BRIO) [BRIO](/papers/2203.16804) trains a summariser with an extra ranking loss, and its decoding uses two existing tools. The first is beam search with a length-normalised score: the sum of a candidate's token log-probabilities divided by its length raised to a penalty α [arxiv:2203.16804 eq. 9]. The paper sets α = 2.0 for CNN/DailyMail and 0.6 for XSum, equal to the beam search's own length penalty [arxiv:2203.16804 tab. 13] [app. B]. The 16 training candidates per document come from diverse beam search, run offline before training [arxiv:2203.16804 app. B] [§3]. The paper states the candidate count but not the beam width for them [arxiv:2203.16804 app. B]. The second is the ranker. BRIO-Ctr is trained with the ranking loss alone, can select among candidates, and cannot generate [arxiv:2203.16804 §3]. BRIO-Mul keeps the cross-entropy term and generates [arxiv:2203.16804 §5.1]. On CNN/DailyMail, BRIO-Ctr scores 47.28 ROUGE-1 against 47.78 for BRIO-Mul [arxiv:2203.16804 tab. 2]. Beam width matters differently for the two models. Author-reported ROUGE-1 on CNN/DailyMail [arxiv:2203.16804 tab. 5]: BART falls from 44.29 with 4 beams to 42.79 with 100 beams, while BRIO-Mul rises from 47.78 to 48.23. The paper reports this on one dataset and does not test other causes. The default beam width for final generation is 4 [arxiv:2203.16804 tab. 5]. ### Decoding-time guidance (controllable text survey) The [controllable text survey](/papers/2201.05337) sorts control methods by where the control signal enters. The post-processing family keeps the language model's weights fixed and reshapes the next-token distribution at decoding time. It covers guided methods that use a separate guide model at inference (PPLM, GeDi, DExperts, FUDGE, COLD and others) and trainable modules that are fitted jointly with the frozen model [arxiv:2201.05337 §3.4]. In GeDi, a class-conditional model's posterior is combined with the language model's distribution by Bayes' rule [arxiv:2201.05337 eq. 9]. The survey says that post-processing can be slow at inference and can lower fluency. PPLM loses some fluency, and Mix and Match and COLD need many iterations [arxiv:2201.05337 §3.4] [§5.1]. The survey runs no experiments of its own, so it gives no comparative numbers for these methods. ### Toxicity filter after top-k and top-p (MoRT) Ordinary top-k and then top-p sampling give a set of candidate tokens. Each candidate is appended to the prefix, and the whole text is scored on the moral direction. Candidates scored at or below a threshold t are removed, but at least m are always kept [arxiv:2103.11790 § Guiding Generative Language Models using the MoralDirection]. The paper uses BERT-large with SBERT pooling for these sentence embeddings [arxiv:2103.11790 § Guiding Generative Language Models using the MoralDirection]. It tests t = -0.5, 0 and 0.5 with m = 5 [arxiv:2103.11790 § Experimental setup]. The paper does not state top-p; the released wrapper uses 0.9 [code: realtoxicityprompts/generation/generation.py#L281]. Author-reported on RealToxicityPrompts, where toxicity is scored by an external API: the share of 100,000 prompts with a generation above toxicity 0.9 falls from 9.82% for GPT-2 to 1.17% with the filter at t = 0 [arxiv:2103.11790 § Reducing Neural Toxic Degeneration in Language Models]. The authors say that such scoring tools are imperfect and biased [arxiv:2103.11790 § Testbed for evaluating the toxicity of text generations]. The filter has a per-step cost. Each candidate is decoded as a full text and embedded with BERT-large, and the whole prefix is embedded again each time [code: MoRT/mcm_textgeneration/mcm_models.py#L439-L444]. The paper reports no runtime or memory figures. ### Tree-based speculative verification (SpecInfer) A speculator, here a small draft model, builds a token tree rather than one continuation. The large model runs once over every node, with a tree attention mask that lets each node see its own ancestors and the verified prefix, not its siblings' branches [arxiv:2305.09781 §4.2]. Greedy verification walks down the tree and takes the child whose token matches the large model's prediction at that node, so its output matches ordinary greedy decoding [arxiv:2305.09781 §4.3] [alg. 2]. The paper also gives a multi-step sampling rule, stated as leaving the output distribution equal to the large model's, and points to a separate reference for the proof [arxiv:2305.09781 §4.3]. Author-reported speed-ups over incremental-decoding serving systems: 1.5-2.5x on one node and 2.4-2.8x on two nodes with eight GPUs [arxiv:2305.09781 §6.2]. Against FlexGen with offloading, OPT-13B and OPT-30B on one A10 go 2.6-3.5x faster per token [arxiv:2305.09781 §6.3]. Multi-step sampling verifies 1.26 to 1.28 times more tokens per stochastic step than naive sampling [arxiv:2305.09781 tab. 3]. The paper estimates that one draft model adds under 1% memory per GPU when it is 100 to 1000 times smaller than the large model [arxiv:2305.09781 §5.3]. For the speculate-then-verify idea itself, see the speculative-decoding concept page. ## When to use which | Situation | Use | Why | |---|---|---| | Tight latency or compute budget, one output needed | Greedy decoding | One path, O(1) [arxiv:2402.10200 tab. 7]. Beats the sampling baselines on Mistral-7B GSM8K [arxiv:2402.10200 tab. 4], but is below CoT-decoding on most tasks reported [arxiv:2402.10200 tab. 5] [arxiv:2402.10200 tab. 6]. | | Pre-trained model, no CoT prompt, no tuning, answer can be parsed from text | CoT-decoding | +15.2 points over greedy on pre-trained Mistral-7B GSM8K, with a prompt-free setup [arxiv:2402.10200 tab. 5]. | | Model is instruction-tuned and reasoning is needed | CoT-decoding, smaller gain | Gain on instruction-tuned Mistral-7B is +7.0 points on GSM8K [arxiv:2402.10200 tab. 5]. The paper says the benefit of more branches is smaller for instruction-tuned models [arxiv:2402.10200 §3.1]. | | A CoT prompt is available and k paths can be afforded | CoT-decoding with zero-shot CoT (agg rule) | Best GSM8K numbers reported on both models [arxiv:2402.10200 tab. 7]. Costs O(k). | | A CoT prompt alone is enough | Zero-shot CoT prompting | On PaLM-2 L it reaches 75.1% on GSM8K, above CoT-decoding alone at 63.2% [arxiv:2402.10200 tab. 7]. | | Short final answer that can be compared after normalization, k samples affordable, no answer span needed for branching | Majority voting over samples (maj1@k), T = 0.6, p = 0.95 | Large gains over greedy on MATH and GSM8k for Minerva 540B [arxiv:2206.14858 tab. 3]. Costs O(k). The reasoning is not checked, only the final answer [arxiv:2206.14858 §2.3]. | | Open-ended chat from a fine-tuned model, no answer span to score | Temperature and top-p sampling, checked by a grader | The only evidence is one doctor's grades on one question. (0.95, 0.7) got the top grade [arxiv:2304.01097 tab. 6]. Test on your own prompts before fixing values. | | Dialogue reply that must vary, no answer span | Temperature 0.8, top-k 400, top-p 0.9, several candidates, bigram filter, pick one | The paper's setting for recommender replies [arxiv:2009.14306 app. D] [arxiv:2009.14306 §6.2]. The card records no comparison against other decoding rules, so treat the values as a starting point. | | Learned routing policy, solution quality matters more than run time | Beam search, one beam per decoder, beam 50 | Lowest gap of the learned methods on CVRP100 in the table, at about 70 times the greedy run time (53 min against 45 s) [arxiv:2012.10638 tab. 1]. | | Learned routing policy, a fast answer is needed | Greedy rollout of the trained policy | 4.86% gap in 45 s for 10,000 instances [arxiv:2012.10638 tab. 1]. | | Open-ended answers, no clear answer span | Not recommended for CoT-decoding | The top-two margin is less precise for open-ended answers [arxiv:2402.10200 §5]. | | Model API hides token probabilities | Not usable as described | The method needs the top-1 and top-2 probability at each answer token [arxiv:2402.10200 §2.2]. | | Vision-language model describes objects that are not in the image, no retraining, one extra forward pass per token is affordable | Visual contrastive decoding, α = 1, β = 0.1 | Author-reported gains on POPE across three models [arxiv:2311.16922 tab. 1]. β = 0.5 scored highest in the paper's MME sweep, so 0.1 is a default, not a tuned value [arxiv:2311.16922 app. B]. Test on your own image set. | | Retrieval-augmented generation, you can train the generator and a critic, and each claim needs a support check | Self-RAG-style segment beam with critique scores, B = 2, δ = 0.2 | Author-reported 13B results beat ChatGPT on PubHealth [arxiv:2310.11511 tab. 2]. Needs GPT-4-labelled critic data and retraining [arxiv:2310.11511 §3.2.1]. Costs K × B generations [arxiv:2310.11511 §3.3]. | | Robot policy outputs action chunks, is stochastic, and must retry a failed subtask | MBR over N = 8 chunks, L2 on summed deltas | Offline estimate: 90.2% with MBR against 80.3% for a random draw, at the 350K checkpoint [arxiv:2601.02295 tab. 4]. Needs a stochastic policy [arxiv:2601.02295 §3]. | | Output must match the large model's own greedy output, and a much smaller draft model is available | Tree speculation with greedy verification (SpecInfer) | Output matches incremental greedy decoding [arxiv:2305.09781 §4.3]. Author-reported 1.5-2.5x on one node and 2.4-2.8x on two nodes, against incremental-decoding systems [arxiv:2305.09781 §6.2]. The released code caps tree width at 3 and has no stochastic verification, so the width-5 and multi-step sampling results cannot be rerun from it [code: include/flexflow/batch_config.h#L196]. | | Model must be offloaded to CPU memory and per-token latency dominates | Tree speculation with offloading | Author-reported 2.6-3.5x against FlexGen for OPT-13B and OPT-30B on one A10 [arxiv:2305.09781 §6.3]. The gain is stated for that setup only. | | Open-ended generation must avoid toxic continuations, a sentence encoder is available, and each step can afford embedding up to 50 candidates | Toxicity filter after top-k and top-p, t = 0 or 0.5, m = 5 | Author-reported: 1.17% of prompts above toxicity 0.9 with t = 0, against 9.82% for GPT-2 [arxiv:2103.11790 § Reducing Neural Toxic Degeneration in Language Models]. The paper's Table 3 marks t = 0.5 as best [arxiv:2103.11790 tab. 3]. Each candidate needs an embedding pass [code: MoRT/mcm_textgeneration/mcm_models.py#L439-L444]. | | Benchmark runs that must agree across hardware, BF16 greedy decoding | Run in FP32, or LayerCast if memory matters | BF16 greedy accuracy spread is up to 9.15% on AIME'24, and FP32 gives 0 [arxiv:2506.09501 tab. 3]. LayerCast tracks FP32 accuracy in the paper's tables [arxiv:2506.09501 app. F]. The released LayerCast script does not run as shipped and changes the attention backend [code: evaluation/eval_layercast.py#L23] [code: evaluation/patch_layercast.py#L179], so compare like with like. | | Multiple-choice problem solving where repeat runs must match | Temperature 0.0 | The authors' recommendation. Mean accuracy shows no significant change from 0.0 to 1.0 on the 1,000-question exam [arxiv:2402.05201 §3.1] [arxiv:2402.05201 §4.1]. Tested on multiple-choice only [arxiv:2402.05201 §4.2]. Greedy output still varies with hardware under BF16 [arxiv:2506.09501 tab. 3]. | | Evaluation that must report how much answers vary, with k samples affordable | Greedy score next to k samples at temperature 0.7, top-p 0.9 | The measures are defined in [arxiv:2601.07239 §3] [arxiv:2601.07239 §4] [arxiv:2601.07239 §6]. Reported gains are author-reported, and the chain-of-thought and safety tables are simulated [arxiv:2601.07239 §5] [arxiv:2601.07239 §6]. Cost is k generations per input [arxiv:2601.07239 §5]. | | Decoding-time hallucination reduction, where you can read per-layer logits or run the model with and without the context | DoLa or CAD, chosen from the primary papers | The survey gives families and tasks, not comparable results [arxiv:2401.01313 §3.1] [arxiv:2401.01313 tab. 1]. Check each method's own paper before use. | | Short generated text, a metric you can compute as reward, and a beam decoder | Beam search with length normalisation, coefficient chosen by you | Author-reported gain for normalisation and post-processing together: 0.96 ROUGE-L points in the ablation [arxiv:2607.06818 tab. 1]. The coefficient and beam width are not stated [arxiv:2607.06818 §3.4]. | | Repetition in open-ended generation, and you can retrain the model with a contrastive loss | SimCTG training, then contrastive search with k = 8, α = 0.6 | Author-reported: 3.93% 2-gram repeat and MAUVE 0.94 on WikiText-103 with GPT-2 small [arxiv:2202.06417 tab. 1]. Training adds about 1.5% compute [arxiv:2202.06417 app. I]. Costs k candidate states per step. | | Repetition in open-ended generation, the model cannot be retrained | Contrastive search on the existing model, tested first | The paper finds it works on its Chinese and GPT-2-large MLE models, which it calls already isotropic, but not on English GPT-2 small, where it is well behind SimCTG [arxiv:2202.06417 §5] [app. H] [tab. 1]. Test on your own model. | | Attribute control (such as sentiment or topic) needed, the model must stay unchanged, extra inference time is acceptable | Guided post-processing (GeDi, PPLM, DExperts-style) | The survey's family for this case, with fixed weights [arxiv:2201.05337 §3.4]. It notes slow inference and possible fluency loss [arxiv:2201.05337 §3.4]. No comparative numbers in the survey. | | Summariser decoded with beam search, more beams wanted | Beam search with the training length penalty (α = 2.0 CNN/DailyMail, 0.6 XSum), beams up to 100 only on the model trained for it | BRIO-Mul ROUGE-1 rises from 47.78 at 4 beams to 48.23 at 100, while BART falls [arxiv:2203.16804 tab. 5]. The paper reports this on CNN/DailyMail only. | | Pool of summary candidates to choose from, no generation needed | BRIO-Ctr as a scorer | It ranks candidates by the length-normalised score and cannot generate [arxiv:2203.16804 §3]. Its ROUGE-1 is 47.28 on CNN/DailyMail, against 47.78 for BRIO-Mul [arxiv:2203.16804 tab. 2]. | | Energy per generated token for a sharded model | Not a decoding choice on this page; keep the maximum length in view | Energy per token is about the same at maximum lengths 512 and 1024 [arxiv:2310.03003 §4.3], and rises with shard count [arxiv:2310.03003 §4.3]. The benchmark fixes temperature 0.8 and top-p 0.95 [arxiv:2310.03003 §3.1]. | Compute is the main limit for CoT-decoding. Each branch is a full decode, and the paper calls the extra cost a limitation [arxiv:2402.10200 §5]. The paper's experiments use Mistral-7B and Gemma-7B on A100 GPUs and PaLM-2 on TPU v4 [arxiv:2402.10200 §9]. PaLM-2 access is not described in the card, so open-weight models are the practical choice for reproduction [arxiv:2402.10200 §9]. ## Implementation sketch Language-agnostic pseudocode of the common cores, from the steps above. Each block is one variant; the paper's answer-span and filter rules are given for its tasks, and a new task must choose its own [arxiv:2402.10200 §9]. Sampling one token, the core of the temperature and top-k/top-p variants: ```pseudocode # Temperature, top-k, then top-p sampling for one token [app. D] # [code: code/interactive.py#L1590-L1591] # Input: logits for the next token, temperature T, k, p # Output: one sampled token id def sample_token(logits, T=0.8, k=400, p=0.9): # Sharpen or flatten the distribution [app. D] logits = logits / T # Keep only the k highest-scoring tokens [app. D] logits = top_k_filter(logits, k) # Keep the smallest set whose probability reaches p [app. D] logits = top_p_filter(logits, p) return sample_from(softmax(logits)) ``` Notes from the dialogue setup: the order of the three steps follows the released code, and the paper gives 0.82 as the baseline temperature [app. D] [code: code/interactive.py#L1541]. CoT-decoding: ```pseudocode # CoT-decoding [arxiv:2402.10200 §2.2] [arxiv:2402.10200 §9] # Input: model, prompt "Q: [question]\nA:", k (default 10) # Output: the answer with the highest score def cot_decoding(model, prompt, k=10): # Take the k most likely tokens at the first step [§2.2] [§3] first_tokens = top_k_tokens(model, prompt, k) # Continue each branch greedily; 128 steps is the PaLM-2 budget [§9] paths = [greedy_decode(model, prompt, first=t, max_new=128) for t in first_tokens] # Drop empty, truncated and question-ending responses [§9] paths = [p for p in paths if is_well_formed(p)] # Find the answer span in each path [§2.2] [§9] for p in paths: p.answer = find_answer_span(p) # Mean top-1 minus top-2 probability over the answer tokens [§2.2] for p in paths: p.score = mean_margin(model, p, p.answer) # Max rule: the best-scoring path's answer [§2.2] [tab. 7] best = max(paths, key=lambda p: p.score) return best.answer ``` Notes from the paper's setup: - Math tasks take the last number as the answer for Mistral; PaLM-2 appends "So the answer is" and aligns that span with the path [arxiv:2402.10200 §9]. - Yes/no tasks compare the probability mass on "yes/true" with "no/false" instead of using the margin [arxiv:2402.10200 §9]. - For Mistral, outputs that look like training data are dropped [arxiv:2402.10200 §9]. Majority voting over samples, with final-answer comparison: ```pseudocode # maj1@k decoding [arxiv:2206.14858 §2.5] [arxiv:2206.14858 §2.3] # Input: model, problem text, k (samples) # Output: the most common final answer def maj1_at_k(model, problem, k): # Draw k independent solutions at T = 0.6, nucleus p = 0.95 [§2.3] responses = [sample(model, problem, T=0.6, p=0.95) for _ in range(k)] # Read the answer from the fixed "Final Answer" sentence [app. D.1] answers = [extract_final_answer(r) for r in responses] # Normalize answers before grouping; the paper's normalizer is in # app. D.1 [app. D.1]. Grouping rule and tie-breaking are not stated, # so the implementer must choose both [§2.5] answers = [normalize(a) for a in answers] return most_common(answers) ``` Beam search over routing tours, one beam per decoder: ```pseudocode # Customised beam search, one beam per decoder [arxiv:2012.10638 §3.1.2] # Input: M decoders; per-decoder beam size B # Output: the shortest complete tour found by any beam def beam_search(decoders, B): for each decoder m in decoders: # Beams stay separate per decoder, so tours stay diverse [§3.1.2] beam = [empty partial tour] while some partial tour is unfinished: # Extend every partial tour with its valid next nodes candidates = expand(beam, decoder_m) # Merge candidates that share the remaining subproblem and # are inferior [§3.1.2] [fig. 2] candidates = merge_inferior(candidates) # Keep the B best by accumulated log-probability [§3.1.2] beam = top_k(candidates, B) return shortest(all finished tours) ``` Notes from the routing setup: - The paper does not state how many successors each beam entry expands. The released search code keeps 3 per entry, so a 30-wide beam sees at most 90 candidates per step [code: nets/model_search.py#L165-L166]. Copy this constant if you want to reproduce the reported beam results. - Per-decoder beams of size B give M times B partial tours in total; the paper reports bs30 and bs50 as B per decoder [arxiv:2012.10638 §4.1]. Visual contrastive decoding, one token, as the paper defines it [arxiv:2311.16922 §3.3] [eq. 3] [eq. 4]: ```pseudocode # One token of visual contrastive decoding [arxiv:2311.16922 §3.3] # Input: image, prompt, prefix, alpha, beta, noised image # Output: next token id def vcd_next_token(model, image, noised, prompt, prefix, alpha, beta): # Logits from the clean image and from the noised copy [§3.3.1] logits = model.next_logits(image, prompt, prefix) logits_noised = model.next_logits(noised, prompt, prefix) # Keep tokens that are plausible under the clean image [§3.3.2] [eq. 4] probs = softmax(logits) keep = probs >= beta * max(probs) # Favour what the clean image supports over the noised one [eq. 3] contrast = (1 + alpha) * logits - alpha * logits_noised contrast = where(keep, contrast, -inf) return sample(softmax(contrast)) ``` MBR selection over N sampled action chunks, as in the robot paper [arxiv:2601.02295 §4.2] [eq. 4]: ```pseudocode # MBR selection over N sampled action chunks [arxiv:2601.02295 §4.2] [eq. 4] # Input: N chunks, each of shape (H, 9) # Output: the chunk to execute def mbr_select(chunks): # Turn each chunk into a path by summing its translation and rotation # deltas, so the feature vector has length 6H [§4.2] [app. C] paths = [to_path(c) for c in chunks] # Risk of a candidate: mean L2 distance to all candidates [eq. 4] risk = [mean([l2(p, q) for q in paths]) for p in paths] return chunks[argmin(risk)] ``` Notes from the robot setup: the released code does not use the plain rule by default. It uses a density-based variant [code: experiments/robot/libero/run_libero_eval_decomposed_progress_mbr.py#L894-L906]. The paper's text describes the plain rule above [arxiv:2601.02295 app. C], but it does not say which rule produced its tables [arxiv:2601.02295 §5.3]. Two more pseudocode blocks follow the same pattern. The first is the MoRT filter, which sits after top-k and top-p sampling: ```pseudocode # Toxicity filter after top-k and top-p sampling # [arxiv:2103.11790 § Guiding Generative Language Models using the MoralDirection] # Input: prefix text, LM logits for the next token, moral score, threshold t, minimum m # Output: one sampled token id def filtered_sample(lm, prefix, t, m, k=50, p=0.9): # Ordinary top-k, then top-p sampling gives the candidate set [§ Guiding ...] candidates = top_p_filter(top_k_filter(lm.next_logits(prefix), k), p) # Score each candidate by the whole text, prefix plus candidate [§ Guiding ...] scores = {c: moral_score(prefix + c) for c in candidates} # Remove candidates scored at or below t [§ Guiding ...] kept = [c for c in candidates if scores[c] > t] # Keep at least m candidates, the best-scoring ones [§ Guiding ...] if len(kept) < m: kept = sorted(candidates, key=lambda c: scores[c])[-m:] return sample(kept) ``` The second is greedy verification in SpecInfer, which the large model checks after one tree-parallel pass: ```pseudocode # Greedy verification over a token tree [arxiv:2305.09781 §4.3] [alg. 2] # Input: token tree from the draft model; large-model prediction at each node # Output: verified tokens, in order def verify_greedy(tree, llm_pred): verified = [] node = tree.root # Walk down while a child's token matches the large model's prediction here [§4.3] child = matching_child(tree, node, llm_pred[node]) while child is not None: verified.append(tree.token[child]) node = child child = matching_child(tree, node, llm_pred[node]) return verified ``` Contrastive search, as SimCTG defines it: ```pseudocode # Contrastive search [arxiv:2202.06417 eq. 5] [§3.2] # Input: context tokens, k (candidates), alpha (penalty weight), length L # Output: L new token ids def contrastive_search(model, context, k=8, alpha=0.6, L=128): for step in range(L): # Top-k next tokens and their model probabilities [§3.2] probs, cands = top_k_next_tokens(model, context, k) # Last-layer states of the context, reused for every candidate [§3.2] past = model.last_layer_states(context) scores = [] for p, v in zip(probs, cands): # Last-layer state of v when appended to the context [§3.2] h_v = model.last_layer_state(context + [v]) # Penalty: largest cosine to any earlier token state [eq. 5] penalty = max(cosine(h_v, h) for h in past) # Probability term minus weighted penalty [eq. 5] scores.append((1 - alpha) * p - alpha * penalty) # Emit the best-scoring candidate [eq. 5] context = context + [cands[argmax(scores)]] return context ``` Notes: the released code batches the k candidates with a key-value cache [code: simctg/utlisgpt.py#L31-L88]. The default k = 8 and α = 0.6 are the document setting [§4], not values the paper gives for every task. ## Contradictions Two within-paper gaps in the reproducibility study matter for implementation: - The reproducibility paper gives 2.2% of FP32 MATH500 examples diverging in its text, and 5.80% in its table for the same model [arxiv:2506.09501 §3.2] [app. C]. - Its random-sampling appendix repeats one model's values in another model's row [arxiv:2506.09501 app. E], which differs from the main table [arxiv:2506.09501 tab. 5]. Papers disagree on whether temperature changes accuracy: - Renze and Guven find no significant change in mean accuracy from temperature 0.0 to 1.0 on multiple-choice questions, and recommend 0.0 [arxiv:2402.05201 §3.1] [arxiv:2402.05201 §4.1]. Stochastic CHAOS argues that a single greedy run understates capability and risk. It reports that majority votes over sampled outputs beat greedy decoding on its GLUE, in-context learning and summarisation tests [arxiv:2601.07239 §2] [arxiv:2601.07239 §3] [arxiv:2601.07239 §4]. Minerva reports pass@1 of 26.5% with greedy decoding and 21.8% at temperature 0.6 on the 62B model [arxiv:2206.14858 tab. 8]. The tasks, metrics and models differ, so these sources do not settle which rule is better. Paper-to-code gaps, each with both sides cited: - Self-RAG's equation 3 adds the LM probability and the critique score. The long-form runner multiplies each child's score by its parent's, so the score is a product over segments [arxiv:2310.11511 eq. 3] [code: retrieval_lm/run_long_form_static.py#L231-L232]. - VCD's equation 2 applies a fixed per-step noise rate. The released code uses a one-shot schedule and never reads the rate [arxiv:2311.16922 eq. 2] [code: vcd_utils/vcd_add_noise.py#L7-L8]. - CycleVLA's equation 4 is the plain average-distance rule, and the released code defaults to a density-based variant [arxiv:2601.02295 eq. 4] [code: experiments/robot/libero/run_libero_eval_decomposed_progress_mbr.py#L894-L906]. - MoRT's paper sets m = 5 [arxiv:2103.11790 § Experimental setup], but the GPT-2 class defaults to 30 [code: realtoxicityprompts/generation/gpt2MCM_generation.py#L39]. The shipped run scripts pass 5, so the reported runs match the paper. The paper gives no top-p value, and the code uses 0.9 [code: realtoxicityprompts/generation/generation.py#L281]. - SpecInfer's tables use tree widths up to 5 [arxiv:2305.09781 tab. 2] [arxiv:2305.09781 tab. 3]. The released code sets the maximum beam width to 3 [code: include/flexflow/batch_config.h#L196]. The paper describes multi-step stochastic verification [arxiv:2305.09781 §4.3], but the released code's only verifier walks the tree by exact token match [code: src/runtime/request_manager.cc#L2581]. Its merge-based tree construction is also disabled in code [code: src/runtime/request_manager.cc#L2820-L2822]. - SimCTG's paper says the penalty uses the "last layer". The code takes the last entry of the model's hidden states, which in the Hugging Face GPT-2 code is after the final layer norm [code: simctg/utlisgpt.py#L51] [code: simctg/utlisgpt.py#L70]. The paper does not say this [arxiv:2202.06417 §3.2]. - BRIO's paper gives 16 candidates and no beam width for them [arxiv:2203.16804 app. B]. The released CNN/DailyMail script uses 16 beams in 16 groups with length penalty 2.0, and the XSum script uses 128 beams and keeps every 8th output [code: gen_candidate.py#L13-L18] [code: gen_candidate.py#L84-L86]. Its generation settings use 4 beams for CNN/DailyMail and 8 for XSum, with no-repeat 3-grams [code: config.py#L30] [code: config.py#L66] [code: main.py#L157]. The paper gives a default of 4 beams only for its CNN/DailyMail beam experiment [arxiv:2203.16804 tab. 5], and no XSum beam width. - Renze and Guven say the other sampling parameters were fixed [arxiv:2402.05201 §2.3], but the code sets only temperature, so provider defaults apply [code: source/models/openai_model.py#L28-L33]. A temperature result from this code therefore depends on each provider's default top-p. The papers test different setups, so their numbers are not set against each other either, apart from the temperature bullet above. Within the first paper, numbers for the same model differ across tables because the subsets and setups differ. For example, PaLM-2 L greedy decoding is 44.0% on a top-100 GSM8K subset [arxiv:2402.10200 tab. 2] and 34.8% on the full test set [arxiv:2402.10200 tab. 7]. Two paper-to-code gaps are listed in the variants rather than here, because they are not disagreements between papers: the INSPIRED baseline temperature (0.82 in the paper, not in the code) [arxiv:2009.14306 app. D] [code: code/interactive.py#L1541], and the number of candidates (five in the paper, three in the code) [arxiv:2009.14306 §6.2] [code: code/interactive.py#L1539]. ## Papers - [Survey of controllable generation with pre-trained transformers](/papers/2201.05337): a survey that files guided and trainable decoding-time methods under post-processing. It runs no experiments. - [A Contrastive Framework for Neural Text Generation](/papers/2202.06417): SimCTG, a contrastive training loss plus contrastive search, which picks among the top-k tokens with a similarity penalty. - [BRIO: Bringing Order to Abstractive Summarization](/papers/2203.16804): a summariser trained with a ranking loss over beam candidates, decoded by length-normalised beam search, with a scorer variant that ranks candidates. - [Chain-of-Thought Reasoning Without Prompting](/papers/2402.10200): CoT-decoding, which branches on the top-k first tokens and picks the answer with the widest top-1 vs top-2 margin. Compares against greedy, sampling baselines and self-consistency. - [DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task](/papers/2304.01097): LoRA fine-tuning of ChatGLM-6B for Chinese medical dialogue. Its decoding content is one temperature and top-p grid, graded by a doctor on one question. - [INSPIRED: Toward Sociable Recommendation Dialog Systems](/papers/2009.14306): a GPT-2 recommender that samples candidate replies with temperature, top-k and top-p, then picks one at random after a bigram filter. - [Multi-Decoder Attention Model with Embedding Glimpse for Solving Vehicle Routing Problems](/papers/2012.10638): beam search over routing tours with one beam per decoder, compared with greedy and sampling for the same learned policy. - [Solving Quantitative Reasoning Problems with Language Models](/papers/2206.14858): Minerva, which answers maths with greedy decoding for single answers and majority voting over T = 0.6 samples (maj1@k) for headline numbers. - [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](/papers/2310.11511): one model that emits reflection tokens, with a segment-level beam scored by those tokens at inference. - [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](/papers/2311.16922): training-free contrast between the clean image and a noised copy, with a plausibility cutoff on each token. - [Large Language Models: A Survey](/papers/2402.06196): a broad survey whose section on decoding gives temperature, top-k and nucleus sampling. It has no experiments of its own, and its tables are compiled from other papers. - [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](/papers/2506.09501): shows that BF16 greedy outputs vary with GPU count, GPU type and batch size, and proposes LayerCast, which stores weights in BF16 and computes in FP32. - [CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decoding](/papers/2601.02295): MBR selection over sampled action chunks, used after a backtrack in a robot policy. - [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](/papers/2103.11790): a moral direction in BERT embeddings, used as a candidate filter after top-k and top-p sampling in GPT-2. - [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](/papers/2305.09781): a draft model proposes a token tree, and the large model verifies it in one pass with tree attention. - [The Effect of Sampling Temperature on Problem Solving in Large Language Models](/papers/2402.05201): temperature sweeps from 0.0 to 1.0 on multiple-choice questions, with no significant accuracy change in that range and a recommendation of 0.0. - [Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition](/papers/2601.07239): argues for reporting sampled outputs next to a greedy score, with stress tests on GLUE, in-context learning, summarisation, chain-of-thought and safety. - [A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models](/papers/2401.01313): a survey that files CAD, DoLa and ITI under decoding, with a table of 32 techniques and no experiments. - [Energy benchmark for LLaMA inference on V100 and A100 GPUs](/papers/2310.03003): energy and GPU use for LLaMA inference at 7B to 65B on V100 and A100 GPUs, with temperature 0.8 and top-p 0.95 fixed. - [Ad Headline Generation using Self-Critical Masked Language Model](/papers/2607.06818): a BERT-Large masked language model fine-tuned with self-critical policy gradient, decoding by length-normalised beam search.