Speculative decoding

Overview

Autoregressive decoding makes one serial call to the target model per token. At small batch sizes each call is limited by memory bandwidth (all weights and the KV cache are streamed for one token), so compute sits idle 2211.17192 §1 2302.01318 §3 2401.10774 §1. Speculative decoding spends that idle compute. Every variant runs the same three steps:

1. Draft. A cheap mechanism proposes \gamma (also called K) future tokens, or a tree or block of them.
2. Verify. The target scores every drafted position in one forward pass. Scoring a few extra positions costs about as much as generating one token, because linear layers stay memory-bound, KV-cache reads do not grow with K, and all-reduces stay latency-bound 2302.01318 §4.1 2211.17192 §3.3.
3. Accept. Keep a prefix of the draft and add one token taken from the target (a correction or a bonus token). Each iteration therefore yields between 1 and \gamma+1 tokens, and never needs more serial target calls than plain decoding 2211.17192 §2.1 2302.01318 §4.2.

A survey places speculative decoding among the ways to split work between a small and a large model. The others are cascading (answer small first, escalate on a failed check) and routing (pick one model per query). It gives no speed comparison between them 2409.06857 §3.1.3.

Shared invariant (lossless variants). Accept draft token x with probability \min(1, p(x)/q(x)), where p is the target and q the drafter. On the first rejection, resample from \mathrm{norm}(\max(0, p - q)). The output is then distributed exactly as if sampled from the target alone, whatever drafter is used 2211.17192 §2.3 2211.17192 app. A.1 2302.01318 §4.2. The drafter only affects speed, through the acceptance rate \alpha = E[\sum_x \min(p(x), q(x))] 2211.17192 §3.2.

Notation warning. 2302.01318 uses the opposite letters: q is the target and p the draft, so its rule is written \min(1, q/p) with residual (q-p)_+ 2302.01318 §4.2. This page uses the 2211.17192 convention (p = target, q = drafter).

Expected gain 2211.17192 §3.1 2211.17192 §3.3. Assuming i.i.d. acceptance, one iteration yields \frac{1-\alpha^{\gamma+1}}{1-\alpha} tokens in expectation. Let c be the cost of one drafter call relative to one target call. The walltime improvement is then \frac{1-\alpha^{\gamma+1}}{(1-\alpha)(\gamma c+1)}. If \alpha > c, some \gamma gives at least (1+\alpha)/(1+c). Total arithmetic work goes up, not down 2211.17192 §3.4 2211.17192 tab. 1.

Variants

The variants below differ mainly in where the draft comes from and whether the draft is a chain, a tree or a block. Verification is always one target pass.

Separate draft model (speculative decoding / speculative sampling)

A smaller autoregressive model with the same tokenizer drafts a chain of \gamma tokens. The target scores all \gamma+1 prefixes in one call, and the rule above accepts a prefix 2211.17192 alg. 1 2302.01318 alg. 2. Two groups developed the same idea independently:

• Leviathan et al. Off-the-shelf T5 drafters for T5-XXL 11B, batch size 1, one TPU-v4. With T5-small (77M) drafting they report 2.3X-3.4X walltime over T5X: 3.4X on EnDe at temp 0 with \gamma=7 and \alpha=0.75, and 2.3X on CNNDM at temp 1 2211.17192 tab. 2. The larger T5-large drafter is the slowest at every setting, because its cost c outweighs its higher \alpha 2211.17192 tab. 2. A drafter about two orders of magnitude smaller than the target usually worked best. Near-free drafters (a bigram table, c=0) still gave 1.25X 2211.17192 §3.6 2211.17192 §4.2. The paper also derives the expected-gain theory above and a lossy "lenience" extension 2211.17192 app. A.5. Card: Fast Inference from Transformers via Speculative Decoding.
• Chen et al. (SpS). Chinchilla 70B target with a custom 4B draft, K=4, both on 16 TPU v4, batch size 1. Per-token latency drops from 14.1 ms to 7.52 / 7.00 / 5.73 ms, a speedup of 1.92x on XSum nucleus, 2.01x on XSum greedy and 2.46x on HumanEval. Benchmark scores stay at parity 2302.01318 tab. 1. The key engineering point is that in distributed serving the draft should be wide and shallow (8 layers, d_\text{model} 6144). Fewer layers means fewer all-reduces, so the draft can share the target's topology 2302.01318 §6 2302.01318 tab. 2. Card: Accelerating Large Language Model Decoding with Speculative Sampling.

Token tree from small models, served at scale (SpecInfer)

A small speculative model (SSM) proposes a token tree rather than a chain. The tree is built by expansion from one SSM, with a fixed number of children per step, such as <1, 1, 3, 1, 1, 1, 1, 1> in the evaluation 2305.09781 §3 2305.09781 §6.1. The construction can also merge trees from several SSMs or fine-tune SSMs by boosting. The paper describes both, but the evaluation uses one SSM 2305.09781 §3 2305.09781 §6.1. The target scores every tree node in one pass. A topology-aware causal mask lets each node see only its ancestors and the verified prefix, and a shared key-value cache is written in depth-first order 2305.09781 §4.2 fig. 4. Greedy verification walks the tree by exact match with the target's own predictions 2305.09781 §4.3 alg. 2. The paper also gives a multi-step stochastic rule, which it states leaves the output distribution equal to the target's 2305.09781 §4.3. The release does not implement that rule (see Contradictions).

Results are author-reported and measured in a serving system, against serving baselines and not against other speculative methods. Distributed serving on one node is 1.5-2.5x faster than incremental-decoding systems, and two nodes give 2.4-2.8x 2305.09781 §6.2. Offloading OPT-13B and OPT-30B on one A10 is 2.6-3.5x faster than FlexGen 2305.09781 §6.3. Within the same system, a tree beats a single sequence by 1.2-1.5x 2305.09781 §6.2. These figures cannot be compared with the T5 or Chinchilla numbers above.

Two things narrow the reuse of the paper's setup. The paper's width-5 settings appear in its tables 2305.09781 tab. 2 2305.09781 tab. 3, but the release caps width at 3. The release also disables merge-based construction, and it does not include boost-tuning code. Its verification is greedy only. These points come from checking the released code, not from the paper, so see the SpecInfer card for the file references.

Extra decoding heads on the target (Medusa)

No separate model. K one-layer residual heads read the target's last hidden state, and head k predicts the token k+1 positions ahead 2401.10774 §2.1.1. The heads' top-k outputs form a static sparse candidate tree (64 nodes, depth 4 in the release). The backbone verifies the whole tree in one pass using a tree attention mask, with position id = prefix length + depth 2401.10774 §2.1.2 2401.10774 §2.3.3 2401.10774 app. C. Because the heads sit on the backbone, the verification pass also yields the next step's drafts, so there is no separate drafting pass 2401.10774 §2.

• Medusa-1 trains only the heads on a frozen backbone, which may be quantized. It is lossless under greedy or rejection-sampling acceptance. Reported speedup on Vicuna-7B / 13B is 2.18x / 2.33x 2401.10774 §2.2.1 2401.10774 §3.1.
• Medusa-2 trains the heads jointly with a LoRA'd backbone, which changes the model. It reaches 2.83x on Vicuna-7B and 13B, with MT-Bench deltas of -0.14 to +0.05 2401.10774 tab. 1. Self-distillation provides training data when the original SFT data is unavailable 2401.10774 §2.3.2.
• Typical acceptance is an optional lossy rule. It accepts candidates whose target probability exceeds an entropy-dependent threshold, trading quality for acceptance length 2401.10774 §2.3.1 2401.10774 §3.3.2.

Setup: MT-Bench, batch size 1, baseline is the default HuggingFace implementation, GPU not stated 2401.10774 §3. Card: Medusa.

Shared-weight MTP head reused as a draft (Nemotron 3 Super)

Multi-token prediction (MTP) trains the model to predict several future tokens. Standard MTP uses one independent head per offset, which caps the draft length and, if one head is reused recursively, causes a train-inference mismatch. Nemotron 3 Super instead shares one head's weights across all offsets, so the same head can run recursively at inference 2604.12374 §2.1.2. The draft is a chain, and the main model verifies it in one pass. Acceptance still falls with draft depth, but more slowly than with independent heads 2604.12374 §2.1.2 tab. 2. The paper's MTP acceptance length averages 3.45 over SPEED-Bench at draft length 7, author-reported. This is an acceptance length, not a walltime speedup 2604.12374 tab. 2. The paper's headline throughput figures compare whole models, so they do not isolate the draft 2604.12374 §1. The heads are trained during SFT 2604.12374 §3.1. A final MTP-healing stage, after RL, trains only the heads, with the rest of the network frozen 2604.12374 §3.2. Check that the heads you deploy went through this stage before you rely on their acceptance. Card: Nemotron 3 Super.

Parallel block drafter (DFlash)

The draft is a small, non-autoregressive model. Each round starts from an anchor, the target's last accepted or bonus token, followed by B-1 mask tokens, and fills all the masks in one forward pass. The target's hidden states from several layers are fused into one context feature. That feature is not added to the draft's input. Each draft layer projects it into extra keys and values, so the drafter reads it as memory 2602.06036 §4.1 2602.06036 app. A.3. The target then verifies the block as in the chain case. The authors call the output lossless 2602.06036 §1. The released code uses exact-match acceptance at temperature 0 and rejection sampling above it (see the DFlash card).

Training keeps the target frozen and uses target-generated responses. Random anchors are drawn per sequence, and the loss weights early block positions more, because an early error invalidates the rest of the block 2602.06036 §4.2 2602.06036 §5.

Results are author-reported, with the Transformers backend on one H200 and greedy decoding. Qwen3-4B averages 4.9x over seven tasks and peaks at 6.1x on MATH-500. The EAGLE-3 baseline with tree size 16 (EAGLE-3, no card) averages 1.8x on the same tasks [arxiv:2602.06036 tab. 1]. On MT-Bench, a chat task, DFlash reaches 2.85x 2602.06036 tab. 1. Card: DFlash: Block Diffusion for Flash Speculative Decoding.

Lossy relaxations (any drafter)

These variants give up exactness for longer accepted prefixes:

• Leviathan's lenience l<1 scales q before the comparison. On EnDe with T5-small at temp 1, speedup goes from 2.5X at l=1 to 5X at l=0.1 2211.17192 app. A.5.
• Medusa's typical acceptance (above) and Medusa-2's backbone fine-tuning are also lossy 2401.10774 §2.3.1 2401.10774 tab. 1.

When to use which

Speedups in this section come from different targets, hardware, baselines and benchmarks: T5X on TPU-v4, a custom TPU serving stack, HuggingFace on GPUs, and Transformers or SGLang on H200 and B200 GPUs. Do not rank methods by comparing headline numbers across papers. The only head-to-head comparisons are inside a single paper, and those are noted below.

Implementation sketch

state: target T, drafter D, accepted prefix x, KV caches for T (and D if separate)

loop until done:
  # 1. Draft
  chain variants: for i in 1..gamma: q_i = D(x + d_1..d_{i-1}); d_i ~ q_i
                  (MTP: D is one shared head, applied recursively)
  tree variants (Medusa): expand a static tree from head top-k / samples
  block (DFlash): one pass, anchor + B-1 masks, target features as extra KV

  # 2. Verify: ONE target forward over all drafted positions
  chain: p_1..p_{gamma+1} = T(x), T(x+d_1), ..., T(x+d_1..d_gamma)    # causal mask
  tree:  flatten nodes after the KV cache; mask = prefix + own ancestors; pos_id = len(x) + depth

  # 3. Accept (lossless rule; apply temperature/top-k/top-p to BOTH p and q first)
  n = 0
  for i in 1..gamma:
      if U(0,1) < min(1, p_i(d_i) / q_i(d_i)): n += 1
      else: break
  if n < gamma: t ~ norm(max(0, p_{n+1} - q_{n+1}))   # guard: normalizer can be ~0
  else:         t ~ p_{gamma+1}                        # bonus token, no extra target call
  tree: at each node, try children in turn (multi-round); pick the longest accepted path

  # 4. Commit
  x = x + d_1..d_n + [t]
  roll back / compact KV entries of rejected positions (target and drafter)
  Medusa: reuse the accepted path's head logits as the next draft

How the variants differ:

• Greedy decoding. With argmax standardization both distributions are one-hot, so the rule reduces to "accept iff draft token == target argmax". The cards infer this from 2211.17192 §2.2. Medusa uses the same exact-match test 2401.10774 §2.3.1.
• Choosing \gamma. Maximize the walltime formula numerically for your measured \alpha and c. c depends on the hardware, so measure it on your own stack 2211.17192 §3.3 2211.17192 §3.5. Chen et al. found that speedup plateaus or regresses at large K, with the best K=3 on XSum nucleus 2302.01318 §6.3.
• Tree size. For Medusa, acceleration grows roughly logarithmically with candidate count, and speed falls past about 64 candidates 2401.10774 §3.3.1 2401.10774 app. G.3.
• Block size (DFlash). Train and test with the same block size. On Math500, a draft trained at 16 and tested at 8 reaches 3.87x, against 4.64x when both are 16 2602.06036 tab. 8.
• KV-cache rollback. None of the four papers specifies it in the main text. Medusa's code compacts the accepted path's KV entries into contiguous slots (see Medusa, Pitfalls). DFlash keeps target-feature entries in each draft layer's KV cache for the whole sequence, and that memory is not quantified 2602.06036 §4.1 2602.06036 app. A.3.

Contradictions

• Speculative-decoding baseline vs Medusa. Medusa reports classic speculative decoding with open small drafters (Llama-68M, Tiny-Vicuna-1B) at only 1.47x / 1.56x / 1.60x on Vicuna 7B / 13B / 33B 2401.10774 tab. 1 2401.10774 app. D. The original papers report 2.3X-3.4X on T5-XXL with a T5-small drafter 2211.17192 tab. 2 and 1.92x-2.46x on Chinchilla 70B 2302.01318 tab. 1, using purpose-picked or purpose-built drafters on TPU stacks. These numbers do not refute each other. They measure different drafter quality, targets and hardware.
• Does rejection sampling help Medusa? The Medusa paper presents rejection sampling as an acceptance option that does not raise acceleration 2401.10774 §2. Its released code does not implement it, only greedy, typical and nucleus acceptance (see the Medusa card).
• Medusa's headline range. The arXiv abstract metadata says Medusa-2 reaches 2.3-3.6x. The LaTeX abstract says 2.3-2.8x. The 3.62x figure is a single MT-Bench category 2401.10774 §3.1.
• Gains at high concurrency. Medusa's simulation predicts losses above batch size 32 2401.10774 app. G.3. DFlash reports 2.2x-3.1x at concurrency 32 for Qwen3 models on SGLang 2602.06036 tab. 3. The two measure different things, a simulated batch size against concurrent requests on a serving stack, so neither result is resolved here.
• Batch size within SpecInfer. The paper says its gains shrink as concurrent requests grow, and that wide trees are worse at large batch, where width 2 or 3 is best 2305.09781 §6.2 2305.09781 §6.4. Its tree-parallel versus sequence comparison is on par at small batch and up to 1.8x faster at large batch 2305.09781 §6.5. The two comparisons are different, and the paper does not reconcile them in the text, so neither is resolved here.
• Stochastic verification in SpecInfer. The paper's multi-step rule leaves the output distribution equal to the target's 2305.09781 §4.3. Its released code has only greedy verification, so the claim is not checked by the release. Treat it as the paper's claim until a stochastic implementation is tested (see the SpecInfer card).
• Acceptance length vs walltime. Nemotron 3 Super reports MTP acceptance length, averaging 3.45 at draft length 7 on SPEED-Bench 2604.12374 tab. 2. Medusa and DFlash report walltime speedups on other benchmarks 2401.10774 §3.1 2602.06036 tab. 1. Acceptance length does not give speedup without the drafter cost c, and the benchmarks differ, so these numbers are not comparable.

Papers

• Fast Inference from Transformers via Speculative Decoding: original draft-model method with the exactness proof, acceptance-rate and walltime theory, and T5-XXL results.
• Accelerating Large Language Model Decoding with Speculative Sampling: concurrent SpS. Distributed serving at 70B scale with a wide, shallow draft model.
• SpecInfer: token trees from small models, verified in one tree-attention pass in a serving system. Reports speedups against serving baselines, with greedy verification in the release.
• Medusa: draft-model-free decoding with extra heads on the target, tree attention verification, and lossy typical acceptance.
• DFlash: Block Diffusion for Flash Speculative Decoding: parallel block drafter that injects target features as keys and values in every draft layer. Author-reported speedups on Qwen3 and LLaMA-3.1 models.
• Nemotron 3 Super: hybrid MoE model whose shared-weight MTP head doubles as a recursive speculative draft. Reports MTP acceptance length, not walltime speedup.
• Survey of small models alongside LLMs: classifies speculative decoding among cascading and routing. Gives the draft-then-verify shape, not the acceptance rule.