---
kind: concept
slug: speculative-decoding
title: Speculative decoding
summary: >-
  Cuts per-token latency of memory-bound LLM decoding: a cheap drafter proposes several tokens, the target
  model checks them all in one forward pass, and an accept/resample rule keeps the target's output
  distribution. Drafters range from a small separate model to target-side heads or one parallel block. The cost is
  extra compute and memory, so the gains shrink as batch size grows.
tags: [inference, decoding, sampling, latency]
papers: ["2211.17192", "2302.01318", "2305.09781", "2401.10774", "2409.06857", "2602.06036", "2604.12374"]
related: [kv-cache-management, llm-serving, decoding-strategies, weight-quantization, low-rank-adaptation, discrete-diffusion, model-routing]
status: draft
updated_at: 2026-10-31
---

## Overview

Autoregressive decoding makes one serial call to the target model per token. At small batch sizes each
call is limited by memory bandwidth (all weights and the KV cache are streamed for one token), so compute
sits idle [arxiv:2211.17192 §1] [arxiv:2302.01318 §3] [arxiv:2401.10774 §1]. Speculative decoding spends
that idle compute. Every variant runs the same three steps:

1. **Draft.** A cheap mechanism proposes $\gamma$ (also called $K$) future tokens, or a tree or block of them.
2. **Verify.** The target scores every drafted position in **one** forward pass. Scoring a few extra
   positions costs about as much as generating one token, because linear layers stay memory-bound,
   KV-cache reads do not grow with $K$, and all-reduces stay latency-bound
   [arxiv:2302.01318 §4.1] [arxiv:2211.17192 §3.3].
3. **Accept.** Keep a prefix of the draft and add one token taken from the target (a correction or a
   bonus token). Each iteration therefore yields between 1 and $\gamma+1$ tokens, and never needs more
   serial target calls than plain decoding [arxiv:2211.17192 §2.1] [arxiv:2302.01318 §4.2].

A survey places speculative decoding among the ways to split work between a small and a large model. The others are cascading (answer small first, escalate on a failed check) and routing (pick one model per query). It gives no speed comparison between them [arxiv:2409.06857 §3.1.3].

**Shared invariant (lossless variants).** Accept draft token $x$ with probability
$\min(1, p(x)/q(x))$, where $p$ is the target and $q$ the drafter. On the first rejection, resample from
$\mathrm{norm}(\max(0, p - q))$. The output is then distributed exactly as if sampled from the target
alone, whatever drafter is used [arxiv:2211.17192 §2.3] [arxiv:2211.17192 app. A.1]
[arxiv:2302.01318 §4.2]. The drafter only affects speed, through the acceptance rate
$\alpha = E[\sum_x \min(p(x), q(x))]$ [arxiv:2211.17192 §3.2].

**Notation warning.** [arxiv:2302.01318] uses the opposite letters: $q$ is the *target* and $p$ the
*draft*, so its rule is written $\min(1, q/p)$ with residual $(q-p)_+$ [arxiv:2302.01318 §4.2]. This
page uses the [arxiv:2211.17192] convention ($p$ = target, $q$ = drafter).

**Expected gain** [arxiv:2211.17192 §3.1] [arxiv:2211.17192 §3.3]. Assuming i.i.d. acceptance, one
iteration yields $\frac{1-\alpha^{\gamma+1}}{1-\alpha}$ tokens in expectation. Let $c$ be the cost of one
drafter call relative to one target call. The walltime improvement is then
$\frac{1-\alpha^{\gamma+1}}{(1-\alpha)(\gamma c+1)}$. If $\alpha > c$, some $\gamma$ gives at least
$(1+\alpha)/(1+c)$. Total arithmetic work goes up, not down
[arxiv:2211.17192 §3.4] [arxiv:2211.17192 tab. 1].

## Variants

The variants below differ mainly in **where the draft comes from** and **whether the draft is a chain,
a tree or a block**. Verification is always one target pass.

### Separate draft model (speculative decoding / speculative sampling)

A smaller autoregressive model with the same tokenizer drafts a chain of $\gamma$ tokens. The target
scores all $\gamma+1$ prefixes in one call, and the rule above accepts a prefix
[arxiv:2211.17192 alg. 1] [arxiv:2302.01318 alg. 2]. Two groups developed the same idea independently:

- **Leviathan et al.** Off-the-shelf T5 drafters for T5-XXL 11B, batch size 1, one TPU-v4. With
  T5-small (77M) drafting they report 2.3X-3.4X walltime over T5X: 3.4X on EnDe at temp 0 with
  $\gamma=7$ and $\alpha=0.75$, and 2.3X on CNNDM at temp 1
  [arxiv:2211.17192 tab. 2]. The larger T5-large drafter is the slowest at every setting, because its
  cost $c$ outweighs its higher $\alpha$ [arxiv:2211.17192 tab. 2]. A drafter about two orders of
  magnitude smaller than the target usually worked best. Near-free drafters (a bigram table, $c=0$)
  still gave 1.25X [arxiv:2211.17192 §3.6] [arxiv:2211.17192 §4.2]. The paper also derives the
  expected-gain theory above and a lossy "lenience" extension [arxiv:2211.17192 app. A.5].
  Card: [Fast Inference from Transformers via Speculative Decoding](/papers/2211.17192).
- **Chen et al. (SpS).** Chinchilla 70B target with a custom 4B draft, $K=4$, both on 16 TPU v4,
  batch size 1. Per-token latency drops from 14.1 ms to 7.52 / 7.00 / 5.73 ms, a speedup of
  1.92x on XSum nucleus, 2.01x on XSum greedy and 2.46x on HumanEval. Benchmark scores stay at parity
  [arxiv:2302.01318 tab. 1]. The key engineering point is that in distributed serving the draft should
  be **wide and shallow** (8 layers, $d_\text{model}$ 6144). Fewer layers means fewer all-reduces, so
  the draft can share the target's topology [arxiv:2302.01318 §6] [arxiv:2302.01318 tab. 2].
  Card: [Accelerating Large Language Model Decoding with Speculative Sampling](/papers/2302.01318).

### Token tree from small models, served at scale (SpecInfer)

A small speculative model (SSM) proposes a **token tree** rather than a chain. The tree is built by
expansion from one SSM, with a fixed number of children per step, such as `<1, 1, 3, 1, 1, 1, 1, 1>`
in the evaluation [arxiv:2305.09781 §3] [arxiv:2305.09781 §6.1]. The construction can also merge trees
from several SSMs or fine-tune SSMs by boosting. The paper describes both, but the evaluation uses one
SSM [arxiv:2305.09781 §3] [arxiv:2305.09781 §6.1]. The target scores every tree node in one pass. A
topology-aware causal mask lets each node see only its ancestors and the verified prefix, and a shared
key-value cache is written in depth-first order [arxiv:2305.09781 §4.2] [fig. 4]. Greedy verification
walks the tree by exact match with the target's own predictions [arxiv:2305.09781 §4.3] [alg. 2]. The
paper also gives a multi-step stochastic rule, which it states leaves the output distribution equal to
the target's [arxiv:2305.09781 §4.3]. The release does not implement that rule (see Contradictions).

Results are author-reported and measured in a serving system, against serving baselines and not against
other speculative methods. Distributed serving on one node is 1.5-2.5x faster than incremental-decoding
systems, and two nodes give 2.4-2.8x [arxiv:2305.09781 §6.2]. Offloading OPT-13B and OPT-30B on one A10
is 2.6-3.5x faster than FlexGen [arxiv:2305.09781 §6.3]. Within the same system, a tree beats a single
sequence by 1.2-1.5x [arxiv:2305.09781 §6.2]. These figures cannot be compared with the T5 or Chinchilla
numbers above.

Two things narrow the reuse of the paper's setup. The paper's width-5 settings appear in its tables
[arxiv:2305.09781 tab. 2] [arxiv:2305.09781 tab. 3], but the release caps width at 3. The release also
disables merge-based construction, and it does not include boost-tuning code. Its verification is greedy
only. These points come from checking the released code, not from the paper, so see the
[SpecInfer](/papers/2305.09781) card for the file references.

### Extra decoding heads on the target (Medusa)

No separate model. $K$ one-layer residual heads read the target's last hidden state, and head $k$
predicts the token $k+1$ positions ahead [arxiv:2401.10774 §2.1.1]. The heads' top-k outputs form a
static sparse candidate tree (64 nodes, depth 4 in the release). The backbone verifies the whole tree in
one pass using a tree attention mask, with position id = prefix length + depth
[arxiv:2401.10774 §2.1.2] [arxiv:2401.10774 §2.3.3] [arxiv:2401.10774 app. C]. Because the heads sit on
the backbone, the verification pass also yields the next step's drafts, so there is no separate drafting
pass [arxiv:2401.10774 §2].

- **Medusa-1** trains only the heads on a frozen backbone, which may be quantized. It is lossless under
  greedy or rejection-sampling acceptance. Reported speedup on Vicuna-7B / 13B is 2.18x / 2.33x
  [arxiv:2401.10774 §2.2.1] [arxiv:2401.10774 §3.1].
- **Medusa-2** trains the heads jointly with a LoRA'd backbone, which changes the model. It reaches
  2.83x on Vicuna-7B and 13B, with MT-Bench deltas of -0.14 to +0.05 [arxiv:2401.10774 tab. 1].
  Self-distillation provides training data when the original SFT data is unavailable
  [arxiv:2401.10774 §2.3.2].
- **Typical acceptance** is an optional *lossy* rule. It accepts candidates whose target probability
  exceeds an entropy-dependent threshold, trading quality for acceptance length
  [arxiv:2401.10774 §2.3.1] [arxiv:2401.10774 §3.3.2].

Setup: MT-Bench, batch size 1, baseline is the default HuggingFace implementation, GPU not stated
[arxiv:2401.10774 §3]. Card: [Medusa](/papers/2401.10774).

### Shared-weight MTP head reused as a draft (Nemotron 3 Super)

Multi-token prediction (MTP) trains the model to predict several future tokens. Standard MTP uses one
independent head per offset, which caps the draft length and, if one head is reused recursively, causes
a train-inference mismatch. Nemotron 3 Super instead shares one head's weights across all offsets, so
the same head can run recursively at inference [arxiv:2604.12374 §2.1.2]. The draft is a chain, and the
main model verifies it in one pass. Acceptance still falls with draft depth, but more slowly than with
independent heads [arxiv:2604.12374 §2.1.2] [tab. 2]. The paper's MTP acceptance length averages 3.45
over SPEED-Bench at draft length 7, author-reported. This is an acceptance length, not a walltime
speedup [arxiv:2604.12374 tab. 2]. The paper's headline throughput figures compare whole models, so they
do not isolate the draft [arxiv:2604.12374 §1]. The heads are trained during SFT [arxiv:2604.12374 §3.1].
A final MTP-healing stage, after RL, trains only the heads, with the rest of the network frozen
[arxiv:2604.12374 §3.2]. Check that the heads you deploy went through this stage before you rely on their
acceptance. Card:
[Nemotron 3 Super](/papers/2604.12374).

### Parallel block drafter (DFlash)

The draft is a small, non-autoregressive model. Each round starts from an anchor, the target's last
accepted or bonus token, followed by $B-1$ mask tokens, and fills all the masks in one forward pass. The
target's hidden states from several layers are fused into one context feature. That feature is not added
to the draft's input. Each draft layer projects it into extra keys and values, so the drafter reads it as
memory [arxiv:2602.06036 §4.1] [arxiv:2602.06036 app. A.3]. The target then verifies the block as in the
chain case. The authors call the output lossless [arxiv:2602.06036 §1]. The released code uses exact-match
acceptance at temperature 0 and rejection sampling above it (see the [DFlash](/papers/2602.06036) card).

Training keeps the target frozen and uses target-generated responses. Random anchors are drawn per
sequence, and the loss weights early block positions more, because an early error invalidates the rest of
the block [arxiv:2602.06036 §4.2] [arxiv:2602.06036 §5].

Results are author-reported, with the Transformers backend on one H200 and greedy decoding. Qwen3-4B
averages 4.9x over seven tasks and peaks at 6.1x on MATH-500. The EAGLE-3 baseline with tree size 16
([EAGLE-3](https://arxiv.org/abs/2503.01840), no card) averages 1.8x on the same tasks [arxiv:2602.06036
tab. 1]. On MT-Bench, a chat task, DFlash reaches 2.85x [arxiv:2602.06036 tab. 1].
Card: [DFlash: Block Diffusion for Flash Speculative Decoding](/papers/2602.06036).

### Lossy relaxations (any drafter)

These variants give up exactness for longer accepted prefixes:

- Leviathan's lenience $l<1$ scales $q$ before the comparison. On EnDe with T5-small at temp 1, speedup
  goes from 2.5X at $l=1$ to 5X at $l=0.1$ [arxiv:2211.17192 app. A.5].
- Medusa's typical acceptance (above) and Medusa-2's backbone fine-tuning are also lossy
  [arxiv:2401.10774 §2.3.1] [arxiv:2401.10774 tab. 1].

## When to use which

Speedups in this section come from different targets, hardware, baselines and benchmarks: T5X on
TPU-v4, a custom TPU serving stack, HuggingFace on GPUs, and Transformers or SGLang on H200 and B200 GPUs.
Do **not** rank methods by comparing headline numbers across papers. The only head-to-head comparisons are
inside a single paper, and those are noted below.

| Situation | Use | Why |
|---|---|---|
| A smaller model with the same tokenizer exists, and you cannot train anything | Separate draft model | No training. Any drafter is lossless. A drafter about 100x smaller usually worked best [arxiv:2211.17192 §3.6] [arxiv:2211.17192 tab. 2] |
| No same-family small model exists, or the only one is too large | Medusa-1 (train heads) | Heads train on the frozen target, in hours to days on one node [arxiv:2401.10774 §2.2.1] |
| Output must exactly match the target's distribution | Draft model, DFlash, or Medusa-1 with greedy / rejection acceptance | Medusa-2, typical acceptance and lenience $l<1$ are lossy [arxiv:2401.10774 §2.3.1] [arxiv:2211.17192 app. A.5]. DFlash's lossless claim is the authors' [arxiv:2602.06036 §1] |
| Can fine-tune the backbone and accept small quality drift | Medusa-2 | 2.83x vs 2.18x for Medusa-1 on Vicuna-7B, same paper and setup [arxiv:2401.10774 §3.1] [arxiv:2401.10774 tab. 1] |
| Backbone is quantized or memory-limited | Medusa-1 | Heads train on a frozen, quantized backbone [arxiv:2401.10774 §2.2.1] |
| Distributed, model-parallel serving | Wide, shallow draft on the target's topology | Fewer layers means fewer all-reduces [arxiv:2302.01318 §6] |
| You run a serving system and can change its attention kernel, with a same-family SSM available | Token tree from SSMs (SpecInfer) | Author-reported 1.5-2.8x distributed and 2.6-3.5x offloaded, against serving baselines [arxiv:2305.09781 §6.2] [arxiv:2305.09781 §6.3]. The release verifies greedily only and caps width at 3 (see the card) |
| Batch size 1, a training budget for a feature-conditioned draft, and fewest serial draft passes | Parallel block drafter (DFlash) | One draft pass per round instead of one per token. Author-reported 4.9x greedy average on Qwen3-4B, but only 2.85x on chat [arxiv:2602.06036 tab. 1] |
| The model ships with MTP heads and you want no separate drafter | Shared MTP head, applied recursively as a chain | Nothing extra to train or serve. Acceptance length falls with depth, so measure it at your draft length [arxiv:2604.12374 §2.1.2] [tab. 2]. Check the heads went through MTP healing [arxiv:2604.12374 §3.2] |
| Large batches / throughput-bound serving | Gains are uncertain and may shrink. Measure first | Spare compute disappears [arxiv:2211.17192 §3.4]. Medusa's simulation predicts losses above bs 32 [arxiv:2401.10774 app. G.3]. DFlash reports 2.2x-3.1x at concurrency 32 on SGLang [arxiv:2602.06036 tab. 3]. SpecInfer reports tree-parallel decoding up to 1.8x faster than sequence-based decoding at large batch [arxiv:2305.09781 §6.5] (see Contradictions) |
| Tail-latency (P99) SLOs | Small $K$ | The variance of sequence time grows with $K$ [arxiv:2302.01318 §6.3] |

## Implementation sketch

```text
state: target T, drafter D, accepted prefix x, KV caches for T (and D if separate)

loop until done:
  # 1. Draft
  chain variants: for i in 1..gamma: q_i = D(x + d_1..d_{i-1}); d_i ~ q_i
                  (MTP: D is one shared head, applied recursively)
  tree variants (Medusa): expand a static tree from head top-k / samples
  block (DFlash): one pass, anchor + B-1 masks, target features as extra KV

  # 2. Verify: ONE target forward over all drafted positions
  chain: p_1..p_{gamma+1} = T(x), T(x+d_1), ..., T(x+d_1..d_gamma)    # causal mask
  tree:  flatten nodes after the KV cache; mask = prefix + own ancestors; pos_id = len(x) + depth

  # 3. Accept (lossless rule; apply temperature/top-k/top-p to BOTH p and q first)
  n = 0
  for i in 1..gamma:
      if U(0,1) < min(1, p_i(d_i) / q_i(d_i)): n += 1
      else: break
  if n < gamma: t ~ norm(max(0, p_{n+1} - q_{n+1}))   # guard: normalizer can be ~0
  else:         t ~ p_{gamma+1}                        # bonus token, no extra target call
  tree: at each node, try children in turn (multi-round); pick the longest accepted path

  # 4. Commit
  x = x + d_1..d_n + [t]
  roll back / compact KV entries of rejected positions (target and drafter)
  Medusa: reuse the accepted path's head logits as the next draft
```

How the variants differ:

- **Greedy decoding.** With argmax standardization both distributions are one-hot, so the rule reduces
  to "accept iff draft token == target argmax". The cards infer this from [arxiv:2211.17192 §2.2]. Medusa
  uses the same exact-match test [arxiv:2401.10774 §2.3.1].
- **Choosing $\gamma$.** Maximize the walltime formula numerically for your measured $\alpha$ and $c$.
  $c$ depends on the hardware, so measure it on your own stack [arxiv:2211.17192 §3.3]
  [arxiv:2211.17192 §3.5]. Chen et al. found that speedup plateaus or regresses at large $K$, with the
  best $K=3$ on XSum nucleus [arxiv:2302.01318 §6.3].
- **Tree size.** For Medusa, acceleration grows roughly logarithmically with candidate count, and
  speed falls past about 64 candidates [arxiv:2401.10774 §3.3.1] [arxiv:2401.10774 app. G.3].
- **Block size (DFlash).** Train and test with the same block size. On Math500, a draft trained at 16 and
  tested at 8 reaches 3.87x, against 4.64x when both are 16 [arxiv:2602.06036 tab. 8].
- **KV-cache rollback.** None of the four papers specifies it in the main text. Medusa's code compacts
  the accepted path's KV entries into contiguous slots (see [Medusa](/papers/2401.10774), Pitfalls).
  DFlash keeps target-feature entries in each draft layer's KV cache for the whole sequence, and that
  memory is not quantified [arxiv:2602.06036 §4.1] [arxiv:2602.06036 app. A.3].

## Contradictions

- **Speculative-decoding baseline vs Medusa.** Medusa reports classic speculative decoding with
  open small drafters (Llama-68M, Tiny-Vicuna-1B) at only 1.47x / 1.56x / 1.60x on Vicuna 7B / 13B / 33B
  [arxiv:2401.10774 tab. 1] [arxiv:2401.10774 app. D]. The original papers report 2.3X-3.4X on T5-XXL with a T5-small drafter
  [arxiv:2211.17192 tab. 2] and 1.92x-2.46x on Chinchilla 70B [arxiv:2302.01318 tab. 1], using
  purpose-picked or purpose-built drafters on TPU stacks. These numbers do not refute each other. They
  measure different drafter quality, targets and hardware.
- **Does rejection sampling help Medusa?** The Medusa paper presents rejection sampling as an
  acceptance option that does not raise acceleration [arxiv:2401.10774 §2]. Its released code does not
  implement it, only greedy, typical and nucleus acceptance (see the [Medusa](/papers/2401.10774) card).
- **Medusa's headline range.** The arXiv abstract metadata says Medusa-2 reaches 2.3-3.6x. The LaTeX
  abstract says 2.3-2.8x. The 3.62x figure is a single MT-Bench category [arxiv:2401.10774 §3.1].
- **Gains at high concurrency.** Medusa's simulation predicts losses above batch size 32
  [arxiv:2401.10774 app. G.3]. DFlash reports 2.2x-3.1x at concurrency 32 for Qwen3 models on SGLang
  [arxiv:2602.06036 tab. 3]. The two measure different things, a simulated batch size against concurrent
  requests on a serving stack, so neither result is resolved here.
- **Batch size within SpecInfer.** The paper says its gains shrink as concurrent requests grow, and that
  wide trees are worse at large batch, where width 2 or 3 is best [arxiv:2305.09781 §6.2]
  [arxiv:2305.09781 §6.4]. Its tree-parallel versus sequence comparison is on par at small batch and up
  to 1.8x faster at large batch [arxiv:2305.09781 §6.5]. The two comparisons are different, and the
  paper does not reconcile them in the text, so neither is resolved here.
- **Stochastic verification in SpecInfer.** The paper's multi-step rule leaves the output distribution
  equal to the target's [arxiv:2305.09781 §4.3]. Its released code has only greedy verification, so the
  claim is not checked by the release. Treat it as the paper's claim until a stochastic
  implementation is tested (see the [SpecInfer](/papers/2305.09781) card).
- **Acceptance length vs walltime.** Nemotron 3 Super reports MTP acceptance length, averaging 3.45 at
  draft length 7 on SPEED-Bench [arxiv:2604.12374 tab. 2]. Medusa and DFlash report walltime speedups on
  other benchmarks [arxiv:2401.10774 §3.1] [arxiv:2602.06036 tab. 1]. Acceptance length does not give
  speedup without the drafter cost $c$, and the benchmarks differ, so these numbers are not comparable.

## Papers

- [Fast Inference from Transformers via Speculative Decoding](/papers/2211.17192): original draft-model
  method with the exactness proof, acceptance-rate and walltime theory, and T5-XXL results.
- [Accelerating Large Language Model Decoding with Speculative Sampling](/papers/2302.01318): concurrent
  SpS. Distributed serving at 70B scale with a wide, shallow draft model.
- [SpecInfer](/papers/2305.09781): token trees from small models, verified in one tree-attention pass in a
  serving system. Reports speedups against serving baselines, with greedy verification in the release.
- [Medusa](/papers/2401.10774): draft-model-free decoding with extra heads on the target, tree attention
  verification, and lossy typical acceptance.
- [DFlash: Block Diffusion for Flash Speculative Decoding](/papers/2602.06036): parallel block drafter
  that injects target features as keys and values in every draft layer. Author-reported speedups on Qwen3
  and LLaMA-3.1 models.
- [Nemotron 3 Super](/papers/2604.12374): hybrid MoE model whose shared-weight MTP head doubles as a
  recursive speculative draft. Reports MTP acceptance length, not walltime speedup.
- [Survey of small models alongside LLMs](/papers/2409.06857): classifies
  speculative decoding among cascading and routing. Gives the draft-then-verify shape, not the acceptance
  rule.