---
kind: concept
slug: test-time-scaling
title: Test-time scaling
summary: >-
  Spending more inference compute per question, either by letting a model write a longer reasoning trace or by sampling several answers and voting. On an edge GPU, longer answers dominate latency and energy, and a 14B reasoning model reached 80.6% on MMLU-Redux at 259 s per question (author-reported). Budget caps and parallel voting trade accuracy against time.
tags: [inference, latency, reasoning-models, edge]
papers: ["2511.01866"]
related: [large-reasoning-models, chain-of-thought-prompting, decoding-strategies, weight-quantization, llm-serving]
status: draft
updated_at: 2026-10-25
---

## Overview

Test-time scaling means giving a model more work at inference time to improve
an answer. There are two ways to do it: make one answer longer (sequential) or
produce several answers and combine them (parallel). The cost is paid in
decode steps, so latency and energy grow with the number of generated tokens
[arxiv:2511.01866 §1] [arxiv:2511.01866 §4.1].

Only one reuse-licensed card covers this concept so far
([EdgeReasoning](/papers/2511.01866)). The results below describe that paper's
setup alone: DeepSeek-R1 distilled models on a Jetson AGX Orin, with
MMLU-Redux as the benchmark [arxiv:2511.01866 §4, §5].

## Variants

### Output budget: hard limit

The prompt asks for a length and generation stops at a fixed number of tokens
[arxiv:2511.01866 §5]. The cut is enforced, so latency has a hard upper
bound. The 14B model at a 256-token hard limit scored 58.6% at 21.5 s, with
112.9 tokens on average [arxiv:2511.01866 app. B]. The 8B model at 128 tokens
scored 37.9% at 7.9 s [arxiv:2511.01866 app. B].

### Output budget: soft limit

The same instruction is given but generation is not stopped
[arxiv:2511.01866 §5]. The paper labels these runs "NC". Accuracy is higher
and so is cost: the 14B model at a 256-token soft limit scored 77.2% at 70.9 s
with 374.2 tokens on average [arxiv:2511.01866 app. B]. The soft limit is not
a latency bound, so budget a soft-limit run with its measured average length.

### No-reasoning injection

A fixed, short thinking block is inserted so the model skips its own reasoning
phase [arxiv:2511.01866 §5]. The paper compares this against the other output-length methods; the exact
string is given in the paper, not on the card.

### Budget-trained model (L1-Max)

A 1.5B model fine-tuned with reinforcement learning to respect a token budget
[arxiv:2511.01866 §2.3] [arxiv:2511.01866 §5]. Its base configuration scored
43.8% at 7.50 s and used 312.6 tokens on average. Under a 256-token hard limit
it scored 18.3% at 1.2 s [arxiv:2511.01866 app. B].

### Parallel sampling with majority vote

Several decoders run on the same prompt and the answers are combined by
majority vote [arxiv:2511.01866 §5.5]. On a 128-token budget, accuracy rose by
roughly 1.5 to 1.8 times when the scaling factor went from 1 to 32. At a
512-token budget the gain levelled off at about 4 times for the large models.
Decode latency rose about 2 times from factor 1 to 64 [arxiv:2511.01866 §5.5]
[fig. 9] [fig. 10]. Energy per question for the 14B model grew by less than
1.5 times up to factor 4 [arxiv:2511.01866 §5.5].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Latency under about 5 s on a board like the Orin | A 1.5B model, with a hard limit or budget-trained model | The paper reads only the 1.5B models as qualifying in this band [arxiv:2511.01866 §5.1] [fig. 7]. Their accuracy is 15.9% to 43.8% on the full set [arxiv:2511.01866 app. B]. |
| Latency between about 15 and 30 s | Non-reasoning 8B model | The paper prefers the non-reasoning 8B models in this band, read from its Pareto plot [arxiv:2511.01866 §5.1] [fig. 7]. |
| Accuracy matters most and latency is loose (above about 30 s) | 14B reasoning model, base or soft limit | Highest accuracy on the set, 80.6% base and 77.2% soft-limited [arxiv:2511.01866 app. B]. |
| Moderate latency with a reasoning model | 14B model with a 256-token hard limit | 58.6% at 21.5 s, versus 80.6% at 259 s for the base run [arxiv:2511.01866 app. B]. |
| Accuracy must rise and there is headroom in energy | Parallel voting, up to a factor of about 4 | Energy per question stays under 1.5 times up to factor 4 [arxiv:2511.01866 §5.5]. The paper says the cheap regime holds to about 8, and energy rises sharply beyond that [arxiv:2511.01866 §5.5]. |
| Memory or latency too tight for fp16 weights | Weight quantization (AWQ W4) | Relative accuracy loss of 1.04% (1.5B), 6.16% (8B) and 0.62% (14B) [arxiv:2511.01866 §5.6]. See [weight-quantization](/concepts/weight-quantization). |

The paper does not give one best setting. It maps which model fits which
latency band, and it reads those bands from its Pareto plot
[arxiv:2511.01866 §5.1] [fig. 7]. Quantization is a separate lever: it changes the cost per token, not the number
of tokens.

## Implementation sketch

Choose the model and output budget that meet a latency target, using the
paper's fitted latency model for the three models it covers. The fits are
specific to that device and those models, so treat them as a template.

```pseudocode
# Pick the most accurate config that meets a latency target  [§4.1] [§5.1]
def pick_config(input_len, target_s, candidates):
    best = None
    for model in candidates:
        for budget in model.budgets:
            # Hard limits cap output; soft limits use the measured average
            # length, not the budget  [app. B]
            if model.hard:
                out_len = min(budget, model.avg_tokens[budget])
            else:
                out_len = model.avg_tokens[budget]
            t = total_latency(input_len, out_len, model)  # eq. 3
            if t > target_s:
                continue
            if best is None or model.acc[budget] > best.acc:
                best = Config(model, budget, t)
    return best
```

For parallel voting, multiply the decode work by the scaling factor and check
the energy-per-question curve before choosing a factor above 4
[arxiv:2511.01866 §5.5].

## Contradictions

- **Paper text against its own tables.** The text says the 8B base model is
  5.7% more accurate than Llama3.1-8B-it [arxiv:2511.01866 §5.3]. The tables
  give 61.7% against 58.3%, which is 3.4 points or about 5.8% relative
  [arxiv:2511.01866 app. B]. The text also gives 34% loss for the 8B 128-token
  run, while the table gives 35% relative (20.4 points) [arxiv:2511.01866 §5.3]
  [arxiv:2511.01866 app. B]. The card's pitfalls section lists more cases.
- **Quantization speed-up.** The text gives 2 to 5 times [arxiv:2511.01866
  §5.6]. The appendix gives 5.9 times for the 8B model [arxiv:2511.01866
  app. B].
- **Smallest model.** The text says the 1.5B non-reasoning configuration is
  the most accurate of the small models [arxiv:2511.01866 §5.2]. In the
  appendix, the L1-Max base configuration scores higher, 43.8% against 41.0%
  [arxiv:2511.01866 app. B]. The two sets of runs are not directly comparable.
- **Decode power coefficients, paper against code.** The appendix gives the
  two decode-power log coefficients in the reverse order to the code. The
  code's reading gives about 26 W at 512 output tokens, and the appendix's
  reading about 58 W [arxiv:2511.01866 app. E]
  [code: power_model.py#L163-L164]. The card says to verify before relying on
  either.
- **Prefill power switch for 14B.** The paper uses 384 tokens
  [arxiv:2511.01866 §4.2] [arxiv:2511.01866 app. E]. The code uses 800
  [code: config/analytic.yaml#L96-L101].

## Papers

- [EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs](/papers/2511.01866): measures output budgets, a budget-trained model, parallel voting and AWQ on one edge GPU, with fitted latency and energy models.