Test-time scaling

Overview

Test-time scaling means giving a model more work at inference time to improve an answer. There are two ways to do it: make one answer longer (sequential) or produce several answers and combine them (parallel). The cost is paid in decode steps, so latency and energy grow with the number of generated tokens 2511.01866 §1 2511.01866 §4.1.

Only one reuse-licensed card covers this concept so far (EdgeReasoning). The results below describe that paper's setup alone: DeepSeek-R1 distilled models on a Jetson AGX Orin, with MMLU-Redux as the benchmark 2511.01866 §4, §5.

Variants

Output budget: hard limit

The prompt asks for a length and generation stops at a fixed number of tokens 2511.01866 §5. The cut is enforced, so latency has a hard upper bound. The 14B model at a 256-token hard limit scored 58.6% at 21.5 s, with 112.9 tokens on average 2511.01866 app. B. The 8B model at 128 tokens scored 37.9% at 7.9 s 2511.01866 app. B.

Output budget: soft limit

The same instruction is given but generation is not stopped 2511.01866 §5. The paper labels these runs "NC". Accuracy is higher and so is cost: the 14B model at a 256-token soft limit scored 77.2% at 70.9 s with 374.2 tokens on average 2511.01866 app. B. The soft limit is not a latency bound, so budget a soft-limit run with its measured average length.

No-reasoning injection

A fixed, short thinking block is inserted so the model skips its own reasoning phase 2511.01866 §5. The paper compares this against the other output-length methods; the exact string is given in the paper, not on the card.

Budget-trained model (L1-Max)

A 1.5B model fine-tuned with reinforcement learning to respect a token budget 2511.01866 §2.3 2511.01866 §5. Its base configuration scored 43.8% at 7.50 s and used 312.6 tokens on average. Under a 256-token hard limit it scored 18.3% at 1.2 s 2511.01866 app. B.

Parallel sampling with majority vote

Several decoders run on the same prompt and the answers are combined by majority vote 2511.01866 §5.5. On a 128-token budget, accuracy rose by roughly 1.5 to 1.8 times when the scaling factor went from 1 to 32. At a 512-token budget the gain levelled off at about 4 times for the large models. Decode latency rose about 2 times from factor 1 to 64 2511.01866 §5.5 fig. 9 fig. 10. Energy per question for the 14B model grew by less than 1.5 times up to factor 4 2511.01866 §5.5.

When to use which

The paper does not give one best setting. It maps which model fits which latency band, and it reads those bands from its Pareto plot 2511.01866 §5.1 fig. 7. Quantization is a separate lever: it changes the cost per token, not the number of tokens.

Implementation sketch

Choose the model and output budget that meet a latency target, using the paper's fitted latency model for the three models it covers. The fits are specific to that device and those models, so treat them as a template.

# Pick the most accurate config that meets a latency target  [§4.1] [§5.1]
def pick_config(input_len, target_s, candidates):
    best = None
    for model in candidates:
        for budget in model.budgets:
            # Hard limits cap output; soft limits use the measured average
            # length, not the budget  [app. B]
            if model.hard:
                out_len = min(budget, model.avg_tokens[budget])
            else:
                out_len = model.avg_tokens[budget]
            t = total_latency(input_len, out_len, model)  # eq. 3
            if t > target_s:
                continue
            if best is None or model.acc[budget] > best.acc:
                best = Config(model, budget, t)
    return best

For parallel voting, multiply the decode work by the scaling factor and check the energy-per-question curve before choosing a factor above 4 2511.01866 §5.5.

Contradictions

• Paper text against its own tables. The text says the 8B base model is 5.7% more accurate than Llama3.1-8B-it 2511.01866 §5.3. The tables give 61.7% against 58.3%, which is 3.4 points or about 5.8% relative 2511.01866 app. B. The text also gives 34% loss for the 8B 128-token run, while the table gives 35% relative (20.4 points) 2511.01866 §5.3 2511.01866 app. B. The card's pitfalls section lists more cases.
• Quantization speed-up. The text gives 2 to 5 times [arxiv:2511.01866 §5.6]. The appendix gives 5.9 times for the 8B model [arxiv:2511.01866 app. B].
• Smallest model. The text says the 1.5B non-reasoning configuration is the most accurate of the small models 2511.01866 §5.2. In the appendix, the L1-Max base configuration scores higher, 43.8% against 41.0% 2511.01866 app. B. The two sets of runs are not directly comparable.
• Decode power coefficients, paper against code. The appendix gives the two decode-power log coefficients in the reverse order to the code. The code's reading gives about 26 W at 512 output tokens, and the appendix's reading about 58 W 2511.01866 app. E power_model.py. The card says to verify before relying on either.
• Prefill power switch for 14B. The paper uses 384 tokens 2511.01866 §4.2 2511.01866 app. E. The code uses 800 analytic.yaml.

Papers

• EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs: measures output budgets, a budget-trained model, parallel voting and AWQ on one edge GPU, with fitted latency and energy models.