---
kind: concept
slug: chain-of-thought
title: Chain of thought
summary: >-
  Make a model write intermediate text before it acts or answers. As a training target, grounded chain of thought adds a subtask, a movement plan and a gripper target before each robot action. On an edge GPU, long chains dominate latency: token caps, parallel samples and quantization trade accuracy for speed, and pruning low-entropy steps cuts thinking tokens by 27-45%.
tags: [reasoning, vision-language-action, latency, edge-inference, token-efficiency]
papers: ["2412.11974", "2508.03346", "2511.01866"]
related: [chain-of-thought-prompting, self-consistency, large-reasoning-models, test-time-scaling, weight-quantization, vision-language-action-model, imitation-learning, action-chunking, task-decomposition, policy-gradient]
status: draft
updated_at: 2026-10-31
---

## Overview

Chain of thought (CoT) here means text that a model writes before its final
output, which is then used as a training target or paid for at inference. The
three cards on wikillm that use the term do this in different settings. In a
robot policy, the written chain is a plan (the subtask, a target and a movement)
that is trained to come before each motor command [arxiv:2412.11974 §3.3]. On
a reasoning language model, the written chain is the bulk of the work, and it
sets the latency of every answer [arxiv:2511.01866 §4.1]. A third card treats
that chain as something to shorten: it scores each step of a finished trace and
replaces the most predictable steps with `[SKIP]` markers [arxiv:2508.03346
§3.2].

This page does not cover few-shot prompting with exemplars, self-consistency
voting over prompt chains or tree search. Those are on
[Chain-of-thought prompting](/concepts/chain-of-thought-prompting). The
invariant shared by the variants below is that the chain is a sequence of
tokens, so its length sets the cost of each answer or action.

## Variants

### Grounded chain of thought as a training target

Emma-X fine-tunes a 7B OpenVLA policy so that, for each frame, it writes a
reasoning text and a subtask, then a 2D gripper target, then a 3D movement
plan, and only then the 7-dimensional action [arxiv:2412.11974 §3.3] [app. F].
The labels are built from existing demonstrations. The trajectory is split into
segments wherever the cluster label or the gripper state changes [arxiv:2412.11974
§3.1]. A vision-language model then labels each segment from its images and the
task instruction, returning a subtask and a justification [arxiv:2412.11974 §3.2].
The authors use images rather than text alone because a text-only labeller,
as in ECoT, can hallucinate what the scene shows [arxiv:2412.11974 §1]. The
movement plan is the displacement to the first frame of the next segment,
written in a fixed text template [arxiv:2412.11974 §3.2] [app. E]. The model
also predicts the gripper position at the start of the next segment before it
picks the action for the current frame, which the authors call look-ahead
[arxiv:2412.11974 §3.2]. Card: [Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning](/papers/2412.11974).

The author-reported result on 12 real WidowX tasks, 10 trials each, is 57.5%
success and 71.66% half-success. OpenVLA scores 33.33% and 45.41%, and ECoT
15.00% success [arxiv:2412.11974 tab. 1] [§4.2]. The ablation separates the
parts. On six prompts, removing the grounded reasoning drops spatial-relation
success from 70% to 10%, and removing the movement plan drops it to 37%
[arxiv:2412.11974 tab. 2]. Each prompt has only 10 rollouts and no confidence
intervals are given [arxiv:2412.11974 §4.5]. The ablation also has no text-only
labelling arm, so the benefit of image-based labels alone is not isolated
[arxiv:2412.11974 tab. 2].

The cost is output length. The model writes about ten times as many tokens per
step as OpenVLA, which slows each control step [arxiv:2412.11974 Limitations].

### Length-capped reasoning on an edge GPU

EdgeReasoning measures a DeepSeek-R1 distilled family (1.5B, 8B and 14B) on a
Jetson AGX Orin with vLLM, using 3,000 MMLU-Redux questions for accuracy
[arxiv:2511.01866 §4.1] [§5]. For these models decoding takes over 99% of
latency [arxiv:2511.01866 §4.1]. The paper compares four ways to cap output
[arxiv:2511.01866 §5]:

- *Hard limit* (`nT`): the prompt asks for a length and generation stops at
  `n` tokens.
- *Soft limit* (`n`-NC): the same instruction, with no enforcement.
- *No reasoning* (NR): a fixed short thinking block is injected so the model
  skips its own chain.
- *L1-Max*: a 1.5B model trained with reinforcement learning to respect a token
  budget [arxiv:2511.01866 §2.3].

On the full set, the hard limit cuts accuracy sharply. The 8B model drops from
61.7% at 811 average tokens to 37.9% at a 128-token hard cap, with latency
falling from 87.2 s to 7.9 s. The 14B model with a 256-token hard cap reaches
58.6% in 21.5 s, against 77.2% in 70.9 s with the soft limit and 80.6% in
259.0 s uncapped [arxiv:2511.01866 app. B]. A separate 150-question motivation
table gives different latencies for the same models, so the two sample sizes
should not be mixed [arxiv:2511.01866 tab. II] [app. B].

### Parallel samples with majority vote

The same paper runs several decoders on one prompt and takes the majority
answer [arxiv:2511.01866 §5.5]. At a 128-token budget, accuracy rises by about
1.5 to 1.8 times when the scaling factor goes from 1 to 32. Decode latency
rises about 2 times from factor 1 to 64. For the 14B model, energy per question
grows by less than 1.5 times up to factor 4 and doubles at factor 16
[arxiv:2511.01866 §5.5] [fig. 9] [fig. 10]. The authors' takeaway is that
parallel scaling stays cheap only up to a factor of about 8 [arxiv:2511.01866
§5.5]. The paper does not state the sampling temperature or how ties are broken.

Card: [EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs](/papers/2511.01866).

### Pruning low-entropy steps from a trace

Making Slow Thinking Faster splits a finished reasoning trace into steps at
blank lines, scores each step by the mean entropy of its tokens, and replaces
the lowest-scoring 80% of steps with `[SKIP]` [arxiv:2508.03346 §3.2] [§4.1]
[eq. 3]. The method rests on the assumption that the model's own per-token
uncertainty is a usable sign of redundancy [arxiv:2508.03346 §3.1]. The score
uses full-vocabulary entropy [eq. 1], so a serving stack that returns only the
probability of the sampled token cannot compute it as written.

In the static setting, the pruned trace is placed after the query and a closing
`</think>` delimiter, and the model writes only the final answer [arxiv:2508.03346
§3.2]. On DeepSeek-R1-
Distill-Qwen-7B, thinking tokens fall by 29.7% on MATH500 and 36.3% on AIME 2024,
with MATH500 accuracy unchanged at 88.17% [arxiv:2508.03346 tab. 1]. The same
cut costs AIME 2024 accuracy: 63.33% falls to 56.67%, and the 14B model drops
about 7 points on AIME 2024 and AIME 2025 [arxiv:2508.03346 tab. 1]. On the full
DeepScaleR and OpenR1-Math sets, the 7B model loses 1.6 and 1.1 points
[arxiv:2508.03346 tab. 2]. The paper's random-pruning and high-entropy-pruning
controls show the direction matters: removing high-entropy steps hurts at once
[arxiv:2508.03346 §4.1] [fig. 1].

The trained variant teaches the model to write `[SKIP]` itself. An SFT stage
on pruned traces, then GRPO with a composite reward, cuts thinking tokens by
35% to 57% on four benchmarks [arxiv:2508.03346 tab. 3]. For the 7B model, the
change in accuracy is +0.6 on GSM8K, −3.2 on MATH500, −6.2 on AIME 2024 and
−2.4 on AIME 2025 [arxiv:2508.03346 tab. 3]. The reward terms are not
interchangeable. Correctness and skip ratio alone drop MATH500 accuracy to
51.00%; adding the skip-count penalty brings it back to 85.00%
[arxiv:2508.03346 tab. 5]. In the paper's comparison with other compression
methods, on AIME 2024 the trained method loses 6.2 points and cuts 57.0% of
tokens, against 15.0 points and 34.6% for CoT-Valve. On MATH500, CoT-Valve cuts
more tokens (48.4%) but loses more accuracy (10.6 points) [arxiv:2508.03346
tab. 4]. These figures are the authors' and were not re-run.

Card: [Making Slow Thinking Faster](/papers/2508.03346).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Robot policy with labelled demonstrations and a vision-language model to label them, and planning matters more than step speed | Grounded chain of thought, as in Emma-X | Author-reported gain over OpenVLA and ECoT on 12 real tasks [arxiv:2412.11974 tab. 1]; about ten times the tokens per step [arxiv:2412.11974 Limitations] |
| Robot policy that must react at high rate, with no planning text | Plain action output, as in OpenVLA | The planning text makes each step slower [arxiv:2412.11974 Limitations] |
| Reasoning model on an edge GPU with a latency target under 5 s | A 1.5B reasoning model | Under 5 s only the 1.5B models qualify on the Pareto plot [arxiv:2511.01866 §5.1] [fig. 7] |
| Same device, target 15 to 30 s | A non-reasoning 8B model | Preferred in that band on the same plot [arxiv:2511.01866 §5.1] |
| Same device, target above 30 s | The 14B reasoning model | Best accuracy above 30 s [arxiv:2511.01866 §5.1] |
| Latency must be bounded for each answer | Hard limit, with the accuracy loss in mind | Hard caps bound latency but lose accuracy at small budgets (8B at 128 tokens: 37.9%) [arxiv:2511.01866 app. B] |
| Accuracy matters and latency allows it | Soft limit or a larger hard budget | Soft 256 gives 64.3% for the 8B model, but at 97.9 s [arxiv:2511.01866 app. B] |
| Spare compute on a device that can batch decodes | Parallel samples with majority vote, up to a factor of about 8 | Accuracy gains flatten and energy rises sharply past that point [arxiv:2511.01866 §5.5] |
| Memory is the binding limit | 4-bit AWQ weights, after checking the target device has kernels for them | Relative accuracy loss of 0.62% (14B) to 6.16% (8B); the Orin runs the W4 weights through INT8 kernels [arxiv:2511.01866 §5.6] |
| Long traces already exist, the serving stack exposes full token distributions, and some accuracy loss on the hardest problems is acceptable | Static step pruning at 80% | Thinking tokens fall 27% to 45% on MATH500 and AIME with no training [arxiv:2508.03346 tab. 1]; AIME 2024 drops 6.7 to 6.9 points on two of the three models tested [arxiv:2508.03346 tab. 1] |
| Same, but the model runs at inference without a pruning step | SFT then GRPO with the skip-count penalty | 35% to 57% fewer tokens on four benchmarks; for the 7B model, AIME 2024 down 6.2 points [arxiv:2508.03346 tab. 3]. Dropping the skip-count penalty cost MATH500 about 34 points [arxiv:2508.03346 tab. 5] |
| Need a hard bound on tokens for each answer | Hard limit, not step pruning | The paper's length rule is a training reward, not an inference cap [arxiv:2508.03346 §3.3] |

## Implementation sketch

The training target for grounded chain of thought keeps the paper's order: the
segment's reasoning and subtask, then the gripper target, then the movement
plan, then the action tokens [arxiv:2412.11974 §3.3].

```pseudocode
# Target text for one frame, in the order the paper uses  [arxiv:2412.11974 §3.3]
def training_target(frame):
    # The subtask and reasoning come from the frame's segment  [§3.2]
    seg = segment_of(frame)
    text = seg.reasoning + seg.subtask
    # 2D gripper position at the start of the next segment  [§3.2]
    text += next_gripper_2d(frame)
    # Displacement to that point, written as template text  [§3.2] [app. E]
    text += movement_plan(frame)
    # The 7-D action, each dimension as one of 256 bins  [arxiv:2412.11974 §4.1]
    text += tokenize(frame.action)
    return text
```

For a reasoning model with a fixed latency target, the decode latency is a
closed form in the input and output lengths, with coefficients fitted per model
on the target device [arxiv:2511.01866 §4.1] [eq. 2]. The budget can then be
chosen before the request is sent.

```pseudocode
# Decode time after an input of I tokens and an output of O tokens  [arxiv:2511.01866 §4.1] [eq. 2]
def decode_latency(I, O, m, n):
    # Per-token time n + m * context, summed over the output steps
    return n * O + m * (I * O + O * (O - 1) / 2)

# Largest output budget that fits the target, for a model fitted on the device  [§5.1]
def pick_budget(I, target_s, m, n, prefill_s, max_tokens):
    best = 0
    for O in range(1, max_tokens + 1):
        # Prefill is small next to decode  [arxiv:2511.01866 tab. VII]
        if prefill_s + decode_latency(I, O, m, n) <= target_s:
            best = O
    return best
```

The paper's own coefficients are for one Orin and are not portable. Its
published sign for the 1.5B decode slope disagrees with the released code, so
fit your own before relying on the numbers [arxiv:2511.01866 Paper vs code].

For static step pruning, the score and the cut are both simple. The paper
splits at blank lines and averages the entropy of each step's tokens
[arxiv:2508.03346 §3.2] [eq. 3].

```pseudocode
# Static step pruning of one finished trace  [arxiv:2508.03346 §3.2]
# Each token carries its full-vocabulary entropy in bits  [eq. 1]
def prune_trace(tokens, kappa=0.8):
    # Split at blank lines; each step keeps its tokens  [§3.2]
    steps = split_on_blank_lines(tokens)
    # Score each step by its mean token entropy  [eq. 3]
    scores = [mean([t.entropy for t in s]) for s in steps]
    # Replace the kappa * N lowest-scoring steps with [SKIP]  [§3.2] [§4.1]
    # The paper does not say how kappa * N is rounded  [§3.2]
    n_cut = int(kappa * len(steps))
    order = sorted(range(len(steps)), key=lambda i: scores[i])
    cut = set(order[:n_cut])
    # Keep the other steps verbatim  [§3.2]
    return ["[SKIP]" if i in cut else text_of(s) for i, s in enumerate(steps)]
```

## Contradictions

None known between the three papers: they measure different things, a robot
action and two kinds of text answer, and do not test the same claim. The two
text-reasoning cards test different interventions, a hard token cap and
entropy-based step removal, at budgets that do not match, so their accuracy
figures cannot be compared directly.

Within the sources, some reported figures do not reconcile:

- Emma-X prints 28.75% as the ECoT half-success average, but the mean of its
  12 rows is 29.17% [arxiv:2412.11974 tab. 1]. Its text gives a 29% gain over
  ECoT on spatial tasks, while Figure 4 and Table 1 imply about 43 points
  [arxiv:2412.11974 §4.4] [fig. 4]. Use the table rows.
- EdgeReasoning gives 91.5 tokens for the 14B model at a 128-token hard cap in
  its text, but 78.2 in its appendix. The 91.5 figure belongs to the 1.5B row
  [arxiv:2511.01866 §5.2] [app. B].
- The Making Slow Thinking Faster paper labels its Table 4 as percentage
  changes, but the accuracy column matches absolute point differences from its
  Table 3 [arxiv:2508.03346 tab. 4] [tab. 3]. Read those accuracy values as
  points.

## Papers

- [Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning](/papers/2412.11974): grounded chain of thought as the target for a robot action model, with segment labels from images.
- [Making Slow Thinking Faster](/papers/2508.03346): entropy-scored pruning of reasoning steps, applied to a finished trace and learned by SFT and GRPO.
- [EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs](/papers/2511.01866): latency and energy of reasoning models on an edge GPU, with length caps, parallel sampling and quantization.