Adaptive halting

Overview

A recurrent model runs the same block several times over its hidden state. Adaptive halting asks when to stop. Instead of a fixed number of passes for every input, the model stops once it judges that more passes will not help. The same question appears in a feed-forward transformer, where the unit is a layer rather than a pass: the model can skip its remaining layers once the answer is already settled 2203.14680 §6.2.

Two reasons drive the idea. Some inputs need more passes than others, and the number needed is often unknown in advance. The second reason is overthinking: more passes can make the answer worse. One source reports that for its dynamic model, OOD accuracy stopped improving beyond about 15 iterations, and that the margin peaked and then declined as iterations grew 2604.07822 §6.2. That source also says adaptive halting does not fix this effect 2604.07822 §6.2.

The sources differ in what they use as the stop signal. One uses the model's output distribution alone, so it needs no extra training. Another trains a dedicated head to predict whether the current answer is correct 2605.19376 app. A.3. The third reads which large sub-updates of each feed-forward layer already promote the final token, and compares them with clusters seen at earlier saturation events 2203.14680 §6.2.

Variants

Output settling (KL and entropy)

Stop when the prediction stops moving and is confident. At each pass after the first, compare the new output distribution with the previous one by KL divergence, and measure the entropy of the new distribution. Halt when both fall below their thresholds 2604.07822 §6.2. The paper sets the KL threshold to 0.01 and the entropy threshold to 3.00 2604.07822 app. C. The paper's rule is described as an extension of an earlier KL-based halting method, to which the entropy term is added 2604.07822 §6.2. Card: Loop, Think, & Generalize.

Two details matter for reproduction. The rule never halts on the first pass, so at least two passes always run 2604.07822 §6.2. The released code uses a different KL default in its adaptive evaluation than the paper states, so the effective threshold depends on which code path is used 2604.07822 app. C.

Learned halting head

Train an auxiliary head on the recurrent state to output the probability that the current answer is correct. At inference, stop when this probability passes one half 2605.19376 app. A.3. Training uses a binary cross-entropy loss on the head's output, with the label set by whether the current answer is correct 2605.19376 alg. 1. The paper says the head follows a Q-style target 2605.19376 app. A.3. Card: Generative Recursive Reasoning.

The paper marks halting as optional in its method 2605.19376 app. A.3. The card reports no run with halting switched off, so the cost and accuracy effect of the head alone is not known from the card.

Dominant-cluster early exit (layer skipping)

This applies to a standard transformer with no recurrence. For each layer, the 10 largest sub-updates of its feed-forward output are mapped to clusters of value vectors. Clusters seen at saturation events in that layer are stored. At inference, the model exits at a layer when the clusters of its current dominant sub-updates overlap the saturation sets of that layer more than the overlap with the non-saturation sets of every later layer, on both the mean and the maximum 2203.14680 §6.2 2203.14680 app. A.5. Card: Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space.

The clusters are precomputed. The paper uses k = 2e2 clusters for this rule 2203.14680 app. A.5. The paper's text describes the rule more loosely than its notebook, which requires both mean and maximum overlaps to win 2203.14680 §6.2 wikilm-earlyexit-ours.ipynb.

When to use which

Implementation sketch

The output-settling rule is the simplest to add to an existing looped model, so the sketch follows it. The learned head adds one training loss and a threshold of one half at inference. The layer-skipping rule needs the cluster sets and per-layer saturation sets from its card, so it is not sketched here.

# Output-settling adaptive halting  [arxiv:2604.07822 §6.2]
# Stop once successive output distributions agree and are confident

def adaptive_predict(prefix, max_recurrence, eps_kl, entropy_thresh):
    hidden = embed(prefix)
    previous_logp = None
    for t in range(max_recurrence):
        # One more pass through the shared layers  [arxiv:2604.07822 §4]
        hidden = apply_shared_layers(hidden)
        logp = log_softmax(readout(hidden))  # answer position only
        if t > 0:
            # How far this pass moved the prediction  [arxiv:2604.07822 §6.2]
            kl = sum(exp(logp) * (logp - previous_logp))
            # How unsure the current prediction is  [arxiv:2604.07822 §6.2]
            entropy = -sum(exp(logp) * logp)
            if kl < eps_kl and entropy < entropy_thresh:
                return logp, t + 1
        previous_logp = logp
    # No halt before the cap: use the last pass  [arxiv:2604.07822 app. C]
    return logp, max_recurrence

Contradictions

No direct conflict between the three sources. They measure different things: output settling, learned correctness and sub-update clusters. No card compares any two of them on the same task.

Within one source, the paper and its released code disagree. The paper sets the KL threshold to 0.01 2604.07822 app. C, while the adaptive evaluation uses a wrapper default of 0.001 (see the card's Paper vs code section). The same card notes that the paper gives the halting entropy as a fixed threshold, while the code also contains a variant that scales it with sequence length 2604.07822 app. C.

For the learned-head source, the paper states that the halting label is whether the prediction equals the target 2605.19376 app. A.3. The released code accepts any valid solution on the puzzle tasks instead (see the card's Paper vs code). The two readings give different training signals on multi-solution tasks.

For the layer-skipping source, the paper reports its rule as "on-par" with per-layer logistic-regression classifiers 2203.14680 §6.2. The classifiers train on 8k examples and the rule on 10k, so the two are not compared on identical data 2203.14680 app. A.5. The table's rows for two of the classifier inputs show identical numbers, which the text does not explain 2203.14680 tab. 6.

Papers

• Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers: output-settling halting with KL and entropy thresholds, with overthinking observed in a dynamic-recurrence model.
• Generative Recursive Reasoning: a learned halting head trained on correctness, inside a latent-variable recursive reasoner.
• Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space: a layer-skipping early exit from dominant feed-forward sub-update clusters, reported on par with per-layer classifiers.