---
kind: concept
slug: looped-transformer
title: Looped transformer
summary: >-
  A transformer that applies one shared block of layers R times in a row, so effective depth is layers times R without new weights. Reported gains: unseen fact combinations and chains longer than training, both via more loops, with overthinking beyond a point. Variants: fixed or sampled R, and stochastic latent guidance with majority voting. All evidence comes from small synthetic or puzzle tasks.
tags: [recurrence, reasoning, compositional-generalization, test-time-compute]
papers: ["2604.07822", "2605.19376"]
related: [adaptive-halting, multi-hop-reasoning, grokking, recurrent-neural-network, variational-autoencoder]
status: draft
updated_at: 2026-10-18
---

## Overview

A looped transformer reuses one stack of layers. Its output is fed back in as
its input, and the same weights run again. The number of passes R sets the
effective depth, which is the layer count times R [arxiv:2604.07822 §4]. A
vanilla transformer is the case R = 1.

Two questions drive the work on this design. The first is whether reusing
weights lets a model combine facts it could not combine at a fixed depth. In a
vanilla stack, a fact stored in shallow layers cannot be read by deeper layers
that have different weights [arxiv:2604.07822 §1]. The second is whether more
passes at inference can reach longer chains than the model was trained on
[arxiv:2604.07822 §1] [arxiv:2604.07822 §6.1].

The name also covers the original work on looped transformers, which neither
card covers, so it is only named here: [Looped Transformers](https://arxiv.org/abs/2311.12424).
The sources below use it as a baseline, and their own reproductions are the
numbers cited.

## Variants

### Fixed recurrence

Every batch and every input uses the same R. The paper trains a four-layer
model with R in {1, 2, 4, 8} on a synthetic multi-hop task
[arxiv:2604.07822 §5.1]. Reported results on that task:

- R = 1 fails on the out-of-distribution (OOD) split, where the atomic facts
  never appear in a training composition [arxiv:2604.07822 §5.2].
- R = 2 reaches non-trivial OOD accuracy, with no number given in the text.
  R = 4 reaches OOD generalization in about 2k epochs, against about 7k for
  R = 2 [arxiv:2604.07822 §5.2] [fig. 3].

For depth extrapolation, the fixed model trains on chains of up to a set hop
count and is tested on longer ones. With matched data (training up to 12
hops, testing 12 to 24), R = 6 generalizes to 14 hops and R = 8 to 19 hops
[arxiv:2604.07822 §6.2]. With the paper's own curriculum, R = 6 reaches 17 hops
and R = 8 reaches 24 hops [arxiv:2604.07822 §6.2]. Raising R at inference past
the training value is what allows the extra depth [arxiv:2604.07822 §6.2]. Card:
[Loop, Think, & Generalize](/papers/2604.07822).

### Dynamic recurrence

Each batch draws its own R from a Poisson distribution with mean 4, clipped to
[2, 8] [arxiv:2604.07822 §4] [arxiv:2604.07822 app. J]. The sampled R does
not depend on the example's complexity, because the authors model cases
where complexity is unknown in advance [arxiv:2604.07822 §4].

The paper reports that the dynamic model matches R = 8 on the matched-data
setup, at 19 hops. In the main setup it exceeds R = 8, but the text gives no
number [arxiv:2604.07822 §6.2]. The same source reports overthinking: beyond
about 15 inference iterations, OOD accuracy stops improving, and the margin
peaks and then declines. The peak is lower for harder chains
[arxiv:2604.07822 §6.2] [fig. 8]. Card: [Loop, Think, & Generalize](/papers/2604.07822).

### Stochastic guidance with deep supervision

This variant keeps the loop but makes each latent step stochastic. A learned
Gaussian term is added to the high-level state at each transition. Its mean
and variance come from the state, so the run can take different paths
[arxiv:2605.19376 §2.1]. Training uses a posterior that sees the target and a
prior that does not, with a KL term between them
[arxiv:2605.19376 §2.2]. Inference drops the target and draws N trajectories,
then returns the most frequent answer [arxiv:2605.19376 §2.3]
[arxiv:2605.19376 alg. 2].

Training follows deep supervision. A supervision step runs several transitions,
gradients flow only through the last one, and the state is detached before the
next step [arxiv:2605.19376 §2.2]. The paper calls this biased but memory
efficient [arxiv:2605.19376 app. A.2].

Reported on Sudoku-Extreme, with the paper's own reproduction of each
baseline: N = 1 accuracy is 86.6 for this variant, against 61.7 for its looped
baseline and 83.6 for its TRM baseline. At N = 20 accuracy is 96.1
[arxiv:2605.19376 tab. 8]. The ablation removes each part in turn. Without the
stochastic guidance, Sudoku drops to 83.6 and N-Queens drops to 66.8. Stochasticity
without guidance gives 96.0 on Sudoku and 62.1 on N-Queens. Guidance without
stochasticity gives 0.0 on both [arxiv:2605.19376 tab. 3]. Card:
[Generative Recursive Reasoning](/papers/2605.19376).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Fixed latency or compute budget on a small structured task | Fixed recurrence, R set by the budget | Cost grows with R, and the R = 4 model reached OOD generalization sooner than R = 2 [arxiv:2604.07822 §5.2] |
| Test chains longer than any seen in training | Fixed recurrence with a high R at inference, trained on a curriculum | R = 8 reached 24 hops in the paper's own curriculum [arxiv:2604.07822 §6.2]. Each curriculum stage is costly: grokking to 4 hops took over 1.3M steps [arxiv:2604.07822 §6.2] |
| Depth needed per input is unknown | Dynamic recurrence, with an inference cap | Sampling R covers many depths [arxiv:2604.07822 §4]. Cap the iterations: the card reports no gain past about 15 [arxiv:2604.07822 §6.2] |
| Several valid outputs for one input | Stochastic guidance with majority vote | A deterministic loop collapses onto one trajectory [arxiv:2605.19376 §1] [fig. 1]. Majority vote costs N forward passes, so any N = 1 comparison must say so [arxiv:2605.19376 §4.1] |
| Matched compute against deeper deterministic models | Stochastic guidance, batched samples | At 8.2 to 8.5 TFLOPs, 16 iterations with N = 20 reached 96.06% in 184 ms, against 92.15% in 2,755 ms for 320 TRM iterations with N = 1 [arxiv:2605.19376 tab. 10]. The latency comes from one puzzle at a time on the paper's hardware |
| Tight training budget | Fixed or dynamic recurrence without deep supervision | Deep supervision makes training sequential, which the paper names as a barrier to scaling [arxiv:2605.19376 §5]. One looped run on ARC-AGI was estimated at about 97 days, so it was not trained [arxiv:2605.19376 app. C.1] |
| Production LLM | No card supports this | Both sources use small synthetic or puzzle tasks. The first says its results do not carry over to LLMs trained at scale [arxiv:2604.07822 app. A] |

## Implementation sketch

The fixed and dynamic variants share one loop. The stochastic variant adds a
noise head to each transition, which is not shown here.

```pseudocode
# Looped forward pass, fixed or sampled recurrence  [arxiv:2604.07822 §4]
# Input: prefix tokens; R, or None to sample it per batch

def looped_forward(prefix, R=None, lam=4, r_min=2, r_max=8):
    # Dynamic: a new R per batch, clipped Poisson  [arxiv:2604.07822 §4]
    if R is None:
        R = clip(poisson(lam), r_min, r_max)
    hidden = embed(prefix)
    for _ in range(R):
        # Same shared layers each pass; output feeds the next  [arxiv:2604.07822 §4]
        hidden = shared_layers(hidden)
    # Answer read at the final prefix position  [arxiv:2604.07822 §3.1]
    return readout(layer_norm(hidden))
```

Three details from the cards matter for a faithful reimplementation. The
output projections of attention and MLP are zeroed at start, so each pass
begins as an identity map [arxiv:2604.07822 §4]. The synthetic task must
use a permutation construction, or a model can read the answer from a short
suffix of the relations and look like it composes all hops [arxiv:2604.07822 app. K]
[arxiv:2604.07822 app. B]. The padded batch layout also decides which column
holds the prediction, so reproduce it exactly
([Loop, Think, & Generalize](/papers/2604.07822), *Pitfalls*).

## Contradictions

None known between the two sources. They use different tasks, and neither
reports the other's measure, so their numbers are not comparable. The
looped baseline in the second source is its own reproduction, not a number
from the first source.

Within the second source, the paper and its code disagree on several training
details, including the halting label and the weight decay used for ARC-AGI.
These are listed on that card (*Paper vs code*), not resolved here.

## Papers

- [Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers](/papers/2604.07822): fixed and dynamic recurrence on a synthetic knowledge-graph task, with depth extrapolation and overthinking.
- [Generative Recursive Reasoning](/papers/2605.19376): stochastic latent guidance on a looped reasoner, trained with variational inference and majority-voted at test time.