---
kind: concept
slug: grokking
title: Grokking
summary: >-
  Delayed generalization: a model fits its training data, then much later learns the rule behind it. Recurrent blocks change when and whether small transformers generalize to unseen two-hop fact pairs. Also covers float32 loss spikes in long unregularised runs, which one paper argues are numerical rather than grokking. Three cards; results are synthetic or small-scale.
tags: [generalization, training-dynamics, reasoning, numerical-stability]
papers: ["2604.07822", "2605.06152", "2512.02004"]
related: [multi-hop-reasoning, adaptive-halting, low-precision-arithmetic, gradient-descent-stability]
status: draft
updated_at: 2026-11-01
---

## Overview

Grokking is a training pattern in which a model first memorizes its training
examples and only much later learns the rule behind them. The gap between the
two can be thousands of epochs [arxiv:2604.07822 §5.2].

For multi-hop facts, the paper cites prior work that vanilla transformers learn
two-hop composition only through grokking, and even then do not generalize to
pairs of facts that never appeared together in training [arxiv:2604.07822 §2]
[§5]. The shared invariant across the variants below: the model must store
atomic facts, and the test that matters is whether it answers correctly for
unseen combinations of them, not just for the examples it memorized
[arxiv:2604.07822 §3.2].

Two other cards touch this regime. A sparse autoencoder trained to bind
two-hop relations shows a grokking-style transition: its diagonal accuracy at
the first hop step is 38.4% at epoch 10 and 100% at epoch 40, and validation
accuracy rises later [arxiv:2512.02004 §7.2]. That is a binding metric in an
autoencoder, not a model's own generalization, so the epoch numbers are not
comparable with those above. The second card covers sudden loss spikes in
long unregularised training. Its title sets grokking against glitching
[arxiv:2605.06152]. The authors attribute the spikes to float32 rounding in the
cross-entropy loss, not to the optimiser [arxiv:2605.06152 §1] [§3.1]. The card
records spikes and stepwise test-accuracy rises in modular division
[arxiv:2605.06152 tab. 1] [fig. 2c], but no generalization timescale to set
against the variants below.

## Variants

### Grokking with fixed recurrence (two-hop)

The model is a 4-layer decoder that applies its shared block R times, with
R = 1 as the vanilla case [arxiv:2604.07822 §4] [§5.1]. On held-out two-hop
facts built from unseen atomic facts, R = 1 fails completely. R = 2 shows
non-trivial generalization, but the paper gives no number for it. R = 4
converges faster: about 2k epochs to OOD generalization, against about 7k for
R = 2 [arxiv:2604.07822 §5.2] [fig. 3]. For R = 4 the paper describes a
three-stage process, with memorization within about 10² epochs and systematic
generalization at about 10⁴ epochs [arxiv:2604.07822 §5.2].
Card: [Loop, Think, & Generalize](/papers/2604.07822).

### Curriculum grokking over hop depth

For chains longer than two hops, the paper trains easy-to-hard. It starts with
atomic and two-hop facts and adds three-hop facts once accuracy on a held-out
two-hop split reaches 95%. The same step repeats for each higher k, and every
stage keeps the earlier facts [arxiv:2604.07822 §6.1]. The stop rule uses the
same split for stage selection and for reporting, so a reported "learnable
depth" is optimistic unless a separate validation split is used
[arxiv:2604.07822 §6.1].

Cost is high at the first stage. Grokking to four hops took over 1.3M steps,
and later stages were fast by comparison: 19-hop followed quickly, and 21-hop
took 50 steps from a checkpoint [arxiv:2604.07822 §6.2]. Card:
[Loop, Think, & Generalize](/papers/2604.07822).

### Late loss spikes in long unregularised runs

This is a neighbouring phenomenon, not a generalization variant: the card
does not show that these spikes are grokking. The setting is training with
cross-entropy and Adam and no regularisation, for a long time. Once the model
is very confident, float32 rounding sets the correct-class gradient to exactly
zero. The wrong-class gradients then drift the classifier mean and the feature
mean until the correct-class gradient returns, and Adam takes a very large
step [arxiv:2605.06152 §3.1] [§3.3]. The mitigations the paper tests are:

- Computing the loss in float64 removes the spikes in its MLP, CNN and ViT
  runs [arxiv:2605.06152 fig. 1a] [§4.2].
- Raising Adam's epsilon to 1e-5 removes them; 1e-6 only reduces them. The
  dataset for that figure is not stated [arxiv:2605.06152 §4.2] [fig. 3].
- Projecting the logit gradient to zero sum removes the spike in a run whose
  setup the paper does not name [arxiv:2605.06152 §4.1].
- A BatchNorm layer directly before the classifier removes the spike in VGG11;
  LayerNorm there does not prevent it [arxiv:2605.06152 fig. 3c] [fig. 3b].

Card: [Grokking or Glitching? How Low-Precision Drives Slingshot Loss
Spikes](/papers/2605.06152).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Two-hop composition of facts that stay in the same distribution | Plain training, R = 1 | The paper reports only in-distribution grokking for vanilla models [arxiv:2604.07822 §2] [§5] |
| Two-hop composition over unseen fact pairs | Recurrence with R ≥ 2, fixed | R = 1 fails out of distribution; R = 2 generalizes non-trivially and R = 4 converges faster [arxiv:2604.07822 §5.2] |
| Chains longer than the training depth | Curriculum training, then raise recurrence at inference | Extrapolation to deeper chains came from more inference-time recurrence [arxiv:2604.07822 §6.2] |
| Recurrence past about 15 iterations | Cap the iteration count | For the dynamic model, more than about 15 iterations stops improving OOD accuracy [arxiv:2604.07822 §6.2] |
| Small compute budget | Expect long first stages or reduce the target depth | Four-hop grokking took over 1.3M steps [arxiv:2604.07822 §6.2] |
| Graphs where a short suffix predicts the answer | Use a permutation constraint on the relations | Without it, a model reached high OOD accuracy at 80 hops by reading a short suffix [arxiv:2604.07822 app. K] |
| Large pretrained LLMs | No evidence here | The authors do not claim results carry over to LLMs trained at scale [arxiv:2604.07822 app. A] |
| Long unregularised cross-entropy training with Adam, spikes are a risk | Compute the loss in float64, or raise Adam eps to 1e-5 | Float64 removed spikes in MLP, CNN and ViT runs [arxiv:2605.06152 fig. 1a]; eps 1e-5 removed them, 1e-6 only reduced them [arxiv:2605.06152 §4.2] |
| Mini-batch training over a long run | Watch logit and weight norms, not only the loss | Collapsed samples keep driving the drift, and no visible spike may appear [arxiv:2605.06152 §4.3] |
| Label smoothing is on | Expect a different spike type | It removes the precision-driven spikes but adds EOS-type spikes [arxiv:2605.06152 §4.2] [app. A.2] |

For choosing the number of passes per input, see [adaptive halting](/concepts/adaptive-halting).
For the numerical side of the loss spikes, see [low-precision arithmetic](/concepts/low-precision-arithmetic).

## Implementation sketch

The curriculum loop from the paper, with the stop rule left open. The paper
ends training when the threshold is missed; the released code moves on after a
cumulative epoch cap [arxiv:2604.07822 §6.1].

```pseudocode
# Easy-to-hard curriculum over hop depth  [arxiv:2604.07822 §6.1]
# Each stage keeps every earlier fact, so nothing is forgotten

def curriculum_train(model, atomic, facts_by_hop, heldout_by_hop, k_max, threshold=0.95):
    train_set = atomic + facts_by_hop[2]  # start with atomic and 2-hop facts
    for k in range(2, k_max + 1):
        # Train until held-out k-hop accuracy reaches the threshold  [§6.1]
        reached = train_until(model, train_set, heldout_by_hop[k], threshold)
        if not reached:
            # Stop rule is a choice; the paper stops here  [§6.1]
            break
        # Next stage adds (k+1)-hop facts  [§6.1]
        train_set = train_set + facts_by_hop[k + 1]
    return model
```

## Contradictions

No direct disagreement between the three cards. Their numbers do not compare
directly, though:

- The two-hop results in [Loop, Think, & Generalize](/papers/2604.07822) count
  epochs for a model trained from scratch. The transition in
  [AlignSAE](/papers/2512.02004) is a diagonal binding accuracy for a sparse
  autoencoder [arxiv:2512.02004 §7.2].
- Within [Grokking or Glitching?](/papers/2605.06152), the growth of the
  update at the spike is "about 50 times" the previous epoch in §3.3 and "more
  than 40-fold" in §4.1. The appendix does not fix one value, so do not
  hard-code either [arxiv:2605.06152 §3.3] [§4.1].
- The same paper reports that it could not reproduce the logit divergence of
  Wortsman et al. [arxiv:2605.06152 app. A.4]. That paper has no card on this
  wiki, so its side is not stated here.

In the released code, the curriculum stop rule also differs from the text. See
"Paper vs code" in [Loop, Think, & Generalize](/papers/2604.07822).

## Papers

- [Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers](/papers/2604.07822): recurrent blocks give OOD two-hop generalization and deeper-chain extrapolation, with grokking timescales reported on a synthetic knowledge graph.
- [Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes](/papers/2605.06152): attributes late loss spikes to float32 rounding in cross-entropy, with mitigations tested on MLP, CNN, ViT and one nanoGPT run.
- [AlignSAE: Concept-Aligned Sparse Autoencoders](/papers/2512.02004): a supervised binding stage for a sparse autoencoder; its two-hop run shows a grokking-style rise in binding accuracy.