Grokking

Overview

Grokking is a training pattern in which a model first memorizes its training examples and only much later learns the rule behind them. The gap between the two can be thousands of epochs 2604.07822 §5.2.

For multi-hop facts, the paper cites prior work that vanilla transformers learn two-hop composition only through grokking, and even then do not generalize to pairs of facts that never appeared together in training 2604.07822 §2 §5. The shared invariant across the variants below: the model must store atomic facts, and the test that matters is whether it answers correctly for unseen combinations of them, not just for the examples it memorized 2604.07822 §3.2.

Two other cards touch this regime. A sparse autoencoder trained to bind two-hop relations shows a grokking-style transition: its diagonal accuracy at the first hop step is 38.4% at epoch 10 and 100% at epoch 40, and validation accuracy rises later 2512.02004 §7.2. That is a binding metric in an autoencoder, not a model's own generalization, so the epoch numbers are not comparable with those above. The second card covers sudden loss spikes in long unregularised training. Its title sets grokking against glitching 2605.06152. The authors attribute the spikes to float32 rounding in the cross-entropy loss, not to the optimiser 2605.06152 §1 §3.1. The card records spikes and stepwise test-accuracy rises in modular division 2605.06152 tab. 1 fig. 2c, but no generalization timescale to set against the variants below.

Variants

Grokking with fixed recurrence (two-hop)

The model is a 4-layer decoder that applies its shared block R times, with R = 1 as the vanilla case 2604.07822 §4 §5.1. On held-out two-hop facts built from unseen atomic facts, R = 1 fails completely. R = 2 shows non-trivial generalization, but the paper gives no number for it. R = 4 converges faster: about 2k epochs to OOD generalization, against about 7k for R = 2 2604.07822 §5.2 fig. 3. For R = 4 the paper describes a three-stage process, with memorization within about 10² epochs and systematic generalization at about 10⁴ epochs 2604.07822 §5.2. Card: Loop, Think, & Generalize.

Curriculum grokking over hop depth

For chains longer than two hops, the paper trains easy-to-hard. It starts with atomic and two-hop facts and adds three-hop facts once accuracy on a held-out two-hop split reaches 95%. The same step repeats for each higher k, and every stage keeps the earlier facts 2604.07822 §6.1. The stop rule uses the same split for stage selection and for reporting, so a reported "learnable depth" is optimistic unless a separate validation split is used 2604.07822 §6.1.

Cost is high at the first stage. Grokking to four hops took over 1.3M steps, and later stages were fast by comparison: 19-hop followed quickly, and 21-hop took 50 steps from a checkpoint 2604.07822 §6.2. Card: Loop, Think, & Generalize.

Late loss spikes in long unregularised runs

This is a neighbouring phenomenon, not a generalization variant: the card does not show that these spikes are grokking. The setting is training with cross-entropy and Adam and no regularisation, for a long time. Once the model is very confident, float32 rounding sets the correct-class gradient to exactly zero. The wrong-class gradients then drift the classifier mean and the feature mean until the correct-class gradient returns, and Adam takes a very large step 2605.06152 §3.1 §3.3. The mitigations the paper tests are:

• Computing the loss in float64 removes the spikes in its MLP, CNN and ViT runs 2605.06152 fig. 1a §4.2.
• Raising Adam's epsilon to 1e-5 removes them; 1e-6 only reduces them. The dataset for that figure is not stated 2605.06152 §4.2 fig. 3.
• Projecting the logit gradient to zero sum removes the spike in a run whose setup the paper does not name 2605.06152 §4.1.
• A BatchNorm layer directly before the classifier removes the spike in VGG11; LayerNorm there does not prevent it 2605.06152 fig. 3c fig. 3b.

Card: Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes.

When to use which

For choosing the number of passes per input, see adaptive halting. For the numerical side of the loss spikes, see low-precision arithmetic.

Implementation sketch

The curriculum loop from the paper, with the stop rule left open. The paper ends training when the threshold is missed; the released code moves on after a cumulative epoch cap 2604.07822 §6.1.

# Easy-to-hard curriculum over hop depth  [arxiv:2604.07822 §6.1]
# Each stage keeps every earlier fact, so nothing is forgotten

def curriculum_train(model, atomic, facts_by_hop, heldout_by_hop, k_max, threshold=0.95):
    train_set = atomic + facts_by_hop[2]  # start with atomic and 2-hop facts
    for k in range(2, k_max + 1):
        # Train until held-out k-hop accuracy reaches the threshold  [§6.1]
        reached = train_until(model, train_set, heldout_by_hop[k], threshold)
        if not reached:
            # Stop rule is a choice; the paper stops here  [§6.1]
            break
        # Next stage adds (k+1)-hop facts  [§6.1]
        train_set = train_set + facts_by_hop[k + 1]
    return model

Contradictions

No direct disagreement between the three cards. Their numbers do not compare directly, though:

• The two-hop results in Loop, Think, & Generalize count epochs for a model trained from scratch. The transition in AlignSAE is a diagonal binding accuracy for a sparse autoencoder 2512.02004 §7.2.
• Within Grokking or Glitching?, the growth of the update at the spike is "about 50 times" the previous epoch in §3.3 and "more than 40-fold" in §4.1. The appendix does not fix one value, so do not hard-code either 2605.06152 §3.3 §4.1.
• The same paper reports that it could not reproduce the logit divergence of Wortsman et al. 2605.06152 app. A.4. That paper has no card on this wiki, so its side is not stated here.

In the released code, the curriculum stop rule also differs from the text. See "Paper vs code" in Loop, Think, & Generalize.

Papers

• Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers: recurrent blocks give OOD two-hop generalization and deeper-chain extrapolation, with grokking timescales reported on a synthetic knowledge graph.
• Grokking or Glitching? How Low-Precision Drives Slingshot Loss Spikes: attributes late loss spikes to float32 rounding in cross-entropy, with mitigations tested on MLP, CNN, ViT and one nanoGPT run.
• AlignSAE: Concept-Aligned Sparse Autoencoders: a supervised binding stage for a sparse autoencoder; its two-hop run shows a grokking-style rise in binding accuracy.