---
kind: concept
slug: systematic-generalization
title: Systematic generalization
summary: >-
  Handling new combinations of parts seen in training. Position papers argue for dynamic binding into reusable object representations, or for causal, verbalisable variables with sparse changes. CREPE reports lower retrieval on unseen compounds. GrailQA's BERT ranker reaches 53.9 F1 on compositional questions (author-reported).
tags: [compositionality, generalization, binding, benchmark, semantic-parsing]
papers: ["2012.05208", "2011.15091", "2212.07796", "2011.07743"]
related: [object-centric-learning, neural-symbolic-integration, graph-neural-network, domain-generalization, vision-language-pretraining, benchmark-validity, inductive-bias, disentangled-representation, knowledge-base-question-answering, semantic-parsing, causal-inference, meta-learning]
status: draft
updated_at: 2026-11-01
---

## Overview

Systematic generalization is the ability to apply known parts in combinations
that were not seen together during training. The survey frames its absence as
the binding problem: a network cannot route information that is spread over
many units into separate, symbol-like entities that can be reused elsewhere
[arxiv:2012.05208 §2.3]. The authors call this the root cause of weak systematic
generalization [arxiv:2012.05208 §1] [arxiv:2012.05208 §2.3]. The claim is argued
rather than tested in the paper [arxiv:2012.05208 §7].

Every mechanism in the survey shares one requirement: object representations
must stay separate from each other and use a common format, so that they can be
recombined without losing their identity [arxiv:2012.05208 §3.1]. The mechanisms
differ in how they achieve this.

A second position paper frames the same goal as a question of inductive biases.
It argues that data, compute and architecture alone are not enough for
out-of-distribution and systematic generalization [arxiv:2011.15091 §1]. It also
argues that a test set drawn from the training distribution cannot reveal
overfitting to that distribution, so evaluation must change as well
[arxiv:2011.15091 §2].

Two papers measure the symptom instead of proposing a mechanism. A benchmark
tests whether vision-language models retrieve the right caption for a
combination of objects, attributes and relations, and finds lower retrieval
accuracy on compounds absent from the training captions [arxiv:2212.07796 §5.1].
A dataset paper on knowledge-base question answering defines three levels of
test questions, by whether the schema items in each query were seen in training
[arxiv:2011.07743 §2.2]. Both results are observational and author-reported, not
a test of the binding account.

## Variants

The first five variants are mechanisms from the binding survey, so each links
the same card. The survey reviews them as options, not as tested methods. The
sixth is a set of design biases from the second position paper, also untested.
The last two are measurement designs, from a benchmark paper and a dataset paper.

### Slot-based object representations

A fixed set of slots, each holding the features of one object. The survey treats
slots as the simplest option [arxiv:2012.05208 §3.3.1] [arxiv:2012.05208 §3.4].
Card: [On the Binding Problem in Artificial Neural Networks](/papers/2012.05208).

### Distributed codes with separation

Keeps distributed neural codes and adds a mechanism that keeps objects apart
[arxiv:2012.05208 §3.3.2]. The survey judges it harder to build than slots but
more capable [arxiv:2012.05208 §3.4]. Card: [On the Binding Problem in Artificial Neural Networks](/papers/2012.05208).

### Tensor product representations

A binding scheme based on tensor products, reviewed as a more structured
alternative to slots [arxiv:2012.05208 §3.3.3]. The survey places it with
distributed codes in terms of build effort and expressive power
[arxiv:2012.05208 §3.4]. Card: [On the Binding Problem in Artificial Neural Networks](/papers/2012.05208).

### Attractor dynamics

Uses attractor states to represent inputs that admit several stable
interpretations [arxiv:2012.05208 §3.3.4]. Card: [On the Binding Problem in Artificial Neural Networks](/papers/2012.05208).

### Graph neural networks for relational composition

Graph neural networks operate on object representations and their relations.
The survey calls them a promising route to relational responding
[arxiv:2012.05208 §5.3.1] [arxiv:2012.05208 §5.4]. Card: [On the Binding Problem in Artificial Neural Networks](/papers/2012.05208).

### Verbalisable causal variables with sparse changes

Not an architecture. The paper proposes inductive biases taken from high-level,
conscious processing. High-level variables should be verbalisable and often
causal [arxiv:2011.15091 §3.4] [arxiv:2011.15091 §3.5]. Changes should be sparse
and local, touching one or a few variables or mechanisms [arxiv:2011.15091 §3.6].
The joint distribution over those variables should be a sparse factor graph
[arxiv:2011.15091 §3.8], and knowledge should be stored as reusable schemas
whose arguments are bound to instances [arxiv:2011.15091 §3.9]. Stable knowledge
is learned slowly and shared, while volatile knowledge is learned fast
[arxiv:2011.15091 §3.7]. The paper reads soft attention as a soft form of
variable binding, and says discrete typed arguments and hard choices are missing
[arxiv:2011.15091 §3.2]. It gives no algorithm and no experiments
[arxiv:2011.15091 §1]. Card: [Inductive Biases for Deep Learning of Higher-Level Cognition](/papers/2011.15091).

### Seen versus unseen compound splits for vision-language models

Not a mechanism. CREPE builds a zero-shot image-to-text retrieval test on
Visual Genome scene graphs. Atoms are objects, relations or attributes, and
compounds are attribute-object or object-relation-object pairs
[arxiv:2212.07796 §3.1]. Each test caption is placed in a split by whether its
atoms and compounds occur in the training captions of CC-12M, YFCC-15M or
LAION-400M [arxiv:2212.07796 §3.2]. Hard negatives either replace one atom
or join two single-compound foils, so a model that matches loose word patterns
is penalised [arxiv:2212.07796 §4.1]. A second axis varies caption complexity
from 4 to 12 atoms [arxiv:2212.07796 §3.3]. Card: [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](/papers/2212.07796).

### Three levels of new questions for knowledge-base question answering

Not a mechanism. GrailQA groups each question by the schema items in its
logical form: relations, classes and functions [arxiv:2011.07743 §2.2]. A
question is i.i.d. if every item occurs in training and the test questions follow
the training distribution. It is compositional if every item occurs in training
but this combination does not. It is zero-shot if at least one item never occurs
in training [arxiv:2011.07743 §2.2]. The dataset has 64,331 questions. Its
validation and test sets are half zero-shot, a quarter compositional and a
quarter i.i.d. [arxiv:2011.07743 §5.1]. The authors' BERT-based ranker scores
67.0 F1 on i.i.d., 53.9 on compositional and 55.7 on zero-shot test questions
[arxiv:2011.07743 tab. 3]. Card: [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](/papers/2011.07743).

## When to use which

The survey and the causal position paper give no measured comparison between
their proposals. The guidance for those rows repeats their stated judgments and
is not benchmark evidence. The CREPE and GrailQA rows rest on their
author-reported numbers only.

| Situation | Use | Why |
|---|---|---|
| Few objects, and a simple build matters most | Slot-based object representations | The survey calls slots the simplest approach [arxiv:2012.05208 §3.4] |
| Objects must be combined in richer ways, and implementation effort is acceptable | Distributed codes with separation, or tensor product representations | The survey says these are harder to build but support more sophisticated use [arxiv:2012.05208 §3.4] |
| Inputs are ambiguous and admit several stable readings | Attractor dynamics | The survey reviews attractors for multi-stable inputs [arxiv:2012.05208 §3.3.4] |
| Relations between objects must be applied recursively | Graph neural networks | The survey names them as promising for relational responding [arxiv:2012.05208 §5.4] |
| Data arrives as a non-stationary stream, and only some parts of the world change | Sparse, local changes and two learning speeds (Goyal and Bengio, [card](/papers/2011.15091)) | Argued, not measured. The paper says the biases are hypotheses and combining them is open [arxiv:2011.15091 §1] [arxiv:2011.15091 §3.6] [arxiv:2011.15091 §3.7] |
| No object labels are available | Unsupervised segregation, refined later with task signal | The survey argues segregation should be learned largely without supervision [arxiv:2012.05208 §4.4] |
| Need to check whether a vision-language retrieval model handles unseen attribute or relation combinations | Seen versus unseen compound splits (CREPE) | Built for this test. Its unseen-compound drop is largest on atom-level foils for LAION-400M models, up to 12.57 points on one subset [arxiv:2212.07796 app. B.1] |
| Need to check whether a query-generating parser handles new combinations or unseen schema items | Three levels of new questions (GrailQA) | Built for this test, with a stated rule per level [arxiv:2011.07743 §2.2]. Without BERT, ranking loses 21.9 F1 on zero-shot questions [arxiv:2011.07743 §5.3] |

## Implementation sketch

The survey gives no implementation. It does name held-out combinations of
features and of parts as the way to measure separation and composition
[arxiv:2012.05208 §3.4] [arxiv:2012.05208 §5.4]. The sketch below turns that
into a split. It is illustrative and not from the paper.

```pseudocode
# Illustrative only: a split that holds out whole pairings of parts
# Method named in [arxiv:2012.05208 §3.4] and [arxiv:2012.05208 §5.4]
def held_out_combination_split(examples, parts_of, test_fraction):
    # Describe each example by the set of parts it contains
    pairings = {frozenset(parts_of(ex)) for ex in examples}
    # Choose whole pairings for test, so no test pairing is seen in training
    test_pairings = sample(pairings, test_fraction)
    train = [ex for ex in examples
             if frozenset(parts_of(ex)) not in test_pairings]
    test = [ex for ex in examples
            if frozenset(parts_of(ex)) in test_pairings]
    # Every test part must still occur in training, so only pairings are new
    train_parts = {p for ex in train for p in parts_of(ex)}
    assert all(p in train_parts for ex in test for p in parts_of(ex))
    return train, test
```

CREPE uses a related but different rule. It holds out compounds that never
occur in a named training caption set, and it also separates atoms that never
occur there [arxiv:2212.07796 §3.2]. Its split is therefore defined against a
training corpus, not against pairings within one dataset.

GrailQA labels each test question by its schema items instead. The zero-shot
level is any question with an item never seen in training. The compositional
level needs every item seen, plus the combination unseen [arxiv:2011.07743 §2.2].

## Contradictions

Tension, not a test. The survey lists disentanglement among the desiderata for
object representations: factors of variation should map to features in a local,
independent way [arxiv:2012.05208 §3.1.3]. The causal position paper rejects the
assumption, common in disentangling work, that high-level variables are
marginally independent, and asks instead for a sparse joint structure
[arxiv:2011.15091 §3.8]. Neither paper tests the other's proposal.

Tension, not a test. GrailQA reports that its BERT encoder matters most on
compositional and zero-shot questions [arxiv:2011.07743 §5.3]. CREPE reports
that the models trained on the largest corpus still lose the most recall on
unseen compounds, for atom-level foils [arxiv:2212.07796 app. B.1]. The tasks,
metrics and models differ, so the two results are not compared.

Otherwise none directly. The sources do not test the same claim. The survey
argues that scaling alone will not suffice [arxiv:2012.05208 §7], and the causal
paper argues that data, compute and architecture are not enough without
further inductive biases [arxiv:2011.15091 §1]. Both are argued, not measured.
CREPE reports that the LAION-400M models score highest in absolute recall on the
combined seen-compound set and still lose recall on unseen compounds
[arxiv:2212.07796 app. B.1]. All three facts are reported as they stand. None of
the sources settles another.

## Papers

- [On the Binding Problem in Artificial Neural Networks](/papers/2012.05208): a
  position survey that splits the binding problem into representation,
  segregation and composition, and reviews mechanisms for each.
- [Inductive Biases for Deep Learning of Higher-Level Cognition](/papers/2011.15091): a position paper that proposes inductive biases from high-level cognition for out-of-distribution and systematic generalization. It gives no experiments.
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](/papers/2212.07796): a retrieval benchmark that splits test captions into seen and unseen compounds and varies caption complexity.
- [Beyond I.I.D.: Three Levels of Generalization for Question Answering on Knowledge Bases](/papers/2011.07743): a dataset and semantic parser that tests knowledge-base questions at i.i.d., compositional and zero-shot levels.