---
kind: concept
slug: object-centric-learning
title: Object-centric learning
summary: >-
  Learn one representation per object from unlabelled inputs. Variants: EM-style iterative pixel grouping (N-EM) and its recurrent version for sequences (RNN-EM); a survey of binding mechanisms with no experiments. MOVi video benchmarks (from Kubric) measure object discovery. Author-reported numbers are on synthetic data, so they do not show real-image behaviour.
tags: [unsupervised-learning, object-centric]
papers: ["1708.03498", "2012.05208", "2203.03570"]
related: [expectation-maximization, clustering-evaluation, recurrent-neural-network, neural-symbolic-integration, instance-segmentation, physically-based-rendering, domain-randomization]
status: draft
updated_at: 2026-10-24
---

## Overview

The goal is to split each input into its entities and give each one its own
representation, in a shared format, so that the parts can be reused and
recombined. The survey calls the failure to do this the binding problem: the
features of one entity must stay together and stay apart from those of other
entities [arxiv:2012.05208 §2.3] [arxiv:2012.05208 §3].

The invariant the variants share is a partition of the input. Each part has its
own representation, and the partition is learned from data rather than given by
labels. The N-EM paper assumes that pixels of one object depend on each other,
while pixels of different objects are roughly independent once their
representations are known [arxiv:1708.03498 §2]. The variants differ in how the
partition is found and whether the model can also predict the next frame.

## Variants

### Iterative grouping with EM (N-EM)

The input is modelled as a mixture of K components. Each component is a neural
network that turns its representation into per-pixel parameters, and each pixel
has a latent assignment to a component [arxiv:1708.03498 §2.1]. An E-step
computes per-pixel responsibilities, and an M-step takes one gradient step on
the expected log-likelihood. Repeating these steps a fixed number of times and
back-propagating through them trains the whole loop end to end [arxiv:1708.03498
§2.2] [arxiv:1708.03498 §2.3]. Card: [Neural Expectation Maximization](/papers/1708.03498).

Author-reported: on static shapes, AMI 0.475 ± 0.043 with 15 EM steps
[arxiv:1708.03498 §4.1] [arxiv:1708.03498 app. A.1].

### Recurrent grouping for sequences (RNN-EM)

RNN-EM replaces the hand-derived M-step with an unrestricted recurrent network
that receives the responsibility-weighted residual at each step. The weights are
not constrained, so convergence of the likelihood is no longer guaranteed. The
recurrent form makes the model applicable to sequences and to next-step
prediction [arxiv:1708.03498 §2.3]. Card: [Neural Expectation Maximization](/papers/1708.03498),
the same card as N-EM.

Author-reported: AMI 0.826 ± 0.005 on static shapes [arxiv:1708.03498 §4.1];
0.970 ± 0.005 on flying shapes with three objects and K = 3 [arxiv:1708.03498
tab. 1]; 0.819 ± 0.022 on two flying digits trained directly and 0.917 ± 0.005
trained in stages [arxiv:1708.03498 §4.3].

### Mechanisms from the survey

The survey lists mechanisms for each of its three parts, with no algorithm or
numbers. For representation: slot-based codes, augmented distributed codes,
tensor product representations and attractor dynamics [arxiv:2012.05208 §3.3].
For segregation: clustering, neural segmentation, sequential attention and
probabilistic generative models [arxiv:2012.05208 §4.3]. For composition:
graph neural networks and neural computers [arxiv:2012.05208 §5.3]. Card:
[On the Binding Problem in Artificial Neural Networks](/papers/2012.05208). Use
it as a map of the design space and then go to the primary sources for code.

### Synthetic video benchmarks (MOVi)

MOVi is a benchmark family, not a method. Kubric renders MOVi-A to MOVi-E, five
video sets of rising complexity. MOVi-A to -C have up to 10 moving objects over
24 frames; MOVi-D and -E have up to 23 objects, with only a few moving. MOVi-C
adds real-world backgrounds and scanned objects [arxiv:2203.03570 §4.1]. The
generator exports instance segmentation for each scene [arxiv:2203.03570 §3],
which is the ground truth these benchmarks score against.

Author-reported FG-ARI (higher is better) on MOVi-A to MOVi-E [arxiv:2203.03570
§4.1]:

| Model | MOVi-A | MOVi-B | MOVi-C | MOVi-D | MOVi-E |
|---|---|---|---|---|---|
| SAVi (uses optical flow in training) | 82.0 | 61.5 | 47.0 | 19.4 | 2.7 |
| SIMONe | 61.8 | 30.7 | 19.8 | 34.1 | 34.9 |
| SAVi + bounding boxes in frame 1 | 95.3 | 85.5 | 73.5 | 9.9 | 7.5 |

SAVi and SIMONe are the methods the Kubric paper compares; their own papers have
no card here, so these numbers are the only content used. The trends differ by
method: SAVi falls steeply from MOVi-A to MOVi-E, while SIMONe scores higher on
MOVi-D and -E than on MOVi-A [arxiv:2203.03570 §4.1]. The benchmark does not
show how either method does on real video.

## When to use which

| Situation | Use | Why |
|---|---|---|
| You want the grouping update to follow the EM derivation | N-EM | The update comes from the EM gradient step [arxiv:1708.03498 §2.2]. Its static-shape score is lower, 0.475 against 0.826 for RNN-EM [arxiv:1708.03498 §4.1] |
| Static images, best reported score on the synthetic benchmark | RNN-EM | 0.826 AMI on static shapes [arxiv:1708.03498 §4.1] |
| Frames in a sequence, and a next-frame prediction objective | RNN-EM | The recurrent model supports next-step prediction [arxiv:1708.03498 §2.3] and reports flying-shape results [arxiv:1708.03498 tab. 1] |
| You need a convergence guarantee for the grouping | N-EM | RNN-EM gives that guarantee up [arxiv:1708.03498 §2.3] |
| The object count changes between training and use | Plan for a drop | Train and test with three objects at K = 3 give AMI 0.970; five objects at K = 3 give 0.614 [arxiv:1708.03498 tab. 1] |
| Labelled video benchmark for object discovery | MOVi (Kubric) | The generator exports instance segmentation [arxiv:2203.03570 §3], and FG-ARI baselines are reported for five difficulty levels [arxiv:2203.03570 §4.1] |
| Many objects per scene, most of them static | Expect low scores | SAVi falls from 82.0 on MOVi-A to 2.7 on MOVi-E [arxiv:2203.03570 §4.1] |
| Real photographs or a large unlabelled dataset | None of the cards settles it | Both N-EM tests use synthetic shapes or digits [arxiv:1708.03498 §4]. MOVi is rendered, and MOVi-C uses real backgrounds and scanned objects [arxiv:2203.03570 §4.1]. The survey reports no experiments |
| Deciding what to build before any code | The survey | It names the requirements and open problems [arxiv:2012.05208 §3.1] [arxiv:2012.05208 §7] |

## Implementation sketch

The loop shared by N-EM and RNN-EM, with the choices that differ marked. The
paper's own equations give the details [arxiv:1708.03498 §2.2] [arxiv:1708.03498 §2.4].

```pseudocode
# Iterative grouping: one input x, K components  [arxiv:1708.03498 §2.2]
# Input:  x (pixels), thetas (K representations), steps (fixed count)
# Output: thetas, responsibilities gamma (per pixel, per component)

def group(x, thetas, steps):
    # Start from equal responsibilities; the paper leaves init open
    # [arxiv:1708.03498 §2.2]
    gamma = uniform_responsibilities(x)
    for t in range(steps):
        # M-step: one gradient step on the expected log-likelihood, or
        # an RNN update for RNN-EM  [arxiv:1708.03498 §2.2] [§2.3]
        thetas = m_step(x, thetas, gamma)
        # E-step: per-pixel responsibilities; each pixel's gammas sum to 1
        # [arxiv:1708.03498 §2.2] [eq. 3]
        gamma = e_step(x, thetas)
    return thetas, gamma
```

```pseudocode
# Training loss for one input and one prediction step  [arxiv:1708.03498 §2.4]
# The inter term is penalised with a positive weight (see Contradictions)

def grouping_loss(x, preds, gamma, prior, w):
    # Intra term: responsibility-weighted negative log-likelihood
    # [arxiv:1708.03498 §2.4] [eq. 5]
    intra = gamma * neg_log_likelihood(x, preds)
    # Inter term: each component is penalised for predicting pixels it
    # does not own, weighted by (1 - gamma)  [arxiv:1708.03498 §2.4]
    inter = (1 - gamma) * kl_divergence(prior, preds)
    return sum(intra) + w * sum(inter)
```

The paper stops gradients through γ in practice, so the loss does not reward
changing responsibilities [arxiv:1708.03498 §2.4]. Whether to stop them is a
choice the implementation must make explicitly.

## Contradictions

- **K for static shapes.** The main text says K = 3 [arxiv:1708.03498 §4.1].
  The appendix uses K = 4 [arxiv:1708.03498 app. A.1]. Neither source says
  which setting produced the reported score.
- **Sign of the inter term.** As written, eq. 5 subtracts the (1 − γ)-weighted
  divergence. The text describes that term as a penalty [arxiv:1708.03498 §2.4]
  [arxiv:1708.03498 eq. 5]. Implement it as a penalty.
- **Stopping gradients through γ.** The paper stops them in general
  [arxiv:1708.03498 §2.4], and its flying-shapes appendix does so [arxiv:1708.03498
  app. A.2]. The released code does not stop them by default; see the
  [N-EM card](/papers/1708.03498), Paper vs code section.
- **Survey claims against measured results.** The survey argues that scaling
  alone will not produce object representations and that segregation should be
  unsupervised [arxiv:2012.05208 §7]. It does not measure either claim, and the
  N-EM paper does not test scaling, so no source here contradicts it directly.

## Papers

- [Neural Expectation Maximization](/papers/1708.03498): iterative
  EM-style grouping with neural components, and RNN-EM for sequences; author-reported
  AMI on synthetic shapes and digits.
- [On the Binding Problem in Artificial Neural Networks](/papers/2012.05208):
  a position survey that splits the binding problem into representation,
  segregation and composition, and lists mechanisms and open problems.
- [Kubric: A scalable dataset generator](/papers/2203.03570): the generator
  behind the MOVi object-discovery benchmarks; author-reported FG-ARI for SAVi
  and SIMONe on MOVi-A to -E.