Object-centric learning

Overview

The goal is to split each input into its entities and give each one its own representation, in a shared format, so that the parts can be reused and recombined. The survey calls the failure to do this the binding problem: the features of one entity must stay together and stay apart from those of other entities 2012.05208 §2.3 2012.05208 §3.

The invariant the variants share is a partition of the input. Each part has its own representation, and the partition is learned from data rather than given by labels. The N-EM paper assumes that pixels of one object depend on each other, while pixels of different objects are roughly independent once their representations are known 1708.03498 §2. The variants differ in how the partition is found and whether the model can also predict the next frame.

Variants

Iterative grouping with EM (N-EM)

The input is modelled as a mixture of K components. Each component is a neural network that turns its representation into per-pixel parameters, and each pixel has a latent assignment to a component 1708.03498 §2.1. An E-step computes per-pixel responsibilities, and an M-step takes one gradient step on the expected log-likelihood. Repeating these steps a fixed number of times and back-propagating through them trains the whole loop end to end [arxiv:1708.03498 §2.2] 1708.03498 §2.3. Card: Neural Expectation Maximization.

Author-reported: on static shapes, AMI 0.475 ± 0.043 with 15 EM steps 1708.03498 §4.1 1708.03498 app. A.1.

Recurrent grouping for sequences (RNN-EM)

RNN-EM replaces the hand-derived M-step with an unrestricted recurrent network that receives the responsibility-weighted residual at each step. The weights are not constrained, so convergence of the likelihood is no longer guaranteed. The recurrent form makes the model applicable to sequences and to next-step prediction 1708.03498 §2.3. Card: Neural Expectation Maximization, the same card as N-EM.

Author-reported: AMI 0.826 ± 0.005 on static shapes 1708.03498 §4.1; 0.970 ± 0.005 on flying shapes with three objects and K = 3 [arxiv:1708.03498 tab. 1]; 0.819 ± 0.022 on two flying digits trained directly and 0.917 ± 0.005 trained in stages 1708.03498 §4.3.

Mechanisms from the survey

The survey lists mechanisms for each of its three parts, with no algorithm or numbers. For representation: slot-based codes, augmented distributed codes, tensor product representations and attractor dynamics 2012.05208 §3.3. For segregation: clustering, neural segmentation, sequential attention and probabilistic generative models 2012.05208 §4.3. For composition: graph neural networks and neural computers 2012.05208 §5.3. Card: On the Binding Problem in Artificial Neural Networks. Use it as a map of the design space and then go to the primary sources for code.

Synthetic video benchmarks (MOVi)

MOVi is a benchmark family, not a method. Kubric renders MOVi-A to MOVi-E, five video sets of rising complexity. MOVi-A to -C have up to 10 moving objects over 24 frames; MOVi-D and -E have up to 23 objects, with only a few moving. MOVi-C adds real-world backgrounds and scanned objects 2203.03570 §4.1. The generator exports instance segmentation for each scene 2203.03570 §3, which is the ground truth these benchmarks score against.

Author-reported FG-ARI (higher is better) on MOVi-A to MOVi-E [arxiv:2203.03570 §4.1]:

SAVi and SIMONe are the methods the Kubric paper compares; their own papers have no card here, so these numbers are the only content used. The trends differ by method: SAVi falls steeply from MOVi-A to MOVi-E, while SIMONe scores higher on MOVi-D and -E than on MOVi-A 2203.03570 §4.1. The benchmark does not show how either method does on real video.

When to use which

Implementation sketch

The loop shared by N-EM and RNN-EM, with the choices that differ marked. The paper's own equations give the details 1708.03498 §2.2 1708.03498 §2.4.

# Iterative grouping: one input x, K components  [arxiv:1708.03498 §2.2]
# Input:  x (pixels), thetas (K representations), steps (fixed count)
# Output: thetas, responsibilities gamma (per pixel, per component)

def group(x, thetas, steps):
    # Start from equal responsibilities; the paper leaves init open
    # [arxiv:1708.03498 §2.2]
    gamma = uniform_responsibilities(x)
    for t in range(steps):
        # M-step: one gradient step on the expected log-likelihood, or
        # an RNN update for RNN-EM  [arxiv:1708.03498 §2.2] [§2.3]
        thetas = m_step(x, thetas, gamma)
        # E-step: per-pixel responsibilities; each pixel's gammas sum to 1
        # [arxiv:1708.03498 §2.2] [eq. 3]
        gamma = e_step(x, thetas)
    return thetas, gamma

# Training loss for one input and one prediction step  [arxiv:1708.03498 §2.4]
# The inter term is penalised with a positive weight (see Contradictions)

def grouping_loss(x, preds, gamma, prior, w):
    # Intra term: responsibility-weighted negative log-likelihood
    # [arxiv:1708.03498 §2.4] [eq. 5]
    intra = gamma * neg_log_likelihood(x, preds)
    # Inter term: each component is penalised for predicting pixels it
    # does not own, weighted by (1 - gamma)  [arxiv:1708.03498 §2.4]
    inter = (1 - gamma) * kl_divergence(prior, preds)
    return sum(intra) + w * sum(inter)

The paper stops gradients through γ in practice, so the loss does not reward changing responsibilities 1708.03498 §2.4. Whether to stop them is a choice the implementation must make explicitly.

Contradictions

• K for static shapes. The main text says K = 3 1708.03498 §4.1. The appendix uses K = 4 1708.03498 app. A.1. Neither source says which setting produced the reported score.
• Sign of the inter term. As written, eq. 5 subtracts the (1 − γ)-weighted divergence. The text describes that term as a penalty 1708.03498 §2.4 1708.03498 eq. 5. Implement it as a penalty.
• Stopping gradients through γ. The paper stops them in general 1708.03498 §2.4, and its flying-shapes appendix does so [arxiv:1708.03498 app. A.2]. The released code does not stop them by default; see the N-EM card, Paper vs code section.
• Survey claims against measured results. The survey argues that scaling alone will not produce object representations and that segregation should be unsupervised 2012.05208 §7. It does not measure either claim, and the N-EM paper does not test scaling, so no source here contradicts it directly.

Papers

• Neural Expectation Maximization: iterative EM-style grouping with neural components, and RNN-EM for sequences; author-reported AMI on synthetic shapes and digits.
• On the Binding Problem in Artificial Neural Networks: a position survey that splits the binding problem into representation, segregation and composition, and lists mechanisms and open problems.
• Kubric: A scalable dataset generator: the generator behind the MOVi object-discovery benchmarks; author-reported FG-ARI for SAVi and SIMONe on MOVi-A to -E.