How to represent part-whole hierarchies in a neural network

In plain words

Neural networks find it hard to show that a picture is made of objects, the objects are made of parts, and the parts are made of smaller parts, when each picture has a different structure and the network must keep the same wiring for every picture. Hinton's answer is to give every spot in the image a vector at each level of that hierarchy. Spots that belong to the same object or part end up with exactly the same vector, so the matching vectors form islands, and the islands together act as the parse tree of the picture.

The paper describes a design, not a working system, and it reports no experiments. Its value lies in the ideas it combines: attention-style smoothing between neighbouring spots, agreement between levels, and one set of weights reused at every spot. Read it as a proposal to test, not as a method to copy.

TL;DR

This paper is a design proposal, not an implementation. It proposes GLOM, a fixed-architecture network of identical columns that keeps one vector per location and per part-whole level, and whose same-level vectors form islands of agreement. Each vector is updated over time from a weighted average of bottom-up, top-down, its own previous value and attention-weighted neighbours. Nothing is measured in the paper, so the parameters and training details below are mostly "not stated".

Problem

A network with a fixed architecture has to represent a parse tree whose shape differs per image [abstract]. A real network cannot allocate a group of neurons to a new node on the fly, nor rewire its weights quickly §1.

Capsule models handled this by dedicating a neuron group to each possible part type in each image region and routing between them §1, §2. The paper argues that this forces hard decisions about part identity, that routing is needed, and that capsule models have not scaled well to larger datasets §2. The paper considers a single fixation on a static image; video is only sketched §2, §11.

Method

GLOM is treated as a recurrent pipeline over a sequence of frames, and a static image is a sequence of identical frames §1. It is built from many columns, and every column uses one shared set of weights. Within a column, each pair of adjacent levels is linked by a small local autoencoder, and each level holds an embedding vector §1, fig. 1. The example in the text is a face: one column may converge on vectors for a nostril, a nose, a face and a person §1.

Steps for one settling iteration, for each column separately §1:

1. Bottom-up prediction: a multilayer net maps the embedding of the level below, from the previous time step, to a prediction for this level §1, fig. 1.
2. Top-down prediction: a multilayer net maps the embedding of the level above, from the previous time step, to a prediction for this level §1, fig. 1.
3. Self: the embedding of this level at the previous time step §1.
4. Same-level smoothing: an attention-weighted average of the same-level embeddings at nearby columns, at the previous time step §1, §3.4.
5. Update: the new embedding is a weighted average of the four contributions above. The weights are not given §1.

# One settling iteration for all columns  [§1]
# Reads only the previous time step, so updates are synchronous  [§1]
# embeddings[x][l]: vector for location x at level l  [§1]

def settle_step(embeddings, bottom_up, top_down, candidates):
    previous = copy(embeddings)
    for location in locations:
        for level in levels:
            # Level 0 is fed by the conv-net output for its patch  [§3.5]
            from_below = bottom_up[level](previous[location][level - 1])
            # Top-level behaviour is not stated  [§1]
            from_above = top_down[level](previous[location][level + 1])
            # Self: this level's embedding at the previous step  [§1]
            itself = previous[location][level]
            # Same-level smoothing over sampled nearby locations  [§3.4] [§6]
            nearby = attention_average(
                previous[location][level],
                candidates[location][level],
                beta[level],
            )
            # Weighted average of the four; weights not stated  [§1]
            embeddings[location][level] = weighted_mean(
                [from_below, from_above, itself, nearby],
                weights[level],
            )
    return embeddings

The same-level interaction is a simplified attention in which query, key and value are all the same embedding §1, §10.2. The weight that location x gives to location y is the softmax of a scaled dot product §3.4, eq. 1:

w_{xy} = \frac{e^{\beta L_x \cdot L_y}}{\sum_z e^{\beta L_x \cdot L_z}} eq. 1

Here L_x is the embedding at location x, \beta is an inverse temperature, and z runs over the locations x attends to at that level §3.4. Candidates are sampled without replacement from a Gaussian centred on x; higher levels use a larger Gaussian §6. The paper allows the sampling to be done once at design time or at each time step §6.

# Attention average over sampled candidates at one level  [§3.4] [eq. 1]
# query = key = value = embedding  [§1]

def attention_average(embedding_x, candidates, beta):
    # Score each candidate by scaled scalar product  [eq. 1]
    scores = [beta * dot(embedding_x, c) for c in candidates]
    # Softmax so the weights sum to one  [eq. 1]
    weights = softmax(scores)
    # Weighted average of the candidate embeddings  [§1]
    return sum(w * c for w, c in zip(weights, candidates))

For a static image, embeddings settle into islands of nearly identical vectors, with larger islands at higher levels §1, fig. 2. Each location holds its own copy of the vector for the node it belongs to, which replaces pointers and RAM look-ups §1, §6.

The bottom level receives an embedding from a convolutional net that sees a larger patch. That output can be refined by backpropagating the reconstruction error through a neural field §3.5. A coarser convolutional net can initialise higher levels §3.5.

Pose is carried as a sub-vector of the embedding, and the bottom-up and top-down nets receive the whole part-level vector, including identity, so they can apply the coordinate transform conditioned on identity §8. Each location is assumed to hold at most one part per level, so binding is done by location §9.

Uncertainty is represented by populations of neurons whose activities add log-scale bumps, so distributions multiply by adding activities §9. Sources are combined with a weighted geometric mean of their distributions, not a product, because the sources are alternative paths and not independent §9.1. The weight of the bottom-up source should be highest first, and top-down and lateral sources should gain weight later §9.1.

Training: the system is trained end-to-end to reconstruct the uncorrupted image at the final time step from an input with missing regions §1, §7. Two regularisers encourage each new embedding to agree with the bottom-up and top-down predictions, so islands form §1. Agreement alone can collapse the embeddings to small, similar vectors, so contrastive learning is applied across locations: same-entity locations at a level should agree, different entities should not §7. The paper suggests that islands formed by attention may make negatives unnecessary, but this is conjecture §7.

Each net is trained to predict a consensus: the weighted geometric mean of the top-down prediction, the bottom-up prediction, the neighbour average and the previous embedding §7, §12.1. The paper calls this co-distillation, with the ensemble of students as the teacher §12.1.

# Consensus target for one level at one location  [§7] [§12.1]
# Inputs: distributions from top-down, bottom-up, neighbours and self  [§7]
# Weights are not stated  [§9.1]

def consensus(distributions, weights):
    # Geometric mean = weighted sum of log-probabilities  [§9.1]
    log_mix = sum(w * log(p) for p, w in zip(distributions, weights))
    # Renormalise so the result has total mass one  [§9.1]
    return normalise(exp(log_mix))

Hyperparameters

No hyperparameters or training settings are given. The values below are the paper's own remarks.

Results

The paper reports no experiments and no benchmark numbers. It says it does not describe a working system [abstract] and that the absence of an implementation raises credibility concerns §13. The only numbers are back-of-envelope biological estimates, such as about 10^13 synapses in the visual system against about 10^9 fixations in the first ten years §12.1. Do not use them as results.

Paper vs code

No official code.

The source has no URL and the raw metadata has no code link. A GitHub repository search for "glom hinton" returned only third-party reimplementations, for example lucidrains/glom-pytorch and Rishit-dagli/GLOM-TensorFlow. None is owned by the author or a Hinton lab, so none was used.

Pitfalls

• Not a system. The author calls GLOM "imaginary" [abstract]. Every mechanism is a proposal, and the paper defers the hard learning problem for the hidden layers to a companion paper that is not part of this source §12.3.
• Missing numbers. Embedding size, network sizes, number of levels, time steps, β, all update weights, geometric-mean weights, regulariser weight, sample count and optimiser are all not stated §1, §3.4, §9.1.
• Collapse. An agreement regulariser alone shrinks embeddings toward small, similar vectors §7. The fix the paper proposes is contrastive learning with negatives §7. Dropping negatives is speculative §7.
• Distribution, not vector. The consensus and the uncertainty are defined over distributions represented by neuron populations §9, §9.1. The paper does not say how a vector embedding maps to such a distribution, and it says the basis functions need not be local §9. An implementer must choose an encoding.
• Geometric mean. The geometric mean is renormalised so that the product has total mass one §9.1. Weights are described only as a direction of change over iterations §9.1.
• Edges of the hierarchy. Behaviour at the top level, which has no level above, and at the bottom level, which has no level below, is not stated beyond the convolutional input at the bottom §3.5.
• Grid and stride. The paper assumes the same location grid at every level and calls this "probably not the best choice" §3.2. If higher levels use a larger stride, the top-down net needs several outputs per location §3.2.
• Neighbours in the bottom-up net. Letting the bottom-up net look at nearby locations is a "less pure" variant. It breaks the property that identical part vectors give identical predictions §3.3.
• No learned projections in same-level attention. Query, key and value are the embedding itself §1, §10.2. The attention temperature β is not scheduled §3.4.
• Cost. Each location holds a vector per level and its own bottom-up and top-down nets §6, §10.1. The paper gives no compute estimate.
• Fast change. Settling needs time. The paper says complicated shapes cannot be read correctly when images change rapidly §11.
• Weight sharing vs distillation. Shared weights across columns are the design. The distillation route is presented as a biological argument, not tested §12.1. Do not treat it as a requirement.
• Backpropagation through time. The straightforward training backpropagates the reconstruction error through the whole settling process §12.3. The paper calls this hard to reconcile with a brain and proposes local consensus signals as an alternative, without a worked method §12.3.
• Transformer equivalence. Unrolling time into layers gives a transformer with tied weights per layer and heads reused as levels §10.2, fig. 4. The four stated changes from a standard transformer are the detail to copy §10.2.
• Biology. The sleep-based negative learning and the spike-timing account are speculation §12.2, §12.3. Leave them out of any implementation.

Lineage

• Builds on: capsule networks, cited as the motivation and as the EM capsule and stacked capsule autoencoder models §2, §5, §10.1; transformers and BERT §1, §10.2; set transformers §2.2; neural fields §2.3; contrastive learning §7; knowledge distillation §12.1.
• Compared against: capsule models §10.1, transformers §10.2, convolutional nets §10.3, 2-D Ising models §10.5 and restricted Boltzmann machines §10.6.
• The paper's own references are bibliography keys with no arXiv ids in the source, so none is linked.