---
kind: concept
slug: relation-network
title: Relation network
summary: >-
  A set-scoring network that applies one learned function to every pair of object embeddings, sums the results and scores the sum with a second network. Variants differ in what feeds the pairs: a CNN trained from scratch, a frozen pretrained encoder, ground-truth factors, or unsupervised disentangled encoders. One cited paper, on Raven-style puzzles.
tags: [abstract-reasoning, relational-reasoning, representation-learning]
papers: ["1905.12506"]
related: [disentangled-representation, variational-autoencoder]
status: draft
updated_at: 2026-10-19
---

## Overview

A relation network scores a set of objects by comparing every pair of them.
One learned function g takes two object embeddings and returns a relation
vector. The relation vectors are summed over all pairs, and a second function f
turns that sum into a score [arxiv:1905.12506 eq. 2] [arxiv:1905.12506 §3]. The
pair sum is the shared core: any two objects can interact, and the model does
not need to know in advance which pairs matter.

The only cited paper uses this design as a Wild Relation Network (WReN) for
3 × 3 Raven-style puzzles. Each of the eight context panels and each candidate
answer is embedded, and each candidate is scored against the context
[arxiv:1905.12506 §3]. The relation network itself comes from Santoro et al.
2017, which the paper cites [arxiv:1905.12506 §3]. No card exists for that
paper, so its content is not used on this page.

## Variants

The paper varies the input to the relation network rather than its form. The
design fixed by eq. 2 stays the same across the variants below.

### Learned from scratch

The panel embedding is a CNN trained together with the reasoner, with the same
architecture as the disentanglement encoders [arxiv:1905.12506 §4.2.1]. The
paper reports that these runs often stall. Some predict every answer with equal
probability, and runs differ widely [arxiv:1905.12506 §4.2.1]. Card:
[Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506).

### Frozen pretrained representation

The reasoner receives a frozen pretrained representation instead of a learned
one [arxiv:1905.12506 §4.2.1]. The card does not report a separate accuracy for
this input. The paper's headline numbers, about 80% on dSprites and about 90%
on 3dshapes, are averages over the models of the initial study, so they are
not the accuracy of any one input or method [arxiv:1905.12506 §4.2.1]. Card:
[Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506).

### Ground-truth factor values

The reasoner receives the true factor values of each panel instead of an image
embedding [arxiv:1905.12506 §4.2.1]. The paper uses this as a comparison point.
The card gives no separate accuracy for it. Card:
[Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506).

### Disentangled encoder embeddings

The paper's main study. Each panel is embedded by an unsupervised disentangled
encoder: beta-VAE, FactorVAE, beta-TCVAE, DIP-VAE-I or DIP-VAE-II
[arxiv:1905.12506 §2]. The study trains 360 encoders [arxiv:1905.12506 §4.1].
Each one feeds 10 sampled relation-network configurations, which gives 3600
reasoners [arxiv:1905.12506 §4.2.2]. The paper then correlates each encoder's
disentanglement scores with reasoning accuracy over training
[arxiv:1905.12506 §4.2.2]. Card:
[Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Few training samples, and you want an encoder that helps the reasoner learn fast | Disentangled encoder embeddings | In the few-sample regime, the FactorVAE and beta-VAE scores correlate strongly with accuracy [arxiv:1905.12506 §4.2.2] |
| Long training, around 100K steps | Check reconstruction error as well as disentanglement scores | By 100K steps, the paper finds no strong correlation with the disentanglement scores, and reconstruction error correlates more [arxiv:1905.12506 §4.2.2] |
| No pretrained encoder, and the CNN must learn from the puzzles | Learned from scratch, with many seeds | Single runs can stall at uniform predictions, so averages over many runs are the meaningful measure [arxiv:1905.12506 §4.2.1] |
| Ground-truth factors are known and you want a reference point | Ground-truth factor values | The paper uses this input as a comparison for the learned encoders [arxiv:1905.12506 §4.2.1] |
| Limited compute | Fewer encoders and seeds than the study | The full study took about 2.73 GPU-years on P100 GPUs [arxiv:1905.12506 §1]. The paper does not say how results change with fewer encoders [arxiv:1905.12506 §1] |
| Real, unlabelled images, where disentanglement cannot be measured | Disentangled encoder embeddings, with the usefulness question left open | The metrics need ground-truth factors, and the paper assumes a disentangled representation can be obtained, asking only whether it helps once obtained [arxiv:1905.12506 §2] |

## Implementation sketch

The shared core is the pair sum in eq. 2. The paper leaves some details open.
The pseudocode marks them.

```pseudocode
# Score one candidate answer against the context  [arxiv:1905.12506 eq. 2]
# context: 8 panels; answer: 1 panel; output: one scalar score
def relation_score(context, answer):
    # Embed all 9 panels with one shared encoder  [arxiv:1905.12506 §3]
    nodes = embed(concatenate(context, answer))   # (9, d)
    # Apply g to every ordered pair of nodes and sum the results
    # [arxiv:1905.12506 eq. 2]
    # Whether a node is paired with itself is not stated in the paper
    relations = sum(g(a, b) for a in nodes for b in nodes)
    # Turn the summed relations into one score  [arxiv:1905.12506 eq. 2]
    return f(relations)
```

The study does not fix one architecture. It samples the edge MLP g with 256 or
512 units and 2 to 4 hidden layers, and the graph MLP f with 128 or 256 units
and 1 or 2 hidden layers [arxiv:1905.12506 app. A] [arxiv:1905.12506 §4.2.2].
Training uses batches of 32 fresh puzzles, Adam, and 100K steps
[arxiv:1905.12506 §4.2] [arxiv:1905.12506 app. A]. The learning rate is
sampled from 1e-2, 1e-3 or 1e-4 [arxiv:1905.12506 app. A]. The card's
Paper vs code section lists further differences between eq. 2 and the released
code, which an implementer should read before copying the equation.

## Contradictions

Only one card covers this concept, so no two cited papers disagree. Within the
paper, the central claim concerns sample efficiency in the few-sample regime
[arxiv:1905.12506 §4.2.2] [arxiv:1905.12506 §5]. At 100K steps, the paper
reports no strong correlation with disentanglement scores, and reconstruction
error correlates more [arxiv:1905.12506 §4.2.2]. Read the claim as limited to
the early phase of training. The paper also finds that MIG and SAP correlate
weakly in the few-sample regime [arxiv:1905.12506 §4.2.2].

## Papers

- [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506): builds two Raven-style puzzle sets and trains a relation-network reasoner on 360 disentangled encoders, to test whether disentanglement helps reasoning.