Disentangled representation

Overview

A disentangled representation gives each separate quality of an input, such as shape, size, colour or position, its own part of the latent code 1905.12506 §1. The unsupervised variants train an encoder without labels, using a penalty added to the usual reconstruction objective 1905.12506 eq. 1. The penalty pushes the latent dimensions to be statistically independent in some sense, and each family measures this in a different way 1905.12506 §2.

The open question is whether the result helps later tasks. Disentanglement is often said to help, but earlier evidence came from simple tasks such as single-factor classification 1905.12506 §1 §2. One study on this page tests the claim on abstract visual reasoning, with puzzles built from two toy datasets 1905.12506 §3.

Three other sources use the idea in different settings. A survey frames disentanglement as a route to interpretable models, with concept-aligned neurons in the supervised case and compositional models in the unsupervised case 2103.11251 §5 §6. MISA splits each modality of a video into a shared code and a private code, and uses that split for sentiment and humour prediction 2005.03545 §3.3.

The third setting is face ageing. A conditional adversarial autoencoder takes a face code from an encoder and an age label, and gives both to the generator 1702.08423 §4.1. Holding the code fixed while changing only the age label is the paper's way of ageing a face 1702.08423 §3 fig. 2. Its evidence is a user study on identity, not a measure of how well the code and the label are separated 1702.08423 §5.3.

The fourth setting is image animation of articulated objects, such as bodies, with no labels. The model learns soft regions of the object, then splits the source's shape from the driving video's pose with a second pair of encoders 2104.11280 §3.6. Its evidence is reconstruction error and a user preference study, not a measure of how well the two codes are separated 2104.11280 tab. 2 tab. 3.

Variants

The four VAE families below share one objective: a reconstruction term plus two weighted regularisers, one per sample and one on the aggregate posterior 1905.12506 eq. 1. The card's sources do not state each family's regulariser in full, so the entries below give only what the paper states. Hyperparameter sweeps are in 1905.12506 app. A.

beta-VAE

A VAE with a fixed beta. Swept over beta in {1, 2, 4, 6, 8, 16} 1905.12506 app. A. Card: Are Disentangled Representations Helpful for Abstract Visual Reasoning?.

beta-VAE with controlled capacity increase

Same family, with a capacity target that grows during training. Swept over c_max in {5, 10, 25, 50, 75, 100}, with an iteration threshold of 100K and gamma of 1000 1905.12506 app. A. Card: Are Disentangled Representations Helpful for Abstract Visual Reasoning?.

FactorVAE

Penalises total correlation through a discriminator 1905.12506 §2. Swept over gamma in {10, 20, 30, 40, 50, 100}; the discriminator uses its own Adam settings 1905.12506 app. A. In the few-sample regime its score tracks reasoning accuracy closely 1905.12506 §4.2.2. Card: Are Disentangled Representations Helpful for Abstract Visual Reasoning?.

beta-TCVAE

Penalises total correlation with a Monte Carlo estimate instead of a discriminator 1905.12506 §2. Swept over beta in {1, 2, 4, 6, 8, 10} 1905.12506 app. A. Card: Are Disentangled Representations Helpful for Abstract Visual Reasoning?.

DIP-VAE-I and DIP-VAE-II

Penalise the moment mismatch of the aggregate posterior 1905.12506 §2. DIP-VAE-I is swept over lambda_od in {1, 2, 5, 10, 20, 50}; DIP-VAE-II over beta in the same set. The paper states lambda_d = 10 for both 1905.12506 app. A. See Caveats for a code difference. Card: Are Disentangled Representations Helpful for Abstract Visual Reasoning?.

Concept-aligned neurons (supervised)

The survey's supervised case: each concept is aligned with its own neuron, which the survey writes as a constraint 2103.11251 eq. 5 §5. Because the case is supervised, the concepts must be given. The survey lists four open problems: making a whole layer interpretable, disentangling all layers at once, choosing which concepts to learn, and making the mapping from concepts to output interpretable 2103.11251 §5. The card contains no results for this route. Card: Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges.

Compositional models (unsupervised)

Generative models and compositional architectures, such as capsules and slot attention, learn parts of an input without given concepts 2103.11251 §6. The survey lists quantitative evaluation of this route as open (6.1), along with transfer to other domains (6.2) 2103.11251 §6. Card: Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges.

Shared and private subspaces (multimodal)

MISA maps each modality to a shared code and a private code. The shared codes are pulled together with a CMD similarity loss, the private codes are kept apart from the shared ones with a difference loss, and a reconstruction loss keeps each modality's information 2005.03545 §3.5. The split serves prediction, and the paper does not measure whether factors are recovered. Card: MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis.

Identity code with a separate age input (face ageing)

The encoder maps a face to a short code, drawn as 50 values in the figure 1702.08423 §4.1 fig. 3. The generator receives that code with a one-hot age label appended 1702.08423 §4.1 §5.1. A discriminator pushes the codes toward a uniform prior, and a second one judges images conditioned on age 1702.08423 §4.3 §4.4. Training needs no paired images of one person at different ages 1702.08423 §1. Card: Age Progression/Regression by Conditional Adversarial Autoencoder.

View-invariant identity code (single image, 3D)

The paper asks the encoder for a code that identifies the object but not the viewpoint 1612.00814 §3.1. It gets there with a rotation curriculum that pre-trains the encoder, and then fixes the encoder while the decoder is trained 1612.00814 §3.3 §4. The split comes from a training schedule, not from a penalty on the latent. The paper does not call this disentanglement. Card: Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision.

Shape and pose codes (unsupervised, animation)

Two fully connected encoders map the source's region parameters (shape) and the driving frame's region parameters (pose) to 64-value codes. A decoder combines them into modified driving parameters, so that the shape comes from the source 2104.11280 §3.6 app. C. During training the driving parameters are scaled by random factors per axis, which pushes shape to come from the source branch; no scaling is applied at test time 2104.11280 §3.6. Card: Motion Representations for Articulated Animation.

Measuring disentanglement

The paper scores each encoder with five disentanglement metrics (BetaVAE, FactorVAE, MIG, DCI and SAP) and reconstruction error 1905.12506 §4.2.2. All five metrics need the ground-truth factors of variation, so they cannot be computed on real data that has no factor labels 1905.12506 §2. The survey lists quantitative evaluation of unsupervised disentanglement as an open problem 2103.11251 §6. MISA reports no disentanglement metric 2005.03545 tab. 1–4. The face-ageing paper reports no disentanglement metric either; its identity evidence is a user study 1702.08423 §5.3. The animation paper also has no factor-level metric. Its tables give L1, AKD, MKR and AED for reconstruction, and user preferences for the shape-pose split 2104.11280 tab. 2 tab. 3.

When to use which

The VAE study measures usefulness after a representation exists. It does not give a procedure for finding one on data without factor labels 1905.12506 §2.

Reported findings

Author-reported throughout.

• Few samples (up to 20K steps): FactorVAE and beta-VAE scores correlate strongly with reasoning accuracy; MIG and SAP correlate weakly 1905.12506 §4.2.2.
• 100K steps: no strong correlation with the disentanglement scores, and reconstruction error correlates more 1905.12506 §4.2.2.
• Models split by FactorVAE score into quartiles: the more disentangled models do better at every phase of training, shown as a figure 1905.12506 §4.2.2.
• The reasoning models reach about 80% average accuracy on dSprites puzzles and about 90% on 3dshapes puzzles across models in the initial study. These are averages over many models, not the score of one method 1905.12506 §4.2.1.
• The central claim: more disentangled representations are more sample-efficient in this setting 1905.12506 §4.2.2 §5.
• MISA ablation: removing the difference loss raises MOSI MAE from 0.783 to 0.824 and lowers MOSI correlation from 0.761 to 0.749 2005.03545 tab. 4.
• The survey runs no experiments. Its statements on disentanglement are positions and open problems 2103.11251 §5 §6.
• Face ageing: 48.38% of votes judge the output and the ground truth the same person, 29.58% not the same, and 22.04% not sure, for 856 pairs with an age gap over 20 years 1702.08423 §5.3. This is an identity judgement by volunteers, not a measure of disentanglement.
• Animation: shape-pose animation is preferred over standard animation in 53.6% of preferences on TaiChiHD (256) and 65.6% on TED-talks, each study with 2,500 preferences 2104.11280 tab. 3. This is a user preference, not a disentanglement score.

Implementation sketch

Common core of the unsupervised encoders: one encoder, one decoder, and a penalty chosen per family. Shared settings are in 1905.12506 app. A.

# Train an unsupervised encoder with a penalty  [arxiv:1905.12506 eq. 1]
# Batch 64, 10 latents, 300K steps  [arxiv:1905.12506 app. A]

def train_encoder(encoder, decoder, penalty, data, optimizer):
    for step in range(300000):
        # Draw a batch of input panels  [arxiv:1905.12506 app. A]
        x = sample(data, batch_size=64)
        # Parameters of q(z|x) for each input  [arxiv:1905.12506 §2]
        mu, sigma = encoder(x)
        z = sample_gaussian(mu, sigma)
        # Reconstruction term, Bernoulli decoder  [arxiv:1905.12506 app. A]
        recon = bernoulli_nll(decoder(z), x)
        # Family-specific penalty on the posterior  [arxiv:1905.12506 §2]
        reg = penalty(encoder, z, mu, sigma)
        optimizer.step(recon + reg)
    return encoder

Where the VAE variants differ is only in penalty, and in the sweep value that sets its weight. The representation that goes downstream is the encoder's posterior output. The paper does not say whether that is the mean or a sample 1905.12506 §2.

The shared and private variant keeps an encoder per modality but replaces the penalty with a set of terms over shared and private codes: a distribution-matching loss on the shared codes, a difference loss between shared and private codes, and a reconstruction loss 2005.03545 §3.5.

The animation variant adds a second stage. After the region network and the generator are trained, the two shape and pose encoders and their decoder are trained on the region parameters, with the driving pose scaled at random 2104.11280 §3.6 app. C.

Caveats

• Synthetic data only, for the VAE study. The puzzles use only the AND relation and factors from two toy datasets 1905.12506 §3. Generalisation to other reasoning tasks is not tested 1905.12506 §5.
• Averages hide spread. Models trained from scratch vary widely and can stall at uniform predictions 1905.12506 §4.2.1. Use averages over many runs.
• Results are in figures. The text gives no correlation coefficients, so numeric comparisons must be read from plots 1905.12506 §4.2.2.
• Code differs from the paper for DIP-VAE-II. The released code uses lambda_d equal to lambda_od for DIP-VAE-II, while the paper states 10. The code also sweeps lambda_od for both DIP variants. Details are in the card's Paper vs code section 1905.12506 app. A.
• Metric choice matters in the few-sample regime. MIG and SAP correlate weakly, which the paper attributes to struggling to separate partially disentangled models 1905.12506 §4.2.2.
• MISA does not test disentanglement. Its tables report sentiment and humour scores only 2005.03545 tab. 1–4. Good task results from a shared and private split do not show that factors were recovered.
• The survey's claims are untested here. It gives no experiments of its own, so its statements on disentanglement are not evidence of method performance 2103.11251 §5 §6.
• Animation split is judged by users. The shape-pose gains come from user preferences, with no factor-level metric 2104.11280 tab. 3. The TED-talks source videos are, per the authors, under CC BY-NC-ND 4.0, which does not allow commercial use or derivatives 2104.11280 app. E. The card does not supply the data; check those terms before reuse.
• Identity code is not a disentanglement result. The PTN card reports voxel IoU only 1612.00814 tab. 1–4. Read its identity/viewpoint split as a design example, not as evidence that factors were separated.
• Age input is not a measured separation. The face-ageing paper gives the age label to the generator, but its results have no face-recognition or image-quality metric and no measure of whether outputs show the requested age 1702.08423 §5.3. The first user study has 63 volunteers and reports 3,208 votes, which is more than its stated design allows (2,835 at 45 questions each); the paper does not explain this 1702.08423 §5.3.

Contradictions

No direct disagreement. The sources test different claims: the VAE study tests sample efficiency on synthetic reasoning puzzles 1905.12506 §4.2.2, MISA tests sentiment and humour prediction with a shared and private split 2005.03545 §5.1, the survey gives open problems without results 2103.11251 §6, PTN reports single-view reconstruction IoU 1612.00814 §4.1, the face-ageing paper reports a user study of identity and age progression 1702.08423 §5.3, and the animation paper reports user preference for shape-pose animation 2104.11280 tab. 3.

Papers

• Are Disentangled Representations Helpful for Abstract Visual Reasoning?: tests whether unsupervised disentangled encoders make a relation-network reasoner more sample-efficient, using RPM-style puzzles from dSprites and 3dshapes.
• Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges: survey of interpretable models; lists open problems on concept-aligned neurons and unsupervised compositional disentanglement.
• MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis: splits each modality into shared and private codes for multimodal sentiment and humour prediction.
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision: single-view 3D reconstruction; its encoder is pre-trained for an identity code that excludes viewpoint. The card does not study disentanglement.
• Age Progression/Regression by Conditional Adversarial Autoencoder: face ageing with a face code and a separate age label; trained without paired ages and judged by a user study.
• Motion Representations for Articulated Animation: unsupervised image animation that splits source shape from driving pose with two encoders and a decoder; judged by user preference over standard animation.