--- kind: concept slug: disentangled-representation title: Disentangled representation summary: >- A disentangled representation gives each independent factor of variation its own part of the latent code. Unsupervised VAEs learn one with a penalty, but scoring needs known factors. One VAE study finds FactorVAE and beta-VAE scores track reasoning accuracy with few samples. MISA and an animation model split shared or source-shape codes from private or pose codes. tags: [representation-learning, disentanglement, unsupervised-learning, abstract-reasoning, interpretability, multimodal, image-animation] papers: ["1905.12506", "2103.11251", "2005.03545", "1612.00814", "1702.08423", "2104.11280"] related: [variational-autoencoder, explainable-ai, distributional-alignment, latent-space-editing, unsupervised-keypoint-detection, equivariance-constraint] status: draft updated_at: 2026-11-01 --- ## Overview A disentangled representation gives each separate quality of an input, such as shape, size, colour or position, its own part of the latent code [arxiv:1905.12506 §1]. The unsupervised variants train an encoder without labels, using a penalty added to the usual reconstruction objective [arxiv:1905.12506 eq. 1]. The penalty pushes the latent dimensions to be statistically independent in some sense, and each family measures this in a different way [arxiv:1905.12506 §2]. The open question is whether the result helps later tasks. Disentanglement is often said to help, but earlier evidence came from simple tasks such as single-factor classification [arxiv:1905.12506 §1] [§2]. One study on this page tests the claim on abstract visual reasoning, with puzzles built from two toy datasets [arxiv:1905.12506 §3]. Three other sources use the idea in different settings. A survey frames disentanglement as a route to interpretable models, with concept-aligned neurons in the supervised case and compositional models in the unsupervised case [arxiv:2103.11251 §5] [§6]. MISA splits each modality of a video into a shared code and a private code, and uses that split for sentiment and humour prediction [arxiv:2005.03545 §3.3]. The third setting is face ageing. A conditional adversarial autoencoder takes a face code from an encoder and an age label, and gives both to the generator [arxiv:1702.08423 §4.1]. Holding the code fixed while changing only the age label is the paper's way of ageing a face [arxiv:1702.08423 §3] [fig. 2]. Its evidence is a user study on identity, not a measure of how well the code and the label are separated [arxiv:1702.08423 §5.3]. The fourth setting is image animation of articulated objects, such as bodies, with no labels. The model learns soft regions of the object, then splits the source's shape from the driving video's pose with a second pair of encoders [arxiv:2104.11280 §3.6]. Its evidence is reconstruction error and a user preference study, not a measure of how well the two codes are separated [arxiv:2104.11280 tab. 2] [tab. 3]. ## Variants The four VAE families below share one objective: a reconstruction term plus two weighted regularisers, one per sample and one on the aggregate posterior [arxiv:1905.12506 eq. 1]. The card's sources do not state each family's regulariser in full, so the entries below give only what the paper states. Hyperparameter sweeps are in [arxiv:1905.12506 app. A]. ### beta-VAE A VAE with a fixed beta. Swept over beta in {1, 2, 4, 6, 8, 16} [arxiv:1905.12506 app. A]. Card: [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506). ### beta-VAE with controlled capacity increase Same family, with a capacity target that grows during training. Swept over c_max in {5, 10, 25, 50, 75, 100}, with an iteration threshold of 100K and gamma of 1000 [arxiv:1905.12506 app. A]. Card: [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506). ### FactorVAE Penalises total correlation through a discriminator [arxiv:1905.12506 §2]. Swept over gamma in {10, 20, 30, 40, 50, 100}; the discriminator uses its own Adam settings [arxiv:1905.12506 app. A]. In the few-sample regime its score tracks reasoning accuracy closely [arxiv:1905.12506 §4.2.2]. Card: [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506). ### beta-TCVAE Penalises total correlation with a Monte Carlo estimate instead of a discriminator [arxiv:1905.12506 §2]. Swept over beta in {1, 2, 4, 6, 8, 10} [arxiv:1905.12506 app. A]. Card: [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506). ### DIP-VAE-I and DIP-VAE-II Penalise the moment mismatch of the aggregate posterior [arxiv:1905.12506 §2]. DIP-VAE-I is swept over lambda_od in {1, 2, 5, 10, 20, 50}; DIP-VAE-II over beta in the same set. The paper states lambda_d = 10 for both [arxiv:1905.12506 app. A]. See Caveats for a code difference. Card: [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506). ### Concept-aligned neurons (supervised) The survey's supervised case: each concept is aligned with its own neuron, which the survey writes as a constraint [arxiv:2103.11251 eq. 5] [§5]. Because the case is supervised, the concepts must be given. The survey lists four open problems: making a whole layer interpretable, disentangling all layers at once, choosing which concepts to learn, and making the mapping from concepts to output interpretable [arxiv:2103.11251 §5]. The card contains no results for this route. Card: [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](/papers/2103.11251). ### Compositional models (unsupervised) Generative models and compositional architectures, such as capsules and slot attention, learn parts of an input without given concepts [arxiv:2103.11251 §6]. The survey lists quantitative evaluation of this route as open (6.1), along with transfer to other domains (6.2) [arxiv:2103.11251 §6]. Card: [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](/papers/2103.11251). ### Shared and private subspaces (multimodal) MISA maps each modality to a shared code and a private code. The shared codes are pulled together with a CMD similarity loss, the private codes are kept apart from the shared ones with a difference loss, and a reconstruction loss keeps each modality's information [arxiv:2005.03545 §3.5]. The split serves prediction, and the paper does not measure whether factors are recovered. Card: [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](/papers/2005.03545). ### Identity code with a separate age input (face ageing) The encoder maps a face to a short code, drawn as 50 values in the figure [arxiv:1702.08423 §4.1] [fig. 3]. The generator receives that code with a one-hot age label appended [arxiv:1702.08423 §4.1] [§5.1]. A discriminator pushes the codes toward a uniform prior, and a second one judges images conditioned on age [arxiv:1702.08423 §4.3] [§4.4]. Training needs no paired images of one person at different ages [arxiv:1702.08423 §1]. Card: [Age Progression/Regression by Conditional Adversarial Autoencoder](/papers/1702.08423). ### View-invariant identity code (single image, 3D) The paper asks the encoder for a code that identifies the object but not the viewpoint [arxiv:1612.00814 §3.1]. It gets there with a rotation curriculum that pre-trains the encoder, and then fixes the encoder while the decoder is trained [arxiv:1612.00814 §3.3] [§4]. The split comes from a training schedule, not from a penalty on the latent. The paper does not call this disentanglement. Card: [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](/papers/1612.00814). ### Shape and pose codes (unsupervised, animation) Two fully connected encoders map the source's region parameters (shape) and the driving frame's region parameters (pose) to 64-value codes. A decoder combines them into modified driving parameters, so that the shape comes from the source [arxiv:2104.11280 §3.6] [app. C]. During training the driving parameters are scaled by random factors per axis, which pushes shape to come from the source branch; no scaling is applied at test time [arxiv:2104.11280 §3.6]. Card: [Motion Representations for Articulated Animation](/papers/2104.11280). ## Measuring disentanglement The paper scores each encoder with five disentanglement metrics (BetaVAE, FactorVAE, MIG, DCI and SAP) and reconstruction error [arxiv:1905.12506 §4.2.2]. All five metrics need the ground-truth factors of variation, so they cannot be computed on real data that has no factor labels [arxiv:1905.12506 §2]. The survey lists quantitative evaluation of unsupervised disentanglement as an open problem [arxiv:2103.11251 §6]. MISA reports no disentanglement metric [arxiv:2005.03545 tab. 1–4]. The face-ageing paper reports no disentanglement metric either; its identity evidence is a user study [arxiv:1702.08423 §5.3]. The animation paper also has no factor-level metric. Its tables give L1, AKD, MKR and AED for reconstruction, and user preferences for the shape-pose split [arxiv:2104.11280 tab. 2] [tab. 3]. ## When to use which The VAE study measures usefulness after a representation exists. It does not give a procedure for finding one on data without factor labels [arxiv:1905.12506 §2]. | Situation | Use | Why | |---|---|---| | Ground-truth factors available, few training samples | Select by FactorVAE or beta-VAE scores | These correlate strongly with reasoning accuracy up to 20K steps; values are only in a figure [arxiv:1905.12506 §4.2.2] | | Long training (around 100K steps) | Do not rely on disentanglement scores alone | No strong correlation with accuracy; reconstruction error correlates more [arxiv:1905.12506 §4.2.2] | | No ground-truth factors | Not addressed by the VAE study | The metrics cannot be computed [arxiv:1905.12506 §2]; the survey lists evaluation as open [arxiv:2103.11251 §6] | | Concept labels exist and the units must map to named concepts | Concept-aligned neurons | The survey presents this as the supervised route, with whole-layer versions still open [arxiv:2103.11251 §5] | | Several input modalities, each with a shared and a private part | Shared and private subspaces | Without the subspaces (separate encoders per modality), MOSI MAE rises from 0.783 to 0.810 [arxiv:2005.03545 tab. 4] | | Single-view input, the code should not depend on the viewpoint, 2D silhouettes for training | Rotation-curriculum pre-training of the encoder | The paper uses it for this purpose [arxiv:1612.00814 §3.3] [§4]; no run without the curriculum appears in the card's tables [arxiv:1612.00814 tab. 1–4] | | One attribute to vary (such as age bins) while identity stays, no paired images across the attribute | Identity code with the attribute as a separate generator input | The paper trains without paired ages [arxiv:1702.08423 §1] [§5.2]; the identity check is a user study, with 48.38% of votes saying output and ground truth are the same person [arxiv:1702.08423 §5.3] | | Articulated object animated from a driving video, no labels, output must keep the source's shape | Shape and pose codes with a pose-to-shape decoder, trained after the main model | The paper's shape-pose animation is preferred over standard animation in 53.6% of votes on TaiChiHD (256) and 65.6% on TED-talks [arxiv:2104.11280 tab. 3]; the standard animation has no shape split [arxiv:2104.11280 §3.6] | | Limited compute | Expect a smaller study to differ | The full study took about 2.73 GPU-years on P100; the paper does not say how results change with fewer encoders or seeds [arxiv:1905.12506 §1] | | Reasoning task other than the AND-relation puzzles | Test before relying on the result | Only one relation type was tested [arxiv:1905.12506 §3] [§5] | ## Reported findings Author-reported throughout. - Few samples (up to 20K steps): FactorVAE and beta-VAE scores correlate strongly with reasoning accuracy; MIG and SAP correlate weakly [arxiv:1905.12506 §4.2.2]. - 100K steps: no strong correlation with the disentanglement scores, and reconstruction error correlates more [arxiv:1905.12506 §4.2.2]. - Models split by FactorVAE score into quartiles: the more disentangled models do better at every phase of training, shown as a figure [arxiv:1905.12506 §4.2.2]. - The reasoning models reach about 80% average accuracy on dSprites puzzles and about 90% on 3dshapes puzzles across models in the initial study. These are averages over many models, not the score of one method [arxiv:1905.12506 §4.2.1]. - The central claim: more disentangled representations are more sample-efficient in this setting [arxiv:1905.12506 §4.2.2] [§5]. - MISA ablation: removing the difference loss raises MOSI MAE from 0.783 to 0.824 and lowers MOSI correlation from 0.761 to 0.749 [arxiv:2005.03545 tab. 4]. - The survey runs no experiments. Its statements on disentanglement are positions and open problems [arxiv:2103.11251 §5] [§6]. - Face ageing: 48.38% of votes judge the output and the ground truth the same person, 29.58% not the same, and 22.04% not sure, for 856 pairs with an age gap over 20 years [arxiv:1702.08423 §5.3]. This is an identity judgement by volunteers, not a measure of disentanglement. - Animation: shape-pose animation is preferred over standard animation in 53.6% of preferences on TaiChiHD (256) and 65.6% on TED-talks, each study with 2,500 preferences [arxiv:2104.11280 tab. 3]. This is a user preference, not a disentanglement score. ## Implementation sketch Common core of the unsupervised encoders: one encoder, one decoder, and a penalty chosen per family. Shared settings are in [arxiv:1905.12506 app. A]. ```pseudocode # Train an unsupervised encoder with a penalty [arxiv:1905.12506 eq. 1] # Batch 64, 10 latents, 300K steps [arxiv:1905.12506 app. A] def train_encoder(encoder, decoder, penalty, data, optimizer): for step in range(300000): # Draw a batch of input panels [arxiv:1905.12506 app. A] x = sample(data, batch_size=64) # Parameters of q(z|x) for each input [arxiv:1905.12506 §2] mu, sigma = encoder(x) z = sample_gaussian(mu, sigma) # Reconstruction term, Bernoulli decoder [arxiv:1905.12506 app. A] recon = bernoulli_nll(decoder(z), x) # Family-specific penalty on the posterior [arxiv:1905.12506 §2] reg = penalty(encoder, z, mu, sigma) optimizer.step(recon + reg) return encoder ``` Where the VAE variants differ is only in `penalty`, and in the sweep value that sets its weight. The representation that goes downstream is the encoder's posterior output. The paper does not say whether that is the mean or a sample [arxiv:1905.12506 §2]. The shared and private variant keeps an encoder per modality but replaces the penalty with a set of terms over shared and private codes: a distribution-matching loss on the shared codes, a difference loss between shared and private codes, and a reconstruction loss [arxiv:2005.03545 §3.5]. The animation variant adds a second stage. After the region network and the generator are trained, the two shape and pose encoders and their decoder are trained on the region parameters, with the driving pose scaled at random [arxiv:2104.11280 §3.6] [app. C]. ## Caveats - **Synthetic data only, for the VAE study.** The puzzles use only the AND relation and factors from two toy datasets [arxiv:1905.12506 §3]. Generalisation to other reasoning tasks is not tested [arxiv:1905.12506 §5]. - **Averages hide spread.** Models trained from scratch vary widely and can stall at uniform predictions [arxiv:1905.12506 §4.2.1]. Use averages over many runs. - **Results are in figures.** The text gives no correlation coefficients, so numeric comparisons must be read from plots [arxiv:1905.12506 §4.2.2]. - **Code differs from the paper for DIP-VAE-II.** The released code uses lambda_d equal to lambda_od for DIP-VAE-II, while the paper states 10. The code also sweeps lambda_od for both DIP variants. Details are in the card's Paper vs code section [arxiv:1905.12506 app. A]. - **Metric choice matters in the few-sample regime.** MIG and SAP correlate weakly, which the paper attributes to struggling to separate partially disentangled models [arxiv:1905.12506 §4.2.2]. - **MISA does not test disentanglement.** Its tables report sentiment and humour scores only [arxiv:2005.03545 tab. 1–4]. Good task results from a shared and private split do not show that factors were recovered. - **The survey's claims are untested here.** It gives no experiments of its own, so its statements on disentanglement are not evidence of method performance [arxiv:2103.11251 §5] [§6]. - **Animation split is judged by users.** The shape-pose gains come from user preferences, with no factor-level metric [arxiv:2104.11280 tab. 3]. The TED-talks source videos are, per the authors, under CC BY-NC-ND 4.0, which does not allow commercial use or derivatives [arxiv:2104.11280 app. E]. The card does not supply the data; check those terms before reuse. - **Identity code is not a disentanglement result.** The PTN card reports voxel IoU only [arxiv:1612.00814 tab. 1–4]. Read its identity/viewpoint split as a design example, not as evidence that factors were separated. - **Age input is not a measured separation.** The face-ageing paper gives the age label to the generator, but its results have no face-recognition or image-quality metric and no measure of whether outputs show the requested age [arxiv:1702.08423 §5.3]. The first user study has 63 volunteers and reports 3,208 votes, which is more than its stated design allows (2,835 at 45 questions each); the paper does not explain this [arxiv:1702.08423 §5.3]. ## Contradictions No direct disagreement. The sources test different claims: the VAE study tests sample efficiency on synthetic reasoning puzzles [arxiv:1905.12506 §4.2.2], MISA tests sentiment and humour prediction with a shared and private split [arxiv:2005.03545 §5.1], the survey gives open problems without results [arxiv:2103.11251 §6], PTN reports single-view reconstruction IoU [arxiv:1612.00814 §4.1], the face-ageing paper reports a user study of identity and age progression [arxiv:1702.08423 §5.3], and the animation paper reports user preference for shape-pose animation [arxiv:2104.11280 tab. 3]. ## Papers - [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506): tests whether unsupervised disentangled encoders make a relation-network reasoner more sample-efficient, using RPM-style puzzles from dSprites and 3dshapes. - [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](/papers/2103.11251): survey of interpretable models; lists open problems on concept-aligned neurons and unsupervised compositional disentanglement. - [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](/papers/2005.03545): splits each modality into shared and private codes for multimodal sentiment and humour prediction. - [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](/papers/1612.00814): single-view 3D reconstruction; its encoder is pre-trained for an identity code that excludes viewpoint. The card does not study disentanglement. - [Age Progression/Regression by Conditional Adversarial Autoencoder](/papers/1702.08423): face ageing with a face code and a separate age label; trained without paired ages and judged by a user study. - [Motion Representations for Articulated Animation](/papers/2104.11280): unsupervised image animation that splits source shape from driving pose with two encoders and a decoder; judged by user preference over standard animation.