--- kind: concept slug: autoencoder title: Autoencoder summary: >- An autoencoder compresses an input into a code and rebuilds the input from it, so the code keeps what matters. Variants change the bottleneck (sparse, variational, discrete), the input (denoising) or the use of the code: an anomaly score, a training-only shape prior, a survival-model input, a generator latent or interpretable features. Choose by what the code is for and how many labels you have. tags: [representation-learning, autoencoder, latent-representation, anomaly-detection, interpretability, generative-models] papers: ["1506.00327", "1601.02376", "1702.08423", "1705.08302", "1803.10122", "1809.01999", "1810.03382", "1812.01718", "1905.12506", "2102.12092", "2103.04922", "2104.06979", "2104.10157", "2107.12626", "2212.04048", "2302.02913", "2309.08600", "2312.11243", "2408.06083", "2412.00949", "2502.12048", "2512.02004", "2512.10092", "2512.12993", "2605.19376"] related: [variational-autoencoder, denoising-autoencoder, sparse-autoencoder, vector-quantization, latent-diffusion, anomaly-detection, generative-adversarial-network, latent-space-editing, disentangled-representation, text-embedding, self-supervised-learning, mechanistic-interpretability, world-model, time-series-imputation, survival-analysis, semantic-segmentation, shape-prior-regularization, diffusion-models, robotic-grasp-generation] status: draft updated_at: 2026-11-01 --- ## Overview An autoencoder has two parts: an encoder that maps an input to a shorter code, and a decoder that maps the code back to the input. Training minimises a reconstruction error [arxiv:2107.12626 §3.3] [arxiv:1810.03382 eq. 3]. The bottleneck is the point: the code has to keep enough of the input for the decoder to rebuild it. All variants keep this encoder-code-decoder shape. They differ in three places: - **The input.** Corrupting it before encoding gives a denoising autoencoder, which must rebuild the clean input [arxiv:2104.06979 §3.1]. - **The code.** A sparsity penalty gives sparse features [arxiv:2309.08600 eq. 4]. A probabilistic code with a KL term gives a variational autoencoder, and a codebook of nearest entries gives a discrete one [arxiv:2102.12092 §2.1]. - **The use of the code.** Reconstruction is often only the training signal. The code may be scored, used as a penalty, read by a downstream model or used as the latent of a generator. ## Variants ### Plain and convolutional autoencoders A convolutional autoencoder with a deterministic code is the simplest form. In CAE-M, windows of multi-sensor data go through a convolutional encoder and a transposed-convolution decoder, and training uses only normal samples [arxiv:2107.12626 §3.1] [arxiv:2107.12626 §3.3.1]. The anomaly score combines the reconstruction error with the code, and a threshold of mean plus standard deviation on training losses sets the decision [arxiv:2107.12626 eq. 6] [arxiv:2107.12626 eq. 21]. On PAMAP2 the author-reported mean F1 is 0.9639. Setting both the reconstruction and MMD weights to zero drops it to 0.5563 [arxiv:2107.12626 tab. 6], so the reconstruction-based parts carry the score. The baselines are re-implementations, using PyOD and a third-party MSCRED repository, so the comparison depends on those choices [arxiv:2107.12626 §4.3]. An autoencoder can also serve only during training. In a cardiac segmentation model, a stacked convolutional autoencoder encodes one-hot label maps into a 64-value code. The squared distance between the codes of predicted and true labels is added to the loss as a shape penalty [arxiv:1705.08302 eq. 1] [arxiv:1705.08302 §2.2]. The autoencoder is not used at inference [arxiv:1705.08302 §2.4], so the extra cost is paid only in training. Author- reported endocardial Dice is 0.939 ± 0.017, against 0.923 for the 3D baseline [arxiv:1705.08302 tab. 1]. ### Denoising autoencoders The input is corrupted and the target is the clean input. See [Denoising autoencoder](/concepts/denoising-autoencoder). Three cards show how the corruption and the decoder's view change the use: - **Survival prediction from motion.** A mask is applied to the input and redrawn at each backward pass. The loss is a weighted sum of reconstruction and a negated Cox partial likelihood [arxiv:1810.03382 § Network Design and Training] [arxiv:1810.03382 eq. 7]. The author-reported optimism-corrected C-index is 0.73, against 0.59 for a Cox model on manual volumes. This is internal bootstrap validation in 302 patients from one centre [arxiv:1810.03382 § Predictive performance]. - **Sentence embeddings.** Each sentence loses 60% of its tokens. A Transformer encoder outputs one vector, and the decoder reads only that vector [arxiv:2104.06979 §3.2] [arxiv:2104.06979 app. A]. The unsupervised average is 55.2, against 52.4 for the contrastive baseline [arxiv:2104.06979 tab. 2]. On general STS it is weaker: 66.0 against 73.9 on STS-B [arxiv:2104.06979 tab. 3]. - **Time-series imputation.** The autoencoder works on an image encoding of the series, not on the raw values. It has one sigmoid hidden layer of 500 units and is trained on inputs with 20% of points set to zero [arxiv:1506.00327 §4.1]. Interpolation error falls: the card's calculation from the table gives reductions of 12.18% to 48.02%, which the paper does not state [arxiv:1506.00327 §4.2]. Full-series error is worse for the image encoding in all five rows [arxiv:1506.00327 tab. 2]. A fourth card calls its pre-training step a denoising autoencoder, but the corruption is not applied. Negative units are sampled per field and the positive unit is kept [arxiv:1601.02376 §3.2]. See Contradictions. ### Variational autoencoders The code is a distribution, trained with reconstruction plus a KL term. See [Variational autoencoder](/concepts/variational-autoencoder). The cards use it in several ways: - **World models.** A convolutional VAE with a 32-value latent for car racing and 64 for Doom is trained without reward. A separate controller sees the reward and is evolved [arxiv:1803.10122 §3.1] [arxiv:1809.01999 app. A.2]. Car racing reaches 906 ± 21 with the latent and the recurrent state, against 632 ± 251 with the latent alone [arxiv:1809.01999 tab. 1]. The paper trains the VAE for one epoch, and the released code runs ten [arxiv:1803.10122 §A.1]. - **Two image domains.** Two convolutional VAEs, one per domain, each with a 64-value latent, are linked by a mixture density network that maps one latent to the other [arxiv:1812.01718 §3.2]. The paper reports no quantitative transfer metric for the link [arxiv:1812.01718 §3.2]. - **Disentanglement for reasoning.** beta-VAE, FactorVAE, beta-TCVAE and DIP-VAE encoders are compared by the accuracy of a downstream reasoner, not by reconstruction [arxiv:1905.12506 §2]. Disentanglement scores track accuracy in the few-sample regime up to 20K steps, and the link weakens by 100K steps [arxiv:1905.12506 §4.2.2]. - **Latent diffusion for motion.** A transformer VAE turns a motion clip into a single 256-value token, and diffusion runs in that latent. The paper reports an HumanML3D FID of 0.473, against 0.544 for MDM, and about 22 times less DDIM-50 inference time than MDM [arxiv:2212.04048 tab. 1] [arxiv:2212.04048 tab. 10]. - **Latent diffusion for grasps.** A conditional VAE produces grasps, and a DDPM prior runs in its frozen latent. Median simulated success is 0.792 with DDPM, 0.756 with DDIM and 0.660 with the VAE alone [arxiv:2312.11243 tab. 1]. - **Goal prior for a policy.** A conditional VAE maps an audio embedding into the goal space of a frozen policy, which is not retrained [arxiv:2412.00949 §3.4]. The paper reports the audio prompt beats the visual one on 4 of 6 tasks [arxiv:2412.00949 §4.1]. - **Terrain latent.** A convolutional VAE compresses a 20 × 40 height map into 16 values with an MSE plus β·KL loss, where β is not stated [arxiv:2512.12993 §II.B] [arxiv:2512.12993 eq. 4]. - **Frozen pretrained VAE.** A Stable Diffusion VAE encodes several exposures of one scene, and the latents are averaged before decoding. The VAE is not modified [arxiv:2408.06083 §3.3]. - **Evaluation and surveys.** A metrics study compares a conditional VAE with a conditional GAN on a two-dimensional design problem [arxiv:2302.02913 tab. 6], and the authors warn that these results may not carry over [arxiv:2302.02913 §4.5]. A deep generative survey gives the ELBO and reparameterised sampling [arxiv:2103.04922 eq. 11] [arxiv:2103.04922 eq. 13]. Its CIFAR-10 table is compiled from other papers and is not like-for-like [arxiv:2103.04922 tab. 1]. An EEG survey trains no model [arxiv:2502.12048 §3]. ### Discrete autoencoders The code is a choice from a fixed set, so the encoder output can be a token for an autoregressive model. See [Vector quantization](/concepts/vector-quantization). - **Gumbel-softmax dVAE.** A 256 × 256 image becomes a 32 × 32 grid of tokens from an 8192-entry dictionary [arxiv:2102.12092 §2.1]. The encoder is trained with a relaxed sample, and the KL weight is raised from 0 to 6.6 [arxiv:2102.12092 app. A.2]. The paper's text-to-image transformer models caption tokens and these image tokens as one stream [arxiv:2102.12092 §2.2]. - **VQ-VAE with EMA codebook.** A VQ-VAE with 3D convolutions and axial attention maps video to a grid of codes. The codebook is updated by an exponential moving average [arxiv:2104.10157 §2.1]. A GPT-style prior models the codes [arxiv:2104.10157 §3]. The author-reported BAIR FVD is 103.3 [arxiv:2104.10157 tab. 1]. ### Sparse autoencoders The code is overcomplete and penalised for activity. See [Sparse autoencoder](/concepts/sparse-autoencoder). The cards use this on model activations, not on data. - **Interpretability of a language model.** A one-hidden-layer model with a ReLU encoder and a tied decoder is trained on the residual stream. The hidden width is R times the input width [arxiv:2309.08600 §2]. The loss is reconstruction plus α times the L1 norm of the code [arxiv:2309.08600 eq. 4]. The paper reports higher autointerpretability scores than baselines, shown only in a plot [arxiv:2309.08600 fig. 2]. - **Supervised concept slots.** AlignSAE splits the code into free features and slots bound to labelled relations. A cross-entropy binding loss is added to the reconstruction and L1 terms [arxiv:2512.02004 app. C.2]. The card reports swap success of 0.847 at alpha = 2 [arxiv:2512.02004 §5.3]. - **Reuse of a public SAE.** A released SAE for a 70B model, at layer 50 with 65,536 latents, is used as a feature extractor for named concepts [arxiv:2512.10092 §4]. The authors report retrieval MAP of 0.361 against 0.412 for an LLM-based baseline on one prompt set [arxiv:2512.10092 §4.4]. ### Adversarial and conditional autoencoders Here the regulariser is adversarial rather than a KL term. - **Age progression.** The conditional adversarial autoencoder encodes a face and decodes it with an age label. A discriminator pulls the code towards a uniform prior [arxiv:1702.08423 §4.3], so the code is not regularised by a KL term. The age label is one-hot over ten bins [arxiv:1702.08423 §5.1]. Training needs no paired images of one person at two ages [arxiv:1702.08423 §1]. ### Noise in a recurrent state - **Generative recursive reasoning.** Each step of a small recursive reasoner adds Gaussian noise to its high-level state. The posterior sees the target, and the prior does not [arxiv:2605.19376 §2.1] [arxiv:2605.19376 §2.2]. The noise sits in the recurrent state, not in an encoder bottleneck, which is why it is listed here. Latency per puzzle is 184 ms, against 2,755 ms for the baseline at matched compute [arxiv:2605.19376 tab. 10]. ## When to use which The rows are chosen by constraint. The numbers are author-reported, and the cards test different tasks, so they are not a ranking. | Situation | Use | Why | |---|---|---| | Anomalies must be found from normal data only, with no labels | Convolutional autoencoder with a reconstruction-based score | Author-reported mean F1 0.9639 on PAMAP2, above the baselines in its table [arxiv:2107.12626 tab. 2]. Removing the reconstruction and MMD terms gives 0.5563 [arxiv:2107.12626 tab. 6] | | A shape or structure prior should cost nothing at inference | Autoencoder used only in the loss | The autoencoder is dropped after training [arxiv:1705.08302 §2.4]. Dice improves over the 3D baseline in one study [arxiv:1705.08302 tab. 1] | | Labelled outcomes exist and the input is high-dimensional | Denoising autoencoder with a supervised loss on the code | C-index 0.73 against 0.59 for the Cox baseline, in one centre [arxiv:1810.03382 § Predictive performance] | | Sentences are unlabelled and the target is domain similarity | Sequential denoising autoencoder | Best unsupervised average on four domain tasks [arxiv:2104.06979 tab. 2]. Weaker on general STS [arxiv:2104.06979 tab. 3] | | A series has gaps and the values must be filled | Denoising autoencoder on an image encoding of the series | Interpolation error drops [arxiv:1506.00327 §4.2]. Full-series error rises [arxiv:1506.00327 tab. 2] | | The code must generate new samples from a learned distribution | Variational autoencoder, with a diffusion prior in its latent when speed matters | Diffusion in the latent is about 22 times faster than MDM in the motion study [arxiv:2212.04048 tab. 10]. The grasp study shows the prior improves success over the VAE alone [arxiv:2312.11243 tab. 1] | | Discrete tokens must feed an autoregressive model | Gumbel-softmax dVAE or VQ-VAE with an EMA codebook | 8192 tokens on a 32 × 32 grid [arxiv:2102.12092 §2.1]. The codebook update and an FVD of 103.3 [arxiv:2104.10157 tab. 1] | | A downstream reasoner must learn from few examples and factors are labelled for scoring | Disentangled VAE, chosen by downstream accuracy | Scores track accuracy in the few-sample regime [arxiv:1905.12506 §4.2.2]. The scores need ground-truth factors [arxiv:1905.12506 §2] | | Interpretable features are needed from a language model you can run | Train a sparse autoencoder on one layer, or reuse a released one | Trained SAEs give higher autointerpretability scores in one plot [arxiv:2309.08600 fig. 2]. A released 70B SAE gives retrieval MAP 0.361 against 0.412 for a baseline [arxiv:2512.10092 §4.4] | | Pairs of inputs are not available and one attribute must change | Conditional adversarial autoencoder | Trained without paired ages [arxiv:1702.08423 §1]. The code is regularised by a discriminator, not a KL term [arxiv:1702.08423 §4.3] | | A small recursive reasoner must answer under a latency budget | Noise in the recurrent state, with voting over samples | 184 ms per puzzle against 2,755 ms at matched compute [arxiv:2605.19376 tab. 10] | ## Implementation sketch Common core of the variants. Each hook marks where a variant differs. ```pseudocode # Corrupt the input (denoising only; identity otherwise) [arxiv:2104.06979 §3.1] x_in = corrupt(x) # Encode to a code z [arxiv:2107.12626 §3.3.1] z = encoder(x_in) # Variational: sample z from a Gaussian, reparameterised # [arxiv:2103.04922 eq. 13] # Discrete: snap z to the nearest codebook entry [arxiv:2102.12092 §2.1] # Decode the code back to the clean input [arxiv:2104.06979 §3.2] x_hat = decoder(z) # Reconstruction against the clean input [arxiv:1810.03382 eq. 3] loss = recon_error(x_hat, x) # Variational: add the KL term [arxiv:2103.04922 eq. 12] # Sparse: add an L1 penalty on the code; check its normalisation # [arxiv:2309.08600 eq. 4] # Task-specific term on the code, e.g. a Cox loss or a shape distance # [arxiv:1810.03382 eq. 5] # Downstream use reads z; the decoder may be dropped at inference # [arxiv:1705.08302 §2.4] ``` Lines are kept at 80 characters or fewer. The decoder is dropped for the shape-prior variant and kept for generation. ## Contradictions - **What "denoising" means.** The label is used for input corruption in [arxiv:1506.00327 §4] and for masking in [arxiv:1810.03382 § Network Design and Training]. The paper calls its pre-training sampler a denoising autoencoder [arxiv:1601.02376 §3.2], but the card for that paper records that the released code sets the corruption level to zero, so no input is corrupted. Compare the paper's wording with the corruption step before you reuse it. - **Sparse penalty normalisation.** The paper writes the L1 term as a sum over the input dimension [arxiv:2309.08600 eq. 4]. The card records that the released code averages it instead. That changes the effective alpha by a factor of the input width, so the stated alpha values are not interchangeable. - **World-model training length.** The paper trains the VAE for one epoch and the MDN for 20 [arxiv:1803.10122 §A.1] [arxiv:1803.10122 §A.2]. The card records that the released code runs ten VAE epochs and 4,000 MDN steps [arxiv:1803.10122 §A.1] [arxiv:1803.10122 §A.2]. - **Denoising mask.** The paper's deletion ratio of 0.6 means 60% of tokens are affected [arxiv:2104.06979 app. A]. The card records that the released STS-B script keeps each word with probability 0.6, so it deletes about 40% of words [arxiv:2104.06979 app. A]. - **Latent size.** One conditional VAE is described with a 128-value latent in the paper, while the card records 64 or 256 in the released code [arxiv:2312.11243 §3.3]. ## Papers - [Imaging Time-Series to Improve Classification and Imputation](/papers/1506.00327): a denoising autoencoder on image encodings for imputation. - [Deep Learning over Multi-field Categorical Data](/papers/1601.02376): autoencoder-based pre-training for click prediction, with a sampler called denoising. - [Age Progression/Regression by Conditional Adversarial Autoencoder](/papers/1702.08423): a conditional adversarial autoencoder with a discriminator on the code. - [Anatomically Constrained Neural Networks](/papers/1705.08302): an autoencoder used only as a training-time shape prior. - [World Models](/papers/1803.10122): a VAE, an MDN-RNN and a controller trained by evolution. - [Recurrent World Models Facilitate Policy Evolution](/papers/1809.01999): the same three-part model, with the car and Doom settings. - [Deep learning cardiac motion analysis for human survival prediction](/papers/1810.03382): a denoising autoencoder with a Cox survival loss. - [Deep Learning for Classical Japanese Literature](/papers/1812.01718): two convolutional VAEs linked by a mixture density network. - [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](/papers/1905.12506): VAE encoders judged by downstream reasoning accuracy. - [Zero-Shot Text-to-Image Generation](/papers/2102.12092): a discrete VAE that tokenises images for a transformer. - [Deep Generative Modelling: A Comparative Review](/papers/2103.04922): a survey of VAE and related families, with compiled numbers. - [TSDAE: Sequential denoising auto-encoder for sentence embeddings](/papers/2104.06979): a denoising autoencoder trained on unlabelled sentences. - [VideoGPT](/papers/2104.10157): a VQ-VAE with an EMA codebook and a GPT-style prior. - [Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals](/papers/2107.12626): a convolutional autoencoder with an MMD penalty for anomaly scores. - [Motion Latent Diffusion](/papers/2212.04048): a transformer VAE with diffusion in its latent space. - [Beyond Statistical Similarity](/papers/2302.02913): metrics for generative design models, including a VAE and a conditional GAN. - [Sparse Autoencoders Find Highly Interpretable Features in Language Models](/papers/2309.08600): sparse autoencoders trained on language model activations. - [GraspLDM](/papers/2312.11243): a conditional VAE with a DDPM prior in its latent space. - [Monocular Depth in Non-Lambertian Surfaces](/papers/2408.06083): a pretrained Stable Diffusion VAE used to fuse exposures. - [STEVE-Audio](/papers/2412.00949): a conditional VAE that maps audio into a policy's goal space. - [EEG and Generative AI: a survey](/papers/2502.12048): a survey that names VAEs among generative options. - [AlignSAE](/papers/2512.02004): a sparse autoencoder with supervised concept slots. - [Interpretable Embeddings with Sparse Autoencoders](/papers/2512.10092): a released sparse autoencoder used as a feature extractor. - [Terrain-Aware Bipedal Locomotion](/papers/2512.12993): a convolutional VAE that compresses height maps for a walking policy. - [Generative Recursive Reasoning](/papers/2605.19376): Gaussian noise injected into a recursive reasoner's state.