Autoencoder

Overview

An autoencoder has two parts: an encoder that maps an input to a shorter code, and a decoder that maps the code back to the input. Training minimises a reconstruction error 2107.12626 §3.3 1810.03382 eq. 3. The bottleneck is the point: the code has to keep enough of the input for the decoder to rebuild it.

All variants keep this encoder-code-decoder shape. They differ in three places:

• The input. Corrupting it before encoding gives a denoising autoencoder, which must rebuild the clean input 2104.06979 §3.1.
• The code. A sparsity penalty gives sparse features 2309.08600 eq. 4. A probabilistic code with a KL term gives a variational autoencoder, and a codebook of nearest entries gives a discrete one 2102.12092 §2.1.
• The use of the code. Reconstruction is often only the training signal. The code may be scored, used as a penalty, read by a downstream model or used as the latent of a generator.

Variants

Plain and convolutional autoencoders

A convolutional autoencoder with a deterministic code is the simplest form. In CAE-M, windows of multi-sensor data go through a convolutional encoder and a transposed-convolution decoder, and training uses only normal samples 2107.12626 §3.1 2107.12626 §3.3.1. The anomaly score combines the reconstruction error with the code, and a threshold of mean plus standard deviation on training losses sets the decision 2107.12626 eq. 6 2107.12626 eq. 21. On PAMAP2 the author-reported mean F1 is 0.9639. Setting both the reconstruction and MMD weights to zero drops it to 0.5563 2107.12626 tab. 6, so the reconstruction-based parts carry the score. The baselines are re-implementations, using PyOD and a third-party MSCRED repository, so the comparison depends on those choices 2107.12626 §4.3.

An autoencoder can also serve only during training. In a cardiac segmentation model, a stacked convolutional autoencoder encodes one-hot label maps into a 64-value code. The squared distance between the codes of predicted and true labels is added to the loss as a shape penalty 1705.08302 eq. 1 1705.08302 §2.2. The autoencoder is not used at inference 1705.08302 §2.4, so the extra cost is paid only in training. Author- reported endocardial Dice is 0.939 ± 0.017, against 0.923 for the 3D baseline 1705.08302 tab. 1.

Denoising autoencoders

The input is corrupted and the target is the clean input. See Denoising autoencoder. Three cards show how the corruption and the decoder's view change the use:

• Survival prediction from motion. A mask is applied to the input and redrawn at each backward pass. The loss is a weighted sum of reconstruction and a negated Cox partial likelihood [arxiv:1810.03382 § Network Design and Training] 1810.03382 eq. 7. The author-reported optimism-corrected C-index is 0.73, against 0.59 for a Cox model on manual volumes. This is internal bootstrap validation in 302 patients from one centre 1810.03382 § Predictive performance.
• Sentence embeddings. Each sentence loses 60% of its tokens. A Transformer encoder outputs one vector, and the decoder reads only that vector 2104.06979 §3.2 2104.06979 app. A. The unsupervised average is 55.2, against 52.4 for the contrastive baseline [arxiv:2104.06979 tab. 2]. On general STS it is weaker: 66.0 against 73.9 on STS-B 2104.06979 tab. 3.
• Time-series imputation. The autoencoder works on an image encoding of the series, not on the raw values. It has one sigmoid hidden layer of 500 units and is trained on inputs with 20% of points set to zero 1506.00327 §4.1. Interpolation error falls: the card's calculation from the table gives reductions of 12.18% to 48.02%, which the paper does not state 1506.00327 §4.2. Full-series error is worse for the image encoding in all five rows 1506.00327 tab. 2.

A fourth card calls its pre-training step a denoising autoencoder, but the corruption is not applied. Negative units are sampled per field and the positive unit is kept 1601.02376 §3.2. See Contradictions.

Variational autoencoders

The code is a distribution, trained with reconstruction plus a KL term. See Variational autoencoder. The cards use it in several ways:

• World models. A convolutional VAE with a 32-value latent for car racing and 64 for Doom is trained without reward. A separate controller sees the reward and is evolved 1803.10122 §3.1 1809.01999 app. A.2. Car racing reaches 906 ± 21 with the latent and the recurrent state, against 632 ± 251 with the latent alone 1809.01999 tab. 1. The paper trains the VAE for one epoch, and the released code runs ten 1803.10122 §A.1.
• Two image domains. Two convolutional VAEs, one per domain, each with a 64-value latent, are linked by a mixture density network that maps one latent to the other 1812.01718 §3.2. The paper reports no quantitative transfer metric for the link 1812.01718 §3.2.
• Disentanglement for reasoning. beta-VAE, FactorVAE, beta-TCVAE and DIP-VAE encoders are compared by the accuracy of a downstream reasoner, not by reconstruction 1905.12506 §2. Disentanglement scores track accuracy in the few-sample regime up to 20K steps, and the link weakens by 100K steps 1905.12506 §4.2.2.
• Latent diffusion for motion. A transformer VAE turns a motion clip into a single 256-value token, and diffusion runs in that latent. The paper reports an HumanML3D FID of 0.473, against 0.544 for MDM, and about 22 times less DDIM-50 inference time than MDM 2212.04048 tab. 1 2212.04048 tab. 10.
• Latent diffusion for grasps. A conditional VAE produces grasps, and a DDPM prior runs in its frozen latent. Median simulated success is 0.792 with DDPM, 0.756 with DDIM and 0.660 with the VAE alone 2312.11243 tab. 1.
• Goal prior for a policy. A conditional VAE maps an audio embedding into the goal space of a frozen policy, which is not retrained [arxiv:2412.00949 §3.4]. The paper reports the audio prompt beats the visual one on 4 of 6 tasks 2412.00949 §4.1.
• Terrain latent. A convolutional VAE compresses a 20 × 40 height map into 16 values with an MSE plus β·KL loss, where β is not stated 2512.12993 §II.B 2512.12993 eq. 4.
• Frozen pretrained VAE. A Stable Diffusion VAE encodes several exposures of one scene, and the latents are averaged before decoding. The VAE is not modified 2408.06083 §3.3.
• Evaluation and surveys. A metrics study compares a conditional VAE with a conditional GAN on a two-dimensional design problem 2302.02913 tab. 6, and the authors warn that these results may not carry over [arxiv:2302.02913 §4.5]. A deep generative survey gives the ELBO and reparameterised sampling 2103.04922 eq. 11 2103.04922 eq. 13. Its CIFAR-10 table is compiled from other papers and is not like-for-like 2103.04922 tab. 1. An EEG survey trains no model 2502.12048 §3.

Discrete autoencoders

The code is a choice from a fixed set, so the encoder output can be a token for an autoregressive model. See Vector quantization.

• Gumbel-softmax dVAE. A 256 × 256 image becomes a 32 × 32 grid of tokens from an 8192-entry dictionary 2102.12092 §2.1. The encoder is trained with a relaxed sample, and the KL weight is raised from 0 to 6.6 2102.12092 app. A.2. The paper's text-to-image transformer models caption tokens and these image tokens as one stream 2102.12092 §2.2.
• VQ-VAE with EMA codebook. A VQ-VAE with 3D convolutions and axial attention maps video to a grid of codes. The codebook is updated by an exponential moving average 2104.10157 §2.1. A GPT-style prior models the codes 2104.10157 §3. The author-reported BAIR FVD is 103.3 2104.10157 tab. 1.

Sparse autoencoders

The code is overcomplete and penalised for activity. See Sparse autoencoder. The cards use this on model activations, not on data.

• Interpretability of a language model. A one-hidden-layer model with a ReLU encoder and a tied decoder is trained on the residual stream. The hidden width is R times the input width 2309.08600 §2. The loss is reconstruction plus α times the L1 norm of the code 2309.08600 eq. 4. The paper reports higher autointerpretability scores than baselines, shown only in a plot 2309.08600 fig. 2.
• Supervised concept slots. AlignSAE splits the code into free features and slots bound to labelled relations. A cross-entropy binding loss is added to the reconstruction and L1 terms 2512.02004 app. C.2. The card reports swap success of 0.847 at alpha = 2 2512.02004 §5.3.
• Reuse of a public SAE. A released SAE for a 70B model, at layer 50 with 65,536 latents, is used as a feature extractor for named concepts 2512.10092 §4. The authors report retrieval MAP of 0.361 against 0.412 for an LLM-based baseline on one prompt set 2512.10092 §4.4.

Adversarial and conditional autoencoders

Here the regulariser is adversarial rather than a KL term.

• Age progression. The conditional adversarial autoencoder encodes a face and decodes it with an age label. A discriminator pulls the code towards a uniform prior 1702.08423 §4.3, so the code is not regularised by a KL term. The age label is one-hot over ten bins 1702.08423 §5.1. Training needs no paired images of one person at two ages 1702.08423 §1.

Noise in a recurrent state

• Generative recursive reasoning. Each step of a small recursive reasoner adds Gaussian noise to its high-level state. The posterior sees the target, and the prior does not 2605.19376 §2.1 2605.19376 §2.2. The noise sits in the recurrent state, not in an encoder bottleneck, which is why it is listed here. Latency per puzzle is 184 ms, against 2,755 ms for the baseline at matched compute 2605.19376 tab. 10.

When to use which

The rows are chosen by constraint. The numbers are author-reported, and the cards test different tasks, so they are not a ranking.

Implementation sketch

Common core of the variants. Each hook marks where a variant differs.

# Corrupt the input (denoising only; identity otherwise) [arxiv:2104.06979 §3.1]
x_in = corrupt(x)
# Encode to a code z [arxiv:2107.12626 §3.3.1]
z = encoder(x_in)
# Variational: sample z from a Gaussian, reparameterised
# [arxiv:2103.04922 eq. 13]
# Discrete: snap z to the nearest codebook entry [arxiv:2102.12092 §2.1]
# Decode the code back to the clean input [arxiv:2104.06979 §3.2]
x_hat = decoder(z)
# Reconstruction against the clean input [arxiv:1810.03382 eq. 3]
loss = recon_error(x_hat, x)
# Variational: add the KL term [arxiv:2103.04922 eq. 12]
# Sparse: add an L1 penalty on the code; check its normalisation
# [arxiv:2309.08600 eq. 4]
# Task-specific term on the code, e.g. a Cox loss or a shape distance
# [arxiv:1810.03382 eq. 5]
# Downstream use reads z; the decoder may be dropped at inference
# [arxiv:1705.08302 §2.4]

Lines are kept at 80 characters or fewer. The decoder is dropped for the shape-prior variant and kept for generation.

Contradictions

• What "denoising" means. The label is used for input corruption in 1506.00327 §4 and for masking in [arxiv:1810.03382 § Network Design and Training]. The paper calls its pre-training sampler a denoising autoencoder 1601.02376 §3.2, but the card for that paper records that the released code sets the corruption level to zero, so no input is corrupted. Compare the paper's wording with the corruption step before you reuse it.
• Sparse penalty normalisation. The paper writes the L1 term as a sum over the input dimension 2309.08600 eq. 4. The card records that the released code averages it instead. That changes the effective alpha by a factor of the input width, so the stated alpha values are not interchangeable.
• World-model training length. The paper trains the VAE for one epoch and the MDN for 20 1803.10122 §A.1 1803.10122 §A.2. The card records that the released code runs ten VAE epochs and 4,000 MDN steps 1803.10122 §A.1 1803.10122 §A.2.
• Denoising mask. The paper's deletion ratio of 0.6 means 60% of tokens are affected 2104.06979 app. A. The card records that the released STS-B script keeps each word with probability 0.6, so it deletes about 40% of words 2104.06979 app. A.
• Latent size. One conditional VAE is described with a 128-value latent in the paper, while the card records 64 or 256 in the released code 2312.11243 §3.3.

Papers

• Imaging Time-Series to Improve Classification and Imputation: a denoising autoencoder on image encodings for imputation.
• Deep Learning over Multi-field Categorical Data: autoencoder-based pre-training for click prediction, with a sampler called denoising.
• Age Progression/Regression by Conditional Adversarial Autoencoder: a conditional adversarial autoencoder with a discriminator on the code.
• Anatomically Constrained Neural Networks: an autoencoder used only as a training-time shape prior.
• World Models: a VAE, an MDN-RNN and a controller trained by evolution.
• Recurrent World Models Facilitate Policy Evolution: the same three-part model, with the car and Doom settings.
• Deep learning cardiac motion analysis for human survival prediction: a denoising autoencoder with a Cox survival loss.
• Deep Learning for Classical Japanese Literature: two convolutional VAEs linked by a mixture density network.
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?: VAE encoders judged by downstream reasoning accuracy.
• Zero-Shot Text-to-Image Generation: a discrete VAE that tokenises images for a transformer.
• Deep Generative Modelling: A Comparative Review: a survey of VAE and related families, with compiled numbers.
• TSDAE: Sequential denoising auto-encoder for sentence embeddings: a denoising autoencoder trained on unlabelled sentences.
• VideoGPT: a VQ-VAE with an EMA codebook and a GPT-style prior.
• Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals: a convolutional autoencoder with an MMD penalty for anomaly scores.
• Motion Latent Diffusion: a transformer VAE with diffusion in its latent space.
• Beyond Statistical Similarity: metrics for generative design models, including a VAE and a conditional GAN.
• Sparse Autoencoders Find Highly Interpretable Features in Language Models: sparse autoencoders trained on language model activations.
• GraspLDM: a conditional VAE with a DDPM prior in its latent space.
• Monocular Depth in Non-Lambertian Surfaces: a pretrained Stable Diffusion VAE used to fuse exposures.
• STEVE-Audio: a conditional VAE that maps audio into a policy's goal space.
• EEG and Generative AI: a survey: a survey that names VAEs among generative options.
• AlignSAE: a sparse autoencoder with supervised concept slots.
• Interpretable Embeddings with Sparse Autoencoders: a released sparse autoencoder used as a feature extractor.
• Terrain-Aware Bipedal Locomotion: a convolutional VAE that compresses height maps for a walking policy.
• Generative Recursive Reasoning: Gaussian noise injected into a recursive reasoner's state.