--- kind: concept slug: denoising-autoencoder title: Denoising autoencoder summary: >- Train an encoder and decoder to rebuild a clean input from a corrupted copy, so the code keeps the structure needed to rebuild it. Variants differ in the corruption, what the decoder may read, and what the code feeds: a survival loss, a sentence vector, a shape penalty, sparse-input pre-training, or time-series image repair for imputation. Choose by input type and label count. tags: [representation-learning, self-supervised-learning, autoencoder, sentence-embeddings, survival-analysis, medical-imaging, ctr-prediction, time-series, imputation] papers: ["1705.08302", "1810.03382", "2104.06979", "1601.02376", "1506.00327"] related: [self-supervised-learning, variational-autoencoder, sparse-autoencoder, masked-language-modeling, text-embedding, contrastive-learning, survival-analysis, cox-proportional-hazards, data-augmentation, semantic-segmentation, factorization-machine, greedy-layerwise-training, time-series-imputation] status: draft updated_at: 2026-10-24 --- ## Overview A denoising autoencoder (DAE) corrupts an input, encodes the corrupted copy into a code, and decodes that code back towards the clean input. Because the target is the clean input, the code has to keep whatever structure is needed to rebuild it, not just a copy of the surface [arxiv:2104.06979 §3.1]. Most variants start from this core. A variant may add a second loss on the same code. Another may keep only the trained encoder and use its code as a penalty on a different network's outputs [arxiv:1705.08302 §2.2]. The sparse-input variant uses the name for something narrower: it pre-trains a first layer on sampled one-hot features, and its "denoising" sampler does not corrupt its input in the public code [arxiv:1601.02376 §3.2]. What changes is the corruption, what the decoder can see, and what the code is used for. In the first three variants the code is kept narrow. In the cardiac paper the code is much smaller than the 11,514-value input [arxiv:1810.03382 § Network Design and Training]. In the sentence paper the whole sentence is squeezed into one fixed-size vector [arxiv:2104.06979 §3.2]. In the segmentation paper a 120 × 120 × 60 label map is compressed to a 64-value code [arxiv:1705.08302 tab. 5]. The sparse-input variant is the exception: the paper's preferred hidden shape has 600 units in total [arxiv:1601.02376 §4.4]. The time-series imaging variant has a single 500-unit hidden layer, and its input is an n × n image of a series of length n [arxiv:1506.00327 §4.1] [§2.1]. Its output is read back as the filled-in values at the missing points [arxiv:1506.00327 §4]. The idea is usually traced to Vincent et al. (2010), which has no card here, so nothing from that work is used. ## Variants ### Feature masking with a survival head Input entries are zeroed at random, with a masking fraction redrawn on every backward pass. The encoder is a hidden ReLU layer and a code layer, and the decoder mirrors it [arxiv:1810.03382 § Network Design and Training]. A linear unit on the code gives a log hazard ratio. The loss is a weighted sum of the squared reconstruction error and the negative Cox partial likelihood, with weight α on reconstruction and 1 − α on survival [arxiv:1810.03382 eq. 7]. The paper's internal bootstrap reports an optimism-corrected Harrell's C of 0.73 for this model, against 0.59 for a Cox model on manual right-ventricle volumetric indices, in 302 patients from one centre [arxiv:1810.03382 § Predictive performance]. These are internal estimates with no external validation. Card: [cardiac motion survival model](/papers/1810.03382). ### Token deletion with a single-vector decoder Each sentence loses a fraction of its tokens. The encoder outputs one [CLS] vector, and a Transformer decoder reconstructs the original sentence. Its cross-attention takes keys and values only from that one vector, so the decoder cannot read per-token encoder states [arxiv:2104.06979 §3.2]. Encoder and decoder share weights, except the cross-attention layers [arxiv:2104.06979 app. A]. Training is unsupervised: 100K steps at batch size 8, AdamW, learning rate 3e-5 [arxiv:2104.06979 app. B]. After training the decoder is discarded and the encoder alone produces embeddings [arxiv:2104.06979 §3.2]. Card: [TSDAE](/papers/2104.06979). ### Label-swap autoencoder as a training-time shape penalty A stacked convolutional autoencoder is trained on segmentation label maps, with a 64-value code [arxiv:1705.08302 tab. 5]. During its training, tissue labels are swapped at random with probability 0.1. The authors use this so that similar label maps land near each other in the code space [arxiv:1705.08302 §3.2]. The segmentation network then trains on its usual cross-entropy loss plus a term: the squared distance between the code of its predicted label maps and the code of the true labels. Only the segmentation weights are updated by that term [arxiv:1705.08302 §2.2 eq. 1]. The autoencoder is used only during training, and the authors report no extra inference cost [arxiv:1705.08302 §2.4]. Two details are left open. The paper does not say how long the autoencoder is trained before use, or whether its encoder is frozen during segmentation training. The label swap is given only in the training details, while the loss is written on clean labels, so whether the corruption enters the loss is not stated [arxiv:1705.08302 §2.2] [§3.2]. The autoencoder cannot encode an intensity image, so for super-resolution the paper adds a predictor that maps an intensity image to the same 64-value code. The autoencoder is first trained alone on ground-truth masks, then the predictor is trained to match its codes [arxiv:1705.08302 §2.4]. The paper calls this T-L model an extension of the denoising autoencoder and trains it with small Gaussian input noise [arxiv:1705.08302 §2.4] [§2.5]. Card: [ACNN](/papers/1705.08302). ### Sampled one-hot features with a sigmoid first layer A sigmoid first layer over all one-hot inputs is pre-trained on a sampled copy of the input. For each field the active unit is kept, negative units are sampled, and the rest are ignored [arxiv:1601.02376 §3.2] [fig. 2]. The paper calls its SGD-trained sampler a denoising autoencoder (SNN-DAE). Its other sampler is an RBM (SNN-RBM) [arxiv:1601.02376 §3.2]. The upper layers are then fine-tuned by back-propagation of cross-entropy [arxiv:1601.02376 §3.2] [eq. 7]. The public code applies no corruption by default and draws one negative per field, not the two the text reports; the card's Paper vs code section has the details [arxiv:1601.02376 §3.2]. Author-reported, the pooled AUC on the iPinYou data is 69.15%, against 68.81% for logistic regression and 70.70% for the paper's FNN [arxiv:1601.02376 tab. 1]. SNN-DAE is best on advertiser 2997 at 61.58% [arxiv:1601.02376 tab. 1]. Card: [Deep Learning over Multi-field Categorical Data](/papers/1601.02376). ### Corrupted series image with a diagonal inverse (imputation) The series is first encoded as a Gramian angular summation field (GASF), an n × n image whose cell (i, j) is the cosine of the sum of two angles [arxiv:1506.00327 §2.1]. For training, 20% of the raw points are set to zero before the image is built, and the image is not corrupted again [arxiv:1506.00327 §4.1]. A single hidden layer of 500 sigmoid units is trained with mean squared error to rebuild the clean GASF image. The paper's stopping rule is an MSE change below 10⁻³ [arxiv:1506.00327 §4.1]. To impute, the paper rescales the series to [0, 1], zeroes the missing points, encodes the result, and reconstructs it. It then reads the diagonal of the reconstructed image, which holds cos(2φ), and inverts it with cos φ = √((cos 2φ + 1) / 2) [arxiv:1506.00327 §4]. This inverse holds only for [0, 1] rescaling, and the paper says the [−1, 1] map is not invertible in this sense [arxiv:1506.00327 §2.1] [§4]. Author-reported over 10 runs, the MSE on the missing points is lower than for a denoising autoencoder on the raw series, on all five datasets in tab. 2, by 12.18% to 48.02% [arxiv:1506.00327 §4.2] [tab. 2]. The whole-series MSE is higher on all five [tab. 2]. The §4.1 training list includes ECG, CBF, Gun Point and SwedishLeaf, which also appear in tab. 2. Whether tab. 2 is scored on held-out series is not stated [arxiv:1506.00327 §4.1] [tab. 2]. Card: [Imaging Time-Series](/papers/1506.00327). ## When to use which | Situation | Use | Why | |---|---|---| | Per-entity numeric inputs with time-to-event labels, few entities | Feature masking with a survival head | The reconstruction term and the Cox term share the code. The reported gain is internal and comes from one cohort [arxiv:1810.03382 § Predictive performance]. | | Sentence embeddings, no labelled sentence pairs | Token deletion with a single-vector decoder | Unsupervised average 55.2 on four domain tasks, against 52.4 for Contrastive Tension and 52.9 for masked language modelling [arxiv:2104.06979 tab. 2]. | | Sentence embeddings, a few labelled pairs are possible | Token deletion, then supervised training | Labelled pairs needed to match TSDAE: about 2,333 on average, 140 on AskUbuntu [arxiv:2104.06979 tab. 10]. The estimate depends on one annotator's timing. | | General STS benchmark is the target | Not this variant alone | TSDAE scores 66.0 on STSb, against 73.9 for Contrastive Tension [arxiv:2104.06979 tab. 3]. The paper argues STS is a weak guide to the downstream tasks it reports [arxiv:2104.06979 §4]. | | Backbone other than a single-encoder checkpoint such as BERT-base | Check first | Average 55.2 with BERT-base, against 47.6 with BART-base, 31.7 with T5-base and 50.5 from scratch [arxiv:2104.06979 tab. 4]. On RoBERTa-base TSDAE scores 50.3 and Contrastive Tension 50.5 [arxiv:2104.06979 tab. 9]. | | Tight compute or no time for a search | Plan for the cost | The cardiac pipeline took 76 hours, including search and bootstrap [arxiv:1810.03382 § Network Design and Training]. TSDAE runs two Transformers per step, and the paper does not report time or memory [arxiv:2104.06979 app. B]. | | Segmentation with dense label maps, implausible shapes to avoid, and no change allowed at inference | Label-swap autoencoder as a training-time penalty | Author-reported endocardial Dice 0.939 against 0.923 for the sub-pixel baseline [arxiv:1705.08302 tab. 1]. The Hausdorff gain is not significant, and the 900 training segmentations come from one cohort [tab. 1] [§3.3]. | | Intensity output such as super-resolution, with label maps available for training | Predictor onto the same code (T-L model) | Author-reported SSIM 0.796 against 0.783 for SR-CNN [arxiv:1705.08302 tab. 3]. The two-stage schedule and the joint update scale are not fully stated [arxiv:1705.08302 §2.4]. | | Many one-hot categorical fields, about a million sparse features, and a wide first layer that is costly to train from a random start | Sampled-feature pre-training of a sigmoid first layer (SNN-DAE) | A dense first layer on this input would need about 100 million weights [arxiv:1601.02376 §1]. Pooled AUC 69.15%, against 68.81% for LR and 70.70% for the paper's FNN, which is its best pooled model [arxiv:1601.02376 tab. 1]. SNN-DAE trails LR on advertisers 2259 and 3386 [arxiv:1601.02376 tab. 1]. | | Gaps in a univariate series, where only the missing points are scored | Corrupted series image with a diagonal inverse | Interpolation MSE below the raw-series autoencoder on all five tab. 2 datasets [arxiv:1506.00327 tab. 2]. Whole-series MSE is higher on all five, so do not use it where the full series must be accurate. Only [0, 1] rescaling inverts cleanly [arxiv:1506.00327 §4]. | ## Implementation sketch Shared core in Python-like pseudocode. The variants differ in the corruption, the decoder's inputs and the optional auxiliary loss. ```pseudocode # Shared core of a denoising autoencoder [arxiv:2104.06979 §3.1] # [arxiv:1810.03382 § Network Design and Training] # Corrupt, encode, decode, reconstruct the clean input. def dae_loss(x, corrupt, encode, decode, aux_loss=None, alpha=0.5): # Corruption is the main design choice: masking entries # [arxiv:1810.03382 § Network Design and Training], deleting tokens # [arxiv:2104.06979 app. A] or swapping labels [arxiv:1705.08302 §3.2] x_noisy = corrupt(x) # Encode to a narrow code; sentence variant: one vector per input # [arxiv:2104.06979 §3.2] code = encode(x_noisy) # Decoder reads the code; sentence variant: cross-attention on one row # [arxiv:2104.06979 §3.2] x_hat = decode(code) # Reconstruction loss against the clean input, not the corrupted one recon = reconstruction_loss(x_hat, x) if aux_loss is None: return recon # Optional second loss on the same code, weighted by alpha # [arxiv:1810.03382 eq. 7]. The label-map variant instead compares # codes of predicted and true labels [arxiv:1705.08302 eq. 1]. return alpha * recon + (1 - alpha) * aux_loss(code) ``` Where the variants and the papers' own code differ: - **Final settings are not reported.** The cardiac paper gives search ranges only, for the masking fraction, widths, latent size, learning rate and α [arxiv:1810.03382 tab. 2]. An implementer has to run the search. - **Loss scale.** The cardiac paper's reconstruction term is a per-patient squared norm, and its Cox term is summed over patients [arxiv:1810.03382 eq. 3] [arxiv:1810.03382 eq. 6]. The public PyTorch code averages the reconstruction over all elements, which shrinks it by the input size. A learning rate tuned under one convention may not transfer to the other [arxiv:1810.03382 § Network Design and Training]. The repository is described on [its card](/papers/1810.03382). - **Risk sets.** The paper's risk set is the whole training sample [arxiv:1810.03382 eq. 5]. The public code builds risk sets inside each mini-batch, which is much smaller. - **Overflow.** Taking exp of the log hazard before a cumulative sum can overflow for large predictions. A log-sum-exp form is safer. This is our suggestion, not the paper's. - **Deletion rate.** In the sentence paper a ratio of 0.6 means 60% of tokens are affected [arxiv:2104.06979 app. A]. The repository's example scripts do not agree with each other on this, so check the convention before copying a script. The [TSDAE card](/papers/2104.06979) gives the file-level detail. - **Decoder initialisation.** When weights are tied, the cross-attention weights are new and the paper does not say how to initialise them [arxiv:2104.06979 app. A]. - **Shape penalty.** The label-map variant does not reconstruct its input at training time. Apply the trained encoder to the predicted label maps and to the true labels, and take the squared distance between the two codes [arxiv:1705.08302 eq. 1]. Whether the encoder is frozen is not stated [arxiv:1705.08302 §2.2]. - **Sampled inputs.** The sparse-input variant samples the input instead of corrupting it, and its first layer is wide. The core above does not describe it [arxiv:1601.02376 §3.2] [§4.4]. - **Image encoding and missing values.** The imputation variant zeroes raw points before it builds the image, so a real zero in the data looks the same as a removed point unless a mask is kept [arxiv:1506.00327 §4.1]. The paper does not say which min and max undo its rescaling [arxiv:1506.00327 §4]. Store the per-series min and max yourself. This is our suggestion, not the paper's. The test-time corruption rate is not stated [arxiv:1506.00327 §4.1]. ## Contradictions No result conflicts between the five papers. They solve different tasks and report different metrics, so their numbers should not be compared directly. Within the segmentation paper, the autoencoder is described as a plain undercomplete model [arxiv:1705.08302 §2.2], while the super-resolution model is described as an extension of the denoising autoencoder [arxiv:1705.08302 §2.4]. This is a difference in wording, not a conflict in results. Two points need care. The sparse-input paper calls its sampler denoising, but its public code applies no corruption, while the cardiac, sentence, segmentation and time-series papers all corrupt their inputs [arxiv:1601.02376 §3.2] [arxiv:1810.03382 § Network Design and Training] [arxiv:2104.06979 §3.2] [arxiv:1705.08302 §3.2] [arxiv:1506.00327 §4.1]. Also within the sparse-input paper, the text says the sampled models beat LR and FM on all tested datasets, but the table shows SNN-DAE below LR on advertisers 2259 and 3386 [arxiv:1601.02376 §4.2] [tab. 1]. ## Papers - [Cardiac motion survival model](/papers/1810.03382): masked-input DAE over cardiac motion, trained jointly with a Cox survival loss. - [TSDAE](/papers/2104.06979): token-deletion DAE with a single-vector decoder, for unsupervised sentence embeddings. - [Anatomically Constrained Neural Networks (ACNN)](/papers/1705.08302): label-map autoencoder used as a training-time shape penalty, with a predictor for intensity outputs. - [Deep Learning over Multi-field Categorical Data](/papers/1601.02376): sigmoid first layer pre-trained on sampled one-hot features, for ad click prediction. - [Imaging Time-Series to Improve Classification and Imputation](/papers/1506.00327): denoising autoencoder on GASF images, used to fill missing points in a series.