Denoising autoencoder

Overview

A denoising autoencoder (DAE) corrupts an input, encodes the corrupted copy into a code, and decodes that code back towards the clean input. Because the target is the clean input, the code has to keep whatever structure is needed to rebuild it, not just a copy of the surface 2104.06979 §3.1. Most variants start from this core. A variant may add a second loss on the same code. Another may keep only the trained encoder and use its code as a penalty on a different network's outputs 1705.08302 §2.2. The sparse-input variant uses the name for something narrower: it pre-trains a first layer on sampled one-hot features, and its "denoising" sampler does not corrupt its input in the public code 1601.02376 §3.2. What changes is the corruption, what the decoder can see, and what the code is used for.

In the first three variants the code is kept narrow. In the cardiac paper the code is much smaller than the 11,514-value input [arxiv:1810.03382 § Network Design and Training]. In the sentence paper the whole sentence is squeezed into one fixed-size vector 2104.06979 §3.2. In the segmentation paper a 120 × 120 × 60 label map is compressed to a 64-value code 1705.08302 tab. 5. The sparse-input variant is the exception: the paper's preferred hidden shape has 600 units in total 1601.02376 §4.4. The time-series imaging variant has a single 500-unit hidden layer, and its input is an n × n image of a series of length n 1506.00327 §4.1 §2.1. Its output is read back as the filled-in values at the missing points 1506.00327 §4. The idea is usually traced to Vincent et al. (2010), which has no card here, so nothing from that work is used.

Variants

Feature masking with a survival head

Input entries are zeroed at random, with a masking fraction redrawn on every backward pass. The encoder is a hidden ReLU layer and a code layer, and the decoder mirrors it 1810.03382 § Network Design and Training. A linear unit on the code gives a log hazard ratio. The loss is a weighted sum of the squared reconstruction error and the negative Cox partial likelihood, with weight α on reconstruction and 1 − α on survival 1810.03382 eq. 7. The paper's internal bootstrap reports an optimism-corrected Harrell's C of 0.73 for this model, against 0.59 for a Cox model on manual right-ventricle volumetric indices, in 302 patients from one centre [arxiv:1810.03382 § Predictive performance]. These are internal estimates with no external validation. Card: cardiac motion survival model.

Token deletion with a single-vector decoder

Each sentence loses a fraction of its tokens. The encoder outputs one [CLS] vector, and a Transformer decoder reconstructs the original sentence. Its cross-attention takes keys and values only from that one vector, so the decoder cannot read per-token encoder states 2104.06979 §3.2. Encoder and decoder share weights, except the cross-attention layers 2104.06979 app. A. Training is unsupervised: 100K steps at batch size 8, AdamW, learning rate 3e-5 2104.06979 app. B. After training the decoder is discarded and the encoder alone produces embeddings 2104.06979 §3.2. Card: TSDAE.

Label-swap autoencoder as a training-time shape penalty

A stacked convolutional autoencoder is trained on segmentation label maps, with a 64-value code 1705.08302 tab. 5. During its training, tissue labels are swapped at random with probability 0.1. The authors use this so that similar label maps land near each other in the code space 1705.08302 §3.2. The segmentation network then trains on its usual cross-entropy loss plus a term: the squared distance between the code of its predicted label maps and the code of the true labels. Only the segmentation weights are updated by that term 1705.08302 §2.2 eq. 1. The autoencoder is used only during training, and the authors report no extra inference cost 1705.08302 §2.4. Two details are left open. The paper does not say how long the autoencoder is trained before use, or whether its encoder is frozen during segmentation training. The label swap is given only in the training details, while the loss is written on clean labels, so whether the corruption enters the loss is not stated 1705.08302 §2.2 §3.2.

The autoencoder cannot encode an intensity image, so for super-resolution the paper adds a predictor that maps an intensity image to the same 64-value code. The autoencoder is first trained alone on ground-truth masks, then the predictor is trained to match its codes 1705.08302 §2.4. The paper calls this T-L model an extension of the denoising autoencoder and trains it with small Gaussian input noise 1705.08302 §2.4 §2.5. Card: ACNN.

Sampled one-hot features with a sigmoid first layer

A sigmoid first layer over all one-hot inputs is pre-trained on a sampled copy of the input. For each field the active unit is kept, negative units are sampled, and the rest are ignored 1601.02376 §3.2 fig. 2. The paper calls its SGD-trained sampler a denoising autoencoder (SNN-DAE). Its other sampler is an RBM (SNN-RBM) 1601.02376 §3.2. The upper layers are then fine-tuned by back-propagation of cross-entropy 1601.02376 §3.2 eq. 7. The public code applies no corruption by default and draws one negative per field, not the two the text reports; the card's Paper vs code section has the details 1601.02376 §3.2. Author-reported, the pooled AUC on the iPinYou data is 69.15%, against 68.81% for logistic regression and 70.70% for the paper's FNN 1601.02376 tab. 1. SNN-DAE is best on advertiser 2997 at 61.58% 1601.02376 tab. 1. Card: Deep Learning over Multi-field Categorical Data.

Corrupted series image with a diagonal inverse (imputation)

The series is first encoded as a Gramian angular summation field (GASF), an n × n image whose cell (i, j) is the cosine of the sum of two angles 1506.00327 §2.1. For training, 20% of the raw points are set to zero before the image is built, and the image is not corrupted again 1506.00327 §4.1. A single hidden layer of 500 sigmoid units is trained with mean squared error to rebuild the clean GASF image. The paper's stopping rule is an MSE change below 10⁻³ 1506.00327 §4.1.

To impute, the paper rescales the series to [0, 1], zeroes the missing points, encodes the result, and reconstructs it. It then reads the diagonal of the reconstructed image, which holds cos(2φ), and inverts it with cos φ = √((cos 2φ + 1) / 2) 1506.00327 §4. This inverse holds only for [0, 1] rescaling, and the paper says the [−1, 1] map is not invertible in this sense 1506.00327 §2.1 §4. Author-reported over 10 runs, the MSE on the missing points is lower than for a denoising autoencoder on the raw series, on all five datasets in tab. 2, by 12.18% to 48.02% 1506.00327 §4.2 tab. 2. The whole-series MSE is higher on all five tab. 2. The §4.1 training list includes ECG, CBF, Gun Point and SwedishLeaf, which also appear in tab. 2. Whether tab. 2 is scored on held-out series is not stated [arxiv:1506.00327 §4.1] tab. 2. Card: Imaging Time-Series.

When to use which

Implementation sketch

Shared core in Python-like pseudocode. The variants differ in the corruption, the decoder's inputs and the optional auxiliary loss.

# Shared core of a denoising autoencoder  [arxiv:2104.06979 §3.1]
# [arxiv:1810.03382 § Network Design and Training]
# Corrupt, encode, decode, reconstruct the clean input.
def dae_loss(x, corrupt, encode, decode, aux_loss=None, alpha=0.5):
    # Corruption is the main design choice: masking entries
    # [arxiv:1810.03382 § Network Design and Training], deleting tokens
    # [arxiv:2104.06979 app. A] or swapping labels [arxiv:1705.08302 §3.2]
    x_noisy = corrupt(x)
    # Encode to a narrow code; sentence variant: one vector per input
    # [arxiv:2104.06979 §3.2]
    code = encode(x_noisy)
    # Decoder reads the code; sentence variant: cross-attention on one row
    # [arxiv:2104.06979 §3.2]
    x_hat = decode(code)
    # Reconstruction loss against the clean input, not the corrupted one
    recon = reconstruction_loss(x_hat, x)
    if aux_loss is None:
        return recon
    # Optional second loss on the same code, weighted by alpha
    # [arxiv:1810.03382 eq. 7]. The label-map variant instead compares
    # codes of predicted and true labels [arxiv:1705.08302 eq. 1].
    return alpha * recon + (1 - alpha) * aux_loss(code)

Where the variants and the papers' own code differ:

• Final settings are not reported. The cardiac paper gives search ranges only, for the masking fraction, widths, latent size, learning rate and α 1810.03382 tab. 2. An implementer has to run the search.
• Loss scale. The cardiac paper's reconstruction term is a per-patient squared norm, and its Cox term is summed over patients [arxiv:1810.03382 eq. 3] 1810.03382 eq. 6. The public PyTorch code averages the reconstruction over all elements, which shrinks it by the input size. A learning rate tuned under one convention may not transfer to the other 1810.03382 § Network Design and Training. The repository is described on its card.
• Risk sets. The paper's risk set is the whole training sample 1810.03382 eq. 5. The public code builds risk sets inside each mini-batch, which is much smaller.
• Overflow. Taking exp of the log hazard before a cumulative sum can overflow for large predictions. A log-sum-exp form is safer. This is our suggestion, not the paper's.
• Deletion rate. In the sentence paper a ratio of 0.6 means 60% of tokens are affected 2104.06979 app. A. The repository's example scripts do not agree with each other on this, so check the convention before copying a script. The TSDAE card gives the file-level detail.
• Decoder initialisation. When weights are tied, the cross-attention weights are new and the paper does not say how to initialise them 2104.06979 app. A.
• Shape penalty. The label-map variant does not reconstruct its input at training time. Apply the trained encoder to the predicted label maps and to the true labels, and take the squared distance between the two codes 1705.08302 eq. 1. Whether the encoder is frozen is not stated 1705.08302 §2.2.
• Sampled inputs. The sparse-input variant samples the input instead of corrupting it, and its first layer is wide. The core above does not describe it 1601.02376 §3.2 §4.4.
• Image encoding and missing values. The imputation variant zeroes raw points before it builds the image, so a real zero in the data looks the same as a removed point unless a mask is kept 1506.00327 §4.1. The paper does not say which min and max undo its rescaling 1506.00327 §4. Store the per-series min and max yourself. This is our suggestion, not the paper's. The test-time corruption rate is not stated 1506.00327 §4.1.

Contradictions

No result conflicts between the five papers. They solve different tasks and report different metrics, so their numbers should not be compared directly. Within the segmentation paper, the autoencoder is described as a plain undercomplete model 1705.08302 §2.2, while the super-resolution model is described as an extension of the denoising autoencoder 1705.08302 §2.4. This is a difference in wording, not a conflict in results.

Two points need care. The sparse-input paper calls its sampler denoising, but its public code applies no corruption, while the cardiac, sentence, segmentation and time-series papers all corrupt their inputs 1601.02376 §3.2 1810.03382 § Network Design and Training 2104.06979 §3.2 1705.08302 §3.2 1506.00327 §4.1. Also within the sparse-input paper, the text says the sampled models beat LR and FM on all tested datasets, but the table shows SNN-DAE below LR on advertisers 2259 and 3386 1601.02376 §4.2 tab. 1.

Papers

• Cardiac motion survival model: masked-input DAE over cardiac motion, trained jointly with a Cox survival loss.
• TSDAE: token-deletion DAE with a single-vector decoder, for unsupervised sentence embeddings.
• Anatomically Constrained Neural Networks (ACNN): label-map autoencoder used as a training-time shape penalty, with a predictor for intensity outputs.
• Deep Learning over Multi-field Categorical Data: sigmoid first layer pre-trained on sampled one-hot features, for ad click prediction.
• Imaging Time-Series to Improve Classification and Imputation: denoising autoencoder on GASF images, used to fill missing points in a series.