Gradient penalty

Overview

A gradient penalty is a regularisation term added to the loss of a critic (a discriminator that scores samples without a real-or-fake output) in a Wasserstein GAN. It penalises the critic when the norm of its input gradient departs from one, evaluated at points sampled on straight lines between a real sample and a generated one 2102.07708 §3 2102.07708 alg. 1.

The penalty is a squared difference between the gradient norm and one, scaled by a coefficient lambda, and it is added to the critic's Wasserstein loss 2102.07708 alg. 1. The generator's loss does not include the penalty; it is the negative critic score of its samples [arxiv:2102.07708 alg. 1]. The paper does not attribute the term to a source; it names the Wasserstein GAN loss as the base it builds on 2102.07708 §3.

Variants

Only one variant appears in the corpus.

Two-sided penalty on random interpolates

The critic's loss is its Wasserstein difference plus lambda times the squared gap between the gradient norm and one. The gradient is taken at a random point between each fake sample and a paired real sample, with the mixing weight drawn uniformly from zero to one 2102.07708 alg. 1. The paper's critic is a 2D network, and each generated 3D volume contributes its slices along three axes, so each slice is one critic input 2102.07708 §3. Card: SliceGAN.

The paper gives no value for lambda; the algorithm names it as a free parameter 2102.07708 alg. 1. Its learning rate, Adam settings and critic update count are also not stated in the main text 2102.07708 alg. 1.

When to use which

The corpus gives one paper, so there is no comparison to draw on. The paper uses the penalty to train a critic on slices of one generated volume against real 2D images, and reports visual and statistical agreement on its own datasets 2102.07708 §5.2. That result shows the term can be used this way. It does not show that the penalty works better than other ways of constraining a critic.

Points the paper does support for choosing this design:

• Each penalty sample needs a gradient of the critic's input gradient, so the critic step costs more than a plain Wasserstein step. The paper does not report this cost on its own 2102.07708 alg. 1.
• The critic must take the same kind of input as the penalty samples: here, a generated slice paired with a real image 2102.07708 alg. 1.

For alternatives to a penalty, see the spectral normalization page. This page does not compare the two.

Implementation sketch

Pseudocode of the critic loss with the penalty, following the paper's algorithm. Lines are kept short; the paper's batch and step counts are not repeated here.

# Critic loss with a gradient penalty  [arxiv:2102.07708 alg. 1]
# Input:  real sample x, fake sample y, critic D, coefficient lam
# Output: critic loss to minimise

def critic_loss(D, x, y, lam):
    # random mixing weight in [0, 1]  [arxiv:2102.07708 alg. 1]
    eps = uniform(0, 1)
    # point on the line between real and fake  [arxiv:2102.07708 alg. 1]
    mix = eps * y + (1 - eps) * x
    # Wasserstein term: fake scores minus real scores  [arxiv:2102.07708 alg. 1]
    wasserstein = D(y) - D(x)
    # squared gap between gradient norm at mix and one  [arxiv:2102.07708 alg. 1]
    penalty = (grad_norm(D, mix) - 1) ** 2
    return wasserstein + lam * penalty

Implementation notes:

• The paper does not state lambda. The official code uses a value of 10, and the official code is MIT-licensed model.py.
• The official code computes the penalty on a subset of the batch, not on one interpolate per fake sample. Algorithm 1 pairs one real image with each fake slice, while the code takes the first eight fake slices and the real batch 2102.07708 alg. 1 model.py. An implementer who follows the algorithm gets a different estimate of the penalty from one who follows the code.

Contradictions

None known. Only one card in the corpus uses this term, so no two papers can disagree about it. The gap between the algorithm and the official code is listed under implementation notes above.

Papers

• SliceGAN: trains a 2D critic on slices of a generated 3D volume with a gradient penalty on interpolates between fake slices and real images.