---
kind: concept
slug: gradient-penalty
title: Gradient penalty
summary: >-
  A term added to a critic's loss that pushes the norm of its input gradient towards one at random points between real and generated samples, weighted by a coefficient lambda. The corpus has one card, SliceGAN, which uses it in a Wasserstein GAN with a 2D critic on 3D-volume slices. No card compares it with other ways of constraining a critic.
tags: [gan, wasserstein-gan, regularization, critic, generative-models]
papers: ["2102.07708"]
related: [generative-adversarial-network, spectral-normalization]
status: draft
updated_at: 2026-11-05
---

## Overview

A gradient penalty is a regularisation term added to the loss of a critic (a
discriminator that scores samples without a real-or-fake output) in a
Wasserstein GAN. It penalises the critic when the norm of its input gradient
departs from one, evaluated at points sampled on straight lines between a real
sample and a generated one [arxiv:2102.07708 §3] [arxiv:2102.07708 alg. 1].

The penalty is a squared difference between the
gradient norm and one, scaled by a coefficient lambda, and it is added to the
critic's Wasserstein loss [arxiv:2102.07708 alg. 1]. The generator's loss does not
include the penalty; it is the negative critic score of its samples [arxiv:2102.07708
alg. 1]. The paper does not attribute the term to a source; it names the
Wasserstein GAN loss as the base it builds on [arxiv:2102.07708 §3].

## Variants

Only one variant appears in the corpus.

### Two-sided penalty on random interpolates

The critic's loss is its Wasserstein difference plus lambda times the squared
gap between the gradient norm and one. The gradient is taken at a random point
between each fake sample and a paired real sample, with the mixing weight drawn
uniformly from zero to one [arxiv:2102.07708 alg. 1]. The paper's critic is a 2D
network, and each generated 3D volume contributes its slices along three axes, so
each slice is one critic input [arxiv:2102.07708 §3]. Card: [SliceGAN](/papers/2102.07708).

The paper gives no value for lambda; the algorithm names it as a free parameter
[arxiv:2102.07708 alg. 1]. Its learning rate, Adam settings and critic update count
are also not stated in the main text [arxiv:2102.07708 alg. 1].

## When to use which

The corpus gives one paper, so there is no comparison to draw on. The paper uses
the penalty to train a critic on slices of one generated volume against real 2D
images, and reports visual and statistical agreement on its own datasets
[arxiv:2102.07708 §5.2]. That result shows the term can be used this way. It does
not show that the penalty works better than other ways of constraining a critic.

Points the paper does support for choosing this design:

- Each penalty sample needs a gradient of the critic's input gradient, so the
  critic step costs more than a plain Wasserstein step. The paper does not report
  this cost on its own [arxiv:2102.07708 alg. 1].
- The critic must take the same kind of input as the penalty samples: here, a
  generated slice paired with a real image [arxiv:2102.07708 alg. 1].

For alternatives to a penalty, see the [spectral normalization](/concepts/spectral-normalization)
page. This page does not compare the two.

## Implementation sketch

Pseudocode of the critic loss with the penalty, following the paper's algorithm.
Lines are kept short; the paper's batch and step counts are not repeated here.

```pseudocode
# Critic loss with a gradient penalty  [arxiv:2102.07708 alg. 1]
# Input:  real sample x, fake sample y, critic D, coefficient lam
# Output: critic loss to minimise

def critic_loss(D, x, y, lam):
    # random mixing weight in [0, 1]  [arxiv:2102.07708 alg. 1]
    eps = uniform(0, 1)
    # point on the line between real and fake  [arxiv:2102.07708 alg. 1]
    mix = eps * y + (1 - eps) * x
    # Wasserstein term: fake scores minus real scores  [arxiv:2102.07708 alg. 1]
    wasserstein = D(y) - D(x)
    # squared gap between gradient norm at mix and one  [arxiv:2102.07708 alg. 1]
    penalty = (grad_norm(D, mix) - 1) ** 2
    return wasserstein + lam * penalty
```

Implementation notes:

- The paper does not state lambda. The official code uses a value of 10, and the
  official code is MIT-licensed [code: model.py#L46].
- The official code computes the penalty on a subset of the batch, not on one
  interpolate per fake sample. Algorithm 1 pairs one real image with each fake
  slice, while the code takes the first eight fake slices and the real batch
  [arxiv:2102.07708 alg. 1] [code: model.py#L109-L111]. An implementer who
  follows the algorithm gets a different estimate of the penalty from one who
  follows the code.

## Contradictions

None known. Only one card in the corpus uses this term, so no two papers can
disagree about it. The gap between the algorithm and the official code is listed
under implementation notes above.

## Papers

- [SliceGAN](/papers/2102.07708): trains a 2D critic on slices of a generated 3D volume with a gradient penalty on interpolates between fake slices and real images.