---
kind: concept
slug: spectral-normalization
title: Spectral normalization
summary: >-
  Capping the spectral norm (largest singular value) of weight matrices after each update, so a network cannot squeeze distinct inputs together. The cap is a hyperparameter found by search, and the one-step power estimate is approximate. Cards: SNGP, one half of a single-model uncertainty method; Gouk et al., an l2 cap as a regulariser; RL-GAN-Net, a setting in a latent GAN.
tags: [regularization, lipschitz, uncertainty, generative-models]
papers: ["2006.10108", "1804.04368", "1904.12304"]
related: [lipschitz-constraint, uncertainty-estimation, calibration, gaussian-process, ensembles, generative-adversarial-network]
status: draft
updated_at: 2026-10-19
---

## Overview

Spectral normalization rescales a weight matrix after an update so that its
spectral norm, the largest singular value, does not exceed a bound `c`. The
purpose is to limit how far a layer can stretch distances between its inputs.
In SNGP the argument runs as follows: if every residual branch is α-Lipschitz
with α ≤ 1, the hidden map is bi-Lipschitz, so inputs that are far apart stay
far apart after the hidden layers [arxiv:2006.10108 §3.2]. The card treats this
distance preservation as the precondition for a distance-aware output layer
[arxiv:2006.10108 §2] [arxiv:2006.10108 §3].

The card cites the origin of the residual-network variant as Miyato et al.
(2018), together with Lipschitz analyses by Bartlett et al. (2018) and
Behrmann et al. (2019). None of these has a card in this wiki
[arxiv:2006.10108 §3.2].

Gouk et al. use the same rescaling rule, with the bound called λ, as a
regulariser for generalisation rather than for uncertainty. They also compare
caps under the ℓ1 and ℓ∞ norms, which are exact to compute where the ℓ2 cap is
an estimate [arxiv:1804.04368 §3.1] [arxiv:1804.04368 §4.1]. The general
method is on the [Lipschitz constraint](/concepts/lipschitz-constraint) page;
this page covers the spectral (ℓ2) case.

In a GAN the same operation serves a different purpose. The RL-GAN-Net card
uses spectral normalization, together with batch normalization, in the
generator and the discriminator of its latent GAN, and states no bound value
for it [arxiv:1904.12304 supp. l-GAN Details].

## Variants

### Every hidden weight matrix, one power iteration per step (SNGP)

After every SGD step, the method estimates the spectral norm of each hidden
weight matrix `W_l` with one power-iteration step. It rescales the matrix only
when the estimate exceeds `c` [arxiv:2006.10108 §3.2] [arxiv:2006.10108 alg. 1].
Card: [SNGP, Liu et al. 2020](/papers/2006.10108).

The paper's residual-network setting uses a bound of 6.0 for the convolutional
layers. The authors state that this is looser than the residual-branch theory
supports [arxiv:2006.10108 tab. 5] [arxiv:2006.10108 app. C].

### Pooler layer only (BERT)

For BERT-Base, only the pooler dense layer is normalized, with `c = 0.95`.
The hidden transformer layers are not normalized [arxiv:2006.10108 app. C].

### Per-layer ℓ2 cap as a regulariser (Gouk et al.)

Each layer gets its own cap λ. After every optimiser step, a weight tensor
whose norm exceeds λ is divided by max(1, norm/λ) [arxiv:1804.04368 §4]
[arxiv:1804.04368 eq. 25]. This is the same rescaling as SNGP with λ in place
of `c`. For convolutions, the norm is estimated by power iteration that applies
the layer's convolution and then its transpose, rather than building the matrix
[arxiv:1804.04368 §3.2] [arxiv:1804.04368 alg. 1]. The paper says only "a small
number" of iterations [arxiv:1804.04368 §4.1]. The estimate is a lower bound on
the true norm, so a layer can end up above its cap [arxiv:1804.04368 §4.1].

On CIFAR-10 with VGG19, this cap rises from 90.43% to 92.57% in the authors'
table. That is the best single-regulariser row for that model, but dropout with
an ℓ∞ cap reaches 92.71%, and on WRN-16-10 the ℓ∞ cap is the best single row
[arxiv:1804.04368 tab. 1]. In the WRN runs the cap also acts on batch-norm
scales with its own λ, so those rows do not isolate the weight cap
[arxiv:1804.04368 §4.2] [arxiv:1804.04368 §5.7].
Card: [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](/papers/1804.04368).

### Generator and discriminator of a latent GAN (RL-GAN-Net)

The latent GAN uses spectral normalization with batch normalization, in both
the generator and the discriminator tables [arxiv:1904.12304 supp. l-GAN
Details]. The GAN is trained with a WGAN-GP adversarial loss, with gradient
penalty weight 10 [arxiv:1904.12304 supp. l-GAN Details]. The card gives no
bound `c`, no power-iteration count and no ablation of spectral normalization
alone, so the setting cannot be transferred as a number. Card: [RL-GAN-Net,
Sarmad et al. 2019](/papers/1904.12304).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Single model must flag unfamiliar inputs and an ensemble is too costly | Bound every hidden matrix, plus a distance-aware output layer (SNGP) | The paper's goal. Full SNGP latency 6.25 ms vs 38.10 ms for a 10-model ensemble on CIFAR-10 [arxiv:2006.10108 tab. 2]. The latency is for the whole model, not for this step alone |
| Residual network where the α ≤ 1 residual-branch argument holds | Bound each hidden matrix at a value you can justify from that argument | This is the only setting with a stated theory [arxiv:2006.10108 §3.2] |
| Convolutional layers | Search over `c`. The paper suggests a grid starting at 0.9, and its own WRN runs use 6.0 | The authors say the conv bound is empirical [arxiv:2006.10108 app. C] [arxiv:2006.10108 §6] |
| Transformer with a pooler | Normalize only the pooler, `c = 0.95` | Scope used for the BERT results [arxiv:2006.10108 app. C] |
| Regulariser for generalisation, and an ℓ2 cap is acceptable | Per-layer ℓ2 cap with λ searched per layer type (Gouk et al.) | Author-reported gain on CIFAR-10 [arxiv:1804.04368 tab. 1]. The useful range of λ is narrow and the optimum differs by dataset [arxiv:1804.04368 §5.7] [arxiv:1804.04368 fig. 2]. The data-scarcity gains are shown for CIFAR-10 only [arxiv:1804.04368 §5.8] |
| Need the cap to hold exactly, not as an estimate | ℓ1 or ℓ∞ cap instead of ℓ2 | These norms are computed exactly. The ℓ2 estimate is a lower bound [arxiv:1804.04368 §4.1] [arxiv:1804.04368 §6] |
| Latent GAN trained with WGAN-GP on latent codes | Spectral norm with batch norm in generator and discriminator, as in the RL-GAN-Net card | The card reports this combination for its GAN, but gives no ablation that isolates spectral norm [arxiv:1904.12304 supp. l-GAN Details] |
| Need a proven distance property at any `c` | Not supported by this card | The bound with `c ≤ 1` is a sufficient condition, and the one-step power iteration makes the bound approximate [arxiv:2006.10108 §3.2] [arxiv:2006.10108 alg. 1] |

## Hyperparameters

Values are from the paper's table of hyperparameters and appendix C
[arxiv:2006.10108 tab. 5] [arxiv:2006.10108 app. C]. The released code's
defaults can differ, see the card's "Paper vs code" section. The Gouk rows are
from the paper text of that card.

| Name | Value | Notes |
|---|---|---|
| Power iterations per step (SNGP) | 1 | Authors call one "sufficient" [arxiv:2006.10108 app. C] |
| Power iterations (Gouk ℓ2 cap) | "a small number", not given | [arxiv:1804.04368 §4.1] |
| Bound `c`, WRN-28-10 | 6.0 | Conv layers [arxiv:2006.10108 tab. 5] |
| Bound `c`, BERT pooler | 0.95 | Pooler only [arxiv:2006.10108 app. C] |
| Cap λ (Gouk ℓ2 cap) | searched per layer type; no values in the text | Separate values for conv, dense and batch norm [arxiv:1804.04368 §5 intro] [arxiv:1804.04368 §5.7] |
| Search grid for `c` | Starts at 0.9 | Recommended in the appendix [arxiv:2006.10108 app. C] |
| GAN normalization | Spectral norm + batch norm | Generator and discriminator; no bound value stated [arxiv:1904.12304 supp. l-GAN Details] |

## Results

The SNGP card gives no results for spectral normalization on its own. The paper
compares two ablations, DNN-SN and DNN-GP, which the card says isolate the two
components [arxiv:2006.10108 §5]. The card records no numbers for either one,
and it does not say which component each ablation keeps. The full-model results
are on the SNGP card.

The Gouk card reports the ℓ2 cap only inside its full regulariser. For WRN
the same runs also cap batch-norm scales, so its numbers are not attributed to
the weight cap alone [arxiv:1804.04368 §4.2] [arxiv:1804.04368 §5.7]. Its CIFAR-10 and CIFAR-100 results are in
[arxiv:1804.04368 tab. 1] and [arxiv:1804.04368 tab. 2].

The RL-GAN-Net card reports no result for spectral normalization. Its numbers
are for the whole completion pipeline [arxiv:1904.12304 tab. 1]
[arxiv:1904.12304 §4.1], so they cannot be attributed to the normalization.

## Implementation sketch

Only the step the SNGP card specifies: the bound applied after each SGD update.
The estimator is one generic power-iteration step; the card does not give its
exact form. The Gouk ℓ2 cap uses the same rescaling rule with λ in place of `c`,
and differs in the number of power iterations and in the cap values
[arxiv:1804.04368 §4]. The GAN variant has no bound step on the card, so it is
not sketched here.

```pseudocode
# Spectral bound on one hidden weight matrix, after an SGD step  [§3.2]
# c is the bound; sigma_est is the spectral-norm estimate from one step
def spectral_bound(W, sigma_est, c):
    # 1. Estimate the largest singular value: one power-iteration step  [alg. 1]
    #    (estimator details not stated on the card)
    # 2. Rescale only when the estimate exceeds the bound  [§3.2]
    if sigma_est > c:
        W = c * W / sigma_est
    return W
```

## Contradictions

- **Theory versus conv bound.** The distance argument needs residual branches
  with α ≤ 1 [arxiv:2006.10108 §3.2]. The convolutional runs use `c = 6.0`,
  which the authors describe as looser than the theory supports
  [arxiv:2006.10108 tab. 5] [arxiv:2006.10108 app. C] [arxiv:2006.10108 §6].
  The card does not resolve this.
- **Starting point of the search.** The appendix recommends a grid that starts
  at 0.9 [arxiv:2006.10108 app. C], while the WRN runs use 6.0. The card does
  not say which search produced 6.0.

## Papers

- [SNGP, Liu et al. 2020](/papers/2006.10108): caps the spectral norm of hidden weights and adds a random-feature GP output layer for single-model uncertainty.
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity, Gouk et al. 2018](/papers/1804.04368): caps each layer's operator norm under ℓ1, ℓ2 or ℓ∞ and projects the weights back after every update, as a regulariser.
- [RL-GAN-Net, Sarmad et al. 2019](/papers/1904.12304): uses spectral norm with batch norm in the generator and discriminator of a latent GAN for point-cloud shape completion.