Spectral normalization

Overview

Spectral normalization rescales a weight matrix after an update so that its spectral norm, the largest singular value, does not exceed a bound c. The purpose is to limit how far a layer can stretch distances between its inputs. In SNGP the argument runs as follows: if every residual branch is α-Lipschitz with α ≤ 1, the hidden map is bi-Lipschitz, so inputs that are far apart stay far apart after the hidden layers 2006.10108 §3.2. The card treats this distance preservation as the precondition for a distance-aware output layer 2006.10108 §2 2006.10108 §3.

The card cites the origin of the residual-network variant as Miyato et al. (2018), together with Lipschitz analyses by Bartlett et al. (2018) and Behrmann et al. (2019). None of these has a card in this wiki 2006.10108 §3.2.

Gouk et al. use the same rescaling rule, with the bound called λ, as a regulariser for generalisation rather than for uncertainty. They also compare caps under the ℓ1 and ℓ∞ norms, which are exact to compute where the ℓ2 cap is an estimate 1804.04368 §3.1 1804.04368 §4.1. The general method is on the Lipschitz constraint page; this page covers the spectral (ℓ2) case.

In a GAN the same operation serves a different purpose. The RL-GAN-Net card uses spectral normalization, together with batch normalization, in the generator and the discriminator of its latent GAN, and states no bound value for it 1904.12304 supp. l-GAN Details.

Variants

Every hidden weight matrix, one power iteration per step (SNGP)

After every SGD step, the method estimates the spectral norm of each hidden weight matrix W_l with one power-iteration step. It rescales the matrix only when the estimate exceeds c 2006.10108 §3.2 2006.10108 alg. 1. Card: SNGP, Liu et al. 2020.

The paper's residual-network setting uses a bound of 6.0 for the convolutional layers. The authors state that this is looser than the residual-branch theory supports 2006.10108 tab. 5 2006.10108 app. C.

Pooler layer only (BERT)

For BERT-Base, only the pooler dense layer is normalized, with c = 0.95. The hidden transformer layers are not normalized 2006.10108 app. C.

Per-layer ℓ2 cap as a regulariser (Gouk et al.)

Each layer gets its own cap λ. After every optimiser step, a weight tensor whose norm exceeds λ is divided by max(1, norm/λ) 1804.04368 §4 1804.04368 eq. 25. This is the same rescaling as SNGP with λ in place of c. For convolutions, the norm is estimated by power iteration that applies the layer's convolution and then its transpose, rather than building the matrix 1804.04368 §3.2 1804.04368 alg. 1. The paper says only "a small number" of iterations 1804.04368 §4.1. The estimate is a lower bound on the true norm, so a layer can end up above its cap 1804.04368 §4.1.

On CIFAR-10 with VGG19, this cap rises from 90.43% to 92.57% in the authors' table. That is the best single-regulariser row for that model, but dropout with an ℓ∞ cap reaches 92.71%, and on WRN-16-10 the ℓ∞ cap is the best single row 1804.04368 tab. 1. In the WRN runs the cap also acts on batch-norm scales with its own λ, so those rows do not isolate the weight cap 1804.04368 §4.2 1804.04368 §5.7. Card: Regularisation of Neural Networks by Enforcing Lipschitz Continuity.

Generator and discriminator of a latent GAN (RL-GAN-Net)

The latent GAN uses spectral normalization with batch normalization, in both the generator and the discriminator tables [arxiv:1904.12304 supp. l-GAN Details]. The GAN is trained with a WGAN-GP adversarial loss, with gradient penalty weight 10 1904.12304 supp. l-GAN Details. The card gives no bound c, no power-iteration count and no ablation of spectral normalization alone, so the setting cannot be transferred as a number. Card: RL-GAN-Net, Sarmad et al. 2019.

When to use which

Hyperparameters

Values are from the paper's table of hyperparameters and appendix C 2006.10108 tab. 5 2006.10108 app. C. The released code's defaults can differ, see the card's "Paper vs code" section. The Gouk rows are from the paper text of that card.

Results

The SNGP card gives no results for spectral normalization on its own. The paper compares two ablations, DNN-SN and DNN-GP, which the card says isolate the two components 2006.10108 §5. The card records no numbers for either one, and it does not say which component each ablation keeps. The full-model results are on the SNGP card.

The Gouk card reports the ℓ2 cap only inside its full regulariser. For WRN the same runs also cap batch-norm scales, so its numbers are not attributed to the weight cap alone 1804.04368 §4.2 1804.04368 §5.7. Its CIFAR-10 and CIFAR-100 results are in 1804.04368 tab. 1 and 1804.04368 tab. 2.

The RL-GAN-Net card reports no result for spectral normalization. Its numbers are for the whole completion pipeline 1904.12304 tab. 1 1904.12304 §4.1, so they cannot be attributed to the normalization.

Implementation sketch

Only the step the SNGP card specifies: the bound applied after each SGD update. The estimator is one generic power-iteration step; the card does not give its exact form. The Gouk ℓ2 cap uses the same rescaling rule with λ in place of c, and differs in the number of power iterations and in the cap values 1804.04368 §4. The GAN variant has no bound step on the card, so it is not sketched here.

# Spectral bound on one hidden weight matrix, after an SGD step  [§3.2]
# c is the bound; sigma_est is the spectral-norm estimate from one step
def spectral_bound(W, sigma_est, c):
    # 1. Estimate the largest singular value: one power-iteration step  [alg. 1]
    #    (estimator details not stated on the card)
    # 2. Rescale only when the estimate exceeds the bound  [§3.2]
    if sigma_est > c:
        W = c * W / sigma_est
    return W

Contradictions

• Theory versus conv bound. The distance argument needs residual branches with α ≤ 1 2006.10108 §3.2. The convolutional runs use c = 6.0, which the authors describe as looser than the theory supports 2006.10108 tab. 5 2006.10108 app. C 2006.10108 §6. The card does not resolve this.
• Starting point of the search. The appendix recommends a grid that starts at 0.9 2006.10108 app. C, while the WRN runs use 6.0. The card does not say which search produced 6.0.

Papers

• SNGP, Liu et al. 2020: caps the spectral norm of hidden weights and adds a random-feature GP output layer for single-model uncertainty.
• Regularisation of Neural Networks by Enforcing Lipschitz Continuity, Gouk et al. 2018: caps each layer's operator norm under ℓ1, ℓ2 or ℓ∞ and projects the weights back after every update, as a regulariser.
• RL-GAN-Net, Sarmad et al. 2019: uses spectral norm with batch norm in the generator and discriminator of a latent GAN for point-cloud shape completion.