Layer normalization

Overview

Layer normalization rescales the activations of one layer using statistics computed from that layer's own output, not from a batch. The card in this wiki that uses it applies LN to the output of each temporal fully connected block, sums the result with the block input, and has no other nonlinearity 2207.01567 §3.2 2207.01567 eq. 5. The LN is therefore the only normalising step in the model. The paper's own change is to keep only fully connected layers, layer norm and transposes, with no other activation 2207.01567 §1.

The paper gives the block as a residual update with LN applied to the affine output of the temporal layer 2207.01567 eq. 5. It does not state which axis is normalised, the epsilon, or whether the LN has a learned gain and offset 2207.01567 §3.2. Those details come from the released code, recorded on the card's Back to MLP: A Simple Baseline for Human Motion Prediction page, and they are not part of the paper's method.

Variants

Only one variant is covered by a card so far.

LN over channels inside residual FC blocks (siMLPe)

Each block of the motion predictor applies a fully connected layer across the frames of every channel, then LN, then a residual addition [arxiv:2207.01567 §3.2] 2207.01567 eq. 5. The network has 48 such blocks in the paper's best Human3.6M setting 2207.01567 §4.4 2207.01567 tab. 5. The card notes that the block has no activation function other than the LN 2207.01567 §1 2207.01567 §3.2.

The authors' component ablation removes the LN from the same network. At 1000 ms on Human3.6M the MPJPE rises from 109.4 mm to 127.3 mm with no LN, a single table row on one benchmark 2207.01567 tab. 6. The table does not test a different normalised axis or an LN with no learned gain and offset.

The card is Back to MLP: A Simple Baseline for Human Motion Prediction.

Other normalisation layers, such as the conditional forms that regress scale and shift from an input, are a different design. They are covered on their own pages, for example adaptive layer normalization and batch normalization.

When to use which

The evidence is one paper on human motion prediction, and it does not compare LN with other normalisers. Read the rows below as constraints to check, not as measured rules.

Implementation sketch

Language-agnostic pseudocode of the common core. The affine gain and offset, the epsilon and the normalised axis are the points where variants differ. The block below follows the siMLPe card; the paper leaves those choices open 2207.01567 §3.2.

# Layer norm over the channel axis of one block output  [arxiv:2207.01567 eq. 5]
# Input:  x (B, C, T), the output of the temporal FC
# Output: (B, C, T), normalised per frame over channels

def layer_norm(x, gain, offset, eps):
    # Mean and variance over the normalised axis (channels)  [§3.2]
    mean = x.mean(axis=1, keepdims=True)
    var = x.var(axis=1, keepdims=True)
    # Normalise; eps avoids division by zero  [§3.2 leaves eps open]
    x_hat = (x - mean) / sqrt(var + eps)
    # Learned per-channel gain and offset  [§3.2 leaves these open]
    return gain * x_hat + offset

# One residual block with LN  [arxiv:2207.01567 eq. 5]
# Input:  z (B, C, T)
# Output: z (B, C, T)

def block(z, temporal_fc, gain, offset, eps):
    # Temporal FC across frames, then LN  [eq. 5]
    mixed = layer_norm(temporal_fc(z), gain, offset, eps)
    # Residual addition; no other activation  [§1] [§3.2]
    return z + mixed

Contradictions

None known between papers: only one card covers this concept. Inside that card, the paper and its released code differ on how LN is parameterised. The paper does not state the normalised axis, the epsilon or the affine terms 2207.01567 §3.2 2207.01567 eq. 5. The card's Paper vs code section records the code's choices and also notes that the training script's normalisation flag is off unless passed; the code is not counted as a second paper (Back to MLP).

Papers

• Back to MLP: A Simple Baseline for Human Motion Prediction: uses LN after each temporal FC block in a fully connected motion predictor with no other activation.