---
kind: concept
slug: layer-normalization
title: Layer normalization
summary: >-
  Layer normalization (LN) rescales a layer's activations with statistics computed inside the layer, then applies a learned gain and offset. Only one card covers it: a fully connected motion predictor (siMLPe) that uses LN after each temporal FC block and no other activation. The paper leaves the normalised axis, epsilon and affine terms unstated, so a reimplementation must choose them.
tags: [normalization, mlp, motion-prediction]
papers: ["2207.01567"]
related: [adaptive-layer-normalization, batch-normalization, multi-layer-perceptron, motion-prediction, discrete-cosine-transform]
status: draft
updated_at: 2026-11-09
---

## Overview

Layer normalization rescales the activations of one layer using statistics
computed from that layer's own output, not from a batch. The card in this
wiki that uses it applies LN to the output of each temporal fully connected
block, sums the result with the block input, and has no other nonlinearity
[arxiv:2207.01567 §3.2] [arxiv:2207.01567 eq. 5]. The LN is therefore the
only normalising step in the model. The paper's own change is to keep only
fully connected layers, layer norm and transposes, with no other activation
[arxiv:2207.01567 §1].

The paper gives the block as a residual update with LN applied to the affine
output of the temporal layer [arxiv:2207.01567 eq. 5]. It does not state which
axis is normalised, the epsilon, or whether the LN has a learned gain and
offset [arxiv:2207.01567 §3.2]. Those details come from the released code,
recorded on the card's [Back to MLP: A Simple Baseline for Human Motion Prediction](/papers/2207.01567) page, and they are not part of the paper's method.

## Variants

Only one variant is covered by a card so far.

### LN over channels inside residual FC blocks (siMLPe)

Each block of the motion predictor applies a fully connected layer across the
frames of every channel, then LN, then a residual addition [arxiv:2207.01567
§3.2] [arxiv:2207.01567 eq. 5]. The network has 48 such blocks in the paper's
best Human3.6M setting [arxiv:2207.01567 §4.4] [arxiv:2207.01567 tab. 5]. The
card notes that the block has no activation function other than the LN
[arxiv:2207.01567 §1] [arxiv:2207.01567 §3.2].

The authors' component ablation removes the LN from the same network. At 1000 ms
on Human3.6M the MPJPE rises from 109.4 mm to 127.3 mm with no LN, a single
table row on one benchmark [arxiv:2207.01567 tab. 6]. The table does not test
a different normalised axis or an LN with no learned gain and offset.

The card is [Back to MLP: A Simple Baseline for Human Motion Prediction](/papers/2207.01567).

Other normalisation layers, such as the conditional forms that regress scale
and shift from an input, are a different design. They are covered on their own
pages, for example [adaptive layer normalization](/concepts/adaptive-layer-normalization)
and [batch normalization](/concepts/batch-normalization).

## When to use which

The evidence is one paper on human motion prediction, and it does not compare
LN with other normalisers. Read the rows below as constraints to check, not as
measured rules.

| Situation | Use | Why |
|---|---|---|
| You want to reproduce the siMLPe baseline | LN over channels, with the code's epsilon and affine terms | The paper does not specify the axis, epsilon or affine terms; the released code normalises over channels per frame [arxiv:2207.01567 §3.2] |
| You are building a fully connected motion predictor of this shape | LN, as in siMLPe | Removing LN worsens the authors' 1000 ms error from 109.4 to 127.3 mm on Human3.6M [arxiv:2207.01567 tab. 6] |
| You need the normalisation to depend on a condition | Adaptive layer normalization | LN here has no conditioning input; see [adaptive layer normalization](/concepts/adaptive-layer-normalization) |
| You need to know whether LN helps on your task | Run your own ablation | The one ablation is on Human3.6M with this network only [arxiv:2207.01567 tab. 6] |

## Implementation sketch

Language-agnostic pseudocode of the common core. The affine gain and offset,
the epsilon and the normalised axis are the points where variants differ.
The block below follows the siMLPe card; the paper leaves those choices open
[arxiv:2207.01567 §3.2].

```pseudocode
# Layer norm over the channel axis of one block output  [arxiv:2207.01567 eq. 5]
# Input:  x (B, C, T), the output of the temporal FC
# Output: (B, C, T), normalised per frame over channels

def layer_norm(x, gain, offset, eps):
    # Mean and variance over the normalised axis (channels)  [§3.2]
    mean = x.mean(axis=1, keepdims=True)
    var = x.var(axis=1, keepdims=True)
    # Normalise; eps avoids division by zero  [§3.2 leaves eps open]
    x_hat = (x - mean) / sqrt(var + eps)
    # Learned per-channel gain and offset  [§3.2 leaves these open]
    return gain * x_hat + offset
```

```pseudocode
# One residual block with LN  [arxiv:2207.01567 eq. 5]
# Input:  z (B, C, T)
# Output: z (B, C, T)

def block(z, temporal_fc, gain, offset, eps):
    # Temporal FC across frames, then LN  [eq. 5]
    mixed = layer_norm(temporal_fc(z), gain, offset, eps)
    # Residual addition; no other activation  [§1] [§3.2]
    return z + mixed
```

## Contradictions

None known between papers: only one card covers this concept. Inside that
card, the paper and its released code differ on how LN is parameterised. The
paper does not state the normalised axis, the epsilon or the affine terms
[arxiv:2207.01567 §3.2] [arxiv:2207.01567 eq. 5]. The card's Paper vs code
section records the code's choices and also notes that the training script's
normalisation flag is off unless passed; the code is not counted as a second
paper ([Back to MLP](/papers/2207.01567)).

## Papers

- [Back to MLP: A Simple Baseline for Human Motion Prediction](/papers/2207.01567): uses LN after each temporal FC block in a fully connected motion predictor with no other activation.