Adaptive layer normalization (AdaLN)

Overview

Adaptive layer normalization (AdaLN) is a layer norm whose gain and offset are not fixed learned vectors. They are computed from a conditioning input, so the same block behaves differently for each condition. The card in this wiki that uses it applies the modulation (γ + 1) · x + β, with γ and β regressed from the sum of two embeddings: the diffusion step and the observation 2409.14411 §3.2 eq. 2. The conditioning therefore enters through the normalisation, not through an extra sequence of tokens the attention reads.

The paper motivates this with the parameter normalisation used in image generation 2409.14411 §1 2409.14411 §3.2. This page does not restate that lineage from other papers, because no other card in the wiki covers it.

Variants

Only one variant is covered by a card so far.

AdaLN in place of cross-attention (ScaleDP)

Each standard layer norm in the transformer action denoiser is replaced by the modulated form above. The earlier transformer baseline (DP-T) instead concatenated the diffusion-step and observation embeddings into a sequence and attended to them with cross-attention, beside the action tokens 2409.14411 §3.2. In the ScaleDP variant the attention sequence is the action chunk alone 2409.14411 §3.2. A final AdaLN and a linear layer decode the action tokens into predicted noise 2409.14411 §3.2. The card is Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation.

The same change is paired with removing the causal mask from action self-attention 2409.14411 §3.2. Both changes are made together in the headline model, so their effects are not separated in the card's main results.

When to use which

The card's evidence is one paper on visuomotor control. Read the rows below as design constraints to test, not as measured rules.

The card reports ScaleDP-H at 87.50% on four Franka tasks against 51.25% for DP-T, and 98.33% against 23.33% on three bimanual UR5 tasks [arxiv:2409.14411 tab. 2] 2409.14411 tab. 3. These are author-reported, with 20 trials per task, and the comparison also changes the model size and the mask. Do not attribute the gain to AdaLN alone.

Implementation sketch

Own-words core of the ScaleDP block, from the card's description. Shapes are given in comments. The card does not state how γ and β are regressed, so the regression below is a placeholder, not the paper's network.

# Modulated layer norm for one sub-layer  [arxiv:2409.14411 §3.2] [eq. 2]
# x:    (B, C, d) action tokens
# cond: (B, d)    sum of diffusion-step and observation embeddings

def adaln(x, cond, regress):
    # Scale and shift from the condition; regress is not stated
    gamma, beta = regress(cond)           # (B, d) each
    # Normalise over the feature axis, then modulate  [eq. 2]
    return (gamma[:, None] + 1) * layer_norm(x) + beta[:, None]

# One transformer block with AdaLN and no causal mask  [arxiv:2409.14411 §3.2]
# Input:  x (B, C, d) action tokens, cond (B, d)

def block(x, cond):
    # Self-attention over the whole chunk, no causal mask  [arxiv:2409.14411 §3.2]
    x = x + self_attention(adaln(x, cond, regress_a), causal=False)
    # Feed-forward sub-layer; the card does not say whether it is modulated
    x = x + mlp(adaln(x, cond, regress_b))
    return x

Contradictions

None known. Only one card covers this technique. Its abstract describes the observation feature as factorised into several affine layers, while the body describes regressing one scale and shift from summed embeddings 2409.14411 §3.2. Implement the body's form, as the card advises.

Papers

• Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation: replaces cross-attention observation conditioning with AdaLN and removes the causal mask, then scales the transformer policy to 1 billion parameters.