---
kind: concept
slug: adaptive-layer-normalization
title: Adaptive layer normalization (AdaLN)
summary: >-
  AdaLN makes a layer norm's scale and shift depend on a conditioning vector, so a transformer can be steered without a cross-attention path. One card uses it in a diffusion policy (ScaleDP), which regresses the scale and shift from the diffusion step and observation and drops the causal mask. Reported gains come from both changes together; no ablation isolates AdaLN.
tags: [normalization, conditioning, transformer, diffusion-policy, imitation-learning, scaling]
papers: ["2409.14411"]
related: [adaptive-instance-normalization, conditional-batch-normalization, diffusion-models, imitation-learning, neural-scaling-laws]
status: draft
updated_at: 2026-10-25
---

## Overview

Adaptive layer normalization (AdaLN) is a layer norm whose gain and offset are
not fixed learned vectors. They are computed from a conditioning input, so the
same block behaves differently for each condition. The card in this wiki that
uses it applies the modulation (γ + 1) · x + β, with γ and β regressed from
the sum of two embeddings: the diffusion step and the observation
[arxiv:2409.14411 §3.2] [eq. 2]. The conditioning therefore enters through the
normalisation, not through an extra sequence of tokens the attention reads.

The paper motivates this with the parameter normalisation used in image
generation [arxiv:2409.14411 §1] [arxiv:2409.14411 §3.2]. This page does not
restate that lineage from other papers, because no other card in the wiki
covers it.

## Variants

Only one variant is covered by a card so far.

### AdaLN in place of cross-attention (ScaleDP)

Each standard layer norm in the transformer action denoiser is replaced by the
modulated form above. The earlier transformer baseline (DP-T) instead
concatenated the diffusion-step and observation embeddings into a sequence and
attended to them with cross-attention, beside the action tokens
[arxiv:2409.14411 §3.2]. In the ScaleDP variant the attention sequence is the
action chunk alone [arxiv:2409.14411 §3.2]. A final AdaLN and a linear layer
decode the action tokens into predicted noise [arxiv:2409.14411 §3.2]. The
card is [Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation](/papers/2409.14411).

The same change is paired with removing the causal mask from action
self-attention [arxiv:2409.14411 §3.2]. Both changes are made together in the
headline model, so their effects are not separated in the card's main results.

## When to use which

The card's evidence is one paper on visuomotor control. Read the rows below as
design constraints to test, not as measured rules.

| Situation | Use | Why |
|---|---|---|
| Conditioning is one vector per step (a diffusion step plus a pooled observation) | AdaLN | The paper regresses scale and shift from a summed vector and gets rid of the cross-attention path [arxiv:2409.14411 §3.2] |
| Conditioning is a set of tokens that must be attended to individually | Cross-attention, or keep the tokens in the sequence | AdaLN takes one vector per block; the card does not test per-token conditioning [arxiv:2409.14411 §3.2] |
| You want a shorter attention sequence | AdaLN | With AdaLN the attention runs over the action chunk alone [arxiv:2409.14411 §3.2] |
| You need the component-level effect of AdaLN | Run your own ablation | The non-causal ablation is the only isolated test in the card, and it changes the mask, not the normalisation [arxiv:2409.14411 tab. 4] |
| Scaling depth or width of a transformer policy | AdaLN with the paper's scaling recipe, as a starting point | The card reports that the baseline stops improving with size and that ScaleDP keeps improving up to 1 billion parameters, on its own tasks [arxiv:2409.14411 §4.2] [arxiv:2409.14411 tab. 2] |

The card reports ScaleDP-H at 87.50% on four Franka tasks against 51.25% for
DP-T, and 98.33% against 23.33% on three bimanual UR5 tasks [arxiv:2409.14411
tab. 2] [arxiv:2409.14411 tab. 3]. These are author-reported, with 20 trials per
task, and the comparison also changes the model size and the mask. Do not
attribute the gain to AdaLN alone.

## Implementation sketch

Own-words core of the ScaleDP block, from the card's description. Shapes are
given in comments. The card does not state how γ and β are regressed, so the
regression below is a placeholder, not the paper's network.

```pseudocode
# Modulated layer norm for one sub-layer  [arxiv:2409.14411 §3.2] [eq. 2]
# x:    (B, C, d) action tokens
# cond: (B, d)    sum of diffusion-step and observation embeddings

def adaln(x, cond, regress):
    # Scale and shift from the condition; regress is not stated
    gamma, beta = regress(cond)           # (B, d) each
    # Normalise over the feature axis, then modulate  [eq. 2]
    return (gamma[:, None] + 1) * layer_norm(x) + beta[:, None]
```

```pseudocode
# One transformer block with AdaLN and no causal mask  [arxiv:2409.14411 §3.2]
# Input:  x (B, C, d) action tokens, cond (B, d)

def block(x, cond):
    # Self-attention over the whole chunk, no causal mask  [arxiv:2409.14411 §3.2]
    x = x + self_attention(adaln(x, cond, regress_a), causal=False)
    # Feed-forward sub-layer; the card does not say whether it is modulated
    x = x + mlp(adaln(x, cond, regress_b))
    return x
```

## Contradictions

None known. Only one card covers this technique. Its abstract describes the
observation feature as factorised into several affine layers, while the body
describes regressing one scale and shift from summed embeddings
[arxiv:2409.14411 §3.2]. Implement the body's form, as the card advises.

## Papers

- [Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulation](/papers/2409.14411): replaces cross-attention observation conditioning with AdaLN and removes the causal mask, then scales the transformer policy to 1 billion parameters.