Paper · 2022-06-15 · cs.CV
XMorpher: Full Transformer for Deformable Medical Image Registration via Cross Attention
XMorpher is a backbone for deformable medical image registration: two parallel U-shaped transformer branches, one per image, exchange features through cross attention computed in windows of two sizes. Used inside VoxelMorph it reports 83.0% mean DSC on whole-heart CT, against 80.2% for VoxelMorph (author-reported).
Image registrationCross-attentionHierarchical vision transformersmedical-image-registrationdeformable-registrationcross-attentionwindow-attentiontransformer
arXivPDFCodeMarkdown14 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Two medical scans of the same body, taken at different times or from different patients, rarely line up. Deformable registration finds a smooth warp that moves one scan so that its anatomy matches the other, which lets a doctor compare them directly.
Many deep networks for this either feed both scans into one network as a stacked pair, or run two networks on the separate scans and join them only at the end. XMorpher runs two parallel encoder-decoder branches, one per scan, and lets them exchange information at every level through attention. Each scan's features can then look for matching parts in the other scan. The attention is restricted to small local windows, which keeps it cheap. The output is a deformation field, the warp itself. The authors report a higher overlap score (Dice) on whole-heart CT scans than the VoxelMorph network they plug it into.
TL;DR
XMorpher is a transformer backbone for deformable registration of two 3D
medical images §1, §2. Each image gets its own U-shaped branch, and the
branches swap information at every level through cross attention computed in
windows of two sizes §2.1 to §2.3. Plugged into unsupervised VoxelMorph and
semi-supervised PC-Reg, it reports mean DSC of 83.0% and 86.9% on whole-heart
CT, against 80.2% and 86.0% for the baselines tab. 1. These are
author-reported numbers on 210 test pairs. The official repository is
Apache-2.0 licensed but incomplete: the model's forward pass, as cloned at the
commit above, raises a NameError (see Pitfalls).
Problem
Deformable registration must estimate a dense, smooth displacement that aligns a moving image with a fixed one, so that corresponding points coincide §1. The authors argue that earlier deep networks handle the two-image case poorly §1, fig. 1:
- Fusion-first networks concatenate the two images and pass them through one network. Feature extraction and matching are mixed together, which distorts features in mixed regions §1, fig. 1a.
- Fusion-last networks run two single-image networks and fuse their features only at the end, so each branch is biased before any matching happens §1, fig. 1b.
- Transformers used for registration keep single-image self-attention and do not model correspondence between the two images §1.
The method rests on a stated premise: the displacements are local, with no far-apart matches between the moving and fixed images, so attention can be restricted to windows §2.3.
Method
XMorpher is an X-shaped network §2.1. Two parallel U-shaped branches, one per image, exchange information at each level through feature-fusion modules §2, §2.1, fig. 2a. Each fusion module applies CAT blocks: base windows of one branch's tokens form queries, and searching windows of the other branch's tokens form keys and values §2.2. The cross attention is added back to the base tokens, and a 2-layer GELU MLP follows §2.2. Window sizes come in two kinds, base and searching, related by eq. 1 §2.3.
1. Two U-shaped branches §2.1, fig. 2a. The moving and fixed images each go through a U-shaped encoder-decoder with skip connections. The paper replaces the convolutions of a U-Net with CAT blocks §2.1. The two branches exchange information at every level, so the final features represent correspondences between the images §2.1.
2. Fusion module with mutual attention §2.1, fig. 2b. Token sets
T_m (moving) and T_f (fixed) from the two branches pass through two CAT
blocks that share parameters, with the order of inputs swapped. Each output is
returned to its own branch §2.1, fig. 2b. The paper states that this
exchange happens k times per module, but does not give k §2.1.
3. CAT block §2.2, fig. 2c. The tokens are partitioned into base windows
S_ba and searching windows S_se, which have the same count n but different
sizes §2.2, eq. 1. Each base window is projected to queries, and each
searching window to keys and values, through linear layers §2.2. W-MCA
computes the cross attention, which is added to the base window §2.2. A
LayerNorm is applied before each W-MCA and before each MLP §2.2.
4. Multi-size windows §2.3, fig. 2d, eq. 1. Base windows have size
h × w × d. Searching windows are obtained by a sliding window whose stride is
the base window size, and their size is α·h × β·w × γ·d, so each searching
set has n windows of that size §2.3, eq. 1. The paper sets h = w = d = 2
and α = β = γ = 3 §3.1.
5. Window-based multi-head cross attention (W-MCA) §2.3, fig. 2e, eq. 2.
Queries have shape n × s × c, with s = h·w·d tokens per base window. Keys
and values have shape n × μ·s × c, with μ = α·β·γ. The output is a softmax
over scaled dot products, weighted sum of values §2.3, eq. 2. Multi-head
attention is used §2.3. The text does not define d in the scaling factor
or the number of heads.
6. Output. The final representation generates a dense displacement field
φ. The moving image is warped by a spatial transform §2. XMorpher is
trained under two strategies: unsupervised, as the backbone of VoxelMorph
(VM-XMorpher), and semi-supervised, as the backbone of PC-Reg (PC-XMorpher)
§3.1. The paper does not state the loss, optimiser or schedule for either.
Key equations, reproduced from the paper:
h_{ba}, w_{ba}, d_{ba} = h, w, d \qquad h_{se}, w_{se}, d_{se} = \alpha h, \beta w, \gamma d eq. 1
Here S_se has size n × αh × βw × γd, and μ = αβγ §2.3, eq. 1.
\text{W-MCA}(Q_{ba}, K_{se}, V_{se}) = \text{softmax}\left(\frac{Q_{ba} K_{se}^{\top}}{\sqrt{d}}\right) V_{se} eq. 2
Pseudocode, in the paper's order. Lines 1 to 3 restate §2.1 and fig. 2a. The
body of cat_block follows §2.2 and §2.3. The official code is cited where it
resolves something the paper leaves open.
# X-shaped forward pass: two streams exchange features at every level [§2.1]
# Input: moving, fixed: (B, 1, D, H, W) volumes; the code needs B = 1
# Output: displacement field: (B, 3, D, H, W) [code: models/XMorpher.py#L680]
def xmorpher_forward(moving, fixed):
# Patch embedding: a strided 3D convolution [code: models/XMorpher.py#L542]
moving = patch_embed(moving)
fixed = patch_embed(fixed)
skips_moving = []
skips_fixed = []
# Encoder: each stage runs its fusion blocks and keeps a skip [§2.1]
for stage in encoder_stages:
skip_m, skip_f, moving, fixed = stage(moving, fixed)
skips_moving.append(skip_m)
skips_fixed.append(skip_f)
# Decoder: upsample, join the encoder skip, fuse again [§2.1]
for stage_index, stage in enumerate(decoder_stages):
if stage_index > 0:
# Join the skip of the same resolution, then halve the channels
# with one linear layer shared by both streams [code: L715-L718]
skip_index = 3 - stage_index
moving = reduce[stage_index](
concat(moving, skips_moving[skip_index]))
fixed = reduce[stage_index](
concat(fixed, skips_fixed[skip_index]))
_, _, moving, fixed = stage(moving, fixed)
# Join the two streams on the channel axis and normalise [§2.1]
joined = layer_norm(concat(moving, fixed))
# Transposed conv back to image size, then 3 x 3 x 3 conv to 3 channels
# (B, 3, D, H, W) [code: models/XMorpher.py#L725] [code: L734, L740]
flow = flow_head(reverse_patch_embed(joined))
return flow# One CAT block: base tokens are queries, search tokens give keys and values
# Input: base, search: (B, D, H, W, C) token grids
# Output: updated base tokens, same shape [code: models/XMorpher.py#L252-L304]
def cat_block(base, search):
# Normalise only the base tokens before attention [§2.2] [code: L256]
normed = layer_norm_1(base)
# Cut base tokens into windows of size h x w x d [§2.3] [eq. 1]
base_windows = partition_windows(normed, window_size)
# Cut search tokens into 3 x 3 x 3 neighbourhoods of base windows [§2.3]
search_windows = partition_search_windows(search, window_size)
# Cross attention, added back onto the base tokens [§2.2] [eq. 2]
attended = w_mca(base_windows, search_windows)
base = base + merge_windows(attended)
# Two-layer GELU MLP with its own normalisation [§2.2] [code: L250]
base = base + mlp(layer_norm_2(base))
return base# Window-based multi-head cross attention [§2.3] [eq. 2]
# Input: queries (n, s, c); keys and values (n, mu*s, c)
# Output: (n, s, c) [code: models/XMorpher.py#L187-L211]
def w_mca(queries, keys, values):
# Project each stream and split the channels into heads [code: L196-L199]
queries = split_heads(linear_q(queries))
keys = split_heads(linear_k(keys))
values = split_heads(linear_v(values))
# Scaled dot products between queries and keys [eq. 2] [code: L201-L202]
scores = queries @ transpose(keys) * head_dim ** -0.5
# Softmax over the key axis [eq. 2] [code: L204]
weights = softmax(scores, axis=-1)
# Weighted sum of values, heads merged, output projection [code: L208-L209]
out = merge_heads(weights @ values)
return linear_proj(out)Hyperparameters
Values in the first block are stated in the paper. The second block gives the
code's defaults, which the paper does not state. The third block lists what
the paper leaves out. Citations are to the paper unless marked code:.
| Name | Value | Where | Notes |
|---|---|---|---|
base window h, w, d | 2, 2, 2 | §3.1 XMorpher.py | the code passes window_size=(2, 2, 2) |
magnification α, β, γ | 3, 3, 3 | §3.1 eq. 1 | the code hard-codes the factor 3 XMorpher.py |
neighbours μ = αβγ | 27 | §2.3 eq. 2 | derived; the code builds 3 × 3 × 3 base windows |
| tokens per searching window | 216 | derived from eq. 1 | 6 × 6 × 6 at h = w = d = 2 |
| framework | PyTorch | §3.1 | |
| hardware | NVIDIA GeForce RTX 3090, 24 GB | §3.1 | |
| stages (encoder and decoder) | 4 each | XMorpher.py | not stated in paper |
| blocks per stage | 2, 2, 6, 2 | XMorpher.py | not stated; the paper's k (§2.1) is not given |
| embedding dimension | 48 | XMorpher.py | channels double per stage (48, 96, 192, 384) |
| attention heads | 3, 6, 12, 24 | XMorpher.py | head dimension 16 at every stage |
| patch size | 4, 4, 4 | XMorpher.py | not stated |
| MLP ratio | 4 | XMorpher.py | not stated |
| qkv bias | on | XMorpher.py | not stated |
| stochastic depth (drop path) | 0.2 | XMorpher.py | not stated |
| optimiser | Adam | Unsup_train.py Semi_train.py | not stated in paper |
| learning rate | 1e-4 | Unsup_train.py Semi_train.py | not stated in paper |
| epochs × iterations per epoch | 400 × 200 (unsup); 200 × 200 per stage (semi) | Unsup_train.py Semi_train.py | code defaults; not stated in paper |
| batch size | 1 | Unsup_train.py | the window code returns -1 for B > 1 XMorpher.py |
| smoothness loss | L2 on flow gradients, weight 1 | losses.py Unsup_train.py | not stated in paper |
| NCC loss | window 9 × 9 × 9, weight 1 | losses.py Unsup_train.py | not stated in paper |
| semi-supervised label term | weight 0.5 | Semi_train.py Semi_train.py | cross-entropy on segmentation output; not stated in paper |
| intensity | clip to [0, 2048], divide by 2048 | dataloader.py | not stated in paper |
| augmentation | rotation ±π/9 per axis, scale 0.75–1.25, mirroring | Unsup_train.py Unsup_train.py | not stated in paper |
| moving/fixed swap | random, p = 0.5 | dataloader.py | not stated in paper |
| dataset | MM-WHS 2017 CT (20 labelled, 40 unlabelled); ASOCA (60 unlabelled) | §3.1 | the data is not in the repository |
| training pairs | 500 labelled–unlabelled; 9900 unlabelled–unlabelled | §3.1 | counts from 100 unlabelled and 5 labelled images |
| test pairs | 210 from the 15 remaining labelled images | §3.1 | |
| preprocessing | affine transform to common coordinates | §3.1 | tool and parameters not stated; not in the code |
| metrics | mean DSC over all labels; |J(ψ)| ≤ 0 (%) | §3.1 | the Jacobian metric is not defined and not in the code |
Results
Author-reported, on the paper's own split of the MM-WHS 2017 CT and ASOCA data §3.1. The paper reports "±" values without saying whether they are standard deviations or another spread. Bold follows the source's marking, which is per setting tab. 1. The compared baselines are not in the repository, so these numbers have not been reproduced here.
| Method | Setting | Backbone | DSC | |J(ψ)| ≤ 0 (%) | Source |
|---|---|---|---|---|---|
| Affine initialisation | - | - | 69.2 ± 7.2 | - | tab. 1 |
| BSpline | Unsupervised | - | 80.9 ± 7.6 | 5.25 ± 3.27 | tab. 1 |
| VoxelMorph | Unsupervised | CNN | 80.2 ± 5.5 | 4.02 ± 0.82 | tab. 1 |
| TransMorph | Unsupervised | CNN + Transformer | 81.1 ± 5.2 | 3.46 ± 0.75 | tab. 1 |
| No-cross XMorpher (ablation) | Unsupervised | Full Transformer | 81.5 ± 5.4 | 0.94 ± 0.26 | tab. 1 |
| VM-XMorpher | Unsupervised | Full Transformer | 83.0 ± 4.7 | 3.15 ± 0.79 | tab. 1 |
| PC-Reg | Semi-supervised | CNN | 86.0 ± 2.5 | 0.36 ± 0.20 | tab. 1 |
| PC-XMorpher | Semi-supervised | Full Transformer | 86.9 ± 2.4 | 0.32 ± 0.18 | tab. 1 |
Quantitative statements in the text §3.2:
- VM-XMorpher is 2.8 percentage points higher than VoxelMorph on DSC and 0.87 points lower on the Jacobian measure §3.2, tab. 1.
- PC-XMorpher is 0.9 points higher than PC-Reg on DSC and has a lower Jacobian measure §3.2, tab. 1.
- Removing the cross attention (no-cross ablation: the two images are concatenated and self-attended) lowers VM-XMorpher's DSC by 1.5 points [§3.2, tab. 1]. Among the unsupervised rows, the ablation has the lowest Jacobian measure tab. 1.
- Window size: larger base windows give better Jacobian results; smaller ones give better DSC. Shown as a plot without numbers §3.2, fig. 4a.
- Qualitatively, the attention spreads evenly over small windows and concentrates on the heart region §3.2, fig. 4b. Visual results are in fig. 3; no numbers are given.
Paper vs code
Official repository: https://github.com/Solemoon/XMorpher, cloned at commit
926866ef9c0c0c0fa8a0eeedfcd6ee7404c4d841. The repository is linked from the
paper's abstract. Its first author's GitHub account owns it, and the README
attributes it to the authors. The code is Apache-2.0.
The repository contains the PyTorch model (models/XMorpher.py), two training
scripts (Unsup_train.py for VM-XMorpher and Semi_train.py for PC-XMorpher),
losses, a spatial transform, data loaders, and one example .mat file. It does
not contain the MM-WHS or ASOCA data, the baseline implementations, the
Jacobian metric, or trained weights. The MindSpore version is a README stub that
says the release is still to come.
Discrepancies, and places where the code resolves what the paper leaves open:
- Window size and magnification match. The paper sets
h = w = d = 2andα = β = γ = 3§3.1; the code uses(2, 2, 2)XMorpher.py and builds 3 × 3 × 3 neighbourhoods of base windows, which equalsα = 3XMorpher.py. The factor 3 is hard-coded: changingα,βorγin the code is not a parameter change. - "Full transformer" is attention only at the token level. The paper says the CAT blocks replace the convolutions of a U-Net §2.1. The code keeps strided 3D convolutions for patch embedding and merging, transposed convolutions for upsampling and output, and a 3 × 3 × 3 convolution that produces the flow XMorpher.py.
- Number of fusion rounds
k. The paper gives no value §2.1. The code uses a stage depth of 2, 2, 6, 2 XMorpher.py. Each block is applied in both directions with the same parameters XMorpher.py, sokequals the block count of its stage. The paper's two blocks sharing parameters §2.1 agree with this. - Normalisation of the searching tokens. The paper applies LayerNorm before each W-MCA §2.2. The code normalises the base tokens only; the searching tokens enter attention unnormalised XMorpher.py.
- Scaling factor. Eq. 2 divides by
√dwithout definingdeq. 2. The code scales by the per-head dimensionC / heads, which is 16 at every stage XMorpher.py. - Boundary handling. The paper does not say what happens at volume edges. The code pads the grid of windows with zero windows and attends to them, with no mask XMorpher.py.
- Training objective. The paper does not state a loss. For VM-XMorpher the code sums a smoothness loss on the flow and an NCC similarity loss between the warped moving image and the fixed image, each with weight 1 [code: Unsup_train.py#L74-L80] losses.py [code: utils/losses.py#L42-L67].
- Sampling of training pairs. The paper gives 500 and 9900 pairs §3.1. The code draws from the two folders with probability 1/2 each, whatever their sizes Unsup_train.py.
- Semi-supervised strategy. The paper names PC-Reg as the framework in one sentence §3.1. The code runs a two-stage schedule. In stage 1, each iteration updates the U-Net segmenter with cross-entropy where labels exist, then trains the registration head on foreground-masked images taken from the segmenter's own predictions. Stage 2 alternates the two networks, with the segmenter loss weighted 0.5 on the warped-label term [code: Semi_train.py#L79-L226] Semi_train.py.
- Mean DSC. The paper says "all labels" §3.1. The code averages over classes 1 to 7 and so excludes background Unsup_train.py.
- Data. The code maps the label values 205 to 850 to classes 1 to 7 [code: utils/dataloader.py#L51-L57] and clips intensities to [0, 2048] before dividing by 2048 dataloader.py. Neither is in the paper.
No discrepancies were found in the stated h, α, β or γ values, the
eq. 2 structure, or the 2-layer GELU MLP.
Pitfalls
- The released forward pass does not run.
BasicLayer.forwardassignsx = next_aXMorpher.py, andBasicLayerUp.forwarddoes the same XMorpher.py.next_ais never defined. The variables assigned just above arenext_xandnext_xa, so the intent is clear, but as released the first call raisesNameError. This was found by reading; the code was not executed here. - The searching-window indices are wrong. In
window_area_partition, index 21 is written twice XMorpher.py, and index 20 is never written, so it stays zero XMorpher.py. One of the 27 neighbours is therefore missing and another is duplicated. The paper's numbers depend on this code path; whether the reported results used this exact version is not stated. - Zero padding enters attention. At the volume edge, the padded neighbour windows are zeros, and the attention reads them with no mask [code: models/XMorpher.py#L77-L80]. Their keys and values are the projections of zero, so they are not empty. The paper is silent on this.
- Batch size must be 1.
window_area_partitionprintswrongand returns-1for B > 1 XMorpher.py. The training scripts use batch size 1 Unsup_train.py. - CUDA is hard-coded. Window code moves tensors to the GPU [code:
models/XMorpher.py#L80], the spatial transform builds its grid on the GPU [code:
utils/STN.py#L17], and NCC allocates its filter on
"cuda"[code: utils/losses.py#L49]. CPU use needs changes. - Argument order in
Head.Head.forwardcallsself.swin(fixed, moving)XMorpher.py, so themovingparameter of the backbone receives the fixed image. The attention is asymmetric in its queries, so this order is part of the reproduced model. - Decoder resize swaps height and width. When a decoder feature does not
match its encoder skip, the code interpolates to
(D, W, H)after unpacking the shape as(B, D, W, H, C)XMorpher.py. That is wrong when the input is not cubic. The paper does not say whether this case arises for its data. - The inverse flow is an approximation. The reverse transform negates the flow sampled at its own displaced positions STN.py. This is only used for the reverse direction, which the training loss does not use.
- The paper overstates one ranking. It says VM-XMorpher is "top-ranked" on the Jacobian measure §3.2. In tab. 1 the no-cross ablation has 0.94, lower than VM-XMorpher's 3.15. Among the rows without the ablation, VM-XMorpher is best at 3.15, ahead of TransMorph's 3.46 but behind the ablation.
- The Jacobian measure is undefined and unreproducible from the repository.
The paper gives
|J(ψ)| ≤ 0 (%)without saying what the percentage is taken over. There is no Jacobian code in the repository. - Data and weights are missing. The 210 test pairs, the MM-WHS and ASOCA
files, the affine preprocessing, and any checkpoints are not in the repository.
The repository contains only
data/one_example/0_1.mat. The loader reads labels only whenlen(data) == 6dataloader.py [code: utils/dataloader.py#L63]. Counting the three header keys thatloadmatreturns, this condition depends on the file's layout. The sample file was not inspected, so check label loading before relying on it.
Lineage
- Builds on: the Swin Transformer [liu2021swin], cited by the paper among single-image transformers §1; its card is at Swin Transformer. The code's block docstring calls it a "Swin Transformer Block" [code: models/XMorpher.py#L215]. The paper does not say which work it extends.
- Framework: VoxelMorph [balakrishnan2018unsupervised] for the unsupervised setting §3.1, and PC-Reg [he2021few] for the semi-supervised setting §3.1. Neither has a card here.
- Compared against: BSpline [rueckert1999nonrigid], VoxelMorph, PC-Reg, and TransMorph (arXiv:2111.10480, no card here) §3.1, tab. 1.
- Superseded by: not stated. The code README names a MindSpore version that is not yet released README.md.
Source and license
Adapted from “XMorpher: Full Transformer for Deformable Medical Image Registration via Cross Attention” by Jiacheng Shi, Yuting He, Youyong Kong, Jean-Louis Coatrieux, Huazhong Shu, Guanyu Yang, Shuo Li (arXiv:2206.07349), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Solemoon/XMorpher at commit 926866ef9c0c (Apache-2.0). Any quoted code is shown under that license.