wikillm

Paper · 2022-06-15 · cs.CV

XMorpher: Full Transformer for Deformable Medical Image Registration via Cross Attention

XMorpher is a backbone for deformable medical image registration: two parallel U-shaped transformer branches, one per image, exchange features through cross attention computed in windows of two sizes. Used inside VoxelMorph it reports 83.0% mean DSC on whole-heart CT, against 80.2% for VoxelMorph (author-reported).

Image registrationCross-attentionHierarchical vision transformersmedical-image-registrationdeformable-registrationcross-attentionwindow-attentiontransformer

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Two medical scans of the same body, taken at different times or from different patients, rarely line up. Deformable registration finds a smooth warp that moves one scan so that its anatomy matches the other, which lets a doctor compare them directly.

Many deep networks for this either feed both scans into one network as a stacked pair, or run two networks on the separate scans and join them only at the end. XMorpher runs two parallel encoder-decoder branches, one per scan, and lets them exchange information at every level through attention. Each scan's features can then look for matching parts in the other scan. The attention is restricted to small local windows, which keeps it cheap. The output is a deformation field, the warp itself. The authors report a higher overlap score (Dice) on whole-heart CT scans than the VoxelMorph network they plug it into.

TL;DR

XMorpher is a transformer backbone for deformable registration of two 3D medical images §1, §2. Each image gets its own U-shaped branch, and the branches swap information at every level through cross attention computed in windows of two sizes §2.1 to §2.3. Plugged into unsupervised VoxelMorph and semi-supervised PC-Reg, it reports mean DSC of 83.0% and 86.9% on whole-heart CT, against 80.2% and 86.0% for the baselines tab. 1. These are author-reported numbers on 210 test pairs. The official repository is Apache-2.0 licensed but incomplete: the model's forward pass, as cloned at the commit above, raises a NameError (see Pitfalls).

Problem

Deformable registration must estimate a dense, smooth displacement that aligns a moving image with a fixed one, so that corresponding points coincide §1. The authors argue that earlier deep networks handle the two-image case poorly §1, fig. 1:

  • Fusion-first networks concatenate the two images and pass them through one network. Feature extraction and matching are mixed together, which distorts features in mixed regions §1, fig. 1a.
  • Fusion-last networks run two single-image networks and fuse their features only at the end, so each branch is biased before any matching happens §1, fig. 1b.
  • Transformers used for registration keep single-image self-attention and do not model correspondence between the two images §1.

The method rests on a stated premise: the displacements are local, with no far-apart matches between the moving and fixed images, so attention can be restricted to windows §2.3.

Method

XMorpher is an X-shaped network §2.1. Two parallel U-shaped branches, one per image, exchange information at each level through feature-fusion modules §2, §2.1, fig. 2a. Each fusion module applies CAT blocks: base windows of one branch's tokens form queries, and searching windows of the other branch's tokens form keys and values §2.2. The cross attention is added back to the base tokens, and a 2-layer GELU MLP follows §2.2. Window sizes come in two kinds, base and searching, related by eq. 1 §2.3.

1. Two U-shaped branches §2.1, fig. 2a. The moving and fixed images each go through a U-shaped encoder-decoder with skip connections. The paper replaces the convolutions of a U-Net with CAT blocks §2.1. The two branches exchange information at every level, so the final features represent correspondences between the images §2.1.

2. Fusion module with mutual attention §2.1, fig. 2b. Token sets T_m (moving) and T_f (fixed) from the two branches pass through two CAT blocks that share parameters, with the order of inputs swapped. Each output is returned to its own branch §2.1, fig. 2b. The paper states that this exchange happens k times per module, but does not give k §2.1.

3. CAT block §2.2, fig. 2c. The tokens are partitioned into base windows S_ba and searching windows S_se, which have the same count n but different sizes §2.2, eq. 1. Each base window is projected to queries, and each searching window to keys and values, through linear layers §2.2. W-MCA computes the cross attention, which is added to the base window §2.2. A LayerNorm is applied before each W-MCA and before each MLP §2.2.

4. Multi-size windows §2.3, fig. 2d, eq. 1. Base windows have size h × w × d. Searching windows are obtained by a sliding window whose stride is the base window size, and their size is α·h × β·w × γ·d, so each searching set has n windows of that size §2.3, eq. 1. The paper sets h = w = d = 2 and α = β = γ = 3 §3.1.

5. Window-based multi-head cross attention (W-MCA) §2.3, fig. 2e, eq. 2. Queries have shape n × s × c, with s = h·w·d tokens per base window. Keys and values have shape n × μ·s × c, with μ = α·β·γ. The output is a softmax over scaled dot products, weighted sum of values §2.3, eq. 2. Multi-head attention is used §2.3. The text does not define d in the scaling factor or the number of heads.

6. Output. The final representation generates a dense displacement field φ. The moving image is warped by a spatial transform §2. XMorpher is trained under two strategies: unsupervised, as the backbone of VoxelMorph (VM-XMorpher), and semi-supervised, as the backbone of PC-Reg (PC-XMorpher) §3.1. The paper does not state the loss, optimiser or schedule for either.

Key equations, reproduced from the paper:

h_{ba}, w_{ba}, d_{ba} = h, w, d \qquad h_{se}, w_{se}, d_{se} = \alpha h, \beta w, \gamma d eq. 1

Here S_se has size n × αh × βw × γd, and μ = αβγ §2.3, eq. 1.

\text{W-MCA}(Q_{ba}, K_{se}, V_{se}) = \text{softmax}\left(\frac{Q_{ba} K_{se}^{\top}}{\sqrt{d}}\right) V_{se} eq. 2

Pseudocode, in the paper's order. Lines 1 to 3 restate §2.1 and fig. 2a. The body of cat_block follows §2.2 and §2.3. The official code is cited where it resolves something the paper leaves open.

Pseudocode
# X-shaped forward pass: two streams exchange features at every level  [§2.1]
# Input:  moving, fixed: (B, 1, D, H, W) volumes; the code needs B = 1
# Output: displacement field: (B, 3, D, H, W)  [code: models/XMorpher.py#L680]

def xmorpher_forward(moving, fixed):
    # Patch embedding: a strided 3D convolution  [code: models/XMorpher.py#L542]
    moving = patch_embed(moving)
    fixed = patch_embed(fixed)
    skips_moving = []
    skips_fixed = []
    # Encoder: each stage runs its fusion blocks and keeps a skip  [§2.1]
    for stage in encoder_stages:
        skip_m, skip_f, moving, fixed = stage(moving, fixed)
        skips_moving.append(skip_m)
        skips_fixed.append(skip_f)
    # Decoder: upsample, join the encoder skip, fuse again  [§2.1]
    for stage_index, stage in enumerate(decoder_stages):
        if stage_index > 0:
            # Join the skip of the same resolution, then halve the channels
            # with one linear layer shared by both streams  [code: L715-L718]
            skip_index = 3 - stage_index
            moving = reduce[stage_index](
                concat(moving, skips_moving[skip_index]))
            fixed = reduce[stage_index](
                concat(fixed, skips_fixed[skip_index]))
        _, _, moving, fixed = stage(moving, fixed)
    # Join the two streams on the channel axis and normalise  [§2.1]
    joined = layer_norm(concat(moving, fixed))
    # Transposed conv back to image size, then 3 x 3 x 3 conv to 3 channels
    # (B, 3, D, H, W)  [code: models/XMorpher.py#L725] [code: L734, L740]
    flow = flow_head(reverse_patch_embed(joined))
    return flow
Pseudocode
# One CAT block: base tokens are queries, search tokens give keys and values
# Input:  base, search: (B, D, H, W, C) token grids
# Output: updated base tokens, same shape  [code: models/XMorpher.py#L252-L304]

def cat_block(base, search):
    # Normalise only the base tokens before attention  [§2.2] [code: L256]
    normed = layer_norm_1(base)
    # Cut base tokens into windows of size h x w x d  [§2.3] [eq. 1]
    base_windows = partition_windows(normed, window_size)
    # Cut search tokens into 3 x 3 x 3 neighbourhoods of base windows  [§2.3]
    search_windows = partition_search_windows(search, window_size)
    # Cross attention, added back onto the base tokens  [§2.2] [eq. 2]
    attended = w_mca(base_windows, search_windows)
    base = base + merge_windows(attended)
    # Two-layer GELU MLP with its own normalisation  [§2.2] [code: L250]
    base = base + mlp(layer_norm_2(base))
    return base
Pseudocode
# Window-based multi-head cross attention  [§2.3] [eq. 2]
# Input:  queries (n, s, c); keys and values (n, mu*s, c)
# Output: (n, s, c)  [code: models/XMorpher.py#L187-L211]

def w_mca(queries, keys, values):
    # Project each stream and split the channels into heads  [code: L196-L199]
    queries = split_heads(linear_q(queries))
    keys = split_heads(linear_k(keys))
    values = split_heads(linear_v(values))
    # Scaled dot products between queries and keys  [eq. 2] [code: L201-L202]
    scores = queries @ transpose(keys) * head_dim ** -0.5
    # Softmax over the key axis  [eq. 2] [code: L204]
    weights = softmax(scores, axis=-1)
    # Weighted sum of values, heads merged, output projection  [code: L208-L209]
    out = merge_heads(weights @ values)
    return linear_proj(out)

Hyperparameters

Values in the first block are stated in the paper. The second block gives the code's defaults, which the paper does not state. The third block lists what the paper leaves out. Citations are to the paper unless marked code:.

NameValueWhereNotes
base window h, w, d2, 2, 2§3.1 XMorpher.pythe code passes window_size=(2, 2, 2)
magnification α, β, γ3, 3, 3§3.1 eq. 1the code hard-codes the factor 3 XMorpher.py
neighbours μ = αβγ27§2.3 eq. 2derived; the code builds 3 × 3 × 3 base windows
tokens per searching window216derived from eq. 16 × 6 × 6 at h = w = d = 2
frameworkPyTorch§3.1
hardwareNVIDIA GeForce RTX 3090, 24 GB§3.1
stages (encoder and decoder)4 eachXMorpher.pynot stated in paper
blocks per stage2, 2, 6, 2XMorpher.pynot stated; the paper's k (§2.1) is not given
embedding dimension48XMorpher.pychannels double per stage (48, 96, 192, 384)
attention heads3, 6, 12, 24XMorpher.pyhead dimension 16 at every stage
patch size4, 4, 4XMorpher.pynot stated
MLP ratio4XMorpher.pynot stated
qkv biasonXMorpher.pynot stated
stochastic depth (drop path)0.2XMorpher.pynot stated
optimiserAdamUnsup_train.py Semi_train.pynot stated in paper
learning rate1e-4Unsup_train.py Semi_train.pynot stated in paper
epochs × iterations per epoch400 × 200 (unsup); 200 × 200 per stage (semi)Unsup_train.py Semi_train.pycode defaults; not stated in paper
batch size1Unsup_train.pythe window code returns -1 for B > 1 XMorpher.py
smoothness lossL2 on flow gradients, weight 1losses.py Unsup_train.pynot stated in paper
NCC losswindow 9 × 9 × 9, weight 1losses.py Unsup_train.pynot stated in paper
semi-supervised label termweight 0.5Semi_train.py Semi_train.pycross-entropy on segmentation output; not stated in paper
intensityclip to [0, 2048], divide by 2048dataloader.pynot stated in paper
augmentationrotation ±π/9 per axis, scale 0.75–1.25, mirroringUnsup_train.py Unsup_train.pynot stated in paper
moving/fixed swaprandom, p = 0.5dataloader.pynot stated in paper
datasetMM-WHS 2017 CT (20 labelled, 40 unlabelled); ASOCA (60 unlabelled)§3.1the data is not in the repository
training pairs500 labelled–unlabelled; 9900 unlabelled–unlabelled§3.1counts from 100 unlabelled and 5 labelled images
test pairs210 from the 15 remaining labelled images§3.1
preprocessingaffine transform to common coordinates§3.1tool and parameters not stated; not in the code
metricsmean DSC over all labels; |J(ψ)| ≤ 0 (%)§3.1the Jacobian metric is not defined and not in the code

Results

Author-reported, on the paper's own split of the MM-WHS 2017 CT and ASOCA data §3.1. The paper reports "±" values without saying whether they are standard deviations or another spread. Bold follows the source's marking, which is per setting tab. 1. The compared baselines are not in the repository, so these numbers have not been reproduced here.

MethodSettingBackboneDSC|J(ψ)| ≤ 0 (%)Source
Affine initialisation--69.2 ± 7.2-tab. 1
BSplineUnsupervised-80.9 ± 7.65.25 ± 3.27tab. 1
VoxelMorphUnsupervisedCNN80.2 ± 5.54.02 ± 0.82tab. 1
TransMorphUnsupervisedCNN + Transformer81.1 ± 5.23.46 ± 0.75tab. 1
No-cross XMorpher (ablation)UnsupervisedFull Transformer81.5 ± 5.40.94 ± 0.26tab. 1
VM-XMorpherUnsupervisedFull Transformer83.0 ± 4.73.15 ± 0.79tab. 1
PC-RegSemi-supervisedCNN86.0 ± 2.50.36 ± 0.20tab. 1
PC-XMorpherSemi-supervisedFull Transformer86.9 ± 2.40.32 ± 0.18tab. 1

Quantitative statements in the text §3.2:

  • VM-XMorpher is 2.8 percentage points higher than VoxelMorph on DSC and 0.87 points lower on the Jacobian measure §3.2, tab. 1.
  • PC-XMorpher is 0.9 points higher than PC-Reg on DSC and has a lower Jacobian measure §3.2, tab. 1.
  • Removing the cross attention (no-cross ablation: the two images are concatenated and self-attended) lowers VM-XMorpher's DSC by 1.5 points [§3.2, tab. 1]. Among the unsupervised rows, the ablation has the lowest Jacobian measure tab. 1.
  • Window size: larger base windows give better Jacobian results; smaller ones give better DSC. Shown as a plot without numbers §3.2, fig. 4a.
  • Qualitatively, the attention spreads evenly over small windows and concentrates on the heart region §3.2, fig. 4b. Visual results are in fig. 3; no numbers are given.

Paper vs code

Official repository: https://github.com/Solemoon/XMorpher, cloned at commit 926866ef9c0c0c0fa8a0eeedfcd6ee7404c4d841. The repository is linked from the paper's abstract. Its first author's GitHub account owns it, and the README attributes it to the authors. The code is Apache-2.0.

The repository contains the PyTorch model (models/XMorpher.py), two training scripts (Unsup_train.py for VM-XMorpher and Semi_train.py for PC-XMorpher), losses, a spatial transform, data loaders, and one example .mat file. It does not contain the MM-WHS or ASOCA data, the baseline implementations, the Jacobian metric, or trained weights. The MindSpore version is a README stub that says the release is still to come.

Discrepancies, and places where the code resolves what the paper leaves open:

  • Window size and magnification match. The paper sets h = w = d = 2 and α = β = γ = 3 §3.1; the code uses (2, 2, 2) XMorpher.py and builds 3 × 3 × 3 neighbourhoods of base windows, which equals α = 3 XMorpher.py. The factor 3 is hard-coded: changing α, β or γ in the code is not a parameter change.
  • "Full transformer" is attention only at the token level. The paper says the CAT blocks replace the convolutions of a U-Net §2.1. The code keeps strided 3D convolutions for patch embedding and merging, transposed convolutions for upsampling and output, and a 3 × 3 × 3 convolution that produces the flow XMorpher.py.
  • Number of fusion rounds k. The paper gives no value §2.1. The code uses a stage depth of 2, 2, 6, 2 XMorpher.py. Each block is applied in both directions with the same parameters XMorpher.py, so k equals the block count of its stage. The paper's two blocks sharing parameters §2.1 agree with this.
  • Normalisation of the searching tokens. The paper applies LayerNorm before each W-MCA §2.2. The code normalises the base tokens only; the searching tokens enter attention unnormalised XMorpher.py.
  • Scaling factor. Eq. 2 divides by √d without defining d eq. 2. The code scales by the per-head dimension C / heads, which is 16 at every stage XMorpher.py.
  • Boundary handling. The paper does not say what happens at volume edges. The code pads the grid of windows with zero windows and attends to them, with no mask XMorpher.py.
  • Training objective. The paper does not state a loss. For VM-XMorpher the code sums a smoothness loss on the flow and an NCC similarity loss between the warped moving image and the fixed image, each with weight 1 [code: Unsup_train.py#L74-L80] losses.py [code: utils/losses.py#L42-L67].
  • Sampling of training pairs. The paper gives 500 and 9900 pairs §3.1. The code draws from the two folders with probability 1/2 each, whatever their sizes Unsup_train.py.
  • Semi-supervised strategy. The paper names PC-Reg as the framework in one sentence §3.1. The code runs a two-stage schedule. In stage 1, each iteration updates the U-Net segmenter with cross-entropy where labels exist, then trains the registration head on foreground-masked images taken from the segmenter's own predictions. Stage 2 alternates the two networks, with the segmenter loss weighted 0.5 on the warped-label term [code: Semi_train.py#L79-L226] Semi_train.py.
  • Mean DSC. The paper says "all labels" §3.1. The code averages over classes 1 to 7 and so excludes background Unsup_train.py.
  • Data. The code maps the label values 205 to 850 to classes 1 to 7 [code: utils/dataloader.py#L51-L57] and clips intensities to [0, 2048] before dividing by 2048 dataloader.py. Neither is in the paper.

No discrepancies were found in the stated h, α, β or γ values, the eq. 2 structure, or the 2-layer GELU MLP.

Pitfalls

  • The released forward pass does not run. BasicLayer.forward assigns x = next_a XMorpher.py, and BasicLayerUp.forward does the same XMorpher.py. next_a is never defined. The variables assigned just above are next_x and next_xa, so the intent is clear, but as released the first call raises NameError. This was found by reading; the code was not executed here.
  • The searching-window indices are wrong. In window_area_partition, index 21 is written twice XMorpher.py, and index 20 is never written, so it stays zero XMorpher.py. One of the 27 neighbours is therefore missing and another is duplicated. The paper's numbers depend on this code path; whether the reported results used this exact version is not stated.
  • Zero padding enters attention. At the volume edge, the padded neighbour windows are zeros, and the attention reads them with no mask [code: models/XMorpher.py#L77-L80]. Their keys and values are the projections of zero, so they are not empty. The paper is silent on this.
  • Batch size must be 1. window_area_partition prints wrong and returns -1 for B > 1 XMorpher.py. The training scripts use batch size 1 Unsup_train.py.
  • CUDA is hard-coded. Window code moves tensors to the GPU [code: models/XMorpher.py#L80], the spatial transform builds its grid on the GPU [code: utils/STN.py#L17], and NCC allocates its filter on "cuda" [code: utils/losses.py#L49]. CPU use needs changes.
  • Argument order in Head. Head.forward calls self.swin(fixed, moving) XMorpher.py, so the moving parameter of the backbone receives the fixed image. The attention is asymmetric in its queries, so this order is part of the reproduced model.
  • Decoder resize swaps height and width. When a decoder feature does not match its encoder skip, the code interpolates to (D, W, H) after unpacking the shape as (B, D, W, H, C) XMorpher.py. That is wrong when the input is not cubic. The paper does not say whether this case arises for its data.
  • The inverse flow is an approximation. The reverse transform negates the flow sampled at its own displaced positions STN.py. This is only used for the reverse direction, which the training loss does not use.
  • The paper overstates one ranking. It says VM-XMorpher is "top-ranked" on the Jacobian measure §3.2. In tab. 1 the no-cross ablation has 0.94, lower than VM-XMorpher's 3.15. Among the rows without the ablation, VM-XMorpher is best at 3.15, ahead of TransMorph's 3.46 but behind the ablation.
  • The Jacobian measure is undefined and unreproducible from the repository. The paper gives |J(ψ)| ≤ 0 (%) without saying what the percentage is taken over. There is no Jacobian code in the repository.
  • Data and weights are missing. The 210 test pairs, the MM-WHS and ASOCA files, the affine preprocessing, and any checkpoints are not in the repository. The repository contains only data/one_example/0_1.mat. The loader reads labels only when len(data) == 6 dataloader.py [code: utils/dataloader.py#L63]. Counting the three header keys that loadmat returns, this condition depends on the file's layout. The sample file was not inspected, so check label loading before relying on it.

Lineage

  • Builds on: the Swin Transformer [liu2021swin], cited by the paper among single-image transformers §1; its card is at Swin Transformer. The code's block docstring calls it a "Swin Transformer Block" [code: models/XMorpher.py#L215]. The paper does not say which work it extends.
  • Framework: VoxelMorph [balakrishnan2018unsupervised] for the unsupervised setting §3.1, and PC-Reg [he2021few] for the semi-supervised setting §3.1. Neither has a card here.
  • Compared against: BSpline [rueckert1999nonrigid], VoxelMorph, PC-Reg, and TransMorph (arXiv:2111.10480, no card here) §3.1, tab. 1.
  • Superseded by: not stated. The code README names a MindSpore version that is not yet released README.md.

Source and license

Adapted from “XMorpher: Full Transformer for Deformable Medical Image Registration via Cross Attention” by Jiacheng Shi, Yuting He, Youyong Kong, Jean-Louis Coatrieux, Huazhong Shu, Guanyu Yang, Shuo Li (arXiv:2206.07349), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Solemoon/XMorpher at commit 926866ef9c0c (Apache-2.0). Any quoted code is shown under that license.