Image registration

Overview

Registration corrects the spatial misalignment between two images so that the same anatomy lines up in both. The 2013 survey lists the corrections as compensation for scale changes, rotations and translations 1401.0166 §2. It places registration as the first of two stages in medical image fusion. Fusion then selects features from the aligned images and combines them 1401.0166 §2 fig. 3.

A 2018 paper addresses a harder case: deformable registration between MR and ultrasound of the prostate, where the organ shape differs between the planning scan and the procedure 1807.03361 §1. Its method learns a dense displacement field, a per-voxel warp, with a 3D CNN 1807.03361 §2.1.

A 2021 paper applies the same idea to camera poses rather than image pixels. BARF jointly fits a neural radiance field and the pose of each photo, and it starts from imperfect or unknown poses 2104.06405 §1. Its alignment step is the classical photometric one, which minimises the brightness error between images. Classical coarse-to-fine alignment blurs the images so the error surface has a wider basin; BARF gets a similar effect by opening the frequency bands of its input encoding over training 2104.06405 §3.1 §3.4. Its own tests were on synthetic and forward-facing real scenes, not on medical data.

All three sources share one invariant: the output must put corresponding content into the same frame, whether that is anatomy, pixels or camera poses in one scene. The survey keeps image resolution equal before and after fusion 1401.0166 §2.3, and it links inaccurate registration to poor feature- or decision-level fusion 1401.0166 §5. The 2018 paper scores a candidate warp by how well corresponding labels overlap after warping, not by how well the intensities match 1807.03361 §2.1. Its intensity-based baselines did poorly on the same data: a median landmark error (TRE) of 19.6 mm after centroid alignment, and 20.6 to 25.6 mm with free-form deformation on top of it 1807.03361 §4.4 tab. 2.

A 2023 review of vision transformers in medical imaging has its own registration section 2301.03505 §8 and names registration among the open problems for these models 2301.03505 §10. The review is a literature survey with no experiments of its own. Its registration comparison table is not transcribed in this wiki, so this page gives it no numbers. Read the paper for those.

The survey does not describe a registration algorithm. It gives no transform model, similarity measure, optimiser or search strategy, and no error figures. The 2018 paper and BARF are the only sources here with a specified method, and each reports numbers from its own datasets. Read the survey as a vocabulary and open-problems map, and the two method papers as worked examples with their limits.

Variants

The survey does not split registration into named methods. It lists families of techniques that it places in the registration stage, and two specific uses it reports. The 2018 paper is a learned, deformable method for one organ and one modality pair. BARF is a photometric method for camera poses in a neural scene model, and it is not an image-to-image warp.

Feature processing toolbox

The pipeline diagram lists the techniques used for feature processing and extraction in the registration stage: probabilistic and fuzzy logic, training algorithms, neural networks, morphological filtering, wavelet filtering, transforms and component analysis 1401.0166 §2 fig. 3. The survey does not say which of these handles the geometric alignment itself, and it does not describe how any of them is applied. The morphological family, for example, is described for feature detection and fusion, with the caveat that it is sensitive to outliers, noise and feature size 1401.0166 §2.1.

Possibilistic clustering

The survey reports possibilistic clustering as an improvement over fuzzy c-means in the registration stage 1401.0166 §2.6. No figures or comparison data are given.

Support vector machines

SVMs are described as useful in both the registration and the fusion stage, and valued for rejecting outliers 1401.0166 §2.6. The survey does not say how the SVM is posed or what it outputs for registration.

Application-specific registration

Some organ sections of the survey name registration problems without a method:

• Brain: 2D-3D registration is listed among uses for image-guided neurosurgery 1401.0166 §4.1.
• Prostate: deformation of the organ between imaging setups is named as the main challenge 1401.0166 §4.3. The 2018 variant below targets this case.
• Liver: registration and fusion are described together as a primary task 1401.0166 §4.5.

Weakly-supervised deformable CNN (MR to TRUS)

The 2018 paper, by Hu et al., registers a T2-weighted MR volume to a 3D transrectal ultrasound (TRUS) volume of the prostate. Card: Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration.

• Input and output. A moving MR and a fixed TRUS volume, each resampled to 0.8 mm isotropic and normalised per image 1807.03361 §3.1. The output is a dense displacement field 1807.03361 §2.1.
• Training signal. Only corresponding anatomical label pairs are used, with no voxel-level correspondence 1807.03361 §2.1. The similarity is a multiscale soft Dice, averaged over seven Gaussian blur scales, so that labels still give a signal when they do not overlap 1807.03361 §2.2 eq. 5. The deformation penalty is bending energy, weighted by α = 0.5 in the baseline 1807.03361 §3.2.1.
• Network. A 3D CNN with four down-sampling and four up-sampling blocks, plus displacement skip layers that sum predictions across resolutions 1807.03361 §2.3.
• Inference. Needs only the image pair, with no labels or initialisation, at about four registrations per second on the authors' GPU 1807.03361 §4.1.
• Reported accuracy. Cross-validated on 108 pairs from 76 patients, with a median TRE of 3.6 mm and a median gland Dice of 0.87 1807.03361 §4.1 tab. 1. The authors' earlier Global-Net and Composite-Net gave 5.8 mm and 4.7 mm under the same loss 1807.03361 §4.3 tab. 1.
• Ablations. Multiscale Dice beat the cross-entropy variant on TRE (p < 0.001), and bending energy beat an L2 gradient penalty on both metrics 1807.03361 §4.2 §4.3.

Photometric pose registration with coarse-to-fine encoding (BARF)

BARF, by Lin et al., fits a neural radiance field and every camera pose together, starting from imperfect or unknown poses. Card: BARF: Bundle-Adjusting Neural Radiance Fields.

• Input and output. Images of a static scene with initial poses. On the real scenes the start is the identity for every camera 2104.06405 §4.3. The output is a radiance field and a refined pose per image, parametrised in se(3) with known intrinsics 2104.06405 §3.2 §4.2.
• Objective. The squared photometric error over all pixels and images is minimised over the network weights and all poses at the same time 2104.06405 §3.2 eq. 8. No correspondence labels are used.
• Coarse-to-fine schedule. The sinusoidal bands of the positional encoding are opened one at a time, with a smooth window per band. The raw coordinates are never masked 2104.06405 §3.4 eq. 10 eq. 13 eq. 14. On the synthetic and real scenes the window opens linearly from iteration 20K to 100K of 200K, with L = 10 bands 2104.06405 §4.2 §4.3. The paper calls this schedule hand-set and says a per-scene optimum is out of scope 2104.06405 app. B.2.
• Reported results. Author-reported, on 8 Blender scenes with perturbed poses: mean rotation error 6.167 degrees with full encoding and 0.193 with BARF; PSNR 22.12 dB and 27.50 dB, against 29.40 dB for NeRF with true poses 2104.06405 §4.2 tab. 2. On 8 LLFF scenes from identity poses: mean rotation error 0.573 degrees with BARF, against 0.699 without any encoding, which is also the baseline with the better translation error 2104.06405 §4.3 tab. 3 tab. 5. On planar patch alignment, the sl(3) error is 0.0096, against 0.0641 without encoding and 0.2949 with full encoding 2104.06405 §4.1 tab. 1.
• Limits. The authors name slow optimisation and rendering, the assumption of a rigid scene, sensitivity to dense sampling and the heuristic schedule 2104.06405 §5. Results depend on the depth parametrisation: with regular metric depth on LLFF, BARF's mean rotation error rises to 1.580 degrees 2104.06405 tab. 6 app. B.2. The LLFF reference poses come from structure-from-motion, so the errors measure agreement with that method rather than ground truth 2104.06405 §4.3.

When to use which

The survey gives no decision rules for registration. The table maps the constraints the sources name to the options they report, and marks where they give nothing.

Sensitivity to the regulariser matters for the learned method. With α = 0.5 the baseline produced no negative Jacobian determinants. An illustrative network with α = 0.01 did 1807.03361 §4.3.

Implementation sketch

The survey does not specify an algorithm, so the first block shows only the interface between the two stages. Every line marked "not stated" needs a primary source.

# Stage 1: estimate the spatial mapping between the two images
# [arxiv:1401.0166 §2, fig. 3]. Inputs: moving image M, fixed image F.
# Transform family, similarity measure and optimiser: not stated.
T = register(M, F)
# Resample M into the frame of F so pixels correspond
# [arxiv:1401.0166 §2.3: fusion keeps equal resolution]
M_aligned = resample(M, T, F.grid)
# Stage 2: fuse the aligned pair [arxiv:1401.0166 §2]
# Fusion rule (wavelet, fuzzy, neural, SVM, ...): see Variants
fused = fuse(F, M_aligned)

The learned variant replaces register with a network that predicts the mapping directly. The block below is one training step of that variant.

# One training step of the learned variant [arxiv:1807.03361 §2.1]
# Inputs: image pair (x_A, x_B) and K sampled label pairs (l_A, l_B)
# Predict a dense displacement field u from the image pair
u = cnn(x_A, x_B; theta)
# Warp the moving labels with u
y_A = warp(l_A, u)
# Score label overlap with multiscale soft Dice [arxiv:1807.03361 §2.2]
J = mean(multiscale_soft_dice(l_B, y_A))
# Penalise non-smooth displacements: bending energy, alpha = 0.5
# in the baseline [arxiv:1807.03361 §3.2.1]
loss = -J + alpha * bending_energy(u)
# Adam update; learning rate 1e-5 in the baseline, schedule not stated
theta = adam_step(theta, grad(loss, theta))

The photometric variant (BARF) keeps the same loop shape but optimises poses as well as the network. Its coarse-to-fine step is the window on each band of the input encoding.

# Weight of positional-encoding band k at progress alpha, in [0, L]
# [arxiv:2104.06405 §3.4] [eq. 13] [eq. 14]
def band_weight(alpha, k):
    # Band k is closed until alpha reaches k, then opens over one unit
    t = clamp(alpha - k, 0, 1)
    return (1 - cos(pi * t)) / 2

# One joint step over pixel rays [arxiv:2104.06405 §3.2] [eq. 8]
# Network weights and per-image poses are updated together
for each batch of rays:
    # Each ray uses the pose of the image it belongs to
    poses = pose_params[image_of(rays)]
    # Encode sampled points with the windowed encoding [§3.4]
    feats = encode(points(rays, poses), alpha, band_weight)
    # Summed squared error of volume-rendered colour [eq. 8]
    loss = sum((render(feats, rays) - target) ** 2)
    theta, pose_params = adam_step(loss, theta, pose_params)

Contradictions

None between the sources on the same claim. The survey has no experiments and predates the 2018 paper and BARF, so it cannot confirm or dispute their numbers.

Across the two method papers, the scope differs, and the sources state it differently:

• Intensity similarity. The 2018 paper says intensity-based similarity is unreliable across MR and ultrasound, because the same anatomy can look different in each 1807.03361 §1. BARF's objective is a photometric error on RGB images of one scene 2104.06405 §3.2. Both are reported as they stand; the sources do not test each other's setting.
• Schedules. BARF's window is hand-set and depends on the total iteration count 2104.06405 app. B.2. The 2018 paper's learning rate starts at 1e-5 with no decay schedule or iteration count stated 1807.03361 §3.2.1. Neither paper says how to choose its schedule for a new dataset.

Two points inside the 2018 paper need care:

• Initialisation. §2.3 describes the displacement skip layers as randomly initialised with zero mean and small variance. §3.2.1 says the final displacement layers were initialised to zero for all reported results 1807.03361 §2.3 §3.2.1. The paper's reported numbers used the zero initialisation.
• Comparisons with earlier work. The 2018 paper's own comparison table lists earlier MR-to-TRUS results with TREs of 1.4 to 2.8 mm. It marks them as not directly comparable, since they come from other datasets, label sets and training strategies 1807.03361 §4.4 tab. 2.

Papers

• Medical Image Fusion: A survey of the state of the art: surveys medical image fusion up to 2013, presents registration as the first stage, and names techniques for it without specifying an algorithm.
• Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration: learns a dense displacement field for prostate MR-to-TRUS registration from sparse anatomical label pairs, with median TRE 3.6 mm on 108 pairs (author-reported).
• BARF: Bundle-Adjusting Neural Radiance Fields: fits a NeRF and its camera poses jointly from imperfect or unknown poses, using a coarse-to-fine positional encoding; mean rotation error 0.193 degrees on synthetic scenes with perturbed poses (author-reported).
• Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review: a survey of vision transformers in medical imaging with a registration section; names registration as an open problem, and its registration numbers are not transcribed here.