Image fusion

Overview

Image fusion merges information from more than one image, or from more than one sensor, of the same scene so that the result is more useful than any single input. The cards cover three settings that share the word but differ in method:

• Pixel- or feature-level fusion of medical scans. The inputs are aligned first, then features from each image are selected and combined. The survey describes this as two steps: registration, then fusion of the selected features or decisions 1401.0166 §2. Motivation: no single modality shows every detail a diagnosis needs, for example MRI for soft tissue and PET for tracer uptake 1401.0166 §1.
• Learned fusion inside a network. A second sensor (LiDAR, SAR, a surface model) is compressed into one classification token, which then attends to tokens from the main sensor 2203.16952 §III-B2. Here the fusion is part of the model rather than a preprocessing step.
• Two-stream fusion of camera and LiDAR in bird's-eye view. Each sensor gets its own network. The camera stream places its features on a top-down grid using predicted depth, the LiDAR stream builds a grid of its own, and the two grids are fused before a detection head 2205.13790 §3. The camera stream never reads LiDAR points, so the detector can still predict when the LiDAR is weak or cropped 2205.13790 §1.

The shared constraint is that the inputs must be comparable. The survey assumes equal resolution before and after fusion for its wavelet methods 1401.0166 §2.3, and it names registration errors as a main cause of poor fusion 1401.0166 §5. The BEV family sidesteps pixel registration by placing the camera in the vehicle frame with the camera-to-LiDAR calibration 2205.13790 §3.1. It then needs both streams to share one grid size, so changing the resolution of one stream changes the other 2205.13790 §3.1.

Variants

Classical rule-based fusion of registered scans

The survey groups these methods into families 1401.0166 §2.1–§2.6:

• Morphological: opening and closing operators applied across scales, with the fused features chosen per scale. Sensitive to noise and to the structuring element, which the survey does not specify 1401.0166 §2.1.
• Knowledge-based: clinical rules guide segmentation and decide what a region should look like in each modality. Compared against human visual standards; relies on expert labelling 1401.0166 §2.2.
• Wavelet-based: detail, usually high-frequency, is taken from one image and injected into the other. Substitution is the simplest injection rule; addition and aggregator functions are alternatives. The number of decomposition levels is not stated 1401.0166 §2.3. Wavelets are often combined with a neural network that does feature processing 1401.0166 §2.3.
• Fuzzy logic: fuzzy operators act as feature or decision operators. Choosing the membership functions is called an open problem 1401.0166 §2.5.
• Neural, SVM and component methods: weights are learned from a training set, and robustness depends on training-data quality 1401.0166 §2.4. PCA with the intensity-hue-saturation transform is reported to keep spatial detail without colour distortion, a qualitative claim without numbers 1401.0166 §2.6 1401.0166 §5.

Card: Medical Image Fusion: A survey of the state of the art. This is a review, not a method paper; no algorithm in it can be implemented from the text alone, and the survey reports no quantitative results.

Cross-modal token fusion in a transformer

The main sensor's patch is tokenised into several tokens. The second sensor's patch becomes one classification token, built by attention pooling, and encoder blocks let only that token form the query, so it gathers information from every token 2203.16952 §III-B2 2203.16952 §III-B3. The authors argue that stacking the second sensor's channels as extra input bands raises the projection size and the risk of overfitting, which is their reason for the token design 2203.16952 §III-B.

On four hyperspectral scenes, with the main sensor alone and with a second sensor added, the paper reports the highest overall accuracy among its ten compared methods in each setting 2203.16952 §IV-B. Examples: 89.80 % on Houston with LiDAR against 85.05 % for a plain ViT 2203.16952 tab. II, and 98.32 % on Trento with LiDAR 2203.16952 tab. V. These are author results on one test protocol; see Contradictions.

Card: Multimodal Fusion Transformer for Remote Sensing Image Classification.

Two-stream BEV fusion of camera and LiDAR

The camera stream predicts a depth distribution over bins for each pixel and multiplies it with per-pixel context features. The product is a frustum of features, which is moved into the ego frame and summed into a voxel grid. The height axis is folded into channels, and a small convolutional encoder gives a camera map in bird's-eye view 2205.13790 §3.1. The LiDAR stream can be any LiDAR BEV network; the paper tests PointPillars, CenterPoint and TransFusion-L 2205.13790 §3.2. The two maps are concatenated, passed through a 3x3 convolution and a channel gate, and the result goes to a detection head 2205.13790 §3.3 2205.13790 eq. 1 2205.13790 eq. 2.

On nuScenes validation, fusion raises mAP from 64.9 to 67.9 with TransFusion-L as the LiDAR stream, and from 35.1 to 53.5 with PointPillars 2205.13790 tab. 1. When LiDAR points are removed by a field-of-view limit, the authors report gains over LiDAR alone for all three streams at both limits; for PointPillars at ±60 degrees, mAP goes from 8.4 to 33.5 2205.13790 tab. 3. Dropping points inside object boxes during training is reported to help the fused model: TransFusion-L with fusion reaches 50.3 mAP with that training and 40.8 without it, and LiDAR alone without it reaches 34.6 2205.13790 tab. 4. The augmented LiDAR-only row is not reported, so the comparison with LiDAR alone is not like for like. The test-split result for the same TransFusion-L fusion model is 69.2 mAP 2205.13790 tab. 2.

Robustness results cover removed or cropped points, not a frame with no LiDAR points at all 2205.13790 §4.4.1.

Card: BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework. Author-reported throughout. The authors' code is public.

When to use which

Implementation sketch

The common core is registration, feature extraction, combination and output. The variants differ in the feature step and in how the combination is learned.

# Common core of image fusion  [arxiv:1401.0166 §2]
# Input:  two or more images or sensor grids of one scene
# Output: a fused image, or features for a downstream model

def fuse(images):
    # 1. Align the inputs: same grid, same resolution  [arxiv:1401.0166 §2.3]
    aligned = register_and_resample(images)
    # 2. Extract features per input (wavelet detail, morphology, CNN, tokens)
    feats = [extract(img) for img in aligned]
    # 3. Combine with a fixed rule or a learned operator  [arxiv:1401.0166 §2]
    fused = combine(feats)
    # 4. Pass the fused result to diagnosis or classification
    return fused

For the learned token variant, the combine step is one attention block in which only the classification token asks the question 2203.16952 §III-B3 2203.16952 eq. 7a–9:

# One encoder block, CLS-query attention  [arxiv:2203.16952 §III-B3]
# Input:  tokens (5, 64): CLS from the second sensor, then 4 main-sensor tokens

def fuse_block(tokens):
    # 1. Only token 0 forms the query; keys and values come from all tokens
    normed = layer_norm(tokens)
    query = normed[0] @ W_q
    keys = normed @ W_k
    values = normed @ W_v
    # 2. Attention over the 5 tokens, 8 heads of width 8  [eq. 7a]
    weights = softmax(query @ keys.T / sqrt(8))
    # 3. Write the attended update back to all tokens  [eq. 7b, 8b]
    tokens = tokens + dropout((weights @ values) @ W_l)
    # 4. MLP with residual  [eq. 9]
    return tokens + mlp(layer_norm(tokens))

The paper does not state the number of blocks, the MLP width or the classifier head 2203.16952 §III-B3. Take those from the code, and treat them as unverified.

For the BEV variant, only the fusion step is shown. The camera and LiDAR branches are the ones described in its variant section.

# Two-stream BEV fusion, fusion step only  [arxiv:2205.13790 §3.3] [eq. 1–2]
# Input:  cam_bev (B, C_cam, H, W) from the camera stream
#         lidar_bev (B, C_lidar, H, W) from any LiDAR BEV network
# Output: fused (B, C_lidar, H, W) for a detection head

def fuse_bev(cam_bev, lidar_bev):
    # 1. Both maps must share one BEV grid  [arxiv:2205.13790 §3.1]
    # 2. Concatenate on channels, then a 3x3 conv to the LiDAR width  [eq. 1]
    # The code also applies BatchNorm and ReLU here; eq. 1 does not  [§3.3]
    mixed = conv3x3(concat([cam_bev, lidar_bev], dim=channels))
    # 3. Channel gate: pool over space, linear map, sigmoid  [eq. 2]
    gate = sigmoid(linear(global_avg_pool(mixed)))
    return gate * mixed

Contradictions

No two cards disagree on the same task. The three cards address different problems: a medical fusion survey, hyperspectral land-cover classification, and 3D object detection on driving data. Their numbers are not comparable, and the survey reports no numbers that the other two could be checked against.

Within one paper, the BEVFusion schedule disagrees with itself. Section 4.1 says the fusion stage runs for 9 more epochs, while Appendix B.1 gives 6 epochs for TransFusion-L and CenterPoint and 12 for PointPillars 2205.13790 §4.1 2205.13790 app. B.1. This affects reproduction, not the comparison tables.

Caveat on the reported MFT numbers: the companion card records that the released notebook picks checkpoints by test-set accuracy, while the paper does not say how checkpoints were selected 2203.16952 §IV-B. Results obtained with a validation split may be lower. See the card's pitfalls.

Papers

• Medical Image Fusion: A survey of the state of the art: survey of registration-then-fusion methods for medical scans, grouped by method family and imaging modality, up to 2013; no experiments.
• Multimodal Fusion Transformer for Remote Sensing Image Classification: transformer that fuses a second sensor into one classification token for hyperspectral land-cover classification; author-reported accuracy on ten dataset settings.
• BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework: two-stream camera and LiDAR fusion in bird's-eye view for 3D detection on nuScenes; author-reported gains, including under simulated LiDAR loss.