--- kind: concept slug: image-fusion title: Image fusion summary: >- Combining images or sensors of one scene into one result. Three families: rule- or wavelet-based fusion of registered medical scans; a second sensor as a transformer token (hyperspectral plus LiDAR); and camera and LiDAR feature maps fused in a shared bird's-eye-view grid. The trade-off is interpretability and data needs against accuracy and compute cost. tags: [image-fusion, medical-imaging, remote-sensing, multimodal, autonomous-driving] papers: ["1401.0166", "2203.16952", "2205.13790"] related: [vision-transformer, self-attention, bev-representation, object-detection, image-registration] status: draft updated_at: 2026-10-18 --- ## Overview Image fusion merges information from more than one image, or from more than one sensor, of the same scene so that the result is more useful than any single input. The cards cover three settings that share the word but differ in method: - **Pixel- or feature-level fusion of medical scans.** The inputs are aligned first, then features from each image are selected and combined. The survey describes this as two steps: registration, then fusion of the selected features or decisions [arxiv:1401.0166 §2]. Motivation: no single modality shows every detail a diagnosis needs, for example MRI for soft tissue and PET for tracer uptake [arxiv:1401.0166 §1]. - **Learned fusion inside a network.** A second sensor (LiDAR, SAR, a surface model) is compressed into one classification token, which then attends to tokens from the main sensor [arxiv:2203.16952 §III-B2]. Here the fusion is part of the model rather than a preprocessing step. - **Two-stream fusion of camera and LiDAR in bird's-eye view.** Each sensor gets its own network. The camera stream places its features on a top-down grid using predicted depth, the LiDAR stream builds a grid of its own, and the two grids are fused before a detection head [arxiv:2205.13790 §3]. The camera stream never reads LiDAR points, so the detector can still predict when the LiDAR is weak or cropped [arxiv:2205.13790 §1]. The shared constraint is that the inputs must be comparable. The survey assumes equal resolution before and after fusion for its wavelet methods [arxiv:1401.0166 §2.3], and it names registration errors as a main cause of poor fusion [arxiv:1401.0166 §5]. The BEV family sidesteps pixel registration by placing the camera in the vehicle frame with the camera-to-LiDAR calibration [arxiv:2205.13790 §3.1]. It then needs both streams to share one grid size, so changing the resolution of one stream changes the other [arxiv:2205.13790 §3.1]. ## Variants ### Classical rule-based fusion of registered scans The survey groups these methods into families [arxiv:1401.0166 §2.1–§2.6]: - **Morphological:** opening and closing operators applied across scales, with the fused features chosen per scale. Sensitive to noise and to the structuring element, which the survey does not specify [arxiv:1401.0166 §2.1]. - **Knowledge-based:** clinical rules guide segmentation and decide what a region should look like in each modality. Compared against human visual standards; relies on expert labelling [arxiv:1401.0166 §2.2]. - **Wavelet-based:** detail, usually high-frequency, is taken from one image and injected into the other. Substitution is the simplest injection rule; addition and aggregator functions are alternatives. The number of decomposition levels is not stated [arxiv:1401.0166 §2.3]. Wavelets are often combined with a neural network that does feature processing [arxiv:1401.0166 §2.3]. - **Fuzzy logic:** fuzzy operators act as feature or decision operators. Choosing the membership functions is called an open problem [arxiv:1401.0166 §2.5]. - **Neural, SVM and component methods:** weights are learned from a training set, and robustness depends on training-data quality [arxiv:1401.0166 §2.4]. PCA with the intensity-hue-saturation transform is reported to keep spatial detail without colour distortion, a qualitative claim without numbers [arxiv:1401.0166 §2.6] [arxiv:1401.0166 §5]. Card: [Medical Image Fusion: A survey of the state of the art](/papers/1401.0166). This is a review, not a method paper; no algorithm in it can be implemented from the text alone, and the survey reports no quantitative results. ### Cross-modal token fusion in a transformer The main sensor's patch is tokenised into several tokens. The second sensor's patch becomes one classification token, built by attention pooling, and encoder blocks let only that token form the query, so it gathers information from every token [arxiv:2203.16952 §III-B2] [arxiv:2203.16952 §III-B3]. The authors argue that stacking the second sensor's channels as extra input bands raises the projection size and the risk of overfitting, which is their reason for the token design [arxiv:2203.16952 §III-B]. On four hyperspectral scenes, with the main sensor alone and with a second sensor added, the paper reports the highest overall accuracy among its ten compared methods in each setting [arxiv:2203.16952 §IV-B]. Examples: 89.80 % on Houston with LiDAR against 85.05 % for a plain ViT [arxiv:2203.16952 tab. II], and 98.32 % on Trento with LiDAR [arxiv:2203.16952 tab. V]. These are author results on one test protocol; see Contradictions. Card: [Multimodal Fusion Transformer for Remote Sensing Image Classification](/papers/2203.16952). ### Two-stream BEV fusion of camera and LiDAR The camera stream predicts a depth distribution over bins for each pixel and multiplies it with per-pixel context features. The product is a frustum of features, which is moved into the ego frame and summed into a voxel grid. The height axis is folded into channels, and a small convolutional encoder gives a camera map in bird's-eye view [arxiv:2205.13790 §3.1]. The LiDAR stream can be any LiDAR BEV network; the paper tests PointPillars, CenterPoint and TransFusion-L [arxiv:2205.13790 §3.2]. The two maps are concatenated, passed through a 3x3 convolution and a channel gate, and the result goes to a detection head [arxiv:2205.13790 §3.3] [arxiv:2205.13790 eq. 1] [arxiv:2205.13790 eq. 2]. On nuScenes validation, fusion raises mAP from 64.9 to 67.9 with TransFusion-L as the LiDAR stream, and from 35.1 to 53.5 with PointPillars [arxiv:2205.13790 tab. 1]. When LiDAR points are removed by a field-of-view limit, the authors report gains over LiDAR alone for all three streams at both limits; for PointPillars at ±60 degrees, mAP goes from 8.4 to 33.5 [arxiv:2205.13790 tab. 3]. Dropping points inside object boxes during training is reported to help the fused model: TransFusion-L with fusion reaches 50.3 mAP with that training and 40.8 without it, and LiDAR alone without it reaches 34.6 [arxiv:2205.13790 tab. 4]. The augmented LiDAR-only row is not reported, so the comparison with LiDAR alone is not like for like. The test-split result for the same TransFusion-L fusion model is 69.2 mAP [arxiv:2205.13790 tab. 2]. Robustness results cover removed or cropped points, not a frame with no LiDAR points at all [arxiv:2205.13790 §4.4.1]. Card: [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](/papers/2205.13790). Author-reported throughout. The authors' code is public. ## When to use which | Situation | Use | Why | |---|---|---| | Two registered medical scans, and clinicians must be able to inspect the rule | Knowledge-based or wavelet fusion | The rule is explicit and can be compared to clinical standards [arxiv:1401.0166 §2.2]. The survey gives no parameters, so expect to derive them from the primary papers in its bibliography [arxiv:1401.0166 §2.3]. | | No labelled training data | Wavelet, morphological or fuzzy fusion | These do not learn weights from labels. Fuzzy membership functions and structuring elements must still be chosen by hand [arxiv:1401.0166 §2.1] [arxiv:1401.0166 §2.5]. | | Labelled data, and sensors with different band counts or dimensions (hyperspectral with LiDAR, SAR or a surface model) | Transformer token fusion | The second sensor enters as one token instead of extra bands [arxiv:2203.16952 §III-B]. The paper reports the best accuracy in its settings [arxiv:2203.16952 tab. I–X]. | | Very few labelled samples | Token fusion, with care | The paper trains on 3–9 % of the training samples and reports that its model stays ahead of the other transformers at each ratio. The numbers appear only in figures [arxiv:2203.16952 §IV-E] [arxiv:2203.16952 fig. 14]. | | Compute or parameter budget is tight | Check the model cost first | The paper says its parameter count and FLOPs are higher than most baselines, lower than ViT on two datasets. The per-model counts appear only in a figure [arxiv:2203.16952 §IV-F] [arxiv:2203.16952 fig. 16]. | | 3D detection from cameras and LiDAR, where the LiDAR may be weak, cropped or thin | Two-stream BEV fusion | The camera branch does not depend on LiDAR points. On validation the fused model keeps 65.1 to 66.2 mAP when the front camera is removed, when only the front camera is kept, or when half the camera frames are frozen, against 67.9 clean [arxiv:2205.13790 tab. 5]. | | The LiDAR stream is already strong | Expect a smaller gain from BEV fusion | With TransFusion-L the gain is 3.0 mAP on validation; with PointPillars it is 18.4 mAP [arxiv:2205.13790 tab. 1]. | | Real-time 3D detection on a fixed compute budget | Measure latency first | On the authors' batch-size-1 timing with one V100, the fused models take about 1.46–1.53 s per frame, against 0.19–0.26 s for LiDAR alone. The camera-to-BEV projection alone costs more than 957 ms [arxiv:2205.13790 app. E] [arxiv:2205.13790 tab. 13]. | | Real-time or volumetric medical use | Plan for an open problem | The survey names speed as an open challenge and points to optimised algorithms, FPGAs and GPUs. It gives no timings [arxiv:1401.0166 §5]. | ## Implementation sketch The common core is registration, feature extraction, combination and output. The variants differ in the feature step and in how the combination is learned. ```pseudocode # Common core of image fusion [arxiv:1401.0166 §2] # Input: two or more images or sensor grids of one scene # Output: a fused image, or features for a downstream model def fuse(images): # 1. Align the inputs: same grid, same resolution [arxiv:1401.0166 §2.3] aligned = register_and_resample(images) # 2. Extract features per input (wavelet detail, morphology, CNN, tokens) feats = [extract(img) for img in aligned] # 3. Combine with a fixed rule or a learned operator [arxiv:1401.0166 §2] fused = combine(feats) # 4. Pass the fused result to diagnosis or classification return fused ``` For the learned token variant, the combine step is one attention block in which only the classification token asks the question [arxiv:2203.16952 §III-B3] [arxiv:2203.16952 eq. 7a–9]: ```pseudocode # One encoder block, CLS-query attention [arxiv:2203.16952 §III-B3] # Input: tokens (5, 64): CLS from the second sensor, then 4 main-sensor tokens def fuse_block(tokens): # 1. Only token 0 forms the query; keys and values come from all tokens normed = layer_norm(tokens) query = normed[0] @ W_q keys = normed @ W_k values = normed @ W_v # 2. Attention over the 5 tokens, 8 heads of width 8 [eq. 7a] weights = softmax(query @ keys.T / sqrt(8)) # 3. Write the attended update back to all tokens [eq. 7b, 8b] tokens = tokens + dropout((weights @ values) @ W_l) # 4. MLP with residual [eq. 9] return tokens + mlp(layer_norm(tokens)) ``` The paper does not state the number of blocks, the MLP width or the classifier head [arxiv:2203.16952 §III-B3]. Take those from the code, and treat them as unverified. For the BEV variant, only the fusion step is shown. The camera and LiDAR branches are the ones described in its variant section. ```pseudocode # Two-stream BEV fusion, fusion step only [arxiv:2205.13790 §3.3] [eq. 1–2] # Input: cam_bev (B, C_cam, H, W) from the camera stream # lidar_bev (B, C_lidar, H, W) from any LiDAR BEV network # Output: fused (B, C_lidar, H, W) for a detection head def fuse_bev(cam_bev, lidar_bev): # 1. Both maps must share one BEV grid [arxiv:2205.13790 §3.1] # 2. Concatenate on channels, then a 3x3 conv to the LiDAR width [eq. 1] # The code also applies BatchNorm and ReLU here; eq. 1 does not [§3.3] mixed = conv3x3(concat([cam_bev, lidar_bev], dim=channels)) # 3. Channel gate: pool over space, linear map, sigmoid [eq. 2] gate = sigmoid(linear(global_avg_pool(mixed))) return gate * mixed ``` ## Contradictions No two cards disagree on the same task. The three cards address different problems: a medical fusion survey, hyperspectral land-cover classification, and 3D object detection on driving data. Their numbers are not comparable, and the survey reports no numbers that the other two could be checked against. Within one paper, the BEVFusion schedule disagrees with itself. Section 4.1 says the fusion stage runs for 9 more epochs, while Appendix B.1 gives 6 epochs for TransFusion-L and CenterPoint and 12 for PointPillars [arxiv:2205.13790 §4.1] [arxiv:2205.13790 app. B.1]. This affects reproduction, not the comparison tables. Caveat on the reported MFT numbers: the companion card records that the released notebook picks checkpoints by test-set accuracy, while the paper does not say how checkpoints were selected [arxiv:2203.16952 §IV-B]. Results obtained with a validation split may be lower. See [the card's pitfalls](/papers/2203.16952). ## Papers - [Medical Image Fusion: A survey of the state of the art](/papers/1401.0166): survey of registration-then-fusion methods for medical scans, grouped by method family and imaging modality, up to 2013; no experiments. - [Multimodal Fusion Transformer for Remote Sensing Image Classification](/papers/2203.16952): transformer that fuses a second sensor into one classification token for hyperspectral land-cover classification; author-reported accuracy on ten dataset settings. - [BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework](/papers/2205.13790): two-stream camera and LiDAR fusion in bird's-eye view for 3D detection on nuScenes; author-reported gains, including under simulated LiDAR loss.