A review of deep learning-based information fusion techniques for multimodal medical image classification

In plain words

Doctors often look at several scans of the same patient, such as an MRI and a PET scan, or two photos of the eye, and combine what they see. Deep learning models can do this too, but there are many ways to wire the pictures together inside a network. This review sorts those wiring choices into a small set of families and checks which families have worked on which kinds of medical data.

The authors read the literature on classifying diseases from several imaging types at once. They describe five families: join the pictures before the network sees them, join them partway through, join them through a hierarchy of joins, let an attention mechanism decide what to join, or let each picture get its own answer and then merge the answers. They point out that the best family depends on how well the pictures line up, how different their shapes are, and how much labelled data exists. A recurring theme is that the models have only partly learned to use what each scan adds, and that missing scans are a practical problem.

TL;DR

A 2024 review of 114 deep-learning papers on combining medical imaging modalities for classification. It sorts methods into five families by where the fusion happens: before the network (input), once inside it (single-level), at several depths (hierarchical), through attention (attention-based), or on the final outputs (output) §4.1. It catalogues 14 public multimodal datasets §2.2 and tabulates ADNI results by fusion type tab. 5. Reported numbers are the original authors'. The paper found no official code for the review itself.

Problem

Single imaging modalities often miss information that another modality carries, so clinicians combine them §1.1. Classical fusion relied on hand crafted features and simple classifiers, which lose information and do not fully exploit complementarity §1.2. The authors note that medical datasets are small, expensive and often incomplete, so fusion designs must cope with missing modalities §5.3. They also note that no earlier review covered deep-learning fusion for classification across organs §1.1.

Scope: any method that assigns class labels to a patient or region of interest is in scope, regardless of application §1.1. Segmentation, registration and synthesis are excluded §1.1.

Method of selection: for 14 public datasets, PubMed was searched for the dataset name with multimodal terms, and abstracts were hand-screened §1.4. Nineteen further papers using private datasets were added for organs without public multimodal data §1.4. The final list is 114 publications §1.4.

Method

Pipeline (the stages every reviewed method shares)

The authors adopt a five-stage pipeline: data pre-processing, DL backbone, information fusion, final classifier, and model evaluation §3, tab. 3. The position of the fusion stage relative to the backbone defines the fusion type §4.1.

Fusion operations come in two kinds: concatenation, which stacks inputs into one tensor, and merge, which combines them by an operation such as addition that yields a smaller output §3.2, fig. 9.

Five fusion types §4.1, fig. 10

The taxonomy is set by the number of backbones and the position of the fusion step. Pre-processing and the final classifier are fixed in all five types §4.1.

# Input fusion  [§4.2] [fig. 11]
# One backbone sees all modalities combined before feature extraction.
def input_fusion(scans):
    # Stack or merge the registered scans into one tensor  [§4.2 (1)-(2)]
    fused = concatenate_channels_or_merge_pixels(scans)
    return classifier(backbone(fused))

# Single-level fusion  [§4.3] [fig. 12] [fig. 13]
# One backbone per modality; features fused once, before the classifier.
def single_level_fusion(scans):
    # Classic: each modality gets its own backbone  [§4.3 (1)]
    features = [backbone_m(scan_m) for m, scan_m in enumerate(scans)]
    # Fuse once: concatenate or merge  [§3.2]
    fused = concatenate_or_merge(features)
    # Network variant: a further backbone reads the fused features  [§4.3 (2)]
    return classifier(fused)

# Hierarchical fusion  [§4.4] [fig. 14] [fig. 20] [fig. 21]
# Fuse at several depths, then fuse again at the end.
def hierarchical_fusion(scans):
    # Per-modality branches  [§4.4 (1)]
    branches = [branch_m(scan_m) for m, scan_m in enumerate(scans)]
    # Fusion branch reads features at several levels  [§4.4 (1)]
    fused_levels = [fuse_level(branches, level) for level in levels]
    # Final fusion of every branch and fusion-branch output  [§4.4 (1)]
    return classifier(concatenate(fused_levels + branches))

# Attention-based fusion  [§4.5] [fig. 15]
# Transformers exchange information between modalities via attention.
def attention_fusion(scans):
    # Tokens from each modality, plus a learnable token  [§4.5]
    tokens = concatenate([tokenize(scan_m) for scan_m in scans])
    # Self- or cross-attention across modality tokens  [§4.5] [§5.7]
    mixed = transformer(tokens)
    return classifier(mixed)

# Output fusion  [§4.6] [fig. 16]
# Each modality gets its own model; their decisions are merged.
def output_fusion(scans):
    # One complete classifier per modality  [§4.6]
    decisions = [model_m(scan_m) for m, scan_m in enumerate(scans)]
    # Vote, average, weight, or train a second classifier  [§4.6]
    return combine(decisions)

Intermediate fusion groups the three middle types (single-level, hierarchical, attention-based) §4.1.

Missing modalities §5.3

Three approaches are described: discard incomplete subjects; impute features with mean or median values; or generate the missing modality with a GAN, the most promising option according to the authors §5.3. Two other designs are described: a branch that is updated only when its modality is present (PAIRED-NET §5.3), and a zero embedding for an absent modality (3MT §5.3).

Attention and Transformers §4.5, §5.7

Attention-based fusion uses cross-modal attention. The authors separate self-attention Transformers, which concatenate modality features before attention, from cross-attention Transformers, which compute attention between modalities §5.7, fig. 23. They recommend hybrid CNN and Transformer designs because medical datasets are small §4.5.

Hyperparameters

Not applicable. This is a review; it reports no training setup of its own. The evaluation metrics are defined in §3.5: accuracy, sensitivity, specificity, F1, positive and negative predictive value, AUC and Cohen's kappa. Kappa is defined using observed accuracy p0 and chance agreement pe §3.5. The metric formulas are unnumbered in the source.

Results

Author-reported. The review does not run experiments; it compiles published results on ADNI (MRI and PET, Alzheimer's disease) §5.1, tab. 5. The authors state that the experiments were not replicated and that each paper used a different patient subset §5.1.

Accuracy (%) as compiled in tab. 5. Columns: NC vs AD, NC vs MCI and the multi-class task (3 classes = NC/MCI/AD; 4 classes = NC/ncMCI/cMCI/AD).

Source: tab. 5. Sensitivity and specificity are in the same table and are omitted here. Overall statements the authors draw from these numbers: multi-level fusion is argued to exploit complementarity best §5.1; output fusion performs well on NC vs AD, which they attribute to pre-trained unimodal branches §5.1.

Other reported comparisons quoted in §5.1: on ABIDE, hierarchical fusion reached 87.2% against 65.5% for input fusion §5.1; on COBRE, input, output and single-level fusion reached 70%, 78% and 95% §5.1; on GAMMA, input, single-level and hierarchical fusion reached 63%, 72% and 80% §5.1.

Corpus statistics §2.2, §3.3, §5.6: 114 publications in total. ADNI is used in 53 articles, TCIA in 11, BraTS in 7, OASIS in 7, COBRE in 4, SPC in 4 and ABIDE in 4 (per §2.2). Backbones: CNN in 65 articles, FCNN in 10, autoencoders in 8 and Transformers in 6 §3.3. 27 of 114 papers use non-image context §5.6.

Paper vs code

No official code. The review does not link a repository for its own analysis. Searched: links in the paper (only TorchMultimodal, the Meta library §5.7, and Hugging Face Transformers §5.7, both third-party), the arXiv abstract, and a GitHub repository search on the title terms, which returned no results. Paper-internal footnotes point to other works' code (e.g. a CLIMAT repository and a vkola-lab repository in tab. 7); these are not the review's code and are not used here.

Pitfalls

• Taxonomy framing differs by section. The text calls single-level, hierarchical and attention-based fusion sub-categories of intermediate fusion §4.1, while the Highlights name intermediate fusion as one prevailing category that is then split three ways §1.5. The two framings are not reconciled.
• Table 5 mixes tasks and cohorts. Subject counts differ per paper (202 to 959), and the multi-class columns use different class sets. The authors say comparisons are only approximate §5.1. Do not rank methods from this table.
• Accuracy claims vs the table. The text reports that single-level fusion is more robust than input fusion and that hierarchical fusion helps mainly on four-class tasks §5.1. Those conclusions rest on single numbers per paper with no variance reported.
• Unstated details. No review-level hyperparameters, no per-paper training protocols, and no statistical tests are given. Reproducing any cited method needs the original paper.
• Dataset counts do not add up. The text reports 14 public datasets §1.4, and tab. 1 lists 14. The per-dataset article counts in §2.2 sum to about 102, and 19 further papers use private data §2.2; the total of 114 §1.4 is not broken down. The paper does not explain the gap.
• Terminology. "Early", "intermediate" and "late" fusion are used as synonyms for input, intermediate and output fusion §1.5. Concatenation and merge are used loosely in §3.2 and §4.2; the card's pseudocode follows the paper's wording.
• Citation keys. Some in-text keys are URLs (for example a doi.org key for a synthesis survey §1.1). Check the reference entries before citing any single work from this review.

Lineage

• Builds on: the five-stage pipeline of Sleeman et al. (2022) §3; the traditional-method survey by Kline et al. (2022) §1.2; the input, intermediate and output taxonomy of earlier fusion surveys §1.5.
• Scope vs other reviews: image fusion [azam2022review], image synthesis, segmentation and registration reviews are cited as covering other tasks §1.1.
• Superseded by: not stated.