Artificial Intelligence-Based Methods for Fusion of Electronic Health Records and Imaging Data

In plain words

Doctors decide with many kinds of information at once: a patient's history, lab results and medications, plus scans such as MRI or CT. A computer model can use several of these sources together, and this review asks how that combining, called fusion, has been done in medical research.

The authors searched four literature databases and kept 34 studies that combine patient records with scans using machine learning or deep learning. They sort the studies by when the two kinds of data are merged, by disease, by the clinical question (diagnosis or prediction), by the public datasets used and by the scores reported. The most common approach joins the features of both sources before a single model is trained. Most of the studies report that the combined model does better than a model that sees only one source.

The review does not build or test a model. It is a map of the literature and a list of resources, with a note on which gaps need filling.

TL;DR

A scoping review, not a method. It sorts 34 studies that fuse EHR with medical images into early, joint and late fusion, and tabulates diseases, clinical tasks, data sources, public datasets and metrics. Use it as an orientation and a source list. Its counts are the review's own, and its performance claims are only the reviewed studies' claims, not tested here.

Problem

Medical images are often hard to read without the clinical context of the patient. The authors cite evidence that missing clinical data lowers the accuracy of radiologists' diagnoses §Introduction. Fusing the two sources is expected to help precision medicine, but there was no recent scoping review for EHR-plus-imaging fusion with conventional ML or DL methods. Earlier reviews covered other pairs of modalities, omics data, or only deep learning, or included fewer studies §Introduction tab. 1.

The review asks four questions: which fusion strategies are used, which diseases they target, which clinical outcomes and ML models go with each, and which public multimodal datasets exist §Introduction.

Method

The review follows PRISMA-ScR §Methods. It has no algorithm, equations or model. What an implementer can reuse is the taxonomy of fusion strategies and the screening protocol, given below.

Fusion taxonomy, adopted from the earlier review by Huang et al. §Preliminaries:

# Early fusion: join inputs before one model  [§Fusion strategies]
# Image features may come from NN, software or manual extraction
def early_fusion(image, record):
    image_features = extract_image_features(image)
    joined = concatenate(image_features, record)
    return single_model(joined)

# Late fusion: one model per modality, then aggregate the decisions
# Aggregation: weighted average, majority vote or meta-classifier
def late_fusion(image, record):
    p_image = image_model(image)
    p_record = record_model(record)
    return aggregate(p_image, p_record)

# Joint fusion: NN features join mid-network; loss reaches extractors
# Only NN-based joint fusion can send loss back into the extractor
def joint_fusion(image, record):
    image_features = image_network(image)
    record_features = record_network(record)
    return head(concatenate(image_features, record_features))

Screening protocol §Methods §Results:

1. Databases: Embase, PubMed, Scopus and Google Scholar. For Google Scholar only the first 110 relevant results were used. MEDLINE is covered through PubMed §Search strategy.
2. Search string: three groups of terms joined with AND. The first group is AI terms (artificial intelligence, machine learning, deep learning), the second is multimodal fusion, the third is medical imaging or EHR terms. Variant forms of each term were used. The full strings are in Appendix 1 of the supplementary material, which is not in the source package §Search strategy.
3. Date window and language: English only, published 1 January 2015 to 6 January 2022. The search covered titles and abstracts and ran 3 to 6 January 2022 §Search strategy.
4. Inclusion: EHR (structured or free text) fused with any medical imaging modality, using an AI model for any clinical task. Peer-reviewed articles and conference proceedings only §Inclusion and exclusion criteria.
5. Exclusion: a single data modality, including several imaging types or imaging plus its extracted features; multi-omics data; classical statistical models such as regression; reviews, abstracts, preprints, editorials and letters §Inclusion and exclusion criteria.
6. Selection: Rayyan, duplicates removed, title and abstract screening, then full text. Two authors selected studies, a third resolved disagreements §Study selection.
7. Extraction: a form piloted on four studies. It records first author, year, first-author country, disease, clinical outcome, imaging and EHR modality, fusion strategy, feature extraction, data source, AI model, metrics and comparison with single-modality models §Data extraction.
8. Quality: not assessed, following scoping-review guidance [§Study quality assessment].

Hyperparameters

Not applicable: the review trains no model. Its numeric settings are the search and screening parameters in the Method steps above §Methods §Search strategy §Study selection.

Results

Author-reported counts of the 34 included studies. The review does not test any method.

Search flow fig. 1: 1158 records retrieved; 971 after removing duplicates; 44 for full-text review after title and abstract screening; 10 removed at full text; 34 included §Search results.

Fusion versus single modality, as reported by the reviewed studies:

• Early fusion: 14 studies compared the fused model with single-modality models; 13 of them reported the fused model as better §Early fusion.
• Joint fusion: 7 of 10 compared with single modality and reported a gain §Joint fusion.
• Late fusion: both studies reported the late-fusion model beating its single-modality models §Late fusion.

Public datasets tab. 5: ADNI (MRI, PET, clinical and genetic data; used in 7 of the 13 public-data studies), TADPOLE, NACC, MIMIC-CXR, MIMIC-IV, the NCI data catalogue and MR CLEAN Trial.

Metrics tab. 6: accuracy 31 studies, sensitivity 20, AUC 17, specificity 15, precision 7, PPV/NPV 3, MCC 2, C-index 1, RMSE 1. Studies may report several metrics.

Comparison with earlier reviews tab. 1: the authors say the two reviews share no or only two studies with theirs, depending on the review.

Paper vs code

No official code. The paper has no code or project link, and the abstract and arXiv metadata name none. GitHub search (API, short queries): EHR medical imaging fusion review returned no results; fusion electronic health records imaging returned ngiesa/fusion_pod, the code of a different study on postoperative delirium, not by an author of this review. It was not used.

Pitfalls

• Source is a preprint. The source says it was accepted at Scientific Reports; the published version may differ. Cite the final version when it exists.
• Figure numbering: the source labels the third figure fig:4 (Fig4.png). In the compiled paper it is Figure 3, and that is the number used here. Fig3.png is in the source folder but never referenced in the text.
• Table numbering: the comparison table comes first in the text (tab:6) and is Table 1 in the compiled paper. The rest follow in source order.
• Neurological count: the header row in the disease table says 18, but its subcategories sum to 16 (AD 7, MCI 4, stroke 2, and one each for demyelinating, neurodevelopmental and epilepsy) and the text says 16. The total of 34 is consistent only with 16.
• Abstract prediction count: "10" is disease prediction only. All prediction tasks are 14 §Early Prediction.
• Country count: the text says 13 countries, the table lists 12, and the rows sum to 33, not 34 tab. 2.
• Private-data share: 21 of 34 is about 62%, but the text says about 59% §Patient data Resources.
• Misplaced sentence: the early-fusion item of the taxonomy also describes the two sub-types of joint fusion (type I and type II). The sub-types belong to joint fusion, not early fusion §Fusion strategies.
• Citation keys: the diagnosis list for neurological disorders ends with a bare ref4 key, which the introduction uses for a general fusion reference. Joint-fusion text cites the same key for two different studies (Chen et al. and Lili et al.). Some citations may be wrong; check each against the reference list before relying on it.
• Duplicate paragraph: the two opening paragraphs of Future directions are identical §Future directions.
• Missing appendices: the search strings and the extraction description are in supplementary Appendices 1 and 2. They are not in the source package, so the search cannot be rerun from this paper alone.
• Mixed performance metrics: studies use different metrics, tasks and datasets, and few report confidence bounds. The review does not compare scores across studies §Limitations.
• Selection bias: only English, peer-reviewed or conference studies in a short window. Positive results are likely over-represented §Limitations.
• Google Scholar: only the first 110 relevant results were used, so coverage of grey literature is partial §Search strategy.

Lineage

• Builds on: the fusion taxonomy (early, joint and late) comes from the earlier review by Huang et al. §Preliminaries §Fusion strategies. Huang et al. covered research up to 2019 and 17 studies. This review's stated difference is its 34 studies and conventional ML as well as DL §Introduction.
• Compared against: Liu et al. (multimodal EHR), and reviews of imaging-only fusion, omics fusion and biosignal fusion tab. 1.
• Superseded by: not known from this wiki.