A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics

In plain words

A doctor judging a chest X-ray also reads what the patient says and the blood test results, and combines all of it. Most AI diagnosis tools handle each kind of input separately, turn the text into structured fields first, and merge the separate answers at the end. IRENE instead turns the X-ray, the words of the complaint, the lab values and the patient's age and sex into one sequence of tokens, and a single Transformer relates them to each other. Parts of the image and words from the complaint can look at each other directly. The authors tested it on detecting eight lung diseases from chest X-rays, and on predicting ICU admission, ventilation and death for COVID-19 patients from CT scans and clinical data. They report better results than image-only models and older combined models.

TL;DR

• IRENE is one Transformer for clinical diagnosis. It reads a chest radiograph, a free-text chief complaint, structured lab values, age and sex as one set of tokens p. 9.
• Two bidirectional multimodal attention blocks let image tokens and text tokens attend within and across modalities. Ten self-attention blocks then run over the joined sequence p. 10 fig. 2a.
• Author-reported: mean AUROC 0.924 over eight pulmonary diseases, against 0.858 for Perceiver tab. 1. Mean AUPRC 0.592 over three COVID-19 outcomes, against 0.521 for early fusion tab. 2.
• The code is inference only. It has the model and the AUROC evaluation, but no training loop, and the pretrained weights are not in the repository irene.py.
• The method assumes every modality is present. The authors name missing modalities as an open problem p. 7.

Problem

Current multimodal diagnosis systems are "non-unified". They divide the input into images, narrative text and structured fields, extract features for each modality separately, and fuse them early or late p. 2. The authors argue that this structure cannot represent links between modalities, and that turning narrative text into structured fields by hand or with NLP tools is slow and labour-intensive p. 2 p. 7.

The method is meant for settings where the input has complementary information from several modalities. The authors' experiments need all modalities at once. Missing-modality handling is left to future work, which they suggest could use masked modelling p. 7.

Method

IRENE has three parts. First, each input is converted into tokens. Image patches become visual tokens, and each word, lab value and demographic field becomes a clinical text token p. 9 p. 10. Second, a Transformer called MDT processes the joint sequence: two bidirectional multimodal attention blocks followed by ten self-attention blocks p. 10 fig. 2a. Third, the unified tokens are averaged and passed to a classification head that produces one logit per disease or outcome p. 10.

Inputs, task 1 (pulmonary disease identification). A radiograph; the chief complaint (history of present and past illness); 92 laboratory-test values; sex; and age p. 9. Targets are eight pulmonary diseases p. 3.

Inputs, task 2 (COVID-19 adverse outcomes). A set of CT slices per patient; a structured chief complaint with 7 comorbidities and 9 symptoms; 19 lab-test values; sex; and age p. 7 p. 10. Targets are ICU admission, need for mechanical ventilation and death p. 4.

Tokens. The radiograph is cut into 16×16 patches by a convolution, and a learnable CLS token and learnable 1D positions are added p. 10 [code: models/embed.py#L44-L59]. Each chief-complaint word is embedded with a pretrained BERT model and then linearly projected p. 10. Each lab value is min-max scaled with statistics from the training set, then linearly projected as one token p. 7 p. 10. Sex and age are each linearly projected as one token p. 10 embed.py.

Bidirectional multimodal attention. For each modality, queries, keys and values come from a linear projection of the layer-normalised tokens p. 10. Each modality attends to itself (intra-modal) and to the other modality (inter-modal), and the two results are summed with weight λ = 1.0 p. 10. Both streams then go through a layer norm and an MLP, with a residual sum p. 10 fig. 2d.

Self-attention and head. The two streams are concatenated into one sequence p. 10 encoder.py. Ten self-attention blocks follow, each with 12 heads p. 10. A final layer norm is applied [code: models/encoder.py#L53]. Tokens are averaged and passed to a two-layer MLP p. 10.

Loss. Binary cross-entropy on the logits, one independent sigmoid per label p. 11 modeling_irene.py.

Equations are unnumbered in the PDF, so they are cited by page. Notation: LP is a linear projection, Norm is layer normalisation, and Att is scaled dot-product attention.

Q_I, K_I, V_I = \mathrm{LP}(\mathrm{Norm}(X_I)), \quad Q_T, K_T, V_T = \mathrm{LP}(\mathrm{Norm}(X_T)) p. 10

\tilde{X}_I = \mathrm{Att}(Q_I, K_I, V_I) + \lambda\,\mathrm{Att}(Q_I, K_T, V_T) p. 10

\tilde{X}_T = \mathrm{Att}(Q_T, K_T, V_T) + \lambda\,\mathrm{Att}(Q_T, K_I, V_I) p. 10

\mathrm{Att}(Q, K, V) = \mathrm{softmax}\!\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V,\quad d_k = 64 p. 10

X_I^{l+1} = \mathrm{MLP}(\mathrm{Norm}(\tilde{X}_I)) + X_I, \quad X_T^{l+1} = \mathrm{MLP}(\mathrm{Norm}(\tilde{X}_T)) + X_T p. 10

The PDF prints the residual terms as "+ +X_I" and "+ +X_T", and the layer index is missing on the right-hand side. The card reads them as the block input.

The forward pass for task 1, in pseudocode:

# IRENE forward pass, task 1 (pulmonary)  [p. 9-10] [fig. 2a]
# Input:  radiograph (B, 3, 224, 224), words (B, 40, 768),
#         labs (B, 92), sex (B, 1), age (B, 1)
# Output: logits (B, 8)

def irene_forward(radiograph, words, labs, sex, age):
    # 1. Image tokens: 16x16 patch convolution, then a CLS token and
    #    learnable 1D positions; 224/16 = 14, so 196 patches (derived)
    #    [p. 10] [p. 11]
    image_tokens = patch_conv(radiograph)           # (B, 196, 768)
    image_tokens = prepend_cls(image_tokens)        # (B, 197, 768)
    image_tokens = image_tokens + position_emb      # (B, 197, 768)

    # 2. Clinical text tokens: one per word, one per lab value, one each
    #    for sex and age; each is a linear projection  [p. 7] [p. 10]
    text_tokens = concat(
        linear_768(words),                 # BERT word features
        scalar_linear(labs),               # one token per lab value
        scalar_linear(sex),
        scalar_linear(age),
    )                                      # (B, 134, 768)

    # 3. Two bidirectional multimodal attention blocks  [p. 10] [fig. 2d]
    for block in bidirectional_blocks:     # 2 blocks
        image_tokens, text_tokens = block(image_tokens, text_tokens)

    # 4. Join both streams into one sequence  [p. 10]
    #    [code: models/encoder.py#L42-L45]
    tokens = concat(image_tokens, text_tokens)      # (B, 331, 768)

    # 5. Ten self-attention blocks over the joined sequence, 12 heads
    #    [p. 10] [fig. 2f]
    for block in self_attention_blocks:    # 10 blocks
        tokens = block(tokens)

    # 6. Average-pool the unified tokens, then predict  [p. 10]
    pooled = mean(tokens, axis=sequence)            # (B, 768)
    return classifier(pooled)                       # (B, 8)

The bidirectional block and its attention, in pseudocode:

# Bidirectional multimodal attention block  [p. 10] [fig. 2d-e]
# Input:  image (B, 197, 768), text (B, 134, 768)
# Output: image (B, 197, 768), text (B, 134, 768)

def bidirectional_block(image, text):
    # Scaled dot-product attention, d_k = 64  [p. 10]
    def attend(queries, keys, values):
        scores = queries @ transpose(keys) / sqrt(64)
        return softmax(scores, axis=-1) @ values

    # Queries, keys and values per modality, after layer norm  [p. 10]
    q_i, k_i, v_i = project_image(layer_norm(image))
    q_t, k_t, v_t = project_text(layer_norm(text))

    # Intra-modal plus inter-modal attention; lam = 1.0 in the paper  [p. 10]
    lam = 1.0
    image_attn = attend(q_i, k_i, v_i) + lam * attend(q_i, k_t, v_t)
    text_attn = attend(q_t, k_t, v_t) + lam * attend(q_t, k_i, v_i)

    # Layer norm, then an MLP per stream, with a residual sum  [p. 10]
    image = mlp_image(layer_norm(image_attn)) + image
    text = mlp_text(layer_norm(text_attn)) + text
    return image, text

Hyperparameters

Values are from the paper unless the Where column says code. "not stated" means the paper does not give a value.

Results

Author-reported. Setup: a retrospective pulmonary cohort of 51,511 patients from West China Hospital, 2008 to 2019, with 72,283 samples over eight diseases, split by admission date into 44,628 training, 3,325 validation and 3,558 test patients p. 3. A COVID-19 CT cohort from 17 hospitals, with 1,164 training, 498 validation and 700 external test cases from 9 centres, and 155, 94 and 59 events in the test set for ICU, mechanical ventilation and death p. 4. Baselines: image-only ViT; early fusion and late fusion of an image model with classical models; GIT-Base fine-tuned; Perceiver p. 7 p. 8 p. 9.

Task 1: pulmonary disease identification, AUROC tab. 1. Mean is the average over eight diseases. Each cell is the value with its 95% CI.

The abstract's headline gains are differences in mean AUROC: IRENE minus image-only is about 0.12, and minus early fusion about 0.09 p. 1 tab. 1.

Task 2: adverse COVID-19 outcomes, AUPRC tab. 2.

The text reports a mean gain of 29 points over image-only, 7 over early fusion and 9 over late fusion, and more than 6 points over the two transformer baselines on average p. 4. Those are differences in AUPRC.

Ablation tab. 3 p. 5. Descriptions follow the paper's text. The check marks in the PDF table are not reproduced.

The text highlights three drops: removing the radiograph (row 7), replacing the bidirectional blocks with self-attention (row 1), and removing the chief complaint (row 4) p. 5.

Qualitative attention analysis fig. 3. For one COPD case, the authors report attention shares of 79.8% for the radiograph, 16.3% for the chief complaint, 2.4% for lab tests and 1.5% for demographics fig. 3a. Cross-attention maps are shown for three words of the complaint. The authors say these maps offer clinical insight only "to a certain extent" p. 5.

Human comparison fig. 1e p. 4. Average performance of two junior and two senior physicians on the eight pulmonary diseases. The authors report IRENE above the junior group on all eight diseases, and comparable to or better than the senior group on six p. 2 p. 4.

Paper vs code

Code: RL4M/IRENE at 128c822, Apache-2.0. The repository is cited in the paper's Code availability statement p. 12. It contains the model, a loader for pickled clinical data, and an AUROC evaluation script. It has no training loop, no pretraining code and no weights.

1. Classification head. Paper: two-layer MLP after averaging p. 10. Code: a single linear layer on the mean token [code: models/modeling_irene.py#L57] modeling_irene.py.
2. Cross-attention combination. Paper: intra-modal plus λ times inter-modal, with λ = 1.0 p. 10. Code: the two terms are averaged (divided by 2) before the output projection attention.py. Because the output projection is linear with a bias, the code equals the paper's sum only if that projection's weight is halved, with the bias unchanged. Pretrained weights follow the code's convention.
3. Residual placement. The paper's equations add the block input only after the MLP p. 10. The code adds the block input after the attention step and again after the MLP block.py [code: models/block.py#L75-L76].
4. Separate output projection. The code declares an output projection for the text stream, but the forward pass does not use it. Both streams share one output projection attention.py [code: models/attention.py#L115-L116]. The paper does not describe either choice.
5. Separate MLPs per stream. Code uses a separate text MLP [code: models/block.py#L43] block.py. Fig. 2d draws two MLPs, but the equations use one MLP symbol p. 10 fig. 2d.
6. Learned positions for text tokens. The code adds learnable positional embeddings to every text token type embed.py. The paper describes positions only for image patches p. 10.
7. Task 2 configuration. Paper: N_cc = 16 and N_lab = 19 for task 2 p. 10. Code: fixed at 40 and 92 configs.py [code: irene.py#L25] irene.py. The shipped code cannot run task 2 as configured.
8. Radiograph resize. Paper: resize to 256×256 p. 11. Code: Resize(256), which scales the shorter side to 256 and keeps the aspect ratio, then a 224 centre crop irene.py.
9. Attention dropout. Not stated for IRENE in the paper. Code uses 0.2 for attention and 0.3 elsewhere configs.py.
10. Weights. The code loads model.pth irene.py, which is not in the repository, so the reported results cannot be reproduced from the code alone.
11. Optimizer in the test script. The test script creates AdamW with learning rate 3e-5 and weight decay 0.01 irene.py. It never calls step(), so these values have no effect on inference. The training schedule is not in the repository.
12. Initialisation. MLP layers use Xavier-uniform weights and near-zero biases mlp.py. The paper does not mention initialisation.

Attention score scaling agrees: both use 1/√64 attention.py p. 10.

Pitfalls

• Padding is not masked. Chief complaints shorter than 40 words are zero-padded p. 7, and non-existent text is zero-padded during pretraining p. 11. The paper does not say whether padded tokens are masked in attention, and the code's attention takes no mask attention.py. Padded tokens take part in every softmax unless you add a mask.
• Missing lab values. Lab values are min-max scaled to [0, 1] with training-set statistics, and -1 means missing p. 7. The paper does not say whether -1 is assigned before or after scaling. Assigning it after scaling keeps the token distinct from any real value.
• Lab feature list. The 92 lab-test items and the 19 task-2 items are listed in supplementary notes that are not in the PDF p. 7. Feature order and names cannot be reconstructed from the paper.
• BERT citation. The text cites reference 23 for BERT p. 10. Reference 23 is a breast-cancer paper. BERT is reference 25, which the baseline section cites correctly p. 8. The model size is not stated, so the feature source is open. The paper does not state the dimension, and the code expects 768 [code: models/embed.py#L48].
• Units differ. The pulmonary dataset has 72,283 samples but 51,511 patients, and the split counts are given per patient p. 3. The paper does not say how many samples a patient contributes or whether one sample is one radiograph.
• Metrics are differences in points. The headline gains are absolute differences in AUROC or AUPRC, not relative gains p. 1 tab. 1 tab. 2.
• Confidence intervals and seeds. CIs come from resampling the test set p. 11. They do not include training variance. The significance test uses five reruns in the text p. 12, but the Fig. 1 caption says ten reruns fig. 1. Only one set of reruns is reported.
• Compute grows with slices. Task 2 runs the MDT once per CT slice, with 16 slices per patient p. 10 p. 11, then pools. Task 1 attends over 331 tokens per image. Attention cost grows with the square of the token count in every block.
• Data is not public. The data restrictions and access on request are stated p. 12. The repository has no data and no way to rebuild the cohorts, so external validation is not possible from this release.
• External CT model. The COVID-19 CT subvolumes depend on a separate lesion segmentation model trained on 14,435 participants p. 11. That model is not in the repository.
• Supplementary material missing. The text refers to Extended Data Fig. 1 and Supplementary Notes 1 to 3 p. 5 p. 7. None of these is in the PDF.
• Preprint vs journal. The code README cites the Nature Biomedical Engineering version README.md. This card follows arXiv v1. Numbers, equations or the code may differ in the journal version.
• Attention maps are not localisation. The authors say cross-attention maps cannot identify the image regions that match a keyword p. 5. Treat them as qualitative.
• Cross-attention weights. The λ value was chosen on preliminary experiments, and the text gives no sensitivity study for λ p. 10.

Lineage

• Builds on: Vision Transformer (ViT; arXiv:2010.11929, link) for the patch backbone p. 8; BERT (arXiv:1810.04805, link) for word features p. 10; the Transformer [ref. 24] for self-attention blocks p. 10. The text tokens and the joint sequence are IRENE's own design.
• Compared against: Perceiver [ref. 30], whose IO variant is Perceiver IO (arXiv:2107.14795, link) p. 9; GIT (arXiv:2205.14100, link) p. 9; early and late fusion of image and text models p. 8.
• Superseded by: not known from content already in this wiki.