On the Automatic Generation of Medical Imaging Reports

In plain words

Radiologists look at an x-ray and then type a short report: a summary verdict, a few sentences on each part of the body they checked, and a list of keywords. Writing these reports is slow for experts and error-prone for beginners, so this paper asks whether a neural network can draft them.

The model works in three stages. First, an image classifier guesses which medical keywords ("tags") fit the image. Second, before writing each sentence, the model decides where to look, both in the image and in its list of guessed keywords. Looking at both at once is the "co-attention" in the title: the keywords tell it what kind of finding to describe, the image regions tell it where. Third, the text is written in two levels: an outer loop picks a topic for the next sentence and decides when the report is finished, and an inner loop writes the words of that sentence.

The two-level writer is what makes long, multi-sentence reports work: a plain single-loop captioner, built for one-sentence image captions, scores clearly worse on report paragraphs. One caveat: the model was scored with standard captioning metrics that count overlapping words, not by checking whether the medical findings are correct.

TL;DR

A CNN-RNN report generator for medical images with three pieces: (1) multi-label tag prediction from VGG-19, (2) co-attention over 196 VGG-19 region features and the embeddings of the 10 most likely tags, and (3) a hierarchical LSTM (sentence LSTM producing topic vectors plus a stop signal, word LSTM generating each sentence). All losses are summed with weight 1.0. On IU X-Ray (impression + findings as one paragraph) it reports BLEU-4 0.247 / CIDEr 0.327 against 0.118 / 0.302 for a re-implemented Show-Attend-Tell; on single-sentence PEIR Gross captions the gain is smaller. No official code.

Problem

A radiology report mixes heterogeneous outputs: an impression sentence, a multi-sentence findings paragraph, and a keyword list §1 fig. 1. Three difficulties are named: producing these different outputs in one framework, localizing abnormal regions and attaching the right description to them, and generating long multi-sentence text, which a single-layer LSTM captioner handles poorly §1. The setting assumes paired image/report data with keyword tags (MTI tags on IU X-Ray, tf-idf-derived tags on PEIR Gross) §4.1.

Method

The encoder is a VGG-19 CNN. Its last convolutional feature map gives N region features v_n; its last two fully connected layers act as the multi-label classifier (MLC) over L tags §3.2. The embeddings of the M most likely tags become semantic features a_m §3.2. At each sentence step s, a co-attention network computes soft attention over the regions and over the tags, conditioned on the previous sentence-LSTM hidden state, and fuses both attended vectors into a joint context vector §3.3. The sentence LSTM consumes the context vector, outputs a topic vector through a "deep output" layer and a STOP/CONTINUE probability from the previous and current hidden states §3.4. The word LSTM receives the topic vector as its first input, the START token second, then the words, and predicts each word from its hidden state §3.5. The final report is the concatenation of all sentences §3.5. An overview diagram shows the image passing through the CNN to the MLC, the predicted tags' embeddings and the region features entering co-attention, and the sentence and word LSTMs below fig. 2.

Steps:

1. Tag prediction §3.2 eq. 1: p(l_i = 1 | {v_n}) ∝ exp(MLC_i({v_n})), a softmax over all L tags (not independent sigmoids). The training loss is cross-entropy against the L1-normalized ground-truth tag vector §3.6, described as a softmax cross-entropy multi-label ranking loss §4.2.
2. Semantic features §3.2 §4.2: take the M = 10 highest-scoring tags and look up their embeddings a_m (a tag embedding matrix separate from the word embedding matrix, since a tag can contain several words).
3. Co-attention §3.3 eq. 2 eq. 3 eq. 4:

\alpha_{\mathbf{v},n} \propto \exp(\mathbf{W}_{\mathbf{v}_{att}}\tanh(\mathbf{W}_\mathbf{v}\mathbf{v}_n + \mathbf{W}_{\mathbf{v},\mathbf{h}}\mathbf{h}_{sent}^{(s-1)})) eq. 2

\alpha_{\mathbf{a},m} \propto \exp(\mathbf{W}_{\mathbf{a}_{att}}\tanh(\mathbf{W}_\mathbf{a}\mathbf{a}_m + \mathbf{W}_{\mathbf{a},\mathbf{h}}\mathbf{h}_{sent}^{(s-1)})) eq. 3

\mathbf{v}_{att}^{(s)} = \sum_{n=1}^N\alpha_{\mathbf{v},n}\mathbf{v}_n, \quad \mathbf{a}_{att}^{(s)} = \sum_{m=1}^M\alpha_{\mathbf{a},m}\mathbf{a}_m §3.3, unnumbered

\mathbf{ctx}^{(s)} = \mathbf{W}_{fc}[\mathbf{v}_{att}^{(s)}; \mathbf{a}_{att}^{(s)}] eq. 4

4. Topic generator §3.4 eq. 5:

\mathbf{t}^{(s)} = \tanh(\mathbf{W}_{\mathbf{t}, \mathbf{h}_{sent}}\mathbf{h}_{sent}^{(s)} + \mathbf{W}_{\mathbf{t}, \mathbf{ctx}}\mathbf{ctx}^{(s)})

5. Stop control §3.4 eq. 6; stop when the probability exceeds a threshold (0.5 §4.2):

p(STOP|\mathbf{h}_{sent}^{(s-1)}, \mathbf{h}_{sent}^{(s)}) \propto \exp\{\mathbf{W}_{stop}\tanh(\mathbf{W}_{stop, s-1}\mathbf{h}_{sent}^{(s-1)}+\mathbf{W}_{stop, s}\mathbf{h}_{sent}^{(s)})\}

6. Word LSTM §3.5 eq. 7: p(word | h_word) ∝ exp(W_out h_word).
7. Loss §3.6 eq. 8 eq. 9, with the stop target I{s = S} (STOP only after the last ground-truth sentence):

\ell(I, \mathbf{l}, \mathbf{w}) = \lambda_{tag}\ell_{tag} + \lambda_{sent}\sum_{s=1}^S\ell_{sent}(p_{stop}^{s}, I\{s=S\}) + \lambda_{word}\sum_{s=1}^{S}\sum_{t=1}^{T_s}\ell_{word}(p_{s,t}, w_{s,t}) eq. 8

\ell_{reg} = \lambda_{reg}\Big[\sum_n^N\big(1-\sum_s^S \alpha_{n,s}\big)^2 + \sum_m^M\big(1-\sum_s^S \beta_{m,s}\big)^2\Big] eq. 9

where α ∈ R^{N×S} and β ∈ R^{M×S} stack the visual and semantic attention weights over all sentence steps §3.6.

# Forward pass and loss for one (image, tags, paragraph) example  [§3]
# Input:  image (3, 224, 224) assumed; tag_vector (L,) binary;
#         sentences: list of S word-id lists
# Output: scalar loss

def report_loss(image, tag_vector, sentences):
    # VGG-19 last conv map -> N = 14*14 region features  [§3.2] [§4.2]
    regions = vgg19_last_conv(image)              # (196, 512)
    # Tag distribution from VGG-19's last two FC layers  [§3.2] [eq. 1]
    tag_logits = vgg19_fc_layers(image)           # (L,)
    tag_probs = softmax(tag_logits)
    # Target: binary tag vector normalized to sum 1  [§3.6]
    tag_target = tag_vector / sum(tag_vector)
    loss_tag = cross_entropy(tag_target, tag_probs)
    # Embeddings of the 10 most likely tags  [§3.2] [§4.2]
    top_tags = argsort_descending(tag_probs)[:10]
    tag_feats = tag_embedding[top_tags]           # (10, E)

    # Initial sentence-LSTM state: not stated in the paper
    h_prev, cell = initial_sentence_state()
    loss_sent = 0
    loss_word = 0
    visual_attn = []
    semantic_attn = []
    for s in range(len(sentences)):
        # Additive attention over regions and over tags  [§3.3] [eq. 2-3]
        alpha = softmax(
            w_v_att @ tanh(w_v @ regions + w_vh @ h_prev))   # (196,)
        beta = softmax(
            w_a_att @ tanh(w_a @ tag_feats + w_ah @ h_prev)) # (10,)
        visual_ctx = weighted_sum(alpha, regions)
        semantic_ctx = weighted_sum(beta, tag_feats)
        # Fuse by concatenation + FC layer  [§3.3] [eq. 4]
        context = w_fc @ concat(visual_ctx, semantic_ctx)
        # One step of the single-layer sentence LSTM  [§3.4]
        h_sent, cell = sentence_lstm(context, (h_prev, cell))
        # Topic vector via deep output layer  [§3.4] [eq. 5]
        topic = tanh(w_t_h @ h_sent + w_t_ctx @ context)
        # STOP vs CONTINUE from previous and current state  [§3.4] [eq. 6]
        stop_logits = w_stop @ tanh(
            w_stop_prev @ h_prev + w_stop_cur @ h_sent)
        is_last = 1 if s == len(sentences) - 1 else 0
        loss_sent += cross_entropy(is_last, softmax(stop_logits))
        # Word LSTM: topic first, START second, then the words  [§3.5]
        inputs = [topic, embed(START)] + embed(sentences[s][:-1])
        word_states = word_lstm(inputs)
        # Each word state predicts the next word  [§3.5] [eq. 7]
        word_logits = w_out @ word_states[1:]
        loss_word += cross_entropy(sentences[s], softmax(word_logits))
        visual_attn.append(alpha)
        semantic_attn.append(beta)
        h_prev = h_sent

    # Attention coverage penalty over sentence steps  [§3.6] [eq. 9]
    loss_reg = (sum((1 - sum_over_steps(visual_attn)) ** 2)
                + sum((1 - sum_over_steps(semantic_attn)) ** 2))
    # All weights lambda set to 1.0  [§4.2] [eq. 8]
    return loss_tag + loss_sent + loss_word + loss_reg

The word-LSTM input alignment above (which output predicts which word, and whether the sentence ends with an end token) is not stated in the paper; it follows the description that the topic and START are the first two inputs §3.5.

Hyperparameters

Results

Author-reported. Setup: IU X-Ray (7,470 image–report pairs; target is impression and findings concatenated into one paragraph) and PEIR Gross (7,442 image–caption pairs from 21 sub-categories, one sentence each) §4.1. Baselines were re-implemented by the authors with VGG-19 encoders §4.3. Metrics from the coco-caption toolkit §4.4. Full table tab. 1:

Ours-no-Attention takes the 4096-d global VGG-19 feature into the hierarchical decoder, so it differs from CNN-RNN only in the decoder §4.3. The ablations Semantic-only and Visual-only feed only one attended vector into the topic generator §4.3.

Share of generated sentences describing normal vs abnormal findings on IU X-Ray, where a sentence counts as "normal" if it contains "no", "normal", "clear" or "stable" §4.5.1 tab. 2:

Qualitatively, generated paragraphs typically open with a global statement and then cover one anatomical area per sentence; the paper shows one partial failure (abnormal area found, abnormality mis-described, extra findings on a darker image) and one failure case §4.5.1 fig. 3. Attention visualizations show different regions and tags attended per sentence, with one mis-description traced to wrong semantic attention §4.5.2 fig. 4.

Paper vs code

No official code.

Pitfalls

• Tag loss is a softmax, not per-tag sigmoids. Eq. 1 normalizes over all L tags, and the target is the binary tag vector divided by its L1 norm §3.6 §4.2. An implementation with BCE per tag is a different model. The tag probabilities are only used to rank and select the top 10; IU X-Ray images average 2.2 ground-truth tags §4.1, so most of the 10 attended tags are usually not ground-truth tags.
• Top-k tag selection is non-differentiable. Gradients reach the MLC only through ℓ_tag; whether gradients from the decoder flow into the selected tag embeddings or into VGG-19 through the region features is not stated. Whether ground-truth or predicted tags are fed during training is also not stated (the text says "predicted").
• Undefined dimensions. Only "all hidden states and word embeddings" are given as 512 §4.2; the tag embedding size E, context size C, topic size K and the attention hidden size are not separately stated. Sentence-LSTM and word-LSTM initial states, and whether one word LSTM is shared across sentences, are not stated.
• Stop control is a 2-way softmax using h^(s-1) and h^(s) eq. 6; at s = 1 the "previous" state is the initial state, whose definition is not stated. The stop target is 1 only at the last ground-truth sentence eq. 8. No maximum sentence count is given for inference, so set one to avoid runaway generation.
• Attention regularizer eq. 9 pushes each region's attention, summed over sentence steps, towards 1. With N = 196 regions and about 5.7 sentences on average §4.1, the total visual attention mass is S, so the visual term cannot reach zero and mostly acts as a spreading penalty; for the 10 tags it is closer to satisfiable. Normalization by S or N is not described.
• Learning rates differ by module (1e-5 for VGG-19, 5e-4 for the LSTMs) §4.2; using one rate for both is a deviation. The MLC layers' rate is not stated separately.
• Evaluation split. IU X-Ray splits are random 500/500 images §4.1; the paper does not say whether images sharing a report or patient are kept in the same split. Results on other splits of IU X-Ray are not directly comparable.
• Impression is part of the target. Impression and findings are concatenated into one paragraph §4.1, so scores are not comparable to findings-only setups.
• Metrics are n-gram overlap, not clinical accuracy. The only clinical-flavoured analysis is the keyword-based normal/abnormal sentence ratio tab. 2, which is a crude heuristic (e.g. "no" also appears in abnormal sentences).
• On PEIR Gross the hierarchical decoder alone does not help: Ours-No-Attention scores about the same as CNN-RNN tab. 1; the gain there comes from attention.

Lineage

• Builds on: Show, Attend and Tell (arXiv:1502.03044) for soft visual attention and the attention-coverage regularizer §3.3 §3.6; Krause et al. 2017 hierarchical paragraph captioning for the topic-then-START word-LSTM input scheme §3.5; deep output layers from Pascanu et al. (arXiv:1312.6026) §3.4; VGG-19 (arXiv:1409.1556) §3.2.
• Compared against: CNN-RNN (Vinyals et al., arXiv:1411.4555), LRCN (Donahue et al., arXiv:1411.4389), Soft ATT (arXiv:1502.03044), ATT-RK (You et al., arXiv:1603.03925), all re-implemented §4.3.