Paper · 2020-04-26 · cs.CL, cs.CV, eess.IV
Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports
Generates chest X-ray reports in two stages: Findings from the image, then an Impression from the image and those findings. Each section is written by three cooperating agents: a planner picks stop, normal writer or abnormal writer, and two LSTM writers produce sentences. Trained by imitation, then REINFORCE on BLEU rewards. Author-reported Findings BLEU-4: 0.545 on CX-CHR, 0.154 on IU-Xray.
Image captioningPolicy gradientMulti-agent reinforcement learningImitation learningmedical-report-generationchest-x-rayhierarchical-generationmulti-agentreinforceabnormality-detection
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A chest X-ray report has two parts. The Findings part describes what the scan shows, both the normal parts and the problems. The Impression part sums up the diagnosis based on those findings. Most sentences in a report describe normal things, so a model that learns from all sentences in one pass tends to repeat normal statements.
The authors split the job in two. First the model writes the Findings from the image. Then it writes the Impression from the image and the Findings it just wrote. For the Findings, three components take turns. A planner decides whether to stop, or which of two writers speaks next: one writer handles normal sentences and the other handles abnormal ones. The model is first trained to copy reference reports, then tuned to score higher on a text-overlap metric.
The authors report better overlap scores than earlier methods on two datasets. They also measure how often the model names the abnormal findings, and they note that overlap scores alone can be high for a model that writes mostly normal text.
TL;DR
- Generates chest X-ray reports in two stages: the Findings section from the image, then the Impression from the image and the generated Findings §3.
- Each section is written by a Co-operative Multi-Agent System (CMAS): a Planner that chooses STOP, the Normality Writer or the Abnormality Writer, plus two single-layer LSTM writers §4.1, §4.2.
- Trained first by cross-entropy imitation, then by REINFORCE with a discounted BLEU-based reward §4.3, §4.4.
- Author-reported, Findings: CMAS-RL reaches BLEU-4 0.545 on CX-CHR (best baseline HGRG-Agent 0.486) and 0.154 on IU-Xray (tied with CoAtt) tab. 1.
- No official code was found, so the implementation details here come only from the paper's text.
Problem
A report has two sections: Findings, which lists normal and abnormal observations, and Impression, which states a diagnostic conclusion drawn from the Findings §1 fig. 1. Earlier methods ignored the relationship between the two sections and the different distributions of normal and abnormal sentences §1.
The normal sentences of a report are usually 4 to 12 times as many as the abnormal ones §4.2.3. A single decoder trained on all sentences therefore tends to produce normal text §4.2.3 §5.5.
BLEU and similar overlap scores can be high for a model that writes normal findings, so the paper also reports precision and false positive rate for abnormality terms §5.4. The abnormality labels rely on patterns that human experts wrote for the most frequent abnormality terms §5.2. The patterns are not given in the paper.
Method
Two-stage framework §3 fig. 3
The Findings module examines areas of the chest X-ray image and writes a description for each §3. The Impression module takes the image and the generated Findings, and writes a conclusion §3 §4.5. The paper says the Impression is a summary of the Findings, and borrows ideas from image captioning and text summarisation for it §1. Both modules are CMAS instances §3.
CMAS overview §4.1 fig. 2
CMAS has three agents: Planner (PL), Normality Writer (NW) and Abnormality
Writer (AW) §4.1. Generation runs in loops. In loop n, the writers share
their local states, which are the words of the previous sentence, into a
global state GS_n = (I, {s_i}_{i=1}^{n-1}), where I is the image and s_i
is the i-th sentence §4.1. PL reads the global state and chooses STOP or
one writer. The chosen writer refreshes its memory from the global state and
writes the next sentence §4.1. The loop ends when PL outputs STOP §4.2.2.
# Generation loop of CMAS for one section [§4.1] [§4.2]
# Input: visual features v (P, C); findings f (Impression only)
# Output: list of generated sentences
def generate_section(v, findings=None):
# 1. Start the global state from the mean visual vector [§4.2.1]
gs, cell = initial_states(mean(v))
# 2. The local state of the last writer; not defined before loop 1
ls = None
sentences = []
while True:
# 3. Update the global state with a new context [eq. 1] [eq. 5]
ctx = context(v, findings, ls, gs)
gs = gse_lstm(gs, ctx)
# 4. Planner picks STOP (0), normal writer (1) or abnormal (2)
# [§4.2.2] [eq. 8] [eq. 9]
idx = planner(gs)
if idx == 0:
break
# 5. The chosen writer writes one sentence from a fresh memory
# initialised with gs and a BOS token [§4.2.3]
writer = nw if idx == 1 else aw
sentence, ls = writer.generate(gs)
sentences.append(sentence)
return sentencesGlobal State Encoder §4.2.1
A single-layer LSTM with soft attention encodes the global state §4.2.1.
The three agents share its weights §4.2.1. At each loop the encoder takes a
multi-modal context vector ctx_n in dimension H, obtained by embedding the
previous sentence's local state and the image §4.2.1:
\mathbf{gs}_n = \text{LSTM}(\mathbf{gs}_{n-1}, \mathbf{ctx}_n) eq. 1
The attention over P visual vectors v_p in C channels §4.2.1:
\mathbf{h}_{p} = \tanh(\mathbf{W}_h[\mathbf{ls}_{n-1}; \mathbf{gs}_{n-1}]) eq. 2
\alpha_{p} = \frac{\exp(\mathbf{W}_{att}\mathbf{h}_{p})}{\sum_{q=1}^{P}\exp(\mathbf{W}_{att}\mathbf{h}_{q})} eq. 3
\mathbf{v}_{att} = \sum_{p=1}^P\alpha_{p}\mathbf{v}_p eq. 4
\mathbf{ctx}_{n} = \tanh(\mathbf{W}_{ctx}[\mathbf{v}_{att}; \mathbf{ls}_{n-1}]) eq. 5
The initial states come from the mean visual vector v̄ = (1/P) Σ v_i
§4.2.1:
\mathbf{gs}_0 = \tanh(\mathbf{W}_{gs}\bar{\mathbf{v}}) eq. 6
\mathbf{c}_{0} = \tanh(\mathbf{W}_{c}\bar{\mathbf{v}}) eq. 7
Planner §4.2.2
PL is the shared encoder followed by a two-layer feed-forward network §4.2.2.
Its output is an index idx_n in {0, 1, 2}: 0 is STOP, 1 is NW and 2 is AW
§4.2.2:
\mathbf{h}_n = \tanh(\mathbf{W}_2\tanh(\mathbf{W}_1\mathbf{gs}_n)) eq. 8
idx_n = \arg\max(\text{softmax}(\mathbf{W}_3\mathbf{h}_n)) eq. 9
Writers §4.2.3
NW and AW are each a single-layer LSTM with hidden size H §4.2.3. Their
architectures may differ, but the paper uses the same design for both
§4.2.3. The chosen writer takes gs_n and a BOS token as its first inputs
§4.2.3. It then generates words until it emits EOS or reaches the maximum
step T:
\mathbf{h}_{t} = \text{LSTM}(\mathbf{h}_{t-1}, \mathbf{W_e}\mathbf{y}_{w_{t-1}}) eq. 10
\mathbf{p}_t = \text{softmax}(\mathbf{W}_{out}\mathbf{h}_t) eq. 11
w_{t} = \arg\max(\mathbf{p}_t) eq. 12
The last hidden state of the writer becomes the local state ls_n, which
feeds the encoder in the next loop §4.2.3.
Reward §4.3
The reward is BLEU-based. The opening of §4.3 and §1 say BLEU-4, and the text after eq. 14 uses BLEU-1 (see Pitfalls). Eqs. 13 and 14 compute BLEU separately over the abnormal sentences and over the normal sentences of a report §4.3:
f(s^{ab}_n) = \text{BLEU}(\{s^{ab}_1,\cdots, s^{ab}_n\}, \mathbf{s}^{\ast ab}) eq. 13
f(s^{nr}_n) = \text{BLEU}(\{s^{nr}_1,\cdots, s^{nr}_n\}, \mathbf{s}^{\ast nr}) eq. 14
The immediate reward of sentence n is r(s_n) = f(s_n) - f(s_{n-1}), and
r(s_1) = BLEU({s_1}, s*) §4.3. The discounted reward is:
R(s_n) = \sum_{i=0}^{\infty}\gamma^ir(s_{n+i}) eq. 15
where γ ∈ [0, 1] §4.3.
# Discounted reward per generated sentence [§4.3] [eq. 15]
# Input: generated sentences, reference sentences, discount gamma
# Output: one discounted reward per generated sentence
def discounted_rewards(sentences, references, gamma):
# 1. BLEU of the first n generated sentences against the reference
# [§4.3]; the text after eq. 14 uses BLEU-1 here
scores = [bleu(sentences[:n], references)
for n in range(1, len(sentences) + 1)]
# 2. Immediate reward: the BLEU gain when sentence n is added [§4.3]
immediate = [scores[0]] + [scores[n] - scores[n - 1]
for n in range(1, len(scores))]
# 3. Discounted sum of the current and all later immediate rewards
# [eq. 15]; a finite report truncates the infinite sum
returns = []
for n in range(len(immediate)):
returns.append(sum(gamma ** (m - n) * immediate[m]
for m in range(n, len(immediate))))
return returnsLearning: reinforcement learning §4.4.1
The loss of a report is the negative expected reward §4.4.1:
L(\theta) = -\mathbb{E}_{n,s_n\sim\pi_\theta}[R(s_n)] eq. 16
Following REINFORCE §4.4.1, the gradient is:
\nabla_\theta L(\theta) = \mathbb{E}_{n, s_n\sim\pi_\theta}[R(s_n)\nabla_\theta -\log \pi_\theta(s_n, idx_n)] eq. 17
The joint negative log-likelihood splits into a writer part and a planner part §4.4.1:
-\log\pi_\theta(s_n, idx_n) = \mathbb{1}_{\{idx_n=AW\}}L_{AW} + \mathbb{1}_{\{idx_n=NW\}}L_{NW} + L_{PL} eq. 18
with L_AW = -Σ_t log p_AW(w_{n,t}), L_NW = -Σ_t log p_NW(w_{n,t}) and
L_PL = -log p_PL(idx_n) eq. 18.
\nabla_\theta L(\theta) = \mathbb{E}[R(s_n)(\mathbb{1}_{\{idx_n=AW\}}\nabla L_{AW} + \mathbb{1}_{\{idx_n=NW\}}\nabla L_{NW} + \nabla L_{PL})] eq. 19
# REINFORCE loss for one report [§4.4.1] [eq. 16] [eq. 18] [eq. 19]
# Input: samples (sentence, idx, R) for each generated sentence
# Output: scalar loss; minimise it with Adam
def reinforce_loss(samples):
loss = 0.0
for sentence, idx, R in samples:
# 1. Planner's negative log-likelihood of the chosen action
nll = planner_nll(idx)
# 2. Add the writer's word-level negative log-likelihood [eq. 18]
if idx == 1:
nll += word_nll(nw, sentence)
if idx == 2:
nll += word_nll(aw, sentence)
# 3. Weight by the discounted reward; no baseline is stated
loss += R * nll
return lossLearning: imitation learning §4.4.2
Before reinforcement learning, the policy is pre-trained with cross-entropy on reference reports §4.4.2:
L_{CE}(\theta) = -\lambda_{PL}\sum_{n=1}^{N}\log p_{PL}(idx_n^\ast) - \lambda_{NW}\sum_{n=1}^{N}\mathbb{1}_{\{idx_n^\ast=NW\}}\sum_{t=1}^{T}\log p_{NW}(w_{n,t}^\ast) - \lambda_{AW}\sum_{n=1}^{N}\mathbb{1}_{\{idx_n^\ast=AW\}}\sum_{t=1}^{T}\log p_{AW}(w_{n,t}^\ast) eq. 20
Here w* and idx* are the reference words and the reference writer label of
each sentence §4.4.2. All three λ are 1.0 §5.2.
# Imitation loss on reference sentences [§4.4.2] [eq. 20]
# Input: reference sentences with gold writer labels (0 is not used)
# Output: scalar loss
def imitation_loss(references, lam_pl, lam_nw, lam_aw):
loss = 0.0
for sentence, gold_idx in references:
# 1. Planner is trained to predict the gold writer [eq. 20]
loss += lam_pl * planner_nll(gold_idx)
# 2. Only the writer of the gold type learns this sentence
if gold_idx == 1:
loss += lam_nw * word_nll(nw, sentence)
if gold_idx == 2:
loss += lam_aw * word_nll(aw, sentence)
return lossImpression module §4.5
The Impression module uses the same CMAS, with a richer global state
GS_n = (I, f, {s_i}), where f is the list of generated Findings sentences
§4.5. The context vector gains a second attention vector over the Findings
§4.5:
\mathbf{ctx}_{n} = \tanh(\mathbf{W}_{ctx}[\mathbf{v}_{att}; \mathbf{f}_{att}; \mathbf{ls}_{n-1}]) eq. 21
The paper says f_att is computed like v_att in eqs. 2 and 3 §4.5. The
rest of the Impression module is the same as the Findings module §4.5.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Hidden size of GSE, PL, NW and AW | 512 | §5.2 | |
| Word embedding dimension | 512 | §5.2 | |
| Image encoder | ResNet-50, pretrained on ImageNet | §5.2 | |
| Visual features | last conv layer, 7 × 7 × 2048 | §5.2 | §4.2.1 says "an intermediate layer" (see Pitfalls) |
| Encoder fine-tuning, IU-Xray | on ChestX-ray14, then frozen | §5.2 | IU-Xray is "too small" to fine-tune on alone |
| Encoder fine-tuning, CX-CHR | on its own training set, then frozen | §5.2 | |
| Imitation loss weights λ_PL, λ_NW, λ_AW | 1.0 each | §5.2 eq. 20 | |
| Imitation optimiser and learning rate | Adam, 5 × 10⁻⁴ | §5.2 | both datasets |
| RL optimiser and learning rate | Adam, 10⁻⁶ fixed | §5.2 | both datasets |
| Discount γ | not stated | §4.3 | range is [0, 1] |
| Reward metric | BLEU-4 (opening of §4.3); BLEU-1 (text after eq. 14) | §4.3 | conflicting, see Pitfalls |
| Maximum sentences per report, maximum words T | not stated | §4.2.3 | |
| Vocabulary and tokenisation | not stated for IU-Xray beyond lower-casing and non-alpha removal; CX-CHR uses jieba and rare-token filtering | §5.1 | rare-token threshold not stated |
| Batch size, epochs, stopping rule | not stated | §5 | |
| Decoding | arg max at each step | eq. 9 eq. 12 | beam search not mentioned |
| Dropout, weight decay, gradient clipping | not stated | §5 | |
| Hardware and run time | not stated | §5 |
Results
Author-reported. Setup: two chest X-ray report datasets, ResNet-50 image encoder, and BLEU-1 to BLEU-4, ROUGE and CIDEr from the standard captioning tool §5.1 §5.2. Baselines are CNN-RNN, LRCN, AdaAtt, Att2in, CoAtt and HGRG-Agent for Findings §5.2, and SoftAtt (text-only and vision-only) for Impression §5.2. CMAS-W, CMAS-NW,AW, CMAS-IL and CMAS-RL are the paper's own variants §5.2. The paper runs one configuration per dataset and reports no variance over seeds.
Table 1, Findings generation tab. 1. Values are copied from the paper.
| Dataset | Method | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE | CIDEr |
|---|---|---|---|---|---|---|---|
| CX-CHR | CNN-RNN | 0.590 | 0.506 | 0.450 | 0.411 | 0.577 | 1.580 |
| CX-CHR | LRCN | 0.593 | 0.508 | 0.452 | 0.413 | 0.577 | 1.588 |
| CX-CHR | AdaAtt | 0.588 | 0.503 | 0.446 | 0.409 | 0.575 | 1.568 |
| CX-CHR | Att2in | 0.587 | 0.503 | 0.446 | 0.408 | 0.576 | 1.566 |
| CX-CHR | CoAtt | 0.651 | 0.568 | 0.521 | 0.469 | 0.602 | 2.532 |
| CX-CHR | HGRG-Agent | 0.673 | 0.587 | 0.530 | 0.486 | 0.612 | 2.895 |
| CX-CHR | CMAS_W | 0.659 | 0.585 | 0.534 | 0.497 | 0.627 | 2.564 |
| CX-CHR | CMAS_NW,AW | 0.657 | 0.579 | 0.522 | 0.479 | 0.585 | 1.532 |
| CX-CHR | CMAS-IL | 0.663 | 0.592 | 0.543 | 0.507 | 0.628 | 2.475 |
| CX-CHR | CMAS-RL | 0.693 | 0.626 | 0.580 | 0.545 | 0.661 | 2.900 |
| IU-Xray | CNN-RNN | 0.216 | 0.124 | 0.087 | 0.066 | 0.306 | 0.294 |
| IU-Xray | LRCN | 0.223 | 0.128 | 0.089 | 0.067 | 0.305 | 0.284 |
| IU-Xray | AdaAtt | 0.220 | 0.127 | 0.089 | 0.068 | 0.308 | 0.295 |
| IU-Xray | Att2in | 0.224 | 0.129 | 0.089 | 0.068 | 0.308 | 0.297 |
| IU-Xray | CoAtt | 0.455 | 0.288 | 0.205 | 0.154 | 0.369 | 0.277 |
| IU-Xray | HGRG-Agent | 0.438 | 0.298 | 0.208 | 0.151 | 0.322 | 0.343 |
| IU-Xray | CMAS_W | 0.440 | 0.292 | 0.204 | 0.147 | 0.365 | 0.252 |
| IU-Xray | CMAS_NW,AW | 0.451 | 0.286 | 0.199 | 0.146 | 0.366 | 0.269 |
| IU-Xray | CMAS-IL | 0.454 | 0.283 | 0.195 | 0.143 | 0.353 | 0.266 |
| IU-Xray | CMAS-RL | 0.464 | 0.301 | 0.210 | 0.154 | 0.362 | 0.275 |
Table 2, Impression generation tab. 2.
| Dataset | Method | BLEU-1 | BLEU-2 | BLEU-3 | BLEU-4 | ROUGE | CIDEr |
|---|---|---|---|---|---|---|---|
| CX-CHR | SoftAtt_text | 0.112 | 0.044 | 0.016 | 0.005 | 0.142 | 0.038 |
| CX-CHR | SoftAtt_vision | 0.408 | 0.300 | 0.247 | 0.208 | 0.466 | 0.932 |
| CX-CHR | CMAS_text | 0.182 | 0.141 | 0.127 | 0.119 | 0.356 | 2.162 |
| CX-CHR | CMAS_vision | 0.415 | 0.357 | 0.323 | 0.296 | 0.511 | 3.124 |
| CX-CHR | CMAS-IL | 0.426 | 0.360 | 0.322 | 0.290 | 0.504 | 3.080 |
| CX-CHR | CMAS-RL | 0.428 | 0.361 | 0.323 | 0.290 | 0.504 | 2.968 |
| IU-Xray | SoftAtt_text | 0.179 | 0.047 | 0.006 | 0.000 | 0.161 | 0.032 |
| IU-Xray | SoftAtt_vision | 0.224 | 0.103 | 0.045 | 0.022 | 0.210 | 0.046 |
| IU-Xray | CMAS_text | 0.316 | 0.235 | 0.187 | 0.148 | 0.537 | 1.562 |
| IU-Xray | CMAS_vision | 0.379 | 0.270 | 0.203 | 0.151 | 0.513 | 1.401 |
| IU-Xray | CMAS-IL | 0.399 | 0.285 | 0.214 | 0.158 | 0.517 | 1.407 |
| IU-Xray | CMAS-RL | 0.401 | 0.290 | 0.220 | 0.166 | 0.521 | 1.457 |
Table 3, abnormality detection for Findings tab. 3. Higher precision and lower FPR are better.
| Dataset | Metric | HGRG-Agent | CMAS_NW,AW | CMAS-IL | CMAS-RL |
|---|---|---|---|---|---|
| CX-CHR | Precision | 0.292 | 0.173 | 0.272 | 0.309 |
| CX-CHR | FPR | 0.059 | 0.076 | 0.063 | 0.051 |
| IU-Xray | Precision | 0.121 | 0.070 | 0.094 | 0.128 |
| IU-Xray | FPR | 0.043 | 0.044 | 0.012 | 0.007 |
Table 4, abnormality detection for Impression tab. 4.
| Dataset | Metric | CMAS_text | CMAS_vision | CMAS-IL | CMAS-RL |
|---|---|---|---|---|---|
| CX-CHR | Precision | 0.067 | 0.171 | 0.184 | 0.187 |
| CX-CHR | FPR | 0.067 | 0.142 | 0.170 | 0.168 |
| IU-Xray | Precision | 0.054 | 0.160 | 0.162 | 0.165 |
| IU-Xray | FPR | 0.023 | 0.024 | 0.024 | 0.024 |
Headline comparisons, as the paper states them §5.3 §5.4:
- Findings, CX-CHR: CMAS-RL has the highest value on all six metrics. Its CIDEr of 2.900 is just above HGRG-Agent's 2.895 tab. 1.
- Findings, IU-Xray: CMAS-RL has the highest BLEU-1, BLEU-2 and BLEU-3. It ties CoAtt on BLEU-4 at 0.154. It is not highest on ROUGE (CMAS_NW,AW has 0.366) or CIDEr (HGRG-Agent has 0.343) tab. 1.
- Findings abnormality detection: CMAS-RL has the highest precision and the lowest FPR on both datasets tab. 3 §5.4.
- Impression, CX-CHR: CMAS_vision has the highest BLEU-4 (0.296) and CIDEr (3.124), above CMAS-RL's 0.290 and 2.968 tab. 2. The Impression text of §5.3 compares CMAS with SoftAtt and does not claim CMAS-RL is best on every metric.
- Impression abnormality detection: CMAS-RL has the highest precision on both datasets, but not the lowest FPR. The paper says a low FPR can be reached by writing normal sentences, which is what CMAS_text does tab. 4 §5.4.
- Qualitative: on five examples from IU-Xray, the paper reports that CMAS-RL finds the main abnormalities but sometimes misses secondary ones, and that CMAS_W tends to write normal findings §5.5 fig. 4.
- Template learning: the paper compares the most frequent Normality Writer sentences with the most used template sentences in IU-Xray reports, and reports that they are similar §5.6 tab. 5 tab. 6. This comparison is qualitative only.
Paper vs code
No official code.
Search steps, all in the order the skill specifies:
- Paper links: the only links are footnotes to the coco-caption evaluation tool and to jieba, a Chinese tokeniser §5.1 §5.2. Neither is the authors' code.
- arXiv abstract page: the comment says "ACL 2019" and lists no code or project link.
- GitHub search (API) queries and results: "show describe conclude chest x-ray" returned 0 repositories; "CMAS chest x-ray report" returned 0; "cmas report generation" returned 0; "chest xray report generation org:petuum" returned 0; "Show Describe Conclude" returned 2 unrelated repositories owned by other users; "chest x-ray report generation" returned 311, all third-party reimplementations. None belongs to the authors, so none is used.
- Petuum org (the authors' affiliation): 10 of 11 repositories were
listed. The closest,
petuum/medical_images, describes a hierarchical LSTM with co-attention for medical report generation. Its README does not mention CMAS or this paper, so it is not used as this paper's code. - The authors' personal GitHub profile lists five repositories, none about
chest X-ray reports.
github.com/baoyujing/CMASreturns 404.
Pitfalls
- Eq. 2 has no visual term. As written,
h_pdepends on the local and global states only, not onv_p. Then everyα_pin eq. 3 is the same,1/P, so the attention does not focus on any region. The soft-attention reference [xu2015show] scores each location, so the missingv_pmay be a typo. No code is available to settle it. - Feature source conflicts. §4.2.1 says the visual features come from "an intermediate layer of a CNN". §5.2 says they come from the last convolutional layer, a 7 × 7 × 2048 map. The second is more specific and is used in the Hyperparameters table.
- Reward definition conflicts. §1 and the opening of §4.3 say BLEU-4. The
text after eq. 14 defines
f(s_n)with BLEU-1. Eqs. 13 and 14 compute separate abnormal and normal BLEU, but the text after eq. 14 uses one BLEU over the whole sentence list. The paper does not say which reward trained CMAS-RL. - Discount factor not stated.
γis in [0, 1], but no value is given. Eq. 15 sums to infinity, and a real report has finitely many sentences. The truncation used is not stated. - Sampling versus greedy decoding. Eq. 17 takes expectations over sampled sentences, but eqs. 9 and 12 use arg max. The paper does not say whether RL rollouts are sampled, which affects the gradient.
- No baseline is mentioned in the RL update (eqs. 16–19). The paper gives no variance-reduction term. The implementation in the pseudocode above follows the equations as written.
- Fixed RL learning rate of 10⁻⁶ with no stated batch size, number of updates or stopping rule. Results depend on how long RL runs, and the paper gives no value.
- Impression training is unspecified. The paper does not say whether the Impression module is trained on reference Findings or on generated Findings, whether the two modules share weights, or which training schedule applies to them.
- STOP targets in imitation. Eq. 20 sums over the
Nreference sentences only. Whether a STOP label is trained after the last sentence is not stated. - The initial local state is undefined. Eq. 5 and eq. 2 use
ls_{n-1}. The paper does not definels_0for the first loop. - Abnormality labels depend on undisclosed patterns. The labels that decide which sentences are abnormal, and which Planner target and which writer gets trained, come from expert-written patterns not given in the paper §5.2. Precision and FPR depend on those patterns.
- Reported numbers have no variance. The paper gives one number per setting and no spread over seeds. Small gaps, such as the CIDEr values for CMAS-RL and HGRG-Agent on CX-CHR (2.900 vs 2.895), cannot be judged as significant from what is reported.
- Bold marks in the paper's tables do not always match the numbers. In Table 1 (IU-Xray), ROUGE 0.362 is bold for CMAS-RL, but CMAS_NW,AW has 0.366 and CMAS_W has 0.365. CIDEr 0.343 for HGRG-Agent is bold, which fits. The card uses the numbers, not the bold marks.
- Reproducibility of data. CX-CHR is a proprietary internal dataset, so its results cannot be reproduced from public data §5.1. The IU-Xray splits follow Li et al. §5.1, and the paper does not spell them out.
- External pretraining data. The IU-Xray image encoder is fine-tuned on ChestX-ray14 §5.2. That dataset is outside this card; its terms of use are not covered here.
- Commented-out source text is not used. The LaTeX source contains commented-out text with other settings: SGD, learning rates 10⁻¹⁰ and 5 × 10⁻⁶, early stopping, and a vocabulary of 1,185 tokens. The published text does not state these, so this card does not use them.
- Equation and table numbers were counted from the LaTeX source, in source order, because the typeset PDF was not checked. The unlabelled equation in §4.2.1 is counted as eq. 1. Check the numbers against the PDF before relying on them.
Lineage
- Builds on: soft attention for the visual and Findings contexts [xu2015show], as cited in §4.2.1 and §4.5; REINFORCE [williams1992simple], as cited in §4.4.1; BLEU [papineni2002bleu], as cited in §4.3.
- Compared against: CNN-RNN [vinyals2015show], LRCN [donahue2015long], AdaAtt [lu2017knowing], Att2in [rennie2017self] (image captioning models); CoAtt [jing2017automatic] (hierarchical co-attention report model); HGRG-Agent [li2018hybrid] (hybrid of template retrieval and generation) §2 §5.2; SoftAtt [xu2015show] for Impression §5.2.
- Superseded by: not known from this wiki.
Source and license
Adapted from “Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports” by Baoyu Jing, Zeya Wang, Eric Xing (arXiv:2004.12274), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.