CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison

In plain words

CheXpert is a large public collection of chest X-rays. Each image comes with a set of labels saying which findings it shows, such as an enlarged heart or fluid around the lungs, and each label is yes, no, or maybe. Nobody labeled the whole collection by hand. A rule-based program read the radiologists' written reports and turned them into labels. When a report hedges, such as "possible" or "cannot exclude", the label is marked as uncertain instead of being forced to yes or no.

The paper then asks how a neural network should learn from those maybes. It compares a few simple choices, and it checks the trained model against radiologists on a small test set that several of them labeled carefully.

TL;DR

• The dataset has chest radiographs from one hospital, with 14 labeled observations per study. Labels come from the report's Impression section through a rule-based labeler § Dataset § Label Extraction from Radiology Reports.
• A DenseNet121 classifier is trained with five ways of handling uncertain labels. The best way depends on the observation § Validation Results tab. 3.
• On a 500-study test set, the model's ROC and PR curves lie above all three radiologists' operating points for Cardiomegaly, Edema and Pleural Effusion. These are author-reported results § Test Results fig. 3.
• The labeler code is released under MIT. The image model's training code is not linked, and the paper states no terms for the dataset itself.

Problem

Chest radiography is the most common imaging exam. Automated interpretation needs large labeled datasets with strong reference standards and expert performance figures to compare against § Introduction.

Labels mined from reports have two problems. Radiologists write uncertain phrases, so a label is not always yes or no. And the paper reports that a MetaMap- and DNorm-based labeler gave weak extractions on its reports, so it built its own phrase lists § Comparison to NIH labeler.

Assumptions: reports are available, and only the Impression section is used § Mention Extraction.

Dataset

• Source. A retrospective collection at Stanford Hospital. The span runs from October 2002 to July 2017 and covers inpatient and outpatient imaging, each study paired with its report § Data Collection and Label Selection.
• Size. 224,316 chest radiographs of 65,240 patients § Dataset. The paper uses "studies" and "radiographs" for the same count, and labels are assigned per study tab. 1.
• Observations. 14 observations: 12 pathologies, "Support Devices" and "No Finding". They were chosen by prevalence and clinical relevance, following the Fleischner glossary where it applies. "Pneumonia" is included as the image pattern suggesting primary infection § Data Collection and Label Selection.
• Views. Models take a single view. Multi-view studies output the maximum probability across views § Model.
• Label counts (training set). Positive, uncertain and negative study counts per observation tab. 1:

• Splits. The paper gives the validation set (200 studies), the test set (500 studies) and the report evaluation set (1000 reports), with no patient overlap between them § Validation Set § Test Set § Report Evaluation Set. The training split size is not stated. Each row of tab. 1 sums to 187,641 studies, so the training set appears to contain that many studies (derived from the table, not stated in the text).

Method

Labeler: report to labels

The labeler runs in three stages on the Impression section § Label Extraction from Radiology Reports.

1. Mention extraction. Radiologists curated a large list of phrases per observation. Any phrase found in the Impression is a mention § Mention Extraction.
2. Mention classification. Each mention is parsed and given one of three labels. Rules run in order: pre-negation uncertainty, then negation, then post-negation uncertainty. A match sets the label (uncertain or negative). A mention that matches no rule is positive § Mention Classification. Sentences are split with NLTK, parsed with the Bllip parser (McClosky's biomedical model), and converted to universal dependencies with Stanford CoreNLP § Mention Classification.
3. Mention aggregation. The mention labels for one observation become a single label. The "No Finding" rule is applied at the end § Mention Aggregation fig. 2.

# Label one observation from its mention labels  [§ Mention Aggregation]
# Input:  mention_labels: list of "positive", "uncertain", "negative"
# Output: 1 (positive), -1 (uncertain), 0 (negative), None (blank)

def aggregate_observation(mention_labels):
    # Any positive mention makes the label positive  [§ Mention Aggregation]
    if "positive" in mention_labels:
        return 1
    # Uncertain mentions without a positive give uncertain  [§ Mention Aggregation]
    if "uncertain" in mention_labels:
        return -1
    # Negative mentions with nothing else give negative  [§ Mention Aggregation]
    if "negative" in mention_labels:
        return 0
    # No mention at all is blank  [§ Mention Aggregation]
    return None

# "No Finding" is positive when no pathology is positive or uncertain
# [§ Mention Aggregation]
# labels: dict from observation name to its aggregated label

def no_finding(labels):
    for observation, label in labels.items():
        # Support Devices and No Finding itself do not count  [§ Mention Aggregation]
        if observation in ["Support Devices", "No Finding"]:
            continue
        if label in [1, -1]:
            return 0
    return 1

Image model and uncertainty approaches

Input is a 320 × 320 pixel radiograph. Output is one probability per observation, or three class probabilities for U-MultiClass. The architecture is DenseNet121, which the paper chose after comparing ResNet152, DenseNet121, Inception-v4 and SE-ResNeXt101 § Training Procedure. All approaches share this architecture and training recipe § Training Procedure.

The training labels are 0 (negative), 1 (positive) or u (uncertain) § Uncertainty Approaches.

• U-Ignore (baseline). Sum over the 14 observations of the binary cross-entropy, masked for observations marked uncertain § Ignoring. The equation is unnumbered in the source.
• U-Zeros and U-Ones. Every u becomes 0 (U-Zeros) or 1 (U-Ones), then ordinary binary cross-entropy is applied, averaged over observations § Binary Mapping.
• U-SelfTrained. Train U-Ignore to convergence. Use that model to replace each u with its predicted probability. Keep 0 and 1 unchanged. Train with the mean binary cross-entropy over observations § Self-Training.
• U-MultiClass. Each observation has a three-way softmax over negative, positive and uncertain. The loss is the mean multi-class cross-entropy over observations. At test time, the positive probability is taken after a softmax restricted to negative and positive § 3-Class Classification.

# U-Ignore: binary cross-entropy summed over observations  [§ Ignoring]
# probs:  (14,) predicted P(Y_o = 1 | X)
# labels: (14,) values 1, 0 or "u"

def u_ignore_loss(probs, labels):
    loss = 0
    for o in range(14):
        # Skip observations marked uncertain for this study  [§ Ignoring]
        if labels[o] == "u":
            continue
        loss -= labels[o] * log(probs[o])
        loss -= (1 - labels[o]) * log(1 - probs[o])
    return loss

# U-Zeros (replacement 0) or U-Ones (replacement 1)  [§ Binary Mapping]
# labels: (14,) values 1, 0 or "u"

def binary_mapping(labels, replacement):
    # Map every uncertain label to the replacement value  [§ Binary Mapping]
    return [replacement if y == "u" else y for y in labels]

# U-SelfTrained targets  [§ Self-Training]
# labels:        (14,) values 1, 0 or "u"
# teacher_probs: (14,) output of the trained U-Ignore model for this study

def self_trained_targets(labels, teacher_probs):
    targets = []
    for o in range(14):
        # Soften each uncertain label to the teacher's probability  [§ Self-Training]
        if labels[o] == "u":
            targets.append(teacher_probs[o])
        else:
            targets.append(labels[o])
    return targets

# U-MultiClass: positive probability at test time  [§ 3-Class Classification]
# class_probs: (3,) softmax over negative, positive, uncertain

def multiclass_positive_prob(class_probs):
    p_negative = class_probs[0]
    p_positive = class_probs[1]
    # Renormalise over negative and positive  [§ 3-Class Classification]
    # (inferred from the text; the paper gives no formula)
    return p_positive / (p_negative + p_positive)

For multi-view studies, the model outputs the maximum probability across views § Model. The paper does not say how a study-level label is used for each view during training.

Model selection

For each approach, the paper trains three runs and keeps the best 10 checkpoints per run, ranked by the mean AUC over the five competition tasks. The 30 checkpoints are averaged on the validation set § Model Selection.

On the test set, each task uses the approach whose validation ensemble scored best § Test Results:

• Atelectasis and Edema: U-Ones.
• Cardiomegaly and Pleural Effusion: U-MultiClass.
• Consolidation: U-SelfTrained.

Hyperparameters

Labeler parameters are the phrase lists and rule sets, not numeric hyperparameters. The phrase lists are in the code repository .

Results

Author-reported throughout.

Labeler versus NIH labeler (report evaluation set, 1000 reports)

The ground truth comes from two radiologists who labeled each report, with disagreements resolved by discussion § Report Evaluation Set. Scores are F1 for mention extraction, negation detection and uncertainty detection tab. 2.

Bold marks the better system in the paper's table, where both systems have a score tab. 2.

Uncertainty approaches (validation set, 200 studies)

Ground truth is the majority vote of three radiologists, binarised so that "present" and "uncertain likely" count as positive and "absent" and "uncertain unlikely" count as negative § Validation Set. AUROC with 95% DeLong intervals tab. 3; bold is the best and italic the worst in the paper's table, for each observation:

Reported significance: on Atelectasis, U-Ones beats U-Zeros (p = 0.03). On Cardiomegaly, U-MultiClass beats U-Ignore (p < 0.01). No significant difference was found for Consolidation, Edema or Pleural Effusion § Results.

Test set (500 studies, five-radiologist majority ground truth)

Eight radiologists labeled each test study. Five of them form the majority-vote ground truth. The other three are benchmarked against it § Test Set.

• AUC. Best on Pleural Effusion (0.97) and worst on Atelectasis (0.85). The others are at least 0.9 § Comparison to Radiologists.
• AUPRC. Best on Pleural Effusion (0.91) and worst on Consolidation (0.44) § Comparison to Radiologists.
• Versus radiologists. On Cardiomegaly, Edema and Pleural Effusion, the model is above all three radiologists but not their majority vote. On Consolidation it beats two of three. On Atelectasis all three radiologists beat the model § Results fig. 3.
• Labeler bounds. The report labels are shown as lower (LabelL) and upper (LabelU) bounds. The lower bound lies below the model curves on all competition tasks. The upper bound lies on or below them on every task except Atelectasis fig. 3.
• Calibration. Average scaled Brier score is 0.110 before post-processing, 0.107 after isotonic regression and 0.101 after Platt scaling § Results.
• Statistics. No statistical test was run between the model and the radiologists § Limitations.

Figure 4 shows Grad-CAM maps for the final model, as qualitative examples only fig. 4.

Paper vs code

The labeler is at stanfordmlgroup/chexpert-labeler (MIT), linked from the competition page. It is pinned to 44ddeb3 LICENSE. The paper's footnote points to the competition page, which links this repository. The competition page links only third-party leaderboard entries for the image model, and no training code for the paper's DenseNet121 models. For those, the paper's description is the only source.

Discrepancies between paper and labeler code:

• Positive, negative and uncertain together. The paper says any positive mention makes the label positive § Mention Aggregation. The code checks the negative-plus-uncertain case first and returns uncertain, even when a positive mention is also present aggregate.py.
• Undocumented "chf" and "heart failure" rule. The code turns mentions of "chf" and "heart failure" into an uncertain Cardiomegaly mention. The paper does not describe this aggregate.py.
• Undocumented unmention phrases. The code excludes Cardiomegaly and Enlarged Cardiomediastinum mentions that follow positional phrases such as "over the" or "in the" with left, right or superior. The paper describes only the curated mention lists extract.py .
• Blank labels. The paper assigns "blank" when an observation is not mentioned § Mention Aggregation. The code writes an empty value there aggregate.py label.py. tab. 1 has no blank column, so blank was mapped to a training value somewhere. The paper does not say how.
• Consistent parts. The three-phase classification order matches the code classify.py. The No Finding rule matches the code's logic aggregate.py.

No official training code for the DenseNet121 models was found. The competition page's model description is the only source for them.

Pitfalls

• Section numbers are suppressed in the source (secnumdepth is 0). This card cites sections by title, e.g. § Mention Aggregation. Equations, including the masked loss, are unnumbered. Figures and tables are numbered in source order: Figure 1 is the task overview, Figure 2 the labeler example, Figure 3 the radiologist comparison and Figure 4 the Grad-CAMs fig. 1–fig. 4.
• Uncertain labels carry clinical meaning, not just missing data. Cardiomegaly uncertainty is mostly borderline phrases such as "minimal cardiac enlargement", and "stable cardiac contours" can fit either an enlarged or a normal heart § Analysis. Atelectasis and Consolidation are often reported together as "atelectasis versus consolidation". The labeler marks both uncertain, but panel review often resolved these as Atelectasis-positive and Consolidation-negative § Analysis.
• Ignoring uncertain labels is a biased choice. For Consolidation the uncertain rate (12.78%) is nearly twice the positive rate (6.78%) § Ignoring tab. 1. The paper's own analysis argues this is why U-Ignore underperforms.
• U-SelfTrained is under-specified. The paper does not give the number of relabelling rounds, the convergence criterion, or whether the teacher runs on unaugmented training images § Self-Training.
• Loss reduction differs between approaches. U-Ignore sums its losses over observations; the other approaches average them § Ignoring § Binary Mapping. The paper does not discuss the effect of this difference.
• Inclusion criteria are missing from the compiled text. The source contains commented-out text about adult studies, one to three views, and manual removal of protected health information. None of this is in the compiled paper, so these rules must not be assumed § Data Collection and Label Selection.
• Training split size is not stated. It is derived from tab. 1 (187,641 studies per row) and is not confirmed by the text.
• Table 2 differs from a commented-out version in the source. The source keeps an older table, commented out, with values such as NIH Atelectasis mention 0.978 and negation 0.556. The compiled tab. 2 uses 0.976 and 0.526. Use the compiled values.
• Table 3 caption says AUROC; the text says AUC. This card treats them as the same metric.
• Reading radiologist numbers. The radiologist operating points appear only in fig. 3, as plotted points. No numeric table is given, so they are not reproduced here.
• Limits on the radiologist comparison. Neither the model nor the radiologists had patient history or prior imaging, which the paper notes is known to lower diagnostic performance § Limitations.
• Multi-view handling is partly stated. Output is the maximum across views § Model. Training on multi-view studies, and any per-view augmentation, are not stated.
• Dataset terms are not in the paper. The paper's CC BY 4.0 licence covers the paper text and its figures. It does not cover the dataset. The competition page links the dataset through Redivis and states no terms, so check the dataset's own access terms before any use.

Lineage

• Builds on: The labeler follows the approach of NegBio (Peng et al., 2018), and the paper cites the parsing approach as similar to NegBio's § Mention Classification. No arXiv ids are linked in this card.
• Compared against: The NIH labeler used for ChestX-ray8 (Wang et al., 2017), as described in § Comparison to NIH labeler and tab. 2. The uncertainty approaches are compared against each other on the validation set tab. 3.
• Superseded by: not stated.