Reading Race: AI Recognises Patient's Racial Identity In Medical Images

In plain words

Doctors reading a chest X-ray or a CT scan usually cannot tell a patient's race from the picture. This paper asks whether a neural network can, and finds that it can: models trained on many kinds of medical scans predict the race a patient reports, and they still do so on data from hospitals the models were not trained on.

The authors then try to find the clue the model uses. They hide body parts, shrink the images, blur or add noise, remove bands of spatial frequency, and look at where the model pays attention. None of these steps removes the signal. The authors conclude that the information is spread across the whole image, so it is hard to filter out.

Why it matters: a model can pick up race in ways no clinician can see, and it could then make worse decisions for some groups without anyone noticing. The paper argues that medical AI needs audits by race, sex and age, and that datasets should record the self-reported race of patients.

TL;DR

• Standard CNNs trained on medical images predict self-reported race, one-vs-all ROC-AUC of about 0.96-0.99 on chest X-ray for Asian, Black and White patients tab. 3.
• The signal transfers to other hospitals and scanners tab. 4, to CT, mammography, limb and cervical-spine images tab. 5, and to models trained for other tasks tab. 6 §A3.
• Hypotheses about body habitus, breast density, disease labels, bone density, resolution, frequency bands, lung masks and single image patches each fail to explain it §B §C.
• All numbers are author-reported. The mechanism is not identified. The released code has notebooks and no data, and no license file.

Problem

Race has been shown to produce unequal AI performance in several clinical areas, but the paper states that no visual correlate of race in medical images is known to clinicians §Introduction, p. 3. Regulators clear medical AI systems that use medical images as inputs, and the public documents of cleared products do not show results by clinical or demographic subgroup p. 3. The paper's question is how such models can produce disparities when race is not visible to experts p. 3. Its premise is that self-reported race is a social construct, and the authors use it as the label throughout p. 3.

Method

The paper is an empirical analysis, not a new architecture. It trains standard CNNs on nine labelled datasets and runs three groups of experiments p. 4 p. 9.

• A: race detection. A1 trains one model per chest X-ray dataset on Black, White and Asian labels, then tests it on the other datasets (external validation) §A1 tab. 4. A2 trains binary Black-vs-White models on non-chest-X-ray data §A2. A3 trains a pathology-detection model and a patient re-identification model on MXR, freezes its convolutional layers, and trains a new race head on top §A3 p. 11.
• B: confounders. B1 uses body mass index (BMI). B2 uses breast density, with and without age. B3 predicts race from disease labels alone, without images. B4 removes bone-density information by clipping bright pixels. B5 splits MXR by age and by sex §B1-§B5 fig. 1.
• C: mechanism. C1 applies low-pass, high-pass, band-pass and notch filters in the Fourier domain. C2 changes image resolution and adds noise or blur at test time. C3 uses Grad-CAM saliency maps, masks the most salient region, and tests lung and non-lung segmentations. C4 trains on a 3×3 grid, removing or keeping one of nine patches §C1-§C4 fig. 2 fig. 3 fig. 4 fig. 5.

The datasets are in tab. 1. Race proportions are as stated in the table. Hispanic labels were excluded because they were coded inconsistently across datasets, and Asian labels were used only where prevalence was above 1% p. 9.

Source: tab. 1. The table gives no sex or age details for the split procedure.

The paper has no algorithm box. The pseudocode below follows the text of §A1, §C1, §C2 and §C3. Each block is a summary of the stated procedure, not code from the authors.

# Race classifier on one chest X-ray  [§A1] [tab. 1] [supp. tab. 1]
# Input:  images (N, 320, 320); races in {Black, White, Asian}
# Output: one-vs-all ROC-AUC per race on the test split

def train_race_model(images, races):
    # Split 60 / 10 / 30 for MXR and CXP  [tab. 1]
    train, valid, test = split(images, races, 0.60, 0.10, 0.30)
    # ImageNet-initialised ResNet34, Adam, learning rate 1e-3  [supp. gen.]
    model = resnet34(weights="imagenet", input_size=(320, 320))
    # Categorical cross-entropy over the three race classes  [supp. tab. 1]
    model.compile(loss="categorical_crossentropy", optimizer=adam(1e-3))
    # Cut the learning rate by 10x after 2 epochs without improvement  [supp. gen.]
    # Stop when validation loss stops improving  [supp. tab. 1]
    model.fit(train, valid)
    return model, test

def race_auc(model, test):
    # One-vs-all AUC per race, with 0.5 as the default operating point  [§A1]
    scores = model.predict(test.images)            # (N, 3)
    return [roc_auc(test.races == race, scores[:, i])
            for i, race in enumerate(["Asian", "Black", "White"])]

Frequency filtering in §C1 uses a 2D Fourier transform and a centred circle of diameter d. The low-pass filter zeroes everything outside the circle. The high-pass filter zeroes everything inside it [supp. C1]. Band-pass and notch filters use two circles [supp. C1].

# Fourier-domain filter on one image  [§C1] [supp. C1] [fig. 6]
# Input:  image (224, 224); kind in {low, high, band, notch}; diameter d
# Output: image rebuilt from the kept frequencies

def filter_image(image, kind, diameter):
    # Move zero frequency to the centre of the spectrum  [supp. C1]
    spectrum = fftshift(fft2(image))                # (224, 224) complex
    # Disk mask: low keeps the inside, high zeroes the inside  [supp. C1]
    disk = inside_circle(spectrum.shape, diameter)  # (224, 224) bool
    kept = where(low_or_band(kind, disk), spectrum, 0)
    # Invert the transform; keep the real part  [supp. C1]
    return real(ifft2(ifftshift(kept)))

Robustness to degraded images in §C2 uses two test-time changes on the trained model. The paper states the noise variance and does not state the blur width.

# Test-time degradation of a trained model  [§C2] [supp. tab. 15]
# Input:  test images (N, 224, 224, 1); trained model
# Output: per-race AUC for clean, noisy and blurred inputs

def degrade_and_score(model, test_images):
    # Additive Gaussian noise with mean 0 and variance 0.1  [§C2]
    noise = normal(0, sqrt(0.1), test_images.shape)
    noisy = test_images + noise
    # Gaussian blur; the width is not stated in the text  [§C2]
    blurred = gaussian_blur(test_images, sigma=...)
    return model.predict(noisy), model.predict(blurred)

Saliency occlusion in §C3 thresholds a Grad-CAM heatmap and blacks out one rectangle covering the thresholded area fig. 3 [supp. C3].

# Occlude the most salient region of a chest X-ray  [§C3] [fig. 3]
# Input:  image (H, W); saliency heatmap (H, W) from Grad-CAM
# Output: image with one black rectangle over the hottest region

def occlude_salient_region(image, heatmap):
    # Keep pixels whose blue channel of the heatmap exceeds 0.1  [supp. C3]
    hot = heatmap > 0.1                             # (H, W) bool
    # One rectangle covering all hot pixels  [supp. C3]
    top, left, bottom, right = bounding_box(hot)
    image[top:bottom, left:right] = 0
    return image

Key equations: the paper has no numbered equations. The only formula it states is the noise variance in §C2.

Hyperparameters

"Where" gives the paper's location first. "Code" entries cite the repository at code_commit. Both sources are needed because the paper and the code disagree on several entries; see Paper vs code.

Results

Author-reported. Setup: "MXR ResNet34" rows are trained and tested within the named dataset. "X to Y" rows are trained on X and tested on Y. The paper is an analysis and has no baseline table, so the rows show the paper's own settings only.

A1: primary race detection, chest X-ray tab. 3 (AUC, one-vs-all):

A1: external validation, chest X-ray tab. 4 (AUC):

A2: binary Black vs White, non-chest-X-ray tab. 5 (AUC):

A3: models trained for other tasks, MXR tab. 6 (AUC, frozen features):

B, confounders (text values, author-reported):

• B1, BMI: a generalised linear model on BMI alone reaches AUC 0.55 (patients with BMI) and 0.52 (BMI present or absent as a binary label) on CXP §B1. Stratified by BMI group, the MXR model stays at AUC 0.89-0.98 §B1 [supp. tab. 3].
• B2, breast density: AUC 0.54 from density alone and 0.61 with age, against 0.91 for the EM-Mammo image model §B2.
• B3, disease labels: AUC 0.54-0.61 on MXR and 0.52-0.57 on CXP, depending on classifier §B3 [supp. tab. 18].
• B4, bone density removed by clipping: Black AUC 0.96 on MXR and 0.94 on CXP §B4.
• B5, age and sex: no significant difference across age groups or between sexes on MXR §B5 [supp. tab. 11] [supp. tab. 13].

C, mechanism:

• C1, frequency filters on MXR: low-pass performance falls sharply around diameter 10. High-pass performance holds up to diameter 100 §C1 fig. 6. Band-pass and notch AUC for White patients is 0.75-0.91 across the grids tested [supp. tab. 7] [supp. tab. 8].
• C2, resolution: AUC above 0.95 at 160 × 160 and above §C2 fig. 7. At 4 × 4 pixels it is 0.63-0.66 [supp. tab. 14].
• C2, degradation, MXR [supp. tab. 15]: noisy 0.64 / 0.72 / 0.70 and blurred 0.59 / 0.64 / 0.62 (Asian / Black / White). The text reports different ranges; see Pitfalls.
• C3, masking the saliency region, MXR [supp. tab. 16]: AUC falls from 0.93 / 0.94 / 0.94 to 0.88 / 0.79 / 0.79 (Asian / Black / White).
• C3, lung segmentation [supp. tab. 17]: original 0.93 / 0.94 / 0.94; non-lung 0.87 / 0.85 / 0.87; lung only 0.68 / 0.74 / 0.73.
• C4, patch removal, MXR tab. 7: removing any one of nine patches leaves AUROC at 0.90-0.91 (White vs others).
• C4, one patch only tab. 8: AUROC 0.60-0.88, depending on the patch. The paper calls most single patches close to the full-image result.

The paper's Discussion states that the authors could not isolate the image features responsible, whether by location, frequency, or known anatomical confounders §Discussion, p. 21. The paper reports race AUC only as a probe, not as a mitigation.

Paper vs code

Repository: https://github.com/Emory-HITI/AI-Vengers, cloned at commit 393519961ed9c8c5031d94574c90e48ea7fbee75. The paper links this URL in its supplement [supp. p. 27]. The owner, Emory-HITI, appears to be the Emory group of co-author Judy W. Gichoya; the paper lists Emory affiliations, but the repository does not state the link itself. The repository has no LICENSE file, so code_license is recorded as not stated. The supplement says the code is available "with an open-source license" [supp. p. 27], but no license text is in the repository, so no code is quoted verbatim in this card. The repository's README says the data and models folders are empty and that some data are proprietary [README.md]. Notebooks use placeholder paths such as /path/to/directory/ MIMIC_resnet34_race_detection_2021_09_21_custom_splits.ipynb. Paths below are relative to the repository root.

• Batch size, CXR DenseNet121. The text gives batch 256 for all CXR models [supp. gen.], and Supp. Table 1 gives 128 for DenseNet121 [supp. tab. 1]. The released DenseNet121 code sets BATCH_SIZE = 128 Race classification_MIMIC_Densenet121.ipynb. The released ResNet34 code uses 256 MIMIC_resnet34_race_detection_2021_09_21_custom_splits.ipynb.
• Epochs, CXR ResNet34. Supp. Table 1 gives 12-16 epochs with early stopping [supp. tab. 1]. The released code sets a fit limit of 100 epochs with early-stopping patience 4 MIMIC_resnet34_race_detection_2021_09_21_custom_splits.ipynb MIMIC_resnet34_race_detection_2021_09_21_custom_splits.ipynb. The notebook's saved log shows validation-loss improvements up to epoch 12, so the 12-16 range may come from a different run. I did not resolve this.
• Epochs, DenseNet121. Supp. Table 1 gives 10 [supp. tab. 1]. The MIMIC race notebook sets 10 Race classification_MIMIC_Densenet121.ipynb. The Emory DenseNet121 notebook sets 10 Race_classification_Emory_Densenet121.ipynb.
• Dropout. Supp. Table 1 lists dropout 0.5 for DenseNet121 [supp. tab. 1]. The Emory DenseNet121 notebook adds Dropout(0.5) before the classifier Race_classification_Emory_Densenet121.ipynb. I found no dropout in the MIMIC DenseNet121 notebook.
• Augmentation. The text describes random horizontal flip, rotation of 15°, and zoom of ±10% [supp. gen.]. The MIMIC ResNet34 notebook sets these MIMIC_resnet34_race_detection_2021_09_21_custom_splits.ipynb. The Emory DenseNet121 notebook has them commented out Race_classification_Emory_Densenet121.ipynb.
• Noise. The text gives variance 0.1 §C2. The released function sets var = 0.1 and then sigma = var**sig. The test call uses sig = 1, so the standard deviation is 0.1 and the variance is 0.01 Race classification_MIMIC_Densenet121.ipynb Race classification_MIMIC_Densenet121.ipynb.
• Blur. The text does not give the width §C2. The released test call uses gaussian_filter(..., sigma=2) Race classification_MIMIC_Densenet121.ipynb Race classification_MIMIC_Densenet121.ipynb.
• Frequency filters. The C1 experiments are described in the Keras-based paper. The filters are in a separate PyTorch pipeline, training_code/frequency_training/, with low, high, band-pass and notch options data.py. The Fourier steps are in utils.py and the disk test is in utils.py. I did not confirm that this pipeline produced Tables 7 and 8, and I did not run it.
• Bone-density clipping (B4). The text describes clipping bright pixels to 60% intensity §B4 fig. 1. I searched training_code/ and bias_pred.ipynb for clip, bright and bone operations and found no such step.
• Lung segmentation. The paper uses TernausNet §C3 [ref. 42]. The repository README lists the lung-segmentation model as U-Net [README.md#L23].
• Patch helper. The released blacken_tensor helper defaults to tile size 256 with offset 256 utils.py. Inputs in the frequency pipeline are 224 × 224 data.py. I did not trace how the helper is called, so I cannot say whether the patch experiment used it.
• Sensitivity and specificity. The text says they are given at 0.5 §A1. I did not find them in any table.

Pitfalls

• Text and table disagree on degradation. The text gives AUC 0.74-0.80 for noisy images and 0.64-0.72 for blurred images §C2. Supp. Table 15 gives 0.64-0.72 for noisy and 0.59-0.64 for blurred [supp. tab. 15]. Use the table if you need the number; the text may refer to a different set.
• Noise definition. "Variance 0.1" means different things in the text and the code (variance 0.1 vs std 0.1). A reproduction that follows the code gives much milder noise.
• Batch size and epochs are not one setting. The text and the table disagree on batch size for DenseNet121, and the CXR ResNet34 epoch range is far from the code's limit. Plan on running a small sweep rather than copying one value.
• Dropout and augmentation vary by notebook. Dropout 0.5 and the augmentation described in the text are not in every released model. Take them from the notebook for the model you reproduce.
• Race is self-reported and used as the only label. The authors state that self-reported race is a social construct and not genetic ancestry, and they treat it as a strong proxy §Discussion, p. 22. Do not read the results as a claim about biology.
• Asian class is small. Asian labels were used only where prevalence exceeded 1% p. 9. Supp. Table 11 attributes a low Asian AUC in the 0-20 age group to a dataset size under 1% [supp. tab. 11]. Per-race AUCs for small groups have wide uncertainty; the paper gives no intervals.
• Confounders are not independent of race. Chi-square tests show that BMI, breast density, age and sex are not independent of race in the datasets used [supp. tab. 2] [supp. tab. 5] [supp. tab. 10] [supp. tab. 12]. The B-experiments test whether these factors alone explain race prediction, not whether they are unrelated to it.
• Saliency maps are not evidence of mechanism. Five radiologists judged the Grad-CAM maps qualitatively §C3. The authors found no consistent anatomical location [supp. fig. 5]. Occluding the top-attention region lowers AUC to 0.79 for Black and White patients, so the signal is spread across the image rather than sitting in one area [supp. tab. 16].
• Frequency naming. The paper says "diameter" in the text, and the caption of fig. 6 says it is a radius. The code tests distance against the diameter utils.py. Use the same convention in your code, and check it with a known image.
• Resolution floor. The AUC at 4 × 4 pixels is above chance, but images at that size show no recognisable anatomy [supp. fig. 3]. The paper uses this to argue the signal is not in fine detail. It does not show the signal is absent at higher resolution.
• Scope. Only ionising-radiation imaging (X-ray and CT) is shown in the main claims. Ultrasound and MRI were not tested §Limitations, p. 23.
• Data access. The paper says CheXpert and RSPECT (Stanford subset) race labels "will be released publicly" p. 5, with no date. Patient subsets for non-public datasets such as NLST are "available on request" p. 5. The repository holds no data.

Lineage

• Builds on: ResNet for the CXR backbone [ref. 27, He et al., 2016; no arXiv id in the paper]; Grad-CAM for saliency [ref. 41, no arXiv id in the paper]; TernausNet for lung segmentation [ref. 42, arXiv:1801.05746].
• Related prior work named in the paper: a fairness-gap study of chest X-ray classifiers [ref. 9, arXiv:2003.00827]; texture-bias work on ImageNet-trained CNNs [ref. 39, arXiv:1811.12231; ref. 40, arXiv:1911.09071]; shortcut selection in COVID-19 detection [ref. 38, no arXiv id in the paper].
• Compared against: none. The paper reports analysis experiments, not a comparison with other methods.
• Superseded by: not stated.