Deep Learning to Improve Breast Cancer Early Detection on Screening Mammography

In plain words

Screening mammograms are huge X-ray images in which a cancer is a tiny spot. The usual way to teach a computer to spot such spots needs a radiologist to outline every lesion, and those outlines are rare and expensive. This paper asks whether a model can learn from a much cheaper label: just "this image has cancer" or "it does not".

The trick is to start small. A network is first trained on small patches cut around the outlined lesions and on background patches. Its convolutional layers are then reused on a whole image, where they produce a map of scores across the image. Convolutional layers added on top look at that map and output one answer for the whole picture. Because the whole pipeline is one network, it can be trained with only whole-image labels after the first step.

The authors trained on a film-based public dataset and then moved the same kind of model to a newer digital dataset with only a few images for fine-tuning. The point is that such models can be adapted to a new imaging setup without new lesion outlines.

TL;DR

• Two-stage recipe: train a patch classifier on lesion-centred patches, then convert it into a whole-image classifier with convolutional top layers and train the whole network on image-level labels §2.1 §3.1.2.
• Best per-image AUC on the DDSM test split is 0.91 for a four-model ensemble with flip-averaged predictions; a single model reaches 0.88 with flip averaging §3.1.4 tab. 2.
• Transferred to INbreast (digital), the four models reach AUC 0.95 and the ensemble 0.98 on the validation split §3.2.2 tab. 4.
• Results are author-reported on small splits (376 DDSM test images, 107 INbreast validation images), with images down-sized to 1152 x 896 §3.1.1.
• The released code is non-commercial and its example scripts do not match the paper's hyperparameters; see Paper vs code.

Problem

Lesions are tiny relative to the image: a full-field digital mammogram is typically 4000 x 3000 pixels while a cancerous region of interest can be about 100 x 100 pixels §1. Standard detectors need lesion boxes, which are rare and costly to build, so most large mammography databases carry only image-level labels §1. Classic patch-based pipelines also split the work into two separately tuned steps: a patch classifier, then a summary step on its output §2.1. The paper assumes that ROI annotations exist for the first training stage, and that the whole-image labels from another database are enough for fine-tuning §1 §3.2.2.

Method

Overview. A patch classifier f is a CNN that maps a patch to class probabilities. Because a CNN has no fixed input size beyond its layers' weights, applying f to a whole image without changing its weights gives a grid of outputs, the "heatmap" f(M) §2.1. The paper calls this converted network an all-convolutional whole-image classifier: a network g of convolutional top layers, applied to f's output, followed by global average pooling and a softmax output §2.1 §2.2. Earlier designs placed a heatmap, a max-pooling layer and fully connected layers between the patch classifier and the output. Those are the variants this paper compares against and finds weaker §3.1.4.

Patch classifier. Two architectures are compared: VGG and ResNet, with the ResNet50 bottleneck design §2.2. The VGG16 variant replaces its two fully connected layers with global average pooling and adds one fully connected output layer §2.2. Patches are 224 x 224 and there are five classes: benign and malignant calcification, benign and malignant mass, and background §3.1.1. ImageNet-pretrained weights are used, since they beat random initialisation on the harder S10 set (Resnet50: 0.89 vs 0.63 accuracy) §3.1.3 tab. 1.

Converting to a whole-image classifier. The patch network's convolutional layers are applied to the full image. Two top blocks of convolutional layers (VGG or ResNet type) follow, then global average pooling and an output layer. The heatmap is removed entirely so the top layers can use all patch features §2.2 fig. 1. Note the stride arithmetic in the text: a 3 x 3 layer on a 224-pixel receptive field with stride 32 widens it to 288 §2.1.

Activations where a heatmap is used. When a heatmap is kept, its activation is softmax eq. 1 in the patch classifier, but ReLU eq. 2 is preferred in intermediate layers because softmax saturates for large inputs §2.2.

# Whole-image classifier built from a patch classifier  [§2.1] [§2.2]
# Input:  mammogram (H, W) gray image, mean already subtracted
# Output: probabilities for (neg, pos), shape (2,)

def whole_image_forward(image):
    # 1. Keep the patch network's convolutional trunk and drop its
    #    pooling and dense head. Run on the whole image, this gives a
    #    spatial grid of features instead of one vector  [§2.1] [§3.1.4]
    features = patch_trunk_conv_layers(image)      # (h, w, C)

    # 2. Two top blocks of convolutional units (ResNet bottleneck or VGG).
    #    No heatmap sits between the patch network and these blocks  [§2.2]
    for block in top_blocks:
        features = block(features)

    # 3. Global average pooling, then a softmax output  [§2.2]
    pooled = spatial_mean(features)                # (C',)
    return softmax(dense(pooled))

Patch sampling (S1 and S10). Mammograms are converted to PNG and down-sized to 1152 x 896 §3.1.1. S1 holds one lesion-centred patch and one random background patch per image. S10 holds 10 patches sampled around each ROI with a minimum overlap ratio of 0.9, plus the same number of background patches from the same image §3.1.1. S10 is used for all later experiments because it gives the whole-image classifier more information §3.1.4 tab. 2.

Patch training schedule. Three stages, each with lower learning rate and more layers unfrozen §3.1.2:

# Patch classifier training, three stages, ImageNet initialisation  [§3.1.2]
# Patches 224 x 224, five classes, Adam, batch size 32, sample weights
# adjusted within each batch to balance the classes  [§3.1.2]

def train_patch_classifier(patch_net, patches):
    # Stage 1: train only the final layer, lr 1e-3, 3 epochs  [§3.1.2]
    freeze_all_except_last_layer(patch_net)
    fit(patch_net, patches, learning_rate=1e-3, epochs=3)

    # Stage 2: unfreeze the top layers, lr 1e-4, 10 epochs.
    # Paper's top-layer index: 46 (ResNet50), 11 (VGG16)  [§3.1.2]
    unfreeze_from_index(patch_net, top_layer_index)
    fit(patch_net, patches, learning_rate=1e-4, epochs=10)

    # Stage 3: unfreeze all layers, lr 1e-5, 37 epochs (50 total)  [§3.1.2]
    unfreeze_all(patch_net)
    fit(patch_net, patches, learning_rate=1e-5, epochs=37)
    return patch_net

Whole-image training schedule. Two stages, the first training only the new top layers and the second training all layers at a lower rate §3.1.2:

# Whole-image training from the converted patch classifier  [§3.1.2]
# Batch size 2 (GPU memory limit). Optimizer not stated.

def train_whole_image(image_model, images, labels):
    # Stage 1: train only the new top layers (function g).
    # lr 1e-4, weight decay 1e-3, 30 epochs  [§3.1.2]
    freeze_patch_layers(image_model)
    fit(image_model, images, labels, learning_rate=1e-4,
        weight_decay=1e-3, epochs=30)

    # Stage 2: train all layers (function h).
    # lr 1e-5, weight decay 1e-2, 20 more epochs (50 total)  [§3.1.2]
    unfreeze_all(image_model)
    fit(image_model, images, labels, learning_rate=1e-5,
        weight_decay=1e-2, epochs=20)

    # If not converged, resume for up to 200 more epochs  [§3.1.2]
    return image_model

Augmentation. Applied on the fly to training data: horizontal and vertical flips, rotation within plus or minus 25 degrees, shear within plus or minus 0.2 radians, zoom 0.8 to 1.2, and channel shift within plus or minus 20 pixel values. The mean grey value of the whole-image training set is subtracted from both patch and whole-image inputs, with no other preprocessing §3.1.2.

Augmented prediction and ensemble. Each test image is predicted in four versions: original, horizontally flipped, vertically flipped, and both flips. The scores are averaged. This raised AUC by 0.01 to 0.03 per model §3.1.4 tab. 2 tab. 3. The ensemble averages the augmented scores of four models: Resnet-Resnet, VGG-VGG, and two hybrids, Resnet-VGG (ResNet50 patch network with VGG top blocks) and VGG-Resnet §3.1.4.

# Augmented prediction: average over four flipped copies  [§3.1.4]
# Ensemble: average the augmented scores of four models  [§3.1.4]

def augmented_score(model, image):
    # Original, horizontal flip, vertical flip, and both flips.
    views = [
        image,
        flip_horizontal(image),
        flip_vertical(image),
        flip_horizontal(flip_vertical(image)),
    ]
    return mean([model(view) for view in views])   # (2,)


def ensemble_score(models, image):
    # Four best models: Resnet-Resnet, VGG-VGG, Resnet-VGG, VGG-Resnet
    return mean([augmented_score(m, image) for m in models])

Transfer to INbreast. INbreast is a digital (FFDM) dataset with different intensity profiles from the film-based DDSM images §3.2.1 fig. 4. The four best DDSM models are fine-tuned directly on INbreast image labels, with ROI annotations ignored §3.2.2. Labels come from BI-RADS: categories 1 and 2 are negative; 4, 5 and 6 are positive; category 3 is excluded §3.2.1. Training used Adam, learning rate 1e-5, 200 epochs and weight decay 0.01 §3.2.2.

Hyperparameters

Results

Author-reported. Setup: DDSM (CBIS-DDSM, film-digitised) for development, with 2478 images from 1249 women and 376 test images; INbreast (FFDM) for transfer, with 107 validation images. Hardware: one 8 GB Quadro M4000 GPU [unnumbered "Computational environment" section] §3.1.1 §3.2.1.

Notes on the table: the abstract's 0.88 matches the augmented (A-AUC) column of the hybrid row in tab. 2, not the non-augmented 0.87. The abstract's 0.95 for INbreast is the AUC shared by the four transferred models in tab. 4.

Paper vs code

Official code: lishen/end2end-all-conv, licensed CC BY-NC-SA 4.0 (the author's own repository; non-commercial, so no verbatim code is quoted here) LICENSE. The scripts are examples and several differ from the paper.

• Patch size and classes. Paper: 224 x 224, five classes §3.1.1. Released example: 256 x 256, three classes (background, malignant, benign) train_patch_clf_im4096_256_3Cls.sh train_patch_clf_im4096_256_3Cls.sh.
• Patch optimizer. Paper: Adam §3.1.2. Example script: nadam train_patch_clf_im4096_256_3Cls.sh. Code default: sgd patch_clf_train.py.
• Patch learning rates and epochs. Paper: lr 1e-3, 1e-4, 1e-5 for 3, 10 and 37 epochs §3.1.2. Example script: initial lr 0.01, top multiplier 0.01, all-layer multiplier 0.0001, epochs 1, 5 and 15 train_patch_clf_im4096_256_3Cls.sh train_patch_clf_im4096_256_3Cls.sh train_patch_clf_im4096_256_3Cls.sh. The code multiplies one base rate per stage: stage 2 uses init_lr * top_layer_multiplier and stage 3 uses init_lr * all_layer_multiplier dm_keras_ext.py dm_keras_ext.py. The paper's three rates correspond to base 1e-3 with multipliers 0.1 and 0.01. This is inferred from the code, not stated in the paper.
• Top-layer index. Paper: 46 (ResNet50), 11 (VGG16) §3.1.2. Code default when no override is given: 162 and 15 dm_keras_ext.py dm_keras_ext.py. The patch example passes --no-top-layer-nb train_patch_clf_im4096_256_3Cls.sh, so it uses 162. The paper's index base is not stated, so these cannot be reconciled.
• Whole-image optimizer. Paper: not stated §3.1.2. Code default: sgd image_clf_train.py. Both whole-image example scripts use adam train_image_clf_im1152.sh train_image_clf_inbreast.sh.
• Whole-image learning rates, weight decay and epochs. Paper: lr 1e-4 and 1e-5, weight decay 1e-3 and 1e-2, 30 and 20 epochs §3.1.2. DDSM example: initial lr 1e-3, all-layer multiplier 0.1, weight decay 1e-4 for both stages, epochs 1 and 2 train_image_clf_im1152.sh train_image_clf_im1152.sh train_image_clf_im1152.sh train_image_clf_im1152.sh train_image_clf_im1152.sh train_image_clf_im1152.sh. The example is a short run, not the paper's schedule.
• Whole-image batch size. Paper: 2 §3.1.2. DDSM example: 20 with multiplier 0.5, so 10 per training step train_image_clf_im1152.sh train_image_clf_im1152.sh. The code computes int(batch_size * train_bs_multiplier) image_clf_train.py.
• Bottleneck width. Paper: top blocks written [512-512-1024] x 2 tab. 2. The bottleneck's last layer has enlarge_factor times the depth, and the default factor is 4 dm_resnet.py image_clf_train.py. That gives 2048, not 1024. The INbreast script sets factor 2 train_image_clf_inbreast.sh, which yields the paper's notation. The paper does not state the factor. The DDSM example uses depth 512, repetitions 3, so it is not the paper's best configuration train_image_clf_im1152.sh train_image_clf_im1152.sh.
• Test-time augmentation. Paper: flip averaging, reported as A-AUC §3.1.4. Code: the flip function exists dm_keras_ext.py, but calc_test_auc defaults to test_augment=False dm_keras_ext.py. No call site in the repository sets it to True; the training script's final test AUROC is computed without flips image_clf_train.py.
• Best-model selection. The checkpoint callback is given the validation set and keeps the epoch with the best AUC image_clf_train.py dm_keras_ext.py. The test set is scored only after the best weights are loaded image_clf_train.py. The paper does not describe checkpoint selection.
• Sampling. Paper: S10 = 10 ROI patches per image plus background, overlap ratio 0.9 §3.1.1. Code: patch size 256, 30 abnormal, 30 background and 15 hard-negative patches by default, with an overlap cutoff of 0.75 sample_patches_combined.py sample_patches_combined.py sample_patches_combined.py. Target height is 4096 sample_patches_combined.py, not 1152 x 896 §3.1.1.
• Mean subtraction. Paper: mean subtracted, value not given §3.1.2. Code constants: 59.6 for patches patch_clf_train.py, 52.16 for DDSM whole images image_clf_train.py, and 44.33 for INbreast train_image_clf_inbreast.sh.
• Input scaling. Whole-image scripts rescale by 255/65535 = 0.003891 train_image_clf_im1152.sh train_image_clf_im1152.sh, which implies 16-bit inputs. The paper says PNG and gives no bit depth §3.1.1.
• INbreast transfer. Paper: lr 1e-5, weight decay 0.01, 200 epochs §3.2.2. Script: initial lr 1e-4 with all-layer multiplier 0.01 (effective 1e-6) train_image_clf_inbreast.sh train_image_clf_inbreast.sh, weight decay 0.001 for the first stage and 0.01 for the second train_image_clf_inbreast.sh train_image_clf_inbreast.sh, 50 epochs train_image_clf_inbreast.sh, and batch size 4 train_image_clf_inbreast.sh.
• Split names. The INbreast script names its folders test (validation directory) and val (test directory) train_image_clf_inbreast.sh train_image_clf_inbreast.sh. The paper describes only a training and a validation split §3.2.1. Check which split the reported numbers come from.
• Random-forest baseline. Paper: described in §3.1.4. The repository has no random-forest code (searched by name).
• Heatmap activation. When a heatmap is added, the code uses ReLU dm_resnet.py, as the paper states eq. 2. The main model does not use a heatmap.

Pitfalls

• Bottleneck notation. [L-M-N] in the paper does not state the expansion factor. With the code default of 4, [512-512-1024] cannot be built. Use factor 2 for the paper's best configuration (see Paper vs code).
• Top-layer index base. The paper gives 46 and 11 without saying how layers are counted. The code's Keras indexing gives 162 and 15 by default. Set the index from the layer list you build, and check it on the frozen/unfrozen boundary.
• Stage learning rates. Implement the stage rates as absolute values (1e-3, 1e-4, 1e-5) or as a base rate with multipliers. The code uses the multiplier form, so copying its example values gives rates far below the paper's.
• Unstated whole-image optimizer. The paper states Adam only for patch training and for INbreast. Choose and report one for the whole-image stages.
• Batch size semantics. The paper's batch sizes (32 patch, 2 whole-image) are per-step training sizes. In the code, batch_size is multiplied by train_bs_multiplier (default 0.5). Check which one you set.
• Class balance. Patch training balances the five classes with per-batch sample weights §3.1.2. The code exposes this as auto_batch_balance, which defaults to off in the patch script and on in the whole-image parser patch_clf_train.py image_clf_train.py. The paper does not say whether whole-image training balances classes.
• Augmented metrics are opt-in in code. Without setting test_augment, the reported AUROC is non-augmented (see Paper vs code). The paper's headline 0.88 and 0.91 need flip averaging.
• Small, single-split evaluation. The DDSM test set has 376 images and the INbreast validation set has 107, each analysed per view, not per patient §3.1.1 §3.2.1. Expect wide intervals; no confidence intervals are given.
• Cross-platform transfer is measured on one pair. Transfer goes from film-digitised DDSM to FFDM INbreast only §3.2.1. No other platform is tested.
• Compute. Patch sampling cost grows linearly with patches per ROI, and the authors note the gain may diminish §4. Images were down-sized to fit an 8 GB GPU; the authors expect full resolution to help but did not test it §4. VGG16 without its dense layers has about 15M weights and Resnet50 about 24M §4.
• Dataset-specific constants. Mean subtraction and the 255/65535 scaling are dataset-specific. Reuse the constants from your own training set, not the code's values (see Paper vs code).
• Comparison claims. The paper cites a commercial CAD AUC of 0.72 and reported AUCs of other deep methods from other studies §4. These numbers are not measured here.

Lineage

• Builds on: VGG §2.2, ResNet with bottleneck blocks and batch normalisation §2.2, and ImageNet pre-training §1 §3.1.2. The paper gives no arXiv ids for these; cite them from their own cards when they exist.
• Compared against: the two-step approach with a binarised heatmap and a random forest, AUC 0.73 §3.1.4; patch-based heatmap methods of Wang et al. and Shen et al. §2.1 §3.1.4; multiple-instance-learning methods of Zhu et al. and Choukroun et al., whose AUCs the paper reports as lower §4.
• Superseded by: not stated.