Improving Computer-aided Detection using Convolutional Neural Networks and Random View Aggregation

In plain words

Computer-aided detection (CADe) software scans medical images and flags spots that might be disease. Existing tools rarely miss a real finding, but they also flag many harmless spots, and each false alarm costs a radiologist time. This paper keeps the existing detector as a first pass and adds a second, more careful classifier after it.

The second classifier does not look at a flagged spot once. It looks at many slightly different crops around it: different zoom levels, small shifts and turns. A convolutional neural network scores each crop, and the scores are averaged into one number for the spot. Averaging over many views makes the verdict steadier and gives the network more training examples from few cases. The authors tested this on CT scans for three problems: spine bone lesions, lymph nodes and colon polyps. For a fixed number of false alarms per patient, more real findings are kept.

TL;DR

• The method is a second-tier false-positive filter for existing CADe systems, which produce candidates with near-100% sensitivity §I §II-D.
• Each candidate gets many random 2D or 2.5D views over several physical scales, with random translations and rotations; N = 100 views for bone and lymph nodes, N = 40 for polyps §III-A.
• The final score per candidate is the plain mean of the ConvNet probabilities over its views eq. 1.
• Author-reported sensitivity at 3 FPs per patient: 57% to 70% (bone), 43% to 77% (lymph nodes), 58% to 75% (polyps) tab. II. Several headline figures need care; see Pitfalls.
• The code release in this repository is the lymph-node pipeline only. The bone and polyp pipelines are not in it.

Problem

• Existing CADe systems in these settings reach high sensitivity only at many false positives per volume or patient, which keeps them out of routine use §I.
• The candidate generators used here reach close to 100% sensitivity at 40 to 60 false positives per volume for the lymph-node and bone sets, and at about 150 for polyps §II-B.
• Hand-crafted features (intensity statistics, HoG, Hessian shape descriptors) are fixed by the designer, which the authors argue limits how well they capture the imaging data §II-A.
• A full 3D ConvNet needs more labelled data than these CADe problems provide. The authors state that 59 patients are too few to train ConvNets to generalise well for the bone task §III-C §III-D §III-G.

Method

The method has two tiers. Tier one is an existing CADe system that outputs candidate locations with high sensitivity and many false positives §II-D. Tier two is a ConvNet that scores each candidate from N random views §II-C. Training uses the tier-one false positives as negative examples, which the authors describe as hard-negative mining §III-D.

Section and subsection numbers follow the IEEE journal layout used in the printed paper (Roman numeral for the section, letter for the subsection).

1. Candidates. The tier-one system gives a centre for each candidate. Bone candidates come from spine segmentation, watershed sub-segments and an SVM committee §II-D. Lymph-node candidates come from a mediastinal and an abdominal system §II-D. Polyp candidates come from colon-surface shape features §II-D. For lymph-node training, labelled nodes are injected into the candidate list so that candidate sensitivity is 100% §III-E.
2. Scales. Each candidate is resampled to isotropic voxels, and a cube is cut around it for each physical edge length s in 30, 35, 40 and 45 mm. The cube is always resampled to the same pixel count, so the physical pixel size changes with s §II-C §III-A.
3. Random views. For each scale, the cube is translated along a random vector N_t times (up to 3 mm) and each translated cube is rotated N_r times by a random angle. This gives N = N_s × N_t × N_r views per candidate §II-C §III-A.
4. Representation. Bone uses 2D axial patches, because the slices are 2.5 to 5 mm thick §III-D. Lymph nodes and polyps use 2.5D patches: three orthogonal central slices form the three channels §II-B fig. 4 §III-E §III-H.
5. ConvNet. Two convolution layers with max-pooling, two locally connected layers, a fully connected layer with DropConnect, and a softmax output that gives a per-view class probability §II-A fig. 1. The released model is named fc512 layers-fc512-11pct-dc.cfg.
6. Training. Stochastic gradient descent with momentum, in four stages totalling 1200 epochs, with mini-batches of 64, 64, 32 and 16 images §II-A §III-A. Each view is mean-subtracted with the per-pixel mean of the training set §II-A.
7. Test. Each of the N views gets a probability from the ConvNet. The per-candidate probability is their mean eq. 1 §II-C. Thresholds on that probability give FROC curves §III-D.

The aggregation is the equation the paper relies on:

p\left(x \mid \{P_1(x), \ldots, P_N(x)\}\right) = \frac{1}{N} \sum_{i=1}^{N} P_i(x) eq. 1

Where P_i(x) is the ConvNet probability for view i of candidate x.

Sampling of one candidate, as the code does it:

# Random views of one candidate  [§II-C] [§III-A]
# Input:  volume (CT, HU), centre (mm), scales (mm), n_trans, n_rot
# Output: N cubes of 32 x 32 x 32 voxels

def sample_views(volume, centre, scales, n_trans, n_rot):
    # Collect every view of this candidate  [§II-C]
    views = []
    # One set of views for each physical edge length  [§III-A]
    for scale in scales:
        # The first translation keeps the candidate centre
        # [code: pyconvnet/ITKapps/itkResampleRegionOfInterestFromCoordinateList.cxx#L409]
        for t in range(n_trans):
            shift = zero_vector_mm()
            if t > 0:
                # Random direction, length uniform up to 3 mm  [§III-A]
                shift = random_direction() * uniform(0.0, 3.0)
            moved = translate(volume, centre + shift)
            # Rotations are applied one after another
            # [code: pyconvnet/ITKapps/itkResampleRegionOfInterestFromCoordinateList.cxx#L434-L458]
            # Axis components uniform in [0, 1); angle in [0, 360)
            # [code: pyconvnet/ITKapps/itkResampleRegionOfInterestFromCoordinateList.cxx#L448-L452]
            for r in range(n_rot):
                if r > 0:
                    moved = rotate(moved, random_axis(), uniform(0.0, 360.0))
                # Cut a 32-voxel cube of physical size scale
                # [code: pyconvnet/ITKapps/itkResampleRegionOfInterestFromCoordinateList.cxx#L171]
                views.append(crop_cube(moved, centre, scale, size=32))
    return views

Turning one cube into a 2.5D patch:

# 2.5D patch: three central orthogonal slices as RGB  [§II-B] [fig. 4]
# Input:  cube (32, 32, 32)
# Output: patch (32, 32, 3)

def make_25d_patch(cube):
    # Index size/2 - 1 is the central slice in the code
    # [code: pyconvnet/ITKapps/Libs/nihHelperFunctions.h#L163]
    centre = size(cube) // 2 - 1
    # Red channel: axial slice along z
    # [code: pyconvnet/ITKapps/Libs/nihHelperFunctions.h#L238]
    red = cube[:, :, centre]
    # Green channel: slice along x; blue channel: slice along y
    # [code: pyconvnet/ITKapps/Libs/nihHelperFunctions.h#L239-L240]
    green = cube[centre, :, :]
    blue = cube[:, centre, :]
    return stack_channels(red, green, blue)

The released network, layer by layer:

# Released fc512 ConvNet  [§II-A] [fig. 1]
# Input: 3-channel patch, 24 x 24 after cropping
# [code: pyconvnet/convdata.py#L250]

def convnet(patch):
    # Convolution, 64 filters of 5x5, padding 2, ReLU
    # [code: pyconvnet/lymphnode-layers/layers-fc512-11pct-dc.cfg#L10-L18]
    maps = relu(conv(patch, filters=64, size=5, pad=2))
    # Max-pool 3x3 with stride 2, then cross-map normalisation
    # [code: pyconvnet/lymphnode-layers/layers-fc512-11pct-dc.cfg#L23-L36]
    maps = cross_map_norm(max_pool(maps, size=3, stride=2), size=9)
    # Second convolution, 64 filters of 5x5, ReLU
    # [code: pyconvnet/lymphnode-layers/layers-fc512-11pct-dc.cfg#L39-L47]
    maps = relu(conv(maps, filters=64, size=5, pad=2))
    # Cross-map normalisation, then max-pool 3x3 with stride 2
    # [code: pyconvnet/lymphnode-layers/layers-fc512-11pct-dc.cfg#L52-L62]
    maps = max_pool(cross_map_norm(maps, size=9), size=3, stride=2)
    # Locally connected layer, 64 filters of 3x3, no weight sharing
    # [code: pyconvnet/lymphnode-layers/layers-fc512-11pct-dc.cfg#L68-L76]
    maps = relu(local_conv(maps, filters=64, size=3, pad=1))
    # Second locally connected layer, 32 filters of 3x3
    # [code: pyconvnet/lymphnode-layers/layers-fc512-11pct-dc.cfg#L79-L87]
    maps = relu(local_conv(maps, filters=32, size=3, pad=1))
    # Fully connected layer of 512 units, DropConnect rate 0.5
    # [code: pyconvnet/lymphnode-layers/layers-fc512-11pct-dc.cfg#L91-L96]
    hidden = relu(dropconnect_fc(maps, units=512, rate=0.5))
    # Two outputs with softmax, positive and negative class
    # [code: pyconvnet/lymphnode-layers/layers-fc512-11pct-dc.cfg#L99-L105]
    return softmax(fc(hidden, units=2))

One training step:

# One training step  [§II-A] [code: pyconvnet/convdata.py#L328-L335]
# Input:  patches (32 x 32 x 3), labels, training mean, learning rate
# Output: loss value

def training_step(patches, labels, training_mean, convnet, learning_rate):
    # Subtract the per-pixel mean of the training set  [§II-A]
    centred = patches - training_mean
    # Code only: random 24 x 24 crop, then a 50% horizontal flip
    # [code: pyconvnet/convdata.py#L328-L335]
    crops = random_crop_and_flip(centred, size=24)
    # SGD with momentum 0.9 and DropConnect in the fc512 layer  [§II-A]
    # [code: pyconvnet/lymphnode-layers/params-fc512-11pct.cfg#L16]
    loss = convnet.update(crops, labels, learning_rate, momentum=0.9)
    return loss

Hyperparameters

Results

Author-reported. Setup: five-fold (bone), three-fold (lymph nodes) and train/test (polyps) cross-validation at patient level, CT imaging, baseline is the tier-one CADe system of each dataset §III-A §III-D §III-E §III-H.

Other results: at 3 FP per volume the lymph-node improvement is significant in both mediastinum and abdomen (Fisher's exact test, p < 0.001) §III-E. Sensitivity saturates quickly with N for lymph nodes and bone §III-D §III-E fig. 9 fig. 12, and the polyp curves saturate after about 10 views §III-H fig. 15. The 2.5D model with random views matches a 3D ConvNet with augmentation on the lymph-node set, and beats a 2D model and a 3D model without augmentation at 3 FPs per patient §III-G fig. 14.

Paper vs code

Checked against LymphNodeRFCNNPipeline/ in the official CADLab repository at code_commit (rsummers11/CADLab, shallow clone). The repository holds the lymph-node pipeline and its ConvNet training scripts. It does not contain the bone or polyp pipelines, so those results are not checked against code. The tier-one lymph-node candidate generator in the same directory was not checked.

Owner rsummers11 is the lab of an author; the pipeline's README and LICENSE credit Kevin Cherry and Holger Roth and ask users to cite this paper README.txt.

Agreements, checked in code:

• Patch size 32 and maximum translation 3 mm LymphNodeGenerateCNNImagePatches.m LymphNodeGenerateCNNImagePatches.m.
• Epochs 700, 300, 100, 100 and mini-batches 64, 64, 32, 16 run_net_lymph_nodes.sh run_net_lymph_nodes.sh.
• Lymph-node window [-100, 200] HU run_LymphNodeRFCNNPipeline.m.
• Aggregation is the mean over views LymphNodeFuseCNNMultiviewPredictions.m LymphNodeFuseCNNMultiviewPredictions.m. Median and maximum were tried in comments and rejected there LymphNodeFuseCNNMultiviewPredictions.m.

Discrepancies:

• Scales: the paper uses four scales §III-A. The code default is [30, 45] mm LymphNodeGenerateCNNImagePatches.m, and a commented lymph-node batch path says "2scales" run_multiple_nets_lymph_nodes.sh.
• The shipped demo is not the paper's setting. It uses one scale (45 mm), 2 translations, 2 rotations and 1.5 mm displacement, giving N = 4 run_LymphNodeRFCNNPipeline.m. The paper's settings appear commented out run_LymphNodeRFCNNPipeline.m.
• Weight init: the paper says "unit Gaussian" §III-A. The code draws weights as initW times a standard normal sample, with initW from 0.0001 to 0.04 layer.py layers-fc512-11pct-dc.cfg.
• Weight decay: the paper says "default" §II-A. The code sets it per layer params-fc512-11pct.cfg.
• Architecture details absent from the paper's text: cross-map response normalisation (size 9) in both convolutional blocks layers-fc512-11pct-dc.cfg layers-fc512-11pct-dc.cfg, DropConnect rate 0.5 layers-fc512-11pct-dc.cfg, and the padding of each layer.
• Training augmentation not described in the paper: random 24 x 24 crops of the 32 x 32 patch, a 50% horizontal flip, and at test time five crops (with flips when the img_flip parameter is set) convdata.py convdata.py.
• Learning-rate factors of 0.1 and 0.01 in the last two stages run_net_lymph_nodes.sh are not in the paper.
• Views: the code keeps the centre patch as one view (first translation, no rotation) itkResampleRegionOfInterestFromCoordinateList.cxx itkResampleRegionOfInterestFromCoordinateList.cxx. Rotations compound within one translation itkResampleRegionOfInterestFromCoordinateList.cxx, and the rotation axis is drawn with components uniform in [0, 1) itkResampleRegionOfInterestFromCoordinateList.cxx. The paper describes none of this.
• Lymph-node split: the paper says three-fold §III-E. The released script trains on batches 1 to 4, validates on 5 and tests on 6 run_net_lymph_nodes.sh. Whether that is the paper's split is not stated.

Pitfalls

• The lymph-node headline mixes subsets. The 43% baseline in the abstract and tab. II is not derived in the text. Section III-E gives 55% (mediastinum) and 30% (abdomen) as the prior results at 3 FP per volume §III-E. The 77% is the mediastinal result with joint training, and the abdominal result is 83% §III-E. The AUC row (0.76 to 0.94) is the abdominal result only §III-E.
• Lymph-node candidate sensitivity is assumed, not measured. Labelled nodes are injected into the candidate list so that candidate sensitivity is 100% §III-E. The reported sensitivities therefore exclude candidate misses.
• Bone sensitivity has a ceiling the paper doesn't mention in its tables. The bone candidate generator reaches at most 88.9% sensitivity §III-D. The paper does not say whether the reported sensitivities are relative to that ceiling.
• Polyp operating points mix units. Polyp FP rates are given per volume and per patient §III-H §III-I. Each polyp patient has two CT volumes, prone and supine §III-H, and the paper does not convert between the two. Do not treat "per volume" and "per patient" as interchangeable.
• Lymph-node true positives differ. tab. I gives 1,966 true positives and §III-E gives 1,956. The paper does not say which is correct.
• Bone slice-thickness rule conflicts. Section II-C uses 2D views only for slices of 5 mm or thicker §II-C. Section III-D uses 2D views for bone at 2.5 to 5 mm thickness, citing the thickness as the reason for axial-only sampling §III-D.
• "N = 100 at each scale" conflicts with N = 100 in total. Section III-E says the views are generated "N = 100 times ... at each scale" §III-E. Section III-A gives N = N_s × N_t × N_r = 100 in total, which is 25 views per scale §III-A.
• The untransformed patch is one of the N views. The released sampling keeps the centre patch as one view itkResampleRegionOfInterestFromCoordinateList.cxx itkResampleRegionOfInterestFromCoordinateList.cxx. An implementation that draws N random views from scratch will not match it.
• Unit Gaussian versus small initW. See Paper vs code. Implement what the code does if you want to reproduce the reported numbers. The paper's statement would give a very different weight scale.
• The trained network sees 24 x 24 inputs. The paper says 32 x 32 patches §III-A. The network input is a 24 x 24 crop convdata.py. Any port has to match the crop, or it will not reproduce the results.
• Hidden costs. Training takes 9 to 37 hours per model on one GPU §III-A. Scoring one volume at N = 100 takes 1 to 5 minutes, plus about 2 minutes of patch extraction per scale §III-A. A full candidate list is therefore minutes of compute per volume, not milliseconds.
• Window and interpolation are incomplete. The polyp window and the interpolation method are not stated in the paper. The code uses BSPLINE interpolation for the lymph-node run run_LymphNodeRFCNNPipeline.m.
• Class balance for lymph nodes is unstated. The paper says the lymph-node training set is balanced but gives no ratio §III-E.
• Sample size. The paper says that 59 patients are too few for the ConvNet to generalise well, and reports overfitting between training and test FROC curves §III-D.

Lineage

• Builds on: the authors' earlier lymph-node and bone-metastasis work, which this paper extends §I; the cuda-convnet implementation §II-A; DropConnect regularisation §II-A; and the coarse-to-fine classification idea of Lu et al. §II-D.
• Compared against: Seff et al. (HoG with a linear classifier as the second tier) and Cherry et al. (random-forest second tier) on the lymph-node set §III-F; Feulner et al. and Nakamura et al. on lymph nodes §III-E; Summers et al. (2005) and Lu et al. on colonic polyps §III-H; Burns et al. on bone §III-D.
• Superseded by: not known from content already in this wiki.