Breast Cancer Classification from Histopathological Images with Inception Recurrent Residual Convolutional Neural Network

In plain words

A pathologist decides whether a stained slice of breast tissue shows cancer by looking at it under a microscope, often at several zoom levels. The work is slow, and people do not always agree. This paper trains a neural network to make that call from the slide images.

The network is built from repeated blocks. Each block looks at its input through filters of two sizes running side by side, plus a pooled branch. Each filter is also looped over a few passes, so its output feeds back into itself, and the block adds its input back onto its output as a shortcut. Between blocks the image is shrunk and the features are pooled.

The authors test the network on two public collections of breast tissue images, sorting them as benign or malignant and, in a second setting, into finer categories. Their reported scores are higher than earlier published methods on the same collections. These scores come from the authors' own experiments, and the paper does not show that the result holds on other data.

TL;DR

• IRRCNN is a CNN whose core block (IRRU) merges inception-style branches built from recurrent convolutional layers, plus a residual add §3 fig. 2 eq. 1 eq. 3 eq. 4.
• It is applied to breast histopathology classification on BreakHis (benign or malignant, and eight subtypes) and on the BCC Classification Challenge 2015 dataset (normal, benign, in situ, invasive) §4 tab. 1 tab. 2.
• Author-reported: 97.95% ± 1.07 image-level binary accuracy on BreakHis at 40x tab. 4; 99.05% image-wise binary accuracy on BCC 2015 with resized and augmented samples tab. 7.
• No official code is public. Several protocol details conflict within the paper (see Pitfalls), so reproducing the exact numbers needs care.

Problem

Manual review of biopsy slides at several magnifications is time-consuming and can lead to inaccurate diagnoses §1. Earlier machine-learning pipelines (hand-crafted texture features with SVM or random forest) were mostly tested on small, often private sample sets §2. Deep networks need large labelled datasets, and annotating histopathology is expensive §1. Whole-slide images are much larger than typical photos (in some cases over 2000 × 2000 pixels), so the images are typically fed to the model as several patches [§4, Experimental setup].

Method

The model is a stack of IRRCNN blocks with transition units between them, followed by a fully connected layer and a softmax §3 §4.

Each IRRU (Inception Recurrent Residual Unit) takes its input through several inception-style branches. The branches apply recurrent convolutional layers (RCLs). The branch outputs are concatenated along the channel axis, and the block input is added to the result §3 fig. 2 eq. 3 eq. 4. The recurrent part is an RCL: at each time step, a feed-forward convolution of the current input is summed with a recurrent convolution of the previous step's output, plus a bias, then passed through ReLU §3 eq. 1 eq. 2. In the experiments, t = 2 means one feed-forward convolution followed by two RCLs §4 fig. 2.

Transition units handle down-sampling. The paper uses overlapping max pooling, a 1 × 1 convolution, and dropout of 0.5 after each convolution in the transition block §3.

The paper's own description of the network in the experiments reads: two convolution layers at the start, four IRRCNN blocks with transition blocks between them, a fully connected layer and softmax, with about 9.3 million parameters §4.

# Recurrent convolution layer (RCL) unrolled over time  [eq. 1] [eq. 2]
# Input: x (B, C, H, W); state from step t-1 (B, K, H, W) or None at t=0
# Output: state at step t, (B, K, H, W)

def rcl_step(x, previous_state, t):
    # Feed-forward convolution on the input at this step  [eq. 1]
    forward = conv_forward(x)                   # (B, K, H, W)
    # Recurrent convolution on the previous output  [eq. 1] [§3]
    # Initial state at t = 0 is not stated; zero is assumed here
    if previous_state is None:
        recurrent = 0
    else:
        recurrent = conv_recurrent(previous_state)
    # Sum, add bias, then ReLU  [eq. 1] [eq. 2]
    return relu(forward + recurrent + bias)

# IRRU block: inception branches with RCLs, merged, plus a residual add
# [§3] [fig. 2] [eq. 3] [eq. 4]
# Input: x (B, C, H, W); output has the same shape (B, C, H, W) [§3]

def irru(x, time_steps):
    # Branches with 1x1 and 3x3 kernels only, each with RCLs  [§3] [fig. 2]
    branch_1x1 = unroll_rcl(conv_1x1(x), time_steps)
    branch_3x3 = unroll_rcl(conv_3x3(x), time_steps)
    # Average pooling followed by a 1x1 convolution  [eq. 3]
    branch_pool = conv_1x1(avg_pool(x))
    # Concatenate branch outputs along channels  [eq. 3]
    merged = concat_channels(branch_1x1, branch_3x3, branch_pool)
    # Residual: add the block input to the merged output  [eq. 4]
    # Order of batch norm relative to this add is not stated; BN is
    # applied after it here  [§3]
    return batch_norm(x + merged)

# Forward pass of the classifier  [§4] [§3] [eq. 5]
# Input: resized or center-cropped patch (B, 3, 128, 128) for BCC 2015 [§4]

def irrcnn_forward(images):
    # Two convolution layers at the start  [§4]
    features = conv_stem(images)
    # Four IRRCNN blocks, each followed by a transition unit  [§4] [§3]
    for stage in range(4):
        features = irru_block(features)
        features = transition(features)
    # Fully connected layer, then softmax over the classes  [§4] [eq. 5]
    return softmax(fully_connected(flatten(features)))

# Training of one trial  [§4 Training Methodology]
# Loss function not named; the network outputs a softmax  [eq. 5]

def train_trial(model, samples, labels, epochs):
    # SGD with momentum 0.9; initial learning rate and batch size not stated
    optimizer = sgd(momentum=0.9, learning_rate=not_stated)
    # Epoch count: 50 per trial in the text; fig. 7 runs ~150 [§4] [fig. 7]
    for epoch in range(epochs):
        # Learning rate divided by 10 after 50 epochs  [§4]
        if epoch == 50:
            optimizer.learning_rate /= 10
        # Training uses the augmented samples  [§4 Data Augmentation]
        for batch in batches(augment(samples), not_stated):
            loss = loss_fn(model(batch), labels_of(batch))
            optimizer.update(loss)

# Image-wise decision by winner-take-all over patches  [§4]
# Input: 200 random 128 x 128 patches from one image

def image_label_wta(patches, model):
    # Predicted class for each patch  [§4]
    votes = [argmax(model(patch)) for patch in patches]
    # The class with the most patches wins; tie rule not stated  [§4]
    return most_common(votes)

Patient-level and image-level scores use the definitions below. Patient score: correctly classified images over the patient's images, averaged over patients eq. 6 eq. 7 (see Pitfalls for the printed form). Image-level rate: correctly classified test samples over all test samples eq. 8.

Data augmentation (both datasets): sequential rotation by 40 degrees, width shift 0.2, height shift 0.2, shear 0.2, zoom range 0.2, horizontal flip and vertical flip §4 Data Augmentation. The paper also notes that some augmented samples contain added noise and reports a center-patch variant to avoid it §4 Data Augmentation fig. 5 fig. 6.

Hyperparameters

Results

Author-reported. Setup: BreakHis (7909 images, 82 patients, four magnifications), with 70/30 patient-disjoint splits and five trials averaged §4 Datasets tab. 1. BCC 2015 (249 images, four classes, binary and four-class tasks) §4 tab. 2. Baselines are taken from the cited works in the paper's tables; the paper states the protocol of those works but its reference list is incomplete in the fetched PDF (see Pitfalls).

Table 3: BreakHis eight-class accuracy (classification rate, %, mean ± SD) tab. 3

Table 4: BreakHis binary (benign vs malignant) accuracy (%, mean ± SD) tab. 4

Figure 8 gives ROC curves for the eight-class BreakHis setting. The legend reports AUC 0.9891 (40×), 0.9880 (100×), 0.9868 (200×) and 0.9871 (400×) fig. 8. Figure 7 plots training and validation accuracy by epoch for the four magnifications; the paper says the 100× setting had the best training accuracy, and the 200× setting was very close fig. 7 §4.

The paper's own summary: 1.05 and 0.55 points above the best earlier image-level (96.9) and patient-level (97.1) binary figures reported in [18] §4 Analysis and Comparison tab. 4.

BCC 2015, patch-based (Table 5: binary, Table 6: four-class), author-reported tab. 5 tab. 6:

BCC 2015, image-wise (Table 7: binary, Table 8: four-class), author-reported tab. 7 tab. 8:

Computation time per sample for training or inference is not specified in the text tab. 9:

Figure 9 plots training and validation accuracy for the four-class BCC setting under five input schemes: resized and augmented (RZ-AUG), center patch cropped and augmented (CRP-AUG), random patches (RP), resized (RZ), and center patch (CRP) fig. 9. The paper says the augmented schemes reach the highest accuracy §4 Results and Discussion.

Author-reported headline numbers on BCC 2015: 99.05% and 98.59% image-wise test accuracy for binary and four-class with resized and augmented samples; 100% with random patches and winner-take-all §4 Analysis and Comparison tab. 7 tab. 8. The text also reports 97.51% and 97.11% for augmented center patches in the binary and four-class cases §4 tab. 5 tab. 6.

Paper vs code

No official code.

Search, in order:

1. The paper text contains no code link. The arXiv metadata record (raw/arxiv/1811.04241.json) has no comments field.
2. No project page is linked.
3. GitHub search: api.github.com/search/repositories for "IRRCNN breast cancer" and "IRRCNN" returned one repository, ah01es/Breast-Cancer-Classification-with-IRRCNN-on-BreakHis-Dataset. Its owner is not an author, it is a from-scratch reimplementation, and its own description says the paper's official code is not public. Not used.
4. GitHub user search for the author "Zahangir Alom" found two accounts. The zahangircse repositories contain no breast-cancer or IRRCNN code. Its closest name is R2U-Net-models, which names a different model and is not linked from this paper. Not used. The Zahangir115 account has only a profile-configuration repository. Not used.

No discrepancies can be checked without the code.

Pitfalls

• No code, so no reference implementation. Several details that an implementer needs are missing from the text: initial learning rate, batch size, weight decay formula, loss function, weight initialization. Do not fill these from other papers; test a range and report the choice.
• Trial count conflicts. The dataset section says five trials are averaged §4 Datasets; the training section says three trials of 50 epochs §4 Training Methodology. Epoch counts also conflict: the text says 50 epochs per trial, while fig. 7 and fig. 9 plot about 150 fig. 7 fig. 9.
• Block count conflicts. The experiments section describes four IRRCNN blocks §4. The architecture section says "three IRRCNN units and one transition unit" for the implementation §3. Read this as a conflict, not a design choice.
• Feature-map assignment is ambiguous. "32 and 64 feature maps for the first three convolutional layers" does not say which layer gets which value §4.
• Branch structure is under-specified. Eq. 3 defines three branch outputs (1 × 1, an inception branch y(x) that the text does not give a kernel size for, and pooled 1 × 1). Fig. 2 shows chains of conv + ReLU with RCLs and does not give channel counts. The order of batch norm relative to the residual add is not stated.
• Residual dimensions. The text says the residual branch has the same number of feature maps as the IRRCNN unit §3. Make the branch output channel count match the input channel count.
• Activation. ELU was tried §3, but the results are not reported separately from ReLU.
• Patient-level equations are inverted as printed. Eq. 6 is written as P_s = N_ncp / N_ntp. The prose defines N_ncp as the number of the patient's images and N_ntp as the number of correctly classified images §4 Results. Read as printed, the ratio is inverted. The intended ratio (correct over total) is inferred from the definitions, not stated.
• Patient-level decision rule is not stated. How a patient score becomes a patient-level label is not described. Eq. 7 gives only the mean score.
• Augmentation counts conflict. The text says 21 augmented samples per input §4 Datasets and 20 for BCC §4. The BreakHis per-magnification totals are 41,895, 43,701, 42,273 and 38,220, which sum to 166,089. The paper states 166,068.
• BreakHis benign count conflicts with Table 1. The text gives 2440 benign and 5429 malignant samples §4 Datasets. The subclass rows of Table 1 sum to 2480 benign and 5429 malignant, which gives 7909 total tab. 1. The table is internally consistent; the text's benign count is not.
• Time per sample for BreakHis is inconsistent. 72.06 s over 8732 samples is about 0.0083 s, but Table 9 lists 0.08 tab. 9. The table does not say whether time covers training or inference.
• Small BCC test set. The 2015 dataset has 249 images. Training uses 229 §4, so the test set is about 20 images. Each image is about 5 percentage points of accuracy, and the 100% image-wise results come from that set. Patch-level test counts are not stated.
• Resize degrades detail. The authors note that resizing 2040 × 1536 images to 128 × 128 "significantly degrades" the information §4. Patches of 128 × 128 are chosen because nuclei radii of 3 to 11 pixels (1.26 to 4.62 µm) fit a patch §4.
• Augmentation noise. Some augmented samples contain added noise §4 Data Augmentation fig. 5. The paper evaluates a center-patch variant partly for this reason.
• Reference list is incomplete. The fetched PDF's reference list stops at [14], while the text cites up to [31]. Citations [15] to [31] (including the baseline results) cannot be checked against the PDF. The text cites "Inception-v4" as [2], but [2] in the list is a clinical pathology paper.
• Baseline numbers may be copied. Several baseline rows are identical or nearly identical across tables. For example, the CNN+fusion [15] image row in Table 4 has the same values as AlexNet+Aug [18] at 40×, 100× and 400× tab. 4. These baseline numbers are not verified against the cited works.
• Naming inconsistency. Fig. 8's legend uses "IR2CNN" while the rest of the paper uses IRRCNN fig. 8.
• Author-reported claims. The paper calls its results state of the art. Treat that as an author claim; the comparisons span different protocols and are not re-run.

Lineage

• Builds on: Inception-v4 (cited in the text as [2]), inception networks [21], residual networks [23] and RCNN [24]; the IRRCNN name is attributed to [19], [20]. None of these entries are linked from the paper, and most of them are not in the fetched PDF.
• Compared against: CNN with fusion [15], Bayramoglu et al. [14], Gupta et al. [13], AlexNet+Aug, LeNet+Aug, and CSDCNN+Aug [18], CNN and CNN+SVM [17], ASSVM [28].
• Superseded by: not known from content in this wiki.