Pathologist-level Classification of Histologic Patterns on Resected Lung Adenocarcinoma Slides with Deep Neural Networks

In plain words

A whole-slide image is a very large digital picture of a tissue slide, and a pathologist reads it by looking for the tissue pattern that covers most of the tumour. Lung adenocarcinoma has several such patterns, and they matter for prognosis. Pathologists often disagree on which pattern dominates, so the authors asked whether a computer model could make the call as consistently as the experts do.

The model works like a careful but mechanical reader. It cuts the slide into many small square tiles, classifies each tile on its own, and throws away the tiles it is unsure about. The pattern that appears most among the remaining tiles is called the main one, and the other tumour patterns are listed as minor. The authors compare the model's answers with three pathologists on slides none of them had seen during training.

TL;DR

Use this card for a whole-slide tile classifier that predicts predominant and minor growth patterns. The patch model is a ResNet-18 trained from scratch, and the slide step is a per-class confidence threshold followed by a counting rule. Author-reported results put the model level with the pathologists on a 143-slide test set. The official code is GPL-3.0, its defaults differ from the paper in several places, and the aggregation rule is not implemented as the paper describes it. Read "Paper vs code" before copying anything.

Problem

Lung adenocarcinoma pattern grading is subjective. Pathologists disagree on the patterns, and the agreement scores reported by earlier studies are moderate §1. Predominant and minor patterns both affect prognosis and treatment §1. The authors frame the task as slide-level classification with no ground truth. Pathologist labels are the only reference, so Cohen's kappa and agreement replace F1 or AUC as the main metrics §4 Statistical analysis. The paper reports no earlier automated system for these patterns §3.

Method

The pipeline has two stages. A patch classifier is trained on annotated crops, then it is run over every tile of a whole slide and its outputs are aggregated §4 Whole-slide inference fig. 1.

Data

Pathologists drew variable-size rectangular crops on 245 training slides, with about 17 crops per slide. Crops are labelled as one of five patterns or as benign. The benign class also includes inflammation, scarring, fibrosis and artifacts §4 Slide Annotation tab. 1. A separate development set of 1,068 fixed-size 224 x 224 patches, drawn from classic examples of each pattern, was checked by two pathologists, and disputed patches were removed [§4 Slide Annotation]. Slides were randomly partitioned into 279 for training and development and 143 for testing. The 279 were then split into 245 training and 34 development slides, a split whose method is not described §4 Data Collection [§4 Slide Annotation] tab. 1.

# Patch classifier training  [§4 Data processing] [§4 Training]
# Input:  crops_by_class: {pattern: [crop, ...]}, six classes with benign
# Output: model: ResNet-18 with six outputs

def train_patch_classifier(crops_by_class):
    # 1. Slide a fixed-size window over each crop; the training window size
    #    is not stated (224 px is used for the development patches)  [§4]
    # 2. Choose the overlap per class so that class sizes are roughly equal;
    #    the paper gives no formula  [§4 Data processing]
    patches = build_patches_per_class(crops_by_class)

    # 3. Normalise each colour channel with the mean and std of the whole
    #    training set  [§4 Data processing]
    mean, std = channel_mean_and_std(patches)

    # 4. Initialise ResNet-18 with He initialisation  [§4 Training]
    model = resnet18(num_classes=6, init="he")

    # 5. Train 50 epochs with multi-class cross-entropy; the learning rate
    #    starts at 0.001 and is multiplied by 0.9 after each epoch  [§4]
    for epoch in range(50):
        for patch, label in shuffled(patches):
            # 5a. Colour jitter on brightness, contrast, saturation and hue
            #     (strengths not stated)  [§4 Data processing]
            patch = colour_jitter(patch)
            # 5b. Random horizontal and vertical flips, then a 90-degree
            #     rotation  [§4 Data processing]
            patch = flip_and_rotate(patch)
            loss = cross_entropy(model(normalise(patch, mean, std)), label)
            update_weights(model, loss)  # optimiser not stated  [§4]
    return model

Whole-slide inference

The authors tile each slide with a sliding window that overlaps by one fifth, giving about 9,267 patches per slide on average §4 Whole-slide inference. Each patch is classified, and the aggregation then runs in three steps. First, predictions below a per-class confidence threshold are discarded. Second, classes that make up less than 5% of the remaining predictions, and the benign class, are dropped. Third, the most frequent class is the predominant pattern and the other remaining cancerous classes are minor patterns [§4 Whole-slide inference]. Thresholds are chosen per class by a grid search that maximises agreement with whole-slide labels on the development set [§4 Whole-slide inference]. Inference takes about 30 seconds per slide on the authors' hardware §4 Whole-slide inference.

# Whole-slide inference  [§4 Whole-slide inference]
# Input:  slide; model; thresholds: {pattern: float}, grid-searched on dev
# Output: predominant pattern and list of minor patterns

def classify_slide(slide, model, thresholds):
    # 1. Tile the slide with a sliding window; overlap "one fifth" is not
    #    defined further (stride or overlapped area)  [§4]
    patches = sliding_windows(slide, overlap=0.2)
    # 2. Classify each tile and keep only confident predictions  [§4]
    kept = [(pattern, conf) for pattern, conf in classify(model, patches)
            if conf > thresholds[pattern]]  # strict ">" not stated
    # 3. Count kept predictions per pattern, then drop benign and any
    #    pattern under 5% of the predictions  [§4]
    shares = share_per_pattern(kept)  # denominator not stated
    shares = drop_benign_and_small(shares, min_share=0.05)
    # 4. The most frequent pattern is predominant; the rest are minor  [§4]
    predominant = argmax(shares)
    minor = [p for p in shares if p != predominant]
    return predominant, minor

Model and training

The patch classifier is ResNet-18, chosen because it had the fewest parameters and fastest training among 18, 34, 50, 101 and 152 layers, all of which gave similar development accuracy §4 Training. The network is trained from initialised weights with no pretrained model mentioned §4 Training. Training took 24 hours on one NVIDIA K40c GPU §4 Training.

Hyperparameters

Values in the paper are cited to the paper. Code defaults are listed in the Notes column and cited to the repository. The paper states the value of only a few of the items below.

Results

Author-reported, on the 143-slide test set. The kappa, agreement and robust agreement compare each annotator with the others, and the model is one of the annotators. Average kappa is computed by averaging over pairs that include that annotator tab. 2. The model's per-pair values are in fig. 3, which the card does not reproduce.

Setup: three pathologists labelled each test slide independently for predominant and minor patterns, and the model was run on the same slides §4 Statistical analysis. Baseline: the slide-averaging method of Coudray et al., extended to multi-label prediction by the authors §2 Results. Metrics are predominant-pattern measures tab. 2.

The authors state that every metric for the model and the three pathologists falls within the 95% confidence intervals of agreement, and they ran two-sample t-tests on each pair of metrics §2 Results §4 Statistical analysis. The paper does not name the interval method for kappa.

Patch-level results on the development set: F1 of 90.4% on the patches, AUC of at least 0.97 for every class §2 Results fig. 2. The per-class precision, recall and F1 values are in fig. 2A and not restated in the text.

Dataset counts tab. 1:

Paper vs code

Official code: BMIRDS/deepslide, GPL-3.0, commit 18e5412f, the default branch HEAD when cloned. It is a v1.0 rewrite dated 2019-12-18, after the paper's arXiv version, so it is not the code that produced the paper's numbers RELEASES.md. The repo is checked as a whole, and examples are cited by path. Nothing is quoted. Searches: the paper's abstract link, then a repository read.

• Model selection. The paper selects the architecture on 1,068 classic development patches §4 Slide Annotation. The code picks the checkpoint with the highest validation accuracy on patches cut from validation slides, which default to 20 slides per class config.py utils_model.py. The classic-patch set is not in the repository.
• Training length and decay. 50 epochs and x0.9 per epoch in the paper, against 20 epochs and x0.85 in the code defaults config.py config.py.
• Optimiser. The paper names none. The code uses Adam with weight decay 1e-4 utils_model.py.
• Balancing. The paper balances classes only through per-class overlap §4 Data processing. The code also copies files until every class matches the largest one utils_processing.py. The overlap formula uses a per-image size of 0.013 MB, and a code comment says part of the constant was added without a clear basis utils_processing.py.
• Background filter. The code keeps only purple-stained windows by default, dropping whitespace and other non-purple tiles. The paper does not mention this step utils_processing.py utils_processing.py config.py.
• Rotation. The paper rotates by 90 degrees §4 Data processing. The code picks at random from 0, 90, 180 and 270 degrees, so 0 degrees is possible utils_model.py utils_model.py.
• Normalisation statistics. The paper uses the training set. The code computes them over the training folder when training starts, so any duplicated copies from balancing count config.py.
• Weight initialisation. The paper states He initialisation. The code sets none explicitly utils_model.py.
• Confidence thresholds. The paper uses one threshold per class. The code applies one shared threshold to all classes and searches 0.3 to 0.9 utils_evaluation.py config.py. The comparison is strict ">" utils_evaluation.py.
• Aggregation rule. The code takes the argmax of the thresholded shares over all classes. It has no 5% rule and no minor-pattern output. A benign class, if present as a folder, can win utils_evaluation.py utils.py.
• Threshold selection metric. The code maximises average per-class accuracy on validation slides, not kappa utils_evaluation.py utils_evaluation.py.
• Statistics. The repository prints accuracy and a confusion matrix on the test set 7_final_test.py. Kappa, robust agreement and the t-tests are not in the repository.
• Window overlap. The paper says one fifth. The code default is slide_overlap 3, so the stride is patch size divided by 3, about 74 px at 224 px config.py utils_processing.py. The README says the default overlapping area is 1/3 README.md. The stride arithmetic and the README do not agree.
• Data availability. The paper says the dataset is not public §7 Data Availability Statement. The README says the 143 test slides and their predominant-pattern labels are public at the dataset webpage README.md.

Pitfalls

• Inter-pathologist kappa. The abstract gives 0.485, but Table 2 and the results text give 0.479 [abstract] tab. 2. Use 0.479 and note the mismatch.
• Averaging. Each table value averages pairwise comparisons that include the model. The row labelled inter-pathologist is not defined as precisely as its name suggests tab. 2 caption.
• Overlap wording. "Overlapped by a factor of one-fifth" does not say whether the stride or the overlapped area is one fifth [§4 Whole-slide inference]. Slide dimensions are not given, so the 9,267 mean patches per slide §4 Whole-slide inference cannot settle the question.
• 5% denominator. The paper does not say whether the 5% share uses predictions before or after the threshold.
• Threshold grid. The grid resolution and the search range for the paper's per-class thresholds are not stated §4 Whole-slide inference.
• Benign handling. The benign class is dropped in the paper's aggregation, but the code keeps it in the argmax utils_evaluation.py.
• Ground truth. No slide has a single ground truth. Each test label is one annotator's view, and the model is scored against each of them §4 Statistical analysis.
• Significance. The text says the model "edges out" the pathologists, but its kappa CI overlaps every pathologist's CI tab. 2. The Discussion's claim is "on par" with pathologists §3 Discussion.
• Rare classes. Papillary has 9 training slides and 5 test slides, and micropapillary 41 and 22 tab. 1. Results for these patterns are the least reliable.
• Single centre. All slides come from one medical centre §3 Discussion.
• Reproducibility. The code sets no random seed for training or for the split. The only seed in the repository is in a preprocessing script 4_anonymize.py. Expect run-to-run variation.
• Fragile comparison. The loader compares a string with is to choose shuffling. This works in CPython by accident, and it should be == utils_model.py.
• Scale. Training took 24 hours on one K40c GPU. Inference takes about 30 seconds per slide and generates thousands of tiles §4 Training [§4 Whole-slide inference].
• Licence. The code is GPL-3.0, so it is cited by path and line only and never quoted. Implement from the pseudocode above and the citations.

Lineage

• Builds on: ResNet (He et al., CVPR 2016), cited as the network in [§4 Residual neural networks]. He initialisation is also cited to its source §4 Training.
• Compared against: the slide-averaging method of Coudray et al. (Nature Medicine, 2018), extended by the authors to multi-label prediction [§2 Results].
• Later work: the repository README points to an end-to-end attention-based model in JAMA Network Open README.md. Its content is not used here.
• Superseded by: not stated in this card's sources.