wikillm

Paper · 2020-02-20 · cs.LG, cs.CV, eess.IV

Comparing Different Deep Learning Architectures for Classification of Chest Radiographs

Trains 15 CNN architectures (ResNet, DenseNet, VGG, SqueezeNet, AlexNet) on five CheXpert findings with 320 px inputs, two batch sizes and five repeats, then scores AUROC and AUPRC on the CheXpert validation set. Shallow models such as ResNet-34, AlexNet and VGG-16 match deeper ones at lower training cost. Results are author-reported and contain several text-table mismatches.

Convolutional neural networkTransfer learningMulti-label classificationData augmentationmedical-imagingchest-radiographmodel-comparisonbenchmark

arXivPDFMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Chest X-rays are grey, mostly show a small set of conditions, and are usually read with models that were first built for large colour photo collections. This paper asks whether those big models are needed. It takes fifteen common image-classifier designs, from small to very deep, and trains each one to spot five findings on chest X-rays from the public CheXpert dataset.

The main finding is that the small designs do about as well as the deep ones on these pictures, while taking less time and memory to train. The numbers are the authors' own and come from one dataset, so they show a trend rather than a settled ranking. The paper also points out several places where its text and its tables disagree, so the tables are the more reliable source.

TL;DR

  • Compares 15 convolutional networks for five chest X-ray findings on the CheXpert dataset, using AUROC and AUPRC on the CheXpert validation set § Methods (Model training) § Methods (Evaluation).
  • Shallow networks (ResNet-34, AlexNet, VGG-16) are competitive with deep ones (ResNet-152, DenseNet-161) on AUROC, and shallow ones score higher on AUPRC § Results (AUROC) § Results (AUPRC).
  • Best pooled AUROC at batch size 32 is 0.882 (ResNet-152), against the CheXpert baseline the authors cite, 0.889 tab. 1.
  • Training time is reported per model; AlexNet trains fastest and DenseNet-201 slowest tab. 3. No inference timing or parameter counts are given.

Problem

Most chest-radiograph classifiers reuse networks built for large colour image sets with 1000 classes. Chest X-rays are greyscale and have few labels, so it is unclear which ImageNet-style architecture suits them. Higher input resolution increases memory use, which pushes training towards smaller batches, and the authors argue that deeper networks cost more to train and deploy § Introduction. The paper asks which of the available architectures perform best on this task, given that cost § Introduction.

Method

Data. The authors use CheXpert: 224,316 radiographs from 65,240 patients, with 14 findings per image labelled present, absent or uncertain § Methods (Data preparation). They train on five findings: cardiomegaly, edema, consolidation, atelectasis and pleural effusion § Methods (Data preparation). Every image with an uncertain label is dropped, and only frontal views are kept, leaving 135,494 training images from 53,388 patients § Methods (Data preparation). The CheXpert validation set is the evaluation set. Its count is stated inconsistently (see Pitfalls) § Methods (Data preparation) § Results.

Preprocessing and augmentation. Images are resized to 320 × 320 pixels with bilinear interpolation and their pixel values normalised § Methods (Data augmentation). During training they are flipped horizontally and vertically, rotated by up to 10°, zoomed by up to 110%, given random lighting changes, or symmetrically wrapped § Methods (Data augmentation). Probabilities for each transform are not stated.

Architectures. Fifteen networks are compared: ResNet-18, 34, 50, 101 and 152; DenseNet-121, 161, 169 and 201; AlexNet; SqueezeNet-1.0 and 1.1; and VGG-13, 16 and 19 tab. 1 tab. 3. The text repeatedly says "14" networks (see Pitfalls) § Methods (Model training).

Training. Each network gets one multilabel classification head for the five findings § Methods (Model training). Training runs in two sessions, one per batch size (16, then 32) § Methods (Model training). In each run, the network body is frozen for the first five epochs and only the head is trained. The model is then unfrozen and trained as a whole for three more epochs, so eight epochs in total § Methods (Model training). A learning rate is found before training and again after epoch five, using the search method of Smith (2017), which the paper cites. The head phase gets a rate between 1e-1 and 1e-2, and the full-model phase between 1e-5 and 1e-6 § Methods (Model training). The exact values picked are not stated. Each configuration is repeated five times, and the validation predictions are exported as CSV § Methods (Model training).

Pseudocode
# Two-phase training of one architecture  [§ Methods] (Model training)
# Input:  architecture, batch_size in {16, 32}, train_set, val_set
# Output: five sets of validation predictions

def train_one_architecture(architecture, batch_size, train_set, val_set):
    # Repeat the whole procedure five times  [§ Methods] (Model training)
    predictions = []
    for run in range(5):
        model = architecture(num_outputs=5)  # one multilabel head
        # Find the head rate in 1e-1..1e-2 before training  [§ Methods]
        head_lr = find_learning_rate(model, low=1e-2, high=1e-1)
        # Phase 1: body frozen, only the head trains, 5 epochs  [§ Methods]
        freeze_body(model)
        train(model, train_set, epochs=5, lr=head_lr, batch=batch_size)
        # Find the full-model rate in 1e-6..1e-5 after epoch 5  [§ Methods]
        full_lr = find_learning_rate(model, low=1e-6, high=1e-5)
        # Phase 2: unfreeze everything, 3 more epochs  [§ Methods] (Model training)
        unfreeze_all(model)
        train(model, train_set, epochs=3, lr=full_lr, batch=batch_size)
        predictions.append(predict(model, val_set))
    return predictions

Evaluation. Predictions are analysed in R with tidyverse and ROCR § Methods (Evaluation). For each run, and for the five runs pooled as one consortium, ROC and precision-recall curves are drawn and AUROC and AUPRC computed § Methods (Evaluation). The two metrics are threshold-free, and the paper uses them because the class prevalences differ (cardiomegaly 33%, edema 21%, consolidation 16%, atelectasis 37%, pleural effusion 32%) § Results (AUROC). The paper does not say how the pooled predictions are combined.

Figures. ROC curves for all models are in fig. 1 to fig. 3; coloured lines are single runs and black lines the pooled result § Figures (Receiver Operating Characteristic Curves). Precision-recall curves are in fig. 4 to fig. 6 § Figures (Precision Recall Curves). The figures are not reproduced here.

Hyperparameters

NameValueWhereNotes
Input size320 × 320 px§ Methods (Data augmentation)bilinear resize
Pixel normalisationapplied§ Methods (Data augmentation)statistics not stated
Batch size16 and 32§ Methods (Model training)two sessions; 32 gave most of the best results § Results (AUROC)
Epochs8 total§ Methods (Model training)first 5 head only, then 3 full
Head learning rate1e-1 to 1e-2§ Methods (Model training)range from search; value used not stated
Full-model learning rate1e-5 to 1e-6§ Methods (Model training)range from search; value used not stated
Learning-rate searchbefore training and after epoch 5§ Methods (Model training)method of Smith (2017), as cited
Repeats5 per architecture and batch size§ Methods (Model training)
Horizontal and vertical flipson§ Methods (Data augmentation)probability not stated
Rotationup to 10°§ Methods (Data augmentation)distribution not stated
Zoomup to 110%§ Methods (Data augmentation)zoom in or out not stated
Lighting and symmetric wrapon§ Methods (Data augmentation)magnitudes not stated
Optimisernot stated
Loss functionnot statedmultilabel head, loss not named
Weight initialisationnot statedfreezing the body for five epochs suggests pretrained weights, but the paper does not say so
Weight decay, momentum, schedulernot stated
Training labels5 findings§ Methods (Data preparation)uncertain-label images excluded
Training images135,494 from 53,388 patients§ Methods (Data preparation)frontal only
Hardware2 × Nvidia GeForce RTX 2080 Ti, 11 GB each§ Methods (Model training)Ubuntu 18.04
SoftwarePython 3.8, PyTorch, FastAI§ Methods (Model training)library versions other than Python not stated

Results

Author-reported. Setup: CheXpert validation set, five findings, 15 architectures, batch sizes 16 and 32, five repeats per configuration. The reference is the CheXpert baseline reported by Irvin et al., which the authors did not retrain § Results (AUROC) tab. 1. Values are from the paper's own tables; the text repeats some of them with mismatches, noted below.

AUROC (pooled, batch size 32) tab. 1 § Results (AUROC)

ModelBatchAtelectasisCardiomegalyConsolidationEdemaEffusionPooled
CheXpert baseline (Irvin et al., reported)160.8180.8280.9380.9340.9280.889
ResNet-18160.8160.7970.9050.8680.8990.857
ResNet-34160.7990.7980.9020.8910.9050.859
ResNet-50160.7980.7990.8900.8800.9130.856
ResNet-101160.8130.8100.9050.8890.9070.865
ResNet-152160.8010.8090.9080.8960.9160.866
DenseNet-121160.8090.7940.8950.8830.9060.857
DenseNet-161160.8000.8170.8850.9000.9230.865
DenseNet-169160.8050.7950.8980.8910.9090.860
DenseNet-201160.8050.8120.8910.8860.9160.862
AlexNet160.7900.7550.8570.8940.8810.835
SqueezeNet-1.0160.7610.7550.8330.9070.8850.828
SqueezeNet-1.1160.7670.7640.8800.9030.8790.839
VGG-13160.7980.7520.8860.8670.8720.835
VGG-16160.8090.7660.8920.8790.8830.846
VGG-19160.8110.7860.9010.8900.8840.854
ResNet-18320.7960.8220.9080.9030.9110.868
ResNet-34320.7970.8400.9030.9020.9190.872
ResNet-50320.8110.8320.9160.9130.9340.881
ResNet-101320.7970.8230.9110.9150.9360.876
ResNet-152320.8020.8360.9170.9200.9370.882
DenseNet-121320.8080.8280.8790.9040.9260.869
DenseNet-161320.8090.8340.9130.9230.9280.881
DenseNet-169320.8090.8160.9000.9220.9340.876
DenseNet-201320.7950.8200.9040.9220.9310.874
AlexNet320.7910.7680.8560.8940.8860.839
SqueezeNet-1.0320.7730.7690.8800.9130.8950.846
SqueezeNet-1.1320.7850.7890.8950.9040.8980.854
VGG-13320.8000.7620.8830.8960.9070.850
VGG-16320.7980.7760.8900.9110.9060.856
VGG-19320.7870.7900.8790.9110.9160.857

AUPRC tab. 2 § Results (AUPRC)

ModelBatchAtelectasisCardiomegalyConsolidationEdemaEffusionPooled
ResNet-18160.5000.5590.8060.7270.5800.634
ResNet-34160.5060.5600.8040.7350.5800.637
ResNet-50160.5010.5570.8020.7330.5850.636
ResNet-101160.4990.5580.7650.7350.5820.628
ResNet-152160.5030.5590.8080.7370.5840.638
DenseNet-121160.5030.5540.8020.7330.5800.634
DenseNet-161160.5010.5570.7990.7360.5870.636
DenseNet-169160.5000.5600.8050.7330.5820.636
DenseNet-201160.3200.5550.4450.7340.5820.527
AlexNet160.5430.5650.4900.7330.5770.582
SqueezeNet-1.0160.5090.5650.4250.7360.5760.562
SqueezeNet-1.1160.5050.5630.4000.7330.5750.555
VGG-13160.5020.5630.7610.7260.5740.625
VGG-16160.5010.5590.7970.7330.5770.633
VGG-19160.5000.5580.8080.7310.5770.635
ResNet-18320.5020.5570.8050.7360.5820.636
ResNet-34320.6520.5560.8060.7370.5850.667
ResNet-50320.4970.5550.8090.7400.5900.638
ResNet-101320.5000.5580.8080.7400.5910.639
ResNet-152320.5020.5590.8100.7410.5910.641
DenseNet-121320.5000.5580.7930.7360.5870.635
DenseNet-161320.4990.5560.8080.7430.5890.639
DenseNet-169320.4990.5560.8050.7430.5880.638
DenseNet-201320.5020.5550.8080.7420.5890.639
AlexNet320.7200.5620.7890.7310.5780.676
SqueezeNet-1.0320.3540.5620.8150.7380.5800.610
SqueezeNet-1.1320.5060.5630.8040.7310.5770.636
VGG-13320.5010.5600.7990.7350.5780.635
VGG-16320.7320.5610.8040.7390.5820.684
VGG-19320.5010.5620.8000.7400.5850.638

Headline AUROC, as the text states it § Results (AUROC)

  • Pooled, batch size 32: ResNet-152 0.882, DenseNet-161 0.881, ResNet-50 0.881. The CheXpert baseline cited is 0.889.
  • Per finding: atelectasis best ResNet-18 0.816 (batch 16); cardiomegaly ResNet-34 0.840; edema DenseNet-161 0.923; effusion ResNet-152 0.937. Each of the best cardiomegaly models and the effusion models exceeds the cited baseline (0.828 and 0.928).

Headline AUPRC as the text states it § Results (AUPRC)

  • Pooled: VGG-16 0.709, AlexNet 0.701, ResNet-34 0.688. Table 2 gives other values for these models (see Pitfalls).
  • Atelectasis: VGG-16 and AlexNet 0.732 (text); Table 2 gives 0.732 for VGG-16 and 0.720 for AlexNet.

Overall, both metrics § Results (Overall best Performance): VGG-16 (AUROC 0.856, AUPRC 0.709), ResNet-34 (0.872, 0.688) and AlexNet (0.839, 0.701), all at batch size 32.

Training time tab. 3 § Results (Training time). Overall, 340 hours. Changing the batch size from 16 to 32 gave, on average, a 29.9% ± 9.34% speed-up (as reported). Table 3 times are the average of five runs, rounded to the minute.

ModelBatchPer epochTotal
ResNet-18166 min50 min
ResNet-341610 min1 h 13 min
ResNet-501611–13 min1 h 40 min
ResNet-1011619–25 min2 h 47 min
ResNet-1521627–28 min4 h 7 min
SqueezeNet-1.0164–6 min39 min
SqueezeNet-1.1164 min37 min
AlexNet163 min24 min
VGG-131612 min1 h 49 min
VGG-161620–21 min2 h 14 min
VGG-191624 min2 h 40 min
DenseNet-1211623–25 min3 h 7 min
DenseNet-1691631–34 min4 h 21 min
DenseNet-1611629–36 min4 h 17 min
DenseNet-2011641 min5 h 11 min
ResNet-18324 min31 min
ResNet-34325–7 min45 min
ResNet-50328 min1 h 16 min
ResNet-1013213 min2 h 8 min
ResNet-1523221–26 min2 h 58 min
SqueezeNet-1.0323–4 min28 min
SqueezeNet-1.1323 min25 min
AlexNet322–3 min20 min
VGG-133210–14 min1 h 31 min
VGG-163217 min1 h 47 min
VGG-193213 min2 h 2 min
DenseNet-1213212–16 min1 h 49 min
DenseNet-1693217 min2 h 25 min
DenseNet-1613221–27 min3 h 6 min
DenseNet-2013220 min2 h 52 min

Conclusions the paper draws § Conclusion § Discussion: smaller networks can match or beat deeper ones on this task; the authors report the best results with networks "up to 95% smaller" than DenseNet-121, a figure the paper does not define. Shallow models need less GPU memory and train faster.

Paper vs code

No official code. The paper gives no repository link. Its only URLs are for Python, PyTorch and FastAI, plus a FastAI GitHub entry in the bibliography § Methods (Model training). Checked: the first author's GitHub profile (26 public repositories, none on chest radiographs or CheXpert), and two GitHub repository searches for the CheXpert architecture comparison (no result owned by an author). The paper's own description of the code is the only source for the pipeline.

Pitfalls

  • Validation set counts do not agree. The Methods say the CheXpert validation set has 235 images, 201 after removing uncertain labels and lateral views § Methods (Data preparation). The Results say 234 studies of 200 patients, with 202 images left after removing 32 lateral views § Results. The per-finding percentages and the AUROC values cannot be matched to one count.
  • Network count. The abstract says fifteen architectures and Table 1 and Table 3 list fifteen (five ResNet, four DenseNet, one AlexNet, two SqueezeNet, three VGG). The Model training section and the Training time section say "14". The training-time section also gives 210 runs, which does not match 15 architectures × 2 batch sizes × 5 repeats = 150 multilabel runs. The text says each network also got a binary head per finding, once each; those runs are never described in Methods or reported in the tables § Results (Training time).
  • Finding count. The Data preparation section says CheXpert has fourteen findings and then lists thirteen names § Methods (Data preparation).
  • Text and Table 2 disagree on pooled AUPRC. The text gives VGG-16 0.709, AlexNet 0.701 and ResNet-34 0.688. Table 2 at batch size 32 gives 0.684, 0.676 and 0.667 § Results (AUPRC) tab. 2. The atelectasis value for AlexNet is 0.732 in the text and 0.720 in Table 2. The text also names "ResNet-24", which is not in the study, and "CGG-16" and "Alexnet-152" (presumably VGG-16 and AlexNet). Trust the tables.
  • "Congestion" in the text is the consolidation column. The AUROC text gives "congestion" values (ResNet-152 0.917, ResNet-50 0.916, DenseNet-161 0.913) that match the consolidation column in Table 1, not edema § Results (AUROC) tab. 1.
  • Longest model at batch 32. The text says DenseNet-169 took the longest time at batch size 32 (3:06 h). Table 3 gives DenseNet-161 3 h 6 min and DenseNet-169 2 h 25 min § Results (Training time) tab. 3.
  • Precision-recall figure references. The precision-recall subsection says "Figures 1, 2 and 3" (the ROC figures), while its headings place precision-recall curves in Figures 4 to 6 § Figures (Precision Recall Curves).
  • Unstated training details. Optimiser, loss, the learning-rate values used, weight decay, scheduler, weight initialisation, normalisation statistics, augmentation probabilities and magnitudes, and the zoom direction are not stated. An implementer must choose them, and the choices will move the results. Treat the phase-one and phase-two learning-rate ranges as search bounds, not settings.
  • Label coding. The paper writes "absent (NA)" and "uncertain (-1)" for the label values § Methods (Data preparation). The coding used in training is not given, and "NA" is ambiguous for absent.
  • Model selection and evaluation split. Batch size 32 was chosen because it gave better results on the same validation set that the paper reports on § Results (AUROC). No separate test split is described. The pooling of the five runs' predictions is not specified.
  • Noise is not reported. Each configuration was run five times, but no standard deviation is given for AUROC or AUPRC. Differences of about 0.001 to 0.01 between the top models (for example 0.881 and 0.882) may be within run variation. The only spread reported is for training time.
  • The baseline is not like-for-like. The 0.889 CheXpert baseline is the number Irvin et al. published, not a rerun. The paper excludes uncertain-label images, which Irvin et al. handle differently. The authors also say their hyperparameters were probably less tuned than the original work § Discussion.
  • Cost claims. "Less computationally demanding" rests on training time on two RTX 2080 Ti cards. Parameter counts, memory use and inference time are not reported. The "up to 95% smaller" figure is not explained.
  • Batch size 16 row in Table 1. The CheXpert baseline row is labelled batch size 16 in the table, but it is the published result, not a model trained here tab. 1.

Lineage

  • Builds on: CheXpert dataset and its baseline, as published by Irvin et al. (cited, no card) § Methods (Data preparation) § Results (AUROC).
  • Learning-rate search: Smith (2017), as cited § Methods (Model training).
  • Compared against in the Discussion, all cited, none carded: Pham et al. (DenseNet-121, hierarchical labels, mean AUROC 0.940 as cited), Allaouzi et al. (DenseNet-121 on CheXpert and ChestX-ray14), Sabottke et al. (ResNet-32) and Raghu et al. (ResNet-50, Inception-v3 and a small custom network) § Discussion.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “Comparing Different Deep Learning Architectures for Classification of Chest Radiographs” by Keno K. Bressem, Lisa Adams, Christoph Erxleben, Bernd Hamm, Stefan Niehues, Janis Vahldiek (arXiv:2002.08991), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.