nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation

In plain words

Medical image segmentation means outlining organs or tumours in 3D scans. Many papers claim that a new neural network design beats the U-Net, the standard architecture for this job. This paper asks whether those claims survive a fair test. Its answer is mostly no. When the comparisons are made carefully, the old U-Net, configured well by the nnU-Net framework and given enough GPU memory, beats most of the newer designs.

The authors list common ways such comparisons go wrong. Examples include giving the new method extra training data or a stronger encoder that the baseline lacks, comparing against a poorly tuned baseline, using a single small test split, and picking only the classes that look good. They also rank six public datasets by how well they tell methods apart, and find that some of the most popular ones are poor for that purpose.

The lesson for an engineer is to check the baseline before trusting a new architecture's gain, and to measure on datasets and splits that can actually show a difference.

TL;DR

• Benchmarks CNN, Transformer and Mamba segmentation models on six 3D datasets, using five-fold cross-validation with nnU-Net's splits §3.2 tab. 1.
• Most recent Transformer and Mamba claims do not hold against the original nnU-Net. The strongest entries are CNN U-Nets: nnU-Net ResEnc presets, MedNeXt and STU-Net §4.
• Author-reported. Code is Apache-2.0 nnU-Net; the other methods' code is not part of this card's source.

Problem

Papers that propose a new 3D segmentation model often compare against a baseline that is not fully tuned, is trained on less data or without the same compute, or is measured on a small or unrepresentative split. The authors argue that recent claims of superiority over the U-Net therefore cannot be trusted without a controlled comparison §1 §2.

They identify four pitfalls: confounding performance boosters (P1), baselines that are not well configured or standardised (P2), insufficient or unsuitable datasets (P3), and inconsistent reporting such as custom splits, ensembles and test-time augmentation (P4) §2.1 §2.2.

Method

The paper has two parts: a list of validation pitfalls with recommendations (R1 to R4), and a benchmark that applies them §2 §3.

Recommendations §2

1. R1: isolate the claimed innovation. Do not couple it with extra data, self-supervised pre-training, larger compute, or ensembling that the baseline lacks §2.1.
2. R2: configure every baseline well. Prefer auto-configuration frameworks, or require that new methods ship adaptation instructions §2.1.
3. R3: check that each dataset can separate methods, and use enough datasets §2.2.
4. R4: use five-fold cross-validation with rotating validation folds. Ideally keep a separate pool of development datasets and an independent pool for testing §2.2.

Benchmark protocol §3.1

1. Every method uses a standard configuration scheme, applied in this order: (a) the method's own self-configuration if it has one; (b) the configuration nearest to the dataset when several are offered; (c) the default configuration when no alternatives exist; (d) where needed, a lower learning rate until training converges §3.1.
2. All models are trained from scratch. The only exception is SwinUNETR inside Auto3DSeg, which by default loads pre-trained weights and was left as is §3.1.
3. All trainings run on one NVIDIA A100 with 40 GB VRAM, which excludes the largest STU-Net variant ("H") §3.1.
4. Most methods run inside the nnU-Net framework. SwinUNETR (V1 and V2) was integrated into nnU-Net because its configuration instructions were incomplete §3.1.
5. The framework comparison runs Auto3DSeg (version 1.3.0, part of MONAI) with three architectures: SegResNet, DiNTS and SwinUNETR §3.1.

Dataset suitability score §3.2 fig. 1 tab. 3

Computed per dataset from the same benchmark runs. The paper defines it in words, and it has no numbered equation.

# Dataset suitability: inter-method SD over intra-method SD  [§3.2]
# dice[method][fold] = DSC of one method on one of the five folds
# Output: suitability ratio (higher = better at separating methods)

def suitability(dice):
    # Intra-method SD: spread across folds, per method, then averaged  [§3.2]
    intra = mean([std(dice[m]) for m in methods])
    # Inter-method SD: spread of each method's fold-mean score  [§3.2]
    inter = std([mean(dice[m]) for m in methods])
    return inter / intra

The paper does not state the SD convention (population or sample).

Evaluation §3.2 tab. 4

• Five-fold cross-validation, with the splits generated by nnU-Net and used for every method §3.2.
• Primary metric: average Dice similarity coefficient (DSC). Secondary metric: Normalized Surface Dice (NSD) at 2 mm tolerance tab. 4.
• Both metrics are averaged over classes and then over folds. For BraTS2021 and KiTS2023, metrics are computed on the hierarchical evaluation regions, not the non-overlapping classes §3.2.

Datasets §3.2: BTCV, ACDC, LiTS, BraTS2021, KiTS2023 and AMOS2022 (post-challenge Task 2). They were chosen for popularity.

Methods compared §3.1 tab. 1

• CNN: nnU-Net original; nnU-Net ResEnc presets M, L, XL (residual-encoder U-Nets sized for three VRAM budgets); MedNeXt L with kernel sizes k3 and k5; STU-Net S, B, L.
• Transformer: SwinUNETR, SwinUNETRV2, nnFormer and CoTr.
• Mamba: U-Mamba Bot and U-Mamba Enc, plus the "No-Mamba Base" ablation that keeps the same setting without Mamba layers.
• Framework comparison: Auto3DSeg SegResNet, DiNTS and SwinUNETR.

Hyperparameters

The paper states few training hyperparameters. The values below come from the official nnU-Net code (commit in the frontmatter), unless noted. They are the framework defaults; the paper does not say which overrides each method used.

Results

Author-reported. Setup: five-fold cross-validation with nnU-Net splits on six 3D datasets, all trained from scratch on one A100 40 GB §3.1 §3.2. RT is GPU-hours on A100 40 GB PCIe, and VRAM is the peak usage the paper reports, in GB.

Table 1: mean DSC (%) per dataset tab. 1. Bold marks the best value in each dataset column, as in the paper.

Table 4: mean NSD (%) at 2 mm tab. 4 (the paper orders the columns LiTS, BTCV, ACDC, BraTS2021, KiTS2023, AMOS2022)

Dataset suitability tab. 3 fig. 1. Average over methods. "Inter/intra" is the suitability ratio above.

Auto3DSeg vs nnU-Net on KiTS fold 0 tab. 2. Manual tuning of the framework approaches but does not reach nnU-Net ResEnc M.

Spacing ablation for MedNeXt tab. 5. Mean DSC (%) with isotropic spacing for ResEnc L:

Transfer ablation for STU-Net L tab. 6. Mean DSC (%); the source caption says the BraTS fine-tune did not converge at the default learning rate of 0.001, so the table cell is 0.

Headline findings §4

• CNN U-Nets built in nnU-Net perform strongest across all six datasets. MedNeXt has the best DSC on every dataset except KiTS, but its gains shrink on the more suitable datasets §4 tab. 1.
• Transformer models (SwinUNETR, nnFormer, CoTr) do not match the CNNs or the original nnU-Net §4.
• The U-Mamba advantage disappears once the missing "No-Mamba Base" ablation is included; the earlier gain came from the residual encoder, not Mamba §4 tab. 1.
• Auto3DSeg methods trail the original nnU-Net. Against the same SwinUNETR, nnU-Net wins on 5 of 6 datasets §4.
• Gains from scaling grow on the harder datasets, AMOS and KiTS §4.

Paper vs code

Code: the official nnU-Net repository, https://github.com/MIC-DKFZ/nnUNet, at commit 1489d746e084459581ff20382c1241d8cc2eb258, Apache-2.0. The paper links this repository in its contributions list §1.

• Scope of code checked. Only the nnU-Net framework was cloned and read. The implementations of MedNeXt, STU-Net, U-Mamba, SwinUNETR, nnFormer, CoTr and Auto3DSeg were not checked, so no code citations are given for them.
• ResEnc presets. The paper describes M, L and XL as presets for three hardware tiers, and says all runs used one 40 GB A100 §3.1. The code sizes the presets for targets of 8, 24 and 40 GB and documents them as roughly 9–11 GB, 24 GB and 40 GB residual_encoder_unet_planners.py residual_encoder_unet_planners.py residual_encoder_unet_planners.py. The paper's own VRAM column (9.10, 22.70 and 36.60 GB) is measured usage tab. 1.
• Dataset coverage cap. The paper does not mention it. The ResEnc presets set the batch-size cap to the whole dataset (coverage 1) residual_encoder_unet_planners.py, while the default planner caps coverage at 5% default_experiment_planner.py. Batch sizes for the paper's ResEnc runs are therefore not determined by the paper's text.
• Learning rate rule. The paper says the learning rate was decreased "until convergence" for some methods §3.1. The defaults shown above fix the initial rate at 1e-2 nnUNetTrainer.py, and no per-method values are given in the paper. Whether the learning rate was changed for any method is not stated.
• Deprecated duplicate planner. A module at nnunetv2/experiment_planning/experiment_planners/resencUNet_planner.py now re-exports the maintained class and emits a deprecation warning. The code's own comment says that an earlier duplicate caused -pl ResEncUNetPlanner to pick a stale class resencUNet_planner.py. Use the residual_unets path.
• Trainer defaults not contradicted. The paper does not state epochs, optimiser or loss settings, so the code defaults cannot be checked against it. The code defaults are the values in the Hyperparameters table.

Pitfalls

• Training hyperparameters are mostly not stated. The paper gives no epochs, learning rates, batch sizes or patch sizes for most methods, except the tables in the appendix. The values above are nnU-Net code defaults and may not be what each method used. The paper says the learning rate was lowered for some methods but does not name them.
• Batch sizes depend on the planner. For ResEnc presets, batch size comes from the VRAM estimate and the coverage cap, not from a fixed value. With the cap set to the whole dataset, small datasets can get large batches. Plan on running the planner rather than copying a batch size.
• Dataset ranking depends on which methods are included. Without Auto3DSeg, the ACDC ratio falls from 357% to 102%, and BTCV falls from 94% to 65% tab. 3. The text names ACDC, AMOS and KiTS as most suitable §4; the ranking of ACDC relative to the others is sensitive to the method set. The paper does not state the SD convention, so ratios may shift slightly by convention.
• Differences between top methods are small relative to fold variability. Intra-method SD is 2.39% on BTCV and 2.07% on KiTS tab. 3. Several top-table gaps are below one point (for example, ResEnc L vs MedNeXt L k3 on BraTS: 91.13 vs 91.35). The paper reports no significance test, so the ranking among the top CNNs on the suitable datasets is not established.
• Spacing confounds architecture. Isotropic ResEnc L (84.01 on BTCV, 92.64 on ACDC, 89.60 on AMOS) comes close to MedNeXt L k3 (84.70, 92.65, 89.62), and the paper concludes that part of MedNeXt's advantage is target spacing §4 tab. 5. Spacing and patch size must be set per dataset to reproduce this.
• Hidden compute cost. MedNeXt L k5 runs for 233 GPU-hours against 18 GB VRAM, while ResEnc M runs for 12 hours tab. 1. The paper's cost comparison uses RT, and the Auto3DSeg manual-tuning run uses about 240 GPU-hours on 8 GPUs for one fold tab. 2 §4. The appendix caption gives 240 hours, the table 247.44 hours; use the table value.
• Appendix table numbers. This card cites appendix tables as tab. 2 to tab. 6, following source order in the LaTeX. The compiled PDF was not available, so verify these numbers against it before relying on them.
• Missing STU-Net value. The pre-trained STU-Net BraTS cell is 0 in the source table, meaning the fine-tune did not converge. Do not average it.
• Auto3DSeg version. The paper uses Auto3DSeg 1.3.0 §3.1. The wider Auto3DSeg code is not part of this card's source, so claims about its internals are not checked.
• Licence for external code. The MedNeXt, STU-Net and U-Mamba repositories are separate projects; their licences were not checked here.

Lineage

• Builds on: nnU-Net (cited in the paper as 2018 and 2021 works) for auto-configuration and the self-configuring U-Net baseline §1 §2.1. Not a wiki card.
• Compared against: U-Mamba §3.1, MedNeXt §3.1, STU-Net §3.1, SwinUNETR and SwinUNETR V2 §3.1, nnFormer §3.1, CoTr §3.1, Auto3DSeg §3.1. None of these have cards in this wiki yet.
• Superseded by: not stated.