A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation

In plain words

Colonoscopy is the main way doctors look for colon cancer, and small or flat polyps (growths on the bowel wall) are easy to miss. A computer can draw an outline around each polyp in a video frame, which is called segmentation. This paper takes an earlier outline-drawing network, ResUNet++, and asks whether two cheap add-ons make its outlines better. The first add-on tidies the outline by smoothing the mask so that stray specks and ragged edges are reduced. The second add-on looks at each frame several times, once as-is and once mirrored left-right and once mirrored top-bottom, and averages the answers.

The authors test these add-ons on six public polyp datasets, including video, and also train on one dataset and test on another. The add-ons help on some datasets and not on others, so the paper's main message is that the result depends on the data. It also releases a subset of 196 small or flat polyps from Kvasir-SEG, which the authors selected with the help of a gastroenterologist.

TL;DR

• ResUNet++ is an encoder-decoder segmentation network with residual, squeeze-and-excitation, atrous-pyramid and attention blocks §III.
• Two post-processing steps are added: a dense CRF on the thresholded mask §III-E and flip-based test-time augmentation (TTA) §III-F.
• Author-reported gains are largest on Kvasir-SEG (DSC 0.8119 to 0.8508 with TTA and CRF) tab. II. On several video and ETIS-Larib results, the extra steps are equal or worse tab. V, tab. VII.
• Implement the network from the code, not the paper alone: the paper's learning rate and the code's schedule disagree, and several details are only in code (see Paper vs code).

Problem

Polyp miss-rates in colonoscopy are high. The paper cites an adenoma miss-rate of about 6 to 27 percent, and says polyps under 10 mm, sessile and flat polyps are missed most often §I. A computer-aided system is expected to help, but it must be robust across easy and hard images and generalise to data from other hospitals §I. Earlier polyp work usually tested on one small dataset, which the authors say makes results hard to compare §II. The paper therefore asks whether ResUNet++ with CRF and TTA gives consistent gains within a dataset, across mixed datasets and across datasets §I, §V.

Method

Section numbers follow the IEEE numbering as printed (§III-E is Conditional Random Field). Tables are numbered I to XII in source order. Equation 1 is the only numbered equation.

Network (ResUNet++). The network is a U-Net-style encoder-decoder with residual units, squeeze-and-excitation (SE) blocks, an atrous spatial pyramid pooling (ASPP) bridge and attention gates on the decoder §III. Each residual unit computes an output as a residual function of its input plus the input itself eq. 1. The filter counts are 32, 64, 128, 256 and 512 across the levels §III.

# ResUNet++ forward pass  [§III] [code: m_resunet.py#L114-L148]
# Input:  image (256, 256, 3), intensity-normalised [code: tf_data.py#L10-L11]
# Output: probability map (256, 256, 1)

def resunetplusplus(image):
    # Stem residual unit with SE, 32 filters  [§III-A, §III-B]
    skip1 = stem_block(image, filters=32, stride=1)
    # Encoder: residual units with SE, stride 2  [§III-A, §III-B]
    skip2 = residual_block(skip1, filters=64, stride=2)
    skip3 = residual_block(skip2, filters=128, stride=2)
    deep = residual_block(skip3, filters=256, stride=2)
    # Bridge: ASPP with 512 filters  [§III-C] [code: m_resunet.py#L127]
    bridge = aspp(deep, filters=512)
    # Decoder level 3: gate skip3 with the bridge, upsample, concatenate
    # [§III-D] [code: m_resunet.py#L130-L133]
    up = attention(skip3, bridge)
    up = upsample_2x(up)
    up = concatenate(up, skip3)
    up = residual_block(up, filters=256)
    # Decoder level 2, same pattern with skip2 and 128 filters
    # [code: m_resunet.py#L135-L138]
    up = attention(skip2, up)
    up = upsample_2x(up)
    up = concatenate(up, skip2)
    up = residual_block(up, filters=128)
    # Decoder level 1, same pattern with skip1 and 64 filters
    # [code: m_resunet.py#L140-L143]
    up = attention(skip1, up)
    up = upsample_2x(up)
    up = concatenate(up, skip1)
    up = residual_block(up, filters=64)
    # Output: ASPP with 32 filters, 1x1 conv, sigmoid  [§III]
    # [code: m_resunet.py#L146-L148]
    up = aspp(up, filters=32)
    return sigmoid(conv1x1(up, filters=1))

Residual unit and SE. The paper describes a pre-activation residual unit built from batch normalisation (BN), ReLU and convolution §III-A. The SE block squeezes each channel to a global average, passes it through a small gating network and rescales the channels §III-B. In the code, each residual unit uses a 1×1 convolution plus BN shortcut, not an identity shortcut, and the SE block is applied after the sum m_resunet.py.

# Residual unit with SE  [§III-A, §III-B] [code: m_resunet.py#L45-L64]
# Input:  features (H, W, c_in); output (H / stride, W / stride, filters)

def residual_block(features, filters, stride=1):
    # Two pre-activation units: BN, ReLU, 3x3 conv  [§III-A]
    residual = bn_relu_conv3x3(features, filters, stride)
    residual = bn_relu_conv3x3(residual, filters, stride=1)
    # Shortcut: 1x1 conv with stride and BN (projection, not identity)
    shortcut = batch_norm(conv1x1(features, filters, stride))
    # Add, then squeeze-and-excite on the sum  [§III-B]
    return squeeze_excite(residual + shortcut, ratio=8)

ASPP. The bridge and the output stage use ASPP. The code runs three dilated 3×3 convolutions with rates 6, 12 and 18, plus one ordinary 3×3 convolution, each followed by BN. It sums the four branches and applies a 1×1 convolution §III-C m_resunet.py. The paper does not state the rates.

Attention gate. The gate takes the encoder skip feature and the decoder feature. The skip branch is max-pooled to the decoder resolution. The two branches are summed, passed through another convolution, and the gate multiplies the decoder feature §III-D m_resunet.py.

# Attention gate  [§III-D] [code: m_resunet.py#L83-L108]
# Input:  skip (2H, 2W, c) from the encoder, decoder (H, W, c)
# Output: decoder features scaled by a learned gate (H, W, c)

def attention(skip, decoder):
    # Skip branch: BN, ReLU, 3x3 conv, then 2x2 max-pool to H x W
    skip_feat = max_pool_2x2(conv3x3(relu(batch_norm(skip))))
    # Decoder branch: BN, ReLU, 3x3 conv  [code: m_resunet.py#L97-L99]
    dec_feat = conv3x3(relu(batch_norm(decoder)))
    # Sum, then BN, ReLU, 3x3 conv gives the gate  [code: m_resunet.py#L103-L105]
    gate = conv3x3(relu(batch_norm(skip_feat + dec_feat)))
    # Multiply the decoder input by the gate  [code: m_resunet.py#L107]
    return gate * decoder

Dense CRF (post-processing). The CRF is applied to the output mask after thresholding, as a further step on the test set §III-E. The paper names a dense CRF and gives no parameters. The code uses a two-label CRF whose unary term trusts the thresholded mask with probability 0.7, and a Gaussian pairwise term over pixel location only, followed by 10 inference steps crf.py.

# Dense CRF refinement of a binary mask  [§III-E] [code: crf.py#L8-L42]
# Input:  mask (H, W) with values in {0, 1}
# Output: refined mask (H, W) with values in {0, 1}

def refine_with_crf(mask):
    # Unary term: each pixel's thresholded label has probability 0.7
    # [code: crf.py#L30]
    unary = unary_from_labels(mask, n_labels=2, gt_prob=0.7)
    # Pairwise term: Gaussian over pixel positions only, sxy=3, compat=3
    # [code: crf.py#L34]. No colour or image term is added.
    pairwise = gaussian_location_kernel(sxy=3, compat=3)
    # Mean-field inference for 10 steps  [code: crf.py#L37]
    beliefs = dense_crf_inference(unary, pairwise, steps=10)
    # Most probable label per pixel  [code: crf.py#L40]
    return argmax(beliefs, axis=label)

Test-time augmentation. TTA runs the network on the image, on its horizontal flip and on its vertical flip. It flips each output back and averages the three probability maps §III-F tta.py.

# Flip-based test-time augmentation  [§III-F] [code: tta.py#L12-L26]
# Input:  image (H, W, 3); model returns probability map (H, W, 1)
# Output: averaged probability map (H, W, 1)

def predict_tta(model, image):
    # Original view, no flip  [code: tta.py#L17]
    original = model(image)
    # Horizontal flip: predict, then flip the map back  [code: tta.py#L18, L22]
    horizontal = flip_horizontal(model(flip_horizontal(image)))
    # Vertical flip: predict, then flip the map back  [code: tta.py#L19, L23]
    vertical = flip_vertical(model(flip_vertical(image)))
    # Plain mean of the three maps  [code: tta.py#L25]
    return (original + horizontal + vertical) / 3.0

Full inference. TTA, if used, averages first. The result is then thresholded at 0.5. The CRF, if used, runs on the thresholded mask §V-B infer.py.

# Inference with optional TTA and CRF  [§V-B] [code: infer.py#L128-L175]
# Input:  image (256, 256, 3)
# Output: binary mask (256, 256)

def segment(model, image, use_tta, use_crf):
    # Average over flips if TTA is on  [§III-F]
    if use_tta:
        prob = predict_tta(model, image)
    else:
        prob = model(image)
    # Threshold the probability map at 0.5  [§V-B]
    mask = prob > 0.5
    # Refine the thresholded mask with the CRF if on  [§III-E]
    if use_crf:
        mask = refine_with_crf(mask)
    return mask

Training. The loss is binary cross-entropy. Training uses Nadam with a stochastic gradient descent with warm restarts (SGDR) schedule, batch size 16 and 300 epochs, with early stopping §IV-D. Training images are augmented offline before training §IV-C.

Hyperparameters

Values in the code are the released defaults at code_commit. Where the paper and code differ, both are listed in Paper vs code.

Datasets tab. I:

Results

Author-reported. Setup: each dataset is trained and tested on its own split unless marked; the cross-dataset tables train on one dataset and test on another; the mixed setting trains on Kvasir-SEG plus CVC-ClinicDB §V. The "Baseline" column is the previous ResUNet++ model, unless another method is named. Metric is the Dice coefficient (DSC) §IV-B.

Other reported numbers:

• Model size: ResUNet++ has 16,228,001 trainable parameters. U-Net has 5,400,289, ResUNet 8,221,121 and ResUNet-mod 2,058,465 tab. XII.
• ROC AUC: the paper reports AUC above 0.93 for all proposed models §VI-A. For CVC-ClinicDB with TTA it reports 0.9814 §V-B.
• The paper's own summary: CRF is best on same-dataset video and ASU-Mayo tests, TTA is best on still images and cross-dataset tests, and the combined model is best on video cross-dataset tests §V-K, §VI-A.

Paper vs code

Code: https://github.com/DebeshJha/ResUNet-with-CRF-and-TTA at commit 8dc333c2f2903e0d36d81844158b8efd7879a85f. The owner is the first author, Debesh Jha, and the paper's footnote links this repository §IV-D. The repository has no LICENSE file and the README names no license, so the code license is not stated. Code is cited by path and line only. No code is quoted.

1. Learning rate. The paper states 1e-5 §IV-D. The code's SGDR schedule starts the optimiser at max LR 1e-2 and cycles between 1e-6 and 1e-2 train.py sgdr.py. The paper's stated rate is not the learning rate the code uses.
2. Batch size. The paper states 16 §IV-D. tf_dataset defaults to batch 8, and train.py calls it without a batch argument tf_data.py train.py. Training uses batch 8 as written.
3. SGDR steps. The schedule is given steps_per_epoch = ceil(epochs / batch_size) = 19 train.py. This is a fixed count and does not depend on the dataset size, so each cycle is measured in those units, not in actual batches.
4. Weights kept. The paper says early stopping was used §IV-D. In the code, ModelCheckpoint has no save_best_only, so the file holds the last epoch's weights train.py. EarlyStopping has restore_best_weights=False train.py. The SGDR callback restores the weights saved at the most recent restart when training ends sgdr.py. Final weights are therefore not chosen by validation loss.
5. Identity shortcut. Equation 1 is written as a residual function plus its input eq. 1. The code adds a 1×1 convolution with BN on the shortcut in every residual unit, including where shapes match m_resunet.py.
6. SE placement. The paper says the SE blocks are in the encoder §III. The code calls the same residual unit, with its SE block, in the decoder too m_resunet.py m_resunet.py m_resunet.py.
7. TTA views. The paper names horizontal and vertical flips §III-F. The code averages the original view with both flips and divides by 3 tta.py.
8. CRF. The paper describes a dense CRF that models shape, connectivity and context §III-E. The code imports a colour (bilateral) pairwise term but does not add it crf.py. The only pairwise term is a Gaussian over position crf.py. The image is passed to the CRF but not used, because only the mask's shape is read crf.py.
9. Augmentation. The paper lists its augmentations §IV-C. The code's augment_data also applies gamma, Gaussian and median blur, Gaussian noise, channel shuffle, and CLAHE is imported but not used infer_image_all.py. The code creates 26 copies per training image: the original and 25 variants infer_image_all.py.
10. Split. The paper states an 80:10:10 split §IV-C. main() splits 80:20 into train and validation infer_image_all.py. The repository shows no test split.
11. Preprocessing. The paper does not mention intensity normalisation. The code subtracts the image median and clips tf_data.py infer.py. The paper does not report how much this step changes the results.
12. Multi-GPU. The paper reports a single Volta GPU §IV-D. train.py has a multi-GPU branch with a call that does not match the TensorFlow API as written, so the except branch is the one that runs train.py. This was read, not run.
13. Dice loss. The code defines a Dice-based loss but trains only with BCE train.py. This agrees with the paper, which reports BCE as the chosen loss.
14. Evaluation resolution. The paper resizes test data to 256 × 256 §IV-C. The code's inference script reads images without resizing infer.py, and resizing appears only in augment_data for the train and validation splits infer_image_all.py. Whether the cross-dataset folder cs_data/CVC-12k was resized is not visible in the repository.
15. Not checked. The repository also has resunet++_pytorch.py. It was not reviewed, so this card describes the Keras/TensorFlow code only.

The 0.5 threshold matches the paper §V-B infer.py.

Pitfalls

• Text and table disagree. The CVC-ClinicDB text says CRF reaches DSC 0.9293 §V-B, but the table shows 0.9203 tab. III. The "2.23%" gain over PraNet matches 0.9203, so use the table value.
• Gains are small, unreplicated and not measured. Many differences in the tables are below 0.005 DSC (for example CVC-ClinicDB 0.9199 vs 0.9203). The paper reports one run per configuration and no error bars in the text. The code fixes the seed at 42, but the paper's numbers are not averaged over seeds.
• TTA is not uniformly helpful. It lowers in-domain DSC on CVC-ClinicDB (0.9020 vs 0.9199) and CVC-VideoClinicDB (0.8125 vs 0.8798) tab. III, tab. VII. It helps on Kvasir-SEG, Kvasir-Sessile and the cross-dataset tables tab. II, tab. VI, tab. X, tab. XI.
• ETIS-Larib favours plain ResUNet++. Its DSC is highest without TTA or CRF tab. V. The authors attribute this to data specifics and note that only 156 of 196 images are used for training §V-D.
• Ground truth is not pixel-precise on video. CVC-VideoClinicDB masks are oval or circular approximations §V-F. The paper attributes low cross-dataset scores partly to this §VI-B.
• ASU-Mayo test data is unavailable. The paper uses the 20 training videos and tests cross-dataset on 10 positive videos §IV-A. The dataset is marked copyrighted tab. I, so reproducing the ASU-Mayo numbers needs access that this card does not provide.
• CRF cost. The CRF runs per image on the CPU after the network, for 10 inference steps crf.py. It needs the pydensecrf package crf.py. TTA triples the forward passes at inference tta.py.
• Input size. Training and validation images are reduced to 256 × 256 §IV-C. The authors name this resize as a cost that may lower performance §VI-D.
• Unstated parameters. The paper gives no SGDR parameters, no ASPP rates, no SE ratio and no CRF parameters. Use the code values in Hyperparameters, and say so when you report them.
• Parameter counts. The paper's claim that ResUNet++ needs fewer parameters than pretrained-encoder models is based on its own comparison §VI-A, tab. XII.
• Other splits. The code's dataset paths point to a local Kvasir-SEG folder and to cs_data/CVC-12k, which are not in the repository. Expect to rebuild the folders.

Lineage

• Builds on: ResUNet++, the authors' earlier model with the same architecture, which this paper extends with CRF and TTA §I, §III. The repository README links its arXiv id as 1911.07067 README.md. Not in this wiki yet.
• Builds on: U-Net §II, tab. II and ResUNet §II, as baselines, and the squeeze-and-excitation, ASPP and attention components §III.
• Compared against: UNet, ResUNet, ResUNet-mod tab. II, PraNet, MultiResUNet, FCN-based methods, SegNet and others in tab. III to tab. XI.
• Superseded by: not known from content in this wiki.