Feature upsampling

Overview

Feature upsampling takes a low-resolution feature map from a backbone and produces a higher-resolution one, so that a dense-prediction head can label boundaries and small regions. The alternative is to keep the backbone at high resolution, which costs compute and memory 1903.11816 §1.

The wiki has one card for this concept. Its paper keeps a standard stride-32 backbone and rebuilds a high-resolution map from the last three stages with a learned module, Joint Pyramid Upsampling (JPU) 1903.11816 §3.2 1903.11816 §3.3. Because there is one source, every result below comes from one author team and one dataset, and nothing is compared across papers.

Variants

Plain bilinear upsampling

The simplest option: resize the coarse map with interpolation and add no parameters. The paper uses it as the lower bound in its ablation 1903.11816 tab. 1. With an Encoding head on ResNet-50 it gives 46.47 mIoU at 45.67 FPS, the fastest of the upsampling rows 1903.11816 tab. 1 1903.11816 tab. 2. Card: FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation.

Feature pyramid upsampling (FPN baseline)

The paper's FPN baseline, named as FPN in its ablation. It is compared at output stride 32 with the Encoding head 1903.11816 tab. 1. On ResNet-50 it reaches 49.59 mIoU at 37.87 FPS 1903.11816 tab. 1 1903.11816 tab. 2. This page does not describe its internals, because the card does not.

Joint Pyramid Upsampling (JPU)

JPU maps Conv3, Conv4 and Conv5 to a common width with a regular convolution block, upsamples Conv4 and Conv5 to Conv3's size, and concatenates the three 1903.11816 §3.3.3 fig. 4. Four separable convolutions with dilation rates 1, 2, 4 and 8 then read the fused map in parallel, and their outputs are concatenated 1903.11816 §3.3.3 fig. 5. The output is at the resolution of Conv3, which the paper places at output stride 8 1903.11816 §1.

With the Encoding head on ResNet-50, the ablation reports 51.05 mIoU at 37.56 FPS 1903.11816 tab. 1 1903.11816 tab. 2. The same module also improves the ASPP and PSP heads over their dilated versions in the paper's ablation, both in mIoU and in FPS 1903.11816 tab. 1 1903.11816 tab. 2. Card: FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation.

When to use which

The paper's evidence is one ablation on PASCAL-Context with ResNet-50 and single scale, plus speed numbers measured on a Titan-Xp at 512 x 512 input. The numbers below come only from that setup.

The speed gain over a dilated backbone depends on the head and the depth. For ResNet-101 with an Encoding head, the paper reports 32.02 FPS for JPU against 10.51 FPS for the dilated version 1903.11816 tab. 2. The paper gives no memory or FLOP table for these comparisons, so this page does not claim a memory figure.

Results on larger benchmarks are mixed. On ADE20K, the JPU model with ResNet-50 reaches 42.75 mIoU and the one with ResNet-101 reaches 44.34, while the EncNet baseline with ResNet-101 reaches 44.65 1903.11816 tab. 4. Where the ADE20K gap matters, check the card's table before choosing JPU.

Implementation sketch

The common core is: bring each level to one width, bring the coarse levels to the finest level's size, concatenate, and fuse with parallel dilated separable convolutions. Details the paper leaves open are marked as such; see the card's Hyperparameters table for the values the released code uses.

# Joint Pyramid Upsampling, reduced to its common core
# [arxiv:1903.11816 §3.3.3] [fig. 4]
# Inputs: stage3 (finest), stage4, stage5; width d is not stated in the paper

def jpu(stage3, stage4, stage5):
    # Map each stage to the same width with a regular conv block.
    # Kernel size and normalisation are not stated in the paper.
    f3 = conv_block(stage3)
    f4 = conv_block(stage4)
    f5 = conv_block(stage5)
    # Bring the coarser stages to the finest stage's size, then concatenate.
    # The interpolation mode is not stated in the paper.
    f4 = resize_to(f4, size_of(f3))
    f5 = resize_to(f5, size_of(f3))
    fused = concat([f5, f4, f3])
    # Parallel separable convolutions with dilation rates 1, 2, 4 and 8.
    # Padding equal to the dilation keeps the spatial size [fig. 5].
    branches = [separable_conv(fused, dilation=r) for r in (1, 2, 4, 8)]
    return concat(branches)

Where the paper and the released code differ, for example in the training losses and the learning rate of the new modules, the card's "Paper vs code" section lists each difference. Implement the paper's described loss unless you are reproducing the released numbers.

Contradictions

• JPU on PASCAL-Context, ResNet-101. The abstract gives 53.13 mIoU 1903.11816 abstract. The SOTA table gives 53.1 with 60 classes including background, multi-scale and flip 1903.11816 tab. 3. The text gives 54.03 on the 59-class protocol without background 1903.11816 §4.3. The paper does not reconcile these; they are different protocols, and the three values should not be compared directly.
• Output stride of JPU rows. The text says the JPU output is at output stride 8 1903.11816 §1. The ablation table labels the JPU rows as output stride 32 1903.11816 tab. 1. The table's column appears to describe the backbone, not the module's output, but the paper does not say so.

Papers

• FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation: introduces Joint Pyramid Upsampling to replace the dilated stage of a segmentation backbone, with ablations against bilinear and FPN upsampling.