Dilated convolution

Overview

A dilated (atrous) convolution spaces its kernel taps apart by a dilation rate instead of reading neighbouring pixels. Its use in this wiki is to keep a feature map at high resolution. The FastFCN paper states the problem this way: a classification backbone downsamples the image 32 times, so the last feature map is too coarse for boundaries. Dilated convolutions avoid that by removing the last two downsampling steps 1903.11816 §1.

The cost is the trade-off. With the enlarged maps, the dilated stage of a ResNet-101 needs about four times the computation and memory of the original stage, and three further blocks need about sixteen times 1903.11816 §1 1903.11816 §3.2.

The paper also gives a 1D analysis in which a dilated convolution equals a split, a regular convolution and a merge, while a strided convolution equals a regular convolution followed by dropping every other element 1903.11816 §3.3.2 fig. 3. The paper uses it as a justification, not as an implementation step.

Only one card covers this concept, so this page describes one paper. The variants below are both in that paper.

Variants

Dilated backbone at output stride 8

The standard way to get dense predictions: the backbone keeps its dilated stage and the head works on that high-resolution map. The paper's baseline is the Encoding head on a dilated ResNet, which it calls EncNet 1903.11816 tab. 1. On PASCAL-Context with ResNet-50, single scale, the paper reports 49.91 mIoU at 18.77 FPS for that baseline 1903.11816 tab. 1 1903.11816 tab. 2. Card: FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation.

Dilated separable convolutions as a fusion module

FastFCN keeps a standard stride-32 backbone and does not dilate it. A joint upsampling module (JPU) reads the last three stages, upsamples them to the size of the highest-resolution one, and concatenates them. Four separable convolutions with dilation rates 1, 2, 4 and 8 then read this map in parallel, and their four outputs are concatenated 1903.11816 §3.3.3 fig. 4. Here the dilation is a set of parallel branches inside a module, not a change to the backbone.

With the same Encoding head and output stride 32, the paper reports 51.05 mIoU and 37.56 FPS for this variant 1903.11816 tab. 1 1903.11816 tab. 2. The paper's ablation shows that plain bilinear upsampling at the same stride gives 46.47 mIoU at 45.67 FPS 1903.11816 tab. 1 1903.11816 tab. 2.

When to use which

The numbers come from one paper, single run each, on one dataset. They do not show the trade-off on other datasets. The gain of JPU over the dilated baseline also depends on the head: +1.14 mIoU with the Encoding head (51.05 against 49.91), +0.88 with ASPP (50.07 against 49.19) and +0.31 with PSP (50.89 against 50.58) 1903.11816 tab. 1.

Implementation sketch

Dilated separable branches of the fusion module, as described in the paper 1903.11816 §3.3.3. Shapes are (batch, channels, height, width).

# Four parallel dilated separable convs over one fused map
# [arxiv:1903.11816 §3.3.3]
# Input:  fused (B, C, H, W)
# Output: (B, 4d, H', W'), width d per branch

def dilated_fusion(fused, d):
    # Each branch reads the same map with its own dilation rate.
    branches = []
    for rate in [1, 2, 4, 8]:
        # Separable conv with dilation `rate`, d channels out
        branches.append(separable_conv(fused, out=d, dilation=rate))
    # Concatenate the branch outputs along channels [arxiv:1903.11816 §3.3.3]
    return concat(branches)

The paper does not state the width d, and the padding of each branch is not stated in the paper; choose both from your constraints. The paper's JPU also has a first conv block before this step, whose kernel is not stated.

Contradictions

None known. Only one card covers this concept, so no two sources disagree. Within the paper, the reported numbers differ by protocol: the ablation is 59 classes without background, and the SOTA table is 60 classes with background 1903.11816 tab. 3 1903.11816 §4.3.

Papers

• FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation: replaces the dilated backbone with a stride-32 backbone and a joint upsampling module that uses parallel dilated separable convolutions; author-reported PASCAL-Context gains.