Road Extraction by Deep Residual U-Net

In plain words

The paper is about finding roads in aerial photos. Each pixel gets a label: road or not. The authors start from U-Net, a popular network for this kind of labelling. U-Net squeezes the picture down to a compact summary and then expands it back to full size, copying fine detail from the early layers across to the late ones so the output keeps sharp edges.

The authors make two changes. The simple building blocks of U-Net are swapped for residual blocks, which pass their input around a small block of layers and add it back. That makes deep stacks easier to train. They also drop a cropping step that U-Net needs, so the input and output have the same size. The result is a smaller network that the authors report labels roads more accurately than U-Net on one public road dataset.

The gains are measured on one dataset and reported by the authors. Whether the same holds for other imagery is not tested in the paper.

TL;DR

• Deep ResUnet is a U-Net-style encoder-decoder whose blocks are full pre-activation residual units, with no cropping §2.1.3 fig. 2.
• It has 15 convolution layers (U-Net: 23) and about 7.8M parameters versus about 30.6M for U-Net, as stated in the text §3.4.
• Trained with MSE and SGD on 30,000 random 224x224 patches from the Massachusetts roads training set, with no augmentation §3.2.
• Author-reported relaxed break-even point on the test set: 0.9187, against 0.9053 for U-Net and 0.9047 for Saito et al. tab. II.
• Useful if you need a small road or binary segmentation baseline for aerial images. The config details are incomplete, so expect to fill gaps (see Pitfalls).

Problem

Road area extraction from high-resolution aerial images is hard because the imagery is noisy and cluttered, and roads are partly occluded, for example by trees §1. Very deep networks are hard to train, especially with few labelled samples §2.1.1. U-Net trains well on small biomedical data by using heavy augmentation and by joining low-level detail to high-level features §2.1.1. The authors argue that residual connections can ease training further, and that the U-Net skip connections can carry information more easily when they are placed alongside residual blocks §2.1.1 §2.1.3.

The paper scopes its work to road area (pixel-wise) extraction, not to road centerline extraction. It assumes labelled road masks for training §1.

Method

Deep ResUnet is a 7-level encoder-decoder: three encoding levels, one bridge level, and three decoding levels §2.1.3 fig. 2. Every level is built from residual units. The steps below follow the paper's text.

1. Residual unit. A unit computes a residual function F on its input x and adds the input back through an identity mapping h(x) = x, following the general form in eq. 1 §2.1.2. The paper uses the full pre-activation order: batch normalisation, ReLU, then a 3x3 convolution, twice, plus the identity path §2.1.2 fig. 1 §2.1.3.
2. Encoder. Levels 1 to 3 each hold one residual unit with two convolution blocks. Downsampling is done by a stride of 2 in the first convolution of a unit, not by pooling §2.1.3 tab. I. Channels double at each level: 64, 128, 256 tab. I.
3. Bridge. Level 4 is one residual unit with 512 filters and a stride of 2 in its first convolution tab. I §2.1.3.
4. Decoder. Levels 5 to 7 each start with an upsampling of the features from the level below, then a concatenation with the encoder features of the same level, then one residual unit §2.1.3. The paper calls these paths "encoding" and "decoding" instead of U-Net's "contracting" and "expansive" §2.1.3.
5. Cropping removed. Because input and output have the same size (224x224 at input and output), U-Net's cropping before concatenation is not needed §2.1.3 §2.3.
6. Output. A 1x1 convolution to one channel, followed by a sigmoid, gives a per-pixel road probability §2.1.3 tab. I.

The full layer table, as given in the paper, is reproduced here tab. I:

The paper also states the loss, eq. 2 §2.2:

\mathcal{L}(W) = \frac{1}{N}\sum_{i=1}^{N} \lVert \mathrm{Net}(I_i; W) - s_i \rVert^2 eq. 2

where N is the number of training samples, I_i an input patch, s_i its ground truth mask, and W the network parameters §2.2.

Residual unit of eq. 1 §2.1.2:

\mathbf{y}_l = h(\mathbf{x}_l) + \mathcal{F}(\mathbf{x}_l, \mathcal{W}_l), \qquad \mathbf{x}_{l+1} = f(\mathbf{y}_l) eq. 1

with f an activation function and h(x) = x in the paper's typical case §2.1.2.

The forward pass, decoder wiring and residual unit are written as pseudocode below. Shapes follow tab. I; the channel counts that the decoder receives after concatenation are not stated.

# Deep ResUnet forward pass  [§2.1.3] [tab. I] [fig. 2]
# Input:  image (224, 224, 3)
# Output: road probability (224, 224, 1)

def deep_resunet(image):
    # Encoder: stride 2 sits on the first conv of levels 2 and 3  [tab. I]
    skip_1 = residual_unit(image, filters=64, stride=1)     # (224, 224, 64)
    skip_2 = residual_unit(skip_1, filters=128, stride=2)   # (112, 112, 128)
    skip_3 = residual_unit(skip_2, filters=256, stride=2)   # (56, 56, 256)

    # Bridge: level 4, one residual unit, stride 2  [tab. I]
    bridge = residual_unit(skip_3, filters=512, stride=2)   # (28, 28, 512)

    # Decoder level 5: upsample, concatenate level 3, residual unit  [§2.1.3]
    up_5 = upsample(bridge)                                 # (56, 56, 512)
    dec_5 = residual_unit(concat(up_5, skip_3), filters=256, stride=1)

    # Decoder level 6: same pattern with level 2 features  [§2.1.3]
    up_6 = upsample(dec_5)                                  # (112, 112, 256)
    dec_6 = residual_unit(concat(up_6, skip_2), filters=128, stride=1)

    # Decoder level 7: same pattern with level 1 features  [§2.1.3]
    up_7 = upsample(dec_6)                                  # (224, 224, 128)
    dec_7 = residual_unit(concat(up_7, skip_1), filters=64, stride=1)

    # 1x1 conv to one channel, then sigmoid  [§2.1.3] [tab. I]
    logits = conv1x1(dec_7, filters=1)
    return sigmoid(logits)

# Full pre-activation residual unit  [§2.1.2] [fig. 1b] [eq. 1]
# Two blocks of BN -> ReLU -> 3x3 conv, plus an identity shortcut  [§2.1.3]

def residual_unit(inputs, filters, stride):
    # First block: the stride is applied in this convolution  [tab. I]
    hidden = relu(batch_norm(inputs))
    hidden = conv3x3(hidden, filters, stride=stride)
    # Second block: stride 1  [tab. I]
    hidden = relu(batch_norm(hidden))
    residual = conv3x3(hidden, filters, stride=1)
    # Identity shortcut as written; how shapes are matched when the
    # stride or filter count changes is not stated  [§2.1.2]
    shortcut = inputs
    return residual + shortcut

Training loop, from the implementation details §3.2 §2.2:

# Training: SGD on fixed-size patches, MSE loss  [§3.2] [§2.2]
# Input: 30,000 random 224x224 patches with ground-truth masks  [§3.2]

def train(network, patches, masks):
    # Batch size 8; initial learning rate 0.001  [§3.2]
    learning_rate = 0.001
    # The paper says convergence in 50 epochs; the stop rule is not stated
    for epoch in range(50):
        for batch in minibatches(patches, masks, size=8):
            predictions = network(batch.images)
            # Mean squared error, see eq. 2  [§2.2] [eq. 2]
            error = mean_squared_error(predictions, batch.masks)
            # Plain SGD update; momentum and weight decay are not stated
            network.weights -= learning_rate * gradient(error, network)
        # Multiply the learning rate by 0.1 every 20 epochs  [§3.2]
        if (epoch + 1) % 20 == 0:
            learning_rate *= 0.1
    return network

Inference on a large image, with overlapping tiles §2.3:

# Sliding-window inference over a large image  [§2.3]
# Tiles are 224x224 with an overlap of 14 pixels  [§2.3]

def segment_large_image(image):
    totals = zeros(height(image), width(image))
    counts = zeros(height(image), width(image))
    # Tile placement at image borders is not stated
    for window in tile(image, size=224, overlap=14):
        probabilities = deep_resunet(window.pixels)
        totals[window.region] += probabilities
        counts[window.region] += 1
    # Overlapping predictions are averaged  [§2.3]
    # Binarisation threshold is not stated
    return totals / counts

Hyperparameters

Results

Author-reported. Setup: Massachusetts roads dataset, test set of 49 images, relaxed precision and recall with slack rho = 3 pixels, reported as the break-even point §3.1 §3.3 §3.4.

The paper says ResUnet beats all three comparison methods on relaxed precision and recall §3.4. It gives the relaxed precision-recall curves for U-Net and ResUnet, with break-even markers fig. 3. It also shows four test-set tiles for Saito et al., U-Net and ResUnet fig. 4. In these examples ResUnet is described as producing less noise, separating two-lane roads, and recovering roads under trees. The one failure shown is a parking lot, where the authors say most roads are unlabelled §3.4 fig. 4.

Paper vs code

No official code.

Searched: the arXiv abstract page, the comments field and the TeX source (none links code or a project page); GitHub repository search for "resunet road", "deep residual unet road extraction" and "zhengxin zhang resunet", and GitHub user search for the second author's name with "beihang". The only repository found is DuFanXin/deep_residual_unet, a Keras reimplementation by someone who is not an author, with no license. It is not used here.

Pitfalls

• Shortcut shape. The identity mapping h(x) = x §2.1.2 cannot be added to a tensor of a different shape. The paper uses stride 2 and filter counts that change between levels tab. I, so it must do something else there (a projection, zero padding, or a strided shortcut). The paper does not say which.
• Activation after the addition. Eq. 1 writes x_{l+1} = f(y_l) with f an activation function §2.1.2, but the unit is described as full pre-activation §2.1.2. Whether an activation is applied after the addition is not stated.
• Decoder wiring. The upsampling method (nearest, bilinear, transposed convolution) is not stated. The channel count of each concatenation is not listed in tab. I. Both must be chosen to implement the decoder.
• Loss scaling. Eq. 2 writes the squared norm of each error, averaged over samples eq. 2. Whether the norm is summed or averaged over pixels, and therefore the effective learning rate, is not stated.
• Schedule and stopping. The learning rate drops by 0.1 every 20 epochs and the network "converges in 50 epochs" §3.2, but the exact drop points and the stopping rule are not stated. An "epoch" over 30,000 random patches at batch size 8 is also not defined.
• Tiling. The paper says "overlap o" with o = 14 §2.3, which the card reads as 14 pixels. The tile stride, the handling of the final tile at the image edge (1500 px does not split evenly into 224-px tiles at this overlap), and any padding at the image border are not stated.
• Thresholding. The sigmoid output is turned into a road mask without a stated threshold §2.1.3, and the break-even point is computed over a precision-recall sweep §3.3. Choose the threshold to match the evaluation.
• Validation set. Fourteen validation images are listed §3.1, but their use (early stopping, model selection, or none) is not stated.
• Random patches. The paper does not say whether random 224 x 224 patches are drawn uniformly, whether patches without road are included, or how many patches of each kind are used §3.2.
• Baselines. The Mnih-CNN, Saito-CNN and U-Net numbers tab. II could be re-run by the authors or copied from earlier work. The text does not say which.
• Variance. Each model has one break-even value in tab. II. The paper reports no seeds, repeated runs or spread. The gap between ResUnet and U-Net is 0.0134 points.
• Parameter count. The 7.8M figure §3.4 is given in the text only. The layer table does not list the decoder concatenation widths, so the count cannot be checked from tab. I.
• Framework and hardware. Keras and an NVIDIA Titan 1080 GPU are named §3.2, but no framework version, batch-norm momentum or epsilon, or inference time is given.
• Failure mode. The paper reports missed roads in parking lots, which it attributes to unlabelled roads in the ground truth §3.4 fig. 4. Expect similar errors wherever the labels omit roads.
• Dataset scope. Results are on one dataset (Massachusetts roads, 1.2 m per pixel) §3.1. The paper does not test other imagery, resolutions or road types.

Lineage

• Builds on: deep residual learning (He et al., CVPR 2016) for the residual unit §2.1.2; identity mappings in deep residual networks (He et al., ECCV 2016) for the full pre-activation order §2.1.2; U-Net (Ronneberger et al., MICCAI 2015) for the encoder-decoder with skip connections, without its cropping §2.1.1 §2.3.
• Compared against: Mnih and Hinton (ECCV 2010) and its CRF and post-processing variants; Saito et al. (2016); U-Net tab. II.
• Related skip-connection idea: fully convolutional networks (Long et al., CVPR 2015) §2.1.1.