---
kind: concept
slug: feature-upsampling
title: Feature upsampling
summary: >-
  Raising a coarse feature map back to a finer resolution with a learned module instead of plain interpolation, so a dense-prediction head gets detail without a dilated backbone. One card (FastFCN's Joint Pyramid Upsampling) compares it with bilinear and FPN upsampling on PASCAL-Context. All numbers are author-reported, from one paper and one dataset.
tags: [upsampling, semantic-segmentation, backbone-efficiency]
papers: ["1903.11816"]
related: [dilated-convolution, semantic-segmentation, guided-depth-upsampling, image-super-resolution]
status: draft
updated_at: 2026-11-01
---

## Overview

Feature upsampling takes a low-resolution feature map from a backbone and
produces a higher-resolution one, so that a dense-prediction head can label
boundaries and small regions. The alternative is to keep the backbone at high
resolution, which costs compute and memory [arxiv:1903.11816 §1].

The wiki has one card for this concept. Its paper keeps a standard stride-32
backbone and rebuilds a high-resolution map from the last three stages with a
learned module, Joint Pyramid Upsampling (JPU) [arxiv:1903.11816 §3.2]
[arxiv:1903.11816 §3.3]. Because there is one source, every result below comes
from one author team and one dataset, and nothing is compared across papers.

## Variants

### Plain bilinear upsampling

The simplest option: resize the coarse map with interpolation and add no
parameters. The paper uses it as the lower bound in its ablation
[arxiv:1903.11816 tab. 1]. With an Encoding head on ResNet-50 it gives 46.47
mIoU at 45.67 FPS, the fastest of the upsampling rows [arxiv:1903.11816 tab. 1]
[arxiv:1903.11816 tab. 2]. Card: [FastFCN: Rethinking Dilated Convolution in the
Backbone for Semantic Segmentation](/papers/1903.11816).

### Feature pyramid upsampling (FPN baseline)

The paper's FPN baseline, named as FPN in its ablation. It is compared at output
stride 32 with the Encoding head [arxiv:1903.11816 tab. 1]. On ResNet-50 it reaches
49.59 mIoU at 37.87 FPS [arxiv:1903.11816 tab. 1] [arxiv:1903.11816 tab. 2].
This page does not describe its internals, because the card does not.

### Joint Pyramid Upsampling (JPU)

JPU maps Conv3, Conv4 and Conv5 to a common width with a regular convolution
block, upsamples Conv4 and Conv5 to Conv3's size, and concatenates the three
[arxiv:1903.11816 §3.3.3] [fig. 4]. Four separable convolutions with dilation
rates 1, 2, 4 and 8 then read the fused map in parallel, and their outputs are
concatenated [arxiv:1903.11816 §3.3.3] [fig. 5]. The output is at the
resolution of Conv3, which the paper places at output stride 8 [arxiv:1903.11816 §1].

With the Encoding head on ResNet-50, the ablation reports 51.05 mIoU at 37.56
FPS [arxiv:1903.11816 tab. 1] [arxiv:1903.11816 tab. 2]. The same module also
improves the ASPP and PSP heads over their dilated versions in the paper's
ablation, both in mIoU and in FPS [arxiv:1903.11816 tab. 1]
[arxiv:1903.11816 tab. 2]. Card: [FastFCN: Rethinking Dilated Convolution in the
Backbone for Semantic Segmentation](/papers/1903.11816).

## When to use which

The paper's evidence is one ablation on PASCAL-Context with ResNet-50 and single
scale, plus speed numbers measured on a Titan-Xp at 512 x 512 input. The
numbers below come only from that setup.

| Situation | Use | Why |
|---|---|---|
| Latency or memory budget is tight, and some loss in boundary quality is acceptable | Plain bilinear | Fastest upsampling row, 45.67 FPS, but the lowest mIoU, 46.47 [arxiv:1903.11816 tab. 1] [arxiv:1903.11816 tab. 2] |
| Need dense features and can afford the dilated stage | Dilated backbone (no upsampling module) | 49.91 mIoU at 18.77 FPS, the reference for the paper's comparisons [arxiv:1903.11816 tab. 1] [arxiv:1903.11816 tab. 2] |
| Want a learned upsampler with the least structural change | FPN-style fusion | 49.59 mIoU at 37.87 FPS, close in speed to JPU [arxiv:1903.11816 tab. 1] [arxiv:1903.11816 tab. 2] |
| Want the highest mIoU from a stride-32 backbone at close to FPN's speed | JPU | 51.05 mIoU at 37.56 FPS, against 49.59 for FPN in the same setting; the 1.46 mIoU gap is the only JPU-versus-FPN comparison in the card [arxiv:1903.11816 tab. 1] |
| Using an ASPP or PSP head rather than Encoding | JPU, with the head's own numbers in view | The paper reports JPU above the dilated version for both heads, at higher FPS [arxiv:1903.11816 tab. 1] [arxiv:1903.11816 tab. 2] |

The speed gain over a dilated backbone depends on the head and the depth. For
ResNet-101 with an Encoding head, the paper reports 32.02 FPS for JPU against
10.51 FPS for the dilated version [arxiv:1903.11816 tab. 2]. The paper gives no
memory or FLOP table for these comparisons, so this page does not claim a
memory figure.

Results on larger benchmarks are mixed. On ADE20K, the JPU model with ResNet-50
reaches 42.75 mIoU and the one with ResNet-101 reaches 44.34, while the EncNet
baseline with ResNet-101 reaches 44.65 [arxiv:1903.11816 tab. 4]. Where the
ADE20K gap matters, check the card's table before choosing JPU.

## Implementation sketch

The common core is: bring each level to one width, bring the coarse levels to
the finest level's size, concatenate, and fuse with parallel dilated separable
convolutions. Details the paper leaves open are marked as such; see the card's
Hyperparameters table for the values the released code uses.

```pseudocode
# Joint Pyramid Upsampling, reduced to its common core
# [arxiv:1903.11816 §3.3.3] [fig. 4]
# Inputs: stage3 (finest), stage4, stage5; width d is not stated in the paper

def jpu(stage3, stage4, stage5):
    # Map each stage to the same width with a regular conv block.
    # Kernel size and normalisation are not stated in the paper.
    f3 = conv_block(stage3)
    f4 = conv_block(stage4)
    f5 = conv_block(stage5)
    # Bring the coarser stages to the finest stage's size, then concatenate.
    # The interpolation mode is not stated in the paper.
    f4 = resize_to(f4, size_of(f3))
    f5 = resize_to(f5, size_of(f3))
    fused = concat([f5, f4, f3])
    # Parallel separable convolutions with dilation rates 1, 2, 4 and 8.
    # Padding equal to the dilation keeps the spatial size [fig. 5].
    branches = [separable_conv(fused, dilation=r) for r in (1, 2, 4, 8)]
    return concat(branches)
```

Where the paper and the released code differ, for example in the training
losses and the learning rate of the new modules, the card's "Paper vs code"
section lists each difference. Implement the paper's described loss unless you
are reproducing the released numbers.

## Contradictions

- **JPU on PASCAL-Context, ResNet-101.** The abstract gives 53.13 mIoU
  [arxiv:1903.11816 abstract]. The SOTA table gives 53.1 with 60 classes including
  background, multi-scale and flip [arxiv:1903.11816 tab. 3]. The text gives
  54.03 on the 59-class protocol without background [arxiv:1903.11816 §4.3].
  The paper does not reconcile these; they are different protocols, and the
  three values should not be compared directly.
- **Output stride of JPU rows.** The text says the JPU output is at output
  stride 8 [arxiv:1903.11816 §1]. The ablation table labels the JPU rows as
  output stride 32 [arxiv:1903.11816 tab. 1]. The table's column appears to
  describe the backbone, not the module's output, but the paper does not say so.

## Papers

- [FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation](/papers/1903.11816): introduces Joint Pyramid Upsampling to replace the dilated stage of a segmentation backbone, with ablations against bilinear and FPN upsampling.