Transposed convolution

Overview

A transposed convolution is a learned layer that increases the spatial size of a feature map. Each input value is spread over a kernel-sized patch of the output, and the stride sets how far apart those patches are placed. The point of the layer is that the network learns the upsampling weights rather than using a fixed rule. The invariant that matters for generators is uniform information density: each output position should be reached by the same kinds of kernel taps, so no region of the generated volume is favoured 2102.07708 §4. The sources found so far use the layer only in a 3D GAN generator, so the rules below come from one paper.

Variants

There is one card with a variant in this concept. More cards are needed before the page can compare alternatives.

Stride-2 transposed convolution with uniform-density settings

SliceGAN builds its 3D generator from transposed convolutions. It states three rules on kernel k, stride s and padding p. Density is uniform only when s < k, k mod s = 0 and p ≥ k − s 2102.07708 §4. Of the practical settings that satisfy these rules, the paper uses (k, s, p) = (4, 2, 2) for most layers, and gives (6, 3, 3) and (6, 2, 4) as the other two 2102.07708 §4. The paper's generator table lists (4, 2, 2) for its first four layers and (4, 2, 3) for the last layer 2102.07708 tab. 1. Card: SliceGAN.

The paper also explains why the latent input is not a single voxel. The first layer takes a latent tensor of spatial size 4 instead of 1, so that kernel overlap is learned during training 2102.07708 §4. It also states that kernels larger than 6 are impractical because the parameter count grows with k³ 2102.07708 §4.

Resize-convolution (named alternative, not used)

The paper names resize-convolution as the alternative to transposed convolution and says it needs more memory for 3D volumes 2102.07708 §4. The card gives no details of how it would be built. An implementer who wants this variant needs a source that describes it.

When to use which

The choice rests on constraints. The rows below state what the cited paper supports, and nothing more.

Implementation sketch

The check below is the paper's three rules. The layer call is the common framework form. The output size depends on the padding convention, which the paper does not state; see Contradictions.

# Check the uniform-density rules before building the layer  [arxiv:2102.07708 §4]
def check_uniform_density(k, s, p):
    # stride below kernel size, so neighbouring outputs overlap  [§4]
    assert s < k
    # kernel is a multiple of stride, so every output sees the same taps  [§4]
    assert k % s == 0
    # padding covers the part of the kernel that does not overlap  [§4]
    assert p >= k - s

# One upsampling stage of a 3D generator  [arxiv:2102.07708 tab. 1]
def upsample_3d(in_channels, out_channels):
    # (4, 2, 2) is the setting used for most layers  [§4]
    check_uniform_density(k=4, s=2, p=2)
    return conv_transpose3d(in_channels, out_channels,
                            kernel=4, stride=2, padding=2)

Contradictions

• Paper versus released code. The paper describes transposed convolutions in every generator layer, with the table given 2102.07708 §4 2102.07708 tab. 1. The official repository's default run script (github.com/stke9/SliceGAN, at the commit recorded on the card) builds a generator that uses four transposed convolutions, then trilinear upsampling and a 3×3×3 convolution. Its fifth transposed-convolution layer is defined but not called. A reimplementation that copies the paper's table gets a different network from the one that was trained in the repository. This finding comes from the card's code review, not from the paper.
• Padding convention. The table gives padding 2 for layers 1 to 4. The paper does not say how padding is defined. Under the common output-size formula, (in − 1)·s − 2p + k, a 4-voxel input with (4, 2, 2) gives 6 voxels, not the 8 that a doubling layer needs. The paper's convention must be confirmed before the sizes are reused 2102.07708 tab. 1 2102.07708 §4.

Papers

• SliceGAN: uses transposed convolutions in a 3D GAN generator and gives the three uniform-density rules and a table of kernel, stride and padding settings.