Spatial pooling

Overview

Spatial pooling reduces the height and width of an activation map by replacing each window of each channel with one value 2101.00440 §1 2101.00440 §3. The stated aims are lower computation and memory in the following layers, a larger receptive field, and as little lost information as possible 2101.00440 §1 2101.00440 §3. The usual baselines are max pooling, which keeps only the largest value in a window, and average pooling, which weights every value equally 2101.00440 §3.2 2101.00440 §3.3.

Variants

Softmax-weighted pooling (SoftPool)

Each output is a softmax-weighted sum over a k × k window. Every activation is weighted by its exponential, normalised over the window, and the output is the weighted sum of the activations 2101.00440 §3.1. Large values dominate, but smaller values still contribute, so the output depends smoothly on every activation in the window 2101.00440 §3.1 2101.00440 §3.3. The operator has no trainable parameters, so parameter counts and GFLOPs match the pooling it replaces 2101.00440 §4.4. The same formulation extends to 3D inputs with a spatio-temporal kernel 2101.00440 §3.4. Card: Refining activation downsampling with SoftPool.

Author-reported results:

• Image similarity: highest SSIM and PSNR in all nine settings (k = 2, 3, 5 on three datasets) against the pooling methods the paper compares 2101.00440 tab. 1.
• ImageNet1K, trained from scratch: top-1 gains over the original network between +0.81 and +1.51 as printed, across 11 networks 2101.00440 tab. 2. Some printed gains disagree with the table's own numbers (see Contradictions).
• ImageNet1K, pooling swapped into a trained network without retraining: printed gains between +0.37 and +0.80, with one row whose printed gain conflicts with its own numbers 2101.00440 tab. 2.
• Same ResNet18 setting, six CNNs: the highest from-scratch top-1 in every column against the other pooling operators 2101.00440 tab. 5.
• Action recognition: SoftPool versions of three video models are reported with higher top-1 on HACS, Kinetics-700 and UCF-101 in most rows. The baselines use different pre-training, so these rows are not controlled comparisons 2101.00440 §4.6 2101.00440 tab. 7.

When to use which

Implementation sketch

Forward pass for one window of one channel. The paper omits the channel index; pooling is applied per channel.

# Forward pass over one window R of one channel  [arxiv:2101.00440 §3.1]
# Input:  window, k*k activations (non-negative, see When to use which)
# Output: pooled scalar

def softpool_window(window):
    # Exponentiate each activation  [arxiv:2101.00440 eq. 1]
    exp_values = exp(window)
    # Normalise so the weights in the window sum to one  [arxiv:2101.00440 eq. 1]
    weights = exp_values / sum(exp_values)
    # Output is the weighted sum of the activations  [arxiv:2101.00440 eq. 2]
    return sum(weights * window)

Subtracting the window maximum before the exponential leaves the weights unchanged, because softmax is invariant to a shift. This matters for half-precision overflow. The card records that the official CUDA kernel does not subtract it.

Contradictions

• Backward pass. The paper gives each input a gradient proportional to its forward weight 2101.00440 §3.2. The exact derivative of 2101.00440 eq. 2 also depends on how the weights change with each input. The card derives this term and notes that the official code uses the weights-only form (card). The paper does not say which form the reported results used 2101.00440 §3.2.
• Operator on CPU and GPU. The paper computes the weights within each channel's window 2101.00440 §3.1. The card records that the official non-CUDA path normalises over the channel dimension, so outputs differ between devices (card).
• 3D windows. The paper states the 3D extension only as a kernel extension 2101.00440 §3.4. The card records that the official 3D forward and backward passes place their windows differently (card).
• Printed gains. Some printed gains in 2101.00440 tab. 2 do not equal the difference of the printed top-1 values. ResNeXt101 32x8d pre-trained reads +0.58, but 78.89 − 79.31 = −0.42. DenseNet161 from scratch reads +0.93, but 78.72 − 77.65 = 1.07.
• Baseline labels. 2101.00440 tab. 5 attaches the Lp and Gate citations to the opposite labels from 2101.00440 tab. 1 and 2101.00440 tab. 3.

Papers

• Refining activation downsampling with SoftPool: replaces max or average pooling with a softmax-weighted sum that has no trainable parameters, tested on image downsampling, ImageNet1K and video.