---
kind: concept
slug: spatial-pooling
title: Spatial pooling
summary: >-
  Spatial pooling shrinks a feature map by reducing each small window of each channel to one value. The one card so far, SoftPool, replaces max or average pooling with a softmax-weighted sum that has no trainable parameters. Author-reported gains on image-similarity and ImageNet1K tests. Its official backward pass omits a term of the exact gradient.
tags: [pooling, cnn, downsampling]
papers: ["2101.00440"]
related: [convolutional-neural-network, feature-upsampling, transposed-convolution]
status: draft
updated_at: 2026-11-08
---

## Overview

Spatial pooling reduces the height and width of an activation map by replacing
each window of each channel with one value [arxiv:2101.00440 §1] [arxiv:2101.00440 §3].
The stated aims are lower computation and memory in the following layers, a
larger receptive field, and as little lost information as possible
[arxiv:2101.00440 §1] [arxiv:2101.00440 §3]. The usual baselines are max
pooling, which keeps only the largest value in a window, and average pooling,
which weights every value equally [arxiv:2101.00440 §3.2] [arxiv:2101.00440 §3.3].

## Variants

### Softmax-weighted pooling (SoftPool)

Each output is a softmax-weighted sum over a k × k window. Every activation is
weighted by its exponential, normalised over the window, and the output is the
weighted sum of the activations [arxiv:2101.00440 §3.1]. Large values dominate,
but smaller values still contribute, so the output depends smoothly on every
activation in the window [arxiv:2101.00440 §3.1] [arxiv:2101.00440 §3.3]. The
operator has no trainable parameters, so parameter counts and GFLOPs match the
pooling it replaces [arxiv:2101.00440 §4.4]. The same formulation extends to 3D
inputs with a spatio-temporal kernel [arxiv:2101.00440 §3.4].
Card: [Refining activation downsampling with SoftPool](/papers/2101.00440).

Author-reported results:

- Image similarity: highest SSIM and PSNR in all nine settings (k = 2, 3, 5 on
  three datasets) against the pooling methods the paper compares
  [arxiv:2101.00440 tab. 1].
- ImageNet1K, trained from scratch: top-1 gains over the original network
  between +0.81 and +1.51 as printed, across 11 networks [arxiv:2101.00440 tab. 2].
  Some printed gains disagree with the table's own numbers (see Contradictions).
- ImageNet1K, pooling swapped into a trained network without retraining: printed
  gains between +0.37 and +0.80, with one row whose printed gain conflicts with
  its own numbers [arxiv:2101.00440 tab. 2].
- Same ResNet18 setting, six CNNs: the highest from-scratch top-1 in every column
  against the other pooling operators [arxiv:2101.00440 tab. 5].
- Action recognition: SoftPool versions of three video models are reported with
  higher top-1 on HACS, Kinetics-700 and UCF-101 in most rows. The baselines use
  different pre-training, so these rows are not controlled comparisons
  [arxiv:2101.00440 §4.6] [arxiv:2101.00440 tab. 7].

## When to use which

| Situation | Use | Why |
|---|---|---|
| A drop-in change to an existing CNN, with no new parameters or FLOPs | SoftPool | Parameters and GFLOPs are unchanged [arxiv:2101.00440 §4.4]; from-scratch gains in [arxiv:2101.00440 tab. 2] |
| The window's weaker responses should still count, not only its peak | SoftPool | Every activation enters the weighted sum [arxiv:2101.00440 §3.1] |
| Training-time pooling cost is the binding constraint | Average pooling | Lowest backward cost in the authors' CUDA test: 76 ms, against 234 ms for SoftPool and 267 ms for max [arxiv:2101.00440 tab. 3] |
| Pooling forward cost matters on either CPU or GPU | SoftPool, or average pooling if cheapest is required | SoftPool forward sits between average and max pooling in the authors' float16 setup [arxiv:2101.00440 tab. 3] [arxiv:2101.00440 app. §7] |
| Inputs can be negative, e.g. no ReLU before pooling | Test first | The paper's overflow and underflow checks assume non-negative activations [arxiv:2101.00440 app. §7] |
| You need the exact gradient of the pooling output | SoftPool with an exact backward, checked against autograd | The paper's backward uses the weights alone [arxiv:2101.00440 §3.2]. The exact form is derived on the card, not stated in the paper ([card](/papers/2101.00440)) |
| You plan to claim a gain of about one top-1 point | Run several seeds | The number of seeds is not stated, and McNemar's test compares paired predictions rather than seed variance [arxiv:2101.00440 app. §6] |

## Implementation sketch

Forward pass for one window of one channel. The paper omits the channel index;
pooling is applied per channel.

```pseudocode
# Forward pass over one window R of one channel  [arxiv:2101.00440 §3.1]
# Input:  window, k*k activations (non-negative, see When to use which)
# Output: pooled scalar

def softpool_window(window):
    # Exponentiate each activation  [arxiv:2101.00440 eq. 1]
    exp_values = exp(window)
    # Normalise so the weights in the window sum to one  [arxiv:2101.00440 eq. 1]
    weights = exp_values / sum(exp_values)
    # Output is the weighted sum of the activations  [arxiv:2101.00440 eq. 2]
    return sum(weights * window)
```

Subtracting the window maximum before the exponential leaves the weights
unchanged, because softmax is invariant to a shift. This matters for
half-precision overflow. The card records that the official CUDA kernel does
not subtract it.

## Contradictions

- **Backward pass.** The paper gives each input a gradient proportional to its
  forward weight [arxiv:2101.00440 §3.2]. The exact derivative of [arxiv:2101.00440 eq. 2]
  also depends on how the weights change with each input. The card derives this
  term and notes that the official code uses the weights-only form
  ([card](/papers/2101.00440)). The paper does not say which form the reported
  results used [arxiv:2101.00440 §3.2].
- **Operator on CPU and GPU.** The paper computes the weights within each
  channel's window [arxiv:2101.00440 §3.1]. The card records that the official
  non-CUDA path normalises over the channel dimension, so outputs differ between
  devices ([card](/papers/2101.00440)).
- **3D windows.** The paper states the 3D extension only as a kernel extension
  [arxiv:2101.00440 §3.4]. The card records that the official 3D forward and
  backward passes place their windows differently ([card](/papers/2101.00440)).
- **Printed gains.** Some printed gains in [arxiv:2101.00440 tab. 2] do not equal
  the difference of the printed top-1 values. ResNeXt101 32x8d pre-trained reads
  +0.58, but 78.89 − 79.31 = −0.42. DenseNet161 from scratch reads +0.93, but
  78.72 − 77.65 = 1.07.
- **Baseline labels.** [arxiv:2101.00440 tab. 5] attaches the Lp and Gate
  citations to the opposite labels from [arxiv:2101.00440 tab. 1] and
  [arxiv:2101.00440 tab. 3].

## Papers

- [Refining activation downsampling with SoftPool](/papers/2101.00440): replaces
  max or average pooling with a softmax-weighted sum that has no trainable
  parameters, tested on image downsampling, ImageNet1K and video.