Bilinear pooling

Overview

Bilinear pooling fuses two feature vectors at each position with an outer product, then sums the products over all positions. Every channel of one stream is paired with every channel of the other, so the result carries pairwise interactions that a plain concatenation or sum does not. Within the paper, the bilinear output is written as x = f_{RGB}^{T} f_N, sum-pooled over spatial positions, for RoI features of an RGB stream and a noise stream 1805.04953 §3.3. The paper then applies a signed square root and L2 normalisation before the classifier 1805.04953 §3.3.

The output has one entry per channel pair. With 1024 channels per stream 1805.04953 §3.4, the full product has about one million entries, which is why the paper uses a compact approximation instead 1805.04953 §3.3.

This page rests on a single paper card. Its only comparisons are against the single streams and late fusion 1805.04953 §4.2, so it says nothing about how bilinear fusion ranks against concatenation, element-wise products or attention-based fusion.

Variants

Full outer product with signed square root

The exact form described in the paper: outer product of the two feature vectors, summed over positions, then signed square root and L2 normalisation 1805.04953 §3.3. The paper presents the compact variant as a cheaper replacement for this step, so the full form is the reference rather than the one the paper uses 1805.04953 §3.3.

Compact bilinear pooling (used by RGB-N)

Replaces the full outer product with a compact approximation (method of Gao et al., cited by the paper but not on a card here) to save memory and time, with a fixed output dimension of 16384 1805.04953 §3.3 1805.04953 §3.4. The code needs a custom FFT operation built from its README (checked when the RGB-N card was written). Card: Learning Rich Features for Image Manipulation Detection.

When to use which

Implementation sketch

Based on the paper's description of the fusion step. The 7×7 grid and the channel count C are the RGB-N values; the paper's compact approximation is not reproduced here.

# Bilinear fusion of two RoI feature maps  [arxiv:1805.04953 §3.3]
# Input:  f_rgb, f_noise: (R, 7, 7, C) per-RoI features from two streams
# Output: fused vector per RoI, L2-normalised

def bilinear_fuse(f_rgb, f_noise, out_dim=16384):
    # flatten the 7x7 grid; the outer product is summed over positions  [§3.3]
    a = reshape(f_rgb, (R, 49, C))
    b = reshape(f_noise, (R, 49, C))
    # compact approximation of the summed outer product, out_dim features  [§3.3]
    # [§3.4] fixes out_dim at 16384 for RGB-N
    x = compact_bilinear_sum(a, b, out_dim)    # (R, out_dim)
    # signed square root, then L2 normalisation  [§3.3]
    x = sign(x) * sqrt(abs(x))
    return x / norm(x, axis=1)

Contradictions

Paper vs code on where pooling happens. The paper pools the 7×7×1024 RoI features 1805.04953 §3.3 1805.04953 §3.4. The released code, as checked on the RGB-N card, runs each stream's conv5 block on its crops first and pools the resulting 2048-channel features resnet_fusion.py. The two sides are stated here, not resolved; an implementation that follows one will not match the other's channel count.

Papers

• Learning Rich Features for Image Manipulation Detection: RGB-N fuses RGB and noise RoI features with compact bilinear pooling and reports it beats either stream alone and late fusion.