---
kind: concept
slug: bilinear-pooling
title: Bilinear pooling
summary: >-
  Fuse two feature maps by summing their channel-wise outer products over spatial positions, then signed square root and L2 normalisation. Compact bilinear pooling keeps the output at a fixed size (16384 in RGB-N). Only one paper card so far, which fuses an RGB and a noise stream per region; no comparison with other fusion methods, runtime or memory numbers are available here.
tags: [fusion, two-stream, pooling, image-forensics]
papers: ["1805.04953"]
related: []
status: draft
updated_at: 2026-10-18
---

## Overview

Bilinear pooling fuses two feature vectors at each position with an outer
product, then sums the products over all positions. Every channel of one
stream is paired with every channel of the other, so the result carries
pairwise interactions that a plain concatenation or sum does not. Within the
paper, the bilinear output is written as $x = f_{RGB}^{T} f_N$, sum-pooled over
spatial positions, for RoI features of an RGB stream and a noise stream
[arxiv:1805.04953 §3.3]. The paper then applies a signed square root and L2
normalisation before the classifier [arxiv:1805.04953 §3.3].

The output has one entry per channel pair. With 1024 channels per stream
[arxiv:1805.04953 §3.4], the full product has about one million entries, which
is why the paper uses a compact approximation instead [arxiv:1805.04953 §3.3].

This page rests on a single paper card. Its only comparisons are against the
single streams and late fusion [arxiv:1805.04953 §4.2], so it says nothing
about how bilinear fusion ranks against concatenation, element-wise products
or attention-based fusion.

## Variants

### Full outer product with signed square root

The exact form described in the paper: outer product of the two feature
vectors, summed over positions, then signed square root and L2 normalisation
[arxiv:1805.04953 §3.3]. The paper presents the compact variant as a cheaper
replacement for this step, so the full form is the reference rather than the
one the paper uses [arxiv:1805.04953 §3.3].

### Compact bilinear pooling (used by RGB-N)

Replaces the full outer product with a compact approximation (method of Gao
et al., cited by the paper but not on a card here) to save memory and time,
with a fixed output dimension of 16384 [arxiv:1805.04953 §3.3]
[arxiv:1805.04953 §3.4]. The code needs a custom FFT operation built from its
README (checked when the RGB-N card was written). Card:
[Learning Rich Features for Image Manipulation Detection](/papers/1805.04953).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Two streams over the same region, one classifier on the joint feature, memory allows a ~16K-dim vector | Compact bilinear pooling, output 16384 | The paper's setting; the fixed output keeps the classifier layer size bounded [arxiv:1805.04953 §3.3] [arxiv:1805.04953 §3.4] |
| Two streams, but a classifier-side cost of 1024×1024 features is acceptable | Full outer product | Described in the paper; the compact form is presented as cheaper, so no paper result favours the full form [arxiv:1805.04953 §3.3] |
| Need the fusion to beat late fusion on all benchmarks | Bilinear fusion, as the paper reports | Late fusion was below bilinear fusion on every F1 and AUC row for the four datasets [arxiv:1805.04953 §4.2] [tab. 3] [tab. 4] |
| Noise-only features for a dataset with uncompressed splices | Check the single-stream numbers first | Noise-only had the best F1 on Columbia (0.705 vs 0.697 for RGB-N), so fusion did not win every row [arxiv:1805.04953 §4.2] [tab. 3] |
| Budget for runtime or memory of the fused model | Not stated | The paper does not report runtime [arxiv:1805.04953 §4.2] |

## Implementation sketch

Based on the paper's description of the fusion step. The 7×7 grid and the
channel count C are the RGB-N values; the paper's compact approximation is not
reproduced here.

```pseudocode
# Bilinear fusion of two RoI feature maps  [arxiv:1805.04953 §3.3]
# Input:  f_rgb, f_noise: (R, 7, 7, C) per-RoI features from two streams
# Output: fused vector per RoI, L2-normalised

def bilinear_fuse(f_rgb, f_noise, out_dim=16384):
    # flatten the 7x7 grid; the outer product is summed over positions  [§3.3]
    a = reshape(f_rgb, (R, 49, C))
    b = reshape(f_noise, (R, 49, C))
    # compact approximation of the summed outer product, out_dim features  [§3.3]
    # [§3.4] fixes out_dim at 16384 for RGB-N
    x = compact_bilinear_sum(a, b, out_dim)    # (R, out_dim)
    # signed square root, then L2 normalisation  [§3.3]
    x = sign(x) * sqrt(abs(x))
    return x / norm(x, axis=1)
```

## Contradictions

Paper vs code on where pooling happens. The paper pools the 7×7×1024 RoI
features [arxiv:1805.04953 §3.3] [arxiv:1805.04953 §3.4]. The released code, as
checked on the RGB-N card, runs each stream's conv5 block on its crops first
and pools the resulting 2048-channel features [code: lib/nets/resnet_fusion.py#L250-L270].
The two sides are stated here, not resolved; an implementation that follows
one will not match the other's channel count.

## Papers

- [Learning Rich Features for Image Manipulation Detection](/papers/1805.04953): RGB-N fuses RGB and noise RoI features with compact bilinear pooling and reports it beats either stream alone and late fusion.