Localizing Visual Sounds the Hard Way

In plain words

Many objects make a characteristic sound, and a video shows both the sound and the picture. This paper teaches a model to point at the object that is making the sound, without anyone drawing boxes around it. The model compares the audio with every part of the frame, and the parts that match the sound are pulled toward it while the rest are pushed away.

The new idea is what counts as "away". Most parts of a frame are background, and background parts of the same frame are the hardest examples to reject, so the model is trained to push them away explicitly. Because nobody says which parts really are the sound source, the model uses rough thresholds, and it ignores a band of uncertain parts between the confident "sound" and "not sound" regions.

The authors also build a new test set of videos with hand-drawn boxes around the sound sources, so that localization can be measured on video clips and not only on still images.

TL;DR

• Self-supervised audio-visual localization with a dual ResNet18 and a cosine-similarity heatmap, trained with an InfoNCE-style contrastive loss.
• Adds same-image background pixels as hard negatives, selected by soft thresholds with an uncertain band (Tri-map) that is ignored in training.
• Reports state of the art on Flickr SoundNet and on the new VGG-SS benchmark (author-reported; see Results).
• Implement the soft masks and the Tri-map. The repository has inference code only, so the training loss has to be built from the paper (see Paper vs code).

Problem

Audio-visual correspondence models can localize sound in a frame, but their attention often covers too little or too much of the sounding object §1. Prior self-supervised formulations use only positives and negatives from other clips §3.3. Explicit background negatives inside the same frame need hard labels, which are not available without supervision §3.4. The paper also reports that existing localization benchmarks are small, use still images, or have no object category labels §4.

Method

Representation §3.1. A video clip gives a centre frame I and the spectrogram of its audio a. Two CNNs f and g produce a visual map V and a pooled audio vector A with the same channel count c eq. 1 eq. 2.

Correspondence §3.2. The audio vector of clip i is probed against each spatial location of the visual map of clip j, with cosine similarity. This gives a map S_{i→j} of size h×w. Both features are L2-normalised first. The same-clip case is j = i.

Oracle contrastive loss §3.3. If a ground-truth mask m_i is available, the positive score is the mean of S over the masked (sounding) pixels. The negative score has two parts: a hard term, the mean of S over unmasked pixels of the same image, and an easy term, the spatial mean of S for every other clip j. The loss is an InfoNCE-style log-ratio of exp(P_i) against exp(P_i) + exp(N_i).

Self-supervised version §3.4. The mask is replaced by a soft threshold of the same-clip map. Two thresholds ε_p > ε_n give a positive mask and a negative mask with the formulas below. Pixels between the two thresholds are the uncertain Tri-map band and receive no weight. The loss is then computed on the same P_i and N_i as above.

# Self-supervised localization loss  [§3.4] [eq. 1] [eq. 2]
# Input:  frames (B, 3, 224, 224); spectrograms (B, 1, F, T)
# Output: scalar loss

def localization_loss(frames, spectrograms, eps_p, eps_n, tau):
    # Encode both streams and L2-normalise the channels  [§3.1] [eq. 1] [eq. 2]
    visual = normalise(visual_encoder(frames), dim=1)   # (B, C, h, w)
    audio = normalise(audio_encoder(spectrograms), dim=1)  # (B, C)
    losses = []
    for i in range(B):
        # Cosine map of clip i's audio on every frame j  [§3.2]
        maps = cosine_map(audio[i], visual)                # (B, h, w)
        own_map = maps[i]                                  # same clip, (h, w)
        # Soft positive weights: locations above eps_p  [§3.4]
        pos_weight = sigmoid((own_map - eps_p) / tau)
        # Soft negative weights: 1 minus locations above eps_n  [§3.4]
        neg_weight = 1 - sigmoid((own_map - eps_n) / tau)
        # Positive score: weighted mean over positive locations  [§3.4]
        pos_score = sum(pos_weight * own_map) / sum(pos_weight)
        # Hard-negative score: weighted mean over background locations  [§3.4]
        hard_score = sum(neg_weight * own_map) / sum(neg_weight)
        # Easy-negative score: spatial mean summed over the other clips  [§3.3]
        easy_score = sum(mean(maps_j) for j != i)
        # InfoNCE term with N_i = hard + easy, as written  [§3.4]
        neg_score = hard_score + easy_score
        losses.append(-log(exp(pos_score) / (exp(pos_score) + exp(neg_score))))
    return mean(losses)

Inference §6.3 test.py. Run the same two encoders on the full clip. The heatmap is the same-clip map; the official test script rescales it to [0, 1] and turns it into a binary mask (see Paper vs code).

Hyperparameters

Results

Author-reported. Setup: Flickr SoundNet test set of 250 image-audio pairs with three boxes per pair §5.2; VGG-SS test set of 5,158 videos over 220 classes §4 tab. 2; cIoU and AUC as defined in app. A.

Flickr SoundNet tab. 3:

Ablation, trained on VGG-Sound 144k, tested on Flickr SoundNet tab. 4:

VGG-SS, all models trained on VGG-Sound 144k tab. 5:

Open-set split, 110 seen and 110 unseen VGG-Sound categories, about 70k samples each §6.5 tab. 6:

Paper vs code

Official repository hche11/Localizing-Visual-Sounds-the-Hard-Way at 8fe8cbbf810e804cf8cc9b244bb5d77ec8aa8315, Apache-2.0. The paper's project page links it. The repository has inference code (test.py, model.py, utils.py, dataloader and backbones) and pretrained checkpoints (download links in its README). It does not contain training code, so the training loss and optimizer below are from the paper only.

• Easy negatives are weighted, not plain. Paper: a plain spatial mean of S_{i→j} for other clips §3.4. Code: each other clip's map is weighted by the positive mask and divided by the mask's sum model.py.
• Negatives are separate logits. Paper: one negative score N_i per clip, with hard and easy terms combined inside one log-ratio §3.3 §3.4. Code: the positive, each other clip, and the hard term are concatenated as separate logits and divided by 0.07 model.py. This is a multi-way softmax rather than the paper's two-term form.
• InfoNCE temperature. Not stated in the paper. Code uses 0.07 model.py.
• Same-clip masking. The paper excludes j = i from the easy sum §3.4. Code multiplies the whole (B, B) matrix by 1 - 100·I, so same-clip entries are scaled by −99 rather than removed model.py model.py.
• Test-time binarisation. Paper: a fixed threshold τ = 0.5 on the prediction map, A(τ) = {i | p_i > τ} app. A. Code: min-max normalises the map, then keeps the pixels above the median, so each image contributes its top 50% of pixels; the 0.5 is applied to that binary map test.py.
• Test audio length. Paper: 20 s of audio per Flickr frame, full length fed at test time §5.2 §5.3. Code: clips are padded or tiled, then truncated to 10 s dataloader.py.
• Image preprocessing. Paper: resize to 224 × 224 without cropping §5.3. Code: resize the shorter side to 224, then centre-crop dataloader.py.
• ImageNet initialisation. Paper: visual encoder pre-trained on ImageNet §5.3. Code: the loader for ImageNet weights is commented out base_models.py; the released checkpoint overrides the weights at test time.
• Matching. Positive and negative thresholds ε_p = 0.65 and ε_n = 0.4 §5.3 match the code defaults test.py. The sigmoid temperature τ = 0.03 matches model.py. The Flickr ground truth (sum of boxes divided by two, clipped at 1) matches app. A utils.py.
• VGG-SS ground truth. The paper does not define the union rule for VGG-SS. Code takes the union of boxes utils.py. The README says some boxes were updated and that VGG-SS results shift by 2–3% in IoU README.md.
• AUC. Paper pseudocode uses > and says "20 different thresholds" while its loop has 21 points alg. 1. Code uses >= on 21 points test.py.
• VGG-SS annotation file. The repository ships metadata/vggss.json. The paper does not state a separate licence for the annotations, so this card does not reproduce them.

Pitfalls

• Training is not in the repository. Loss reduction, batch handling and optimizer schedule beyond Adam and the learning rate come from the paper only §5.3. Test the implementation against the released checkpoint before trusting a retrain.
• Easy-negative magnitude grows with batch size under the literal sum over j §3.4. The code's per-clip logits (see Paper vs code) do not sum over j. Decide which form you implement and document it.
• Denominators can vanish. If no location exceeds ε_p, the positive score divides by near-zero [model.py#L69]. The same applies to the hard-negative score when almost every location exceeds ε_n. Code adds no guard, so add a small epsilon if you rewrite this.
• Sharp masks. With τ = 0.03 the masks are close to binary, and their gradient is significant only within a narrow band around each threshold §3.4. Training depends on where the heatmap values fall relative to 0.65 and 0.4.
• Uncertain band. The Tri-map band is ignored in training. Ablation row b (no band) is worse than row a (no negatives) tab. 4, so the band is not optional.
• Units in the text. §6.1 writes "0.546% vs. 0.582%", but the values are cIoU fractions from tab. 3, not percentages.
• Benchmark change. VGG-SS CIoU values are much lower than Flickr values §6.3 tab. 3 tab. 5. Compare only within one benchmark.
• Hidden costs. Training uses 224 × 224 frames, 512-D audio features and a full batch of 256 with a cross-clip matrix (B × B spatial maps, per §3.4). The paper does not report memory or wall-clock time.
• Data availability. The released VGG-SS annotations are in the repository without a separate licence, and the VGG-Sound dataset is under its own terms §4.

Lineage

• Builds on: the audio-visual correspondence formulation with a cosine heatmap [Arandjelovic18objects], and the attention and max-pooling variants [senocak2018learning] [Afouras20b] §3.2 §3.4; none of these has a card.
• Compared against: Attention10k [senocak2018learning], AVObject [Afouras20b], CoarsetoFine [qian2020multiple], DMC [Hu_2019_CVPR] tab. 3 tab. 5.
• Superseded by: not stated in this wiki.