Saliency prediction

Overview

Saliency prediction, also called fixation prediction, outputs a map over the pixels of an image or video frame. Each value estimates how likely a person viewing the scene is to look at that spot. The maps are scored against recorded eye-tracking fixations, using metrics such as AUC-J, SIM, s-AUC, CC and NSS 1801.07424 tab. 5. The video form is called dynamic fixation prediction: the model must track where attention moves across frames, not only what one frame looks like 1801.07424 §1.

The word also appears in video moment retrieval, where a model scores each clip for how well it matches a text query and calls that score saliency 2303.13874 §3. That target is query relevance, not where people look. The first three variants below use the fixation meaning; the last one uses the query-relevance meaning. The two scores are not comparable.

Variants

The ACLNet variants come from Revisiting Video Saliency: A Large-scale Benchmark and a New Model. The QD-DETR variant comes from Query-Dependent Video Representation for Moment Retrieval and Highlight Detection.

ACLNet: attentive CNN-convLSTM for video

A VGG-16 backbone (first five convolution blocks, with the last two pooling layers removed) feeds a convolutional LSTM. An attention branch on the last backbone block produces a map, trained on static fixation images. The map gates the features in residual form before the recurrent step, and a 1x1 convolution gives the output map 1801.07424 §4. On the DHF1K test set it reports AUC-J 0.890 and CC 0.434 when trained on DHF1K, Hollywood-2 and UCF sports together 1801.07424 tab. 5. Card: Revisiting Video Saliency: A Large-scale Benchmark and a New Model.

Attention branch alone, per frame

The attention map can be used as the output without the recurrent part. The paper reports AUC-J 0.870 and CC 0.380 on DHF1K for this output, under the same training setting 1801.07424 tab. 5. The branch is trained on static images, and it also receives gradients from the video loss, so it is not a purely static model 1801.07424 §4.

Video training without the attention module

The ablation removes the attention branch and trains on video only, with 1.5K videos. It reports AUC-J 0.847 on DHF1K, against 0.890 for the full model 1801.07424 tab. 6. The video count is smaller in this run too, so the drop mixes the effect of removing attention with the effect of less video data 1801.07424 §5.4.

QD-DETR: query-conditioned clip scores and moment spans

This variant does not predict fixations. Given a video and a text query, it scores each clip for how well it matches the query (highlight detection) and predicts the start and end of the matching moment (moment retrieval) 2303.13874 §1 §3. The model takes frozen, pre-extracted video and text features §3.

Clips first attend to the query words through cross-attention in the first encoder layers, so each clip's representation depends on the query 2303.13874 §3.1 eq. 1. A learned saliency token then gives one score per clip through a scaled dot product §3.3 eq. 3. During training, a video is also paired with the query of another item in the batch, and that pair's scores are pushed toward zero. The paper argues this stops the model from ranking clips without using the query §3.2 eq. 2. A DETR-style decoder with dynamic anchors predicts the span §3.4.

Reported results are author-reported. On the QVHighlights test split, the video-only model reaches R1@0.7 of 44.98 ± 0.8 over five runs, against 33.02 ± 1.7 for Moment-DETR 2303.13874 tab. 1. With per-category training on TVSum, its video-only average mAP is 85.0, against 73.3 for SL-Module 2303.13874 tab. 2. The ablation with only the cross-attention encoder reaches R1@0.7 of 38.71 2303.13874 tab. 4. That table is labelled as the validation split, but its baseline row equals the Moment-DETR test numbers in tab. 1, so treat the ablation as unverified for split 2303.13874 tab. 1 tab. 4. Card: Query-Dependent Video Representation for Moment Retrieval and Highlight Detection.

When to use which

Implementation sketch

Common core for the ACLNet variant, one frame at a time. The recurrent state carries from frame to frame. Gating form and loss weights are from the paper.

# Per-frame saliency step  [arxiv:1801.07424 §4]
# Input: frame (224, 224, 3); state from the previous frame
# Output: saliency map (28, 28); new state; attention map (28, 28)

def saliency_step(frame, state):
    # Backbone features from the first five VGG-16 conv blocks [§4]
    feats = backbone(frame)                    # (28, 28, 512)
    # Attention map from a branch on the last block [§4]
    att = attention_branch(feats)              # (28, 28)
    # Residual gating: each channel times (1 + att) [§4] [eq. 7]
    feats = (1 + att) * feats                  # (28, 28, 512)
    # Recurrent state with 3x3 convolutional gates [§4] [eq. 1-5]
    state = conv_lstm(feats, state)
    # 1x1 convolution head gives the map [§4]
    saliency = head(state.hidden)              # (28, 28)
    return saliency, state, att

Loss on one map, with weights 0.1 for the correlation and NSS terms 1801.07424 eq. 8-11:

# Saliency loss for one map  [arxiv:1801.07424 eq. 8-11]
# y: predicted map; fix: binary fixations; gt: blurred fixation map

def saliency_loss(y, fix, gt):
    # KL divergence, the primary term [eq. 9]
    kl = kl_divergence(gt, y)
    # Correlation and NSS, both negated so that lower is better [eq. 10-11]
    return kl + 0.1 * (-cc(y, gt)) + 0.1 * (-nss(y, fix))

Clip scoring for the QD-DETR variant. The negative-pair term is the only suppression signal; the rest of the model is a transformer 2303.13874 §3.2 eq. 2-3.

# Clip saliency and negative-pair loss  [arxiv:2303.13874 §3.2] [eq. 2-3]
# clip_tokens: (L, d) after cross-attention to a query
# neg: the same video scored against the query of another batch item

def clip_scores(saliency_token, clip_tokens):
    # Scaled dot product between projected token and each projected clip [eq. 3]
    return dot(w_v(clip_tokens), w_s(saliency_token)) / sqrt(d)

def negative_loss(neg_scores):
    # -log(1 - S) pushes mismatched-pair scores toward zero [eq. 2]
    return mean(-log(1 - neg_scores))

Gaps the paper does not fill, which an implementation has to decide:

• The size of the blur kernel for the ground-truth map is not stated 1801.07424 §4.
• The fixation and ground-truth maps are at video resolution, while the output is 28x28. How they are resized is not stated, and the choice changes the CC and NSS terms 1801.07424 §4.
• The attention output activation is not described 1801.07424 §4.
• Whether the backbone starts from ImageNet weights is not stated 1801.07424 §4.

For the QD-DETR variant:

• The margin Δ in the hinge loss is not stated 2303.13874 §3.4 eq. 5.
• The number of rank levels R in the contrastive loss is not stated, nor how the per-clip saliency levels are derived 2303.13874 §3.4 eq. 6.
• The paper does not say whether S is passed through a sigmoid before the negative loss 2303.13874 §3.2 eq. 2.
• With a batch of one, no other query exists to pair with the video, so the negative term has no valid pair 2303.13874 §3.2.

Contradictions

No direct contradiction. The two papers score different targets: fixations on free viewing (DHF1K, scored with AUC-J, SIM, s-AUC, CC and NSS 1801.07424 tab. 5) and query-relevant clips (QVHighlights and TVSum, scored with mAP, HIT@1 and R1 2303.13874 tab. 1 tab. 2). Numbers from one target do not compare with numbers from the other. Each target has one card so far.

Papers

• Revisiting Video Saliency: A Large-scale Benchmark and a New Model: introduces the DHF1K video fixation benchmark and the ACLNet model with a supervised static attention branch.
• Query-Dependent Video Representation for Moment Retrieval and Highlight Detection: QD-DETR, with query cross-attention, negative query pairs, a learned saliency token and a span decoder for moment retrieval and highlight detection.