---
kind: concept
slug: saliency-prediction
title: Saliency prediction
summary: >-
  Scoring image or video content for saliency, as a pixel-wise fixation map or a per-clip relevance score. Covers DHF1K video fixation prediction with ACLNet, a CNN with a convolutional LSTM and an attention branch, and QD-DETR, which scores clips against a text query. Results are author-reported, from two papers; the two scores measure different things.
tags: [saliency, eye-tracking, video, attention, highlight-detection, moment-retrieval]
papers: ["1801.07424", "2303.13874"]
related: [attention-mechanism, convolutional-recurrent-network, long-short-term-memory, segmentation-evaluation, cross-attention, contrastive-learning, object-queries]
status: draft
updated_at: 2026-11-06
---

## Overview

Saliency prediction, also called fixation prediction, outputs a map over the
pixels of an image or video frame. Each value estimates how likely a person
viewing the scene is to look at that spot. The maps are scored against
recorded eye-tracking fixations, using metrics such as AUC-J, SIM, s-AUC, CC
and NSS [arxiv:1801.07424 tab. 5]. The video form is called dynamic fixation
prediction: the model must track where attention moves across frames, not only
what one frame looks like [arxiv:1801.07424 §1].

The word also appears in video moment retrieval, where a model scores each clip
for how well it matches a text query and calls that score saliency
[arxiv:2303.13874 §3]. That target is query relevance, not where people look.
The first three variants below use the fixation meaning; the last one uses the
query-relevance meaning. The two scores are not comparable.

## Variants

The ACLNet variants come from [Revisiting Video Saliency: A Large-scale Benchmark and a New Model](/papers/1801.07424). The QD-DETR variant comes from [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](/papers/2303.13874).

### ACLNet: attentive CNN-convLSTM for video

A VGG-16 backbone (first five convolution blocks, with the last two pooling
layers removed) feeds a convolutional LSTM. An attention branch on the last
backbone block produces a map, trained on static fixation images. The map
gates the features in residual form before the recurrent step, and a 1x1
convolution gives the output map [arxiv:1801.07424 §4]. On the DHF1K test set
it reports AUC-J 0.890 and CC 0.434 when trained on DHF1K, Hollywood-2 and UCF
sports together [arxiv:1801.07424 tab. 5]. Card: [Revisiting Video Saliency: A Large-scale Benchmark and a New Model](/papers/1801.07424).

### Attention branch alone, per frame

The attention map can be used as the output without the recurrent part. The
paper reports AUC-J 0.870 and CC 0.380 on DHF1K for this output, under the same
training setting [arxiv:1801.07424 tab. 5]. The branch is trained on static
images, and it also receives gradients from the video loss, so it is not a
purely static model [arxiv:1801.07424 §4].

### Video training without the attention module

The ablation removes the attention branch and trains on video only, with 1.5K
videos. It reports AUC-J 0.847 on DHF1K, against 0.890 for the full model
[arxiv:1801.07424 tab. 6]. The video count is smaller in this run too, so the
drop mixes the effect of removing attention with the effect of less video data
[arxiv:1801.07424 §5.4].

### QD-DETR: query-conditioned clip scores and moment spans

This variant does not predict fixations. Given a video and a text query, it
scores each clip for how well it matches the query (highlight detection) and
predicts the start and end of the matching moment (moment retrieval)
[arxiv:2303.13874 §1] [§3]. The model takes frozen, pre-extracted video and
text features [§3].

Clips first attend to the query words through cross-attention in the first
encoder layers, so each clip's representation depends on the query
[arxiv:2303.13874 §3.1] [eq. 1]. A learned saliency token then gives one score
per clip through a scaled dot product [§3.3] [eq. 3]. During training, a video
is also paired with the query of another item in the batch, and that pair's
scores are pushed toward zero. The paper argues this stops the model from
ranking clips without using the query [§3.2] [eq. 2]. A DETR-style decoder
with dynamic anchors predicts the span [§3.4].

Reported results are author-reported. On the QVHighlights test split, the
video-only model reaches R1@0.7 of 44.98 ± 0.8 over five runs, against 33.02 ±
1.7 for Moment-DETR [arxiv:2303.13874 tab. 1]. With per-category training on
TVSum, its video-only average mAP is 85.0, against 73.3 for SL-Module
[arxiv:2303.13874 tab. 2]. The ablation with only the cross-attention encoder
reaches R1@0.7 of 38.71 [arxiv:2303.13874 tab. 4]. That table is labelled as
the validation split, but its baseline row equals the Moment-DETR test numbers
in tab. 1, so treat the ablation as unverified for split [arxiv:2303.13874 tab. 1]
[tab. 4]. Card: [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](/papers/2303.13874).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need a fixed, held-out video benchmark to score models against | DHF1K: 600 training, 100 validation, 300 test clips | Test labels stay hidden and are scored by a server; validation scores are not the benchmark numbers [arxiv:1801.07424 §3] [arxiv:1801.07424 §5.1] |
| You want the highest DHF1K score the paper reports and can train on video and static images | ACLNet, training setting (iv) | AUC-J 0.890, against 0.870 for the attention branch alone [arxiv:1801.07424 tab. 5] |
| You need a per-frame map with less structure | Attention branch output | AUC-J 0.870 on DHF1K, a loss of 0.020 against the full model [arxiv:1801.07424 tab. 5] |
| You have little or no static fixation data | Keep the attention branch and its static training, if you can; otherwise expect a drop | The no-attention ablation drops to AUC-J 0.847 [arxiv:1801.07424 tab. 6]. The drop also includes the smaller video count |
| You must avoid optical flow | ACLNet | The model uses no optical flow, unlike the two-stream baseline the paper compares against [arxiv:1801.07424 §5.2] |
| Your target footage is sports-like, as in UCF sports | Train on UCF sports alone (setting iii) | Reports AUC-J 0.905 on UCF sports, above the all-data setting; the authors attribute this to the similarity of the clips, so it may not carry to other domains [arxiv:1801.07424 tab. 5] [arxiv:1801.07424 §5.2] |
| Compute is limited | Plan for about 30 hours on one TITAN X for setting (iv), and about 0.08 s per 224x224 frame at inference | Authors' hardware and timing, without pre- or post-processing [arxiv:1801.07424 §5.1] |
| You need clip scores for a text query, plus a time span, and you have moment-annotated video | QD-DETR | Outputs both a span and per-clip scores; video-only QVHighlights R1@0.7 is 44.98 against 33.02 for Moment-DETR [arxiv:2303.13874 tab. 1] |
| You need where people look in free viewing, with no text | ACLNet | QD-DETR scores query relevance, not fixations [arxiv:2303.13874 §3] [arxiv:1801.07424 §3] |
| You can use only pre-extracted, frozen features | QD-DETR, with those features | The method is set up on frozen encoders; end-to-end training is not evaluated [arxiv:2303.13874 §3] |

## Implementation sketch

Common core for the ACLNet variant, one frame at a time. The recurrent state
carries from frame to frame. Gating form and loss weights are from the paper.

```pseudocode
# Per-frame saliency step  [arxiv:1801.07424 §4]
# Input: frame (224, 224, 3); state from the previous frame
# Output: saliency map (28, 28); new state; attention map (28, 28)

def saliency_step(frame, state):
    # Backbone features from the first five VGG-16 conv blocks [§4]
    feats = backbone(frame)                    # (28, 28, 512)
    # Attention map from a branch on the last block [§4]
    att = attention_branch(feats)              # (28, 28)
    # Residual gating: each channel times (1 + att) [§4] [eq. 7]
    feats = (1 + att) * feats                  # (28, 28, 512)
    # Recurrent state with 3x3 convolutional gates [§4] [eq. 1-5]
    state = conv_lstm(feats, state)
    # 1x1 convolution head gives the map [§4]
    saliency = head(state.hidden)              # (28, 28)
    return saliency, state, att
```

Loss on one map, with weights 0.1 for the correlation and NSS terms
[arxiv:1801.07424 eq. 8-11]:

```pseudocode
# Saliency loss for one map  [arxiv:1801.07424 eq. 8-11]
# y: predicted map; fix: binary fixations; gt: blurred fixation map

def saliency_loss(y, fix, gt):
    # KL divergence, the primary term [eq. 9]
    kl = kl_divergence(gt, y)
    # Correlation and NSS, both negated so that lower is better [eq. 10-11]
    return kl + 0.1 * (-cc(y, gt)) + 0.1 * (-nss(y, fix))
```

Clip scoring for the QD-DETR variant. The negative-pair term is the only
suppression signal; the rest of the model is a transformer
[arxiv:2303.13874 §3.2] [eq. 2-3].

```pseudocode
# Clip saliency and negative-pair loss  [arxiv:2303.13874 §3.2] [eq. 2-3]
# clip_tokens: (L, d) after cross-attention to a query
# neg: the same video scored against the query of another batch item

def clip_scores(saliency_token, clip_tokens):
    # Scaled dot product between projected token and each projected clip [eq. 3]
    return dot(w_v(clip_tokens), w_s(saliency_token)) / sqrt(d)

def negative_loss(neg_scores):
    # -log(1 - S) pushes mismatched-pair scores toward zero [eq. 2]
    return mean(-log(1 - neg_scores))
```

Gaps the paper does not fill, which an implementation has to decide:

- The size of the blur kernel for the ground-truth map is not stated
  [arxiv:1801.07424 §4].
- The fixation and ground-truth maps are at video resolution, while the output
  is 28x28. How they are resized is not stated, and the choice changes the CC
  and NSS terms [arxiv:1801.07424 §4].
- The attention output activation is not described [arxiv:1801.07424 §4].
- Whether the backbone starts from ImageNet weights is not stated
  [arxiv:1801.07424 §4].

For the QD-DETR variant:

- The margin Δ in the hinge loss is not stated [arxiv:2303.13874 §3.4] [eq. 5].
- The number of rank levels R in the contrastive loss is not stated, nor how
  the per-clip saliency levels are derived [arxiv:2303.13874 §3.4] [eq. 6].
- The paper does not say whether S is passed through a sigmoid before the
  negative loss [arxiv:2303.13874 §3.2] [eq. 2].
- With a batch of one, no other query exists to pair with the video, so the
  negative term has no valid pair [arxiv:2303.13874 §3.2].

## Contradictions

No direct contradiction. The two papers score different targets: fixations
on free viewing (DHF1K, scored with AUC-J, SIM, s-AUC, CC and NSS
[arxiv:1801.07424 tab. 5]) and query-relevant clips (QVHighlights and TVSum,
scored with mAP, HIT@1 and R1 [arxiv:2303.13874 tab. 1] [tab. 2]). Numbers from
one target do not compare with numbers from the other. Each target has one card
so far.

## Papers

- [Revisiting Video Saliency: A Large-scale Benchmark and a New Model](/papers/1801.07424): introduces the DHF1K video fixation benchmark and the ACLNet model with a supervised static attention branch.
- [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](/papers/2303.13874): QD-DETR, with query cross-attention, negative query pairs, a learned saliency token and a span decoder for moment retrieval and highlight detection.