Revisiting Video Saliency: A Large-scale Benchmark and a New Model

In plain words

People looking at a moving scene do not look everywhere equally. Their eyes settle on some spots more than others, and a saliency model tries to predict those spots frame by frame. Until this work, the video versions of this task had few test clips and no fixed, held-out test set to score against, so results were hard to compare.

The paper contributes two things. The first is a benchmark: about a thousand short, varied video clips with eye-tracking data from a group of viewers, and a test set whose answers stay hidden until a model is submitted. The second is a neural model that looks at each frame with a convolutional network and remembers earlier frames with a recurrent cell. A separate branch learns where the eye goes in still images, and its output is used to boost the features the video model sees. The idea is to reuse large image datasets so the video model does not have to learn everything from few video clips.

TL;DR

Use DHF1K as a held-out video fixation benchmark and ACLNet as an end-to-end video saliency baseline. The model is a VGG-16 CNN with a convLSTM, and a static attention branch trained on SALICON images. Results are author-reported and best on DHF1K, Hollywood-2 and UCF sports in the paper's tables. The model code is not in the repository and was not read for this card.

Problem

Dynamic fixation prediction on video lacked a large, varied dataset with a held-out test set and an evaluation server §1. Existing video eye-tracking sets were small, limited in content, or in several cases built around a task rather than free viewing §2 tab. 1. The paper also argues that deep video models ignored the large static fixation datasets and did not use attention §2.2. The table below summarises the dataset statistics the authors report.

Source: tab. 1.

Method

Dataset (DHF1K). The 1,000 clips were chosen from YouTube search results for about 200 key terms. Each clip is 30 fps, resized to 640x360, and the set totals 582,605 frames §3. Each clip has a main category label (7 super-classes, 150 sub-classes) and attributes for motion and object count §3 tab. 2 tab. 3 tab. 4. Seventeen participants viewed the clips on an SMI RED 250 eye tracker at 250 Hz; the clips were split into 10 sessions and shown in random order §3.

Split. 600 training, 100 validation and 300 test clips, where the test annotations are held out for a benchmark server §3 §5.1.

Model (ACLNet). The pipeline is as follows §4 fig. 3.

1. Each frame, at 224x224 pixels, goes through the first five convolution blocks of VGG-16. Pool4 and pool5 are removed, giving 28x28 features with 512 channels, an 8x downsampling instead of 32x §4.
2. An attention branch on conv5-3 pools the features down to a 7x7 map, which gives a receptive field of 260x260. The map is upsampled 4x to 28x28 §4. The branch is supervised by static fixation images, and also receives gradients from the dynamic loss, which the paper calls implicit training §4.
3. The attention map M is combined with the features in residual form: each channel is multiplied by (1 + M) §4 eq. 7. The earlier form, M times the features eq. 6, is the variant tested as "without residual connection" in the ablation §5.4.
4. A convLSTM with gates for input, forget and output runs on these features, using convolutions instead of matrix products and a 3x3 kernel §4 eq. 1-5.
5. A 1x1 convolution of the hidden state gives the 28x28 dynamic saliency map §4.

# Attentive CNN-LSTM, one frame  [§4] [eq. 1-7]
# Input:  frame (224, 224, 3); lstm_state from the previous frame
# Output: saliency (28, 28); lstm_state; attention (28, 28)

def attentive_cnn_lstm_step(frame, lstm_state):
    # VGG-16 conv1 to conv5-3, with pool4 and pool5 removed  [§4]
    features = vgg16_conv5_3(frame)              # (28, 28, 512)
    # Attention branch on conv5-3: pool to 7x7, upsample x4  [§4]
    attention = attention_branch(features)       # (28, 28), intended in [0, 1]
    # Residual gating: each channel times (1 + attention)  [§4] [eq. 7]
    enhanced = (1 + attention) * features        # (28, 28, 512)
    # convLSTM over the enhanced features, 3x3 kernels  [§4] [eq. 1-5]
    lstm_state = convlstm(enhanced, lstm_state)  # hidden state (28, 28, C)
    # 1x1 convolution of the hidden state  [§4]
    saliency = conv1x1(lstm_state.hidden)        # (28, 28)
    return saliency, lstm_state, attention

Loss. The loss sums three metrics with weights 1, 0.1 and 0.1 §4 eq. 8. Y is the predicted map in [0, 1], P the binary fixation map and Q the continuous map made by blurring the fixations with a small Gaussian §4.

# Saliency loss for one map  [§4] [eq. 8-11]
# Inputs: y (28, 28) in [0, 1]; p (28, 28) binary fixations;
#         q (28, 28) continuous ground-truth map

def saliency_loss(y, p, q):
    # KL divergence, the primary term  [eq. 9]
    kl = sum(q * log(q / y))
    # Linear correlation, negated so that minimising improves it  [eq. 10]
    cc = -cov(y, q) / (std(y) * std(q))
    # NSS: z-scored prediction averaged over fixated pixels  [eq. 11]
    y_norm = (y - mean(y)) / std(y)
    nss = -sum(y_norm * p) / sum(p)
    # Weights alpha_1 = alpha_2 = 0.1  [§4] [eq. 8]
    return kl + 0.1 * cc + 0.1 * nss

Training protocol. Training alternates between two batch types §4.

# Alternating training: video batch, then image batch  [§4] [eq. 12-13]
# Video batch: 20 consecutive frames from one random video, loss over all
#   of them. Image batch: 20 static images from the SALICON training set.

def train(video_batches, image_batches):
    for video_batch, image_batch in zip(video_batches, image_batches):
        # Dynamic loss summed over the T frames  [eq. 12]
        loss_dynamic = 0
        for t in range(T):
            saliency_t = model_forward(video_batch.frames[t])
            loss_dynamic += saliency_loss(saliency_t, p_t, q_t)
        update(model, loss_dynamic)  # which parameters: not stated

        # Static loss on the attention map only  [eq. 13]
        attention = attention_forward(image_batch.images)
        loss_static = saliency_loss(attention, p_static, q_static)
        update(attention_branch, loss_static)  # LSTM is not trained here

Hyperparameters

Dataset splits §5.1: DHF1K 600/100/300; Hollywood-2 823 training and 884 test videos, with 10 percent of training videos held out for validation; UCF sports 103 training and 47 test videos, with 10 percent held out.

Results

Author-reported. "Ours" rows are trained on the stated training setting and tested on the stated test set. Baselines use public implementations, except the two-stream model, which the authors re-implemented §5.1 §5.2 tab. 5.

DHF1K test set (300 clips):

Hollywood-2 (884 test clips) and UCF sports (47 test clips), best setting per dataset tab. 5:

Ablation on DHF1K, training setting (iv) baseline AUC-J 0.890 tab. 6 §5.4:

Reported comparisons: deep video models beat classic dynamic ones on DHF1K, and classic dynamic models do not beat their static counterparts §5.3. The The model itself uses no optical flow, unlike Fang et al. and the two-stream baseline §5.2. Full baseline tables: tab. 5.

Paper vs code

Searched: the paper's header links github.com/wenguanwang/DHF1K, and the repository's owner is the first author. It contains the dataset-side material, not the model. Cloned at commit 35570c7c01128f1acb4de161965e7a260cfc162f.

• Model code not in the repository. The README says the ACLNet code and the training code are on Google Drive and Baidu Pan README.md README.md. Those links were not read. Everything below comes from the repository files only, so the model's hyperparameters are not checked against code.
• License. The repository has no LICENSE file. The README states the dataset and code are CC BY 4.0 README.md. The model code on Drive has no verified license.
• Gaussian blur. The paper says "a small Gaussian kernel" §4. The DHF1K generator uses a Gaussian with full width at half maximum of 30 pixels at video resolution, which gives sigma of about 12.7 pixels, and normalises the result to a maximum of 1 make_gauss_masks.m make_gauss_masks.m. The README says the UCF sports and Hollywood-2 maps use a 150x150 Gaussian with sigma 20 instead README.md.
• Hollywood-2 clips. The paper says 884 test clips §5.1. The README says the split clips are used, each containing a single shot, not the full videos README.md.
• Frameworks. The README names Keras 2.2.2 and TensorFlow 1.10.0 README.md. The paper says Keras only §5.1; no conflict, but the versions are not in the paper.
• Corrections. The README says the UCF sports numbers for setting (iii) were corrected in the arXiv version README.md. This card uses arXiv v3.

Pitfalls

• Equation reference. The ablation text refers to "Eq. 2" for the version without residual connection §5.4. In the printed paper, Eq. (2) is the forget gate of the convLSTM. The intended equation is the attention-gating equation, printed as (6) eq. 6.
• Output resolution. The model outputs 28x28 maps, but the fixations and Gaussian maps are 640x360 at video resolution. The paper never says how P and Q are resized to 28x28. An implementation has to choose this, and the choice changes the CC and NSS terms.
• Normalisation of Y and Q. The KL term sums Q times log(Q/Y) eq. 9. It is written without normalising Y or Q to sum to one. The repository's Gaussian maps are normalised to a maximum of 1, not to a sum of 1 [code: make_gauss_masks.m#L38]. The paper does not say what is done in training.
• Attention output. The text says M is in [0, 1] §4, but the activation of the attention branch is not described. With residual gating, the output of (1 + M) lies in [1, 2] if M is in [0, 1].
• Parameters updated by the dynamic loss. The paper does not say whether the dynamic loss eq. 12 updates the attention branch as well as the CNN and convLSTM. The text says the branch is trained implicitly through this loss §4.
• VGG-16 weights. Whether the backbone is pretrained on ImageNet is not stated.
• Convolutional LSTM peepholes. Equations 1 and 2 use the previous cell state C_{t-1}, while equation 3 uses the current C_t eq. 1-3. Implement this as written, not as a symmetric peephole.
• Cost of training. About 30 hours on a single TITAN X in setting (iv) §5.1. The 20-frame video batches are the main memory cost.
• Test-set caveat. The DHF1K test annotations are held out. Results on the validation split are not the same as the benchmark numbers.
• UCF sports. Setting (iii), with only 103 training clips, beats setting (iv), which has far more data. The authors explain this by the similarity of the UCF clips §5.2. Treat that as a dataset-specific effect, not a general result.
• Baseline re-runs. The two-stream model was re-implemented by the authors because the official code could not run §5.1 footnote.
• Version. The repository cites a later TPAMI 2019 paper with the same dataset. It was not read for this card.

Lineage

• Builds on: convolutional LSTM and VGG-16, both cited in §4; the loss combination is from SALICON, cited in §4. No arXiv ids are given in this card.
• Compared against: PQFT, Seo et al., Rudoy et al., Hou et al., Fang et al., OBDL, AWS-D, OM-CNN and two-stream, which are the dynamic models; ITTI, GBVS, SALICON, DVA, Shallow-Net and Deep-Net, which are the static ones §5.1.
• Superseded by: not known from content in this wiki.