Deep Visual Attention Prediction

In plain words

When you look at a picture, your eyes land on a few spots that stand out, such as a face, a bright colour or a sign with text. Saliency models try to predict those spots by computing a heat map over the image. This paper trains one neural network to produce that heat map.

The network starts from a standard image classifier (VGG16). It takes intermediate layers from three depths. Shallow layers see small detail, and deep layers see large regions. Each of the three layers gets its own small upsampling branch and its own training signal, so each branch learns to draw a map at its own scale. A final learned layer blends the three maps into one answer.

The point is that one network does the job of several, instead of running several networks on resized copies of the image. The authors report that it is accurate on several eye-tracking benchmarks and fast enough for practical use.

TL;DR

• Task: predict where people look in a natural image, as a per-pixel map §1.
• Architecture: VGG16 encoder without its last pooling layer, plus one deconvolution decoder per side output on conv3-3, conv4-3 and conv5-3 §3.1 §3.3.1 §3.3.2.
• Training: cross-entropy on every side output and on the fused output, trained on SALICON only §3.2 §3.3.3.
• Result: author-reported best on most metrics for MIT1003, TORONTO, PASCAL-S and DUT-OMRON, not on MIT300 §4.3.
• Speed: about 0.1 s per image on a GPU, author-reported §3.3.3.

Problem

Saliency models have long been built from hand-designed features and centre-surround contrast §2.1. Deep saliency models use one of three layouts: a single stream, several streams that each see a rescaled input, or a skip-layer network that taps several layers of one stream §2.2. The authors point out that multi-stream models learn scale "outside" the network by training several networks, which they call redundant §2.2 §1. Pooling in a standard CNN also destroys spatial detail, which the paper recovers with a decoder §3.1.

The paper proposes to learn multi-scale and multi-level saliency inside one network and to supervise each level directly, which the authors describe as "deep supervision" §1 §2.2.

Method

1. Encoder. Use the convolutional layers of VGG16, with the fully connected layers dropped and the final pooling layer (pool5) removed. The last feature map is then h/16 by w/16 §3.3.1. The paper states that the first 13 convolution layers are the encoder §1.

2. Side outputs. Pick three encoder layers, conv3-3, conv4-3 and conv5-3, as side outputs (M = 3). Deeper layers have larger receptive fields, so the three give a fine-to-coarse set of predictions §3.3.1. More layers gave no gain and cost more computation §3.3.1.

3. Decoder per side output. Each side output gets its own decoder of transposed convolutions (called deconvolutions in the paper). Each deconvolution doubles the spatial size and is followed by a ReLU §3.3.2 eq. 2 eq. 3. The number of deconvolutions is 2 for conv3-3, 3 for conv4-3 and 4 for conv5-3, so every branch ends at the input resolution §3.3.2. The channel count falls at each step, and the paper gives the conv4-3 path as 256, 128, 64, 32, then 1 §3.3.2.

4. Classifier per side output. A 1×1 convolution with a sigmoid turns the decoder output into one saliency map at the input size §3.2. The code uses a 3×3 kernel without padding instead (see Paper vs code).

5. Fusion. The three side maps are merged by an attention fusion layer. It is a learned weighted sum, F = Σ w_f^m S^m, and the paper describes it as a 1×1 convolution §3.2. The code instead concatenates the three pre-sigmoid side outputs and applies a 3×3 convolution with padding 1 (see Paper vs code).

6. Loss. Each side output gets a pixel-wise cross-entropy against the ground-truth map, summed over all pixels eq. 7. The fused output gets the same loss eq. 8. The total objective weights the side losses by 1/M and adds the fused loss eq. 9.

7. Initialisation. The encoder starts from VGG16 weights pre-trained on ImageNet. All other layers start from a zero-mean Gaussian with standard deviation 0.01 §3.3.3.

8. Training. Use SGD with momentum on the SALICON training split, with the schedule and batch size listed under Hyperparameters §3.3.3.

9. Testing. One forward pass per image. The output is the fused map, and no prior knowledge of the image is needed §1 §3.3.3.

# Forward pass at test time
# Input:  image (H, W, 3), RGB
# Output: saliency map (H, W), values in [0, 1]
# [§3.2] [§3.3.3] [code: utils/fixationmap_detect.m#L3-L31]

def predict_saliency(image, encoder, decoders, classifiers, fuse):
    # Subtract the mean image at the original size
    # [code: utils/prep_im_for_blob.m#L6]
    centred = image - mean_image
    # Resize to the fixed network input size. Table 3 says 256 for the
    # largest side; see Pitfalls. [code: utils/prep_im_for_blob.m#L14]
    x = resize(centred, (224, 224))
    # Reorder channels from RGB to BGR for Caffe
    # [code: utils/fixationmap_detect.m#L6]
    x = x[:, :, [2, 1, 0]]
    # Encoder: VGG16 convolutions up to conv3-3, conv4-3 and conv5-3 [§3.3.1]
    feats3, feats4, feats5 = encoder(x)
    # Each side branch: deconvolutions up to the input size [§3.3.2] [eq. 3]
    side_logits = []
    for feats, depth in [(feats3, 2), (feats4, 3), (feats5, 4)]:
        up = feats
        for step in range(depth):
            # Kernel 4, stride 2, then ReLU
            # [code: models/attention_test.prototxt#L389-L392]
            up = relu(deconv(up, kernel=4, stride=2))
        # One convolution gives one logit map [§3.2]
        # [code: models/attention_test.prototxt#L570-L572]
        logit = conv3x3(up)
        # Crop to the input size [code: models/attention_test.prototxt#L584]
        side_logits.append(crop_to_input(logit, x))
    # Fuse the side logits with a learned convolution [§3.2]. The code
    # uses a 3x3 kernel here; see Paper vs code.
    # [code: models/attention_test.prototxt#L845-L876]
    fused_logit = conv3x3(concat(side_logits), padding=1)
    # Sigmoid gives the saliency probability
    # [code: models/attention_test.prototxt#L898-L903]
    final = sigmoid(fused_logit)
    # Divide by the maximum so the map peaks at 1
    # [code: utils/fixationmap_detect.m#L30-L31]
    final = final / max(final)
    # Resize back to the original image size [code: deep_attention_test.m#L53]
    return resize(final, (H, W))

# Training objective
# Input:  side_logits (list of M maps), fused_logit (map), gt (map in [0, 1])
# Output: scalar loss
# [§3.2] [eq. 7] [eq. 8] [eq. 9]

def training_loss(side_logits, fused_logit, gt):
    # Pixel-wise cross-entropy for each side output, summed over pixels
    # [eq. 7]
    side_losses = [bce_sum(logit, gt) for logit in side_logits]
    # Cross-entropy for the fused output, summed over pixels [eq. 8]
    fuse_loss = bce_sum(fused_logit, gt)
    # Side losses are averaged by M. The code weights every term by 1;
    # see Paper vs code. [eq. 9]
    M = len(side_logits)
    return sum(side_losses) / M + fuse_loss

Hyperparameters

Results

Author-reported. The model is trained on SALICON only §3.3.3 and evaluated on five eye-tracking datasets against 13 baselines: six classical models (ITTI, GBVS, Judd, BMS, CAS, AIM) and seven deep models (DeeFix, SALICON, Mr-CNN, SalNet, DeepGaze I, eDN, SU) §4.3.

Datasets

Ground-truth maps are fixations blurred with a Gaussian whose parameters the dataset authors set §4.1. DUT-OMRON fixations are cleaned of outliers §4.1.

Metrics

• EMD (lower is better): minimum cost to move the predicted distribution onto the ground-truth map. The paper gives no formula §4.2.
• NSS (higher is better): the mean of the zero-mean, unit-variance saliency map at fixation locations eq. 10.
• CC (higher is better): linear correlation between the predicted map and the continuous ground truth eq. 11.
• SIM (higher is better): sum of per-pixel minima after each map is normalised to sum to 1 eq. 12.
• AUC-Judd, AUC-Borji (higher is better): ROC area with fixations as positives and uniform random points as negatives §4.2.
• s-AUC (higher is better): as AUC, but negatives are fixations from other images §4.2.

Table 3: model characteristics (author-reported runtime)

Runtime measured on TORONTO §4.3 tab. 3. Values are as printed; the paper marks some as not released.

Table 4: MIT300

Source: tab. 4. In the paper, DeeFix is bold on every MIT300 column, SALICON on three, and DVA on none. The paper says DeeFix uses a more complex network and a centre prior, and is trained on more images §4.3.

Table 5: MIT1003

Source: tab. 5. * DeeFix is fine-tuned on 900 MIT1003 images and evaluated on the other 103 tab. 5. Not comparable with the other rows.

Table 6: TORONTO

Source: tab. 6. Bold as printed in the paper.

Table 7: PASCAL-S

Source: tab. 7. The BMS and eDN s-AUC values (1.32, 1.29) are above 1, which the metric cannot produce; they are printed that way in the paper and are reproduced unchanged here.

Table 8: DUT-OMRON

Source: tab. 8. Bold as printed in the paper.

Table 9: ablation on TORONTO

Source: tab. 9 §4.4. The Δ columns are as printed.

Qualitative results

The paper shows example maps on MIT1003, TORONTO, PASCAL-S and DUT-OMRON and says the model follows colour contrast as well as faces, humans and text, and that its maps are more localised than those of the other methods §4.3 fig. 4.

Paper vs code

The repository is https://github.com/wenguanwang/deepattention at code_commit d66e2db4a9dc0ec5ebc4eb275e3199f9a59f6752. It belongs to the first author's GitHub account and is linked from the abstract §1. The repository holds a MATLAB and Caffe test pipeline, network definitions, a model file and a training configuration. It does not contain a solver file or any training script, so training settings can only be checked where they appear in the prototxt and config files.

• Fusion operates on logits, with a different kernel. The paper fuses the side maps S^m with a 1×1 convolution and writes F as a weighted sum of maps §3.2 eq. 8. The code concatenates the three pre-sigmoid side logits and applies a 3×3 convolution with padding 1 before the sigmoid attention_test.prototxt attention_test.prototxt.
• Side classifiers use a 3×3 kernel without padding. The paper says 1×1 with sigmoid §3.2. The code uses a 3×3 convolution with padding 0 attention_test.prototxt. Crop layers then align the output with the input attention_test.prototxt.
• Loss weights differ. The paper weights the side losses by 1/M eq. 9. Every loss layer in the training prototxt has loss_weight: 1 train_val.prototxt.
• Decoder channels on the conv4-3 path. The paper gives 256, 128, 64, 32, 1 §3.3.2. The code has three deconvolutions with 256, 128 and 64 channels, followed by a 3×3 convolution to one channel attention_test.prototxt attention_test.prototxt.
• Input size and aspect ratio. The paper's Table 3 gives max(w, h) = 256 tab. 3. The code resizes every image to a fixed 224 × 224, which changes the aspect ratio prep_im_for_blob.m attention_config.m attention_config.m. The size-preserving resize (prep_im_for_blob_size) is commented out prep_im_for_blob.m.
• First convolution padding. The paper uses the standard VGG16 encoder §3.3.1. The code sets padding 35 on conv1_1 attention_test.prototxt. With a 224 px input, conv1_1 then outputs 292 px instead of 224 px, so the encoder feature maps are not the sizes the paper derives §3.3.1. The paper does not mention this.
• Mini-batch. The paper uses 16 images per iteration §3.3.3. The config sets batch_size to 64 and ims_per_batch to 1 attention_config.m. The config does not say whether these are images or regions.
• Deconvolution bias. The code uses bias constant 0.1 attention_test.prototxt. The paper does not state a bias initialisation.
• Output normalisation. Each test map is divided by its maximum after the sigmoid fixationmap_detect.m. The paper does not say this. The reported metrics may depend on it.
• Horizontal-flip augmentation. The config enables it attention_config.m. The paper does not mention augmentation.
• Pretrained model. The README asks the user to download a model named attention_final [README.md#L16-L22]. The repository also includes models/model.mat. This card did not check that the two are the same model, nor which one produces the paper's numbers.
• Caffe version. The Caffe version is in external/caffe.rar, which is an archive and was not extracted. The loss normalisation of Caffe's sigmoid cross-entropy layer is therefore not checked against eq. 7.
• Training code. No solver, no training script and no SALICON data loader is in the repository. The learning rate schedule, momentum and total iterations are not checked.
• Licence. The repository has no LICENSE file [code_license is therefore none]. No code is quoted here, and the card uses only citations.

Pitfalls

• Dataset and training split. The model is trained on SALICON only §3.3.3. Results on the five test datasets are cross-dataset. The paper does not say whether any test image overlaps SALICON.
• Comparisons are not like-for-like. The paper ran ITTI, CAS and AIM with their public code and recommended parameters §4.3. For the other baselines, it does not say whether the numbers were recomputed or copied from earlier papers. DeeFix was fine-tuned on MIT1003 tab. 5, and the SALICON baseline was fine-tuned with a more complex objective and OSIE images §4.3.
• DVA does not win on MIT300. DeeFix and SALICON score higher there §4.3 tab. 4. The state-of-the-art claims rest on the other datasets.
• Metric details are missing. EMD is described in words only, with no formula §4.2. The number of sampled negative points for AUC-Judd, AUC-Borji and s-AUC is not stated §4.2. Centre bias and border cut affect AUC §4.2, and the paper does not say how it handled them.
• Printed values out of range. The BMS and eDN s-AUC values on PASCAL-S are above 1 tab. 7. Treat those cells as unreliable.
• Test-time normalisation. Dividing by the maximum changes the map's absolute values. SIM and EMD treat maps as distributions, so their scores depend on this step. The paper does not say how each metric normalised its inputs.
• Fixed square input. Resizing to 224 × 224 distorts wide or tall images. Output maps are resized back to the original size with MATLAB imresize at its default settings deep_attention_test.m. Whether this is the same as the paper's pipeline is not stated.
• Feature map sizes. The paper's sizes (h/8 at conv4-3, h/16 at conv5-3) assume standard VGG padding §3.3.1 §3.3.2. The padding of 35 in the code changes them (see Paper vs code).
• Crop alignment. Deconvolutions with kernel 4 and stride 2 and no padding produce outputs larger than the input attention_test.prototxt. The Crop layers trim them to the input size attention_test.prototxt. The paper does not describe the alignment, and a re-implementation has to pick one.
• Ablation is on TORONTO only. All ablation numbers come from one dataset, with s-AUC and CC only §4.4 tab. 9.
• Inference time is for one setup. The 0.1 s figure is for a TITAN X GPU with 32 GB RAM §3.3.3. Other hardware and CPU timings are not reported.
• Figure-level content. Figures 1 to 4 are described here only in words. No image is included.

Lineage

• Builds on: VGG16 for the encoder §3.3.1 (arXiv:1409.1556, linked from the paper's bibliography). The skip-layer and multi-scale side-output design follows holistically-nested edge detection and deeply-supervised nets §3.2 §2.2; the paper gives no arXiv identifiers for those two, so none are linked.
• Compared against: ITTI, GBVS, Judd model, BMS, CAS, AIM, DeeFix (arXiv:1510.02927), SALICON, Mr-CNN, SalNet, DeepGaze I, eDN and SU §4.3. DeepGaze II (arXiv:1610.01563) is discussed in related work §2.1 but is not in the comparison tables.
• Superseded by: not known from content in this wiki.