Video Salient Object Detection via Fully Convolutional Networks

In plain words

Finding the most eye-catching object in each frame of a video is useful for segmenting and summarising footage. Older methods usually worked out motion between frames first, which takes most of their time. This paper trains a convolutional network that looks at two neighbouring frames and a rough still-image saliency map, and outputs the saliency map directly.

The hard part is training data: labelled video is scarce and costly to make. The authors therefore build training videos from annotated still photos by moving parts of each image around, which gives the network motion examples without filming anything. Their model runs at a usable speed and, by their own measurements, is more accurate than the older methods they compare against.

TL;DR

• The model is a fully convolutional network with two parts: a static network that scores one frame, and a dynamic network that takes a frame pair plus that score §3.1.
• The dynamic network does not use optical flow; the paper argues short-term frame pairs are enough for saliency §1 §3.3.
• Training uses synthetic video made from MSRA10K and DUT-OMRON images, plus about 3K real frame pairs §4 §5.1.
• Author-reported MAE is 6.36% on DAVIS and 7.65% on FBMS, at about 2 fps with all steps tab. 2 §5.4.
• The released code is a Keras re-implementation whose network differs from the text in several places; see Paper vs code.

Problem

Pixel-level video saliency needs dense labels. Existing video segmentation datasets are small, and their frames are highly correlated, so they are far smaller than image datasets such as ImageNet tab. 1 §1.

Most earlier video saliency methods spend most of their time on optical flow or other motion computation, which limits their speed §1 §2 §5.4. Deep video models that take optical flow as input inherit that cost §1.

Method

Architecture

The model has two modules. The static module takes one frame and produces a saliency map. The dynamic module takes the frame pair (I_t, I_{t+1}) and the static map P_t, and produces the final spatiotemporal map for frame t §3.1 fig. 1.

Static network

The static network is an FCN: a stack of convolution and pooling layers with total stride S, followed by deconvolution layers that upsample by S back to the input size §3.2 eq. 1 eq. 2 fig. 2. A 1 × 1 convolution with a sigmoid gives a per-pixel probability map §3.2. The figure caption uses a 224 × 224 input fig. 2. The text says the FCN accepts arbitrary input sizes §3.2.

# Static network: one frame in, one saliency map out  [§3.2]
def static_saliency(frame):
    # F_S: convolution and pooling stack, total stride S  [§3.2] [eq. 2]
    features = conv_pool_stack(frame)            # (h/S, w/S, c)
    # D_S: deconvolution layers upsample by S to the input size  [§3.2]
    features = deconv_stack(features)            # (h, w, c')
    # 1x1 convolution and sigmoid give probabilities in [0, 1]  [§3.2]
    return sigmoid(conv_1x1(features))           # (h, w)

Loss

Training minimises a weighted cross-entropy over the pixels of one sample. Salient pixels are weighted by (1 − α) and non-salient pixels by α, where α is the fraction of salient pixels in that sample's ground truth §3.2 eq. 3. The paper gives no epsilon for the logarithms.

# Weighted cross-entropy for one training sample  [§3.2] [eq. 3]
def weighted_loss(pred, gt):
    # alpha: share of salient pixels in this ground-truth map  [§3.2]
    alpha = mean(gt)
    # Positive pixels get weight (1 - alpha), negatives get alpha  [eq. 3]
    positive = (1 - alpha) * gt * log(pred)
    negative = alpha * (1 - gt) * log(1 - pred)
    return -sum(positive + negative)

Initialisation

The convolution layers of the static network start from the first five convolutional blocks of VGG-16, trained on ImageNet. The remaining layers start at random §3.2 §5.1. The dynamic network also starts from VGG, except that its first convolution uses Gaussian initialisation because its input has seven channels §5.1.

Dynamic network

The frame pair and the static map are concatenated along the channel axis into a tensor with seven channels (3 + 3 + 1). The first convolution is changed to sum three convolutions, one per input, plus a bias §3.3 eq. 4. The rest of the network mirrors the static network §3.3 fig. 3. It is trained with the same loss, with the static map P_t as a prior §3.3.

# Dynamic network: frame pair plus static prior in, saliency map out  [§3.3]
def dynamic_saliency(frame, next_frame, static_prior):
    # Stack I_t, I_{t+1} and P_t on the channel axis: 3 + 3 + 1 = 7  [§3.3]
    stacked = concat([frame, next_frame, static_prior], channels)
    # Eq. 4 sums one convolution per input. One convolution over the
    # stacked tensor gives the same result  [§3.3] [eq. 4]
    features = first_conv(stacked)               # (h, w, c)
    # Remaining layers mirror the static network  [§3.3] [fig. 3]
    features = encoder_decoder(features)         # (h, w, c')
    return sigmoid(conv_1x1(features))           # (h, w)

Test-time procedure

For each frame t, the static network gives P_t, and the dynamic network then uses (I_t, I_{t+1}, P_t). The process repeats until the end of the video §3.3. The text does not say what happens to the last frame, which has no successor.

# Run the model over one video  [§3.3]
def run_video(frames):
    outputs = []
    # Frame t needs frame t + 1, so the loop stops one frame early;
    # the text does not say how the last frame is handled  [§3.3]
    for t in range(len(frames) - 1):
        # Static saliency of frame t  [§3.3]
        prior = static_saliency(frames[t])
        # Spatiotemporal saliency of frame t  [§3.3]
        outputs.append(dynamic_saliency(frames[t], frames[t + 1], prior))
    return outputs

Synthetic video data

The dynamic network takes frame pairs, so each training image only needs to become a two-frame video §4. For each image, the authors generate ten simulated frames §4. The synthetic frame is I'(x) = I(x + v(x)), where v is an optical flow field §4 eq. 5 fig. 4.

The flow is built on superpixels. Background superpixels are split into seeds and the rest. Foreground superpixels get one main motion plus per-part variation §4. The flow is then smoothed by minimising a quadratic energy with a unary term that keeps seeds near their initial values and a smoothness term between neighbours §4 eq. 6 eq. 7 eq. 8 fig. 4.

# Build a two-frame training video from one annotated still image  [§4]
def synthesize_pair(image, mask):
    # Over-segment into superpixels, then mark each as foreground or
    # background from the mask  [§4]
    superpixels, is_foreground = split_superpixels(image, mask)
    # Random initial motion for the seeds; all other parts start at zero  [§4]
    initial_flow = random_seed_motion(superpixels, is_foreground)
    # Smooth the flow with the energy in eq. 6, using the weights of eq. 7
    # and eq. 8  [§4] [eq. 6] [eq. 7] [eq. 8]
    flow = smooth_flow(initial_flow, superpixels, is_foreground)
    # Warp the image and its mask to produce the second frame  [§4] [eq. 5]
    second_frame = warp(image, flow)
    second_mask = warp(mask, flow)
    return image, second_frame, mask, second_mask

# Initial vertical motion; the horizontal part is analogous  [§4]
def random_seed_motion(superpixels, is_foreground, height):
    # d = h / 10 bounds the vertical displacement  [§4]
    d = height / 10
    # All background parts start at zero motion  [§4]
    vertical_flow = zeros(len(superpixels))
    # Ten percent of background parts are seeds with random motion  [§4]
    for part in sample_fraction(background(superpixels), 0.10):
        vertical_flow[part] = uniform(-d, d)
    # One main motion m for the foreground, then per-part variation  [§4]
    m = uniform(-d, d)
    for part in foreground(superpixels):
        vertical_flow[part] = uniform(m - d / 10, m + d / 10)
    return vertical_flow

Smoothing weights §4:

# Unary weight lambda_i: seeds and foreground keep their initial motion  [§4]
# eq. 7: 1 for foreground and seed parts, 1e-4 for the rest
# Smoothness weight w_ij between neighbours  [§4] [eq. 8]
# eq. 8: exp(-||C_i - C_j||^2) when both parts are foreground or both
# are background, and 0 otherwise, where C is the mean colour

Hyperparameters

Results

Author-reported. Setup: test set of FBMS (30 sequences) and the whole DAVIS set (50 sequences) §5.1 §5.2. Baselines: MD, MC, TIMP, GAFL, SAGE and RWRV §5.2.

Ablation, author-reported, MAE in %. Deltas are as printed tab. 2.

The text says set ii uses 0.03×10^5 frame pairs §5.3. The counts in the other rows are printed without a unit.

Paper vs code

The official repository is https://github.com/wenguanwang/ViSalientObject. The paper names no code link. The repository is owned by the first author, and its description names this paper. The repository has no LICENSE file. It contains inference code and model definitions only. No training loop, loss, optimiser or synthesis code is in the Python files main.py models.py. The synthesis code is a MATLAB archive, synthesizevideo.rar, which we did not extract README.md. Model weights are not in the repository; the README links external downloads README.md.

• Framework. Paper: Caffe §5.1. Code: Keras, "Keras Implementation (v1)" README.md models.py.
• Static decoder. Paper: deconvolution layers upsample to the input size, then a 1 × 1 sigmoid convolution §3.2 eq. 2. Code: nearest-neighbour UpSampling2D with no learned weights, concatenated with 64-channel sigmoid features, and 3 × 3 sigmoid heads models.py. No Conv2DTranspose is used.
• Output resolution. Code: the last pool is after conv4, not conv5 models.py models.py. The code uses the finest head, predictions[2] main.py. Computed from the layer strides: 224 input gives a 56 × 56 map, not full input size. This is computed from the code, not run. The map is resized afterwards utilities.py.
• Multi-scale heads. Paper: one output, with loss on the final map §3.2 eq. 3. Code: three outputs, sal_conv_5, sal_conv4 and sal_conv3, returned together models.py models.py models.py models.py. Whether training sums the loss over all three heads cannot be checked because the training code is absent.
• Head kernels. Paper: 1 × 1 convolution §3.2 fig. 2. Code: 3 × 3 kernels models.py models.py models.py.
• Input size. Paper: the FCN "operates on input images of arbitrary sizes" §3.2. Code: every frame is resized to 224 × 224, which does not keep the aspect ratio, and the output is resized back main.py main.py.
• Dynamic first layer. Paper: sum of three convolutions eq. 4. Code: concatenate, then one convolution models.py. These are mathematically equivalent.
• Channel means. Config lists the means as [103.939, 116.779, 123.68] config.py. get_test subtracts index 0 from the RGB red channel, index 1 from green and index 2 from blue, and only then reverses to BGR main.py. So red gets 103.939 and blue gets 123.68. The usual BGR-ordered VGG means would give the reverse. The paper states no preprocessing. The effect on accuracy was not measured here.
• Frame-pair alignment. Paper: frame t is paired with t + 1 §3.3. Code: os.listdir results are sliced first, [0:-1] for the first frame and [1:] for the second, and only then sorted and zipped main.py. Directory order is arbitrary, so pairs can be misaligned. The static-map list has the same issue main.py. The last frame of each video is dropped main.py main.py.
• Grayscale bug. get_dynamic_test reuses the variable copy for the second frame. For a grayscale first frame, img1 and copy are the same array, so img1 is overwritten with the second frame main.py. Color inputs are not affected.
• Output normalisation. Code: each map is resized and then divided by its own maximum and scaled to 255 utilities.py. Paper: MAE is computed on maps in [0, 1] §5.2. The code's normalisation affects any metric computed on its raw output.
• Reported numbers. README: MAE 0.0587 on DAVIS and 0.0619 on FBMS, described as "slightly improved" over the paper README.md. Paper: 6.36% and 7.65% tab. 2. The README values are about 0.5 and 1.6 points lower. These are the authors' numbers, not ours.
• Training configuration. Config sets batch size 5 and 12 epochs config.py. The paper gives iteration counts, 100K and 300K §5.1, and no batch size or epochs. No script in the repository uses the config values.

Pitfalls

• Labels are the authors' own. FBMS annotations were relabelled to cover only the main salient objects, and about 500 extra annotations were added §5.1. FBMS scores are therefore not comparable with the original motion-segmentation labels.
• Split. DAVIS is used only for testing. Training uses MSRA10K, DUT-OMRON, SegTrackV2 and the FBMS training set §5.1.
• Synthetic data is not a full substitute. Training on image-derived video alone raises FBMS MAE from 7.65 to 9.27 tab. 2 §5.3. Video only overfits badly (24.5) tab. 2 §5.3.
• Occlusion and borders are not described. The warp I'(x) = I(x + v(x)) eq. 5 does not say how out-of-frame or occluded pixels are filled.
• Warp interpolation for the image and mask is not stated. Mask warping could produce non-binary values unless it is nearest-neighbour.
• Superpixels. The over-segmentation method and the number of superpixels are not stated §4. Colour space for C(r) is not stated.
• Energy solver. Eq. 6 is described as convex and "efficiently solved" §4. The solver, its tolerance and the iteration count are not stated.
• Horizontal motion bound. Only d = h / 10 is given, for vertical motion. The horizontal bound is not stated §4.
• Loss numerics. α is computed per sample §3.2. If a sample has almost no salient pixels, α is near 0 and the negative term is almost unweighted. No clipping or epsilon is stated.
• Frame pairs only. The model sees two frames, not longer context. The paper argues this is enough for saliency §1, but it does not test longer context.
• Cost. Each frame runs the static network and the dynamic network. Training takes about 40 hours on one TITAN X §5.1. Synthesis takes about 3 hours for 150K clips §4.
• Runtime figures vary by setup. The 0.47 s figure is "as little as", for 480p and one platform §5.4. The 2 fps figure is stated without a resolution [abstract].

Lineage

• Builds on: the FCN architecture [long2015fully] and VGG-16 [simonyan2014very], both cited in the paper's bibliography. Neither has a card here.
• Compared against: MD [li2015visual], MC [jiang2013saliency], TIMP [zhou2014time], GAFL [wang2015consistent], SAGE [wang2015saliency] and RWRV [kim2015spatiotemporal] §5.2. The paper compares timing with four of these video methods §5.4.
• Superseded by: not determined from this card. The repository README links other video saliency and segmentation repositories, several by the same first author README.md. None has a card here.