Dynamic Multimodal Instance Segmentation Guided by Natural Language Queries

In plain words

The task is to outline one object in a photo, given a sentence such as "the man on the right". Unlike classic instance segmentation, there is no fixed list of object categories: the sentence decides which object is meant, so the model has to read the sentence and match it against what it sees.

Earlier approaches did this in two broad ways. Some glued the sentence's features onto the image features and then ran ordinary convolutions. Others turned the sentence into a small filter that is slid over the image, where a strong response marks the object. This method uses both ideas. A reader for the sentence, one word at a time, produces filters that scan the image. A second stage combines those filter responses with the image and with the words, again word by word, into one low-resolution map. A final stage reuses early image features to sharpen that map back into a detailed outline.

The point is that the sentence is read in order, and the fine detail of the outline comes from the image's own intermediate features rather than from the low-resolution map alone.

TL;DR

• Referring-expression segmentation: output one mask for the object a sentence describes.
• Architecture: DPN92 image encoder, SRU word reader that emits dynamic 1×1 filters, a recurrent multimodal SRU over the words, and a U-Net-style upsampler with skip connections.
• Author-reported: beats prior methods on 6 of 8 splits across ReferIt, UNC, UNC+ and GRef §5.2 tab. 2. It does not beat them on ReferIt test or UNC testB.
• The official code differs from the paper in several hyperparameters and in the upsampler and recurrence details; see Paper vs code.

Problem

Segmenting an object from a free-form expression needs the sentence and the image combined in one model. CNNs alone do not model sequences, and the cornerstone is combining information from two different domains §1. Prior work merges by concatenation, uses recursion, or maps the sentence to a filter §2. Those methods, as the paper argues, either merge too simply, upsample coarsely with post-processing, or do not use sequential structure in the language fully §2, "Our approach". Deconvolution upsampling is also named as a source of checkerboard artifacts §2.

Method

Overview: the input is an image and a query of T words §3.1. The modules are:

1. Visual Module (VM). A DPN92 backbone produces feature maps at scales 1/2^n for n = 1..N, with N = 5, so I_5 is at 1/32 of the image §3.2 eq. 1.
2. Language Module (LM). Each word gets an embedding e_t. An SRU reads the embeddings to give a hidden state h_t §3.3. The paper's SRU is the standard one with g set to sigmoid §3.3 eq. 2–6. Per word, r_t = [e_t, h_t] §3.3, and K = 10 dynamic filters are computed from r_t with a sigmoid §3.3 eq. 7 eq. 8.
3. Synthesis Module (SM). The filters respond over I_N. For each word t, a 1×1 convolution merges I_N, the filter responses F_t, the coordinate maps LOC, and r_t into M_t §3.4 eq. 9. A multimodal SRU (described as a 1×1 convolution with SRU gates) runs over M_1..M_T, and the final states form R_N at 1/32 scale §3.4 eq. 10.
4. Upsampling Module (UM). Starting from R_N, at each scale the module concatenates the current map with I_n, applies a 3×3 convolution, then doubles the size by bilinear interpolation. A 1×1 convolution to one channel and a sigmoid give the output §3.5.

Numbered steps for the whole model:

1. Run the VM on the image to get I_1..I_5 §3.2 eq. 1.
2. Run the LM on the words to get r_t (concatenated embedding and hidden state) and K filters per word §3.3 eq. 7 eq. 8.
3. Compute the spatial coordinate maps LOC and concatenate them with I_N §3.4.
4. For each word, compute the filter responses, then M_t §3.4 eq. 9.
5. Run the multimodal SRU over M_1..M_T to get R_N §3.4 eq. 10.
6. Run the UM from R_N back to the top scale using I_4..I_1 as skips §3.5.

Key equations (as written in the paper):

f_{k,t} = \sigma\left(W_{f_{k}} r_{t} + b_{f_{k}}\right), \quad k = 1..K eq. 7

M_{t} = \text{Conv}_{1\times1}([I_{N}, F_{t}, LOC, r_{t}]) eq. 9

R_{N} = \text{mSRU}\left(\{M_{t}\}_{t=1}^{T}\right) eq. 10

The SRU update, as the paper writes it (g = sigmoid):

# One SRU step  [§3.3] [eq. 2-6]
# x_t: input for word t; c_prev: previous cell state

def sru_step(x_t, c_prev):
    # Candidate, forget gate and reset gate  [eq. 2-4]
    x_tilde = W * x_t
    forget = sigmoid(W_f * x_t + b_f)
    reset = sigmoid(W_r * x_t + b_r)
    # Cell update mixes the old state and the candidate  [eq. 5]
    c_t = forget * c_prev + (1 - forget) * x_tilde
    # Output mixes the squashed cell and the raw input  [eq. 6]
    h_t = reset * sigmoid(c_t) + (1 - reset) * x_t
    return h_t, c_t

The Language Module, the filter generation for each word:

# Language Module  [§3.3] [eq. 7-8]
# Input: word ids, length T
# Output: r (T, 2000), filters (T, K, 2688)

def language_module(words):
    # Word embedding e_t for each word  [§3.3]
    embeddings = embed(words)                 # (T, 1000)
    # Two-layer SRU over the embeddings gives h_t  [§3.3] [§4.3]
    hidden = sru_stack(embeddings)            # (T, 1000)
    # Enriched word feature r_t = [e_t, h_t]  [§3.3]
    word_feats = concat(embeddings, hidden)   # (T, 2000)
    # K sigmoid-activated dynamic filters per word  [§3.3] [eq. 7]
    filters = sigmoid(linear(word_feats))     # (T, K, 2688)
    return word_feats, filters

The Synthesis Module:

# Synthesis Module  [§3.4] [eq. 9-10]
# Output: state R_N, (1000, H/32, W/32)

def synthesis_module(last, loc, word_feats, filters):
    # One merged map per word, in word order  [§3.4]
    maps = []
    for t in range(T):
        # Response of each dynamic filter over I_N  [§3.4]
        responses = filter_response(last, filters[t])   # (K, H/32, W/32)
        # 1x1 merge; word feature broadcast over all positions  [eq. 9]
        merged = conv1x1(concat([last, responses, loc,
                                 word_feats[t]]))       # M_t
        maps.append(merged)
    # Multimodal SRU over M_1..M_T; final states form R_N  [§3.4] [eq. 10]
    state = msru(maps)
    return state

The Upsampling Module as the paper describes it. The code differs; see Paper vs code.

# Upsampling Module as the paper describes it  [§3.5]
# Input: state R_N at 1/32 scale; feats[n-1] = I_n for n = 1..5
# Output: sigmoid mask at 1/2 scale (R_1)

def upsampling_module(state, feats):
    # Start from the coarsest map  [§3.5]
    current = state                                   # R_N
    # Walk from scale N back to scale 2  [§3.5]
    for n in range(N, 1, -1):
        # Fuse with the visual map of the same scale  [§3.5]
        fused = concat([current, feats[n - 1]])
        # 3x3 convolution, then bilinear x2 upsampling  [§3.5]
        current = upsample2x(conv3x3(fused))          # R_{n-1}
    # 1x1 convolution to one channel, then sigmoid  [§3.5]
    return sigmoid(conv1x1(current))

Training, as the paper states it: two stages, low resolution first (without the UM), then fine-tuning the UM for high resolution §4.3. Loss is not stated in the paper. Evaluation thresholds the sigmoid output and reports overall IoU §4.2.

Hyperparameters

Results

Author-reported. Setup: four datasets (ReferIt, UNC, UNC+, GRef) §4.1; train on the train split, evaluate on the val or test splits §4.2. Metric: overall IoU summed over all images, reported as mIoU §4.2. Baselines are the three methods in tab. 2.

Ablation on UNC testA, at low resolution. Each row removes one component from the full DMN tab. 1:

Source: tab. 1. The paper bolds the best entries per column; the full model is best on mIoU and Pr@0.5, Pr@0.8, Pr@0.9.

Comparison with state of the art, mIoU tab. 2. Blank entries mean the authors did not report that split:

Source: tab. 2. The paper says DMN is best on six of the eight splits: it is not best on ReferIt test or UNC testB §5.2. The text notes that the gains on UNC/UNC+ testB are smaller than on testA §5.2.

Speed and size, UNC testA, low resolution §5.3 fig. 7: with SRUs the LM has 9M trainable parameters and the SM has 10M. Swapping in LSTMs raises these to 24M and 24.2M. The LSTM version trains about three times slower and reaches similar mIoU, with a small edge for SRUs §5.3.

Qualitative results: the network segments fine details such as legs, heads and hands, and handles attributes, location and role §5.4 fig. 8. Typical failures are ambiguous expressions, long sentences where the model focuses on one word, and cases where two similar objects are both segmented §5.4 fig. 9.

Paper vs code

Repository: BCV-Uniandes/query-objseg, commit 9fa3a3a2, MIT licence. Checked: dmn_pytorch/models/dmn.py, dmn_pytorch/train.py, dmn_pytorch/utils/losses.py, dmn_pytorch/utils/transforms.py, dmn_pytorch/utils/word_utils.py, README.md. Not checked: the sru package, which is a dependency pinned in the README and not in the repository, and the dataset loader's split handling beyond the parts cited.

• Language depth. Paper: two-layer SRU §4.3. Code: default 3 train.py; the DMN class default is 2 dmn.py; the README command uses 3 README.md.
• mSRU hidden size. Paper: 1000 §4.3. Code: 1005 train.py. With 1000 in and 1005 out, the highway term in eq. 6 needs a projection that the paper does not describe.
• Word embedding in r_t. Paper: r_t always concatenates e_t and h_t §3.3. Code: concatenation is switched by --mix-we, default off train.py dmn.py; the README turns it on README.md.
• Number of dynamic filters in the class default. Paper: K = 10 §4.3. Code: the DMN class defaults to 1 dmn.py; the train script and README use 10 train.py.
• Filter convolution. Paper: F_t = {f_{k,t} * I_N}, with * not defined further §3.4. Code: each filter is a 1×1 kernel applied per pixel, over I_N concatenated with 8 coordinate channels dmn.py.
• Coordinate maps (LOC). Paper: "spatial coordinates" without a definition §3.4. Code: 8 channels: corners of each cell in [-1, 1], centre, and 1/W and 1/H dmn.py.
• mSRU recurrence. Paper: the mSRU runs over {M_t} and the final hidden states form R_N §3.4. Code flattens the feature grid and the words into one sequence, with words innermost, so the recurrence carries across pixel boundaries dmn.py. The output keeps the last H·W states of that sequence, not one state per location dmn.py. Read from the code, not run.
• Upsampling order. Paper: concatenate, 3×3 convolution, then bilinear ×2 §3.5. Code: concatenate, then bilinear ×2, then the 3×3 convolution, because nn.Sequential(self.up, conv) upsamples first dmn.py dmn.py.
• Upsampling stages. Paper: log2(N) times §3.5, which is not an integer for N = 5. Code: log2(amplification) = 5 stages, reaching full image size from 1/32 dmn.py README.md. The paper's own output scale is 1/2 (R_1) §3.5.
• Upsampling channels. Not stated in paper. Code: 512, 256, 128, 64, 32 dmn.py.
• Output nonlinearity. Paper: sigmoid in the UM §3.5. Code: the model outputs logits; training uses BCE with logits train.py; sigmoid is applied only at evaluation train.py.
• Loss. Not stated in paper. Code: BCE with logits, computed after bilinear resizing of the logits to the mask size train.py; an IoU loss is available behind --iou-loss train.py losses.py.
• Batch and accumulation. Paper: batch size 1 §4.3. Code: the optimiser steps every --accum-iters pairs, default 100 train.py train.py; the README uses 1 README.md.
• Scheduler. Paper: cut the LR by 10 after 2 epochs of loss stagnation §4.3. Code: ReduceLROnPlateau, default factor 0.1, stepped on 1 − validation mIoU when --val is given train.py train.py. Released UNC weights used a constant LR README.md.
• Threshold selection. Paper: optimal thresholds from the train or validation split, respectively §4.2. Code: the threshold maximising mIoU is picked from the split passed to --val train.py train.py. If that split is test, the test numbers use a threshold chosen on test.
• Input size. Not stated in paper. Code: longest side fitted to 512 with no padding; the batch size is 1 so sizes can vary train.py transforms.py.
• Query length. Not stated in paper. Code: no truncation at batch size 1 train.py. Words are lowercased and split on non-word characters word_utils.py.
• Backbone weights. Not stated in paper. Code: ImageNet-pretrained extra weights by default dmn.py dpn_model.py.
• Released numbers. The README reports high-resolution weights with mIoU close to the paper: ReferIt test 0.5281 vs 52.81, UNC testB 0.4520 vs 45.13 README.md README.md. The released training schedule is not the paper's schedule (see the scheduler point).

Pitfalls

• Recurrence in the SM. The paper says the mSRU runs over the words and yields final states per location. The code runs one SRU over pixels and words together and keeps the last H·W outputs dmn.py. An implementer has to choose: follow the paper's per-location recurrence, or reproduce the code. The released weights are trained with the code's behaviour. A shape test on a small input would settle what the code does before you copy it.
• Parameter counts. Paper: LM has 9M trainable parameters §5.3. With the README setting (--mix-we) the LM's filter layer maps 2000 features to K × (2688 + 8) outputs, roughly 54M weights dmn.py README.md. The paper does not say what its 9M counts. Expect a much larger LM than the paper's figure if you use the code's filter layer.
• Memory grows with query length. The code expands the image features once per word and concatenates them, so the fused tensor is about T times the feature map size, with roughly 2700 channels per word dmn.py. Long queries and large images cost a lot of memory. The paper gives no memory figures.
• Compute grows with image size. The mSRU sequence length is the number of feature positions times T dmn.py. Larger inputs mean longer sequences, which the paper does not discuss.
• mIoU is not a per-image average. It is the total intersection over the total union across all images §4.2 train.py train.py. Averaging per image gives different numbers.
• Unstated training details. Loss, input size, epochs, backbone weights and embedding initialisation are not stated. The code's choices are listed in Paper vs code; they are not the paper's.
• Ambiguous upsampling count. "log2(N) times" §3.5 does not describe a working schedule for N = 5. Five doublings from 1/32 are needed to reach full resolution, as the code does dmn.py.
• Gap to the abstract. "Six of eight splits" holds on the paper's table, but the gains are uneven. ReferIt test and UNC testB are not better than the best baseline, and UNC+ testB is ahead by a much smaller margin than UNC+ testA tab. 2 §5.2. Do not read the average gain as uniform across datasets.
• SRU dependency. The code needs the sru package at a pinned commit that is not in the repository README.md. The paper's SRU uses sigmoid for g §3.3; the code passes no activation argument, so the package default applies. That default was not checked.

Lineage

• Builds on: Hu et al. 2016 (segmentation from natural language expressions; the concatenation of language and visual features and the coordinate maps) §2 §3.4. Li et al. 2017 (dynamic filters from language) §2 §3.4. Liu et al. 2017 (recurrent multimodal interaction) §2 §3.4. The paper also cites skip-connection work for upsampling §3.5 and DPN92 §3.2; those papers are not cards here.
• The SRU unit is cited to Lei et al.; the bibliography key is abs-1709-02755 §3.3. No card exists for it yet.
• Compared against: Hu et al. 2016, Li et al. 2017 and Liu et al. 2017 tab. 2.
• Superseded by: not known from this wiki.