End-to-End Video Instance Segmentation with Transformers

In plain words

Video instance segmentation means finding each object of interest in a short video, outlining it on every frame, and keeping the same label on the same object over time. Most systems solve this in several stages. They detect objects frame by frame and then link them across frames with hand-written rules, or they group pixels by similarity in extra passes. Those steps are slow and easy to break.

VisTR handles the whole clip in one go. A Transformer looks at every frame of the clip at once. A fixed set of learned "slots" each produces one object for every frame, in frame order. Because a slot always describes the same object across frames, tracking comes out of the design rather than from a separate linking step. Training fits the whole sequence of one slot to one ground-truth object as a unit.

The authors report that this gives a simpler pipeline with no post-processing to link instances, and that it is the fastest VIS model and the most accurate single-model method on the YouTube-VIS benchmark.

TL;DR

VisTR casts video instance segmentation as direct sequence prediction: a CNN and a Transformer encoder-decoder over a 36-frame clip emit 10 instance sequences of masks in one pass §3.1 §4.1. Hungarian matching supervises each sequence as a whole, and 3D convolutions refine the masks over time §3.2 §3.3. Author-reported: 40.1 mask AP on YouTube-VIS validation with ResNet-101, at 27.7 FPS with data loading and 57.7 FPS without tab. 2. Several loss weights and inference details exist only in the released code, so read the Paper vs code section before reimplementing.

Problem

Top-down methods follow tracking-by-detection. They run image-level instance segmentation and link instances with hand-designed rules §1. Bottom-up methods cluster learned pixel embeddings, often in several iterative steps, which the authors say makes them slow §1. Both depend on the quality of dense per-frame prediction §1. The paper aims for a simple, end-to-end trainable VIS model in which instance order is kept by construction §1 §3.2.

Assumptions: the default design uses a fixed clip length T = 36, which the authors set to the longest YouTube-VIS video §4.1. The paper does not describe how videos longer than T are handled at test time (see Pitfalls).

Method

VisTR has four parts: a CNN backbone, an encoder-decoder Transformer, an instance sequence matching module that supervises the output, and an instance sequence segmentation module that produces the masks fig. 2 §3.1.

Backbone and clip features §3.1

1. A standard CNN produces a low-resolution map for each frame. The per-frame maps are stacked in frame order into a clip feature map of T frames §3.1.
2. A 1×1 convolution reduces the channel count from C to d, where d < C §3.1.

Transformer encoder §3.1 3. The time and space dimensions are flattened into one sequence of length T·H·W, keeping the input frame order §3.1. 4. A fixed 3D sine/cosine positional encoding is added. Each of the three axes (time, height, width) gets d/3 channels, so d must be divisible by 3 §3.1 eq. 1. 5. Each encoder layer is a standard self-attention block followed by a feed-forward network §3.1.

Transformer decoder §3.1 6. The decoder takes N = n·T learned instance queries, with n = 10 instances per frame. Each query is a separate learned embedding and has the same width as the pixel features. The decoder outputs N instance features O in one pass §3.1 §4.1. Predictions follow the input frame order, and the instance order is the same in every frame, so slot k of each frame is one instance sequence §3.1.

Box and class heads §3.2 7. A 3-layer feed-forward network with ReLU predicts each box as normalised centre, height and width. A linear layer with softmax predicts the class, with an extra background class for "no object". This follows DETR §3.2.

Instance sequence matching §3.2 8. Each instance sequence is the k-th slot of every frame. Its cost against a ground-truth sequence combines the class probability and the box loss, computed over all frames §3.2 eq. 6. 9. Assignment is one-to-one, found with the Hungarian algorithm over the sequence-level cost §3.2 eq. 2.

Hungarian loss §3.2 10. For each matched pair the loss sums a class negative log-likelihood, a box loss and a mask loss §3.2 eq. 7. 11. The box loss is an average over T frames of a weighted L1 term plus a weighted generalised-IoU term, normalised by the number of instances in the batch §3.2 eq. 8.

Instance sequence segmentation §3.3 12. For each frame, the object predictions O and the encoder features E of that frame go through self-attention to form an attention map. The map is fused with the backbone features B and the encoder features E of the same frame. The last fusion layer is a deformable convolution. This gives a mask feature per instance per frame §3.3. 13. For each instance, the T frame mask features are stacked into a sequence of channel count a and spatial size H₀/4 × W₀/4 §3.3. 14. Three 3D convolution layers with GroupNorm and ReLU then produce a mask sequence. The last layer has no normalisation or activation and outputs one channel §3.3. 15. The mask loss is λ_mask times the average over T frames of a Dice term plus a focal term §3.3 eq. 9.

# Training step for one clip of T = 36 frames, from [§3.1] to [§3.3]
# Input:  clip (T, 3, H, W); targets: instance sequences, each with a
#         class, T boxes and T masks (absent frames flagged)
# Output: scalar loss

def training_step(clip, targets):
    # Per-frame CNN features, stacked in frame order  [§3.1]
    feats = backbone(clip)                          # (T, C, h, w)
    # 1x1 conv to width d = 384, then flatten time and space  [§3.1]
    tokens = conv1x1(feats)                         # (T, d, h, w)
    memory = tokens.reshape(T * h * w, d)           # (T*h*w, d)
    # Fixed 3D sine/cosine positions, added to each attention  [§3.1]
    pos = positional_encoding(T, h, w)              # (T*h*w, d)
    memory = encoder(memory, pos)                   # 6 layers  [§4.1]
    # N = 10 * T learned queries, decoded in one pass  [§3.1] [§4.1]
    slots = decoder(queries, memory, pos)           # (N, d)
    logits = class_head(slots)                      # (N, C + 1)
    boxes = box_head(slots).sigmoid()               # (N, 4)  [§3.2]
    # Mask features per frame, then 3D convs over time  [§3.3]
    frame_feats = mask_features(slots, memory, feats)
    masks = sequence_mask_head(frame_feats)         # (n, T, h/4, w/4)
    # Instance sequence k is slot k of every frame  [§3.2]
    seq_probs = softmax(logits).reshape(T, n, C + 1)
    seq_probs = seq_probs.transpose(1, 0, 2)        # (n, T, C + 1)
    seq_boxes = boxes.reshape(T, n, 4).transpose(1, 0, 2)
    # Match cost: class prob, L1 and GIoU, averaged over frames  [§3.2]
    cost = match_cost(seq_probs, seq_boxes, targets)
    # One-to-one assignment of sequences to targets  [§3.2] [eq. 2]
    pairs = hungarian(cost)                         # (sequence, target)
    # Class NLL over all slots; unmatched slots use the background  [§3.2]
    loss = class_nll(seq_probs, pairs, background_weight)
    for seq_index, target in pairs:
        # Box loss: L1 and GIoU, averaged over frames  [§3.2] [eq. 8]
        loss += box_loss(seq_boxes[seq_index], target.boxes)
        # Mask loss: Dice and focal, averaged over frames  [§3.3] [eq. 9]
        loss += mask_loss(masks[seq_index], target.masks)
    return loss

# Inference on one clip of T = 36 frames  [§4.1]
# Input:  clip (T, 3, H, W) with L <= T real frames
# Output: per instance, a score, a category and L masks or None

def predict_instances(clip, L):
    # Class probabilities, background column removed  [§4.1]
    probs = softmax(logits)[:, :-1]                 # (N, C)
    # Back to frame-major layout; keep the first L frames  [§4.1]
    probs = probs.reshape(T, n, C)[:L]              # (L, n, C)
    frame_scores = max(probs, axis=-1)              # (L, n)
    frame_labels = argmax(probs, axis=-1)           # (L, n)
    results = []
    for k in range(n):
        # Skip the instance if its masks are empty in every frame  [§4.1]
        if all(masks[k, :L] <= 0.5):
            continue
        # Instance score is the mean of the per-frame scores  [§4.1]
        score = mean(frame_scores[:, k])
        # Category is the most frequent per-frame label  [§4.1]
        category = most_common(frame_labels[:, k])
        # Frames scoring below 0.001 get no mask  [§4.1]
        masks_out = []
        for t in range(L):
            if frame_scores[t, k] < 0.001:
                masks_out.append(None)
            else:
                # Upsample, sigmoid, then threshold at 0.5  [§4.1]
                mask = upsample(masks[k, t], image_size)
                masks_out.append(sigmoid(mask) > 0.5)
        results.append((score, category, masks_out))
    return results

Key equations, written as in the paper:

\mathrm{PE}(\mathrm{pos}, i) = \begin{cases} \sin(\mathrm{pos}\cdot\omega_k) & i = 2k \\ \cos(\mathrm{pos}\cdot\omega_k) & i = 2k+1 \end{cases}, \qquad \omega_k = \frac{1}{10000^{2k/(d/3)}} eq. 1

\hat\sigma = \arg\min_{\sigma \in S_n} \sum_{i}^{n} \mathcal{L}_{\text{match}}\bigl(y_i, \hat y_{\sigma(i)}\bigr) eq. 2

\mathcal{L}_{\text{match}}\bigl(y_i, \hat y_{\sigma(i)}\bigr) = -\hat p_{\sigma(i)}(c_i) + \mathcal{L}_{\text{box}}\bigl(b_i, \hat b_{\sigma(i)}\bigr) eq. 6

\mathcal{L}_{\text{Hung}}(y, \hat y) = \sum_{i=1}^{N} \Bigl[ -\log \hat p_{\hat\sigma(i)}(c_i) + \mathcal{L}_{\text{box}}\bigl(b_i, \hat b_{\hat\sigma(i)}\bigr) + \mathcal{L}_{\text{mask}}\bigl(m_i, \hat m_{\hat\sigma(i)}\bigr) \Bigr] eq. 7

\mathcal{L}_{\text{box}} = \frac{1}{T} \sum_{t=1}^{T} \Bigl[ \lambda_{\text{iou}}\,\mathcal{L}_{\text{iou}}\bigl(b_{i,t}, \hat b_{\sigma(i),t}\bigr) + \lambda_{\text{L1}}\,\bigl\|b_{i,t} - \hat b_{\sigma(i),t}\bigr\|_1 \Bigr] eq. 8

\mathcal{L}_{\text{mask}} = \lambda_{\text{mask}} \frac{1}{T} \sum_{t=1}^{T} \Bigl[ \mathcal{L}_{\text{Dice}}\bigl(m_{i,t}, \hat m_{\sigma(i),t}\bigr) + \mathcal{L}_{\text{Focal}}\bigl(m_{i,t}, \hat m_{\sigma(i),t}\bigr) \Bigr] eq. 9

Equation numbers are counted in order of appearance in the compiled paper, since the source labels differ. The paper gives no values for λ_iou, λ_L1 or λ_mask. For the matching cost, the paper gives no weights for the three terms of eq. 6.

Hyperparameters

Results

Author-reported. Setup: YouTube-VIS validation set, average precision (AP) and average recall (AR) with video mask IoU, 40 categories §4. Main results use 18 epochs on 8 V100 GPUs §4.1. The ablations train for 10 epochs with ResNet-50 tab. 1 caption. Hardware for the FPS measurement is not stated.

Main results, YouTube-VIS validation tab. 2

For VisTR, FPS is given with and without data loading. Images are loaded serially, which the authors say costs much time tab. 2 caption. For the first three baselines, the numbers are taken from the re-implementations in the MaskTrack R-CNN paper tab. 2 caption.

Ablations, ResNet-50, 10 epochs, YouTube-VIS validation tab. 1

"Default" marks the setting the paper uses for its main model: T = 36, prediction-level queries, time order, positional encoding, and the 3D convolutions §4.1 §4.2.

Paper vs code

Checked against the official repository at code_commit (Apache-2.0). Each difference is cited on both sides.

• Default backbone. The paper says ResNet-50 is the default backbone §4.1. The code's default is ResNet-101 main.py. The README gives both checkpoints.
• What DETR initialises. The paper says only the backbone is initialised from COCO-pretrained DETR §4.1. The code loads the --pretrained_weights checkpoint with strict=False, skipping only the class and query heads main.py. If that checkpoint is the converted DETR model the README describes, the transformer, input projection and box head also start from DETR. A key that does not match is skipped without an error.
• Class-loss normalisation. Eq. 7 is an unnormalised sum over slots §3.2. The code takes a mean of the cross-entropy over all N slots, with background weight 0.1 vistr.py. The box and mask losses divide by the number of instance-frame entries in the batch instead vistr.py vistr.py vistr.py. Implement the two normalisers separately to match the code.
• Auxiliary decoder losses. The paper does not mention them. The code enables them by default, and mask losses are skipped on intermediate decoder layers main.py vistr.py.
• Background class for unmatched slots. The paper's eq. 7 is written for matched targets with c_i ≠ ∅ §3.2. It does not say how unmatched slots are trained. The code trains them toward the background class with weight 0.1 vistr.py main.py.
• Loss weights. The paper's λ terms have no values. The code uses GIoU 2, L1 5, mask 1 and Dice 1 in the loss main.py, and class 1, L1 5 and GIoU 2 in the matcher main.py.
• Instance score threshold. The paper says instances with scores above 0.001 are kept §4.1. The code applies 0.001 per frame, so a low-scoring frame gets no mask, and an instance is dropped only if all its masks are empty inference.py.
• Inference frame size. The paper says inference uses the same scale as training §4.1. The code's inference transform resizes the short side to 300 with no maximum inference.py. Training uses a long side limit of 540 ytvos.py, so frames are not always exactly 300 × 540.
• Long videos at inference. The paper's clip-length ablation relies on matching instances across overlapping clips §4.2. The released inference script takes only the first 36 frames of each video inference.py and loops over all length frames, reading scores with only 36 rows inference.py inference.py. As written, a video longer than 36 frames would index past the array. The paper's overlap matching is not in the released inference code.
• Positional encoding. The paper gives a fixed sinusoidal encoding §3.1. The code normalises each coordinate by the clip extent and scales it to [0, 2π] position_encoding.py position_encoding.py.
• Three 3D convolutions. The paper describes three 3D convolution layers §3.3. The code has three 3×3×3 dilated convolutions with GroupNorm and ReLU, then a 1×1×1 convolution to one channel segmentation.py.

Checked and consistent: the matching structure (class probability plus box terms, then Hungarian assignment), the feature sizes (d = 384, 6 + 6 layers, 8 heads, 360 queries), the optimiser, and the learning-rate schedule main.py main.py matcher.py transformer.py.

Source of the code: the abstract links git.io/VisTR. The TeX source comments say this is an alias of github.com/Epiphqny/VisTR, which is the repository cloned for this card. No other repository was checked.

Pitfalls

• Missing frames. Some instances lack a box or mask in some frames. The loader sets those frames to label 0 with a zero box and flags them invalid ytvos.py. The matcher's class and box costs still include those frames matcher.py, and the assignment skips them matcher.py. The classification loss then trains the matched slot toward background on the invalid frames vistr.py.
• Loss normaliser. The normaliser is the number of instance-frame entries, counting invalid frames too vistr.py ytvos.py. The paper says the normaliser is the number of instances §3.2. The two agree only when every frame of every instance is annotated.
• Hard-coded slot count. The matcher assumes 10 slots per frame and frame-major query order matcher.py. Changing num_queries requires changing this line too.
• Training clip sampling is unspecified in the paper. The loader builds the clip for frame f from indices f − 35 through f ytvos.py. For f < 35 these indices are negative, so Python indexing wraps to the end of the video. Videos shorter than 36 frames repeat frames through modulo indexing ytvos.py. Reproduce this only if you want the same early-frame behaviour.
• Silent mask-loss fallback. If the mask interpolation fails, the code catches the exception and uses the prediction itself as the target vistr.py. That trains the mask head against its own output, with no error raised.
• Positional encoding depends on clip size. Because positions are normalised by the clip extent position_encoding.py, the encoding changes with the input resolution. Use a fixed clip size, or reproduce the normalisation.
• Speed. The paper reports two FPS figures: 27.7 with data loading and 57.7 without, both for ResNet-101 tab. 2. Images are loaded serially, and the authors say parallel loading would be faster tab. 2 caption. The hardware for the measurement is not stated.
• Ablations are not the main results. The ablations run 10 epochs with ResNet-50 tab. 1 caption. The main table uses 18 epochs §4.1. Do not compare ablation rows with the main table.
• The "best single model" claim. In the same table, MaskProp with ResNet-101 reaches 42.5 AP, above VisTR's 40.1 tab. 2. The paper argues that MaskProp combines several networks and post-processing, so it is excluded from the single-model comparison §4.3. This is the authors' argument, not something the table settles.
• Encoded-feature ablation. CNN features give higher AP50 (54.5 against 53.4) but lower AP and AP75 than Transformer-encoded features tab. 1. The paper's conclusion rests on the overall AP.
• Larger datasets. The paper says VisTR "can achieve even better results" on a larger dataset §4.2. This is a conjecture with no experiment behind it.
• Mask resolution. Mask predictions are at a quarter of the input resolution §3.3. The loss upsamples them bilinearly to the target size vistr.py, and inference upsamples them the same way inference.py.

Lineage

• Builds on: DETR (arXiv:2005.12872) for object queries, the Hungarian matching loss and the default hyperparameters §3.1 §3.2 §4.1.
• Builds on: the Transformer, used with parallel decoding instead of autoregressive decoding §1 §3.1.
• Uses, citing the original works: deformable convolution, Dice loss, focal loss and GroupNorm §3.3.
• Compared against: DeepSORT, FEELVOS, OSMN, MaskTrack R-CNN, STEm-Seg and MaskProp tab. 2.
• Superseded by: not stated in this wiki.