Putting the Object Back into Video Object Segmentation

In plain words

In video object segmentation, you mark an object in the first frame and the model has to trace its outline through the rest of the video. Many earlier methods remember past frames at the pixel level: each new pixel is matched to similar-looking remembered pixels. That works, but other objects that look alike ("distractors") get matched too, so the result gets noisy, especially when objects are crowded or hide each other.

Cutie adds a second, object-level memory. A small set of learned "slots" stands for the target as a whole. The slots look at the noisy pixel-level match, work out which parts belong to the object, and push that understanding back into the image features. Half of the slots only look at the foreground and the other half only at the background, so the object and its surroundings stay separated. A compact summary of the object is also stored in memory, so later frames can use it.

The authors report better accuracy than earlier methods, most clearly on a hard benchmark with occlusion and crowds. These figures are their own.

TL;DR

• Cutie is an online video object segmentation network: a first-frame mask goes in, masks for later frames come out. It keeps a pixel memory (derived from XMem) and a compact object memory of 16 object queries §3.1 §3.4.1 §3.3.
• The new part is object-level readout. Three transformer blocks let object queries read pixel features under foreground/background masked attention and write them back, without self-attention on the high-resolution pixels §3.2 fig. 3.
• Author-reported: on MOSE validation, Cutie-base trained with MOSE reaches 68.3 J&F, against 64.1 for DeAOT-R50 and 59.6 for XMem. On YouTubeVOS-2019 the margin is small tab. 1.
• Ablations (small model, MOSE training): removing the object transformer costs about 2.2 J&F, and removing masked attention costs about 3.5 J&F tab. 3 tab. 6.
• The code is public under MIT, but several details differ from the paper text (pooling masks, positional embeddings, an extra projection, the staggered sensory update, the multi-object masking rule). See Paper vs code.

Problem

Pixel-level matching maps each query pixel independently to a combination of memory pixels. The result lacks high-level consistency and is prone to matching noise when distractors are present. The authors attribute the drop on MOSE, a benchmark with occlusion and distractors, to this missing object-level reasoning §1 fig. 1. Full-resolution attention between pixels is also ruled out, since it is too costly in memory and compute §3.2.

Method

Overview. Cutie encodes each memory frame into a high-resolution pixel memory and a high-level object memory. For a new query frame it first reads the pixel memory by low-level matching, giving an initial readout R_0. The object transformer then enriches R_0 with object-level semantics from the object memory and from a set of object queries. The final readout R_L goes to the decoder §3.1 fig. 2. The paper treats a single target object; multiple objects are handled by running objects in parallel and soft-aggregating their probabilities §3.1 app. G.1.

Object transformer (three blocks). The inputs are R_0 (C x H x W), N learned queries X and the object memory S (N x C). Before the first block the static queries and the dynamic object memory are summed, X_0 = X + S §3.2.1. In each block, in order: (1) queries read from pixels with masked cross-attention; (2) queries run self-attention and a feed-forward layer; (3) pixels are updated by a reversed cross-attention from the queries, which puts object semantics back into the pixel features; (4) a pixel feed-forward layer follows, with no pixel self-attention §3.2.1.

Foreground/background masked attention. The first half of the queries may attend only to pixels predicted as foreground, and the second half only to background. The prediction comes from a mask head on the previous pixel features. The same mask is shared across heads §3.2.2 fig. 3. The attention is masked with the 0/-infinity matrix of eq. 3, and the queries still communicate through self-attention afterwards §3.2.2.

X_l' = \text{softmax}(\mathcal{M}_l + Q_l K_l^T) V_l + X_l eq. 2

\mathcal{M}_l(q,i) = \begin{cases} 0 & \text{if } q \le N/2 \text{ and } M_l(i) \ge 0.5 \\ 0 & \text{if } q > N/2 \text{ and } M_l(i) < 0.5 \\ -\infty & \text{otherwise} \end{cases} eq. 3

Positional embeddings. Queries and keys get a positional embedding but values do not. The query embedding is a learned embedding plus a learned linear projection of the object memory, P_X = E_X + f_ObjEmbed(S) eq. 4. The pixel embedding is a fixed 2D sinusoidal embedding plus a learned projection of R_0, P_R = R_sin + f_PixEmbed(R_0) eq. 5 §3.2.3.

Object memory. The memory is N summaries. Each is a weighted average of object features over all pixels of all memory frames, using pooling weights from the pooling masks. The first N/2 summaries pool over the object and the rest over the background, and the sinusoidal positional embedding enters the pooling-weight input, so pooling is location-aware. The average in eq. 6 is computed as a streaming average at inference, so its cost does not grow with video length §3.3 app. G.2.

S_q = \frac{\sum_{i} U(i)\, W_q(i)}{\sum_{i} W_q(i)} eq. 6

Object features are U = f_ObjFeat(F), a two-layer MLP on the mask-encoder features eq. 7. The pooling weight for pixel i is zero where the mask says the pixel is outside the allowed side of the split, and otherwise sigmoid of a two-layer MLP on F(i) + R_sin(i) eq. 8 §3.3. A summary whose pooling weights are all zero is left unchanged, which stops features drifting when an object is occluded §3.3.

Pixel memory (from XMem; no claimed contribution). Keys and values are stored per memory frame, and a recurrent hidden state is kept. The readout uses an anisotropic L2 affinity with top-k filtering, and the readout fuses the attention output with the hidden state §3.4.1 app. G.5.

A^{pix}_{ij} = \frac{\exp(d(q_i, k_j))}{\sum_m \exp(d(q_i, k_m))} eq. 9

R_0 = f_{fuse}\left(A^{pix} v + h\right) eq. 10

Per-frame procedure. The steps below follow §3.1 and §3.4.4, with the code citations in the comments.

# Segment one query frame  [§3.1] [fig. 2]
# [code: cutie/model/cutie.py#L172-L203]
# Input:  frame (3, H, W), pixel memory, object memory S (N, C)
# Output: probabilities (K + 1, H, W), background channel first

def segment_frame(frame, pixel_memory, object_memory):
    # Query encoder: keys, pixel features, skip connections  [§3.4.2]
    keys, pixel_feats, skips = encode_query(frame)
    # Initial readout R_0 by low-level matching  [eq. 9] [eq. 10]
    readout = pixel_memory.read(keys)
    # Object-level refinement  [§3.2]
    readout = object_transformer(readout, object_memory)
    # Decode per-object logits  [§3.4.2]
    logits = decode(readout, skips)
    # Sigmoid per object, then soft aggregation over objects and background
    # [code: cutie/model/cutie.py#L194-L201]
    probs = aggregate_and_softmax(sigmoid(logits))
    return probs

# Object transformer, L blocks  [§3.2] [eq. 2] [eq. 3] [eq. 4] [eq. 5]
# [code: cutie/model/transformer/object_transformer.py#L114-L167]
# Input:  R_0 (C, H, W), object memory S (N, C), learned queries X (N, C)
# Output: R_L (C, H, W)

def object_transformer(readout_0, object_memory):
    # Queries start from the learned queries plus a projection of S
    # [§3.2.1] [code: cutie/model/transformer/object_transformer.py#L133-L137]
    queries = learned_queries + project(object_memory)
    # Query positional embedding: learned plus projected S  [eq. 4]
    query_pe = learned_query_pe + project_pe(object_memory)
    # Pixel positional embedding: sine plus projected R_0  [eq. 5]
    pixel_pe = sine_pe + project_pixel_pe(readout_0)
    # Pixels start from R_0 after an extra 1x1 projection (code only)
    # [code: cutie/model/transformer/object_transformer.py#L141-L147]
    pixels = project_pixels(readout_0)
    # Fg/bg mask predicted from the current pixels  [§3.2.2] [eq. 3]
    aux_mask = foreground_background_mask(pixels)
    for index, block in enumerate(blocks):
        queries, pixels = query_block(queries, pixels,
                                      query_pe, pixel_pe, aux_mask)
        # Refresh the mask for the next block from the updated pixels
        # [code: cutie/model/transformer/object_transformer.py#L164-L167]
        if index < len(blocks) - 1:
            aux_mask = foreground_background_mask(pixels)
    return pixels

# One query-transformer block  [§3.2.1] [eq. 2]
# [code: cutie/model/transformer/object_transformer.py#L35-L73]
# Input and output: queries (N, C), pixels (C, H, W)

def query_block(queries, pixels, query_pe, pixel_pe, aux_mask):
    # Queries read the pixels under fg/bg masked attention  [eq. 2]
    queries = masked_cross_attention(queries, pixels,
                                     query_pe, pixel_pe, aux_mask)
    # Queries reason among themselves, unmasked  [§3.2.1]
    queries = self_attention(queries, query_pe)
    # Query feed-forward, hidden width 8C = 2048  [§3.4.2]
    queries = feed_forward(queries)
    # Pixels take the object semantics back (reversed cross-attention)
    # [§3.2.1] [code: cutie/model/transformer/object_transformer.py#L60-L64]
    pixels = reversed_cross_attention(pixels, queries, pixel_pe, query_pe)
    # Pixel feed-forward: two 3x3 convs with channel attention, no
    # pixel self-attention  [§3.2.1] [§3.4.2]
    pixels = pixel_feed_forward(pixels)
    return queries, pixels

# Object memory write for one memory frame  [§3.3] [eq. 6] [eq. 7] [eq. 8]
# [code: cutie/model/transformer/object_summarizer.py#L55-L84]
# Input:  predicted mask (1, H0, W0), mask-encoder features F (C, H, W)
# Output: N summaries, each a C-vector plus one area value

def write_object_summaries(mask, features):
    # Bring the mask to feature resolution with area interpolation
    # The paper gates pooling with a hard 0.5 threshold; the code keeps
    # the soft value  [eq. 8] [code: .../object_summarizer.py#L62-L70]
    mask = resize_area(mask, (H, W))
    # First N/2 pooling masks use the mask, the rest use 1 - mask
    pool_masks = concat(repeat(mask, N // 2), repeat(1 - mask, N // 2))
    # Code adds the sine embedding to the input of both MLPs
    # [code: .../object_summarizer.py#L72-L81]
    values = input_proj(features) + sine_pe
    # Object features U = f_ObjFeat(F)  [eq. 7]
    object_feats = feature_mlp(values)
    # Pooling weight: sigmoid of an MLP, times the pool mask  [eq. 8]
    weights = sigmoid(weight_mlp(values)) * pool_masks
    sums = weighted_sum(weights, object_feats)  # (N, C)
    area = weights.sum_over_pixels()  # (N, 1)
    return concat(sums, area)

Reading the object memory uses the accumulated sums divided by the accumulated area, with a small constant added to the denominator eq. 6 object_transformer.py.

Training. Training runs in two stages. Pre-training uses static segmentation datasets with three-frame synthetic sequences and loss on frames two and three. Main training uses video sequences of eight frames. The loss is cross-entropy plus soft dice with equal weight, computed at sampled points (importance sampling), and auxiliary losses at intermediate masks with weight 0.01 §3.4.3 app. G.3 losses.py base.yaml.

Inference. A memory frame is written every r-th frame (r = 5 by default). The first frame is always kept. The other memory frames use first-in-first-out replacement, up to T_max = 5 frames in total. Pixel-memory attention keeps the top 30 entries. Long videos can use XMem's long-term memory without retraining. Inference is online; with the default FIFO memory, the paper says compute and memory per frame stay constant with sequence length §3.4.4 app. E.

Hyperparameters

Architecture and training values come from the paper. Where the paper does not state a value and the code does, the code value is given and labelled.

Results

Author-reported. Setup: each method is re-run on the authors' hardware with its official code where available. FPS is measured on YouTubeVOS with a V100 §4.1. The paper states that three competitors (SimVOS, JointFormer, ISVOS) were not reproducible, so their numbers are quoted as reported tab. S2. J&F is the mean of region (J) and contour (F) accuracy. G is the mean over seen and unseen YouTubeVOS classes §4.

Table 1: common benchmarks tab. 1 (the MEGA rows are from tab. S2).

The paper also reports YouTubeVOS seen and unseen scores. Cutie-base trained without MOSE reaches 85.5 J&F on seen classes and 80.6 on unseen classes tab. 1.

Table 2: long videos, BURST validation and test tab. 2. All methods trained with MOSE. Mem. is peak GPU memory. FIFO is the default memory, INF is unbounded, LT is XMem's long-term memory. Com. and Unc. are common and uncommon object classes.

Ablations (small model, MOSE training, J&F on MOSE validation; FPS on YouTubeVOS-2019 validation) §4.2. Rows with a ± value are means over several seeds: the paper says the baseline was trained five times and selected ablations three times. Other rows use the default seed only.

Table 3: hyperparameters tab. 3. The paper highlights the defaults: L = 3, N = 16, r = 5, T_max = 5.

Table 4: bottom-up versus top-down tab. 4. Mean ± standard deviation over seeds.

Table 5: object memory and object queries tab. 5. Mean ± standard deviation over seeds.

Table 6: masked attention tab. 6. Mean ± standard deviation over seeds.

Table 7: positional embeddings tab. 7. Single runs, no deviation reported.

Supplement tables. Running time per component on a 2080 Ti, in the units the table gives (seconds) tab. S1:

The paper reports LVOS tab. S3. On LVOS validation, Cutie-base without MOSE scores 60.1 J&F, against 60.7 for the LVOS baseline DDMemory; on LVOS test it scores 56.2 against 55.0. The authors say LVOS has many tiny objects, which the method is not designed for app. F.3. Across five seeds with MOSE training, Cutie-small scores a median of 67.3 ± 0.36 on MOSE validation and 86.2 ± 0.11 G on YouTubeVOS-2019 tab. S5.

Qualitative comparisons are described in words app. A and the feature progression through the blocks in words fig. 4 app. D. Limitations are in §4.3 and app. B. The authors name similar objects in close proximity or occluding each other as the main failure case, and say competitors fail the same way app. B.

Paper vs code

The official repository is https://github.com/hkchengrex/Cutie, at commit ec5cdd4cf16f75c73ad785a2f96fb97dbad4125a (2024-11-08). The abstract's project page points to it. The owner is the first author, and the repository is MIT-licensed, copyright Ho Kei Cheng LICENSE. The paper says code will be released app. G; the repository includes training, evaluation and a GUI.

• Pooling gates are soft, not thresholded. The paper uses a hard 0.5 test on the mask to decide where each summary may pool eq. 8 §3.3. The code resizes the mask with area interpolation and multiplies the sigmoid weights by the mask value itself object_summarizer.py object_summarizer.py. At inference the mask is the model's own soft probability, written to memory inference_core.py inference_core.py.
• Positional embedding in the object features. The paper adds the sine embedding only inside the pooling-weight input eq. 8. The code adds it before both the object-feature MLP and the weight MLP object_summarizer.py.
• Multi-object foreground rule. The paper's rule per object is M(i) >= 0.5 eq. 3. The code assigns each pixel to the object whose aggregated log-odds are largest, against background and other objects object_transformer.py cutie.py. For one object the two rules agree, because the aggregated score is above background exactly when the probability is at least 0.5. For several objects they can differ. The paper does not describe the multi-object rule.
• Fully masked rows are unmasked. If a query has no allowed pixel, the code lifts the mask for that query object_transformer.py. The paper does not mention this.
• Query initialisation passes through a projection. The paper sums the static queries and the object memory directly, X_0 = X + S §3.2.1. The code first applies a learned linear layer to the object memory object_transformer.py object_transformer.py.
• Extra projection of the initial readout. The code applies a 1x1 convolution to R_0 before the first block object_transformer.py object_transformer.py. The paper's block description takes R_0 directly.
• Sensory memory updates are staggered. The paper says the recurrent state is updated every frame app. G.5. The code updates it only at a set of offsets inside each memory interval; with r = 5 and stagger_updates = 5 these cover every frame, but with r = 10 (BURST) only five of the ten offsets are used eval_config.yaml inference_core.py inference_core.py. The paper does not mention this.
• Deep updates during training. The paper describes deep updates at memory insertion. The training code applies them to a random 20% of memory frames train_config.yaml train_wrapper.py. At inference they are always used inference_core.py.
• Default checkpoint name is the MEGA model. The default evaluation config points to output/cutie-base-mega.pth eval_config.yaml. The name indicates MEGA training, which the paper reports separately. Check which weights you load before comparing with a given row of the paper.
• Mixed precision in pre-training. The paper says AMP is used for training app. G. The pre-training configuration turns it off train_config.yaml; main training turns it on train_config.yaml.
• Pixel memory details. The paper gives no key dimension, positional embedding scale or temperature; the code uses 64, 32 and 128 base.yaml base.yaml. These values are used but not stated in the paper.

No further discrepancies were found in the learning-rate schedule, optimiser, gradient clipping, loss weights, point counts, frame skip schedule, memory interval, top-k or BURST and LVOS settings. Each matches the paper's stated value train_config.yaml eval_config.yaml.

Pitfalls

• Abstract wording about speed. The abstract says Cutie runs at "similar running time" to XMem. The table gives Cutie-base 36.4 FPS and XMem 22.6 FPS tab. 1. Use the table. The 8.7 J&F gain over XMem in the abstract is Cutie-base against XMem, both trained with MOSE (68.3 against 59.6) tab. 1.
• Speed numbers depend on hardware. The main table uses a V100; the running time breakdown uses a 2080 Ti §4.1 tab. S1. Do not compare them directly.
• Ablation differences are small. Most differences in Tables 3, 5 and 7 are under one J&F point, and the default's standard deviation is 0.36. The bottom-up-only, top-down-only and no-masked-attention settings are clearly lower tab. 4 tab. 6.
• Competitor numbers are partly quoted. Competitors re-run by the authors are on their hardware. SimVOS, JointFormer and ISVOS are copied from their papers, and the authors could not reproduce them tab. 1 tab. S2.
• Object count in training is unstated. The main text trains one target object, and the supplement describes multi-object extension without giving a training object count. The code trains the main stage with up to three objects per sample train_config.yaml.
• MEGA schedule is not in the paper. The paper gives "an additional 50K iterations" app. F.2. The code uses 175K total with learning-rate drops at 140K and 160K mega.yaml. Use the code values if reproducing MEGA.
• Object memory denominator. The code adds 1e-4 to the area before dividing object_transformer.py. The paper's averaging formula has no such term, so use the code's guard if you need the same numerics.
• Long-term memory values. The paper refers to XMem's defaults. The code states the values eval_config.yaml. Use those.
• Cutie+ is a speed trade-off, not a new model. It changes memory size, memory interval and input resolution at inference, with no retraining app. F.1.
• Failure modes. The method struggles when similar objects move close together or occlude each other, and the authors suspect that neither memory gives enough distinctive features in these cases §4.3 app. B.

Lineage

• Builds on: XMem for the pixel memory, sensory memory and long-term memory §3.4.1 app. G.5 app. E; masked attention extended from a prior foreground-only form §3.2.2; object queries from query-based detection and segmentation §1; point-supervised loss with importance sampling §3.4.3 app. G.3; training tricks from DEVA §3.4.3.
• Compared against: STCN, AOT, RDE, XMem, DeAOT, DEVA on common benchmarks; SimVOS, JointFormer and ISVOS in the supplement; DDMemory on LVOS tab. 1 tab. S2 tab. S3.
• Superseded by: not stated in this wiki.