---
kind: concept
slug: masked-attention
title: Masked attention
summary: >-
  Attention with a 0 or minus-infinity mask added to its scores, so each query reads only an allowed subset of keys. Cutie splits 16 object queries: the first half reads only predicted foreground pixels, the second half only background. Removing the masks cost 3.5 J&F on MOSE, and foreground-only masking cost 0.6.
tags: [attention, transformer, masking, video-segmentation]
papers: ["2310.12982"]
related: [object-queries, cross-attention, self-attention, attention-mechanism]
status: draft
updated_at: 2026-11-05
---

## Overview

Masked attention switches off some query-key pairs before the softmax. The
mask holds 0 for each allowed pair and minus infinity for each blocked pair.
Added to the scores, it gives blocked keys zero weight after the softmax.
Cutie writes the update as a softmax over the mask plus the query-key scores,
applied to the values, with a residual connection [arxiv:2310.12982 §3.2.2]
[eq. 2]. The mask is a rule about which pixels each query may see, and the
rule can depend on the data. In Cutie it comes from a foreground prediction
made inside the network [arxiv:2310.12982 §3.2.2].

Only one card on this wiki lists this concept, so the page covers one
design. It will grow as more cards are added.

## Variants

### Foreground/background split of object queries (Cutie)

Cutie has 16 object queries [arxiv:2310.12982 §3.4.2]. The first half may
attend only to pixels predicted as foreground. The second half may attend only
to pixels predicted as background [arxiv:2310.12982 §3.2.2] [eq. 3]. The mask
comes from a mask head on the previous pixel features, and the same mask is
shared across all attention heads [arxiv:2310.12982 §3.2.2]. The released code
refreshes the mask after each block except the last, from the updated pixels.
The paper's text does not spell this out, so treat it as a code detail
[arxiv:2310.12982 §3.2.2]. The queries still exchange
information afterwards through unmasked self-attention [arxiv:2310.12982 §3.2.1].
Card: [Putting the Object Back into Video Object Segmentation](/papers/2310.12982).

The paper says this extends a masked form that used only the foreground
[arxiv:2310.12982 §3.2.2]. That earlier work has no card on this wiki, so its
content is not described here.

### Foreground-only masking

All queries read only predicted foreground pixels. Cutie reports this as an
ablation, not as its own model: 66.7 ± 0.21 J&F against 67.3 ± 0.36 for the
split [arxiv:2310.12982 tab. 6].

## When to use which

The evidence is from Cutie's ablations on MOSE validation, using the small
model and MOSE training [arxiv:2310.12982 §4.2] [tab. 6]. It covers video
object segmentation only. No card here tests masked attention on another task.

| Situation | Use | Why |
|---|---|---|
| Queries must separate a target from its surroundings, and the network can predict a foreground mask | Foreground/background split | Without masks the score falls to 63.8 J&F, and the split beats foreground-only masking by 0.6 [arxiv:2310.12982 tab. 6] |
| The same, but you want the simplest mask | Foreground-only masking | 66.7 ± 0.21 against 67.3 ± 0.36 for the split. The gap is small, and the spread is of the same order [arxiv:2310.12982 tab. 6] |
| No predicted mask exists to build the rule from | Masked attention in this form does not apply | The Cutie mask is built from a predicted foreground [arxiv:2310.12982 §3.2.2]. The cards here give no alternative rule |
| Stable results across training runs matter | Use the mask | Reported spread is ± 0.36 with masks and ± 1.06 without [arxiv:2310.12982 tab. 6] |
| Speed matters | Masks cost little in Cutie | 45.5 FPS with the default masking against 46.3 without, in the ablation table [arxiv:2310.12982 tab. 6] |

## Implementation sketch

The sketch follows the Cutie form [arxiv:2310.12982 §3.2.2] [eq. 2] [eq. 3].
Shapes: N queries, C channels, HW pixels. Projections of queries, keys and
values are as in standard attention, and the positional terms are left out.

```pseudocode
# Build the allowed-pair mask for one block  [arxiv:2310.12982 §3.2.2] [eq. 3]
# Input:  predicted foreground probability M (HW,), N queries
# Output: additive mask (N, HW), 0 where allowed, -inf where blocked

def build_mask(M, N):
    allowed = zeros(N, HW, dtype=bool)
    # First half of the queries: pixels predicted as foreground  [eq. 3]
    allowed[: N // 2] = (M >= 0.5)
    # Second half: pixels predicted as background  [eq. 3]
    allowed[N // 2 :] = (M < 0.5)
    # Rows with no allowed pixel are a special case; see the note below
    return where(allowed, 0.0, -inf)

# Masked cross-attention from queries to pixels  [arxiv:2310.12982 eq. 2]
# Input:  queries X (N, C), pixels R (HW, C), additive mask (N, HW)
# Output: X plus the attended values

def masked_cross_attention(X, R, mask):
    # Scores between queries and pixels
    scores = Q(X) @ K(R).T
    # Add the mask, so blocked pixels get zero weight after softmax
    weights = softmax(scores + mask, axis=-1)
    # Read the values and add the residual  [eq. 2]
    return weights @ V(R) + X
```

Notes for implementers:

- Rebuild the mask from the pixels after each block except the last, as the
  released code does. The paper's text does not state this schedule
  [arxiv:2310.12982 §3.2.2].
- If a query has no allowed pixel, its softmax has only -inf inputs and is
  undefined. The released code lifts the mask for such rows. The paper does
  not mention this [arxiv:2310.12982 §3.2.2].
- The paper does not describe how to mask several objects at once. Its
  per-object rule is the 0.5 threshold above [arxiv:2310.12982 eq. 3].

## Contradictions

None between cards: only one card lists this concept. The paper and its
released code do differ on the rule for several objects. The paper's rule for
each object is the 0.5 threshold in eq. 3. The code assigns each pixel to the
object with the largest aggregated score against background. The two agree for
a single object and can differ for several. The paper does not describe the
multi-object rule [arxiv:2310.12982 eq. 3] [§3.2.2].

## Papers

- [Putting the Object Back into Video Object Segmentation](/papers/2310.12982):
  introduces the foreground/background masked attention between 16 object
  queries and pixel features, inside a three-block object transformer.