Cross-attention

Overview

Cross-attention is attention where the queries come from one feature set and the keys and values come from another. The seven papers on this page use it for seven different jobs:

• two views of the same video at different temporal resolutions, where the coarse view asks and the fine view answers 2201.04288 §3.3.2 eq. 5;
• a text stream that reads visual and acoustic streams, in a sentiment model 2107.13669 §3.3;
• a small set of learned object queries that read dense pixel features, and pixels that read the queries back 2310.12982 §3.2.1;
• an image feature map that reads event-camera features, with attention over channels rather than tokens, in a deblurring network 2112.00167 §3.4;
• video clip features that read the words of a query before the encoder, for moment retrieval and highlight detection 2303.13874 §3.1;
• a small set of learned latent codes that read the points of each LiDAR voxel, and the points that read the latents back, in a 3D detector 2203.10314 §3.2;
• the feature map of a frozen encoder, upsampled to any output size, where the full image supplies the queries and keys 2506.11136 §3.1.

In most of these, the query stream keeps its own identity and gains information from the other stream, through a residual connection 2201.04288 §3.3.2 2107.13669 §3.3 2310.12982 §3.2.1 2112.00167 §3.4 2303.13874 §3.1. Two do not follow that pattern. VoxSeT concatenates its input with the decoded features, where the paper's text says residual 2203.10314 §3.3 voxset.py. JAFAR returns the attention output alone, a weighted average of the low-resolution features 2506.11136 §3.1. The papers differ in what they exchange, when, and how the exchange is gated, masked or normalised. Each card reports the result of its own ablation, and the seven tasks do not share a benchmark, so the numbers are not comparable across variants.

Joint self-attention over the tokens of all views is the other option the MTV authors weigh. They judge it infeasible for video, because its cost grows with the square of the token count 2201.04288 §3.3.

Variants

Cross-view attention (CVA)

Used by MTV. Each adjacent pair of views is fused at chosen layers. The coarser view's tokens form the queries. The finer view's tokens, after a projection, form the keys and values. The attention output is added back to the coarser view with a residual connection. The output projection starts at zero, so the cross term adds nothing when training begins 2201.04288 §3.3.2 eq. 4 eq. 5. The attention is a softmax over scaled query-key products, applied to the values eq. 5. The authors chose CVA for their main experiments, because it performed best among the three options they tried 2201.04288 §4.2. Card: Multiview Transformers for Video Recognition.

On Kinetics-400 at 4x3 crops, CVA reaches 81.1 top-1 at 195 GFLOPs and 314M parameters for the three-view model 2201.04288 tab. 1 (c). Each run is single, and the paper reports no spread 2201.04288 §4.2.

Bottleneck tokens and MLP fusion

These are the two alternatives to CVA in the MTV ablation. They are not cross-attention in the strict sense.

• Bottleneck tokens. A few learned tokens per view are concatenated to that view's tokens and passed on to the next view 2201.04288 §3.3.2. On the same ablation this gives 81.0 at 188 GFLOPs and 306M parameters 2201.04288 tab. 1 (c). The paper does not state the number of bottleneck tokens for its main models 2201.04288 §4.2.
• MLP fusion. The other view's tokens are concatenated along the hidden dimension inside the view's MLP block 2201.04288 §3.3.2. This gives 80.6 at 202 GFLOPs and 323M parameters on the same ablation 2201.04288 tab. 1 (c).

Gated cross-modal attention

Used by BBFN. The text stream is the "master" and reads the visual or acoustic stream. The query comes from the master, and the keys and values come from the other modality, which the paper calls heterogeneous multi-head attention 2107.13669 §3.3. The attention output m is not added directly. It is combined with the master's own input X as a layer-normalised sum with two sigmoid gates: a compound gate on m and a retain gate on X. Both gates come from the two pooled vectors of the layer 2107.13669 §3.3. A feed-forward block with residual connection follows eq. 4. Two such modules, text with visual and text with acoustic, run in parallel and share the text input 2107.13669 §3.1. Card: Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis.

On CMU-MOSEI, the paper's ablation shows a small effect of the gates. Without them the MAE is 0.531 and binary accuracy 85.8, against 0.529 and 86.2 for the full model. Without gates and separator together the MAE is 0.540 2107.13669 tab. 3. The table gives one value per row.

Masked cross-attention from object queries

Used by Cutie. Learned object queries, 16 in the default model, start from the learned queries plus a projection of the object memory. Each of three blocks runs, in order: masked cross-attention from queries to pixels, self-attention among the queries, a query feed-forward layer, reversed cross-attention that updates the pixels from the queries, and a pixel feed-forward layer with no pixel self-attention 2310.12982 §3.2.1. Full-resolution attention between pixels is ruled out as too costly 2310.12982 §3.2.

The mask splits the queries. The first half may attend only to pixels predicted as foreground, and the second half only to background. The prediction comes from the previous pixel features, and the same mask is shared across heads 2310.12982 §3.2.2 eq. 3. Queries and keys get a positional embedding, but values do not 2310.12982 §3.2.3. Card: Putting the Object Back into Video Object Segmentation.

The ablations are on MOSE with the small model. The default, with foreground and background masking, scores 67.3 ± 0.36 J&F. Foreground-only masking gives 66.7 ± 0.21. No masked attention gives 63.8 ± 1.06 2310.12982 tab. 6. Removing the object transformer gives 65.0 ± 0.44 at 56.6 FPS, against 67.3 at 45.5 FPS 2310.12982 tab. 4. The FPS values depend on hardware, so compare them only within one table.

Channel-wise cross-modal attention (EICA)

Used by EFNet, a deblurring network that also receives the events from an event camera. At each fused level, the image features supply the queries and the event features supply the keys and values. Both branches first pass through normalisation and 1×1 convolutions. The attention map is computed over channels, so it is c × c and not over pixels, which the paper says keeps the cost linear in the number of pixels 2112.00167 §3.4. The attention output is added to the image features, and a GELU MLP with its own residual follows 2112.00167 §3.4. Card: Event-Based Fusion for Motion Deblurring with Cross-modal Attention.

On GoPro with simulated events, the fusion ablation gives 35.31 dB PSNR for the channel-wise attention without the event-mask gate. The best of three simple multi-level fusions, by multiplication, gives 34.86 dB, and addition 34.78 dB 2112.00167 tab. 3. The full model, with the gate, reaches 35.46 dB 2112.00167 tab. 3. The table gives one value per row.

Text-to-clip cross-attention before the encoder (CATE)

Used by QD-DETR, for moment retrieval and highlight detection on video. The first encoder layers are cross-attention layers. Video clip features are the queries, and the query words are the keys and values. Each clip is a softmax-weighted mix of word values, with weights taken over the words only 2303.13874 §3.1 eq. 1. The paper uses two such layers, followed by two self-attention layers 2303.13874 §6. The words are not passed to the self-attention encoder or the decoder transformer.py, and the paper argues the decoder needs no text 2303.13874 §3.4. Card: Query-Dependent Video Representation for Moment Retrieval and Highlight Detection.

On the QVHighlights ablation, adding the cross-attentive encoder alone raises video-only R1@0.7 from 33.02 to 38.71. Adding negative pairs on top gives 42.12, and the full model gives 46.66 2303.13874 tab. 4. The paper replaces the cross-attention layers with four self-attention layers. The query-only variant then reaches 45.21 R1@0.7, against 46.66 for the cross-attentive encoder, so the paper attributes the gain to the query conditioning, not to extra depth 2303.13874 tab. 5 §4.3. The split of the ablation table is unclear, as the Contradictions section explains.

Latent-code set attention per voxel (VSA)

Used by VoxSeT, a LiDAR detector. Points are grouped into voxels. Each voxel has a small set of latent codes, eight per block in the paper's KITTI setup. The latents are the queries of the encoder step, the points give the keys and values, and a softmax runs over the points of each voxel. Each voxel is then summarised by its latents 2203.10314 §3.2 eq. 6 eq. 7 eq. 8. The decoder runs in the other direction: each point queries the latents of its own voxel 2203.10314 §3.2 eq. 10 eq. 11 eq. 12. The paper argues that this keeps cost linear in the number of points, and that no points are dropped or padded, which grouping-based transformers must do 2203.10314 §1. Card: Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds.

Between the two attention steps, a convolutional feed-forward network mixes neighbouring voxels. Replacing it with a plain feed-forward network lowers KITTI validation car scores sharply, from 88.31 / 79.56 / 77.84 to 70.12 / 69.54 / 54.23 for Easy, Moderate and Hard 2203.10314 tab. 5. The number of latents also matters. Four, eight and sixteen latents give mAP 76.63, 78.25 and 78.74 in the paper's ablation 2203.10314 tab. 6. The table gives one value per row, and the paper does not report spread.

Frozen-feature upsampling with a cross-attention block (JAFAR)

Used by JAFAR, which upsamples the features of any frozen vision encoder to any output grid. The queries come from a small encoder of the guidance image, which is the full image, average-pooled to the requested output grid. The keys come from a second encoder of the same image, pooled to the low-resolution grid and modulated by the low-resolution features through spatial feature transform 2506.11136 §3.1 eq. 2. Rotary positional encoding is added to both queries and keys 2506.11136 §3.1. The values are the raw low-resolution features, with no learned projection, so every output vector is a weighted average of the input features 2506.11136 §3.1 eq. 1. In this design the queries and keys come from the image, not from the feature stream, so the definition at the top of this page fits only in the sense that the values come from another source. Training needs no labels. A random factor shrinks each ImageNet image, the frozen encoder gives low-resolution features from the small copy, and the module is trained to predict the high-resolution features of the full image 2506.11136 §3.2 eq. 3. Card: JAFAR: Jack up Any Feature at Any Resolution.

The paper's ablation on the key construction, on the VOC, ADE20K and Cityscapes linear probes, places the spatial feature transform above two alternatives. With it, VOC mIoU is 84.44. Without it, it is 83.25. A plain linear projection of the low-resolution features as keys gives 80.02 2506.11136 tab. 5. Using four heads is at or near the best of the head counts tested, with 84.44 on VOC and 61.47 on Cityscapes, against 83.82 on VOC for eight heads [arxiv:2506.11136 tab. 5]. Removing rotary positional encoding at a 448-pixel output drops VOC from 83.55 to 67.22 2506.11136 app. D.2.

When to use which

Implementation sketch

Two sketches follow. The first is CVA, the variant the MTV paper uses for its main results. The second is masked cross-attention, as in Cutie. The notes below each show where the variants differ.

# Cross-view attention between adjacent views  [§3.3.2] [eq. 4] [eq. 5]
# Input:  tokens[v] of shape (B, N_v, d_v), views ordered coarse to fine
# Output: tokens[v] with information from the next finer view added

def cross_view_attention(tokens):
    # Walk adjacent pairs from the finest view back to the coarsest  [§3.3.2]
    for coarse in reversed(range(len(tokens) - 1)):
        fine = coarse + 1
        # Queries from the coarse view, keys and values from the fine view  [eq. 5]
        kv = project(tokens[fine])                       # projection to a common width
        update = attention(tokens[coarse], kv)
        # Residual add, with an output projection that starts at zero  [§3.3.2]
        tokens[coarse] = tokens[coarse] + zero_init_proj(update)
    return tokens

# Masked cross-attention, queries read pixels  [§3.2.1] [eq. 2] [eq. 3]
# Input:  queries (N, C), pixels (HW, C), fg (HW,) probability of foreground
# Output: queries (N, C)

def masked_cross_attention(queries, pixels, fg, q_pe, k_pe):
    # Foreground pixels for the first half of the queries, background for
    # the rest, thresholded at 0.5  [eq. 3]
    is_fg = fg >= 0.5
    allowed = concat(repeat(is_fg, N // 2), repeat(~is_fg, N - N // 2))
    # Queries and keys carry positional embeddings, values do not  [§3.2.3]
    q = W_q(queries + q_pe)
    k = W_k(pixels + k_pe)
    v = W_v(pixels)
    scores = q @ k.T                  # the equation in the paper has no scale
    # Rows with no allowed pixel are unmasked in the code, not in the paper
    # [code: cutie/model/transformer/object_transformer.py#L203]
    allowed = allowed | ~any(allowed, axis=1, keepdims=True)
    # Disallowed pairs get -inf, so softmax gives them zero weight  [eq. 2]
    logits = where(allowed, scores, -inf)
    # Residual add to the queries  [eq. 2]
    return queries + softmax(logits) @ v

Points where variants differ. In CVA, the attention can be replaced by bottleneck tokens or by MLP fusion 2201.04288 §3.3.2. The layers that fuse are a free choice 2201.04288 §3.3.2. The MTV paper sets different fusion layers for its larger models and states none for MTV-B 2201.04288 §4.2. Fusion costs more than its name suggests, because each fusion layer runs one extra attention sub-block on top of the view's usual layer, and the FLOPs in the ablation include this 2201.04288 tab. 1.

In gated cross-modal attention, the attention output is scaled by a compound gate and the residual input by a retain gate, and the two are summed before layer normalisation 2107.13669 §3.3. In masked cross-attention, the reversed direction is also used: after the queries have read the pixels, the pixels read the queries back 2310.12982 §3.2.1. Writing a masked cross-attention block without that second step gives the query-side behaviour only.

Channel-wise attention (EICA) is not a token-to-token map. Its weights are c × c per head 2112.00167 §3.4, so the first sketch's token shapes do not apply to it. CATE, in the released code, is multi-head and wrapped in a residual, a LayerNorm and a feed-forward block transformer.py, so a single-head sketch leaves out parts that the code uses.

The VoxSeT block has the shape of its paper's induced set attention, in two steps: a fixed set of latents reads the points, then the points read the latents back 2203.10314 §3.2 eq. 1 eq. 3. The latents are learned in the paper, but the released code has no latent parameter (see Contradictions). JAFAR differs from the first sketch in three places: the queries and keys come from encoders of the image, the values are the raw low-resolution features, and the output grid is set by the requested size 2506.11136 §3.1. Its card gives its own pseudocode, which follows the same order.

Contradictions

• Projection in CVA, paper vs code. Eq. 4 projects the finer view's tokens with a learned matrix before attention 2201.04288 eq. 4. The released code has no separate projection. The attention width is the query view's width, and heads come from the query view model.py.
• Fusion layer index, paper vs code. The paper says fusion happens "at the 5th and 11th layers" 2201.04288 §4.2. The released config sets indices 5 and 11, and the loop counts from 0, so the code fuses at the 6th and 12th layers model.py. The paper does not say whether its own indices start at 0 or 1 2201.04288 tab. 1 (f).
• Normalisation between view pairs, paper vs code. The paper fuses adjacent pairs "sequentially" 2201.04288 §3.3.2. The code layer-normalises all views once, before the pair loop, so each pair reads the tensors from before the previous pair's update model.py. With three or more views, the finest view does not reach the coarsest one within one block.
• MLP fusion availability. The paper reports MLP fusion in its ablation 2201.04288 tab. 1 (c). The released code does not implement it model.py.
• Merging views of different widths. The paper does not state how the per-view class tokens are merged when the views have different widths 2201.04288 §3.3.3. The released code concatenates them along the channel axis model.py. This merge sits in the global encoder, not in CVA.
• Layer normalisation in gated cross-modal attention. The paper normalises after the gated sum and after the feed-forward residual 2107.13669 §3.3 eq. 4. The released code sets a pre-normalisation layout, so the query, key and value inputs are normalised, and the residual sums are not transformer.py transformer.py.
• Extra projections in Cutie, paper vs code. The paper takes the initial readout directly into the first block, and sets the initial queries to the sum of the learned queries and the object memory 2310.12982 §3.2.1. The code applies a 1x1 convolution to the readout and a learned linear layer to the object memory first object_transformer.py.
• Fully masked rows, paper vs code. The paper does not handle a query with no allowed pixel. The code lifts the mask for that query object_transformer.py.
• EICA formula, paper vs code. The paper's attention formula is a softmax over Q^T K scaled by sqrt(d_k), with no output projection shown 2112.00167 §3.4. The released code L2-normalises queries and keys, uses a learned per-head temperature, and adds a 1×1 output projection arch_util.py arch_util.py.
• CATE projections, paper vs code. The paper's Eq. 1 is a single softmax with sqrt(d) and separate learned matrices for the keys and values 2303.13874 §3.1 eq. 1. The released code uses an 8-head attention module and takes keys and values from the word tokens inside its own projections transformer.py.
• QD-DETR ablation split. The caption of the ablation table says "val split". Its baseline row equals the Moment-DETR test numbers in the main table, and its full-model row equals the query-only QD-DETR row in the encoder ablation 2303.13874 tab. 1 2303.13874 tab. 4 2303.13874 tab. 5. Treat the ablation numbers as unverified for split.
• VoxSeT latent codes, paper vs code. The paper describes learned latent codes and separate key and value projections 2203.10314 §3.1 2203.10314 §3.2. The released code has no latent parameter. A single linear layer gives one score per latent for each point, and the value is the per-point MLP output voxset.py voxset.py. A reimplementation that follows the text is a different network from the released one.
• VoxSeT skip connection, paper vs code. The paper wraps each VSA module in a residual block 2203.10314 §3.3. The code concatenates the block input with the decoded features, so the block output is twice as wide voxset.py.
• JAFAR query pooling, paper vs code. The paper pools the queries only during training, so inference keeps the full output grid [arxiv:2506.11136 §3.1]. The code pools to the requested output size in every call jafar.py. An implementer who follows the paper has no defined query grid at inference.
• JAFAR attention scale, paper vs code. The paper's attention formula scales by sqrt(d) 2506.11136 eq. 1. PyTorch's multi-head attention scales by the per-head dimension, which is 32 with d = 128 in the code attentions.py. The paper does not define d.
• JAFAR extra layers and loss, paper vs code. The code adds a GroupNorm, a bias-free 3x3 convolution on the query branch, and L2 normalisation of the low-resolution features, none of which the paper describes jafar.py attentions.py jafar.py. Its training loss is a cosine term plus a mean squared error of min-max-normalised features, not the L2 form of 2506.11136 eq. 3 loss.py.
• JAFAR guidance resolution. Training uses a 224-pixel guidance image 2506.11136 §4.1 train.py. Probing uses the full 448-pixel image train_probes.py. Results at one guidance size do not carry over to the other.
• Reported numbers across the cards. No two cards report incompatible numbers for the same setup, because each one measures a different task. The MTV figures are single-run accuracies on Kinetics-400, the BBFN figures are author-reported sentiment metrics, the Cutie figures are J&F on MOSE, the EFNet figures are PSNR on GoPro and REBlur, the QD-DETR figures are recall and mAP on QVHighlights, the VoxSeT figures are 3D detection mAP and AP on KITTI and Waymo, and the JAFAR figures are linear-probe mIoU and Grad-CAM ADCC.

Papers

• Multiview Transformers for Video Recognition: fuses several tubelet-size views of a video with cross-view attention, bottleneck tokens or MLP fusion, at selected layers, and reports 81.8% top-1 on Kinetics-400 for MTV-B (author-reported).
• Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis: text-centred cross-modal attention with retain and compound gates, and a per-layer separator, for sentiment and humour benchmarks (author-reported).
• Putting the Object Back into Video Object Segmentation: learned object queries read masked pixel features and write back to them, with foreground and background masks, in the Cutie segmentation network.
• Event-Based Fusion for Motion Deblurring with Cross-modal Attention: channel-wise cross-modal attention between image and event features at several U-Net levels, with an event-mask gate between two stages, for deblurring (author-reported PSNR on GoPro and REBlur).
• Query-Dependent Video Representation for Moment Retrieval and Highlight Detection: text-to-clip cross-attention in the first encoder layers, with negative video-query pairs and a learned saliency token, for moment retrieval and highlight detection on QVHighlights and other benchmarks (author-reported).
• Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds: per-voxel latent-code set attention with a convolutional feed-forward network, as the backbone of a LiDAR detector on KITTI and Waymo (author-reported).
• JAFAR: Jack up Any Feature at Any Resolution: a cross-attention upsampler for any frozen encoder, guided by the image and trained without labels, evaluated on linear probes and Grad-CAM (author-reported).