--- kind: concept slug: cross-attention title: Cross-attention summary: >- Cross-attention lets the queries of one stream read the keys and values of another. Seven cards use it: fusing video views (MTV), text-centred sentiment fusion (BBFN), object queries reading masked pixels (Cutie), event-to-image fusion for deblurring (EFNet), text-to-clip attention for moment retrieval (QD-DETR), learned latents reading LiDAR points (VoxSeT) and frozen-feature upsampling (JAFAR). tags: [attention, transformer, multi-view, multimodal, video, feature-fusion, deblurring, moment-retrieval, point-cloud, feature-upsampling] papers: ["2201.04288", "2107.13669", "2310.12982", "2112.00167", "2303.13874", "2203.10314", "2506.11136"] related: [self-attention, vision-transformer, multi-scale-feature-fusion, multimodal-fusion, attention-mechanism, gated-skip-connection, object-queries, channel-attention, feature-upsampling, bev-representation] status: draft updated_at: 2026-11-06 --- ## Overview Cross-attention is attention where the queries come from one feature set and the keys and values come from another. The seven papers on this page use it for seven different jobs: - two views of the same video at different temporal resolutions, where the coarse view asks and the fine view answers [arxiv:2201.04288 §3.3.2] [eq. 5]; - a text stream that reads visual and acoustic streams, in a sentiment model [arxiv:2107.13669 §3.3]; - a small set of learned object queries that read dense pixel features, and pixels that read the queries back [arxiv:2310.12982 §3.2.1]; - an image feature map that reads event-camera features, with attention over channels rather than tokens, in a deblurring network [arxiv:2112.00167 §3.4]; - video clip features that read the words of a query before the encoder, for moment retrieval and highlight detection [arxiv:2303.13874 §3.1]; - a small set of learned latent codes that read the points of each LiDAR voxel, and the points that read the latents back, in a 3D detector [arxiv:2203.10314 §3.2]; - the feature map of a frozen encoder, upsampled to any output size, where the full image supplies the queries and keys [arxiv:2506.11136 §3.1]. In most of these, the query stream keeps its own identity and gains information from the other stream, through a residual connection [arxiv:2201.04288 §3.3.2] [arxiv:2107.13669 §3.3] [arxiv:2310.12982 §3.2.1] [arxiv:2112.00167 §3.4] [arxiv:2303.13874 §3.1]. Two do not follow that pattern. VoxSeT concatenates its input with the decoded features, where the paper's text says residual [arxiv:2203.10314 §3.3] [code: pcdet/models/backbones_3d/vfe/voxset.py#L193]. JAFAR returns the attention output alone, a weighted average of the low-resolution features [arxiv:2506.11136 §3.1]. The papers differ in what they exchange, when, and how the exchange is gated, masked or normalised. Each card reports the result of its own ablation, and the seven tasks do not share a benchmark, so the numbers are not comparable across variants. Joint self-attention over the tokens of all views is the other option the MTV authors weigh. They judge it infeasible for video, because its cost grows with the square of the token count [arxiv:2201.04288 §3.3]. ## Variants ### Cross-view attention (CVA) Used by MTV. Each adjacent pair of views is fused at chosen layers. The coarser view's tokens form the queries. The finer view's tokens, after a projection, form the keys and values. The attention output is added back to the coarser view with a residual connection. The output projection starts at zero, so the cross term adds nothing when training begins [arxiv:2201.04288 §3.3.2] [eq. 4] [eq. 5]. The attention is a softmax over scaled query-key products, applied to the values [eq. 5]. The authors chose CVA for their main experiments, because it performed best among the three options they tried [arxiv:2201.04288 §4.2]. Card: [Multiview Transformers for Video Recognition](/papers/2201.04288). On Kinetics-400 at 4x3 crops, CVA reaches 81.1 top-1 at 195 GFLOPs and 314M parameters for the three-view model [arxiv:2201.04288 tab. 1 (c)]. Each run is single, and the paper reports no spread [arxiv:2201.04288 §4.2]. ### Bottleneck tokens and MLP fusion These are the two alternatives to CVA in the MTV ablation. They are not cross-attention in the strict sense. - **Bottleneck tokens.** A few learned tokens per view are concatenated to that view's tokens and passed on to the next view [arxiv:2201.04288 §3.3.2]. On the same ablation this gives 81.0 at 188 GFLOPs and 306M parameters [arxiv:2201.04288 tab. 1 (c)]. The paper does not state the number of bottleneck tokens for its main models [arxiv:2201.04288 §4.2]. - **MLP fusion.** The other view's tokens are concatenated along the hidden dimension inside the view's MLP block [arxiv:2201.04288 §3.3.2]. This gives 80.6 at 202 GFLOPs and 323M parameters on the same ablation [arxiv:2201.04288 tab. 1 (c)]. ### Gated cross-modal attention Used by BBFN. The text stream is the "master" and reads the visual or acoustic stream. The query comes from the master, and the keys and values come from the other modality, which the paper calls heterogeneous multi-head attention [arxiv:2107.13669 §3.3]. The attention output m is not added directly. It is combined with the master's own input X as a layer-normalised sum with two sigmoid gates: a compound gate on m and a retain gate on X. Both gates come from the two pooled vectors of the layer [arxiv:2107.13669 §3.3]. A feed-forward block with residual connection follows [eq. 4]. Two such modules, text with visual and text with acoustic, run in parallel and share the text input [arxiv:2107.13669 §3.1]. Card: [Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis](/papers/2107.13669). On CMU-MOSEI, the paper's ablation shows a small effect of the gates. Without them the MAE is 0.531 and binary accuracy 85.8, against 0.529 and 86.2 for the full model. Without gates and separator together the MAE is 0.540 [arxiv:2107.13669 tab. 3]. The table gives one value per row. ### Masked cross-attention from object queries Used by Cutie. Learned object queries, 16 in the default model, start from the learned queries plus a projection of the object memory. Each of three blocks runs, in order: masked cross-attention from queries to pixels, self-attention among the queries, a query feed-forward layer, reversed cross-attention that updates the pixels from the queries, and a pixel feed-forward layer with no pixel self-attention [arxiv:2310.12982 §3.2.1]. Full-resolution attention between pixels is ruled out as too costly [arxiv:2310.12982 §3.2]. The mask splits the queries. The first half may attend only to pixels predicted as foreground, and the second half only to background. The prediction comes from the previous pixel features, and the same mask is shared across heads [arxiv:2310.12982 §3.2.2] [eq. 3]. Queries and keys get a positional embedding, but values do not [arxiv:2310.12982 §3.2.3]. Card: [Putting the Object Back into Video Object Segmentation](/papers/2310.12982). The ablations are on MOSE with the small model. The default, with foreground and background masking, scores 67.3 ± 0.36 J&F. Foreground-only masking gives 66.7 ± 0.21. No masked attention gives 63.8 ± 1.06 [arxiv:2310.12982 tab. 6]. Removing the object transformer gives 65.0 ± 0.44 at 56.6 FPS, against 67.3 at 45.5 FPS [arxiv:2310.12982 tab. 4]. The FPS values depend on hardware, so compare them only within one table. ### Channel-wise cross-modal attention (EICA) Used by EFNet, a deblurring network that also receives the events from an event camera. At each fused level, the image features supply the queries and the event features supply the keys and values. Both branches first pass through normalisation and 1×1 convolutions. The attention map is computed over channels, so it is c × c and not over pixels, which the paper says keeps the cost linear in the number of pixels [arxiv:2112.00167 §3.4]. The attention output is added to the image features, and a GELU MLP with its own residual follows [arxiv:2112.00167 §3.4]. Card: [Event-Based Fusion for Motion Deblurring with Cross-modal Attention](/papers/2112.00167). On GoPro with simulated events, the fusion ablation gives 35.31 dB PSNR for the channel-wise attention without the event-mask gate. The best of three simple multi-level fusions, by multiplication, gives 34.86 dB, and addition 34.78 dB [arxiv:2112.00167 tab. 3]. The full model, with the gate, reaches 35.46 dB [arxiv:2112.00167 tab. 3]. The table gives one value per row. ### Text-to-clip cross-attention before the encoder (CATE) Used by QD-DETR, for moment retrieval and highlight detection on video. The first encoder layers are cross-attention layers. Video clip features are the queries, and the query words are the keys and values. Each clip is a softmax-weighted mix of word values, with weights taken over the words only [arxiv:2303.13874 §3.1] [eq. 1]. The paper uses two such layers, followed by two self-attention layers [arxiv:2303.13874 §6]. The words are not passed to the self-attention encoder or the decoder [code: qd_detr/transformer.py#L124-L131], and the paper argues the decoder needs no text [arxiv:2303.13874 §3.4]. Card: [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](/papers/2303.13874). On the QVHighlights ablation, adding the cross-attentive encoder alone raises video-only R1@0.7 from 33.02 to 38.71. Adding negative pairs on top gives 42.12, and the full model gives 46.66 [arxiv:2303.13874 tab. 4]. The paper replaces the cross-attention layers with four self-attention layers. The query-only variant then reaches 45.21 R1@0.7, against 46.66 for the cross-attentive encoder, so the paper attributes the gain to the query conditioning, not to extra depth [arxiv:2303.13874 tab. 5] [§4.3]. The split of the ablation table is unclear, as the Contradictions section explains. ### Latent-code set attention per voxel (VSA) Used by VoxSeT, a LiDAR detector. Points are grouped into voxels. Each voxel has a small set of latent codes, eight per block in the paper's KITTI setup. The latents are the queries of the encoder step, the points give the keys and values, and a softmax runs over the points of each voxel. Each voxel is then summarised by its latents [arxiv:2203.10314 §3.2] [eq. 6] [eq. 7] [eq. 8]. The decoder runs in the other direction: each point queries the latents of its own voxel [arxiv:2203.10314 §3.2] [eq. 10] [eq. 11] [eq. 12]. The paper argues that this keeps cost linear in the number of points, and that no points are dropped or padded, which grouping-based transformers must do [arxiv:2203.10314 §1]. Card: [Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds](/papers/2203.10314). Between the two attention steps, a convolutional feed-forward network mixes neighbouring voxels. Replacing it with a plain feed-forward network lowers KITTI validation car scores sharply, from 88.31 / 79.56 / 77.84 to 70.12 / 69.54 / 54.23 for Easy, Moderate and Hard [arxiv:2203.10314 tab. 5]. The number of latents also matters. Four, eight and sixteen latents give mAP 76.63, 78.25 and 78.74 in the paper's ablation [arxiv:2203.10314 tab. 6]. The table gives one value per row, and the paper does not report spread. ### Frozen-feature upsampling with a cross-attention block (JAFAR) Used by JAFAR, which upsamples the features of any frozen vision encoder to any output grid. The queries come from a small encoder of the guidance image, which is the full image, average-pooled to the requested output grid. The keys come from a second encoder of the same image, pooled to the low-resolution grid and modulated by the low-resolution features through spatial feature transform [arxiv:2506.11136 §3.1] [eq. 2]. Rotary positional encoding is added to both queries and keys [arxiv:2506.11136 §3.1]. The values are the raw low-resolution features, with no learned projection, so every output vector is a weighted average of the input features [arxiv:2506.11136 §3.1] [eq. 1]. In this design the queries and keys come from the image, not from the feature stream, so the definition at the top of this page fits only in the sense that the values come from another source. Training needs no labels. A random factor shrinks each ImageNet image, the frozen encoder gives low-resolution features from the small copy, and the module is trained to predict the high-resolution features of the full image [arxiv:2506.11136 §3.2] [eq. 3]. Card: [JAFAR: Jack up Any Feature at Any Resolution](/papers/2506.11136). The paper's ablation on the key construction, on the VOC, ADE20K and Cityscapes linear probes, places the spatial feature transform above two alternatives. With it, VOC mIoU is 84.44. Without it, it is 83.25. A plain linear projection of the low-resolution features as keys gives 80.02 [arxiv:2506.11136 tab. 5]. Using four heads is at or near the best of the head counts tested, with 84.44 on VOC and 61.47 on Cityscapes, against 83.82 on VOC for eight heads [arxiv:2506.11136 tab. 5]. Removing rotary positional encoding at a 448-pixel output drops VOC from 83.55 to 67.22 [arxiv:2506.11136 app. D.2]. ## When to use which | Situation | Use | Why | |---|---|---| | Two streams of the same input at different resolutions, and the top reported accuracy is the goal | Cross-view attention at a few layers | 81.1 against 81.0 for bottleneck tokens and 80.6 for MLP fusion on Kinetics-400, single runs [arxiv:2201.04288 tab. 1 (c)]. Fusion at layers 5 and 11 gives 81.14, against 81.08 at layer 5 alone [arxiv:2201.04288 tab. 1 (f)]. The gaps are small and the paper reports no spread | | Compute is tight, and a 0.1 point loss is acceptable | Bottleneck tokens | 188 GFLOPs against 195 for CVA, at 0.1 points lower top-1 [arxiv:2201.04288 tab. 1 (c)] | | The per-stream network must behave as before at the start of training | CVA with a zero-initialised output projection | The cross term is zero at initialisation [arxiv:2201.04288 §3.3.2] | | Fusion layer choice is open | Start from the layers the MTV ablation favours | Layers 5 and 11 are the best row in the layer ablation [arxiv:2201.04288 tab. 1 (f)]. The spread is 0.3 points, from single runs | | One modality is judged most informative, and fusion should be text-centred | Gated cross-modal attention | The paper reports the best MAE and accuracy among its baselines on CMU-MOSEI and CMU-MOSI [arxiv:2107.13669 tab. 1]. The gates add 0.002 MAE in its own ablation [arxiv:2107.13669 tab. 3], so the gates are not the main source of the gain. The results are author-reported | | Each query slot must attend to one side of a mask, for example foreground or background | Masked cross-attention with split queries | Without masking the J&F falls from 67.3 to 63.8 [arxiv:2310.12982 tab. 6] | | A degraded image comes with a second, event-based sensor stream, and token-level attention would be too costly | Channel-wise cross-modal attention (EICA) at several levels | 35.31 dB against 34.86 dB for the best simple fusion, on GoPro with simulated events [arxiv:2112.00167 tab. 3]. Real-event results are on REBlur only [arxiv:2112.00167 tab. 2] | | A text query must condition every clip before the decoder, and the decoder should see no text | Text-to-clip cross-attention in the first encoder layers | Video-only R1@0.7 rises from 33.02 to 46.66 in the paper's ablation, against 45.21 for four self-attention layers in the query-only setting [arxiv:2303.13874 tab. 4] [arxiv:2303.13874 tab. 5]. The ablation split is unclear | | Frame rate is the binding constraint in online segmentation | Fewer object-transformer blocks | With one block, 66.0 J&F at 51.1 FPS; with three, 67.4 at 45.5 FPS [arxiv:2310.12982 tab. 3] | | Reproducing released checkpoints | CVA or bottleneck tokens | The released MTV code accepts only these two types [code: scenic/projects/mtv/model.py#L380-L388]. MLP fusion has no code path | | Views of different widths | Use the query view's width for the attention | The MTV paper does not say how to merge views of different widths. The released code's rule is in Contradictions | | Points are unevenly spread, and grouping them into fixed-size clusters would drop or pad points | Latent-code set attention per voxel (VSA) | Single-stage Waymo LEVEL_1 3D mAP of 76.02, against 70.30 for PV-RCNN [arxiv:2203.10314 tab. 1]. Latency is 34 ms and memory 2381 MB on a 2080 Ti, against 48 ms and 6093 MB for SECOND [arxiv:2203.10314 tab. 8]. On KITTI validation car Moderate, SECOND scores 78.62 and VoxSeT 78.48 [arxiv:2203.10314 tab. 2], so the gain is not uniform. Author-reported | | Reproducing the released code, or its two-stage numbers | Check the code's configurations first | The repository has single-stage configurations only [code: tools/cfgs/kitti_models/voxset.yaml#L51-L54] [code: tools/cfgs/waymo_models/centerpoint_voxset_1x.yaml#L33-L38]. The paper's two-stage rows use an RoI head [arxiv:2203.10314 §3.3] that the repository does not contain | | A frozen encoder's coarse feature map must be upsampled to an arbitrary grid, without labels | Cross-attention upsampling with the image as guide (JAFAR) | COCO-Stuff linear-probe mIoU 60.78, against 60.10 for FeatUp and 59.03 for bilinear [arxiv:2506.11136 tab. 1]. Grad-CAM ADCC 73.3 against 64.3 for FeatUp [arxiv:2506.11136 tab. 2]. Author-reported | | Inference speed or training memory is tight, especially at high output resolution | Check the cost before choosing JAFAR | At a 448-pixel output JAFAR takes 94.0 ms against 64.9 ms for FeatUp, and it needs 26.0 GB to train against 7.4 GB [arxiv:2506.11136 app. E]. The cost grows with the size of the guidance grid [arxiv:2506.11136 app. F] | | The full image is not available at inference, for example only stored features | Not JAFAR | The guidance image is an input of the module at inference [arxiv:2506.11136 §3.1] | ## Implementation sketch Two sketches follow. The first is CVA, the variant the MTV paper uses for its main results. The second is masked cross-attention, as in Cutie. The notes below each show where the variants differ. ```pseudocode # Cross-view attention between adjacent views [§3.3.2] [eq. 4] [eq. 5] # Input: tokens[v] of shape (B, N_v, d_v), views ordered coarse to fine # Output: tokens[v] with information from the next finer view added def cross_view_attention(tokens): # Walk adjacent pairs from the finest view back to the coarsest [§3.3.2] for coarse in reversed(range(len(tokens) - 1)): fine = coarse + 1 # Queries from the coarse view, keys and values from the fine view [eq. 5] kv = project(tokens[fine]) # projection to a common width update = attention(tokens[coarse], kv) # Residual add, with an output projection that starts at zero [§3.3.2] tokens[coarse] = tokens[coarse] + zero_init_proj(update) return tokens ``` ```pseudocode # Masked cross-attention, queries read pixels [§3.2.1] [eq. 2] [eq. 3] # Input: queries (N, C), pixels (HW, C), fg (HW,) probability of foreground # Output: queries (N, C) def masked_cross_attention(queries, pixels, fg, q_pe, k_pe): # Foreground pixels for the first half of the queries, background for # the rest, thresholded at 0.5 [eq. 3] is_fg = fg >= 0.5 allowed = concat(repeat(is_fg, N // 2), repeat(~is_fg, N - N // 2)) # Queries and keys carry positional embeddings, values do not [§3.2.3] q = W_q(queries + q_pe) k = W_k(pixels + k_pe) v = W_v(pixels) scores = q @ k.T # the equation in the paper has no scale # Rows with no allowed pixel are unmasked in the code, not in the paper # [code: cutie/model/transformer/object_transformer.py#L203] allowed = allowed | ~any(allowed, axis=1, keepdims=True) # Disallowed pairs get -inf, so softmax gives them zero weight [eq. 2] logits = where(allowed, scores, -inf) # Residual add to the queries [eq. 2] return queries + softmax(logits) @ v ``` Points where variants differ. In CVA, the attention can be replaced by bottleneck tokens or by MLP fusion [arxiv:2201.04288 §3.3.2]. The layers that fuse are a free choice [arxiv:2201.04288 §3.3.2]. The MTV paper sets different fusion layers for its larger models and states none for MTV-B [arxiv:2201.04288 §4.2]. Fusion costs more than its name suggests, because each fusion layer runs one extra attention sub-block on top of the view's usual layer, and the FLOPs in the ablation include this [arxiv:2201.04288 tab. 1]. In gated cross-modal attention, the attention output is scaled by a compound gate and the residual input by a retain gate, and the two are summed before layer normalisation [arxiv:2107.13669 §3.3]. In masked cross-attention, the reversed direction is also used: after the queries have read the pixels, the pixels read the queries back [arxiv:2310.12982 §3.2.1]. Writing a masked cross-attention block without that second step gives the query-side behaviour only. Channel-wise attention (EICA) is not a token-to-token map. Its weights are c × c per head [arxiv:2112.00167 §3.4], so the first sketch's token shapes do not apply to it. CATE, in the released code, is multi-head and wrapped in a residual, a LayerNorm and a feed-forward block [code: qd_detr/transformer.py#L431-L439], so a single-head sketch leaves out parts that the code uses. The VoxSeT block has the shape of its paper's induced set attention, in two steps: a fixed set of latents reads the points, then the points read the latents back [arxiv:2203.10314 §3.2] [eq. 1] [eq. 3]. The latents are learned in the paper, but the released code has no latent parameter (see Contradictions). JAFAR differs from the first sketch in three places: the queries and keys come from encoders of the image, the values are the raw low-resolution features, and the output grid is set by the requested size [arxiv:2506.11136 §3.1]. Its card gives its own pseudocode, which follows the same order. ## Contradictions - **Projection in CVA, paper vs code.** Eq. 4 projects the finer view's tokens with a learned matrix before attention [arxiv:2201.04288 eq. 4]. The released code has no separate projection. The attention width is the query view's width, and heads come from the query view [code: scenic/projects/mtv/model.py#L161-L168]. - **Fusion layer index, paper vs code.** The paper says fusion happens "at the 5th and 11th layers" [arxiv:2201.04288 §4.2]. The released config sets indices 5 and 11, and the loop counts from 0, so the code fuses at the 6th and 12th layers [code: scenic/projects/mtv/model.py#L319-L320]. The paper does not say whether its own indices start at 0 or 1 [arxiv:2201.04288 tab. 1 (f)]. - **Normalisation between view pairs, paper vs code.** The paper fuses adjacent pairs "sequentially" [arxiv:2201.04288 §3.3.2]. The code layer-normalises all views once, before the pair loop, so each pair reads the tensors from before the previous pair's update [code: scenic/projects/mtv/model.py#L148-L179]. With three or more views, the finest view does not reach the coarsest one within one block. - **MLP fusion availability.** The paper reports MLP fusion in its ablation [arxiv:2201.04288 tab. 1 (c)]. The released code does not implement it [code: scenic/projects/mtv/model.py#L380-L388]. - **Merging views of different widths.** The paper does not state how the per-view class tokens are merged when the views have different widths [arxiv:2201.04288 §3.3.3]. The released code concatenates them along the channel axis [code: scenic/projects/mtv/model.py#L527-L541]. This merge sits in the global encoder, not in CVA. - **Layer normalisation in gated cross-modal attention.** The paper normalises after the gated sum and after the feed-forward residual [arxiv:2107.13669 §3.3] [eq. 4]. The released code sets a pre-normalisation layout, so the query, key and value inputs are normalised, and the residual sums are not [code: src/modules/transformer.py#L210] [code: src/modules/transformer.py#L277-L278]. - **Extra projections in Cutie, paper vs code.** The paper takes the initial readout directly into the first block, and sets the initial queries to the sum of the learned queries and the object memory [arxiv:2310.12982 §3.2.1]. The code applies a 1x1 convolution to the readout and a learned linear layer to the object memory first [code: cutie/model/transformer/object_transformer.py#L133-L147]. - **Fully masked rows, paper vs code.** The paper does not handle a query with no allowed pixel. The code lifts the mask for that query [code: cutie/model/transformer/object_transformer.py#L203]. - **EICA formula, paper vs code.** The paper's attention formula is a softmax over Q^T K scaled by sqrt(d_k), with no output projection shown [arxiv:2112.00167 §3.4]. The released code L2-normalises queries and keys, uses a learned per-head temperature, and adds a 1×1 output projection [code: basicsr/models/archs/arch_util.py#L309-L312] [code: basicsr/models/archs/arch_util.py#L292]. - **CATE projections, paper vs code.** The paper's Eq. 1 is a single softmax with sqrt(d) and separate learned matrices for the keys and values [arxiv:2303.13874 §3.1] [eq. 1]. The released code uses an 8-head attention module and takes keys and values from the word tokens inside its own projections [code: qd_detr/transformer.py#L388]. - **QD-DETR ablation split.** The caption of the ablation table says "val split". Its baseline row equals the Moment-DETR test numbers in the main table, and its full-model row equals the query-only QD-DETR row in the encoder ablation [arxiv:2303.13874 tab. 1] [arxiv:2303.13874 tab. 4] [arxiv:2303.13874 tab. 5]. Treat the ablation numbers as unverified for split. - **VoxSeT latent codes, paper vs code.** The paper describes learned latent codes and separate key and value projections [arxiv:2203.10314 §3.1] [arxiv:2203.10314 §3.2]. The released code has no latent parameter. A single linear layer gives one score per latent for each point, and the value is the per-point MLP output [code: pcdet/models/backbones_3d/vfe/voxset.py#L144-L145] [code: pcdet/models/backbones_3d/vfe/voxset.py#L172-L177]. A reimplementation that follows the text is a different network from the released one. - **VoxSeT skip connection, paper vs code.** The paper wraps each VSA module in a residual block [arxiv:2203.10314 §3.3]. The code concatenates the block input with the decoded features, so the block output is twice as wide [code: pcdet/models/backbones_3d/vfe/voxset.py#L193]. - **JAFAR query pooling, paper vs code.** The paper pools the queries only during training, so inference keeps the full output grid [arxiv:2506.11136 §3.1]. The code pools to the requested output size in every call [code: src/upsampler/jafar.py#L71-L74]. An implementer who follows the paper has no defined query grid at inference. - **JAFAR attention scale, paper vs code.** The paper's attention formula scales by sqrt(d) [arxiv:2506.11136 eq. 1]. PyTorch's multi-head attention scales by the per-head dimension, which is 32 with d = 128 in the code [code: src/layers/attentions.py#L14-L26]. The paper does not define d. - **JAFAR extra layers and loss, paper vs code.** The code adds a GroupNorm, a bias-free 3x3 convolution on the query branch, and L2 normalisation of the low-resolution features, none of which the paper describes [code: src/upsampler/jafar.py#L55-L56] [code: src/layers/attentions.py#L43] [code: src/upsampler/jafar.py#L79]. Its training loss is a cosine term plus a mean squared error of min-max-normalised features, not the L2 form of [arxiv:2506.11136 eq. 3] [code: src/loss.py#L127-L145]. - **JAFAR guidance resolution.** Training uses a 224-pixel guidance image [arxiv:2506.11136 §4.1] [code: train.py#L148]. Probing uses the full 448-pixel image [code: evaluation/train_probes.py#L152-L157]. Results at one guidance size do not carry over to the other. - **Reported numbers across the cards.** No two cards report incompatible numbers for the same setup, because each one measures a different task. The MTV figures are single-run accuracies on Kinetics-400, the BBFN figures are author-reported sentiment metrics, the Cutie figures are J&F on MOSE, the EFNet figures are PSNR on GoPro and REBlur, the QD-DETR figures are recall and mAP on QVHighlights, the VoxSeT figures are 3D detection mAP and AP on KITTI and Waymo, and the JAFAR figures are linear-probe mIoU and Grad-CAM ADCC. ## Papers - [Multiview Transformers for Video Recognition](/papers/2201.04288): fuses several tubelet-size views of a video with cross-view attention, bottleneck tokens or MLP fusion, at selected layers, and reports 81.8% top-1 on Kinetics-400 for MTV-B (author-reported). - [Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis](/papers/2107.13669): text-centred cross-modal attention with retain and compound gates, and a per-layer separator, for sentiment and humour benchmarks (author-reported). - [Putting the Object Back into Video Object Segmentation](/papers/2310.12982): learned object queries read masked pixel features and write back to them, with foreground and background masks, in the Cutie segmentation network. - [Event-Based Fusion for Motion Deblurring with Cross-modal Attention](/papers/2112.00167): channel-wise cross-modal attention between image and event features at several U-Net levels, with an event-mask gate between two stages, for deblurring (author-reported PSNR on GoPro and REBlur). - [Query-Dependent Video Representation for Moment Retrieval and Highlight Detection](/papers/2303.13874): text-to-clip cross-attention in the first encoder layers, with negative video-query pairs and a learned saliency token, for moment retrieval and highlight detection on QVHighlights and other benchmarks (author-reported). - [Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds](/papers/2203.10314): per-voxel latent-code set attention with a convolutional feed-forward network, as the backbone of a LiDAR detector on KITTI and Waymo (author-reported). - [JAFAR: Jack up Any Feature at Any Resolution](/papers/2506.11136): a cross-attention upsampler for any frozen encoder, guided by the image and trained without labels, evaluated on linear probes and Grad-CAM (author-reported).