Shifted-window attention

Overview

Full self-attention lets every token look at every other token, which costs more as the number of tokens grows. For a video the token count grows with the number of frames and with the frame size, so global attention is too expensive 2106.13230 §1 2106.13230 §3.2.

Windowed attention fixes the cost by splitting the token grid into non-overlapping windows and letting each token attend only inside its own window 2106.13230 §3.2. The catch is that a plain window grid never lets information cross a window edge. Shifted-window attention fixes this by offsetting the grid in every second block. The windows of that block then straddle the edges of the previous block's windows, so neighbouring regions can exchange information 2106.13230 §3.2.

Four papers on this page use the idea or one of its parts. Video Swin applies it to 3D windows over time, height and width 2106.13230 §3.1. UniMatch applies the 2D form, with windows and a half-window shift, to the feature maps of its matching network for optical flow, stereo and depth 2211.05783 §3.2. FAST-VQA builds on the Video Swin-T backbone for video quality assessment, with a 3D window size per variant 2207.02595 §3.2 2207.02595 tab. 2. GRL uses Swin V2-style window attention for half of its channels in image restoration, and its text states no window shift 2303.00748 §4. The FAST-VQA card does not describe the shift at all. The 2D form itself comes from the Swin Transformer card, which does not list this concept yet, so this page does not draw on that paper.

Variants

Shifted 3D windows in Video Swin

Each block pair has a regular block and a shifted block. The regular block partitions the grid into windows of P×M×M tokens, with P=8 in time and M=7 in height and width by default. The shifted block offsets the grid by half a window in each axis. The paper's formula is (P/2, M/2, M/2), which gives 3.5 for M=7 2106.13230 §3.2 2106.13230 §3.3. See Pitfalls for the integer rounding. Card: Video Swin Transformer.

The paper keeps the batched form of the Swin computation, so the number of windows does not change between the two blocks 2106.13230 §3.2. Each window still attends only inside itself, with a relative position bias 2106.13230 §3.2 2106.13230 eq. 2.

Shift in space only, or in time and space

The paper compares three settings on Swin-T and Kinetics-400 2106.13230 tab. 6:

• Full 3D shift, in time and space: 78.8 top-1, 93.6 top-5.
• Spatial shift only, no temporal shift: 78.5 top-1, 93.5 top-5.
• No shift at all: 78.1 top-1, 93.3 top-5.

The full 3D shift matches the reported default result of 78.8 2106.13230 tab. 6. The gaps are 0.3 and 0.7 points, from single runs 2106.13230 tab. 6.

Two-dimensional windows with a half-window shift in UniMatch

UniMatch uses the 2D form in the Transformer that enhances the features of each image 2211.05783 §3.2. At 1/8 resolution each image is split into K×K windows, with K = 2, so each window is H/2 by W/2. The partition is then shifted by half a window, (H/2K, W/2K), to link neighbouring windows 2211.05783 §3.2 2211.05783 tab. 2. The same network serves flow, stereo and depth. The 1/4-resolution refinement uses 8×8 windows, and the card does not say whether the shift applies there 2211.05783 §3.4.1. Card: Unifying Flow, Stereo and Depth Estimation.

The author-reported split ablation on flow compares the two settings 2211.05783 tab. 2:

• 2×2 windows: 6.67 Things EPE at 53 ms.
• 1×1 split: 6.34 Things EPE at 105 ms.

Lower EPE is better, so the 2×2 split is slightly less accurate and about twice as fast in this table. The card records no run with the shift removed, so this table does not isolate the shift itself 2211.05783 tab. 2.

Window sizes in the FAST-VQA backbone

FAST-VQA runs Video Swin-T on fragment videos, which are small canvases built from raw-resolution patches 2207.02595 §3.1 2207.02595 §3.2. Its window size is (8, 7, 7) for the full model and (4, 4, 4) for the lower-cost FAST-VQA-M 2207.02595 tab. 2. The card gives the first size as "adapted to the input size" 2207.02595 §4.1, and it does not say how the adaptation works. The card does not mention a shift, so this page does not say whether the shifted blocks are kept there 2207.02595 §3.2.

The two variants also differ in frame count (32 against 16) and grid count (7 against 4), so the window size alone does not explain their FLOPs gap (279G against 46G) 2207.02595 tab. 2. Card: FAST-VQA.

Window branch beside anchored stripes in GRL

GRL's layer splits its channels equally. One half goes to window attention in the Swin V2 style, the other half to anchored stripe attention, and the two outputs are concatenated 2303.00748 §4. The stripes are a different mechanism. They attend along horizontal or vertical bands and alternate across layers, with shifted stripe modes 2303.00748 §3.3. The paper gives no window size for GRL itself. The released task configs use window sizes from 8 to 36 tokens, set per task 2303.00748 §3.1. The GRL card lists those code lines.

The text does not say whether the GRL window branch is shifted. The released code shifts it on even layers, which the paper does not state. The GRL card lists the line. The paper's ablations cover the similarity function and the anchor projection, not the window shift, so no reported result isolates the shift in GRL 2303.00748 tab. 9 2303.00748 tab. 10. Card: Efficient and Explicit Modelling of Image Hierarchies for Image Restoration.

When to use which

The paper's numbers are on Kinetics-400 with Swin-T, and each comparison is a single run. The paper gives no run-to-run variance, so it does not show how much of a gap is noise 2106.13230 §4 2106.13230 tab. 4. The UniMatch and GRL numbers are author-reported, and the dense-matching and restoration rows above are trade-offs from one table each, not general rules.

Implementation sketch

The sketch follows the block pair in the Video Swin paper. It is written from the paper's description and is not a copy of any code 2106.13230 §3.2.

# Shifted-window attention for one block  [arxiv:2106.13230 §3.2]
# x: tokens (T', H', W', C); window (P, M, M); shift = (P//2, M//2, M//2)
def shifted_window_attention(x, shift, use_shift):
    if use_shift:
        # Cyclically move the grid so the offset windows line up with
        # the fixed grid  [arxiv:2106.13230 §3.2]
        x = roll(x, by=-shift, axes=(T, H, W))
    # Split into non-overlapping windows of P*M*M tokens  [arxiv:2106.13230 §3.2]
    windows = partition(x, (P, M, M))
    # Attention inside each window, with a learned bias per head  [arxiv:2106.13230 eq. 2]
    scores = attention_scores(windows) + relative_bias
    if use_shift:
        # Tokens that were far apart before the roll now share a window,
        # so block attention between them  [arxiv:2106.13230 §3.2]
        scores = scores + region_mask
    windows = softmax(scores) @ values(windows)
    x = reverse_partition(windows)
    if use_shift:
        # Undo the roll  [arxiv:2106.13230 §3.2]
        x = roll(x, by=shift, axes=(T, H, W))
    return x

Notes for implementers:

• Label each token with the original region it came from, and add a large negative value to scores between different labels 2106.13230 §3.2.
• Alternate use_shift between the two blocks of a pair 2106.13230 eq. 1.
• Pad each axis to a multiple of the window size if the input does not divide evenly. The paper does not describe this 2106.13230 §3.2.
• For UniMatch's 2D form, the window and shift are the H/2K and W/2K sizes given above, on a 2D feature map 2211.05783 §3.2.
• For GRL's window branch, apply window attention to half of the channels only. Whether to shift it is not in the paper; see Pitfalls before you choose 2303.00748 §4.

Contradictions

None known between the papers' stated rules. Video Swin and UniMatch both use a half-window shift, so their shift settings agree. FAST-VQA's card states no shift rule, and it reports a different task, so its numbers are not compared. GRL's text states no window shift, but its released code shifts windows on even layers. That is a gap between paper and code, listed under Pitfalls below, not a disagreement between papers. Where other paper and code differences exist, they are listed under Pitfalls as well.

Pitfalls

• Half-window shift with odd window size. The paper's formula gives 3.5 when M=7 2106.13230 §3.2. The official code uses integer halving, which gives 3. The paper does not say which to use 2106.13230 §3.2.
• Window larger than the token count. With 32 frames and temporal stride 2 there are 16 temporal tokens. A temporal window of 8 gives two windows along time and a window of 16 gives one 2106.13230 §4.1 2106.13230 tab. 5. The official code clamps the window and shift when the input is smaller than the window. The paper does not describe this 2106.13230 §4.1.
• Mask value and padding. The paper does not give the value used for masked scores or the padding rule. The official code uses a large negative value and pads each axis to a multiple of the window size 2106.13230 §3.2.
• Single runs. The ablation gaps are 0.1 to 0.7 points, each from a single run. The paper gives no seeds or variance 2106.13230 tab. 4 2106.13230 tab. 6.
• Stereo cross-attention is windowed in released code. The UniMatch paper describes plain 1D horizontal cross-attention 2211.05783 §3.2. The released stereo scripts use a windowed, shifted 1D variant instead. The UniMatch card lists the code lines. Implement the variant the released checkpoints use.
• Split ablation only. The UniMatch ablation changes the window split and reports no run without the shift 2211.05783 tab. 2. It cannot show how much the shift adds by itself.
• GRL window shift is in code only. The paper describes window attention in the Swin V2 style for half the channels 2303.00748 §4 and does not state a shift for it. The released code shifts it on even layers. The GRL card lists the lines. Match the released code if you want to reproduce its weights; the paper gives no evidence that the shift helps.
• GRL window size varies by task. The paper gives no window size for GRL itself. The released configs use 8 to 36 tokens per task, and the denoising test script changes the window size to 32 for GRL-B. The GRL card lists the lines.
• GRL results are author-reported. The card's tables are the authors' own results, against baselines in the same tables. Supplementary training details are not in the arXiv source, so the training protocol behind them is not stated 2303.00748 §5.

Papers

• Video Swin Transformer: extends shifted-window attention to 3D windows over space and time, and reports ablations of the shift and window size on Kinetics-400 and Something-Something v2.
• Unifying Flow, Stereo and Depth Estimation: uses 2×2 windows with a half-window shift in its Transformer at 1/8 resolution, for flow, stereo and depth, and reports one window-split ablation on flow.
• FAST-VQA: uses the Video Swin-T backbone with (8, 7, 7) windows for video quality assessment, and (4, 4, 4) windows in its lower-cost variant; the card does not describe the shift.
• Efficient and Explicit Modelling of Image Hierarchies for Image Restoration: uses Swin V2-style window attention on half the channels, alongside anchored stripe attention, for image restoration; the text states no window shift, and the released code shifts windows on even layers.