---
kind: concept
slug: shifted-window-attention
title: Shifted-window attention
summary: >-
  Shifted-window attention limits self-attention to non-overlapping windows and moves the window grid every other block, so tokens can reach across window edges. Video Swin uses 3D windows, UniMatch uses 2D windows at 1/8 resolution, and GRL pairs Swin V2-style windows with anchored stripes. Ablation gains are author-reported, without run-to-run variance.
tags: [attention, locality, transformer, video, windowed-attention, dense-matching, image-restoration]
papers: ["2106.13230", "2211.05783", "2207.02595", "2303.00748"]
related: [vision-transformer, hierarchical-vision-transformer, self-attention, relative-positional-encoding, local-global-attention, transfer-learning, optical-flow, dense-correspondence, image-super-resolution, image-denoising]
status: draft
updated_at: 2026-11-01
---

## Overview

Full self-attention lets every token look at every other token, which costs
more as the number of tokens grows. For a video the token count grows with
the number of frames and with the frame size, so global attention is too
expensive [arxiv:2106.13230 §1] [arxiv:2106.13230 §3.2].

Windowed attention fixes the cost by splitting the token grid into
non-overlapping windows and letting each token attend only inside its own
window [arxiv:2106.13230 §3.2]. The catch is that a plain window grid never
lets information cross a window edge. Shifted-window attention fixes this by
offsetting the grid in every second block. The windows of that block then
straddle the edges of the previous block's windows, so neighbouring regions
can exchange information [arxiv:2106.13230 §3.2].

Four papers on this page use the idea or one of its parts. Video Swin applies it to 3D windows over
time, height and width [arxiv:2106.13230 §3.1]. UniMatch applies the 2D form,
with windows and a half-window shift, to the feature maps of its matching
network for optical flow, stereo and depth [arxiv:2211.05783 §3.2]. FAST-VQA
builds on the Video Swin-T backbone for video quality assessment, with a 3D
window size per variant [arxiv:2207.02595 §3.2] [arxiv:2207.02595 tab. 2]. GRL
uses Swin V2-style window attention for half of its channels in image
restoration, and its text states no window shift [arxiv:2303.00748 §4]. The
FAST-VQA card does not describe the shift at all. The 2D form itself comes
from the [Swin Transformer](/papers/2103.14030) card, which does not list this
concept yet, so this page does not draw on that paper.

## Variants

### Shifted 3D windows in Video Swin

Each block pair has a regular block and a shifted block. The regular block
partitions the grid into windows of P×M×M tokens, with P=8 in time and M=7 in
height and width by default. The shifted block offsets the grid by half a
window in each axis. The paper's formula is (P/2, M/2, M/2), which gives 3.5
for M=7 [arxiv:2106.13230 §3.2] [arxiv:2106.13230 §3.3]. See Pitfalls for
the integer rounding. Card: [Video Swin Transformer](/papers/2106.13230).

The paper keeps the batched form of the Swin computation, so the number of
windows does not change between the two blocks [arxiv:2106.13230 §3.2]. Each
window still attends only inside itself, with a relative position bias
[arxiv:2106.13230 §3.2] [arxiv:2106.13230 eq. 2].

### Shift in space only, or in time and space

The paper compares three settings on Swin-T and Kinetics-400 [arxiv:2106.13230 tab. 6]:

- Full 3D shift, in time and space: 78.8 top-1, 93.6 top-5.
- Spatial shift only, no temporal shift: 78.5 top-1, 93.5 top-5.
- No shift at all: 78.1 top-1, 93.3 top-5.

The full 3D shift matches the reported default result of 78.8 [arxiv:2106.13230 tab. 6]. The gaps are
0.3 and 0.7 points, from single runs [arxiv:2106.13230 tab. 6].

### Two-dimensional windows with a half-window shift in UniMatch

UniMatch uses the 2D form in the Transformer that enhances the features of
each image [arxiv:2211.05783 §3.2]. At 1/8 resolution each image is split into
K×K windows, with K = 2, so each window is H/2 by W/2. The partition is then
shifted by half a window, (H/2K, W/2K), to link neighbouring windows
[arxiv:2211.05783 §3.2] [arxiv:2211.05783 tab. 2]. The same network serves
flow, stereo and depth. The 1/4-resolution refinement uses 8×8 windows, and
the card does not say whether the shift applies there [arxiv:2211.05783 §3.4.1].
Card: [Unifying Flow, Stereo and Depth Estimation](/papers/2211.05783).

The author-reported split ablation on flow compares the two settings
[arxiv:2211.05783 tab. 2]:

- 2×2 windows: 6.67 Things EPE at 53 ms.
- 1×1 split: 6.34 Things EPE at 105 ms.

Lower EPE is better, so the 2×2 split is slightly less accurate and about
twice as fast in this table. The card records no run with the shift removed,
so this table does not isolate the shift itself [arxiv:2211.05783 tab. 2].

### Window sizes in the FAST-VQA backbone

FAST-VQA runs Video Swin-T on fragment videos, which are small canvases built
from raw-resolution patches [arxiv:2207.02595 §3.1] [arxiv:2207.02595 §3.2].
Its window size is (8, 7, 7) for the full model and (4, 4, 4) for the
lower-cost FAST-VQA-M [arxiv:2207.02595 tab. 2]. The card gives the first size
as "adapted to the input size" [arxiv:2207.02595 §4.1], and it does not say
how the adaptation works. The card does not mention a shift, so this page does
not say whether the shifted blocks are kept there [arxiv:2207.02595 §3.2].

The two variants also differ in frame count (32 against 16) and grid count (7
against 4), so the window size alone does not explain their FLOPs gap
(279G against 46G) [arxiv:2207.02595 tab. 2]. Card: [FAST-VQA](/papers/2207.02595).

### Window branch beside anchored stripes in GRL

GRL's layer splits its channels equally. One half goes to window attention in
the Swin V2 style, the other half to anchored stripe attention, and the two
outputs are concatenated [arxiv:2303.00748 §4]. The stripes are a different
mechanism. They attend along horizontal or vertical bands and alternate across
layers, with shifted stripe modes [arxiv:2303.00748 §3.3]. The paper gives no
window size for GRL itself. The released task configs use window sizes from 8
to 36 tokens, set per task [arxiv:2303.00748 §3.1]. The
[GRL card](/papers/2303.00748) lists those code lines.

The text does not say whether the GRL window branch is shifted. The released
code shifts it on even layers, which the paper does not state. The
[GRL card](/papers/2303.00748) lists the line. The paper's ablations cover the
similarity function and the anchor projection, not the window shift, so no
reported result isolates the shift in GRL [arxiv:2303.00748 tab. 9]
[arxiv:2303.00748 tab. 10]. Card: [Efficient and Explicit Modelling of Image
Hierarchies for Image Restoration](/papers/2303.00748).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Video input, long clips, a transformer backbone that must stay affordable | Shifted 3D windows | The paper's Swin-T uses windows of 8×7×7 tokens, and the joint 3D form gives the best accuracy among the attention layouts it tests [arxiv:2106.13230 tab. 4] |
| Video with a tighter compute budget and a small loss of accuracy acceptable | Keep the shift and reduce the window | With 16 temporal tokens, a window of 4 gives 78.6 top-1 at 79 G, against 78.8 at 88 G for a window of 8 and 79.1 at 106 G for a window of 16 [arxiv:2106.13230 tab. 5] |
| Need the attention layout the paper tested in full, with temporal and spatial layers kept apart | Not supported by this page's sources | The split temporal design drops to 76.4 top-1 with more parameters [arxiv:2106.13230 tab. 4] |
| Image-only pre-training is the starting point | Shifted 3D windows with inflated 2D weights | The paper inflates the 2D checkpoint and reports the same top-1 for both inflation choices [arxiv:2106.13230 tab. 8] |
| Dense 2D matching (flow, stereo or depth) with a Transformer at 1/8 resolution, where time per pair matters | 2×2 windows with a half-window shift | The split ablation trades 0.33 Things EPE for about half the time (53 ms against 105 ms) [arxiv:2211.05783 tab. 2]. The windows do not shrink the matching step, which still scores every pair of positions [arxiv:2211.05783 eq. 1] |
| High-resolution image restoration where regional detail and long-range context both matter, with a limited parameter budget | Window branch beside anchored stripes (GRL) | GRL-B reports 33.93 dB on GoPro against 32.92 dB for Restormer [arxiv:2303.00748 tab. 2], with 19.81 M parameters against Restormer's 26.13 M [arxiv:2303.00748 tab. 4]. The card does not isolate the window branch, so these are whole-model results |

The paper's numbers are on Kinetics-400 with Swin-T, and each comparison is
a single run. The paper gives no run-to-run variance, so it does not show how
much of a gap is noise [arxiv:2106.13230 §4] [arxiv:2106.13230 tab. 4]. The
UniMatch and GRL numbers are author-reported, and the dense-matching and
restoration rows above are trade-offs from one table each, not general rules.

## Implementation sketch

The sketch follows the block pair in the Video Swin paper. It is written from
the paper's description and is not a copy of any code [arxiv:2106.13230 §3.2].

```pseudocode
# Shifted-window attention for one block  [arxiv:2106.13230 §3.2]
# x: tokens (T', H', W', C); window (P, M, M); shift = (P//2, M//2, M//2)
def shifted_window_attention(x, shift, use_shift):
    if use_shift:
        # Cyclically move the grid so the offset windows line up with
        # the fixed grid  [arxiv:2106.13230 §3.2]
        x = roll(x, by=-shift, axes=(T, H, W))
    # Split into non-overlapping windows of P*M*M tokens  [arxiv:2106.13230 §3.2]
    windows = partition(x, (P, M, M))
    # Attention inside each window, with a learned bias per head  [arxiv:2106.13230 eq. 2]
    scores = attention_scores(windows) + relative_bias
    if use_shift:
        # Tokens that were far apart before the roll now share a window,
        # so block attention between them  [arxiv:2106.13230 §3.2]
        scores = scores + region_mask
    windows = softmax(scores) @ values(windows)
    x = reverse_partition(windows)
    if use_shift:
        # Undo the roll  [arxiv:2106.13230 §3.2]
        x = roll(x, by=shift, axes=(T, H, W))
    return x
```

Notes for implementers:

- Label each token with the original region it came from, and add a large
  negative value to scores between different labels [arxiv:2106.13230 §3.2].
- Alternate `use_shift` between the two blocks of a pair [arxiv:2106.13230 eq. 1].
- Pad each axis to a multiple of the window size if the input does not
  divide evenly. The paper does not describe this [arxiv:2106.13230 §3.2].
- For UniMatch's 2D form, the window and shift are the H/2K and W/2K sizes
  given above, on a 2D feature map [arxiv:2211.05783 §3.2].
- For GRL's window branch, apply window attention to half of the channels only.
  Whether to shift it is not in the paper; see Pitfalls before you choose
  [arxiv:2303.00748 §4].

## Contradictions

None known between the papers' stated rules. Video Swin and UniMatch both use
a half-window shift, so their shift settings agree. FAST-VQA's card states no
shift rule, and it reports a different task, so its numbers are not compared.
GRL's text states no window shift, but its released code shifts windows on
even layers. That is a gap between paper and code, listed under Pitfalls below,
not a disagreement between papers. Where other paper and code differences
exist, they are listed under Pitfalls as well.

## Pitfalls

- **Half-window shift with odd window size.** The paper's formula gives 3.5
  when M=7 [arxiv:2106.13230 §3.2]. The official code uses integer halving,
  which gives 3. The paper does not say which to use [arxiv:2106.13230 §3.2].
- **Window larger than the token count.** With 32 frames and temporal stride 2
  there are 16 temporal tokens. A temporal window of 8 gives two windows along
  time and a window of 16 gives one [arxiv:2106.13230 §4.1] [arxiv:2106.13230 tab. 5].
  The official code clamps the window and shift when the input is smaller
  than the window. The paper does not describe this [arxiv:2106.13230 §4.1].
- **Mask value and padding.** The paper does not give the value used for
  masked scores or the padding rule. The official code uses a large negative
  value and pads each axis to a multiple of the window size
  [arxiv:2106.13230 §3.2].
- **Single runs.** The ablation gaps are 0.1 to 0.7 points, each from a
  single run. The paper gives no seeds or variance [arxiv:2106.13230 tab. 4] [arxiv:2106.13230 tab. 6].
- **Stereo cross-attention is windowed in released code.** The UniMatch paper
  describes plain 1D horizontal cross-attention [arxiv:2211.05783 §3.2]. The
  released stereo scripts use a windowed, shifted 1D variant instead. The
  [UniMatch card](/papers/2211.05783) lists the code lines. Implement the
  variant the released checkpoints use.
- **Split ablation only.** The UniMatch ablation changes the window split and
  reports no run without the shift [arxiv:2211.05783 tab. 2]. It cannot show
  how much the shift adds by itself.
- **GRL window shift is in code only.** The paper describes window attention
  in the Swin V2 style for half the channels [arxiv:2303.00748 §4] and does not
  state a shift for it. The released code shifts it on even layers. The
  [GRL card](/papers/2303.00748) lists the lines. Match the released code if
  you want to reproduce its weights; the paper gives no evidence that the shift
  helps.
- **GRL window size varies by task.** The paper gives no window size for GRL
  itself. The released configs use 8 to 36 tokens per task, and the denoising
  test script changes the window size to 32 for GRL-B. The
  [GRL card](/papers/2303.00748) lists the lines.
- **GRL results are author-reported.** The card's tables are the authors' own
  results, against baselines in the same tables. Supplementary training
  details are not in the arXiv source, so the training protocol behind them is
  not stated [arxiv:2303.00748 §5].

## Papers

- [Video Swin Transformer](/papers/2106.13230): extends shifted-window attention to 3D windows over space and time, and reports ablations of the shift and window size on Kinetics-400 and Something-Something v2.
- [Unifying Flow, Stereo and Depth Estimation](/papers/2211.05783): uses 2×2 windows with a half-window shift in its Transformer at 1/8 resolution, for flow, stereo and depth, and reports one window-split ablation on flow.
- [FAST-VQA](/papers/2207.02595): uses the Video Swin-T backbone with (8, 7, 7) windows for video quality assessment, and (4, 4, 4) windows in its lower-cost variant; the card does not describe the shift.
- [Efficient and Explicit Modelling of Image Hierarchies for Image Restoration](/papers/2303.00748): uses Swin V2-style window attention on half the channels, alongside anchored stripe attention, for image restoration; the text states no window shift, and the released code shifts windows on even layers.