Paper · 2022-11-21 · cs.CV
N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution
Adds an N-gram context to Swin window attention for lightweight super-resolution. Neighbouring windows are embedded by a group convolution, interact through sliding scaled-cosine attention, and their output is added back as a per-window bias. NGswin (about 1M parameters, 140.4G Mult-Adds at x2) and SwinIR-NG report the top PSNR on most benchmarks compared.
Image super-resolutionShifted-window attentionHierarchical vision transformersSelf-attentionimage-super-resolutionwindow-attentionswin-transformerlightweight-model
arXivPDFCodeMarkdown23 min read
AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Super-resolution turns a small, blurry picture into a larger, sharper one. A Swin Transformer handles this by cutting the image into small square patches and looking at each patch mostly on its own. That makes it hard for a patch to use the texture of the patches next to it. This paper borrows the idea of an N-gram from language models, where a word is read together with its neighbours, and applies it to patches. Each patch gathers a summary of its neighbourhood through a cheap step, the neighbourhood summaries exchange information through a second attention step, and the result is added back to each patch as a bias.
The authors build a small new network, NGswin, and add the same step to two existing Swin-based models, which then score higher. The goal is a model that stays small and efficient. The extra step costs only a little more computation than the model it extends.
TL;DR
A method for lightweight single-image super-resolution. Each Swin window receives an N-gram context built from its neighbouring windows: a channel- reducing group convolution, sliding scaled-cosine attention over 2 x 2 neighbourhoods, and a 1 x 1 merge §3.4. NGswin is a U-Net-like network with hierarchical encoder stages, an SCDP bottleneck and a small decoder §3.3 alg. 1. Reported numbers are author-reported on five benchmarks; they are only comparable under the same training data: some rows use DIV2K alone and others DF2K tab. 2 tab. 3. Training uses only the 800 DIV2K images app. B.
Problem
Window self-attention (WSA) in Swin only mixes tokens inside one window, so the receptive field is small §3.1. The authors visualise deep feature maps and report two failure modes: windows in the same region end up with homogeneous patterns that differ from their neighbours, which distorts the output, and the shallow layers cannot use patterns outside the window at all §3.1 fig. 2. Lightweight super-resolution also has to keep multiply-add counts low, since the authors tie the time and energy cost of a network to these operations §1.
Method
The network has four parts: a shallow convolution, three hierarchical encoder stages of Swin-style blocks with patch merging, an SCDP bottleneck that fuses all encoder scales, and a decoder with a reconstruction head §3.3 fig. 3a. Each Swin-style block is an NSTB (N-Gram Swin Transformer Block) §3.4 fig. 3b.
-
Shallow module. One 3 x 3 convolution maps the low-resolution input to a feature map z_s with D = 64 channels §3.3.
-
Encoder stages. Stage i runs K_i NSTBs, with K = (6, 4, 4) by default, at resolution HW / (2^(i-1))^2 §3.3 eq. 1. Between stages a 2 x 2 patch merging halves the resolution. It follows Swin, except that the width falls from 4D to D instead of 2D §3.3 main_branch.py.
-
Across-stage pooling cascade. Before stages 2 and 3, the outputs of the earlier stages (including z_s) are max-pooled 2 x 2 and concatenated with the current stage output, in the manner of CARN's global cascade §3.3. The code then projects the concatenation back to D with a linear layer main_branch.py. See Paper vs code.
-
N-gram context and NSTB. Each NSTB first builds an N-gram context from the unshifted window grid, adds it to every window, applies window attention, and then uses post-normalisation residuals and a feed-forward network §3.4 fig. 3b-c. The four sub-steps of the context are described in the pseudocode below. Shifted windows (size M/2) are used in every second block, as in Swin §3.4. Inside a stage, blocks and patch merging are connected by residual paths rather than dense links; the third encoder stage and the decoder have no residual link to their patch merging §3.4 app. A.2.
-
SCDP bottleneck. Takes the last-block output of each encoder stage plus z_s, upsamples each by pixel shuffle, concatenates them along channels, and applies a 3 x 3 depth-wise convolution and a point-wise projection §3.5 alg. 1.
-
Decoder. One decoder stage with K_dec = 6 NSTBs at full resolution. Its input is the bottleneck output plus the first encoder stage's output. A final layer norm follows, and the shallow map z_s is added as a global skip connection §3.3 app. A.2.
-
Reconstruction. A convolution produces 3 r^2 channels, a pixel shuffle with factor r gives the upscaled image, and a further 3 x 3 convolution maps it to RGB §3.3 app. A.3 reconstruction.py.
-
SwinIR-NG and HNCT-NG. The same N-gram context is applied to SwinIR-light and HNCT §3.4. The improved versions are named SwinIR-NG and HNCT-NG §4.2.
Window attention. Scaled-cosine attention with a relative position bias §3.4 eq. 2 eq. 3:
\mathrm{WSA}(Q, K, V) = \mathrm{Softmax}\!\left(\frac{\cos(Q, K)}{\tau} + B\right) V eq. 2
\cos(Q, K)_{ij} = \frac{q_i \cdot k_j}{\lVert q_i \rVert_2 \, \lVert k_j \rVert_2} eq. 3
Here tau is a learnable scalar greater than 0.01 §3.4. The encoder mapping of stage i is written as a recursion over its K_i blocks eq. 1.
The N-gram context is a sliding attention over unit windows, computed in both directions with shared weights:
# N-gram context for one block [§3.4] [fig. 3c] [fig. 6]
# Input: tokens (B, H, W, D); window M = 8; N-gram size N = 2
# Output: context (B, H/M, W/M, D), one vector per window
def ngram_context(tokens):
# Channel-reducing group conv: each MxM window -> one D/2 vector [§3.4]
unigram = group_conv(tokens, M, stride=M, groups=D // 2)
# Pad N-1 units on the lower and right edges [app. A.1]
forward_pad = seq_refl_pad(unigram, "lower_right", N - 1)
# Pad N-1 units on the upper and left edges, same rule [app. A.1]
backward_pad = seq_refl_pad(unigram, "upper_left", N - 1)
# Sliding NxN cosine attention, then NxN average pooling [§3.4] [eq. 2]
forward_feat = avg_pool(sliding_wsa(forward_pad, N), N)
# The backward pass reuses the same sliding-attention weights [§3.4]
backward_feat = avg_pool(sliding_wsa(backward_pad, N), N)
# Concatenate both directions and merge with a 1x1 conv [§3.4]
context = conv1x1(concat([forward_feat, backward_feat], dim=-1))
return contextThe NSTB applies the context to the block's windows and then runs attention:
# One NSTB [§3.4] [fig. 3b] [code: my_model/ngswin_model/nstb.py#L114-L120]
# Input: tokens (B, H, W, D); shifted: True in every second block
def nstb(tokens, shifted):
# Add one context value to all M*M pixels of its window [§3.4] [fig. 3c]
context = ngram_context(tokens)
windows = partition(tokens, M) + context[:, :, :, None, None]
# Cyclic shift by M/2 for shifted blocks [§3.4]
grid = unpartition(windows)
if shifted:
grid = roll(grid, -(M // 2))
# Scaled-cosine window attention [eq. 2] [eq. 3]
attn = wsa(partition(grid, M), mask if shifted else None)
grid = unpartition(attn)
if shifted:
grid = roll(grid, M // 2)
# Post-normalisation residuals, then feed-forward network [§3.4]
tokens = tokens + layer_norm_1(grid)
tokens = tokens + layer_norm_2(ffn(tokens))
return tokensThe SCDP bottleneck follows the paper's algorithm alg. 1 with its description in §3.5:
# SCDP bottleneck [alg. 1] [§3.5] [app. B]
# Input: last-block outputs of the 3 encoder stages, shallow map z_s
# Output: tokens (B, H*W, D)
def scdp_bottleneck(stage_outputs, shallow):
pieces = []
for i in range(3):
# Max-pool z_s to stage i's size, then LeakyReLU [§3.5] [alg. 1]
guide = leaky_relu(max_pool_2x2_times(shallow, i))
# Add the guide, then pixel-shuffle by 2**i [§3.5] [alg. 1]
piece = pixel_shuffle(stage_outputs[i] + guide, 2 ** i)
pieces.append(piece)
# Concatenate along channels [§3.5] [alg. 1]
x = concat(pieces, dim=channels)
# 3x3 depth-wise convolution, then GELU [alg. 1] [app. B]
x = gelu(depthwise_conv3x3(x))
# Point-wise linear projection to D, then LayerNorm [§3.5] [alg. 1]
return layer_norm(linear(x, out_dim=D))Hyperparameters
Values are the paper's unless marked "code". Code values are read from the
official repository at code_commit. The paper gives many values only in
its supplement app. B, so both places are cited.
| Name | Value | Where | Notes |
|---|---|---|---|
| Feature width D | 64 | §3.3 app. B | encoder and decoder width; code embed_dim argfunc_utils.py |
| Encoder NSTBs K1, K2, K3 | 6, 4, 4 | §3.3 app. B | code depths argfunc_utils.py |
| Decoder NSTBs K_dec | 6 | §3.3 app. B | code dec_depths argfunc_utils.py |
| Attention heads, encoder | 6, 4, 4 | app. B | code num_heads argfunc_utils.py |
| Attention heads, decoder | 6 | app. B | code dec_num_heads argfunc_utils.py |
| Per-head dimension | not stated | — | code uses integer D // heads win_attention.py; see Pitfalls |
| Window size M | 8 | §3.4 app. B | code window_size argfunc_utils.py |
| N-gram size N | 2 | §3.4 app. B | code ngrams argfunc_utils.py |
| FFN hidden width | 128 (ratio 2) | app. B | code mlp_ratio argfunc_utils.py |
| Shift size | M/2 = 4 in alternate blocks | §3.4 app. B | code shifts blocks with index i odd, 0-based main_branch.py |
| Attention temperature tau | learnable, > 0.01 | §3.4 | code: logit scale starts at log 10 (tau = 0.1), clamped to tau >= 0.01 win_attention.py win_attention.py |
| Relative bias table init | truncated normal, std 0.02 | — | code win_attention.py |
| Other init | Linear: truncated normal, std 0.02; LayerNorm: weight 1, bias 0 | — | code ngswin.py |
| Activations | GELU in FFN and after the SCDP depth-wise conv; LeakyReLU after the bottleneck pooling | app. B | none elsewhere |
| Dropout, drop-path | 0 | app. B | code defaults argfunc_utils.py |
| Training patch | 64 x 64 LR patch | §4.1 | code training_patch_size argfunc_utils.py |
| Augmentation | random horizontal flip; rotation by 0, 90, 180 or 270 degrees | §4.1 | code dataset.py |
| Repeats per epoch | 80 copies of each image | app. B | code dataset.py; 800 x 80 = 64,000 crops per epoch (derived) |
| Input normalisation | per-channel mean and std from the 800 LR images, per task | app. B | code uses constants: mean (0.4485, 0.4375, 0.4045) for every scale; std (0.2397, 0.2290, 0.2389) for x2, (0.2373, 0.2265, 0.2367) for x3, (0.2352, 0.2244, 0.2349) for x4 ngswin.py |
| Loss | L1 between de-normalised output and HR | §4.1 app. B | code default criterion L1 argfunc_utils.py; loss taken after de-normalisation ngswin.py |
| Batch size | 64 in total (4 GPUs) | app. B | code default argfunc_utils.py; DDP README uses 16 per GPU |
| Optimizer, scratch | Adam, beta = (0.9, 0.999), eps = 1e-8 | app. B | code sets lr only dp_builder.py; betas and eps come from PyTorch defaults |
| Initial learning rate | 4e-4 | app. B | code default argfunc_utils.py |
| Warm-up, scratch | 20 epochs, linear from 0 | app. B | the same paragraph writes "10^-4" for the target lr; the code uses 4e-4 argfunc_utils.py |
| LR schedule, scratch | halved after epochs 200, 300, 400, 425, 450, 475 | app. B | code default half_list argfunc_utils.py |
| Total epochs, scratch | 500 | app. B | code default argfunc_utils.py |
| Cosine schedule | tried, worse on SR | app. B | code supports it train_utils.py |
| Weight decay | 0 | app. B | paper: 0.05 hurt. Code default argfunc_utils.py |
| Gradient clipping | none | app. B | paper: clipping at 5.0 hurt. Code default max_norm None argfunc_utils.py |
| Warm-start (x3, x4) | start from x2 weights; first 50 epochs train only the reconstruction head at lr 4e-4 | app. B | code freezes all but to_target dp_builder.py; the warm-start lr branch returns before setting the lr train_utils.py |
| Whole fine-tuning | AdamW, same hyper-parameters; 10 warm-up epochs; halved after 50, 100, 150, 175, 200, 225 epochs of this phase; 300 epochs total with warm-start | app. B | AdamW dp_builder.py; code defaults differ, see Paper vs code |
| Seed | not stated | — | code default 100 argfunc_utils.py |
| Hardware | 4 NVIDIA TITAN Xp, PyTorch | app. B | README lists PyTorch >= 1.10.1 |
| Metric | PSNR and SSIM on Y of YCbCr | §4.1 | code crops scale pixels from every border before metrics dp_engine_test.py dp_engine_test.py |
Results
Setup: 800 DIV2K training pairs; test sets Set5, Set14, BSD100, Urban100 and Manga109; LR images from MATLAB bicubic resizing; PSNR and SSIM on the Y channel §4.1. All numbers are author-reported. The paper does not state how many runs each number averages or report any variance §4.1. Mult-Adds are for a 1280 x 720 HR image tab. 1 tab. 2.
Efficiency at the three scales tab. 1:
| Scale | NGswin | SwinIR-light (corrected) | ESRT | DiVANet | ELAN-light |
|---|---|---|---|---|---|
| x2 | 140.4G | 243.7G | 191.4G | 189.0G | 168.4G |
| x3 | 66.6G | 109.5G | 96.4G | 89.0G | 75.7G |
| x4 | 36.4G | 61.7G | 67.7G | 57.0G | 43.2G |
Efficient SR, x2, all rows tab. 2. PSNR / SSIM:
| Method | Train | Mult-Adds | Params | Set5 | Set14 | BSD100 | Urban100 | Manga109 |
|---|---|---|---|---|---|---|---|---|
| EDSR-baseline | D2K | 316.3G | 1,370K | 37.99 / 0.9604 | 33.57 / 0.9175 | 32.16 / 0.8994 | 31.98 / 0.9272 | 38.54 / 0.9769 |
| MemNet | 291 | 2,662.4G | 677K | 37.78 / 0.9597 | 33.28 / 0.9142 | 32.08 / 0.8978 | 31.31 / 0.9195 | - |
| CARN | D2K+291 | 222.8G | 1,592K | 37.76 / 0.9590 | 33.52 / 0.9166 | 32.09 / 0.8978 | 31.92 / 0.9256 | 38.36 / 0.9765 |
| IMDN | D2K | 158.8G | 694K | 38.00 / 0.9605 | 33.63 / 0.9177 | 32.19 / 0.8996 | 32.17 / 0.9283 | 38.88 / 0.9774 |
| LatticeNet | D2K | 169.5G | 756K | 38.06 / 0.9607 | 33.70 / 0.9187 | 32.20 / 0.8999 | 32.25 / 0.9288 | 38.94 / 0.9774 |
| RFDN-L | D2K | 145.8G | 626K | 38.08 / 0.9606 | 33.67 / 0.9190 | 32.18 / 0.8996 | 32.24 / 0.9290 | 38.95 / 0.9773 |
| SRPN-Lite | DF2K | 139.9G | 609K | 38.10 / 0.9608 | 33.70 / 0.9189 | 32.25 / 0.9005 | 32.26 / 0.9294 | - |
| HNCT | D2K | 82.4G | 357K | 38.08 / 0.9608 | 33.65 / 0.9182 | 32.22 / 0.9001 | 32.22 / 0.9294 | 38.87 / 0.9774 |
| FMEN | DF2K | 172.0G | 748K | 38.10 / 0.9609 | 33.75 / 0.9192 | 32.26 / 0.9007 | 32.41 / 0.9311 | 38.95 / 0.9778 |
| NGswin | D2K | 140.4G | 998K | 38.05 / 0.9610 | 33.79 / 0.9199 | 32.27 / 0.9008 | 32.53 / 0.9324 | 38.97 / 0.9777 |
Efficient SR, x3 tab. 2:
| Method | Train | Mult-Adds | Params | Set5 | Set14 | BSD100 | Urban100 | Manga109 |
|---|---|---|---|---|---|---|---|---|
| EDSR-baseline | D2K | 160.2G | 1,555K | 34.37 / 0.9270 | 30.28 / 0.8417 | 29.09 / 0.8052 | 28.15 / 0.8527 | 33.45 / 0.9439 |
| MemNet | 219 (as printed) | 2,662.4G | 677K | 34.09 / 0.9248 | 30.00 / 0.8350 | 28.96 / 0.8001 | 27.56 / 0.8376 | - |
| CARN | D2K+291 | 118.8G | 1,592K | 34.29 / 0.9255 | 30.29 / 0.8407 | 29.06 / 0.8034 | 28.06 / 0.8493 | 33.50 / 0.9440 |
| IMDN | D2K | 71.5G | 703K | 34.36 / 0.9270 | 30.32 / 0.8417 | 29.09 / 0.8046 | 28.17 / 0.8519 | 33.61 / 0.9445 |
| LatticeNet | D2K | 76.3G | 765K | 34.40 / 0.9272 | 30.32 / 0.8416 | 29.10 / 0.8049 | 28.19 / 0.8513 | 33.63 / 0.9442 |
| RFDN-L | D2K | 65.6G | 633K | 34.47 / 0.9280 | 30.35 / 0.8421 | 29.11 / 0.8053 | 28.32 / 0.8547 | 33.78 / 0.9458 |
| SRPN-Lite | DF2K | 62.7G | 615K | 34.47 / 0.9276 | 30.38 / 0.8425 | 29.16 / 0.8061 | 28.22 / 0.8534 | - |
| HNCT | D2K | 37.8G | 363K | 34.47 / 0.9275 | 30.44 / 0.8439 | 29.15 / 0.8067 | 28.28 / 0.8557 | 33.81 / 0.9459 |
| FMEN | DF2K | 77.2G | 757K | 34.45 / 0.9275 | 30.40 / 0.8435 | 29.17 / 0.8063 | 28.33 / 0.8562 | 33.86 / 0.9462 |
| NGswin | D2K | 66.6G | 1,007K | 34.52 / 0.9282 | 30.53 / 0.8456 | 29.19 / 0.8078 | 28.52 / 0.8603 | 33.89 / 0.9470 |
Efficient SR, x4 tab. 2:
| Method | Train | Mult-Adds | Params | Set5 | Set14 | BSD100 | Urban100 | Manga109 |
|---|---|---|---|---|---|---|---|---|
| EDSR-baseline | D2K | 114.0G | 1,518K | 32.09 / 0.8938 | 28.58 / 0.7813 | 27.57 / 0.7357 | 26.04 / 0.7849 | 30.35 / 0.9067 |
| MemNet | 291 | 2,662.4G | 677K | 31.74 / 0.8893 | 28.26 / 0.7723 | 27.40 / 0.7281 | 25.50 / 0.7630 | - |
| CARN | D2K+291 | 90.9G | 1,592K | 32.13 / 0.8937 | 28.60 / 0.7806 | 27.58 / 0.7349 | 26.07 / 0.7837 | 30.47 / 0.9084 |
| IMDN | D2K | 40.9G | 715K | 32.21 / 0.8948 | 28.58 / 0.7811 | 27.56 / 0.7353 | 26.04 / 0.7838 | 30.45 / 0.9075 |
| LatticeNet | D2K | 43.6G | 777K | 32.18 / 0.8943 | 28.61 / 0.7812 | 27.57 / 0.7355 | 26.14 / 0.7844 | 30.54 / 0.9075 |
| RFDN-L | D2K | 37.4G | 643K | 32.28 / 0.8957 | 28.61 / 0.7818 | 27.58 / 0.7363 | 26.20 / 0.7883 | 30.61 / 0.9096 |
| SRPN-Lite | DF2K | 35.8G | 623K | 32.24 / 0.8958 | 28.69 / 0.7836 | 27.63 / 0.7373 | 26.16 / 0.7875 | - |
| HNCT | D2K | 22.0G | 373K | 32.31 / 0.8957 | 28.71 / 0.7834 | 27.63 / 0.7381 | 26.20 / 0.7896 | 30.70 / 0.9112 |
| FMEN | DF2K | 44.2G | 769K | 32.24 / 0.8955 | 28.70 / 0.7839 | 27.63 / 0.7379 | 26.28 / 0.7908 | 30.70 / 0.9107 |
| NGswin | D2K | 36.4G | 1,019K | 32.33 / 0.8963 | 28.78 / 0.7859 | 27.66 / 0.7396 | 26.45 / 0.7963 | 30.80 / 0.9128 |
Lightweight SR, x2 tab. 3. SwinIR-NG is SwinIR-light with the N-gram context. Years are as printed:
| Method | Year | Mult-Adds | Params | Set5 | Set14 | BSD100 | Urban100 | Manga109 |
|---|---|---|---|---|---|---|---|---|
| SwinIR-light | 2021 | 243.7G | 910K | 38.14 / 0.9611 | 33.86 / 0.9206 | 32.31 / 0.9012 | 32.76 / 0.9340 | 39.12 / 0.9783 |
| ESRT | 2022 | 191.4G | 677K | 38.03 / 0.9600 | 33.75 / 0.9184 | 32.25 / 0.9001 | 32.58 / 0.9318 | 39.12 / 0.9774 |
| ELAN-light | 2022 | 168.4G | 582K | 38.17 / 0.9611 | 33.94 / 0.9207 | 32.30 / 0.9012 | 32.76 / 0.9340 | 39.12 / 0.9783 |
| DiVANet | 2023 | 189.0G | 902K | 38.16 / 0.9612 | 33.80 / 0.9195 | 32.29 / 0.9012 | 32.60 / 0.9325 | 39.08 / 0.9775 |
| SwinIR-NG | 2023 | 274.1G | 1,181K | 38.17 / 0.9612 | 33.94 / 0.9205 | 32.31 / 0.9013 | 32.78 / 0.9340 | 39.20 / 0.9781 |
Lightweight SR, x3 tab. 3:
| Method | Year | Mult-Adds | Params | Set5 | Set14 | BSD100 | Urban100 | Manga109 |
|---|---|---|---|---|---|---|---|---|
| SwinIR-light | 2021 | 109.5G | 918K | 34.62 / 0.9289 | 30.54 / 0.8463 | 29.20 / 0.8082 | 28.66 / 0.8624 | 33.98 / 0.9478 |
| ESRT | 2022 | 96.4G | 770K | 34.42 / 0.9268 | 30.43 / 0.8433 | 29.15 / 0.8063 | 28.46 / 0.8574 | 33.95 / 0.9455 |
| ELAN-light | 2022 | 75.7G | 590K | 34.61 / 0.9288 | 30.55 / 0.8463 | 29.21 / 0.8081 | 28.69 / 0.8624 | 34.00 / 0.9478 |
| DiVANet | 2023 | 89.0G | 949K | 34.60 / 0.9285 | 30.47 / 0.8447 | 29.19 / 0.8073 | 28.58 / 0.8603 | 33.94 / 0.9468 |
| SwinIR-NG | 2023 | 114.1G | 1,190K | 34.64 / 0.9293 | 30.58 / 0.8471 | 29.24 / 0.8090 | 28.75 / 0.8639 | 34.22 / 0.9488 |
Lightweight SR, x4 tab. 3. Rows marked "DF2K" train on DIV2K and Flickr2K instead of DIV2K alone; "reduced" means 48 channels instead of 60:
| Method | Year | Mult-Adds | Params | Set5 | Set14 | BSD100 | Urban100 | Manga109 |
|---|---|---|---|---|---|---|---|---|
| SwinIR-light | 2021 | 61.7G | 930K | 32.44 / 0.8976 | 28.77 / 0.7858 | 27.69 / 0.7406 | 26.47 / 0.7980 | 30.92 / 0.9151 |
| ESRT | 2022 | 67.7G | 751K | 32.19 / 0.8947 | 28.69 / 0.7833 | 27.69 / 0.7379 | 26.39 / 0.7962 | 30.75 / 0.9100 |
| ELAN-light | 2022 | 43.2G | 601K | 32.43 / 0.8975 | 28.78 / 0.7858 | 27.69 / 0.7406 | 26.54 / 0.7982 | 30.92 / 0.9150 |
| DiVANet | 2023 | 57.0G | 939K | 32.41 / 0.8973 | 28.70 / 0.7844 | 27.65 / 0.7391 | 26.42 / 0.7958 | 30.73 / 0.9119 |
| SwinIR-NG reduced (DIV2K) | 2023 | 42.5G | 770K | 32.44 / 0.8978 | 28.80 / 0.7863 | 27.70 / 0.7407 | 26.47 / 0.7977 | 30.97 / 0.9147 |
| SwinIR-NG reduced (DF2K) | 2023 | 42.5G | 770K | 32.48 / 0.8979 | 28.83 / 0.7868 | 27.71 / 0.7411 | 26.54 / 0.7998 | 31.12 / 0.9158 |
| SwinIR-NG | 2023 | 63.0G | 1,201K | 32.44 / 0.8980 | 28.83 / 0.7870 | 27.73 / 0.7418 | 26.61 / 0.8010 | 31.09 / 0.9161 |
N-gram ablation tab. 4. Top: NGswin with and without the context. Bottom: HNCT and HNCT-NG. PSNR / SSIM on Set14, Urban100, Manga109. The x4 block also has two parameter-matched variants of NGswin without the context:
| Model | N-gram | Scale | Mult-Adds | Params | Set14 | Urban100 | Manga109 |
|---|---|---|---|---|---|---|---|
| NGswin | w/o | x2 | 138.20G | 750K | 33.70 / 0.9194 | 32.39 / 0.9304 | 38.86 / 0.9775 |
| NGswin | w/ | x2 | 140.41G | 998K | 33.79 / 0.9199 | 32.53 / 0.9324 | 38.97 / 0.9777 |
| NGswin | w/o | x3 | 65.53G | 759K | 30.48 / 0.8451 | 28.37 / 0.8573 | 33.81 / 0.9464 |
| NGswin | w/ | x3 | 66.56G | 1,007K | 30.53 / 0.8456 | 28.52 / 0.8603 | 33.89 / 0.9470 |
| NGswin | w/o | x4 | 35.89G | 771K | 28.70 / 0.7844 | 26.25 / 0.7918 | 30.70 / 0.9123 |
| NGswin, more channels | w/o | x4 | 53.71G | 1,189K | 28.75 / 0.7854 | 26.28 / 0.7927 | 30.73 / 0.9129 |
| NGswin, more depth | w/o | x4 | 47.88G | 1,061K | 28.75 / 0.7853 | 26.37 / 0.7946 | 30.78 / 0.9133 |
| NGswin | w/ | x4 | 36.44G | 1,019K | 28.78 / 0.7859 | 26.45 / 0.7963 | 30.80 / 0.9128 |
| HNCT | w/o | x2 | 82.39G | 357K | 33.65 / 0.9182 | 32.22 / 0.9294 | 38.87 / 0.9774 |
| HNCT-NG | w/ | x2 | 83.19G | 424K | 33.64 / 0.9195 | 32.35 / 0.9306 | 38.94 / 0.9774 |
| HNCT | w/o | x3 | 37.78G | 363K | 30.44 / 0.8439 | 28.28 / 0.8557 | 33.81 / 0.9459 |
| HNCT-NG | w/ | x3 | 38.14G | 431K | 30.48 / 0.8450 | 28.38 / 0.8573 | 33.81 / 0.9464 |
| HNCT | w/o | x4 | 22.01G | 373K | 28.71 / 0.7834 | 26.20 / 0.7896 | 30.70 / 0.9112 |
| HNCT-NG | w/ | x4 | 22.21G | 440K | 28.72 / 0.7846 | 26.23 / 0.7912 | 30.71 / 0.9114 |
Interaction and direction ablation, x2 tab. 5. "Direction" is how many neighbour directions are seen; "Type" is the interaction method:
| Direction | Type | Mult-Adds | Params | Urban100 | Manga109 |
|---|---|---|---|---|---|
| 1 | WSA | 152.41G | 1,238,056 | 32.54 / 0.9322 | 38.90 / 0.9777 |
| 4 | WSA | 139.56G | 935,272 | 32.52 / 0.9317 | 38.92 / 0.9776 |
| 1 | CNN | 139.80G | 1,327,528 | 32.45 / 0.9316 | 38.86 / 0.9775 |
| 2 | CNN | 139.38G | 998,568 | 32.54 / 0.9321 | 38.90 / 0.9776 |
| 4 | CNN | 139.17G | 936,488 | 32.52 / 0.9320 | 38.93 / 0.9777 |
| 2 | WSA (default) | 140.41G | 998,384 | 32.53 / 0.9324 | 38.97 / 0.9777 |
SCDP and extra stages tab. 6. The default has encoder depths 6, 4, 4 at 64, 32, 16 pixels and decoder depth 6 at 64 pixels. The "extra" variant uses four encoder stages of 4 blocks each (down to 8 pixels) and two decoder stages of 2 blocks; the total stays at 20 blocks. Rows give the stage design and SCDP setting at each scale:
| Stages | SCDP | Scale | Mult-Adds | Params | Urban100 | Manga109 |
|---|---|---|---|---|---|---|
| extra | w/o | x2 | 87.98G | 997K | 32.28 / 0.9298 | 38.72 / 0.9773 |
| default | w/o | x2 | 138.88G | 992K | 32.48 / 0.9321 | 38.92 / 0.9776 |
| default | w/ | x2 | 140.41G | 998K | 32.53 / 0.9324 | 38.97 / 0.9777 |
| extra | w/o | x3 | 42.10G | 1,006K | 28.33 / 0.8562 | 33.67 / 0.9453 |
| default | w/o | x3 | 65.85G | 1,001K | 28.47 / 0.8596 | 33.81 / 0.9464 |
| default | w/ | x3 | 66.56G | 1,007K | 28.52 / 0.8603 | 33.89 / 0.9470 |
| extra | w/o | x4 | 23.33G | 1,018K | 26.22 / 0.7900 | 30.46 / 0.9090 |
| default | w/o | x4 | 36.06G | 1,013K | 26.38 / 0.7954 | 30.71 / 0.9121 |
| default | w/ | x4 | 36.44G | 1,019K | 26.45 / 0.7963 | 30.80 / 0.9128 |
Supplement tables. Table A lists which learning strategies the authors found to help NGswin: warm-start over scratch for x3 and x4; normalisation with data-derived std over 1.0; de-normalisation before the loss; half-decay over cosine; and no weight decay, gradient clipping, layer-wise lr decay, dropout or drop-path app. B tab. A. Table B covers other ablations at x2, x3 and x4 (PSNR / SSIM on Set5, Set14, BSD100, Urban100, Manga109) app. C:
Swin version (B(a)), NGswin, Mult-Adds 140.41G / 66.56G / 36.44G:
| Version | Scale | Params | Set5 | Set14 | BSD100 | Urban100 | Manga109 |
|---|---|---|---|---|---|---|---|
| V1 | x2 | 998,176 | 37.99 / 0.9606 | 33.71 / 0.9192 | 32.20 / 0.9000 | 32.28 / 0.9301 | 38.69 / 0.9770 |
| V2 | x2 | 998,384 | 38.05 / 0.9610 | 33.79 / 0.9199 | 32.27 / 0.9008 | 32.53 / 0.9324 | 38.97 / 0.9777 |
| V1 | x3 | 1,006,831 | 34.42 / 0.9273 | 30.44 / 0.8445 | 29.13 / 0.8066 | 28.35 / 0.8569 | 33.66 / 0.9456 |
| V2 | x3 | 1,007,039 | 34.52 / 0.9282 | 30.53 / 0.8456 | 29.19 / 0.8078 | 28.52 / 0.8603 | 33.89 / 0.9470 |
| V1 | x4 | 1,018,948 | 32.20 / 0.8946 | 28.69 / 0.7836 | 27.61 / 0.7380 | 26.26 / 0.7916 | 30.53 / 0.9090 |
| V2 | x4 | 1,019,156 | 32.33 / 0.8963 | 28.78 / 0.7859 | 27.66 / 0.7396 | 26.45 / 0.7963 | 30.80 / 0.9128 |
Padding (B(b)), NGswin, compared with zero padding:
| Padding | Scale | Set5 | Set14 | BSD100 | Urban100 | Manga109 |
|---|---|---|---|---|---|---|
| zero-pad | x2 | 37.82 / 0.9599 | 33.38 / 0.9160 | 32.06 / 0.8983 | 31.58 / 0.9231 | 38.10 / 0.9759 |
| seq-refl-win-pad | x2 | 38.05 / 0.9610 | 33.79 / 0.9199 | 32.27 / 0.9008 | 32.53 / 0.9324 | 38.97 / 0.9777 |
| zero-pad | x3 | 34.14 / 0.9249 | 30.22 / 0.8400 | 28.99 / 0.8030 | 27.74 / 0.8439 | 33.01 / 0.9410 |
| seq-refl-win-pad | x3 | 34.52 / 0.9282 | 30.53 / 0.8456 | 29.19 / 0.8078 | 28.52 / 0.8603 | 33.89 / 0.9470 |
| zero-pad | x4 | 31.90 / 0.8906 | 28.45 / 0.7782 | 27.47 / 0.7332 | 25.72 / 0.7740 | 29.89 / 0.9016 |
| seq-refl-win-pad | x4 | 32.33 / 0.8963 | 28.78 / 0.7859 | 27.66 / 0.7396 | 26.45 / 0.7963 | 30.80 / 0.9128 |
Warm-start (B(c)), SwinIR-NG, scratch against warm-start:
| Training | Scale | Set5 | Set14 | BSD100 | Urban100 | Manga109 |
|---|---|---|---|---|---|---|
| scratch | x3 | 34.65 / 0.9291 | 30.59 / 0.8471 | 29.23 / 0.8090 | 28.71 / 0.8636 | 34.17 / 0.9485 |
| warm-start | x3 | 34.64 / 0.9293 | 30.58 / 0.8471 | 29.24 / 0.8090 | 28.75 / 0.8639 | 34.22 / 0.9488 |
| scratch | x4 | 32.45 / 0.8979 | 28.80 / 0.7867 | 27.71 / 0.7413 | 26.51 / 0.7992 | 31.02 / 0.9158 |
| warm-start | x4 | 32.44 / 0.8980 | 28.83 / 0.7870 | 27.73 / 0.7418 | 26.61 / 0.8010 | 31.09 / 0.9161 |
Table C (ablations on all benchmarks) is cited but not reproduced here app. C; the main-text ablations above are its primary rows.
Paper vs code
Code: official repository rami0205/NGramSwin at code_commit. Paths below
are relative to that repository. The SCDP algorithm, N-gram context, padding,
patch merging, decoder and reconstruction match the text. The differences are:
- Fine-tuning defaults differ from the paper's stated schedule. The paper
states 300 epochs in total with warm-start (50 + 250), 10 warm-up epochs in
whole fine-tuning, and halving after 50, 100, 150, 175, 200, 225 epochs of
that phase app. B. The fine-tuning script defaults to 500 total epochs
ddp_main_finetune.py, 20 warm-up epochs ddp_main_finetune.py,
and a halving list of 200, 300, 400, 425, 450, 475 ddp_main_finetune.py.
The half-list is compared against epochs counted from the end of warm-start
dp_main_finetune.py train_utils.py. The
README commands do not pass
--total_epochs,--warmup_epochor--half_listREADME.md. Running them reproduces the code defaults, not the stated schedule, unless the flags are set explicitly. - Warm-start learning rate. The paper says the lr stays at 4e-4 during warm-start. The code computes a linear ramp for that phase but returns before writing it to the optimizer train_utils.py. The lr therefore stays at its initial value, which matches the text; the ramp is dead code.
- Learning-rate value typo. The paper's warm-up sentence says the target lr is 10^-4 app. B, but everywhere else it is 4e-4. The code uses 4e-4 argfunc_utils.py.
- Cross-stage projection. The code applies a linear layer to each concatenated cascade input, mapping 2D or 3D channels back to D main_branch.py main_branch.py. The text describes the concatenation without this projection §3.3.
- Per-head dimension. The paper gives head counts per stage but no
per-head width app. B. The code uses
D // headswith integer division win_attention.py. See Pitfalls. - Evaluation border. The paper does not say whether border pixels are
excluded. The code removes
scalepixels from each border before Y-channel metrics dp_engine_test.py dp_engine_test.py. - Normalisation constants. The paper says the statistics come from the 800 LR training images app. B. The code hard-codes one mean and one std per scale ngswin.py. The repository contains no script that computes them.
- Compute count. The paper reports Mult-Adds on a 1280 x 720 HR image tab. 2.
The code's MAC printout pads the LR input up to multiples of 32 and then
calls the model's own
flops()estimate dp_builder.py. This is an estimate, not a profiler. - SwinIR-NG configuration. The N-gram size is fixed to 2 inside the SwinIR-NG block swinirng.py. Channel count 60 and blocks [6, 6, 6, 6] are defaults swinirng.py. The text does not restate these for SwinIR-light.
- Pretrained weights. The repository ships six checkpoints under
pretrain/(NGswin and SwinIR-NG at x2, x3, x4). They were not loaded or checked against the tables.
No discrepancy was found in the core modules (win_partition.py, pool.py,
main_branch.py, reconstruction.py) beyond the items above.
Pitfalls
- Head width truncation.
D // headsis integer division. With D = 64 and 6 heads, the encoder's first stage and the decoder use 10-dimensional heads, so attention runs at width 60 and projects back to 64. With 4 heads (stages 2 and 3) the width is 64. The N-gram attention runs at width 32 and shows the same truncation (5-dimensional heads at 6 heads) [code: my_model/ngswin_model/win_attention.py#L45-L46]. Reproduce this exactly or the numbers will differ; the text does not mention it. - Padding index ranges. The N-gram padding copies the N-1 units before the last unit, in forward order, for the lower and right edges. The upper and left edges copy the N-1 units after the first, also in forward order. For N = 2 this equals a mirror; for N > 2 a mirror would give a different result. Copy the code's index ranges win_partition.py.
- Zero padding hurts. The paper says zero padding is worse than no N-gram context at all app. C tab. B(b), so the padding is part of the method.
- Context ordering. The context is computed from the unshifted grid, added to the windows, and only then is the grid cyclically shifted win_partition.py. Applying the context after shifting would change which neighbours each window sees.
- Shift mask. The shifted-window mask uses -100 for cross-region pairs and 0 otherwise, added before softmax nstb.py. The paper does not state the value.
- Temperature. Tau is learnable with a floor of 0.01 and starts at 0.1 in the code win_attention.py [code: my_model/ngswin_model/win_attention.py#L84]. The paper gives only the floor.
- Inference padding. Inputs are padded to multiples of 32 on the right and bottom before normalisation, and the output is cropped back to the original size times the scale ngswin.py ngswin.py. Training patches (64 x 64) need no padding.
- Schedule trap. The README fine-tuning commands produce 500 total epochs
by default, which gives 450 epochs of whole fine-tuning instead of the 250
stated in the paper (see Paper vs code). To match the text, set
--total_epochs 300 --warmup_epoch 10 --half_list "(50,100,150,175,200,225)"on the fine-tuning script. The half-list is compared with epochs counted from the end of warm-start, so these values are the paper's own numbers train_utils.py app. B. - Claim vs table. The text says NGswin outperformed previous models on all benchmarks §4.2. Table 2 contradicts this at x2: Set5 PSNR is 38.05 against 38.10 for SRPN-Lite and FMEN, and Manga109 SSIM is 0.9777 against 0.9778 for FMEN tab. 2.
- SOTA claim vs table. The text says SwinIR-NG sets state-of-the-art results on all benchmarks §4.2. At x2, Set14 SSIM is 0.9205 against 0.9207 for ELAN-light, and Manga109 SSIM is 0.9781 against 0.9783 for SwinIR-light and ELAN-light tab. 3.
- DF2K rows are a different protocol. The SwinIR-NG reduced (DF2K) row and FMEN and SRPN-Lite at x2 and x3 use DF2K data, not DIV2K alone tab. 2 tab. 3. Compare them only with that in mind.
- Run counts. The number of runs behind each figure is not stated, and no variance is reported. Many ablation gaps are 0.01 to 0.1 dB tab. 4 tab. 5. Treat them as author-reported.
- Cost of the context. The N-gram block adds about 2G Mult-Adds at x2 and about 250K parameters for NGswin tab. 4. The authors note that SwinIR-NG needs more parameters and operations than ELAN-light app. E.
- Cosine decay underfits. Cosine decay, weight decay, gradient clipping, layer-wise lr decay and dropout all reduced performance in the authors' runs app. B. The code supports cosine decay train_utils.py but the defaults use half-decay with no regularisation.
- Data scale. Training uses only the 800 DIV2K pairs, with each image repeated 80 times per epoch app. B dataset.py. Adding Flickr2K is a different experiment tab. 2 tab. 3.
- Crop sampling. Random crop offsets are sampled over the range
0 .. lr_size - cropwith an exclusive upper bound, so the last valid offset is never drawn dataset.py. This has negligible effect on the numbers but differs from a standard inclusive sampler.
Lineage
- Builds on: Swin Transformer [arXiv:2103.14030] (window attention, shifted windows, patch merging), Swin V2 [arXiv:2111.09883] (post-normalisation and scaled-cosine attention) and SwinIR-light [arXiv:2108.10257] §3.3 §3.4 README.md. HNCT §2 is the other Swin-based model that receives the N-gram context.
- Also compared against (named in the tables): EDSR-baseline, MemNet, CARN, IMDN, LatticeNet, RFDN-L, SRPN-Lite, FMEN, ESRT, ELAN-light and DiVANet tab. 2 tab. 3. Swin2SR is excluded from the SwinIR-NG comparison because it was trained on more data app. E.
- Superseded by: not known from the content of this wiki.
Source and license
Adapted from “N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution” by Haram Choi, Jeongmin Lee, Jihoon Yang (arXiv:2211.11436), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/rami0205/NGramSwin at commit 8d0d6490954e (Apache-2.0). Any quoted code is shown under that license.