wikillm

Paper · 2021-08-17 · cs.CV

Light Field Image Super-Resolution with Transformers

LFT super-resolves 5x5 light-field images with two alternating Transformer blocks: an angular one that attends across the views at each pixel, and a spatial one that attends across pixels within each view, restricted to a local window in the released code. Author-reported PSNR/SSIM is best or near-best on five public datasets at 2x and 4x, with about 1.1-1.2M parameters.

Image super-resolutionSelf-attentionMasked attentionVision Transformer (ViT) backboneslight-fieldsuper-resolutiontransformerself-attention

arXivPDFCodeMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A light field photo records the scene from many slightly shifted viewpoints at once, like a small grid of cameras. Making each of those views sharper is the task here: take the low-resolution grid and produce a high-resolution grid.

The method treats the views as a set of pictures that should agree with one another. At each pixel position it lets the views compare notes, so a detail seen clearly in one view can help another view where that detail is hidden, for example behind an occluding edge. Within each view it also lets pixels compare with other pixels nearby, so the network can use context that is farther away than a small filter would see. These two kinds of comparison alternate a few times, and the result is a sharper grid that the model produces with a comparatively small number of parameters.

TL;DR

LFT is a Transformer for light-field super-resolution that alternates an angular attention block (across the 5x5 views at each pixel) with a spatial attention block (across pixels inside each view). Its spatial block is restricted to a 5x5 neighbourhood in the released code, which the paper does not say. The paper reports the best or near-best PSNR/SSIM on five public datasets at 2x and 4x, with the lowest FLOPs in its comparison table and a parameter count near the smallest baselines. Everything is author-reported.

Problem

Light-field SR must use two kinds of information: angular information (the same scene point seen from several views) and spatial information (structure within each view). The paper argues that CNN-based methods use only part of the views, or mix angular information without regard to view position and content, and that their convolutions have a local receptive field, so they miss long-range spatial dependencies §I §II-A. The paper assumes a square array of views, U = V = A, with A = 5 in all experiments §III.

Method

The network has three stages: initial feature extraction by cascaded 3x3 convolutions, a stack of alternating angular and spatial Transformer blocks, and up-sampling §III fig. 1.

  1. Input and initial features. Each sub-aperture image (SAI) is a view. Cascaded 3x3 convolutions produce features F with shape U x V x H x W x C §III-A. The released code splits this into a first convolution and three more with LeakyReLU, plus a skip LFT.py.

  2. Angular Transformer (per pixel, across views) §III-A. Features are reshaped into angular tokens T_Ang with N_A = U V = A^2 tokens per pixel location. Sinusoidal angular position codes P_A are added to the tokens eq. 1 eq. 2. Queries and keys are the layer-normalised sum of token and code; values are the raw tokens §III-A. Multi-head self-attention runs over the views, then a layer-normalised MLP with a residual connection eq. 4 eq. 5 eq. 6.

  3. Spatial Transformer (per view, across pixels) §III-B. Each pixel is expanded to its 3x3 neighbourhood and mapped by an MLP into a spatial token eq. 7. Sinusoidal 2D position codes P_S are added eq. 8 eq. 9. Queries and keys come from the layer-normalised sum; values are the tokens eq. 10 eq. 11. Multi-head self-attention is followed by the same layer-normalised MLP with a residual connection eq. 12 eq. 13. The paper describes these tokens as "overlapping" and gives no window size for the attention.

  4. Alternation. The angular and spatial blocks alternate, and features pass to the next block §III-A §III-B. The number of blocks is not stated in the paper; the code uses four LFT.py.

  5. Up-sampling. The paper says pixel shuffling produces the super-resolved light field of size U x V x alpha H x alpha W §III-B. The released code also adds a bicubic-upscaled copy of the input (see Paper vs code).

Pseudocode
# LFT forward pass  [§III] [fig. 1]
# Input:  lr (B, 1, A*h, A*w), Y channel, A = 5
# Output: sr (B, 1, A*h*s, A*w*s)

def lft_forward(lr, scale):
    # Bicubic copy of the input, added back at the end  [code: LFT.py#L54]
    upscaled = bicubic_upscale(lr, scale)
    # Split the mosaic into views: (B, C, A2, h, w)  [§III]
    views = to_views(lr)
    # Initial convolutions with a skip  [§III-A] [code: LFT.py#L65-L66]
    feats = init_conv0(views)
    feats = init_convs(feats) + feats
    # Position codes, shared by all blocks  [eq. 1] [eq. 2] [eq. 8] [eq. 9]
    ang_pos = angular_position(A * A, C)
    spa_pos = spatial_position(h, w, C)
    # Alternate angular and spatial blocks, with a skip around the stack
    # [§III]; the block count is four in the code [LFT.py#L15]
    skip = feats
    for block in range(num_blocks):
        feats = angular_block(feats, ang_pos)     # [§III-A]
        feats = spatial_block(feats, spa_pos)     # [§III-B]
    feats = feats + skip                          # [code: LFT.py#L76]
    # Pixel shuffle to the target grid, then a 3x3 conv  [§III-B]
    # [code: LFT.py#L39-L44]
    sr = upsample(from_views(feats), scale)
    return sr + upscaled                          # [code: LFT.py#L81]
Pseudocode
# Angular Transformer block  [§III-A] [eq. 4] [eq. 5] [eq. 6]
# tokens: (A2, B*h*w, C); one token per view at each pixel

def angular_block(feats, ang_pos):
    tokens = to_angular_tokens(feats)              # (A2, B*h*w, C)
    # Queries and keys: layer norm of token plus position; values: raw token
    qk = layer_norm(tokens + ang_pos)
    # Multi-head self-attention over the views, with residual  [eq. 4] [eq. 5]
    tokens = multi_head_attention(qk, qk, tokens) + tokens
    # Layer norm, MLP, residual  [eq. 6]
    tokens = mlp(layer_norm(tokens)) + tokens
    return from_angular_tokens(tokens, feats)
Pseudocode
# Spatial Transformer block  [§III-B] [eq. 7] to [eq. 13]
# tokens: (B*A2, h*w, C_s); one token per pixel of each view

def spatial_block(feats, spa_pos):
    # Local embedding: 3x3 neighbourhood of each pixel, then an MLP [eq. 7]
    patches = unfold_3x3(feats)                    # (B*A2, h*w, 9C)
    tokens = linear_no_bias(patches)               # (B*A2, h*w, C_s)
    # Queries and keys: layer norm of token plus position  [eq. 10]
    qk = layer_norm(tokens + spa_pos)
    # Attention with a mask: keys outside the local window are removed
    # [code: model/LFT.py#L118-L162, L177-L187]; the paper states no window
    mask = local_window_mask(h, w, radius=2)       # (h*w, h*w)
    tokens = masked_attention(qk, qk, tokens, mask) + tokens  # [eq. 12]
    # Layer norm, MLP, residual  [eq. 13]
    tokens = mlp(layer_norm(tokens)) + tokens
    return linear_to_channels(tokens, feats)       # back to C channels
Pseudocode
# Local window mask for spatial attention  [code: model/LFT.py#L148-L162]
# A key is allowed when it lies within `radius` of the query pixel;
# radius = 2 gives a 5x5 window (the code's kernel_search = 5)

def local_window_mask(height, width, radius=2):
    rows = arange(height * width) // width         # (h*w,)
    cols = arange(height * width) % width          # (h*w,)
    near_rows = abs(rows[:, None] - rows[None, :]) <= radius
    near_cols = abs(cols[:, None] - cols[None, :]) <= radius
    return near_rows & near_cols                   # (h*w, h*w), True = allowed
Pseudocode
# Sinusoidal angular position code  [§III-A] [eq. 1] [eq. 2]
# The paper uses p = 1..A^2 and alpha = 10000; even channels use sin,
# odd channels use cos, with the same frequency for channels 2i and 2i+1

def angular_position(num_views, dim):
    p = arange(1, num_views + 1)                   # (A2,)
    i = arange(dim // 2)                           # (dim/2,)
    angle = p[:, None] / 10000 ** (2 * i / dim)    # (A2, dim/2)
    code = empty(num_views, dim)
    code[:, 0::2] = sin(angle)
    code[:, 1::2] = cos(angle)
    return code                                    # (A2, dim)

Key equation: the angular attention is softmax over scaled query-key products, then multiplied by values eq. 3. The paper defines it per head with square projections of size d_A / N_H eq. 3.

Hyperparameters

Values marked "not stated" do not appear in the paper. Where the released code sets a value, the code citation is given, and the code is the only source for it.

NameValueWhereNotes
angular resolution A5 x 5§IV-Aall train and test images
training patch (HR)64 (2x), 128 (4x)§IV-ALR patch 32 x 32 §IV-A; data generated with 50% overlap Generate_Data_for_Training.m
batch size4 (4x), 8 (2x)§IV-Acode default is 4 option.py; README uses 8 for 2x
learning rate2e-4§IV-Aoption.py
lr schedulehalved every 15 epochs§IV-AStepLR, gamma 0.5 train.py
epochs80§IV-Acode default is 50 option.py; see Paper vs code
optimizerAdam, betas (0.9, 0.999), eps 1e-8§IV-Ano weight decay train.py
weight initXavier§IV-Acode's weights_init does nothing; see Paper vs code
lossnot stated§IV-Acode uses L1 LFT.py
metricPSNR, SSIM§IV-Aaveraged over M x A^2 scores; code uses Y channel utils.py
sinusoid base alpha10000§III-A §III-Bsame in the code LFT.py
channels Cnot statednot in papercode default 64 option.py
attention headsnot statednot in papercode uses 8 LFT.py
number of blocksnot statednot in papercode uses 4 LFT.py
spatial token dim d_Snot statednot in papercode uses 2C LFT.py
spatial attention windownot statednot in papercode uses 5 x 5 LFT.py
MLP hidden widthnot statednot in papercode uses 2x width with ReLU, no bias LFT.py
test patch / stridenot statednot in papercode: 32 / 16 in LR pixels option.py

Results

Author-reported. Five public light-field datasets (EPFL, HCInew, HCIold, INRIA, STFgantry), all with 5 x 5 views. The authors retrained every baseline on the same training data. Baselines: VDSR, EDSR, RCAN (single-image) and resLF, LFSSR, LF-ATO, LF-InterNet, LF-DFnet, MEG-Net, LF-IINet, DPT (light-field) §IV-B tab. I. The paper does not name the colour channel; the code computes PSNR/SSIM on Y utils.py.

Headline comparison, LFT against LF-IINet, the strongest baseline in most cells tab. I:

DatasetLFT 2xLF-IINet 2xLFT 4xLF-IINet 4x
EPFL34.80 / 0.97834.68 / 0.97729.25 / 0.92129.11 / 0.920
HCInew37.84 / 0.97937.74 / 0.97931.46 / 0.92231.36 / 0.921
HCIold44.52 / 0.99544.84 / 0.99537.63 / 0.97437.62 / 0.974
INRIA36.59 / 0.98636.57 / 0.98531.20 / 0.95231.08 / 0.952
STFgantry40.51 / 0.99439.86 / 0.99431.86 / 0.95531.21 / 0.950

The paper says LFT is "competitive" on all five datasets §IV-B. On HCIold at 2x LF-IINet scores higher PSNR (44.84 against 44.52) tab. I. The STFgantry gain of 0.65 dB at 2x over LF-IINet matches the text §IV-B.

Model size and cost, average over the five datasets tab. II. FLOPs are for a 5 x 5 x 32 x 32 input tab. II.

MethodParams 2xFLOPs 2xPSNR/SSIM 2xParams 4xFLOPs 4xPSNR/SSIM 4x
LF-IINet4.84M56.16G38.74 / 0.9864.89M57.42G32.08 / 0.944
DPT3.73M57.44G38.40 / 0.9853.78M58.64G31.92 / 0.941
LFT (ours)1.11M28.00G38.85 / 0.9861.16M29.45G32.28 / 0.945

Ablation at 4x, PSNR on EPFL / HCIold / INRIA tab. III:

ModelAngTrAngPosSpaTrSpaPosParamsEPFLHCIoldINRIA
11.49M28.6337.0030.66
2x1.42M28.8537.2930.93
3xx1.42M28.9837.3830.93
4x1.28M28.9337.3030.97
5xx1.28M28.9537.4130.98
6 (LFT)xxxx1.16M29.2537.4631.20

Qualitative results and a demo video are described §IV-B and shown in fig. 2 and fig. 3. fig. 4 shows angular attention maps for one scene (Cards): the maps of the angular Transformer adapt to occlusion, while the maps of LF-InterNet link mainly adjacent views §IV-C.

Paper vs code

Code: https://github.com/ZhengyuLiang24/LFT, commit 964fc60bf5ababc6a29e6bf2b1ec6d7387f630c1, MIT. The repository owner (ZhengyuLiang24) matches the first author, and the paper's abstract names the link.

  • Spatial attention window. The paper says the spatial Transformer captures "long-range dependencies" within each SAI §I §III-B. The code builds a mask so each pixel attends only to a 5 x 5 neighbourhood LFT.py LFT.py LFT.py. Reach beyond two pixels then comes from stacking blocks; this is inferred from the code, not stated in the paper.
  • Bicubic skip. The output is the pixel-shuffled features plus a bicubic upscaling of the input LFT.py LFT.py. The paper's output description has only pixel shuffling §III-B.
  • Up-sampling head. The code uses a 1x1 convolution to C s^2 channels, pixel shuffle, LeakyReLU(0.2), and a 3x3 convolution to one channel LFT.py. The paper names only pixel shuffling §III-B.
  • Global skips. The code adds a skip around the stack of blocks and around the initial convolutions LFT.py LFT.py. The paper does not mention them.
  • Initial convolutions. The paper says "cascaded 3x3 convolutions" §III-A. The code's first convolution has no activation, and the next three are separated by LeakyReLU(0.2) LFT.py.
  • Positional encoding scale. Equations 8 and 9 sum a row term and a column term eq. 8 eq. 9. The code averages them, dividing by 2 LFT.py.
  • Position index. The paper uses p = 1..A^2 for angular and (1,1)..(H,W) for spatial positions §III-A §III-B. The code's grid starts at 0 LFT.py.
  • Channel layout of the codes. The paper interleaves sin and cos on alternate channels eq. 1 eq. 2. The code puts all sin terms first and all cos terms second LFT.py. With learned projections after the add, this changes which channel the weights see; treat it as a different parameterisation, not an equivalent one.
  • Attention projections. The paper defines per-head projections of size d_A/N_H eq. 3 §III-A. The code uses PyTorch nn.MultiheadAttention with full-width query, key and value projections and no bias LFT.py LFT.py.
  • Spatial tokens. The paper calls the spatial tokens "overlapping" and cropped §III-B. The code makes one token per pixel from a 3x3 unfold LFT.py, so N_S = h x w. For a token count, use the code.
  • Epochs. The paper says training stops after 80 epochs §IV-A. The code's default is 50 option.py, and the released checkpoints are named epoch_50 [README]. Whether those checkpoints were trained for 80 epochs is not stated.
  • Initialisation. The paper says Xavier §IV-A. The code's weights_init is empty, so layers keep PyTorch defaults; only the attention input projections use Kaiming-uniform LFT.py LFT.py LFT.py.
  • Loss and augmentation. The paper states no loss and no augmentation. The code uses L1 loss LFT.py and random flips and a transpose of views and pixels utils_datasets.py.
  • Colour space. The paper does not say. The code trains and tests on the Y channel of YCbCr Generate_Data_for_Training.m utils.py.
  • Training device. train.py and test.py set the device to CPU and comment out the CUDA line, although they call torch.cuda.set_device train.py test.py. Running on GPU needs an edit.
  • Test patching. The code cuts overlapping 32 x 32 LR patches with stride 16, mirror-extends the border, and stitches the central part back utils.py utils.py. The paper states no test-time patching.

Pitfalls

  • Equation numbers. Equations are numbered here in source order from the LaTeX file (13 display equations, eq. 1 to eq. 13); the paper's labels do not match the printed numbers. Check against the PDF before citing a number.
  • Local attention, dense cost. The mask keeps spatial attention local, but the implementation still builds a full hw by hw attention matrix and applies the mask after the scores LFT.py. Memory and FLOPs grow with (h*w)^2, not with the window size. Each forward pass also rebuilds the mask in Python loops LFT.py.
  • Angular cost. Angular attention runs over A^2 = 25 tokens per pixel, which is cheap, but it repeats for every pixel location §III-A LFT.py.
  • Unstated dimensions. Channels, heads, number of blocks, MLP width, and the spatial token size d_S are not in the paper. Implementing from the paper alone gives a model with different sizes and different parameter counts from the 1.11M and 1.16M in tab. II.
  • Bicubic residual. The bicubic skip changes what the network learns: it predicts a correction to bicubic LFT.py LFT.py. Without it, a re-implementation computes a different function.
  • Mismatched training schedule. The paper's 80 epochs and the code's 50-epoch default differ. Results at either length are not reproduced by the other setting, and the paper does not say which setting produced Table I.
  • Mask values. The code's mask adds -inf to disallowed keys and 0 to allowed ones LFT.py. An implementation that multiplies by a 0/1 mask instead changes the softmax.
  • Table I typo. The VDSR row's 2x INRIA cell reads 0,974 instead of 0.974 tab. I; read it as 0.974.
  • Code bug in the stitching step. LFintegrate slices with v*ph where the column offset should use pw utils.py. Results are only correct for square inputs.
  • Author-reported only. The comparisons use baselines the authors retrained themselves §IV-B. No variance or repeat runs are reported. The paper's cost claims come from its FLOPs and parameter counts tab. II.

Lineage

  • Builds on: multi-head self-attention and Transformers, for which the paper cites standard references §II-B; the local 3x3 feature embedding from LIIF §III-B; pixel shuffling §III-B; ViT for patch tokens and classification §II-B. Light-field SR is built on CNN methods: resLF, LFSSR, LF-InterNet, LF-DFnet, LF-IINet, MEG-Net, LF-ATO, HDDRNet §II-A.
  • Compared against: VDSR, EDSR, RCAN, resLF, LFSSR, LF-ATO, LF-InterNet, LF-DFnet, MEG-Net, LF-IINet, DPT §IV-B tab. I tab. II.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “Light Field Image Super-Resolution with Transformers” by Zhengyu Liang, Yingqian Wang, Longguang Wang, Jungang Yang, Shilin Zhou (arXiv:2108.07597), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ZhengyuLiang24/LFT at commit 964fc60bf5ab (MIT). Any quoted code is shown under that license.