Local Texture Estimator for Implicit Representation Function

In plain words

Super-resolution means producing a sharp, larger version of a small image. A popular trick is a small neural function that, given any position in the picture, outputs the colour there. Because it can be asked about any position, one network can enlarge an image by any factor. The catch is that such functions tend to learn smooth shapes first and struggle with fine texture such as stripes, text edges and grids.

This paper adds a small helper that looks at the low-resolution image around each spot and works out which repeating patterns dominate there, how strong they are and where they line up. The colour function then uses this description to draw fine detail at any zoom level. The plain enlarged image is also added back in, so the network only has to supply the missing detail.

The authors report higher scores than earlier arbitrary-scale methods on most entries of their test tables, and faster timings in their own tests. Those numbers come from the authors' own experiments.

TL;DR

• Arbitrary-scale super-resolution with an implicit MLP tends to miss high-frequency texture (spectral bias) and distorts structure at large scales §1 §2.
• LTE predicts, per latent cell, Fourier amplitudes and frequencies plus a scale-dependent phase, then decodes the sum with a shared 4-layer MLP and adds the bilinearly upscaled low-resolution input §4.1 §3.
• Trained on DIV2K with scales from x1 to x4 and L1 loss, it reports the best DIV2K PSNR among the compared arbitrary-scale methods at every tested scale, including out-of-scale x6 to x30 tab. 1.
• Speed claims depend on the query setting; the abstract's 40% figure is not a direct reading of tab. 4 (see Pitfalls).
• The official code (BSD-3-Clause) matches most of the paper, but the frequency count, the phase bias and the cell clamp differ in detail (see Paper vs code).

Problem

• Storing one super-resolution model per scale factor is costly, and sub-pixel upsamplers are fixed to their training scale §1.
• A standalone MLP learns low frequencies first and fails on fine detail (spectral bias). Fourier feature mappings and sinusoidal activations address this with fixed or random frequencies §1 §2.
• Local implicit image functions (LIIF) handle arbitrary scales but show structural distortion at extreme factors §2.
• Large outputs are memory-heavy at evaluation time, which makes the memory and time cost of arbitrary-scale decoders a practical concern §1 §6.

Method

Overview

The network has four parts §4.1 fig. 2:

1. Encoder E_φ: EDSR-baseline, RDN or SwinIR with their upsampling layers removed, so the feature map has the LR resolution §4.1.
2. LTE h_ψ: an amplitude estimator h_a and a frequency estimator h_f, each a 3x3 convolution with 256 output channels (2K) that is equivalent to a fully connected layer on the unfolded 3x3 neighbourhood. A phase estimator h_p is one fully connected layer with hidden size 128 that takes the cell size §4.1 eq. 5 eq. 7.
3. Decoder f_θ: a 4-layer MLP with ReLU and 256 hidden units, shared by all images §4.1.
4. LR skip connection: the bilinearly upscaled LR image is added to the decoder output §4.1 eq. 8.

The paper's central equation for a query point x is a weighted sum over the four nearest latent cells j eq. 4:

\mathbf s(\mathbf x,\mathbf I^{LR};\Theta,\psi)=\sum_{j\in\mathcal J} w_j\, f_\theta\big(h_\psi(\mathbf z_j,\mathbf x-\mathbf x_j)\big) eq. 4

The LTE term, with the scale-dependent phase added eq. 5 eq. 7, is

h_\psi(\hat{\mathbf z}_j,\boldsymbol\delta,\mathbf c)=\mathbf A_j\odot \begin{bmatrix}\cos\big(\pi(\mathbf F_j\boldsymbol\delta+h_p(\hat{\mathbf c}))\big)\\ \sin\big(\pi(\mathbf F_j\boldsymbol\delta+h_p(\hat{\mathbf c}))\big)\end{bmatrix} eq. 7

with A_j = h_a(z_j), F_j = h_f(z_j) eq. 5 and ĉ = max(c, c_tr), where c_tr is the smallest cell size seen in training §3 eq. 7. The final prediction adds the LR skip eq. 8:

\hat{\mathbf s}(\mathbf x)=\mathbf s(\mathbf x,\mathbf I^{LR};\Theta,\psi)+\mathbf I^{LR}_{\uparrow}(\mathbf x) eq. 8

Steps

1. Encode the LR image into a feature map z of the same height and width as the LR image §4.1 §3.
2. For each latent cell, let the 3x3 convolutional estimators see the 3x3 neighbourhood. The amplitude vector has 2K entries and the frequency matrix has K x 2 entries §3 eq. 5.
3. For each query coordinate, take the four nearest latent cells. Compute the offset δ from each cell centre, using the local grid §3.
4. Compute the phase term from the cell size, clamped to the training minimum §3 eq. 7.
5. Form the Fourier basis, cos and sin of π(Fδ + phase), and multiply by the amplitudes eq. 5 eq. 7.
6. Decode each latent's contribution with the shared MLP, then blend the four results with local ensemble weights w_j eq. 4 §3.
7. Add the bilinearly upscaled LR value at x eq. 8.

# Predict a colour at query coordinate x with cell size c  [§3] [eq. 4]
def query_rgb(x, c, lr_image, features):
    # Encode and predict per-cell amplitudes and frequencies  [§4.1] [eq. 5]
    amplitudes = conv3x3_amp(features)           # (B, 2K, H, W), 2K = 256
    frequencies = conv3x3_freq(features)         # (B, 2K, H, W)
    total = 0
    # Loop over the 4 nearest latent cells  [§3] [code: models/lte.py#L51-L57]
    for j in four_nearest_cells(x):
        # Offset from the cell centre on the local grid  [§3]
        delta = x - centre_j
        # Phase from the clamped cell size  [§3] [eq. 7]
        phase = phase_fc(max(c, c_train))         # (K,)
        # Scaled frequency dot offset, plus phase  [eq. 7]
        angles = pi * (freq_j_matrix @ delta + phase)   # (K,)
        # Fourier basis gated by amplitudes  [eq. 5]
        basis = amp_j * concat(cos(angles), sin(angles))  # (2K,)
        # Decode and blend with local ensemble weight w_j  [eq. 4]
        total = total + w_j * decoder(basis)
    # LR skip connection  [eq. 8]
    return total + bilinear(lr_image, x)

Training minimises the L1 distance between predicted and ground-truth pixels eq. 3 §5.1:

\hat\Theta=\arg\min_\Theta\sum_{m,n}\big\|\mathbf I^{HR}_n(\mathbf x_m)-\mathbf s(\mathbf x_m,\mathbf I^{LR}_n;\Theta)\big\|_1 eq. 3

Hyperparameters

Results

Author-reported. All values are PSNR in dB. Table 1 uses the DIV2K validation set. Table 2 uses four benchmark sets. † marks the authors' own implementation; the paper does not say how the other rows were obtained tab. 1 tab. 2. The paper's red and blue highlighting is not reproduced.

Table 1: DIV2K validation, arbitrary-scale SR tab. 1

Table 2: Benchmark sets, arbitrary-scale SR tab. 2. Columns are x2, x3, x4 (in-scale) and x6, x8 (out-of-scale). The table is laid out here with one row per dataset and method.

Table 3: Ablation on Set14, EDSR-baseline encoder tab. 3 §5.3. -A removes the amplitude estimator, -F keeps 128 estimated frequencies, -P removes the phase estimator and -L removes the LR skip §5.3.

Table 4: Memory and time, x2 SR, 624 x 624 input, EDSR-baseline encoder, RTX 3090 24 GB tab. 4 §6.

Figures are described in the text only: the qualitative comparisons are fig. 3 to fig. 6, the Fourier-space visualisations are fig. 7 to fig. 9, the qualitative ablation is fig. 10, and the discussion figures on DFT, aliasing and Gibbs ringing are fig. 11 to fig. 13. Figure numbers follow the order of appearance in the LaTeX source.

Paper vs code

Repository: github.com/jaewon-lee-b/lte at a8815d6, linked from the paper's author block. The GitHub owner matches the first author's name, and the LICENSE credits jaewon-lee-b LICENSE. Licence is BSD-3-Clause.

• Frequency count. The paper says the amplitude and frequency convolutions produce 2K = 256 channels §4.1. The code applies the same 256 channels as 128 (frequency, pair) entries, so K = 128 lte.py lte.py. The ablation text, however, says -F estimates "only 128 frequencies (not 256)" §5.3, which implies the full model uses 256. The code matches K = 128, not 256. Follow the code.
• Phase estimator. The paper describes it as a single FC layer with hidden size 128 §4.1. The code is Linear(2, 128, bias=False) lte.py, so it has no bias term and maps the cell size to a 128-vector.
• Cell clamp. The paper defines ĉ = max(c, c_tr) for the phase input §3 eq. 7. The model itself does not clamp. The clamp is applied only at test time, by multiplying the cell by max(scale / scale_max, 1), with scale_max = 4 test.py test.py. Training scales lie in [1, 4], so the clamp never changes a training cell.
• Clamp only in batched evaluation. When eval_bsize is unset, the test path passes the raw cell and skips the clamp test.py. The shipped configs set eval_bsize: 30000, so the clamped path is the one used test-div2k-6.yaml. Apply the clamp in every path if you reimplement it.
• Downsampling. The paper says bicubic resizing "in PyTorch" §5.1. The code resizes with PIL's bicubic filter via torchvision wrappers.py.
• Undocumented training details. The code adds random flips and a transpose wrappers.py, normalises inputs to [-1, 1] train_edsr-baseline-lte.yaml, and repeats the training set 20 times train_edsr-baseline-lte.yaml. None of these appear in the paper.
• PSNR protocol. The paper does not state it. The code computes DIV2K PSNR on RGB with a border of scale + 6 pixels shaved utils.py. For the benchmark sets it converts to luminance with fixed weights and shaves scale pixels utils.py.
• SwinIR evaluation. The code pads the LR input by reflection to a multiple of the window size before inference test.py. The README runs SwinIR with window size 8 README.md. The paper does not describe this padding.
• Decoder depth. The paper says "4-layer MLP" §4.1. The code has three hidden layers plus an output layer train_edsr-baseline-lte.yaml mlp.py. Under the "output layer counted" reading, these agree.
• Local ensemble weights. The paper calls w_j a bilinear interpolation weight §3. The code weights each corner by the area of the opposite corner, swapping the indices lte.py. This is the standard local-ensemble rule of LIIF. Implement it as the code does.
• Training loss and schedule. L1 loss, Adam, learning rates, batch sizes, epochs and milestones match train.py train_edsr-baseline-lte.yaml train_swinir-lte.yaml.
• LTE+. The paper describes it only in the discussion and in tab. 4, as a 1x1-convolution decoder §6. The code is lte-fast ltep.py, with three 1x1 hidden layers and a 1x1 output ltep.py. The paper gives no training details for it.
• Ablations and Fourier figures. No ablation configs are shipped. The Fourier visualisations are in Eval-Fourier-Feature-Space.ipynb, which was not read for this card.

Pitfalls

• Frequency count. Implement K = 128 (256 amplitude channels) as the code does. The ablation text's "256 frequencies" reading is not what the code computes.
• The 40% claim. The abstract says memory and time fall by 40%. Table 4 does not show that for LTE itself. At 9216 queries LTE is 36% faster than LIIF (2912 vs 4559 ms) but uses more memory (2.3 vs 1.9 GB). At 1.6M queries LTE takes 925 ms against LIIF's 873 ms, which is slower, but uses less memory (10.2 GB against 11.4 GB). The large saving appears only for LTE+ (483 ms, 7.1 GB), which is a different decoder §6 tab. 4.
• Four decoder calls per query. The local ensemble runs the decoder four times per query point lte.py. Cost grows with the number of queries, which is why the paper's timing depends on how many points are queried at once §6.
• Cell size is a hidden input. The phase term reads the cell size. Training cells are 2 / (HR patch side) wrappers.py, and test cells are 2 / (LR side x scale) test.py. A different convention shifts the phase, so keep cells consistent with training.
• Out-of-scale results. Training scales stop at x4. Results at x6 to x30 depend on the test-time cell clamp described above §5.2. Apply it, or those results will not match.
• In-scale comparisons are not like-for-like. RDN and SwinIR are trained at fixed scales, so they win some in-scale entries in Table 2. For example, RDN scores 38.24 at Set5 x2, against 38.23 for RDN-LTE, and SwinIR scores 38.35, against 38.33 for SwinIR-LTE tab. 2 §5.2.
• Encoder dependence. EDSR-baseline-LTE captures mostly low-frequency components, while SwinIR-LTE captures diagonal frequencies §5.4 fig. 8. Encoder choice changes the Fourier behaviour, not just the PSNR.
• Ablation gaps are small. In tab. 3 most differences are at most 0.15 dB. -A is 0.30 dB below LTE at x3 and within 0.01 dB at x4. The number of runs and their variance are not reported.
• Failure modes the paper names. Severe aliasing in the LR input makes the estimated dominant frequencies disagree with the ground truth §6 fig. 12. Ringing (Gibbs) overshoot appears at large scale, such as x12 §6 fig. 13.
• Training hardware and time are not stated, so the training budget cannot be reproduced from the paper alone.

Lineage

• Builds on: LIIF (local implicit image function, cited as chen2021learning), which the paper extends by adding the Fourier-style estimator, the phase term and the LR skip §2 §3. Also on EDSR-baseline, RDN and SwinIR as encoders §4.1. Fourier feature mapping and NeRF-style position encoding inspired the frequency term §3.
• Compared against: MetaSR (hu2019meta), LIIF, and RDN and SwinIR trained at fixed scales §5.2.
• Superseded by: not stated in the paper.