Fusformer: A Transformer-based Fusion Approach for Hyperspectral Image Super-resolution

In plain words

A hyperspectral camera records many bands of light for every pixel, but the picture it makes is coarse. A normal colour photo of the same scene is sharp, but it has only a few bands. Fusformer combines the two: it takes the coarse multi-band image and the sharp colour image and produces a sharp multi-band image.

The network treats every pixel as one item in a sequence and lets each item look at all the others. Older networks of this kind looked only at a small neighbourhood around each pixel. This global view is the transformer's main contribution here.

The network does not draw the full output from scratch. It starts from a stretched-up copy of the coarse image and learns only the missing detail, which is a smaller job. The model is small, about a tenth of a million weights according to the authors. On their test set it scores above several earlier methods, but all of these numbers are the authors' own.

TL;DR

• Fuses a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) into a high-resolution hyperspectral image §1 §2.2.
• Each pixel is one token. A 2-block encoder and a 2-block decoder use self-attention and MLPs, and a small refine head predicts a residual §2.2.1 eq. 2 eq. 3 model.py.
• Author-reported: best PSNR, SAM and ERGAS on the CAVE test set, and best PSNR on the Harvard generalisation set, both at 0.10M parameters tab. 1 tab. 2.
• Code-level caveats: the reported test path runs attention on 64×64 patches, and the code's decoder reading differs from Eq. 3 main.py.

Problem

Hyperspectral images have many spectral bands but poor spatial resolution. Sensors cannot give high spatial and high spectral resolution at once, so the usual remedy is to fuse a low-resolution hyperspectral image with a high-resolution multispectral image §1. Model-based fusion methods need the blur and spectral response operators, which are hard to obtain in practice, and their hand-chosen regularisers must be tuned for each device §1. CNN methods do better, but their convolutions see only a local neighbourhood, so the global structure is left out §2 §2.1.

The paper's premise is that the low-resolution hyperspectral input already carries most of the spectral structure, so a network only needs to estimate the spatial detail, which shrinks the mapping it has to learn §1 §2.2.

Method

Overview. The network takes the bicubically upsampled LR-HSI and the HR-MSI as input, concatenates them along the spectral axis, embeds each pixel, applies a transformer, reshapes the result to an image and refines it into a residual that is added to the upsampled input fig. 1 §2.2 §2.2.1.

Steps.

1. Upsample the LR-HSI to the HR-MSI grid. The paper says only "upsampled"; the code uses bicubic interpolation with a factor of 4 and clips to [0, 1] §2.2 model.py.
2. Concatenate the upsampled LR-HSI and the HR-MSI along the spectral axis, giving a cube with S + s channels §2.2.1.
3. Flatten the cube so each pixel is one token of length S + s. The paper argues this suits pixel-wise super-resolution, unlike ViT-style patches, because a patch of a 31-band image would be very long §2.2.1.
4. Embed each token with one fully connected layer to F = 48 channels §2.2.1.
5. Encoder: a stack of pre-norm blocks. Each block applies multi-head self-attention with a residual connection, then an MLP with a residual connection eq. 2 §2.2.2.
6. Decoder: a stack of pre-norm blocks with two attention layers and one MLP each eq. 3 §2.2.2. The code's decoder does not match this reading, see Paper vs code.
7. Reshape the F-channel tokens back to H × W × F and apply the refine module (shown as "Reshape & Refine" in fig. 1). The code uses a 3×3 convolution, a LeakyReLU, and a 3×3 convolution down to S bands model.py.
8. Add the upsampled LR-HSI to the refined output. This residual learning is the paper's stated reason the network learns a smaller mapping §2.2 §3.2.

Self-attention. Queries, keys and values are linear maps of the same token matrix. Attention applies a softmax to scaled query-key products and weights the values with the result. The paper divides by √d_k and does not state how d_k relates to the heads eq. 1 §2.2.2.

\mathrm{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V}) = \operatorname{softmax}\!\left(\frac{\mathbf{Q}\mathbf{K}^{T}}{\sqrt{d_k}}\right)\mathbf{V} eq. 1

Encoder block. With layer normalisation LN, multi-head attention MHA and MLP written as in the paper:

\mathbf{X}' = \operatorname{MHA}(\operatorname{LN}(\mathbf{X})),\quad \mathbf{X} = \mathbf{X} + \mathbf{X}',\quad \mathbf{X}' = \operatorname{MLP}(\operatorname{LN}(\mathbf{X})),\quad \mathbf{X} = \mathbf{X} + \mathbf{X}' eq. 2

Decoder block. As in eq. 2, with a second MHA before the MLP:

\mathbf{X}' = \operatorname{MHA}(\operatorname{LN}(\mathbf{X})),\quad \mathbf{X} = \mathbf{X} + \mathbf{X}',\quad \mathbf{X}' = \operatorname{MHA}(\operatorname{LN}(\mathbf{X})),\quad \mathbf{X} = \mathbf{X} + \mathbf{X}',\quad \mathbf{X}' = \operatorname{MLP}(\operatorname{LN}(\mathbf{X})),\quad \mathbf{X} = \mathbf{X} + \mathbf{X}' eq. 3

Forward pass as implemented (the paper does not give this level of detail; the code fixes the depth and widths):

# Fusformer forward pass  [§2.2] [code: model.py#L50-L65]
# Input:  lr_hsi (B, S, h, w) in [0, 1]; hr_msi (B, s, H, W)
# Output: hr_hsi (B, S, H, W) in [0, 1]; H = 4h in the code

def fusformer(lr_hsi, hr_msi):
    # Bicubic upsampling by 4; factor not stated in the paper
    # [code: model.py#L52]
    up_hsi = bicubic_upsample(lr_hsi, scale=4)     # (B, S, H, W)
    # Clip the upsampled input to [0, 1]  [code: model.py#L53]
    up_hsi = clip(up_hsi, 0, 1)                    # (B, S, H, W)
    # Concatenate along the spectral axis  [§2.2.1] [code: model.py#L55]
    data = concat(up_hsi, hr_msi, axis=channel)    # (B, S+s, H, W)
    # One token per pixel  [§2.2.1] [code: model.py#L56]
    tokens = flatten_pixels(data)                  # (B, HW, S+s)
    # Linear embedding to F = 48 channels  [§2.2.1] [code: model.py#L57]
    tokens = linear_embed(tokens, out_dim=48)      # (B, HW, F)
    # Encoder: 2 pre-norm blocks  [eq. 2] [code: model.py#L143-L159]
    code = encoder(tokens, blocks=2)               # (B, HW, F)
    # Decoder: 2 blocks, two attention layers each  [eq. 3]
    # [code: model.py#L162-L181]
    features = decoder(code, blocks=2)             # (B, HW, F)
    # Back to an image grid  [§2.2] [code: model.py#L60]
    features = unflatten_pixels(features, H, W)    # (B, F, H, W)
    # Refine to S bands  [fig. 1] [code: model.py#L61]
    residual = refine(features)                    # (B, S, H, W)
    # Residual learning: add the upsampled input  [§2.2] [code: model.py#L62]
    output = residual + up_hsi
    # Clip the output to [0, 1]  [code: model.py#L63]
    return clip(output, 0, 1)

Training step as implemented. The paper does not state a loss; the code uses an L1 loss on the output against the ground truth.

# One training step  [code: main.py#L107-L123]
# The paper does not state the loss; the code uses L1  [code: main.py#L114]

def train_step(model, optimizer, gt, lr_hsi, hr_msi):
    # Clear gradients from the previous step  [code: main.py#L110]
    optimizer.zero_grad()
    # Predict the high-resolution hyperspectral image  [code: main.py#L112]
    output = model(lr_hsi, hr_msi)
    # Mean absolute error over pixels and bands  [code: main.py#L114]
    loss = mean(abs(output - gt))
    # Backpropagate and update the weights  [code: main.py#L122-L123]
    loss.backward()
    optimizer.step()
    return loss

Hyperparameters

Values marked "not stated" are absent from the compiled paper. Values from the code are the defaults in the repository at code_commit; they are not in the paper.

Results

Author-reported. Setup: 20 CAVE images for training; testing on 11 CAVE images tab. 1 and on 10 Harvard images as a generalisation test tab. 2 §3. Baselines: FUSE, GLP-HS, CSTF and CNN-FUS (traditional); SSRNet, ResTFNet, MHF-Net and HSRnet (deep learning) §3. Hardware, training time and the parameter-count method are not stated.

CAVE, 11 test images tab. 1

Harvard, 10 test images (generalisation) tab. 2

On CAVE, Fusformer has the best PSNR, SAM and ERGAS, and ties HSRnet on SSIM at 0.995 tab. 1. On Harvard it has the best PSNR and ties HSRnet on SAM and SSIM. The paper says ERGAS is the only index where it is not the smallest; CSTF has 2.43 against 2.48 tab. 2 §3.1. The paper says HSRnet's generalisation is close to Fusformer's but uses far more parameters §3.1.

Ablation: residual learning on CAVE tab. 3 (mean ± standard deviation, as stated in the caption):

The paper reads this as showing that adding the upsampled input is vital and improves stability §3.2. Only this one ablation is reported.

Visual comparison on one CAVE and one Harvard sample, with residual maps, is in fig. 2; the paper says the residuals of Fusformer are the darkest §3.

Paper vs code

Searched for an official repository. The paper and arXiv metadata have no code link and no comment field. A GitHub search for "Fusformer" returned J-FHu/Fusformer. Its owner is J-FHu, whose profile name is "Jinfan Hu" (the first author's name without the hyphen), and the account forks liangjiandeng/HSRnet, a repository under the account of co-author Liang-Jian Deng. The match rests on name and the forked repository, not on a link from the paper, so check it before relying on it. Cloned at b075443ec51b958c6ae0f3cfc01eaf3a15ea7257. The repository has no LICENSE file, so no code is quoted here; code is cited by path and line only.

• Decoder. Paper: decoder as in eq. 3, with two MHA layers per block, "using both the encoder and decoder part of the original transformer" §2.2.2. Code: the decoder takes only the encoder output, and both attention layers in each block are self-attention on that same tensor; there is no separate memory input model.py.
• Depth. Paper: not stated. Code: 2 encoder blocks and 2 decoder blocks model.py model.py model.py.
• Positional information. Paper: silent. Code: each transformer defines a learned positional embedding, but its addition is commented out, so the model uses no position information model.py model.py model.py model.py.
• Test-time patching. Paper: relations are modelled "globally" over the image §1 §2.2.1. Code: test() splits each test image into 64 patches of 64×64 and runs the network on each, and reshape2big.m reassembles them main.py main.py reshape2big.m. The README says patches are for limited GPU memory README.md. Whether the reported numbers used patches is not stated.
• Loss, optimizer, schedule, epochs, batch size. Paper: none stated. Code: L1 loss, Adam at 1e-3, StepLR, 1000 epochs, batch size 3 main.py main.py.
• Input bands. Paper: the number of MSI bands s is not stated. Code: 3 channels model.py; README: the demo MSI is RGB README.md.
• Parameter count. Paper: 0.10M tab. 1 tab. 2. A hand count of the weights used in the forward pass gives about 0.11M (not run). The two unused positional tensors, each 4096 × 48, add about 0.39M more to any parameter count taken from the module. The shipped checkpoint Trained_model/1000.pth is 6.1 MB, which fits about 0.5M float32 weights plus Adam state. The paper's 0.10M matches the forward-pass weights.
• Weight initialisation. Paper: silent. Code: a He-init helper is defined but never called, so PyTorch defaults apply model.py.
• Trained weights. The repository ships Trained_model/1000.pth. Its license is not known, so this card does not use it.

Pitfalls

• Upsampling factor and scaling are not stated. The paper says only "upsampled". The code uses ×4 bicubic and no value scaling, and clips to [0, 1] model.py data.py. An implementer who feeds 0–255 data gets clipped output and no gradient through the clipped values.
• Clipping kills gradients. clamp on the upsampled input and on the output has zero gradient outside [0, 1] model.py model.py. Pixels that saturate receive no training signal.
• Attention cost is quadratic in pixels. Each token is one pixel, so a 512×512 image gives 262,144 tokens (derived: 512 × 512). The attention map then has about 6.9 × 10^10 entries per head per block (derived: 262,144²). That is why the code's test path works on 64×64 patches main.py. Global attention over a whole scene is not what the reported test path computes.
• No positional information. Attention on pixel tokens without positional embeddings is blind to where a pixel sits; only the pixel's own values carry location, plus the refine convolutions after the transformer. The code keeps the positional tensor but does not add it model.py.
• README sizes are inconsistent. The demo whole-image file lists the RGB input as 128×128, but the target is 512×512; the patch file lists 63×64 for 64×64 targets README.md. The model concatenates the MSI with the upsampled LR-HSI along channels model.py, so the MSI must already match the upsampled size. Treat the README sizes as unreliable.
• Training does not run as shipped. main.py sets train_or_not = 0 main.py, so running it only tests. The training and validation HDF5 files (train_cave.h5, validation_cave.h5) are not in the repository and are only available through external links README.md. The resume path loads an epoch-500 checkpoint that is not shipped main.py.
• Attention scale is ambiguous. The paper writes √d_k without saying whether d_k is the full width or the per-head width eq. 1. The code uses the per-head width 16 model.py.
• Refine module is ambiguous. The text says an MLP is "plotted in fig. 1" (c), but the panel is titled "Reshape & Refine" and the code uses two 3×3 convolutions §2.2.2 model.py.
• Quality index definitions are missing. The compiled paper names PSNR, SAM, ERGAS and SSIM in the table headers but does not define them or state units. The definitions appear only in commented-out source text, which is not part of the paper.
• Commented-out source is not used. The .tex file contains commented text, including a degradation model, a related-work section and a toy figure. The compiled paper does not include them, so this card ignores them.
• Small evidence base. Results come from two 31-band datasets, CAVE and Harvard. Both are reported by the authors; no other sensors or sampling factors are given. Runtime and memory are not reported.

Lineage

• Builds on: the transformer §2.1, vision transformer and image processing transformer as cited in §2.1. Fusformer adapts them to one token per pixel instead of image patches §2.2.1.
• Residual learning on the upsampled input, tested in the paper's own ablation §2.2 §3.2.
• Compared against: FUSE, GLP-HS, CSTF, CNN-FUS, SSRNet, ResTFNet, MHF-Net and HSRnet §3.
• Superseded by: not known from content already in this wiki.