Paper · 2022-03-31 · cs.CV, eess.IV
Deep Hyperspectral Unmixing using Transformer Network
Hyperspectral unmixing model: a convolutional autoencoder reduces the bands, a two-block transformer whose attention is driven by one class token over image patches predicts abundance maps, and a 1x1 convolution decoder holds the endmember spectra. Trained with an MSE plus spectral-angle loss. Author-reported lowest overall RMSE on Samson, Apex and Washington DC Mall.
AutoencoderCNN-Transformer hybridSelf-attentionhyperspectral-unmixingendmember-extractiontransformerautoencoderremote-sensing
arXivPDFCodeMarkdown14 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A satellite or airborne hyperspectral camera records, for every pixel, the light reflected at hundreds of wavelengths. A pixel that is coarser than the ground it covers often holds several materials at once, so its spectrum is a blend. Spectral unmixing asks two questions: which pure materials are in the scene, and how much of each one is in every pixel.
This method treats the problem with a neural network that squeezes the spectrum of each pixel down, looks at the whole image through a transformer, and then rebuilds the image. The part that rebuilds the image is a single layer whose weights are the material spectra, so the network learns those spectra while it learns to reconstruct the data. The transformer is the new piece. It lets the network relate distant parts of the image, which a stack of small convolutions sees only slowly. Training rewards both an accurate reconstruction and spectra that point in the right direction.
TL;DR
A convolutional autoencoder plus a small transformer for hyperspectral unmixing. The transformer's attention is driven by a single class token that attends over image patches, and the decoder's weights are the endmember spectra. It is trained on reconstruction error and a spectral-angle loss, and the paper reports lower RMSE and SAD than the baselines it compares with on three benchmark scenes. Only the authors' own numbers are shown here.
Problem
Linear unmixing needs known endmembers or pure pixels, and blind unmixing is a non-convex problem that is sensitive to the choice of its regularisation parameter §I. Sparse unmixing needs a spectral library and suffers from library mismatch §I. Earlier deep autoencoders for unmixing are mostly built from convolutions, which see local neighbourhoods only, and several of them ignore spatial information or fail when no pure pixels exist §I §I.A. The authors argue that the bottleneck of a convolutional autoencoder loses context, because there are far fewer endmembers than spectral bands §I.A.
Method
The model has four parts. The input cube of shape B × H × W is first reshaped to B × n pixels. The abundance cube has R maps and the endmember matrix has shape B × R §II.
- Linear mixing model and constraints. Each pixel is Y = EA + N eq. 1. Abundances must be non-negative and sum to one, which the paper writes as ANC and ASC eq. 2a eq. 2b §II.A. Endmember entries are also non-negative §II.A.
- Encoder. Three 1×1 convolutions, each followed by batch normalisation. The first also has dropout, and the first two use Leaky ReLU. The band count goes B → 128 → 64 → C §II.B tab. I eq. 3. A 1×1 kernel keeps the parameter count low §II.B. Dropout is used against vanishing gradients §II.B; its rate is not stated.
- Patch tokens. The encoder output (H × W × C) is cut into non-overlapping p × p patches. Each patch is flattened to a vector of length D = p²·C, and the N' patch vectors form the tokens §II.C. A worked example for Samson (95 × 95 × 24, p = 5) gives 361 patches of length 600 §II.C. A learnable class token is prepended, and a learnable positional embedding is added to all N = N' + 1 tokens, both randomly initialised §II.C eq. 4.
- Multihead self-patch attention (MPA). Each transformer encoder block
works as follows §II.D alg. 1.
- Layer-normalise all tokens. Compute the query from the class token only, and the keys and values from all tokens §II.D, Step 1.
- Split the width D into h heads. In each head, the class-token query scores the N tokens, scaled by the square root of D/h, and softmax gives the weights §II.D, Step 2 eq. 5 eq. 6.
- Concatenate the heads, apply a linear map W_l, and add the class token as a residual §II.D, Step 3 eq. 7.
- Concatenate the new class token with the layer-normalised patch tokens §II.D, Step 4 eq. 8.
- Apply a normalisation and an MLP with a residual connection to all tokens §II.D eq. 9. Two such blocks are stacked §II.D.
- Decoder and abundances. Only the class token leaves the transformer. It is reshaped into R rows of length D/R, each row is upscaled to H × W, a 3 × 3 convolution smooths the result, and a softmax across the R maps enforces ANC and ASC §II.E.
- Endmembers. A single 1×1 convolution maps the R abundance maps back to B bands. Its weights are the endmember matrix. They are initialised with VCA and then learned §II.E §I.A.
Training minimises a weighted sum of a mean squared reconstruction error eq. 10 and a spectral-angle term eq. 11, with weights β and γ eq. 12 §II.F. The spectral-angle term is motivated because it does not depend on the absolute magnitude of an endmember §II.F.
Key equations, written out from the paper:
L_{RE} = \frac{1}{HW} \sum_{i=1}^{H} \sum_{j=1}^{W} (\hat{I}_{ij} - I_{ij})^2 eq. 10
L_{SAD} = \frac{1}{R} \sum_{i=1}^{R} \arccos\left( \frac{\langle I_i, \hat{I}_i \rangle}{\lVert I_i \rVert_2 \, \lVert \hat{I}_i \rVert_2} \right) eq. 11
L = \beta L_{RE} + \gamma L_{SAD} eq. 12
Evaluation uses an RMSE over all abundance values eq. 13 and a mean spectral angle over the endmembers eq. 14 §III.D.
# Self-patch transformer unmixing, forward pass [alg. 1] [§II.B-§II.E]
# Input: image (B, H, W) with B bands; P endmembers; patch size p
# Output: abundances (P, H, W), endmembers (B, P), reconstruction (B, H, W)
def unmix(image):
# Encode: three 1x1 conv layers reduce B bands to C [§II.B] [eq. 3]
latent = encoder(image) # (C, H, W)
# Cut the latent cube into non-overlapping p x p patches [§II.C]
patches = to_patch_tokens(latent, p) # (N', D), D = p*p*C
tokens = concat(class_token, patches) + pos # (N, D), N = N' + 1
# Class token and positions are learned, random at start [eq. 4]
# Two encoder blocks with self-patch attention [§II.D] [eq. 5]-[eq. 9]
for block in range(2):
tokens = self_patch_block(tokens) # (N, D)
# Only the class token leaves the transformer [§II.E]
class_out = tokens[0] # (D,)
# Split the class token into P rows of length D / P [§II.E]
rows = reshape(class_out, (P, D // P)) # (P, D/P)
# Upscale each row to the full image size [§II.E]
maps = upscale(rows, H, W) # (P, H, W)
# 3x3 convolution suppresses upscaling noise [§II.E]
maps = conv3x3(maps) # (P, H, W)
# Softmax over endmembers enforces ANC and ASC [§II.E] [eq. 2a] [eq. 2b]
abundances = softmax(maps, axis=0) # (P, H, W)
# Linear decoder: 1x1 conv whose weights are the endmembers [§II.E]
# (B, P) @ (P, H*W) -> (B, H*W), then reshape to the image [§II.E]
reconstruction = endmembers @ abundances.reshape(P, H * W)
return abundances, endmembers, reconstruction
def self_patch_block(tokens):
# Step 1: layer-normalise all tokens [§II.D] [alg. 1]
normed = layer_norm(tokens) # (N, D)
# Query from the class token only; keys and values from all tokens
query = normed[0] @ W_q # (D,)
keys = normed @ W_k # (N, D)
values = normed @ W_v # (N, D)
# Step 2: per head, scores of the query against the N tokens [eq. 6]
# head_dot and head_weighted_sum split the width D into h heads
scores = head_dot(query, keys) / sqrt(D / h) # (h, N)
weights = softmax(scores, axis=1) # (h, N)
head_out = head_weighted_sum(weights, values) # (h, D/h)
attended = reshape(head_out, (D,)) # (D,)
# Step 3: project and add the class token back [eq. 7]
cls_new = attended @ W_l + tokens[0] # (D,)
# Step 4: new class token next to normalised patch tokens [eq. 8]
tokens = concat(cls_new, layer_norm(tokens[1:])) # (N, D)
# Feed-forward over all tokens, with a residual [eq. 9]
# Hidden width of the MLP is not stated in the paper
tokens = tokens + mlp(layer_norm(tokens)) # (N, D)
return tokensHyperparameters
Table values are the paper's; "code" entries are what the authors' repository
uses at code_commit where the paper is silent.
| Name | Samson | Apex | Washington DC | Where | Notes |
|---|---|---|---|---|---|
| Patch size p | 5 | 5 | 10 | tab. II §III.C | |
| Encoder output bands C | 24 | 32 | 24 | tab. II §III.C | Paper calls it a hyperparameter; code derives it, see Paper vs code |
| Transformer width D = p²·C | 600 | 800 | 2400 | §II.C Trans_mod.py | Derived, not stated in the text for Apex or DC |
| Encoder layers | 3 | 3 | 3 | §II.B tab. I | Band counts 128 and 64 in the middle |
| Transformer encoder blocks | 2 | 2 | 2 | §II.D | |
| Heads h | not stated | not stated | not stated | Trans_mod.py | Code uses 8 |
| Block MLP hidden width | not stated | not stated | not stated | Trans_mod.py | Code uses 12, a narrow bottleneck |
| Dropout in encoder layer 1 | not stated | not stated | not stated | §II.B Trans_mod.py | Code uses 0.25 |
| Reconstruction weight β | 5e3 | 5e3 | 5e3 | tab. II | Text says 4e3 for Samson and Apex, see Paper vs code |
| Spectral-angle weight γ | 3e-2 | 5e-2 | 1e-4 | tab. II §III.G | Text says 5e-3 (Samson) and 2e-3 (DC) |
| Epochs | 200 | 200 | 150 | tab. II | One epoch is one full-image step, see Pitfalls |
| Initial learning rate | 6e-3 | 9e-3 | 6e-3 | tab. II §III.C | |
| LR schedule | ×0.8 every 15 epochs | same | same | §III.C Trans_mod.py | Paper says reduced by 20 % |
| Weight decay | 4e-5 | 4e-5 | 3e-5 | tab. II §III.C | |
| Optimiser | not stated | not stated | not stated | Trans_mod.py | Code uses Adam with L2 weight decay |
| Gradient clipping | not stated | not stated | not stated | Trans_mod.py | Code clips the L1 norm at 10 |
| Initial endmembers | VCA | VCA | VCA | §II.E | Code loads a matrix from the data file, see Paper vs code |
| Seed | not stated | not stated | not stated | main.py | Seed 1 is set in the DC entry point |
Results
Author-reported. Setup: three real scenes (Samson, 95 × 95 pixels, 156 bands, 3 endmembers; Apex, 110 × 110, 285 bands, 4 endmembers; Washington DC Mall, 290 × 290, 191 bands, 6 endmembers) §III.A. Ground-truth endmembers were picked by hand, and ground-truth abundances were produced by applying FCLSU §III.A. Baselines are FCLSU, NMF-QMV, Collaborative LASSO, uDAS, UnDIP and CyCUNet §III.B; the SAD tables also list SiVM and VCA tab. IV tab. VI tab. VIII. Each entry is the single reported run, and no spread is given.
| Dataset | Metric | Best baseline (overall) | Proposed (overall) | Source |
|---|---|---|---|---|
| Samson | RMSE | 0.1159 (Collab) | 0.0783 | tab. III |
| Samson | SAD (rad) | 0.0796 (Collab) | 0.0510 | tab. IV |
| Apex | RMSE | 0.1543 (FCLSU) | 0.1264 | tab. V |
| Apex | SAD (rad) | 0.1994 (SiVM) | 0.0867 | tab. VI |
| Washington DC Mall | RMSE | 0.2322 (NMF) | 0.1307 | tab. VII |
| Washington DC Mall | SAD (rad) | 0.2338 (Collab) | 0.1537 | tab. VIII |
Per endmember, the paper highlights the Road endmember in Apex, where no baseline recovers the abundance map well §III.E fig. 8, and the Grass and Tree endmembers in Washington DC Mall, which have nearly identical spectra §III.E. UnDIP has a lower RMSE than the proposed method for Apex Road tab. V. On the abundance and endmember figures, the method is described as visually closest to the ground truth §III.F fig. 6 fig. 7 fig. 8 fig. 9 fig. 10 fig. 11.
Sensitivity: fig. 12 varies γ, and the paper suggests a range of 1e-4 to 1e-2, with more endmembers favouring a smaller γ §III.G. fig. 13 varies the learning rate and weight decay, with the best learning rate between 0.006 and 0.009 and the weight decay near 3e-5 §III.G.
Paper vs code
The authors' repository (code) was cloned at code_commit. It has Trans_mod.py,
transformer.py, datasets.py, utils.py and main.py, and it contains no
LICENSE file. The paper links it as its code.
- Hyperparameter text disagrees with the paper's own table and with the code. The text gives β = 4e3 for Samson and Apex §III.C, but the table gives 5e3 tab. II, and the code uses 5e3 for all three datasets Trans_mod.py Trans_mod.py Trans_mod.py. The text gives γ = 5e-3 for Samson and 2e-3 for DC §III.C, but the table gives 3e-2 and 1e-4 tab. II. The code matches the table Trans_mod.py Trans_mod.py.
- Text disagrees with table on one result. The text reports a mean Samson SAD of 0.0608 §III.E. The table gives 0.0510 tab. IV, and the stated 35.93 % improvement fits 0.0510 against 0.0796. Use the table value.
- C is a hyperparameter in the paper but derived in the code. The code sets the encoder output width to dim·P / p², with dim = 200 for Samson and Apex and 400 for DC Trans_mod.py Trans_mod.py Trans_mod.py Trans_mod.py. This gives 24, 32 and 24, the paper's values, so it is consistent, but changing C alone breaks the code.
- Class-token residual uses the normalised token. Eq. 7 adds the raw class
token X'_cls eq. 7 §II.D. In the code, the block receives the
layer-normalised input from
PreNorm, so the residual is the normalised class token transformer.py transformer.py transformer.py. - One layer norm is shared across both blocks. The code creates one
LayerNormfor the patch tokens in eachTransformerand reuses it in every block transformer.py transformer.py. The paper does not say whether the norm is shared. - Feed-forward width. The code's block MLP is 600→12→600 for Samson, with GELU transformer.py transformer.py Trans_mod.py. The paper gives no width.
- Decoder has a ReLU. The paper's decoder is a single convolution §II.E. The code adds ReLU after it Trans_mod.py.
- Endmember weights are clamped. After each step the decoder weights are clamped to the interval [1e-6, 1] Trans_mod.py Trans_mod.py. The paper only requires non-negativity §II.A.
- Loss normalisation. The code's reconstruction term is
MSELosswith mean reduction over all bands and pixels, so it divides by B·H·W rather than H·W Trans_mod.py Trans_mod.py. The spectral-angle term is the sum over all pixels of a per-pixel angle, in radians, with no 1/R factor Trans_mod.py utils.py. - Upscaling is a linear layer. The paper calls the upscaling step only "upscale". The code uses one linear layer, shared across endmembers, that maps D/P to H·W Trans_mod.py. The 3 × 3 smoothing and the softmax match Trans_mod.py.
- Stochastic depth. The attention residual uses DropPath with probability 0.15 during training transformer.py transformer.py. The paper does not mention it.
- Gradient clipping uses the L1 norm at 10 Trans_mod.py.
- Ordering of endmembers. The endmember and abundance orders are fixed permutations per dataset, set by hand Trans_mod.py Trans_mod.py Trans_mod.py Trans_mod.py. The paper does not describe any matching.
- Initial endmembers come from a file. The code loads
M1from each dataset's.matfile datasets.py datasets.py and uses it as the decoder's initial weights Trans_mod.py. It does not compute VCA. The source ofM1is not stated in the repository. - Dataset band counts.
datasets.pylists 258 bands for Apex datasets.py, while the training code uses 285, as the paper does Trans_mod.py. - Repository logs. The committed
log1.csvfiles record a hyperparameter sweep, not the final configuration. Some rows with identical learning rate and weight decay give different results. For Samson at learning rate 6e-3 and weight decay 4e-5 the RMSE is 0.0770 in one row and 0.0963 in another. For Apex at 9e-3 and 4e-5 it is 0.1687 and 0.1432. Seeds are not recorded for these rows.
Pitfalls
- Only the class token attends. The query comes from the class token alone, so patch tokens do not attend to each other §II.D eq. 5 eq. 6. They pass only through the normalisation and the feed-forward layer eq. 8 eq. 9. An implementer who expects full self-attention among patches will build the wrong model. The name "self-patch attention" refers to this reduced attention.
- Epochs are full-image steps. In the code, one batch holds every pixel, so one epoch is one optimiser step. Samson therefore gets 200 steps in total datasets.py Trans_mod.py Trans_mod.py. Learning rate and weight decay tuned for mini-batch training will not transfer directly.
- Loss scaling is unclear. Eq. 10 and Eq. 11 do not say whether the reconstruction error is averaged over bands, and Eq. 11 writes the average over R, which is ambiguous for a per-pixel angle eq. 10 eq. 11. The code's scale is described above. β and γ are only meaningful together with this scale.
- Spectral angle is in radians. The paper says degrees §III.D, but the code returns the mean of arccos values in radians utils.py. The reported magnitudes match the radian output. Read them as radians.
- Overall RMSE is not a mean. It is the root of the mean of squared errors over all classes utils.py. For Samson, the root mean square of the three per-class values is 0.0783 and their simple mean is 0.0775, so the tables use the root-mean-square.
- Percentage improvements use different denominators. Samson RMSE 48.02 % is relative to the proposed value; Samson SAD 35.93 % is relative to the baseline; DC RMSE 43.71 % is relative to the baseline; DC SAD 52.11 % is relative to the proposed value §III.E. Use the raw values in the tables.
- Ground truth comes from one of the baselines. The reference abundances were produced by FCLSU, which is also a compared method §III.A §III.B. The paper does not discuss how this affects FCLSU's position in the tables.
- No variance. Each table is one run. The repository's sweep rows show large run-to-run differences for the same settings (see Paper vs code).
- Hyperparameters chosen on the training loss. The paper selects the best combination by the value of the total training loss on the same image, using a grid search §III.G eq. 12. The ground truth is not used for this step, but the reported test is on the same image.
- Numerically sensitive spots. The SAD term calls arccos on a cosine
computed from norms, with no epsilon or clamp in
utils.SADutils.py. If a cosine rounds to exactly 1, the gradient is infinite. The endmember clamp to [1e-6, 1] assumes reflectance-scale data, and the paper does not state its data scaling. - Hidden costs. For DC, D = 2400, so each D×D projection (W_q, W_k, W_v, W_l) has about 5.8 million weights per block. The shared upscaling layer maps 400 values to 290² = 84,100 per row, which is about 33.6 million weights in one layer. Both figures are computed from the shapes in the code Trans_mod.py transformer.py.
- Stochastic depth is active only in training. Evaluation uses
net.eval(), which switches off dropout and DropPath Trans_mod.py.
Lineage
- Builds on: the transformer and multi-head attention of the original transformer and of the Vision Transformer, both cited in the introduction, and the cross-attention idea of CrossViT, which the paper cites as its inspiration §I.A. Named only; no card exists for these papers.
- Compares against: FCLSU, NMF-QMV, Collaborative LASSO, uDAS, UnDIP and CyCUNet §III.B, plus SiVM and VCA in the SAD tables tab. IV. Named only.
- Superseded by: not known from content in this wiki.
Source and license
Adapted from “Deep Hyperspectral Unmixing using Transformer Network” by Preetam Ghosh, Swalpa Kumar Roy, Bikram Koirala, Behnood Rasti, Paul Scheunders (arXiv:2203.17076), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/preetam22n/DeepTrans-HSU at commit a1cefda156e3 (No license file (all rights reserved by default)). Any quoted code is shown under that license.