wikillm

Paper · 2022-03-31 · cs.CV, eess.IV

Deep Hyperspectral Unmixing using Transformer Network

Hyperspectral unmixing model: a convolutional autoencoder reduces the bands, a two-block transformer whose attention is driven by one class token over image patches predicts abundance maps, and a 1x1 convolution decoder holds the endmember spectra. Trained with an MSE plus spectral-angle loss. Author-reported lowest overall RMSE on Samson, Apex and Washington DC Mall.

AutoencoderCNN-Transformer hybridSelf-attentionhyperspectral-unmixingendmember-extractiontransformerautoencoderremote-sensing

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A satellite or airborne hyperspectral camera records, for every pixel, the light reflected at hundreds of wavelengths. A pixel that is coarser than the ground it covers often holds several materials at once, so its spectrum is a blend. Spectral unmixing asks two questions: which pure materials are in the scene, and how much of each one is in every pixel.

This method treats the problem with a neural network that squeezes the spectrum of each pixel down, looks at the whole image through a transformer, and then rebuilds the image. The part that rebuilds the image is a single layer whose weights are the material spectra, so the network learns those spectra while it learns to reconstruct the data. The transformer is the new piece. It lets the network relate distant parts of the image, which a stack of small convolutions sees only slowly. Training rewards both an accurate reconstruction and spectra that point in the right direction.

TL;DR

A convolutional autoencoder plus a small transformer for hyperspectral unmixing. The transformer's attention is driven by a single class token that attends over image patches, and the decoder's weights are the endmember spectra. It is trained on reconstruction error and a spectral-angle loss, and the paper reports lower RMSE and SAD than the baselines it compares with on three benchmark scenes. Only the authors' own numbers are shown here.

Problem

Linear unmixing needs known endmembers or pure pixels, and blind unmixing is a non-convex problem that is sensitive to the choice of its regularisation parameter §I. Sparse unmixing needs a spectral library and suffers from library mismatch §I. Earlier deep autoencoders for unmixing are mostly built from convolutions, which see local neighbourhoods only, and several of them ignore spatial information or fail when no pure pixels exist §I §I.A. The authors argue that the bottleneck of a convolutional autoencoder loses context, because there are far fewer endmembers than spectral bands §I.A.

Method

The model has four parts. The input cube of shape B × H × W is first reshaped to B × n pixels. The abundance cube has R maps and the endmember matrix has shape B × R §II.

  1. Linear mixing model and constraints. Each pixel is Y = EA + N eq. 1. Abundances must be non-negative and sum to one, which the paper writes as ANC and ASC eq. 2a eq. 2b §II.A. Endmember entries are also non-negative §II.A.
  2. Encoder. Three 1×1 convolutions, each followed by batch normalisation. The first also has dropout, and the first two use Leaky ReLU. The band count goes B → 128 → 64 → C §II.B tab. I eq. 3. A 1×1 kernel keeps the parameter count low §II.B. Dropout is used against vanishing gradients §II.B; its rate is not stated.
  3. Patch tokens. The encoder output (H × W × C) is cut into non-overlapping p × p patches. Each patch is flattened to a vector of length D = p²·C, and the N' patch vectors form the tokens §II.C. A worked example for Samson (95 × 95 × 24, p = 5) gives 361 patches of length 600 §II.C. A learnable class token is prepended, and a learnable positional embedding is added to all N = N' + 1 tokens, both randomly initialised §II.C eq. 4.
  4. Multihead self-patch attention (MPA). Each transformer encoder block works as follows §II.D alg. 1.
    • Layer-normalise all tokens. Compute the query from the class token only, and the keys and values from all tokens §II.D, Step 1.
    • Split the width D into h heads. In each head, the class-token query scores the N tokens, scaled by the square root of D/h, and softmax gives the weights §II.D, Step 2 eq. 5 eq. 6.
    • Concatenate the heads, apply a linear map W_l, and add the class token as a residual §II.D, Step 3 eq. 7.
    • Concatenate the new class token with the layer-normalised patch tokens §II.D, Step 4 eq. 8.
    • Apply a normalisation and an MLP with a residual connection to all tokens §II.D eq. 9. Two such blocks are stacked §II.D.
  5. Decoder and abundances. Only the class token leaves the transformer. It is reshaped into R rows of length D/R, each row is upscaled to H × W, a 3 × 3 convolution smooths the result, and a softmax across the R maps enforces ANC and ASC §II.E.
  6. Endmembers. A single 1×1 convolution maps the R abundance maps back to B bands. Its weights are the endmember matrix. They are initialised with VCA and then learned §II.E §I.A.

Training minimises a weighted sum of a mean squared reconstruction error eq. 10 and a spectral-angle term eq. 11, with weights β and γ eq. 12 §II.F. The spectral-angle term is motivated because it does not depend on the absolute magnitude of an endmember §II.F.

Key equations, written out from the paper:

L_{RE} = \frac{1}{HW} \sum_{i=1}^{H} \sum_{j=1}^{W} (\hat{I}_{ij} - I_{ij})^2 eq. 10

L_{SAD} = \frac{1}{R} \sum_{i=1}^{R} \arccos\left( \frac{\langle I_i, \hat{I}_i \rangle}{\lVert I_i \rVert_2 \, \lVert \hat{I}_i \rVert_2} \right) eq. 11

L = \beta L_{RE} + \gamma L_{SAD} eq. 12

Evaluation uses an RMSE over all abundance values eq. 13 and a mean spectral angle over the endmembers eq. 14 §III.D.

Pseudocode
# Self-patch transformer unmixing, forward pass  [alg. 1] [§II.B-§II.E]
# Input:  image (B, H, W) with B bands; P endmembers; patch size p
# Output: abundances (P, H, W), endmembers (B, P), reconstruction (B, H, W)

def unmix(image):
    # Encode: three 1x1 conv layers reduce B bands to C  [§II.B] [eq. 3]
    latent = encoder(image)                          # (C, H, W)

    # Cut the latent cube into non-overlapping p x p patches  [§II.C]
    patches = to_patch_tokens(latent, p)             # (N', D), D = p*p*C
    tokens = concat(class_token, patches) + pos      # (N, D), N = N' + 1
    # Class token and positions are learned, random at start  [eq. 4]

    # Two encoder blocks with self-patch attention  [§II.D] [eq. 5]-[eq. 9]
    for block in range(2):
        tokens = self_patch_block(tokens)            # (N, D)

    # Only the class token leaves the transformer  [§II.E]
    class_out = tokens[0]                            # (D,)

    # Split the class token into P rows of length D / P  [§II.E]
    rows = reshape(class_out, (P, D // P))           # (P, D/P)
    # Upscale each row to the full image size  [§II.E]
    maps = upscale(rows, H, W)                       # (P, H, W)
    # 3x3 convolution suppresses upscaling noise  [§II.E]
    maps = conv3x3(maps)                             # (P, H, W)
    # Softmax over endmembers enforces ANC and ASC  [§II.E] [eq. 2a] [eq. 2b]
    abundances = softmax(maps, axis=0)               # (P, H, W)

    # Linear decoder: 1x1 conv whose weights are the endmembers  [§II.E]
    # (B, P) @ (P, H*W) -> (B, H*W), then reshape to the image  [§II.E]
    reconstruction = endmembers @ abundances.reshape(P, H * W)
    return abundances, endmembers, reconstruction


def self_patch_block(tokens):
    # Step 1: layer-normalise all tokens  [§II.D] [alg. 1]
    normed = layer_norm(tokens)                      # (N, D)
    # Query from the class token only; keys and values from all tokens
    query = normed[0] @ W_q                          # (D,)
    keys = normed @ W_k                              # (N, D)
    values = normed @ W_v                            # (N, D)

    # Step 2: per head, scores of the query against the N tokens  [eq. 6]
    # head_dot and head_weighted_sum split the width D into h heads
    scores = head_dot(query, keys) / sqrt(D / h)     # (h, N)
    weights = softmax(scores, axis=1)                # (h, N)
    head_out = head_weighted_sum(weights, values)    # (h, D/h)
    attended = reshape(head_out, (D,))               # (D,)

    # Step 3: project and add the class token back  [eq. 7]
    cls_new = attended @ W_l + tokens[0]             # (D,)
    # Step 4: new class token next to normalised patch tokens  [eq. 8]
    tokens = concat(cls_new, layer_norm(tokens[1:])) # (N, D)

    # Feed-forward over all tokens, with a residual  [eq. 9]
    # Hidden width of the MLP is not stated in the paper
    tokens = tokens + mlp(layer_norm(tokens))        # (N, D)
    return tokens

Hyperparameters

Table values are the paper's; "code" entries are what the authors' repository uses at code_commit where the paper is silent.

NameSamsonApexWashington DCWhereNotes
Patch size p5510tab. II §III.C
Encoder output bands C243224tab. II §III.CPaper calls it a hyperparameter; code derives it, see Paper vs code
Transformer width D = p²·C6008002400§II.C Trans_mod.pyDerived, not stated in the text for Apex or DC
Encoder layers333§II.B tab. IBand counts 128 and 64 in the middle
Transformer encoder blocks222§II.D
Heads hnot statednot statednot statedTrans_mod.pyCode uses 8
Block MLP hidden widthnot statednot statednot statedTrans_mod.pyCode uses 12, a narrow bottleneck
Dropout in encoder layer 1not statednot statednot stated§II.B Trans_mod.pyCode uses 0.25
Reconstruction weight β5e35e35e3tab. IIText says 4e3 for Samson and Apex, see Paper vs code
Spectral-angle weight γ3e-25e-21e-4tab. II §III.GText says 5e-3 (Samson) and 2e-3 (DC)
Epochs200200150tab. IIOne epoch is one full-image step, see Pitfalls
Initial learning rate6e-39e-36e-3tab. II §III.C
LR schedule×0.8 every 15 epochssamesame§III.C Trans_mod.pyPaper says reduced by 20 %
Weight decay4e-54e-53e-5tab. II §III.C
Optimisernot statednot statednot statedTrans_mod.pyCode uses Adam with L2 weight decay
Gradient clippingnot statednot statednot statedTrans_mod.pyCode clips the L1 norm at 10
Initial endmembersVCAVCAVCA§II.ECode loads a matrix from the data file, see Paper vs code
Seednot statednot statednot statedmain.pySeed 1 is set in the DC entry point

Results

Author-reported. Setup: three real scenes (Samson, 95 × 95 pixels, 156 bands, 3 endmembers; Apex, 110 × 110, 285 bands, 4 endmembers; Washington DC Mall, 290 × 290, 191 bands, 6 endmembers) §III.A. Ground-truth endmembers were picked by hand, and ground-truth abundances were produced by applying FCLSU §III.A. Baselines are FCLSU, NMF-QMV, Collaborative LASSO, uDAS, UnDIP and CyCUNet §III.B; the SAD tables also list SiVM and VCA tab. IV tab. VI tab. VIII. Each entry is the single reported run, and no spread is given.

DatasetMetricBest baseline (overall)Proposed (overall)Source
SamsonRMSE0.1159 (Collab)0.0783tab. III
SamsonSAD (rad)0.0796 (Collab)0.0510tab. IV
ApexRMSE0.1543 (FCLSU)0.1264tab. V
ApexSAD (rad)0.1994 (SiVM)0.0867tab. VI
Washington DC MallRMSE0.2322 (NMF)0.1307tab. VII
Washington DC MallSAD (rad)0.2338 (Collab)0.1537tab. VIII

Per endmember, the paper highlights the Road endmember in Apex, where no baseline recovers the abundance map well §III.E fig. 8, and the Grass and Tree endmembers in Washington DC Mall, which have nearly identical spectra §III.E. UnDIP has a lower RMSE than the proposed method for Apex Road tab. V. On the abundance and endmember figures, the method is described as visually closest to the ground truth §III.F fig. 6 fig. 7 fig. 8 fig. 9 fig. 10 fig. 11.

Sensitivity: fig. 12 varies γ, and the paper suggests a range of 1e-4 to 1e-2, with more endmembers favouring a smaller γ §III.G. fig. 13 varies the learning rate and weight decay, with the best learning rate between 0.006 and 0.009 and the weight decay near 3e-5 §III.G.

Paper vs code

The authors' repository (code) was cloned at code_commit. It has Trans_mod.py, transformer.py, datasets.py, utils.py and main.py, and it contains no LICENSE file. The paper links it as its code.

  • Hyperparameter text disagrees with the paper's own table and with the code. The text gives β = 4e3 for Samson and Apex §III.C, but the table gives 5e3 tab. II, and the code uses 5e3 for all three datasets Trans_mod.py Trans_mod.py Trans_mod.py. The text gives γ = 5e-3 for Samson and 2e-3 for DC §III.C, but the table gives 3e-2 and 1e-4 tab. II. The code matches the table Trans_mod.py Trans_mod.py.
  • Text disagrees with table on one result. The text reports a mean Samson SAD of 0.0608 §III.E. The table gives 0.0510 tab. IV, and the stated 35.93 % improvement fits 0.0510 against 0.0796. Use the table value.
  • C is a hyperparameter in the paper but derived in the code. The code sets the encoder output width to dim·P / p², with dim = 200 for Samson and Apex and 400 for DC Trans_mod.py Trans_mod.py Trans_mod.py Trans_mod.py. This gives 24, 32 and 24, the paper's values, so it is consistent, but changing C alone breaks the code.
  • Class-token residual uses the normalised token. Eq. 7 adds the raw class token X'_cls eq. 7 §II.D. In the code, the block receives the layer-normalised input from PreNorm, so the residual is the normalised class token transformer.py transformer.py transformer.py.
  • One layer norm is shared across both blocks. The code creates one LayerNorm for the patch tokens in each Transformer and reuses it in every block transformer.py transformer.py. The paper does not say whether the norm is shared.
  • Feed-forward width. The code's block MLP is 600→12→600 for Samson, with GELU transformer.py transformer.py Trans_mod.py. The paper gives no width.
  • Decoder has a ReLU. The paper's decoder is a single convolution §II.E. The code adds ReLU after it Trans_mod.py.
  • Endmember weights are clamped. After each step the decoder weights are clamped to the interval [1e-6, 1] Trans_mod.py Trans_mod.py. The paper only requires non-negativity §II.A.
  • Loss normalisation. The code's reconstruction term is MSELoss with mean reduction over all bands and pixels, so it divides by B·H·W rather than H·W Trans_mod.py Trans_mod.py. The spectral-angle term is the sum over all pixels of a per-pixel angle, in radians, with no 1/R factor Trans_mod.py utils.py.
  • Upscaling is a linear layer. The paper calls the upscaling step only "upscale". The code uses one linear layer, shared across endmembers, that maps D/P to H·W Trans_mod.py. The 3 × 3 smoothing and the softmax match Trans_mod.py.
  • Stochastic depth. The attention residual uses DropPath with probability 0.15 during training transformer.py transformer.py. The paper does not mention it.
  • Gradient clipping uses the L1 norm at 10 Trans_mod.py.
  • Ordering of endmembers. The endmember and abundance orders are fixed permutations per dataset, set by hand Trans_mod.py Trans_mod.py Trans_mod.py Trans_mod.py. The paper does not describe any matching.
  • Initial endmembers come from a file. The code loads M1 from each dataset's .mat file datasets.py datasets.py and uses it as the decoder's initial weights Trans_mod.py. It does not compute VCA. The source of M1 is not stated in the repository.
  • Dataset band counts. datasets.py lists 258 bands for Apex datasets.py, while the training code uses 285, as the paper does Trans_mod.py.
  • Repository logs. The committed log1.csv files record a hyperparameter sweep, not the final configuration. Some rows with identical learning rate and weight decay give different results. For Samson at learning rate 6e-3 and weight decay 4e-5 the RMSE is 0.0770 in one row and 0.0963 in another. For Apex at 9e-3 and 4e-5 it is 0.1687 and 0.1432. Seeds are not recorded for these rows.

Pitfalls

  • Only the class token attends. The query comes from the class token alone, so patch tokens do not attend to each other §II.D eq. 5 eq. 6. They pass only through the normalisation and the feed-forward layer eq. 8 eq. 9. An implementer who expects full self-attention among patches will build the wrong model. The name "self-patch attention" refers to this reduced attention.
  • Epochs are full-image steps. In the code, one batch holds every pixel, so one epoch is one optimiser step. Samson therefore gets 200 steps in total datasets.py Trans_mod.py Trans_mod.py. Learning rate and weight decay tuned for mini-batch training will not transfer directly.
  • Loss scaling is unclear. Eq. 10 and Eq. 11 do not say whether the reconstruction error is averaged over bands, and Eq. 11 writes the average over R, which is ambiguous for a per-pixel angle eq. 10 eq. 11. The code's scale is described above. β and γ are only meaningful together with this scale.
  • Spectral angle is in radians. The paper says degrees §III.D, but the code returns the mean of arccos values in radians utils.py. The reported magnitudes match the radian output. Read them as radians.
  • Overall RMSE is not a mean. It is the root of the mean of squared errors over all classes utils.py. For Samson, the root mean square of the three per-class values is 0.0783 and their simple mean is 0.0775, so the tables use the root-mean-square.
  • Percentage improvements use different denominators. Samson RMSE 48.02 % is relative to the proposed value; Samson SAD 35.93 % is relative to the baseline; DC RMSE 43.71 % is relative to the baseline; DC SAD 52.11 % is relative to the proposed value §III.E. Use the raw values in the tables.
  • Ground truth comes from one of the baselines. The reference abundances were produced by FCLSU, which is also a compared method §III.A §III.B. The paper does not discuss how this affects FCLSU's position in the tables.
  • No variance. Each table is one run. The repository's sweep rows show large run-to-run differences for the same settings (see Paper vs code).
  • Hyperparameters chosen on the training loss. The paper selects the best combination by the value of the total training loss on the same image, using a grid search §III.G eq. 12. The ground truth is not used for this step, but the reported test is on the same image.
  • Numerically sensitive spots. The SAD term calls arccos on a cosine computed from norms, with no epsilon or clamp in utils.SAD utils.py. If a cosine rounds to exactly 1, the gradient is infinite. The endmember clamp to [1e-6, 1] assumes reflectance-scale data, and the paper does not state its data scaling.
  • Hidden costs. For DC, D = 2400, so each D×D projection (W_q, W_k, W_v, W_l) has about 5.8 million weights per block. The shared upscaling layer maps 400 values to 290² = 84,100 per row, which is about 33.6 million weights in one layer. Both figures are computed from the shapes in the code Trans_mod.py transformer.py.
  • Stochastic depth is active only in training. Evaluation uses net.eval(), which switches off dropout and DropPath Trans_mod.py.

Lineage

  • Builds on: the transformer and multi-head attention of the original transformer and of the Vision Transformer, both cited in the introduction, and the cross-attention idea of CrossViT, which the paper cites as its inspiration §I.A. Named only; no card exists for these papers.
  • Compares against: FCLSU, NMF-QMV, Collaborative LASSO, uDAS, UnDIP and CyCUNet §III.B, plus SiVM and VCA in the SAD tables tab. IV. Named only.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “Deep Hyperspectral Unmixing using Transformer Network” by Preetam Ghosh, Swalpa Kumar Roy, Bikram Koirala, Behnood Rasti, Paul Scheunders (arXiv:2203.17076), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/preetam22n/DeepTrans-HSU at commit a1cefda156e3 (No license file (all rights reserved by default)). Any quoted code is shown under that license.