A Survey: Deep Learning for Hyperspectral Image Classification with Few Labeled Samples

In plain words

Hyperspectral cameras record many narrow colour bands for every pixel, so a classifier can tell materials apart that look alike to the eye. Deep learning does this well, but it usually needs a large number of labelled examples, and labelling hyperspectral pixels by hand is slow and expensive.

This survey looks at how researchers have tried to make deep models work with only a few labels. It sorts those attempts by how the model learns: reusing a model trained on other data, letting the model choose which pixels a person should label, or learning to compare samples so that a handful of examples is enough. It also summarises the main network designs used for these images. The authors then run eight existing methods on three public scenes with the same small label budgets, so that the methods can be compared on equal terms.

TL;DR

A survey with one benchmark, not a new method. It groups deep hyperspectral classifiers by model family (autoencoder, CNN, RNN) and by label-efficient paradigm (transfer, active and few-shot learning), then tests eight models on PaviaU, Salinas and KSC with 10, 50 and 100 labels per class. The best model changes with dataset and label count. Several sentences in the text disagree with the authors' own tables (see Pitfalls), and most training settings are not stated, so use the tables for numbers and the original papers for implementation.

Problem

Hyperspectral labels are costly to collect, while deep models have many parameters and need many samples §1 fig. 1. The authors say earlier reviews sort the field by feature-fusion type, whereas this one sorts few-label work by learning paradigm §1. They also say that different papers use different experimental settings, so they aim to compare methods under one protocol §4.

Method

The paper defines no new algorithm. It has three parts: a taxonomy of model families §2, a taxonomy of learning paradigms §3, and a benchmark §4.

Model families §2.

• Autoencoder §2.1: encoder and decoder trained to reproduce the input. The base loss is written as eq. 1. A sparsity penalty on the hidden layer and a denoising corruption of the input are the two standard ways to stop the network from learning the identity map §2.1.
• CNN §2.2: local kernels with shared weights; the review covers 2D, 1D+2D and 3D variants, residual and dense connections, attention, and multi-modal (HSI + LiDAR) models §2.2. The paper states no equations for this section.
• RNN §2.3: a recurrent state update eq. 2. The LSTM gates are eq. 3 to eq. 5, the candidate input eq. 6, the memory update eq. 7 and the output eq. 8. The section explains that a classic RNN suffers from vanishing or exploding gradients over long sequences, which is the reason for gated units §2.3.

Learning paradigms §3.

• Transfer learning §3.1: a source domain with many labels and a target domain with few labels, where P(X_s) ≠ P(X_t). Fine-tuning pretrains on the source and tunes on the target fig. 4. Distribution adaptation adds a discrepancy term, such as MMD, or an adversarial domain classifier (DANN) fig. 5.
• Active learning §3.2: a tuple A = (C, L, U, Q, S) of classifier, labelled set, unlabelled pool, query function and expert. An initial random set is labelled, then the query function picks pool samples for labelling each round §3.2 fig. 6. Two query families are reviewed: an entropy measure over a committee of classifiers eq. 9, and a posterior-based "breaking ties" margin eq. 10.
• Few-shot learning §3.3: metric-based models. Prototype networks average the embeddings of each class eq. 11 fig. 7. Relation networks replace the fixed distance with a learned scoring function eq. 12 fig. 8. Siamese networks compare two samples by Euclidean distance of their embeddings eq. 13 fig. 9; the review notes that an SVM is usually used as the final classifier for them §3.3.

Benchmark protocol §4. Three datasets, eight models, three label budgets (10, 50, 100 labelled pixels per class) §4. For each budget the experiment is repeated 10 times with a different random training set, and AA and OA are averaged §4 §4.3. The protocol is summarised below. Steps not in the paper are marked.

# Benchmark protocol  [§4] [§4.3]
# Input:  labelled pixels (X, y), models, label budgets, runs
# Output: AA and OA per model and budget  [tab. 5] [tab. 6] [tab. 7]

def benchmark(models, X, y, budgets=(10, 50, 100), runs=10):
    # One result table per budget  [§4]
    for per_class in budgets:
        for run in range(runs):
            # Draw a new random training set each run  [§4]
            train_idx = sample_per_class(y, per_class)
            # Remaining labelled pixels as test: inferred, not stated  [§4]
            test_idx = labelled_minus(y, train_idx)
            for model in models:
                model.fit(X[train_idx], y[train_idx])
                predictions = model.predict(X[test_idx])
                # Accuracies are averaged over the runs  [§4.3]
                record(model, per_class, run, predictions, y[test_idx])
    return summarise_aa_oa(models, budgets)

The active-learning query step, as the review describes it with the margin of eq. 10 §3.2. The batch size per round is not stated in the review.

# Breaking-ties query step  [§3.2] [eq. 10]
# Input:  classifier, labelled set L, unlabelled pool U, batch size b
# Output: updated L and U

def query_step(classifier, labelled, pool, b):
    # Posterior probabilities for every pool sample  [§3.2]
    probs = classifier.predict_proba(pool)  # (|U|, n_classes)
    # Margin between the top two posterior probabilities  [eq. 10]
    margin = top1(probs) - top2(probs)  # (|U|,)
    # The smallest margin is the most uncertain sample  [§3.2]
    picked = argsort(margin)[:b]
    # An expert labels the picked samples  [§3.2]
    labelled = labelled + label_by_expert(pool[picked])
    pool = remove(pool, picked)
    return labelled, pool

Equations as printed in the paper. Numbering follows the compiled paper.

eq. 1 autoencoder loss, written with a minimum over the norm §2.1:

\mathcal{L}(\bm{x},\hat{\bm{x}}) = \min \vert \bm{x}-\hat{\bm{x}} \vert

eq. 2 recurrent state update §2.3:

\bm{h}_t = f(\mathbf{W}_{hh}\bm{h}_{t-1} + \mathbf{W}_{xh}\bm{x}_t + \mathbf{b})

eq. 3–eq. 5 LSTM input, forget and output gates, each a sigmoid of a linear map of [\bm{x}_t, \bm{h}_{t-1}] §2.3:

\bm{i}_t = \sigma(\mathbf{W_{i}} \cdot [\bm{x}_t, \bm{h}_{t-1}])

\bm{f}_t = \sigma(\mathbf{W_{f}} \cdot [\bm{x}_t, \bm{h}_{t-1}])

\bm{o}_t = \sigma(\mathbf{W_{io}} \cdot [\bm{x}_t, \bm{h}_{t-1}])

eq. 6 candidate input §2.3:

\bm{g}_t = \tanh(\mathbf{W_{g}} \cdot [\bm{x}_t, \bm{h}_{t-1}])

eq. 7 memory update §2.3:

\hat{\bm{h}}_t = \bm{f}_t \odot \hat{\bm{h}}_{t-1} + \bm{i}_t \odot \bm{g}_t

eq. 8 output §2.3:

\bm{h}_t = \bm{o}_t \odot \tanh(\hat{\bm{h}}_t)

eq. 9 entropy query in the committee-based method §3.2:

\bm{x}^{EQB}=\mathop{\arg\min}_{x_i \in U}\frac{H^{EQB}(x_i)}{\log(N_i)}

eq. 10 breaking-ties margin query §3.2:

\bm{x}^{BT}=\mathop{\arg\min}_{x_i \in U} \left\{ \mathop{\max}_{w \in N}p(y_i^*=w \vert x_i) - \mathop{\max}_{w \in N\setminus w^+}p(y_i^*=w \vert x_i)\right\}

eq. 11 prototype of class i as the mean embedding of its support samples §3.3:

\bm{c}_i = \frac{1}{\vert S^i \vert}\sum_{(\bm{x}_j, y_j)\in S^i}f_\phi(\bm{x}_j)

eq. 12 relation score §3.3:

r_{i,j} = f_\psi(\mathcal{C}(f_\psi(\bm{x}_i), f_\psi(\bm{x}_j)))

eq. 13 Siamese distance §3.3:

D(\bm{x}_i, \bm{x}_j) = \Vert f_{\phi 1}(\bm{x}_i) - f_{\phi 1}(\bm{x}_j) \Vert_2

eq. 14 SSLstm fusion of the spectral and spatial branch posteriors, with weights that sum to one §4.2:

P(y=j \vert x_i) = w_{spe}P_{spe}(y=j \vert x_i) + w_{spa}P_{spa}(y=j \vert x_i)

Hyperparameters

Datasets §4.1. All three are hyperspectral scenes from airborne sensors.

Per-model settings §4.2. Values as the review states them. "not stated" means the review gives no value.

Global settings §4 §4.5.

Results

Author-reported. Setup: eight models, three datasets, three label budgets, 10 runs each §4. Six of the eight models are the authors' re-implementations; S-DMM and 3DCAE use code from their original papers tab. 4. The paper gives no standard deviations. The per-class accuracies are in tab. 5–tab. 7.

Overall accuracy (OA) and average accuracy (AA) for all models, copied from the tables. Bold marks the best OA in each column.

PaviaU tab. 5

Salinas tab. 6

KSC tab. 7

Best OA per cell: PaviaU S-DMM 84.55 / CNN_HSI 95.21 / CNN_HSI 97.35; Salinas CNN_HSI 91.31 / 3DVSCNN 95.30 / 3DVSCNN 96.89; KSC S-DMM 94.48 / SSLstm 98.72 / SSLstm 99.68 tab. 5–tab. 7.

Other reported results:

• Trainable parameters, including decoder tab. 8: SAE_LR is smallest on all three datasets; TwoCnn is largest (1,379,399 on PaviaU) §4.4.
• Accuracy rises with label count for all models §4.3 fig. 10.
• Training-loss curves for the first 200 epochs are only in figures fig. 20– fig. 22; no numbers are given §4.5.

Paper vs code

The abstract links https://github.com/ShuGuoJ/HSI-Classification. The owner name matches author Shuguo Jiang (author list, front matter), so the repository is treated as the authors' own. At commit 7e32cf2 it contains only a README and a figure. The README lists eight per-method repositories (SAE-LR, S-DMM, SSDL, 3DCAE, TwoCNN, 3DVSCNN, SSLstm, CNN-HSI) and a BibTeX entry README.md. It has no LICENSE file.

The per-method repositories were not cloned or read, so no hyperparameter or algorithm step was compared against code. Everything in Hyperparameters comes from the paper text.

Discrepancies found inside the paper itself are listed under Pitfalls.

Pitfalls

• Text and table disagree on one number. The text gives S-DMM's OA on PaviaU at 10 labels as 84.45% §4.3; tab. 5 gives 84.55.
• Claimed best model on Salinas at 10 labels. The text says S-DMM has the best AA and OA on Salinas at 10 labels §4.3. tab. 6 shows 3DVSCNN with higher AA (94.07 vs 93.92) and CNN_HSI with higher OA (91.31 vs 89.69).
• Rank on PaviaU. The text says S-DMM "still wins the third place" on PaviaU §4.3. tab. 5 ranks it first on both AA and OA at 10 labels, and the text earlier says the same §4.3.
• "3DCAE achieves the worst performance on all data sets." §4.3 Not at Salinas with 10 labels: SAE_LR has lower AA (67.91 vs 75.58) and OA (67.43 vs 71.50) tab. 6.
• Autoencoder claim. The conclusion that few-shot, active, transfer and augmentation methods beat autoencoder methods "in all experiments" §4.3 does not hold for SSDL, a two-stream model with a 1D autoencoder §4.2. SSDL's OA is above 3DVSCNN (active learning) on KSC at 50 labels (96.88 vs 96.03) and at 100 labels (98.75 vs 98.55) tab. 7.
• Parameter-count claims. The text says 3DCAE has at most eight times the parameters of SSDL §4.4. tab. 8 gives ratios of about 9.2 on Salinas (447,315 / 48,718) and 9.5 on KSC (425,139 / 44,967). The text names 3DCAE and then SSLstm as the models with the most parameters after TwoCnn §4.4; on PaviaU SSLstm (367,506) is larger than 3DCAE (256,563) tab. 8.
• Count of paradigms. The introduction says "four mainstream learning paradigms" §1 but names three (transfer, active, few-shot); the section list in §3 has three subsections.
• Eq. 9 and its description disagree. eq. 9 is written with argmin of entropy, which picks the lowest-entropy sample. The text says samples with large entropy are selected §3.2. The printed form and the prose cannot both be right.
• Eq. 12 notation. The relation network embedding is written f_ψ, but f_ψ is defined as the relation function in §3.3. The embedding is f_φ.
• Eq. 1 writes the reconstruction loss as "min" of a norm. Read it as the norm to be minimised §2.1.
• "Exact same experimental setting". §4 says the protocol makes the comparison fair. The per-model neighbourhoods range from 5×5 to 64×64, and PCA is used by some models and not others §4.2. Only the label counts and runs are fixed across models.
• Per-class budgets on KSC. Hardwood swamp has 105 labelled pixels tab. 3. At 100 labels per class, 5 of them would remain for testing if the test set is the rest of the labelled pixels, which the paper does not state for this case §4.
• Validation and tuning. The text says each model's "settings have been optimally tuned" §4.2. No validation split or tuning procedure is described, so this claim cannot be checked.
• Averaging. The text says results are averages over 10 runs §4.3, and it also says the map label is a vote over the 10 runs §4.3. The tables do not say whether they are means, and no standard deviation is given.
• 3DVSCNN selection wording. The review says the SVM picks, from 80% of the samples, a valuable set of 4 samples per iteration §4.2. Whether 80% is the candidate pool, and how many iterations run, is not stated.
• TwoCnn sources. The source datasets (Pavia Centre, Indian Pines) are not described in §4.1, and the number of bands kept after the variance filter is not stated §4.2.
• Missing training details. No optimiser, learning rate, epochs, batch size or weight decay is stated for seven of the eight models; CNN_HSI's batch size of 16 §4.2 is the only one given. The models cannot be reimplemented from this text alone. Use the per-model repositories (not reviewed here) or the original papers.
• Model origin. tab. 4 marks S-DMM and 3DCAE as original code and the other six as the authors' re-implementations. Results for the six are therefore not the original authors' numbers.
• Convergence claims are figure-based. Statements such as "after approximately 3 epochs the loss stabilises" §4.5 come from figures only.
• Seeds. The protocol draws random pixel sets §4; no fixed train/test files or random seeds are given, so the exact splits cannot be reproduced.

Lineage

• Contrasted with: earlier HSI reviews, which the paper says sort work by feature-fusion type rather than by label scarcity §1.
• Benchmarks: the eight methods listed in §4.2.
• Compared against: the same eight models, benchmarked here. No other baseline is used.
• Superseded by: not stated.