Dual Contrastive Network for Few-Shot Remote Sensing Image Scene Classification

In plain words

The task is remote sensing scene classification: deciding which scene category an aerial or satellite image belongs to, when each new category has only a few labelled images. The method first trains an image encoder on classes that have many labels, then reuses it for the new classes.

The paper adds two extra training signals to that first stage. Both push images of the same class closer together and images of different classes further apart. One signal works on a summary of the whole scene, to separate classes that look alike. The other works on small local details, to make a class recognisable even when its images vary a lot. Both signals are used only during training. At test time the encoder is kept, each new class is represented by the average of its few examples, and a query image goes to the closest class.

TL;DR

• Transfer-based few-shot method: pre-train an encoder on base classes, then classify novel classes by prototypes.
• Training adds a context branch (Condenser Network) and a detail branch (Smelter Network), each with a supervised contrastive loss.
• Auxiliary heads and contrastive modules are discarded at meta-test time §III-E.
• Author-reported gains over the baselines in their tables, on four remote sensing benchmarks tab. 1 tab. 2. Not every 1-shot result is best: IDLN is higher on NWPU-RESISC45 1-shot tab. 2.

Problem

Few-shot scene classification with a base set (classes with many labels) and a novel set (disjoint classes, few labels each). The paper follows the transfer pipeline: supervised pre-training of an encoder on the base set, then meta-test on novel tasks with a support set of C classes and K samples per class, and a query set of Q samples per class §III-A. Motivation: remote sensing scenes have small differences between classes and large variation within a class §I fig. 1. The number of query samples Q is not given numerically for the test tasks §III-A; see the Pitfalls.

Method

The encoder is a ResNet-12 feature extractor f_\theta §IV-B. Its output feature maps are Z \in \mathbb{R}^{h \times w \times c} §III-B1. Two branches read these maps during pre-training. Both are supervised contrastive learning tasks, and both also feed a classifier §III-D.

Context branch (CCL)

Condenser Network §III-B1 fig. 3. Steps, in order:

1. Squeeze the channels with a 1×1 convolution to a single spatial map Z_s eq. 2.
2. Map Z_s to weights in 0–1 with softmax and multiply them into Z eq. 3. The paper says "each pixel" is mapped to 0–1; the softmax axis is not stated.
3. Squeeze again, apply layer normalisation and ReLU, then expand back to c channels with a 1×1 convolution §III-B1.
4. Add the result to the input Z eq. 4.
5. Global average pooling of \tilde{Z} gives the context vector Z_c \in \mathbb{R}^{c} §III-B1.

Z_{s} = f_{sq}(Z) \qquad \text{[eq. 2]}

Z_{g} = \mathrm{softmax}(Z_{s}) \times Z \qquad \text{[eq. 3]}

\tilde{Z} = f_{ex}\big(\mathrm{ReLU}(\mathrm{LN}(f_{sq}(Z_{g})))\big) + Z \qquad \text{[eq. 4]}

Contrastive learning on context §III-B2. Each training image gets two augmentations, giving 2N samples per batch. The context vectors go through a non-linear projection and L2 normalisation:

F = f_{pc}(Z_{c}) \qquad \text{[eq. 5]}

The CCL loss is a supervised contrastive loss. The positives of anchor i are the other samples with the same label. d is a normalised dot product, that is cosine similarity §III-B2:

L_{CCL} = -\sum_{i=1}^{2N} \frac{1}{P_i} \sum_{\substack{j=1 \\ j \ne i,\; y_i = y_j}}^{2N} \log \frac{\exp\big(d(F_i, F_j)/\tau\big)} {\sum_{n=1,\, n \ne i}^{2N} \exp\big(d(F_i, F_n)/\tau\big)} \qquad \text{[eq. 6]}

Here P_i is the number of positives for i and \tau is a temperature §III-B2.

Detail branch (DCL)

Smelter Network §III-C1 fig. 4. It builds two attention maps and gates the input with them:

1. Channel squeeze: average over channels, giving Z_{cs} \in \mathbb{R}^{h \times w \times 1} eq. 7.
2. Spatial squeeze: average over positions, giving Z_{ss} \in \mathbb{R}^{1 \times 1 \times c} eq. 8.
3. Spatial attention: a 3×3 convolution, an upsampling layer and a 1×1 convolution on Z_{cs} eq. 9.
4. Channel attention: two 1×1 convolutions on Z_{ss} eq. 10.
5. Multiply the two maps, apply a 1×1 convolution, pass through a sigmoid, multiply with Z, and add Z back eq. 11.

Z_{cs} = f_{c}(Z) \qquad \text{[eq. 7]}

Z_{ss} = f_{s}(Z) \qquad \text{[eq. 8]}

Z_{sp} = \mathrm{conv}_{1\times1}\big(\mathrm{up}(\mathrm{conv}_{3\times3}(Z_{cs}))\big) \qquad \text{[eq. 9]}

Z_{ch} = \mathrm{conv}_{1\times1}\big(\mathrm{conv}_{1\times1}(Z_{ss})\big) \qquad \text{[eq. 10]}

Z_{d} = \varepsilon\big(\mathrm{conv}_{1\times1}(Z_{sp} \times Z_{ch})\big) \times Z + Z \qquad \text{[eq. 11]}

Self-attention alignment §III-C2. The detail maps Z_d go through a projection f_{pf} to give one tensor used as query, key and value, each of shape hw \times \bar{c} eq. 12. For a pair (x_i, x_j), the value of each sample is re-expressed through the other sample's query and key eq. 13 eq. 14. The pair distance is the mean of two cosine similarities eq. 15.

Q = K = V = f_{pf}(Z_{d}) \qquad \text{[eq. 12]}

V_{i \mid j} = \mathrm{softmax}\!\Big(\frac{Q_{j} K_{i}^{\top}}{\sqrt{\bar{c}}}\Big) \times V_{i} \qquad \text{[eq. 13]}

V_{j \mid i} = \mathrm{softmax}\!\Big(\frac{Q_{i} K_{j}^{\top}}{\sqrt{\bar{c}}}\Big) \times V_{j} \qquad \text{[eq. 14]}

d(V_{i}, V_{j}) = \tfrac{1}{2}\big(\cos(V_{i}, V_{j \mid i}) + \cos(V_{j}, V_{i \mid j})\big) \qquad \text{[eq. 15]}

The DCL loss has the same form as eq. 6, with d(V_i, V_j) and temperature \bar{\tau} eq. 16:

L_{DCL} = -\sum_{i=1}^{2N} \frac{1}{P_i} \sum_{\substack{j=1 \\ j \ne i,\; y_i = y_j}}^{2N} \log \frac{\exp\big(d(V_i, V_j)/\bar{\tau}\big)} {\sum_{n=1,\, n \ne i}^{2N} \exp\big(d(V_i, V_n)/\bar{\tau}\big)} \qquad \text{[eq. 16]}

Total loss

Contrastive loss is the sum of the two branches, weighted by \alpha eq. 17. The cross-entropy term applies one classifier to each branch, and \beta weights the detail branch eq. 18. The full objective weights the contrastive part by \gamma eq. 19.

L_{CL} = L_{CCL} + \alpha L_{DCL} \qquad \text{[eq. 17]}

L_{CE} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{c=1}^{C} y_{ic} \log \frac{\exp(o_{ic})}{\sum_{j=1}^{C} \exp(o_{jc})} - \beta \frac{1}{N} \sum_{i=1}^{N} \sum_{c=1}^{C} y_{ic} \log \frac{\exp(\tilde{o}_{ic})}{\sum_{j=1}^{C} \exp(\tilde{o}_{jc})} \qquad \text{[eq. 18]}

L_{total} = L_{CE} + \gamma L_{CL} \qquad \text{[eq. 19]}

Here o and \tilde{o} are the logits from the last layer of the Condenser and Smelter Networks §III-D. C is the number of base classes §III-D.

Meta-test

The encoder, Condenser and Smelter are frozen. Both contrastive modules are dropped §III-E. Each novel class has a prototype from the K support samples, once from context vectors and once from the averaged detail maps eq. 20:

w_{c} = \frac{1}{K} \sum_{i=1}^{K} Z_{i}^{c}, \qquad \tilde{w}_{c} = \frac{1}{K} \sum_{i=1}^{K} \mathrm{avg}(\tilde{Z}_{i}^{c}) \qquad \text{[eq. 20]}

A query gets the average of two softmax scores over the classes eq. 21:

P(y = c \mid x_q) = \tfrac{1}{2}\Big( \mathrm{softmax}\big(\langle Z_q, w_c \rangle\big) + \mathrm{softmax}\big(\langle \mathrm{avg}(\tilde{Z}_q), \tilde{w}_c \rangle\big) \Big) \qquad \text{[eq. 21]}

The paper gives the prototype-classifier probability in eq. 1 as well: P(y=c \mid x) = \mathrm{softmax}(\langle f_\theta(x), w_c \rangle) eq. 1. The paper calls \langle\cdot,\cdot\rangle a distance function; the softmax form only makes sense if larger values mean more similar §III-A.

Pre-training step

The pseudocode below follows the equations above. It is not a copy of the paper's algorithm, because the paper gives no algorithm block. Values that the paper does not give are marked not stated.

# DCN pre-training loss for one batch  [§III-B] [§III-C] [§III-D]
# Input:  images (N, 3, 84, 84); labels (N,)
# Output: total loss (scalar)  [eq. 19]

def dcn_loss(images, labels):
    # Two augmented views per image, so 2N samples  [§III-B2]
    # Augmentation recipe: not stated (cites SimCLR-style work [22])
    views, view_labels = augment_twice(images, labels)
    # Feature encoder, ResNet-12  [§IV-B]
    maps = encoder(views)                           # Z: (2N, h, w, c)
    # Condenser: squeeze, softmax weights, squeeze, LN, ReLU, expand  [eq. 2-4]
    context_maps = condenser(maps)                  # Z~: (2N, h, w, c)
    # Global average pooling gives the context vector  [§III-B1]
    context = mean_over_positions(context_maps)     # Z_c: (2N, c)
    # Projection and L2 normalisation  [eq. 5]
    context_emb = l2_normalise(proj_context(context))  # F: (2N, 128)
    # Context contrastive loss  [eq. 6]
    # Temperature tau: not stated
    loss_ccl = supcon(context_emb, view_labels, tau)
    # Smelter: two squeezes, two attention maps, sigmoid gate  [eq. 7-11]
    detail_maps = smelter(maps)                     # Z_d: (2N, h, w, c)
    # Self-attention alignment, pairwise similarity  [eq. 12-15]
    # Projection output size c_bar: not stated
    sim_detail = aligned_similarity(proj_detail(detail_maps))  # (2N, 2N)
    # Detail contrastive loss  [eq. 16]
    # Temperature tau_bar: not stated
    loss_dcl = supcon_from_similarity(sim_detail, view_labels, tau_bar)
    # Total contrastive loss  [eq. 17]
    loss_cl = loss_ccl + alpha * loss_dcl           # alpha = 1  [§IV-B]
    # Cross-entropy on both branches; which views are used: not stated  [eq. 18]
    logits_context = head_context(context)          # head shape: not stated
    logits_detail = head_detail(mean_over_positions(detail_maps))
    loss_ce = cross_entropy(logits_context, labels)
    # beta = 0.1  [§IV-B]
    loss_ce = loss_ce + beta * cross_entropy(logits_detail, labels)
    # Total loss  [eq. 19]
    return loss_ce + gamma * loss_cl                # gamma = 1  [§IV-B]

# Meta-test: prototype classification  [§III-E] [eq. 20-21]
# Input:  support (C*K, 3, 84, 84) with labels; query (Q, 3, 84, 84)
# Output: predicted class per query

def predict(support, support_labels, query):
    # Encoder, Condenser, Smelter frozen; contrastive heads dropped  [§III-E]
    support_context, support_detail = frozen_branches(support)
    query_context, query_detail = frozen_branches(query)
    # Prototypes from the K support samples per class  [eq. 20]
    proto_context = class_means(support_context, support_labels)
    proto_detail = class_means(support_detail, support_labels)
    # Average the two softmax scores over classes  [eq. 21]
    score_context = softmax(similarity(query_context, proto_context))
    score_detail = softmax(similarity(query_detail, proto_detail))
    return argmax(score_context + score_detail)

Hyperparameters

Results

Author-reported. Setup: four public remote sensing datasets, ResNet-12 backbone, 5-way classification with 1 or 5 shots, 600 test tasks, 95% confidence intervals §IV-B. Baselines: some numbers are borrowed from the cited papers, the rest were re-implemented by the authors, so the protocols are not fully matched §IV-C.

Table 1 tab. 1: WHU-RS19 and UC Merced, accuracy (%)

Table 2 tab. 2: NWPU-RESISC45 and AID, accuracy (%)

The CSDL row is cited in the table as [41], which is a different entry in the bibliography. The text cites CSDL as [2] §I §IV-C. See Pitfalls.

Headline claims in the text (author-reported):

• WHU-RS19 1-shot: 81.74, at least 0.68 above the next best (SPNet, 81.06) §IV-C1.
• UC Merced 1-shot: 58.64, at least 1 above the next best (SPNet, 57.64) §IV-C1.
• 5-shot against SPNet: +3.63 (WHU-RS19), +3.09 (UC Merced) §IV-C1.
• NWPU-RESISC45 5-shot: +3.56 over CSDL (85.66) §IV-C2.
• AID 5-shot: +4.1 over DeepDBC (84.15) §IV-C2.
• NWPU-RESISC45 1-shot: 0.85 below IDLN (75.25) §IV-C2.

Ablation tab. 3 (WHU-RS19 and UC Merced) and tab. 4 (NWPU-RESISC45 and AID). The baseline is cross-entropy-only pre-training, following [14] §IV-D.

Table 3: ablation, WHU-RS19 and UC Merced, accuracy (%)

Table 4: ablation, NWPU-RESISC45 and AID, accuracy (%)

Training time and size tab. 5 tab. 6:

The paper states the maximum training time is under 6 hours §IV-G. The parameter overhead is small in tab. 6. The runtime and parameter counts do not include the batch size, so they cannot be reproduced exactly.

Qualitative analyses (no numbers): Grad-CAM shows the Condenser focusing on scene-relevant regions fig. 6; the Smelter's gated maps focus on local parts fig. 7; a feature-space variance comparison on NWPU-RESISC45 shows larger inter-class and smaller intra-class variance for the respective branch fig. 5.

Sensitivity on NWPU-RESISC45 only §IV-F:

• α peaks at 1 for both shot settings fig. 8 §IV-F1.
• β peaks at 0.1; β = 0 gives a small drop fig. 9 §IV-F2.
• γ peaks at 1 fig. 10 §IV-F3.

Paper vs code

No official code.

Searched: the paper text and the arXiv abstract page for links (none), and GitHub repository search for dual contrastive network remote sensing, few-shot remote sensing scene classification, Condenser Smelter contrastive, Zhong Ji Tianjin few-shot and dual contrastive few-shot. None of the results belong to the authors or their lab. No repository was cloned, and no hyperparameter could be checked against code.

Pitfalls

• The loss scale is not defined. L_{CCL} and L_{DCL} in eq. 6 and eq. 16 sum over the anchors i without averaging, while L_{CE} in eq. 18 averages over N. The relative weight of the contrastive and cross-entropy terms depends on the batch size, and the paper does not say whether the sum is intended. The temperatures \tau and \bar{\tau} are not stated.
• Sign and direction of d. Eq. 6 and eq. 16 put d in the numerator with a plus sign, so d must be a similarity: larger means same class. The paper calls d a distance in the text, and the eq. 1 and eq. 21 scores have the same wording. Implement as cosine similarity, not as a distance.
• Softmax axes. The Condenser's softmax (eq. 3) is not said to run over the h \times w positions. If it does, the weights sum to 1 over hw positions, so Z_g is much smaller than Z. The axis of the softmax in eq. 13 and eq. 14 is also not stated.
• Q = K = V from one projection (eq. 12). The paper writes all three as the same f_{pf}(Z_d). Separate projections for query, key and value are not mentioned. Implement as written, and expect that the official code may differ, since there is none.
• Projection names do not match. The method uses f_{pc} and f_{pf}, and the setup section describes f_{pg} (640 hidden, 128 out) and f_{pl} (single 128 layer) §IV-B. The mapping from these names to the method's projections is not stated. Note also that \bar c is not given.
• Classifier heads for eq. 18. The paper says the logits come from "the last layer" of the Condenser and Smelter Networks, but does not say whether these are separate linear layers on Z_c and on the averaged \tilde Z. The sizes are not given.
• Views in the CE term. The paper's N is the batch size, and the contrastive terms use 2N views. It is not stated whether the cross-entropy uses the original images, one augmented view, or both.
• Test-time prototype features. Eq. 20 defines prototypes from Z_i^c, and eq. 21 uses Z_q for the query. Whether Z_q is the pooled context vector or the map before pooling is not stated, though the prototypes are built from pooled vectors.
• Ambiguous sensitivity sweeps. The α, β and γ sweeps are shown on NWPU-RESISC45 only §IV-F. The paper does not say whether they use the validation or the test split. If the test split was used, the chosen values were selected on test data.
• Mismatched numbers in the introduction. The introduction says DCN beats the state of the art by at least 5.38% on AID 5-shot §I. Table 2 and §IV-C2 give 4.1 (88.25 against 84.15). The NWPU 5-shot figure of 3.56 agrees with the table.
• CSDL citation. Table 2 cites CSDL as [41], which the bibliography titles "A Baseline for Few-Shot Image Classification". The text cites CSDL as [2]. The CSDL row's citation should be read as unreliable.
• Dataset description. The UC Merced text says "100 images of 21 classes". This can be read as 100 images per class or 100 in total. The paper does not say which. WHU-RS19 says "at least 50 images" per class §IV-A.
• Mixed baseline protocols. MAML and RelationNet numbers are borrowed from [2] and [12]. The other baselines are re-implemented following [3] and [29] §IV-C. SPNet uses ResNet-18 tab. 1 tab. 2, not ResNet-12. Comparisons across rows are therefore not fully controlled.
• Augmentation recipe. The two augmentations are not named. Only the citation to [22] is given.
• Cost of the detail branch. Each pair in eq. 13–15 needs an hw \times hw attention, and the batch contains 2N views. The feature-map size h \times w is not stated, so the cost cannot be computed from the paper. The training time in tab. 5 is the only cost figure.
• Encoder frozen at test. At meta-test, the encoder, Condenser and Smelter are frozen, and no fine-tuning on the support set is described §III-E.

Lineage

• Builds on: CSDL, a transfer-based few-shot approach for remote sensing with a mirror-based feature extractor §I [2]. Also the supervised-pretraining baseline following [14] §IV-D and contrastive learning with augmentations [22].
• Compared against: Optimization methods (LLSR [55], MAML [17]), metric methods (ProtoNet [19], MatchingNet [69], RelationNet [20], deepEMD [66], FRN [67], PSM [83], DLA-MatchNet [52], SCL-MLNet [53], IDLN [9], SPNet [10]) and transfer methods (S2M2 [11], DeepDBC [12], CSDL [2]) tab. 1 tab. 2.
• Superseded by: not known from content in this wiki.