Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval

In plain words

Satellite and aerial photos are hard to search with a sentence. A single aerial picture holds many small objects, and much of it is background that the description does not mention. Methods built for everyday photos tend to treat the whole picture alike and miss the small details that matter.

The paper's model, AMFMN, first looks for the most important parts of the image at several sizes and keeps only those. It then lets that image summary steer how the sentence (or a list of keywords) is read, so the words that matter for this picture get more weight. Images and text are compared with a single similarity score, and training pushes matching pairs above mismatched ones.

Many aerial photos look alike, so one caption can fit several images. The authors therefore adjust how strongly the training pushes a pair apart, based on how similar the captions are. They also release a new dataset, RSITMD, with more detailed captions and keywords, which is the main benchmark for the results.

TL;DR

• Task: retrieve remote-sensing images from a text query (a sentence or keywords), and the reverse §I, §V-A.
• Method: a ResNet-18 image encoder with a multiscale visual self-attention module (MVSA), a visual-guided gate on the text, and a triplet loss whose margin depends on caption similarity §III.
• Headline (author-reported): on RSITMD, sentence-only mean recall 29.72 vs 26.92 for MTFN; with keywords added, 30.58 tab. 2, tab. 4.
• Caveat: the released code does not contain the dynamic margin or the keyword branch, and several details disagree with the paper (see Paper vs code).

Problem

Traditional text-image retrieval for remote sensing (RS) suffers from three issues §I:

1. RS images contain many objects, many unrelated to the query, and targets appear at very different scales. Global CNN features average these away, and small targets fade in deeper layers §III-B.
2. RS scenes of one class look very similar. One caption can match several images, so a hard match / no-match label is ambiguous during training §I; §III-E.
3. Existing RS caption datasets (Sydney, UCM, RSICD) use coarse or repeated sentences, so they are poor tests of fine-grained matching §IV-A.

The authors also argue that retrieval by keywords and by sentence are different tasks, and that a joint query can help §IV-B; §V-D.

Method

The paper uses the following symbols. \hat v, \hat t: projected image and text features; \sigma: sigmoid; \odot: element-wise product; \oplus: vector summation; \mathrm{Cat}: channel concatenation §III-A.

1. Global image feature. A ResNet-18 pre-trained on ImageNet gives one global vector per image §III-B eq. 5; §V-B.

   v^{(g)} = \mathrm{CNN}(I, \theta_I) eq. 5

2. Multiscale maps. Feature maps from the first three convolution stages are upsampled and concatenated into a low-level map. Maps from the last two stages are concatenated into a high-level map §III-B eq. 6-9.

3. Multiscale visual self-attention (MVSA). §III-C; fig. 3

   • Project the low-level map with a 3x3 convolution and the high-level map with a 1x1 convolution, then apply PReLU eq. 14-15.
   • Build a joint map: the mean of the high-level features, plus the concatenation of the two projected maps eq. 16.
   • Apply an L2 normalisation, then two 1x1 convolutions, one of which is passed through a sigmoid. Multiply the two results eq. 17-19.
   • Turn the result into a salient mask with a sigmoid and multiply it with the global vector to get the final image feature F_v §III-C, after eq. 19.

   v^{(lh)} = \mathrm{Mean}(v^{(h)}) \oplus \mathrm{Cat}(\hat v^{(l)}, \hat v^{(h)}) eq. 16

   v^{(i)} = \mathrm{conv}_{1\times1}(L_2(v^{(lh)})), \quad v^{(w)} = \sigma(\mathrm{conv}_{1\times1}(L_2(v^{(lh)}))), \quad \hat v^{(i)} = v^{(i)} \odot v^{(w)} eq. 17-19

4. Sentence encoder. Words are embedded and read by a bidirectional GRU. The state for each step is the average of the forward and backward states. The sentence vector is the mean over steps §III-B eq. 10-12.

5. Keyword encoder. Keywords share the word-embedding matrix with the sentence. A GRU reads the keywords, and an MLP maps the mean of the keyword states to a vector §III-B eq. 13.

6. Visual-guided attention (VGA) lets the image feature steer the text. The paper gives three variants §III-D; fig. 4:

   • Soft: the text is multiplied by a sigmoid gate computed from the image eq. 20.
   • Fusion: image and text are concatenated, then two linear layers form a gated product eq. 21-22.
   • Similarity: image and text are projected, a cosine similarity is passed through a sigmoid, and the result gates the text eq. 23-25.

   t^{(vg)} = \sigma(\mathrm{linear}(v)) \odot t eq. 20

   t^{(vg)} = \sigma\big(S(f^{(v)}, f^{(t)})\big) \odot f^{(t)} eq. 25; S is cosine similarity, §III-D

7. Dynamic fusion of sentence and keywords. A gate computed from the two L2-normalised text vectors weights the sentence feature and the keyword feature, which are then summed §III-D eq. 26-27. The final text feature F_t is the output.

   F_t = \big(\sigma(s^{(w)}) \odot s^{(v)}\big) \oplus \big((1 - \sigma(s^{(w)})) \odot s^{(k)}\big) eq. 27; the gate weights follow the text of §III-D

8. Similarity. The paper writes the image-text score as an element-wise product of projected features §III-A eq. 1-4. It does not define how that product becomes one number. The released code uses cosine similarity; see Paper vs code.

9. Training loss. The base loss is a triplet hinge over all negatives in the batch, in both directions (the VSE++ form) §III-E eq. 28. The paper's variant replaces the fixed margin \alpha with a per-pair margin \alpha_{ct} that shrinks as the captions become more similar §III-E eq. 29-30; fig. 5-6.

   \alpha_{ct} = \gamma\,\frac{e^{\beta} - e^{\beta S(T_I, T)}}{e^{\beta} - 1}, \qquad \gamma \in (0, 1) eq. 29

   L_{ct}(I, T) = \sum_{\hat T} [\alpha_{ct} - S(I, T) + S(I, \hat T_h)]_+ + \sum_{\hat I} [\alpha_{ct} - S(I, T) + S(\hat I_h, T)]_+ eq. 30

   Here S is a caption-similarity score in (0, 1), computed from BLEU and METEOR §III-E. \gamma is the maximum margin and \beta controls how fast the margin falls §III-E; fig. 6.

The full pseudocode below follows §III. It covers the sentence-only soft-VGA variant. The keyword branch (eq. 13, 26-27) is left out. The paper gives no algorithm block.

# AMFMN training step, sentence-only, soft visual-guided attention
# Assembled from §III: the paper has no algorithm block.
# Input:  images (B, 3, 256, 256); captions (B, T) token ids; lengths (B,)
# Output: scalar loss over the B x B image-caption score matrix

def training_step(images, captions, lengths):
    # 1. Global CNN feature of each image  [§III-B eq. 5]
    global_feat = global_cnn(images)                      # (B, 512)

    # 2. Low-level and high-level multiscale maps  [§III-B eq. 6-9]
    low_maps = upsample_and_concat(first_stages(images))  # (B, C1, H1, W1)
    high_maps = upsample_and_concat(last_stages(images))  # (B, C2, H2, W2)

    # 3. Project both maps to a matching size  [§III-C eq. 14-15]
    low_feat = prelu(conv3x3_downsample(low_maps))        # (B, C, h, w)
    high_feat = prelu(conv1x1(high_maps))                 # (B, C, h, w)

    # 4. Joint map: high-level mean plus concatenation  [§III-C eq. 16]
    joint = mean_high(high_feat) + concat(low_feat, high_feat)

    # 5. Redundancy filter: L2, 1x1 convs, sigmoid gate  [§III-C eq. 17-19]
    filtered_feat = conv1x1(l2_norm(joint))
    gate_map = sigmoid(conv1x1_gate(l2_norm(joint)))
    filtered_feat = filtered_feat * gate_map

    # 6. Salient mask applied to the global feature  [§III-C, after eq. 19]
    salient_mask = sigmoid(stretch(filtered_feat))        # "stretch" undefined
    visual = global_feat * salient_mask                   # F_v, (B, 512)

    # 7. Sentence vector from the bi-GRU, averaged  [§III-B eq. 10-12]
    word_vecs = embed_words(captions)                     # (B, T, 300)
    states = bigru(word_vecs, lengths)                    # (B, T, 512)
    sentence = mean_over_time(states)                     # (B, 512)

    # 8. Visual-guided soft gate for every image-caption pair  [§III-D eq. 20]
    gate = sigmoid(linear(visual))                        # (B, 512)
    text_vis = gate[:, None, :] * sentence[None, :, :]    # (B, B, 512)

    # 9. Score each pair by cosine similarity  [code: AMFMN.py#L55-L57]
    # The paper's eq. 1-4 use an element-wise product instead.
    scores = cosine(visual[:, None, :], text_vis, dim=-1) # (B, B)

    # 10. Per-pair margin from caption similarity  [§III-E eq. 29]
    # Not in the released code; see Paper vs code.
    margin = gamma * (exp(beta) - exp(beta * caption_sim)) / (exp(beta) - 1)

    # 11. Positive score on the diagonal  [§III-E eq. 28]
    pos = diagonal(scores)                                # (B,)

    # 12. Text-side hinge: other captions are negatives  [§III-E eq. 30]
    cost_text = relu(margin[:, None] - pos[:, None] + scores)

    # 13. Image-side hinge: other images are negatives  [§III-E eq. 30]
    cost_image = relu(margin[None, :] - pos[None, :] + scores)

    # 14. Remove the positive pairs themselves  [§III-E eq. 28]
    cost_text = cost_text * (1 - eye(B))
    cost_image = cost_image * (1 - eye(B))
    return sum(cost_text) + sum(cost_image)

Hyperparameters

Values marked "paper" come from the text or tables. "Code" entries come from the released files. Where the two differ, Paper vs code gives details.

Results

Author-reported. Setup: RSITMD test split unless stated; ResNet-18 backbone; mR = mean of six recall values §V-A. Baselines as listed in the paper's tables; the paper does not say whether it re-ran them §V-C.

The paper's text gives RSITMD keyword-retrieval results as AMFMN 20.80 mR against MTFN 17.77 §V-D. Table II's column names ("Sentence Retrieval", "Image Retrieval") are not defined in the text as image-to-text or text-to-image.

Paper vs code

Code at commit c09bcfba90e8283f98317af3c0e35005c3587925 (repository from the author's account; the top-level README names Zhiqiang Yuan as author [code: README.md#L10]). The repository has no LICENSE file, so the code can be cited but not copied.

• Dynamic margin not released. The paper's loss uses alpha_ct (eq. 29-30). The code's loss takes a constant margin, and the text_sim_matrix argument of calcul_loss is never used utils.py. The README says the Rct component, which computes Lct, "is not published this time" [code: AMFMN/README.md#L47].
• Margin value. Paper: constant margins 0.4-0.8 in Table V tab. 5. Code: 0.2 RSITMD_AMFMN.yaml.
• Keyword branch and dynamic fusion absent from the forward pass. The model builds vgmf_gate but forward only uses the sentence path [code: AMFMN/layers/AMFMN.py#L37, #L50-L53]. The keyword files are not in the repository . Keyword results (Table III, Table IV, the 30.58 headline) cannot be reproduced from this code.
• Similarity inside the model. Paper: element-wise products (eq. 1-4), and the VGA similarity variant uses cosine (eq. 23-25). Code: cosine between the visual and text vectors AMFMN.py. The similarity-VGA variant uses an element-wise product, not cosine [code: AMFMN/layers/AMFMN_Modules.py#L198-L206].
• Text similarity for eq. 29. Paper: BLEU and METEOR §III-E. The code generator for the similarity matrix uses a four-gram BLEU and the overlap of content words. It does not use METEOR generate_t2i_sim_matrix.py.
• Encoder. Paper: bi-GRU with 300-d embeddings and 512 hidden units [§III-B, §V-B]. Code: a skip-thoughts encoder imported from an external skipthoughts module that is not in the repository, then a linear layer from 2400 to 512 with ReLU AMFMN_Modules.py. Its pre-trained weights come from a third-party file host named in the README.
• Low- and high-level maps. Paper: first three stages form the low-level map, last two form the high-level map §III-B eq. 6-9. Code: low = stage 1 plus upsampled stage 2; high = stage 3 plus upsampled stage 4 [code: AMFMN/layers/AMFMN_Modules.py#L52-L63].
• Residual mean. Paper: Mean(v^(h)) eq. 16. Code: mean over the channel axis of the projected high-level map, broadcast onto all channels [code: AMFMN/layers/AMFMN_Modules.py#L105].
• Attention in MVSA. Paper: L2 regularisation, then 1x1 convolutions and a sigmoid gate eq. 17-19. Code: no L2 step. A single-channel spatial sigmoid map, then a linear layer over the flattened 16 x 16 map and a sigmoid gate on the global vector AMFMN_Modules.py. The 3x3 convolution has stride 4 AMFMN_Modules.py; the paper gives no stride.
• Global vector. Paper: the CNN output directly eq. 5. Code: mean-pooled stage 4 followed by a learned linear layer, 512 to 512 [code: AMFMN/layers/AMFMN_Modules.py#L44, #L67-L68].
• Batch negatives. The dataset is indexed per caption [code: AMFMN/data.py#L53, #L77-L78]. calcul_loss masks only the diagonal [code: AMFMN/utils.py#L157-L162]. Two captions of the same image in one batch are therefore treated as negatives. Paper: silent. The hard-negative helper collect_neg is defined but never called utils.py.
• Learning schedule and batch. Paper: lr 1e-4, batch at least 128, 150 epochs §V-B. Code RSITMD: lr 2e-4, batch 100, 70 epochs [code: AMFMN/option/RSITMD_AMFMN.yaml#L22-L27]. Code RSICD: lr 1e-4 [code: AMFMN/option/RSICD_AMFMN.yaml#L27]. The comment on the LR function still says "decayed by 10 every 30 epochs" utils.py; the code does x0.7 every 20 epochs utils.py.
• Data split. Paper: 80 / 10 / 10 for each dataset §V-A. Code: a fixed test file of 452 RSITMD images (2260 captions) [code: AMFMN/data/rsitmd_precomp/test_caps.txt]. The rest is split randomly 80 / 20 into train and validation, redrawn for each of 5 folds with no seed set [code: AMFMN/train.py#L141-L179, #L198-L209]. The test script averages the 5 folds test.py. The paper mentions neither the folds nor the averaging.
• Checkpoint choice. Code: the epoch with the best validation mean recall is kept train.py. Paper: not stated.
• Caption buffer. Validation allocates 47 token slots per caption [code: AMFMN/engine.py#L107]. Longer captions would break the copy.
• Attention in VGMF. Code normalises the sentence and keyword vectors before mixing them, which eq. 27 does not show AMFMN_Modules.py. This path is not called in forward.
• Searches. The repository comes from the paper's footnote p. 2 n. 1; I did not run a GitHub search. The owner's GitHub profile matches the first author (name, PhD affiliation, and the repository README naming Zhiqiang Yuan). The repository description names both the model code and the RSITMD dataset.

Pitfalls

• The headline numbers cannot all be reproduced from the released code. The sentence-only path has no dynamic margin, the keyword branch is missing, and the text encoder needs an external package and weights. Start from the paper's equations and note where the code disagrees.
• Eq. 29 is undefined at beta = 0 (0/0). The paper gives no limit. Taking the limit gives gamma(1 - S), which is my derivation, not the paper's.
• S (eq. 29-30) is under-specified. The paper says S uses BLEU and METEOR, but not how the two are weighted, how the five captions of image I are combined, or how S is normalised to (0, 1). The released generator uses different components generate_t2i_sim_matrix.py.
• Subscript h in eq. 30 is not defined. The text does not say whether the negatives marked with h are hard negatives. The base loss (eq. 28) uses plain negatives, and no hard-negative mining appears in the code.
• Mean in eq. 16 and the sigmoid mask in eq. 19 are ambiguous. The paper does not say over which axis Mean(v^(h)) is taken, nor how a 2-D map is added to a concatenated tensor (the code averages over channels). "Stretching" in eq. 19 is undefined. The code uses a linear layer over the flattened 16 x 16 map.
• Score definition. Eq. 1-4 are written as element-wise products of projected vectors. The paper does not say how that becomes one score per pair. The code uses cosine similarity.
• Text vs table. The text says AMFMN improves mR by 1.95 over MTFN on RSICD §V-C. Table II gives 16.42 for AMFMN-fusion and 14.81 for MTFN, a difference of 1.61 tab. 2. I checked both mR values against the six recalls in each row.
• Two headline settings. Table II reports sentence-only AMFMN (29.72 at best). The joint sentence-plus-keyword model reports 30.58 tab. 4; tab. 6. Cite the setting you use.
• Keyword cleaning is vague. The authors remove words that "frequently appeared" in keywords during training. The threshold is not stated §V-D.
• Split and folds. Paper: 80 / 10 / 10 §V-A. Code: a 452-image test set (about 9.5% of the 4743 images), and training on about 72% of the images [code: AMFMN/data/rsitmd_precomp/]. Whether the paper's numbers are means over folds is not stated. Seeds are not fixed, so reruns will differ.
• Single learning rate. Adam sets one learning rate for the backbone and the heads train.py. The paper does not say whether the backbone gets a different rate.
• Rotation and crop. A 256 crop from a 278 image rotated by 45 degrees includes pixels outside the rotated image (fill colour not stated, default black in torchvision). The paper gives no rotation range data.py. This is my reading of the geometry, not something the paper reports.
• Code is written for old PyTorch. Calls such as yaml.load without a loader (AMFMN/train.py#L35), Variable(..., volatile=True) (AMFMN/utils.py#L261) and F.sigmoid will need changes on current versions. I read the code and did not run it.
• Hard-coded CUDA. VGMF_Fusion builds its ones tensor with .cuda() AMFMN_Modules.py.
• Dataset provenance. Some RSITMD images come from Google Earth §IV-B. The paper does not state the redistribution terms for these images or for the captions. This card does not republish the dataset.
• Repository licence. The repository has no LICENSE file, so the code is cite-only. This card quotes no code.

Lineage

• Builds on: the triplet hinge loss of VSE++ (arXiv:1707.05612) §III-E eq. 28; visual-guided soft attention (Xu et al., cited as [42] in the paper) §III-D; the mean-recall measure of Huang et al. [48] §V-A.
• Compared against: VSE++ [34], SCAN [35], CAMP [51], and MTFN [36], as listed in the paper's tables §V-C; tab. 2.
• Superseded by: not stated in this wiki.