Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information

In plain words

Searching satellite and aerial photos with a sentence ("a baseball field next to four tennis courts") is harder than searching ordinary photos. Most such systems compare one summary vector per image with one vector per sentence. That summary mostly describes the whole scene, so small objects and how they relate to each other get lost, and aerial scenes are full of near-identical objects such as rows of cars.

GaLR keeps both views. One branch is a standard image encoder that summarises the whole scene. The other finds the objects in the image, treats them as nodes in a small graph, and lets a graph network turn them into a second vector. A fusion module then uses each view to correct or fill in the other, and learns how much to trust each one. Redundant objects are merged and large objects are given more weight in the graph, so the network focuses on salient targets.

At search time the method adds a cheap post-processing step. It looks at the top results, runs the search backwards from those results to the query, and combines the two rankings. No extra training is needed for this step.

TL;DR

• Text-image retrieval for remote sensing images, trained with a triplet loss.
• Image side: a global CNN embedding plus a graph convolution over detected objects, fused by a multi-level module (MIDF) §3.A §3.C.
• Text side: a bidirectional GRU encoder §3.A.
• Post-processing: a "multivariate rerank" that uses reverse search §3.D.
• Author-reported: best mean recall on RSICD and RSITMD among the compared methods, with the largest gain on RSICD tab. 1 §4.C.
• Not reproducible from the public code: the object detector and the graph construction (DREA) are not in the repository §3.B note.txt.

Problem

Global image embeddings ignore the relations between targets and let redundant targets dominate, which hurts fine-grained matching §1 §3.A. Using only local object features loses the overall scene, and the authors report that local features alone perform poorly §4.F. Earlier retrieval models also ignore the bidirectional structure of the similarity matrix at inference time §1 §3.D.

Method

Section letters follow the paper: §3.C means Section III-C. Equations and tables are cited as printed.

Overview. Images and text are mapped to vectors of the same size, and the score is the cosine similarity eq. 1 eq. 2. The image vector comes from two branches fused by MIDF eq. 6; the text vector comes from a bidirectional GRU §3.A eq. 7-eq. 10 fig. 2.

Global branch §3.A. A CNN with a multi-level visual self-attention module (MVSA) produces a global vector eq. 3. The CNN is described as fine-tuned from a pre-trained backbone; the paper does not name the backbone here (see Hyperparameters).

Local branch §3.B. An object detector (PP-YOLO with a ResNet-50 backbone, pre-trained on DOTA) returns the objects in the image §3.F eq. 4. A graph convolution network (GCN) over these objects gives a local vector eq. 5. Before the GCN, DREA builds a denoised representation matrix and an enhanced adjacency matrix §3.B:

• Pair filtering: two objects are merged when their similarity passes a threshold and they share a category. The merged object gets the average centre, the summed area and the maximum probability eq. 13-eq. 16.
• Adjacency: edge weights decay with the distance between objects, so distant objects are weakened and close ones strengthened eq. 12.
• Area boost: objects are ranked by area eq. 17. The adjacency distance of a pair is rescaled by a factor that depends on the product of their area ranks, so that relations between large objects are strengthened eq. 18 §3.B.
• The GCN layer is the standard normalised propagation rule eq. 11.

Text branch §3.A. Each word is embedded eq. 7; a forward GRU and a backward GRU give hidden states eq. 8 eq. 9. The text vector is an MLP applied to the mean of the averaged forward and backward states over all positions eq. 10.

MIDF fusion §3.C. The global and local vectors pass through a self-attention block each eq. 19. Each is then guided by the other through a guided-attention block eq. 20. The global vector is gated by the sigmoid of the local one, and the local vector is extended by adding the global one eq. 21. The two are summed eq. 22, and a two-value softmax gives the weights γ1 and γ2 eq. 23. The output mixes the two streams with those weights eq. 24. The paper describes ⊗ as a dot product and ⊕ as addition §3.C.

# MIDF: fuse one global and one local vector  [§3.C] [eq. 19-24]
# Input:  global_vec (512,), local_vec (512,)
# Output: visual vector (512,)

def midf(global_vec, local_vec):
    # Self-attention within each stream  [eq. 19]
    global_sa = self_attention(global_vec)
    local_sa = self_attention(local_vec)
    # Each stream guided by the other  [eq. 20]
    global_sga = guided_attention(global_sa, local_sa)
    local_sga = guided_attention(local_sa, global_sa)
    # Global gated by local; local extended by global  [eq. 21]
    global_gated = global_sga * sigmoid(local_sga)
    local_extended = local_sga + global_sga
    # Mixed vector used to predict the dynamic weights  [eq. 22] [eq. 23]
    mixed = global_gated + local_extended
    weights = softmax(sigmoid(mixed @ W_alpha) @ W_beta)   # (2,)
    # Weighted sum of the two streams  [eq. 24]
    return weights[0] * global_gated + weights[1] * local_extended

Multivariate rerank (MR) §3.D. A post-processing step over the full similarity matrix, with no retraining §3.D fig. 4. For an image query eq. 25:

• Take the top k texts. Each gets a forward score that decays with its rank eq. 26.
• For each candidate text, run a reverse search over the image collection and record where the query image ranks. Only the top l reverse images contribute eq. 27 eq. 28.
• Confidence: the candidate's similarity to the query, divided by its total similarity to all images eq. 29.
• Final score: the forward term plus two weighted terms, the reverse term and the confidence term eq. 30.

# Multivariate rerank for image-to-text queries  [§3.D] [eq. 25-30]
# Input:  sims (num_images, num_texts), top k, weights w_1, w_2
# Output: reranked scores (num_images, num_texts)

def multivariate_rerank(sims, k, w_1, w_2, xi):
    # Candidate texts and their forward rank  [eq. 25] [eq. 26]
    for each image i:
        candidates = top_k_texts(sims[i], k)
        for rank, text in enumerate(candidates):
            forward[i, text] = exp(-xi * (rank + 1))
            # Confidence of this candidate over all images  [eq. 29]
            confidence[i, text] = sims[i, text] / sum(sims[:, text])
            # Reverse search: where image i ranks for this text  [eq. 27]
            reverse_rank[i, text] = rank_of(i, among_images_of(text))
            # Turn the reverse position into a score  [eq. 28]
            reverse[i, text] = exp(-xi * (reverse_rank[i, text] + 1))
    # Weighted sum of the three components  [eq. 30]
    return forward + w_1 * reverse + w_2 * confidence

The paper's text sets the two weights to 0.5 and 1.25 and k to 25, and does not state ξ §4.B §4.E. The reverse term above follows the paper. The code uses a different reverse term, which is listed in Paper vs code.

Training objective §3.E eq. 31. A bidirectional triplet loss with margin ε, summed over all in-batch negatives in both directions.

# Bidirectional triplet loss over one batch  [§3.E] [eq. 31]
# Input:  scores (batch, batch), cosine similarities
#         scores[i, j] is image i against text j; margin is 0.2
# Output: scalar loss

def triplet_loss(scores, margin):
    # Matched pairs sit on the diagonal  [§3.E]
    positive = diagonal(scores)                       # (batch,)
    # Each image against every wrong text  [eq. 31] first sum
    cost_text = relu(margin + scores - positive[:, None])
    # Each text against every wrong image  [eq. 31] second sum
    cost_image = relu(margin + scores - positive[None, :])
    # Matched pairs are not negatives  [§3.E]
    cost_text[diagonal_mask] = 0
    cost_image[diagonal_mask] = 0
    return sum(cost_text) + sum(cost_image)

Inference §3.F alg. 2. Compute the visual and text vectors for every test pair, take the cosine similarity matrix, then apply the MR step to that matrix. Training is alg. 1: the CNN with MVSA, the GCN, the MIDF module and the GRU are trained jointly with the triplet loss.

Hyperparameters

Values in the Value column come from the paper unless the Where column names a code file. Code values are listed where they differ.

Results

Author-reported. Setup: RSICD (10,921 images, 224×224, five sentences each) and RSITMD (4,743 images, 256×256), split 80/10/10 §4.A. Metrics are R@1, R@5, R@10 for sentence and image retrieval, and mR, their mean §4.A. VSE++, SCAN, CAMP and MTFN were run by the authors from their published code, fine-tuned on these datasets. AMFMN and LW-MCR numbers are taken from the literature §4.C.

Ablations on RSITMD mR §4.F tab. 4: global only 24.83 (CNN), 27.46 (CNN with MVSA); local only 7.61 (GCN), 4.19 (GCN with DREA); global plus local with addition 29.90 (GCN), 30.85 (GCN with DREA), 31.00 with MIDF; 31.41 with MR on top of MIDF.

Re-ranking on RSITMD §4.E tab. 3, k = 25: source similarity 31.00, rerank 31.21, multivariate rerank 31.41.

Cost §4.G tab. 5: GaLR takes 46.34 s (RSICD) and 8.82 s (RSITMD) for the ET row and 4.20 ms for the IT row, against 2.83 ms for LW-MCR and 3.51 ms for AMFMN. Units are as printed; the paper does not define ET or IT beyond the table.

Qualitative results: fig. 7 shows top-5 retrieval; fig. 8 shows semantic localisation with and without the local branch and MIDF §4.H.

Paper vs code

Repository: https://github.com/xiaoyuan1996/GaLR, MIT licence, linked from the abstract §1, at commit 60d65760297fd3ea7d857bb0a8b69a887cd8ec0d. Its README calls it the official PyTorch code README.md. The code header says it depends on AMFMN GaLR.py.

• DREA is not in the repository. The graph construction, the representation matrix and the detector are absent. Adjacency and object features are loaded from precomputed .npy files data.py note.txt. Equations 12 to 18 cannot be checked against code, and the column meanings of the 20×20 feature arrays are not documented.
• Dynamic weights. The paper gives two weights γ1 and γ2 eq. 24. The code applies γ1 to both streams and never uses γ2. Both streams are weighted by the first softmax output GaLR.py.
• Guided attention order. The paper's guided attention for the global stream uses the local self-attention output eq. 20. The code feeds it the already-updated local vector GaLR.py.
• Element-wise product. The paper calls ⊗ a dot product §3.C. The code multiplies element-wise GaLR.py.
• Attention over one token. The MIDF blocks add a sequence axis of length 1 before attention GaLR.py. A softmax over a single key returns 1, so each attention block reduces to its value projection and residual layers. The paper does not mention this.
• Image backbone. The paper says ResNet-50 is used for the image extractor for all compared models except LW-MCR §4.C. The code builds the CNN from ResNet-18 GaLR_utils.py; the 192 and 768 channel sizes in the code match ResNet-18 GaLR_utils.py.
• Learning rate. The paper says 1e-4 §4.B; both configs say 2e-4 RSICD_GaLR.yaml RSITMD_GaLR.yaml.
• Epochs. The paper says 70 for both datasets §4.B; the RSITMD config says 20 RSITMD_GaLR.yaml.
• Repeats. The paper says training and verification were repeated five times §4.B; the configs use three folds RSICD_GaLR.yaml. Each fold draws a new random 80/20 train/validation split of the training file train.py; the paper describes an 80/10/10 split §4.A.
• Augmentation. The paper says rotation and flip §4.B. The code uses random rotation from 0 to 90 degrees and random cropping, with no flip data.py.
• Multivariate rerank defaults. The paper uses k = 25 and weights 0.5 and 1.25 §4.E. The script's main block uses k = 30 and weights 0.30 and 1.1 rerank.py. Its functions default to 1.0 and 0.7, and the code raises k to at least 10 rerank.py.
• Weight mapping. Equation 30 assigns w_c1 to the reverse term and w_c2 to the confidence term. In the code the first weight multiplies the confidence term and the second the reverse term rerank.py rerank.py.
• Reverse term. The paper's reverse term is zero outside the top l reverse images eq. 28. The code gives every candidate a score from its position in the reverse ordering, using the decay constant 0.05 rerank.py.
• MR is not part of the test run. test_ave.py evaluates the saved model on the test set and does not call the rerank code test_ave.py. The rerank script loads a similarity file (file/rsicd.npy) that the repository does not include rerank.py.
• Loss. The code sums the hinge over all in-batch negatives, which matches eq. 31 utils.py.
• Text-side dimensions. Word-embedding and GRU classes come from an external skip-thoughts package that is not in the repository seq2vec.py.

Pitfalls

• DREA cannot be reproduced from this repository. The detector, the graph construction and the object features are missing. Implementers have to build the object graph from the paper's equations eq. 12-eq. 18, and the paper does not state the feature layout or the value of ξ. The GCN multiplies by the stored adjacency directly, and the code does not show whether that matrix was normalised as in eq. 11.
• Table II labels look swapped. In the printed table the row group labelled "LF with DREA" peaks at mR 29.90 (w_l = 0.5). The group labelled "LF w/o DREA" peaks at 30.85 (w_l = 0.4), which is the value the text gives as the best setting §4.D tab. 2. The "growth" column agrees with the swapped reading. Check against the PDF before using either.
• Reverse search needs the whole similarity matrix. The reverse ranking is computed over all images and texts §3.D rerank.py, so MR is a batch step over a stored matrix, not a per-query step. The paper does not report its run time.
• Weights and k are sensitive. The paper shows mR varying with k and with each weight fig. 6 §4.E. The code's defaults differ from the paper.
• Dataset split. The paper's 80/10/10 split and the code's 80/20 train-validation split differ, and the test file is read from a data directory that is not in the repository.
• Local branch helps only at small weights. The text says local information helps when its weight is between 0.2 and 0.6 and hurts beyond that §4.D. Under the Table II issue above, the reported shape depends on which group is correct.
• Author-reported numbers. Every table is from the authors' runs. Several baselines were re-run by the authors, and the code has settings that differ from the paper (learning rate, epochs, backbone).
• Timing units. Table V's ET and IT units are not defined; treat the comparison as indicative.

Lineage

• Builds on: AMFMN, the asymmetric multimodal feature matching network whose code the repository states it depends on GaLR.py. The paper also uses the bidirectional re-ranking idea from Wang et al. §3.D.
• Compared against: VSE++, SCAN, CAMP, MTFN, AMFMN and LW-MCR on RSICD and RSITMD tab. 1 §4.C.
• Superseded by: not known from content in this wiki.