Cross-modal retrieval

Overview

Given a query in one modality, such as a sentence, return the items in the other modality, such as images, that match it best. The shared invariant is that every image and every caption is scored against the others, and the gallery is ranked by that score. Evaluation then uses recall at k in each direction, often averaged into one mean recall 2204.09868 §V-A.

The three cards below all come from remote sensing. Two are retrieval models (GaLR, AMFMN); the third is a pretraining dataset that also reports retrieval as one evaluation (SkyScript).

Variants

Global plus object-graph fusion (GaLR)

The image side has two branches. A CNN gives a global vector for the whole scene. A graph convolution over detected objects gives a second vector. A fusion module (MIDF) combines them, and a bidirectional GRU encodes the caption 2204.09860 §3.A 2204.09860 eq. 1-2. Training uses a bidirectional triplet loss with margin 0.2 in the released config 2204.09860 §3.E 2204.09860 eq. 31. Card: GaLR.

A post-processing step, multivariate re-ranking, uses reverse search over the gallery. It needs the full similarity matrix at query time [arxiv:2204.09860 §3.D].

Multiscale attention and a text gate with a learned margin (AMFMN)

A ResNet-18 gives a global image vector. A multiscale visual self-attention module picks salient regions from low- and high-level maps and multiplies the result into that vector 2204.09868 §III-C. A visual-guided gate reweights the text, in one of three forms (soft, fusion, similarity) 2204.09868 §III-D. The triplet margin is per pair and depends on how similar the captions are, with its strength set by a constant gamma between 0 and 1 2204.09868 §III-E eq. 29-30. Card: AMFMN.

The released code uses a constant margin of 0.2 and does not include the keyword branch, so the paper's headline setting cannot be reproduced from it. The AMFMN card lists these gaps in detail.

Pretraining on a large geo-tagged pair set (SkyScript and SkyCLIP)

Instead of changing the architecture, this variant builds data. Satellite images are matched by coordinates to map tags, which become captions, and the top half by CLIP similarity is kept 2312.12856 §Dataset. CLIP is then continually pretrained on the result (SkyCLIP) 2312.12856 §Experiments. Card: SkyScript.

The retrieval numbers here are zero-shot on benchmarks the model did not train on. The authors say SkyCLIP-30 beats CLIP and CLIP-laion-RS on those benchmarks by +2.57 to +6.48 mean recall 2312.12856 §Experiments (Cross-modal retrieval).

When to use which

Caveats on the table. All headline numbers are author-reported on remote-sensing sets (RSICD, RSITMD, UCM, Sydney). They are not from a common protocol across papers, and the cards do not say whether baselines were re-run 2204.09868 §V-C 2204.09860 §4.C.

Implementation sketch

Language-agnostic core shared by the variants. The branch details differ, as noted in each variant above.

# Encode the image into one vector; GaLR fuses two branches here [arxiv:2204.09860 §3.A]
v = image_encoder(image)
# Encode the caption into a vector of the same size [arxiv:2204.09868 §III-B]
t = text_encoder(caption)
# Score the pair by cosine similarity [arxiv:2204.09860 eq. 1-2]
s = cosine(v, t)
# Bidirectional triplet hinge over in-batch negatives with margin eps [arxiv:2204.09860 eq. 31]
loss = sum over negatives of relu(eps - s[pos] + s[neg_text])
     + sum over negatives of relu(eps - s[pos] + s[neg_image])
# Rank the gallery for each query; mR is the mean of six recall values [arxiv:2204.09868 §V-A]
rank = argsort(-scores, axis=gallery)
mR = mean(recall_at_1_5_10_both_directions)

Contradictions

• AMFMN, RSITMD, sentence retrieval. The AMFMN card reports mR 29.72 with the dynamic margin 2204.09868 tab. 2. SkyScript reports AMFMN on RSITMD as 25.74 (i→t) and 33.69 (t→i), in parentheses, which the table marks as in-domain 2312.12856 tab. 3. The two papers report different quantities, so these figures are not directly comparable. Neither source resolves this.
• Direction labels. The GaLR card notes that its column names ("Sentence retrieval", "Image retrieval") are not defined in the text as image-to-text or text-to-image 2204.09860 tab. 1. SkyScript labels GaLR's RSITMD values as i→t 29.65 and t→i 33.17 2312.12856 tab. 3. Read the direction of any reported number with that in mind.

Papers

• GaLR: global and object-graph fusion with multivariate re-ranking on RSICD and RSITMD.
• AMFMN: multiscale visual self-attention, visual-guided text gate and a caption-aware triplet margin.
• SkyScript: geo-tagged remote-sensing image-text dataset and SkyCLIP continual pretraining, with zero-shot retrieval results.