---
kind: concept
slug: cross-modal-retrieval
title: Cross-modal retrieval
summary: >-
  Retrieving images from a text query, or text from an image, by scoring image-text pairs and ranking them. Variants differ in how they build the image vector (global, multiscale or object-graph), how they gate the text, and how much pretraining data they use. Author-reported results are on remote-sensing benchmarks.
tags: [retrieval, multimodal, remote-sensing, metric-learning]
papers: ["2204.09860", "2204.09868", "2312.12856"]
related: [dense-retrieval, multimodal-fusion, metric-learning, vision-language-pretraining, zero-shot-learning, graph-neural-network, attention-mechanism, contrastive-learning]
status: draft
updated_at: 2026-11-08
---

## Overview

Given a query in one modality, such as a sentence, return the items in the
other modality, such as images, that match it best. The shared invariant is
that every image and every caption is scored against the others, and the
gallery is ranked by that score. Evaluation then uses recall at k in each
direction, often averaged into one mean recall [arxiv:2204.09868 §V-A].

The three cards below all come from remote sensing. Two are retrieval models
([GaLR](/papers/2204.09860), [AMFMN](/papers/2204.09868)); the third is a
pretraining dataset that also reports retrieval as one evaluation
([SkyScript](/papers/2312.12856)).

## Variants

### Global plus object-graph fusion (GaLR)

The image side has two branches. A CNN gives a global vector for the whole
scene. A graph convolution over detected objects gives a second vector. A
fusion module (MIDF) combines them, and a bidirectional GRU encodes the
caption [arxiv:2204.09860 §3.A] [arxiv:2204.09860 eq. 1-2]. Training uses a
bidirectional triplet loss with margin 0.2 in the released config
[arxiv:2204.09860 §3.E] [arxiv:2204.09860 eq. 31]. Card:
[GaLR](/papers/2204.09860).

A post-processing step, multivariate re-ranking, uses reverse search over the
gallery. It needs the full similarity matrix at query time [arxiv:2204.09860
§3.D].

### Multiscale attention and a text gate with a learned margin (AMFMN)

A ResNet-18 gives a global image vector. A multiscale visual self-attention
module picks salient regions from low- and high-level maps and multiplies the
result into that vector [arxiv:2204.09868 §III-C]. A visual-guided gate
reweights the text, in one of three forms (soft, fusion, similarity)
[arxiv:2204.09868 §III-D]. The triplet margin is per pair and depends on how
similar the captions are, with its strength set by a constant gamma between 0
and 1 [arxiv:2204.09868 §III-E eq. 29-30]. Card:
[AMFMN](/papers/2204.09868).

The released code uses a constant margin of 0.2 and does not include the
keyword branch, so the paper's headline setting cannot be reproduced from it.
The [AMFMN card](/papers/2204.09868) lists these gaps in detail.

### Pretraining on a large geo-tagged pair set (SkyScript and SkyCLIP)

Instead of changing the architecture, this variant builds data. Satellite
images are matched by coordinates to map tags, which become captions, and the
top half by CLIP similarity is kept [arxiv:2312.12856 §Dataset]. CLIP is then
continually pretrained on the result (SkyCLIP) [arxiv:2312.12856 §Experiments].
Card: [SkyScript](/papers/2312.12856).

The retrieval numbers here are zero-shot on benchmarks the model did not train
on. The authors say SkyCLIP-30 beats CLIP and CLIP-laion-RS on those
benchmarks by +2.57 to +6.48 mean recall [arxiv:2312.12856 §Experiments]
(Cross-modal retrieval).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need a model for scene-level captions and have no detector | Global-only or AMFMN-style branch | On RSITMD, the global-only ablation gives mR 24.83 (CNN) and 27.46 (CNN with MVSA) [arxiv:2204.09860 §4.F] [arxiv:2204.09860 tab. 4] |
| Objects and their relations matter, and you can run a detector | GaLR-style graph branch | Local-only features alone score far lower (mR 7.61 with a GCN), and the fused model reaches 31.00 on RSITMD [arxiv:2204.09860 §4.F] |
| You can afford the reverse-search step at query time | GaLR with multivariate re-ranking | Re-ranking adds 0.41 mR on RSITMD (31.00 to 31.41) [arxiv:2204.09860 tab. 3] |
| Query time is tight and you need a small model | Compare costs before choosing | GaLR reports 4.20 ms for one timing row against 3.51 ms for AMFMN, with row labels that the paper does not define [arxiv:2204.09860 tab. 5] |
| You have no paired remote-sensing data and can run CLIP-scale training | SkyCLIP-style pretraining | The authors' zero-shot retrieval gains are on held-out benchmarks [arxiv:2312.12856 tab. 3] |
| You plan to use the released AMFMN code as-is | Rebuild the dynamic margin from the paper's equations | The released code lacks the dynamic margin, so its results will not match the headline figure [arxiv:2204.09868 §III-E eq. 29-30] |

Caveats on the table. All headline numbers are author-reported on
remote-sensing sets (RSICD, RSITMD, UCM, Sydney). They are not from a common
protocol across papers, and the cards do not say whether baselines were re-run
[arxiv:2204.09868 §V-C] [arxiv:2204.09860 §4.C].

## Implementation sketch

Language-agnostic core shared by the variants. The branch details differ, as
noted in each variant above.

```pseudocode
# Encode the image into one vector; GaLR fuses two branches here [arxiv:2204.09860 §3.A]
v = image_encoder(image)
# Encode the caption into a vector of the same size [arxiv:2204.09868 §III-B]
t = text_encoder(caption)
# Score the pair by cosine similarity [arxiv:2204.09860 eq. 1-2]
s = cosine(v, t)
# Bidirectional triplet hinge over in-batch negatives with margin eps [arxiv:2204.09860 eq. 31]
loss = sum over negatives of relu(eps - s[pos] + s[neg_text])
     + sum over negatives of relu(eps - s[pos] + s[neg_image])
# Rank the gallery for each query; mR is the mean of six recall values [arxiv:2204.09868 §V-A]
rank = argsort(-scores, axis=gallery)
mR = mean(recall_at_1_5_10_both_directions)
```

## Contradictions

- **AMFMN, RSITMD, sentence retrieval.** The AMFMN card reports mR 29.72 with
  the dynamic margin [arxiv:2204.09868 tab. 2]. SkyScript reports AMFMN on
  RSITMD as 25.74 (i→t) and 33.69 (t→i), in parentheses, which the table marks
  as in-domain [arxiv:2312.12856 tab. 3]. The two papers report different
  quantities, so these figures are not directly comparable. Neither source
  resolves this.
- **Direction labels.** The GaLR card notes that its column names ("Sentence
  retrieval", "Image retrieval") are not defined in the text as image-to-text
  or text-to-image [arxiv:2204.09860 tab. 1]. SkyScript labels GaLR's RSITMD
  values as i→t 29.65 and t→i 33.17 [arxiv:2312.12856 tab. 3]. Read the
  direction of any reported number with that in mind.

## Papers

- [GaLR](/papers/2204.09860): global and object-graph fusion with multivariate re-ranking on RSICD and RSITMD.
- [AMFMN](/papers/2204.09868): multiscale visual self-attention, visual-guided text gate and a caption-aware triplet margin.
- [SkyScript](/papers/2312.12856): geo-tagged remote-sensing image-text dataset and SkyCLIP continual pretraining, with zero-shot retrieval results.