Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation

In plain words

Working out where a photo was taken, without GPS metadata, can be done by matching the photo against satellite images of the same area. The street photo and the satellite tile look very different, so the paper trains one image network that turns both kinds of picture into lists of numbers, called vectors, such that the two pictures of the same spot end up close together.

Training compares each picture with all the others in its batch and asks it to pick out its true match, in both directions (street to satellite and back). The tricky part is choosing look-alike wrong answers to learn from. Early in training the method uses places that are geographically close. Later it uses pictures that the network already finds visually similar, re-checked every few epochs.

The authors report that this simple setup beats earlier designs on four public benchmarks. It does not need the polar warping step or the extra aggregation modules that earlier work relied on.

TL;DR

Sample4Geo trains a single ConvNeXt-B encoder, shared by the ground and the satellite view, with a symmetric InfoNCE loss over each batch. Its two hard- negative strategies are GPS-based neighbour sampling and Dynamic Similarity Sampling. The paper reports the best R@1 among the compared methods on CVUSA, CVACT (validation and test), University-1652 and VIGOR, without polar transformation. All numbers are author-reported.

Problem

Cross-view geo-localisation matches a query image from one viewpoint (ground street view, or a drone image) against a database of satellite images with known positions §1. Earlier work often needed a polar transform to bridge the geometry gap, aggregation modules to pool features, or two separate encoders §1 §2. A triplet loss with one negative per anchor is the standard objective, and the paper states that hard negatives with it can make the model collapse §1 §3.

The paper also argues that current benchmarks are saturating, and that generalisation to unseen cities is still weak §2 §4.1.3.

Method

The method has four parts. Each is described below with its section.

1. Shared encoder. One ConvNeXt backbone embeds both the street view and the satellite image. Features are mean-pooled, with no attention pooling and no refinement modules §3.2 fig. 2. ConvNeXt is chosen over a Vision Transformer on the basis of the ablation in §4.3.1 tab. 4.

2. Symmetric InfoNCE loss. For a batch, each street image is a query against the satellite images as references, and each satellite image is a query against the street images. The matching pair is the positive. All other items in the batch are negatives. The loss is the cross-entropy in both directions §3.1 eq. 1. The paper uses the symmetric form because it helps both view directions, and the supplement reports that a one-way loss does worse [supp. "Symmetric InfoNCE Loss", tab. 8].

3. Batch construction by GPS (early epochs). Near neighbours are found from the training coordinates and put in the same batch. This gives a starting set of hard-ish negatives before the model has learned anything §3.3.

4. Dynamic Similarity Sampling, DSS (later epochs). After a number of epochs, a full inference pass over the training data gives cosine similarities between all samples. For each query, the top K most similar samples are kept. From these, the first k/2 by similarity and k/2 drawn at random from the rest of the pool are chosen. The pool is recomputed only every e epochs. The hyperparameters are k = 64, K = 128 and e = 4 §3.4.

The paper has no algorithm environment. The pseudocode below summarises §3.1, §3.3 and §3.4. Augmentation and training settings are under Hyperparameters.

# Symmetric InfoNCE over one batch  [§3.1] [eq. 1]
# Input:  street (B, D), satellite (B, D); row i of each is a matching pair
# Output: scalar loss

def symmetric_info_nce_loss(street, satellite, temperature):
    # Dot-product scores between every street and every satellite image
    # (the paper writes a dot product; normalisation is not stated)  [§3.1]
    scores = street @ satellite.T / temperature    # (B, B)
    # Street as query, satellite images as references; diagonal = positive
    loss_street_to_sat = cross_entropy(scores, arange(B), smoothing=0.1)
    # Satellite as query, street images as references  [§3.1]
    # Label smoothing 0.1 is stated for the InfoNCE loss  [§4.2]
    loss_sat_to_street = cross_entropy(scores.T, arange(B), smoothing=0.1)
    # How the two directions are combined is not stated  [§3.1]
    return combine(loss_street_to_sat, loss_sat_to_street)

# Dynamic Similarity Sampling for one set of queries  [§3.4]
# Input:  cosine similarity matrix sims (N, N) from one inference pass
# Output: for each query, k sampled indices, no repeats within one epoch

def dss_candidates(sims, k=64, K=128):
    # Only recomputed every e = 4 epochs, to shorten training  [§3.4]
    for each query q:
        # Top-K most similar images to q, sorted by similarity  [§3.4]
        pool = top_k_indices(sims[q], K)
        # First k/2 from the sorted pool are the nearest ones  [§3.4]
        nearest = pool[: k // 2]
        # Remaining k/2 drawn at random from the rest of the pool  [§3.4]
        random_rest = random_sample(pool[k // 2 :], k // 2)
        # Skip indices already used in this epoch  [§3.4]
        selected[q] = unused(nearest + random_rest)
    # How selected indices are packed into batches of 128 is not stated  [§3.4]
    return selected

Batch construction for University-1652 cannot use GPS, so it uses plain shuffling §3.3. Because that dataset has several positives per query, the paper also uses a custom sampler that keeps one image per class in a batch §3.1 §4.2. The sampler details are not stated.

Hyperparameters

Results

Author-reported. Metrics: R@k is recall at k. R@1% is recall within the top 1% of the gallery. Hit rate is R@1 with the semi-positives masked §4.1.3. AP is average precision, used for University-1652 §4.1.2.

Setup: ConvNeXt-B, single shared encoder, symmetric InfoNCE, batch 128, 40 epochs §4.2. GPS sampling plus DSS is used for CVUSA and CVACT §4.3.2 [supp. "Sampling Strategies"]. Baselines are the methods listed in each table. Rows marked † use the polar transformation on the satellite input, and the paper says so explicitly tab. 1.

CVUSA and CVACT tab. 1

University-1652 tab. 2

VIGOR tab. 3

Parameter efficiency, VIGOR SAME tab. 4. "Shared W." is shared weights between the two views. "#Params" is in millions. The per-view count is shown for two-encoder rows.

The "Nano" row's R@1% (94.19) and Hit Rate (99.40) look inconsistent with its neighbours. They are reproduced as printed tab. 4.

Sampling strategies, VIGOR tab. 5

Generalisation, trained on one dataset and tested on the other tab. 6

Supplementary tables (table numbers continue from the main text in the source; see Pitfalls).

Neighbours k on VIGOR SAME [supp. "Hyperparameter", tab. 7]: k = 16: R@1 76.84, R@5 95.24, R@10 96.98, R@1% 99.68, Hit Rate 88.55. k = 32: 77.39, 95.61, 97.19, 99.65, 89.46. k = 64 (chosen): 77.86, 95.66, 97.21, 99.61, 89.82. k = 128: 77.84, 95.42, 97.07, 99.58, 89.69.

Loss direction on VIGOR SAME [supp. "Symmetric InfoNCE Loss", tab. 8]: Satellite to street only: 74.69, 93.80, 95.81, 99.40, 86.27. Street to satellite only: 77.49, 95.66, 97.18, 99.58, 89.57. Symmetric: 77.86, 95.66, 97.21, 99.61, 89.82.

Loss function on CVUSA [supp. "Loss comparison", tab. 9]: Triplet loss: R@1 0.00, R@5 0.06, R@10 0.10, R@1% 1.26. Soft-margin triplet: 91.83, 97.87, 98.75, 99.67. InfoNCE: 98.68, 99.68, 99.78, 99.87.

Sampling on CVUSA and CVACT test [supp. "Sampling Strategies", tab. 10]: CVUSA, random: 97.83, 99.63, 99.75, 99.89. GPS: 97.83, 99.53, 99.72, 99.87. DSS: 98.51, 99.69, 99.79, 99.85. GPS + DSS: 98.68, 99.68, 99.78, 99.87. CVACT test, random: 60.57, 89.50, 92.99, 98.92. GPS: 71.13, 90.28, 92.47, 98.09. DSS: 71.04, 92.26, 94.33, 98.58. GPS + DSS: 71.51, 92.42, 94.45, 98.70.

Paper vs code

No official code.

The paper's text, abstract and arXiv page name no code repository or project page. Searching GitHub for the method name returned a repository, Skyy93/Sample4Geo, whose README says "our ICCV accepted work" and links the arXiv paper. It is not cited from the paper, its owner's identity as an author could not be verified, and no LICENSE file is visible. We therefore cannot record code_license and did not clone it. Other GitHub results were third-party forks or extensions. The implementation details in this card are from the paper only.

Pitfalls

• Equation 1 is ambiguous. The paper writes the sum index from 0 to R, but R is also the name of the reference set §3.1 eq. 1. The denominator is assumed to include the positive, as in standard InfoNCE. The printed equation has an unbalanced closing parenthesis, which we read as exp(q · r+ / τ).
• Normalisation is not stated. The text calls the similarity a dot product §3.1. DSS uses cosine similarity §3.4. Whether embeddings are L2-normalised before the loss is not stated, and the code is not available here to check. Pick one and state it in any reimplementation.
• Symmetric combination is not stated. §3.1 says the loss is computed in both directions but does not give how the two terms are combined (sum, mean, or weights).
• Temperature initial value is not stated. It is learnable §4.2, but the starting value and any constraints are not given.
• Label smoothing scope. The paper says label smoothing 0.1 is used "in the InfoNCE loss" §4.2. It is not stated whether it applies to both directions.
• GPS phase length and switch epoch are not stated §3.3 §3.4. The text says only "early epochs" and "after a certain amount of epochs". Where the handover happens matters for reproducing the reported results.
• DSS batch assembly is ambiguous. Each query yields k selected samples, and the paper says these are "added to the batch" §3.4. How candidates from up to K = 128 queries fill a batch of 128 is not stated. Using K equal to batch size §3.4 suggests one query per slot, but this is not confirmed.
• DSS is expensive at intervals. A full inference pass over the training data is needed every e = 4 epochs §3.4. The paper frames this as a shortening trick, but the pass still costs one forward pass per image over the set.
• Deduplication inside an epoch is said to use a lookup §3.4. The data structure and what happens on collision are not stated.
• University-1652 sampling. The paper says GPS sampling is not possible there and uses shuffling §3.3. It does not say whether DSS is used on that dataset, nor how the one-image-per-class sampler is implemented §3.1 §4.2.
• Parameter count mismatch. The text says 88M §4.2, while tab. 4 says 88.5M for the same ConvNeXt-B model tab. 4.
• Polar transform comparisons are not like-for-like. Rows marked † use it tab. 1. The paper notes that polar-transform methods usually gain a little on the benchmarks §4.1.1. Compare with the † rows when assessing the no-polar setting.
• Triplet baseline setup is not stated. The collapse result (R@1 0.00) [supp. "Loss comparison", tab. 9] does not give the margin, mining scheme or batch composition for the triplet run.
• Visualisation method is not stated. The heatmap method behind fig. 3 and the supplementary maps is not named §4.3.4. The supplement applies an inverse polar transform to street-view maps for display only [supp. "Visualisation"].
• Hit rate definition. The text defines it as R@1 with semi-positives masked §4.1.3. Which semi-positives are masked, and how the masking is applied to each query, is not stated.
• Input resizing and augmentation ranges are not stated. The paper gives input sizes §4.1.1 §4.1.3 but not the resize method, rotation range, flip probability or dropout and colour-jitter strengths §4.2.
• Supplementary numbering. The LaTeX source resets the section and figure counters before the supplement, but not the table counter. We cite supplement tables as 7 to 10 by source order and cite supplement sections by title. We have not checked these numbers against a compiled PDF.
• Hard negatives and collapse. The paper says it did not observe collapse with DSS even when k = K §3.4. This is an observation, not a guarantee.
• Generalisation is weaker than same-area results. The paper states that transfer to unseen cities is not yet as strong as same-area performance §4.3.3.
• Evaluation is author-run. Baselines are the authors' reported numbers from each cited paper, and the paper does not re-run them under one protocol.

Lineage

• Builds on: symmetric contrastive loss from multimodal pre-training §3.1; ConvNeXt as the backbone §3.2; the observation that hard negatives with triplet loss can collapse §3.1; the advice to use synchronous augmentation for orientation §4.2.
• Compared against: LPN, SAFA, TransGeo, GeoDTR, SAIG-D, L2LTR, DWDR, MBF tab. 1 tab. 2 tab. 3 tab. 6.
• Superseded by: not stated in the paper.