--- kind: concept slug: hard-negative-mining title: Hard negative mining summary: >- Choosing non-matching examples that lie close to the anchor, to train a metric or contrastive loss or to build hard evaluation candidates. Batch-hard mining picks the hardest pair per anchor within a batch. Sampler-based methods use GPS neighbours, then model similarities. A self-supervised model thresholds a same-frame map. Question-similarity distractors make ranked tests harder. tags: [metric-learning, contrastive-learning, sampling, retrieval, self-supervised, evaluation] papers: ["1901.09107", "2011.04530", "2104.02691", "2303.11851"] related: [metric-learning, contrastive-learning, convolutional-neural-network, self-supervised-learning, conversational-question-answering] status: draft updated_at: 2026-11-06 --- ## Overview A hard negative is a non-matching example that lies close to the anchor, usually under the current model's similarity. The three training variants below differ in where the selection happens. MinkLoc3D selects inside the loss, among the items of a batch [arxiv:2011.04530 §3.2]. Sample4Geo selects in the sampler, deciding which items share a batch [arxiv:2303.11851 §3.3] [arxiv:2303.11851 §3.4]. Localizing Visual Sounds the Hard Way selects inside the loss, among the locations of one frame, using soft thresholds on a similarity map [arxiv:2104.02691 §3.4]. All three use the model's own similarities to decide what counts as hard, and none uses a fixed set of mined negatives. A fourth variant builds hard distractors once, for ranked evaluation, from question similarity rather than from a trained model. It is not used in training [arxiv:1901.09107 §4]. Two choices sit under every variant: the distance or similarity used to rank candidates, and the rule that separates a positive from a negative. MinkLoc3D uses Euclidean distance and a fixed radius, with positives within 10 m and negatives beyond 50 m. Clouds in between are ignored [arxiv:2011.04530 §4.1]. Sample4Geo uses cosine similarity to rank candidates [arxiv:2303.11851 §3.4], and its positive is the matched pair, so it needs no threshold [arxiv:2303.11851 §3.1]. The soft-threshold method uses cosine similarity [arxiv:2104.02691 §3.2] and two thresholds with an ignored band between them [arxiv:2104.02691 §3.4]. ## Variants ### Batch-hard mining inside each batch Batches are built from pairs of similar clouds, so a batch of size n holds n/2 pairs [arxiv:2011.04530 §3.2]. For each anchor, the hardest positive is the farthest similar cloud in the batch, and the hardest negative is the closest dissimilar one. The triplet hinge uses Euclidean distance and margin 0.2 [arxiv:2011.04530 §3.2] [arxiv:2011.04530 tab. 3]. The batch size starts at 32 for baseline training in Table 3, and is multiplied by 1.4 after any epoch in which less than 70% of triplets still have non-zero loss. It is capped at 256 [arxiv:2011.04530 tab. 3] [arxiv:2011.04530 §3.2]. The paper also says that large batches can collapse [arxiv:2011.04530 §3.2]. Card: [MinkLoc3D: Point Cloud Based Large-Scale Place Recognition](/papers/2011.04530). The reported result is 97.9 AR@1% on Oxford with baseline training [arxiv:2011.04530 tab. 4]. The paper's ablations vary the aggregation layer and the descriptor size, not the mining rule [arxiv:2011.04530 tab. 7] [arxiv:2011.04530 tab. 8], so these numbers do not isolate the effect of hard-negative mining. ### GPS-neighbour batches, then similarity-based sampling Sample4Geo trains with a symmetric InfoNCE loss over a batch of 128 [arxiv:2303.11851 §3.1] [arxiv:2303.11851 §4.2]. Hard negatives enter through the batch contents, in two stages. In the early epochs, training locations that are geographically close are put in the same batch, using their coordinates. The paper says this gives a first set of hard-ish negatives before the model has learned anything [arxiv:2303.11851 §3.3]. In the later epochs, Dynamic Similarity Sampling (DSS) takes over. Every e = 4 epochs, a full inference pass over the training data gives the cosine similarities between all samples. For each query, the top K = 128 most similar samples are kept. From these, the first k/2 by similarity and k/2 drawn at random from the rest are selected, with k = 64 [arxiv:2303.11851 §3.4]. The paper reports no collapse with DSS, even when k equals K, and presents this as an observation rather than a guarantee [arxiv:2303.11851 §3.4]. Card: [Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation](/papers/2303.11851). Not stated in the paper: the epoch at which GPS sampling switches to DSS [arxiv:2303.11851 §3.4], how the selected candidates fill batches of 128 [arxiv:2303.11851 §3.4], and how the two directions of the InfoNCE loss are combined [arxiv:2303.11851 §3.1]. The paper uses GPS sampling for CVUSA and CVACT. For University-1652, where coordinates cannot be used, it uses plain shuffling [arxiv:2303.11851 §3.3]. ### Soft-threshold background negatives in the same frame Localizing Visual Sounds the Hard Way has no labelled sound regions, so it takes its negatives from the same frame as the positive. The cosine map between a clip's audio and each location of its own frame is thresholded. Locations above ε_p = 0.65 get positive weight, locations below ε_n = 0.4 get negative weight, and locations between the two get no weight [arxiv:2104.02691 §3.4] [arxiv:2104.02691 §5.3]. The hard negative score is the weighted mean over the background locations [arxiv:2104.02691 §3.4]. The paper chose these two values by ablation [arxiv:2104.02691 tab. 4]. Card: [Localizing Visual Sounds the Hard Way](/papers/2104.02691). In that ablation, models are trained on VGG-Sound 144k and tested on Flickr SoundNet. The row with both thresholds and the band reaches cIoU 0.719, against 0.675 with no negatives [arxiv:2104.02691 tab. 4]. Row b uses a negative threshold of 0.6 without the band, and it scores 0.667 cIoU and 0.544 AUC, below row a's 0.675 and 0.568 [arxiv:2104.02691 tab. 4]. The ablation varies only the thresholds and the band. It does not compare against batch-hard mining [arxiv:2104.02691 tab. 4]. All numbers are author-reported. ### Question-similarity distractors for ranked evaluation This variant is for testing, not training. The AVSD dialog benchmark ranks 100 candidate answers for each test question. The candidates are the true answers to the 50 training questions nearest to the query, the 30 most frequent answers in the dataset, the answers to 19 random questions, and the true answer [arxiv:1901.09107 §4]. Nearness is the Euclidean distance between question vectors, built from word vectors, not from a trained embedding [arxiv:1901.09107 §4]. The positive is the true answer, so no threshold is used. Card: [Audio-Visual Scene-Aware Dialog](/papers/1901.09107). The paper does not ablate this choice. Its results tables vary the input features, the dialog order and the question subsets [arxiv:1901.09107 tab. 3] [arxiv:1901.09107 tab. 4] [arxiv:1901.09107 tab. 5]. The best test MRR, 53.41 with dialog history and video, is therefore a score on this candidate set, and the paper does not show how it would change with another set [arxiv:1901.09107 tab. 3]. The repository's default composition differs from the paper's, and the candidate files it would need are not included, so the exact test candidates cannot be rebuilt from it [Audio-Visual Scene-Aware Dialog](/papers/1901.09107). ## When to use which | Situation | Use | Why | |---|---|---| | Labelled positives, a margin-based metric loss, and a batch large enough to hold many candidates | Batch-hard mining inside the batch [arxiv:2011.04530 §3.2] | Needs no extra pass over the data. The batch grows only when few triplets remain active [arxiv:2011.04530 tab. 3]. | | Training items have coordinates, and early training needs candidates before the model is useful | GPS-neighbour batches for the early epochs [arxiv:2303.11851 §3.3] | Gives negatives without a trained model. Not available when there are no coordinates, as for University-1652 [arxiv:2303.11851 §3.3]. | | The model is trained enough that its similarities rank candidates usefully, and a full inference pass every few epochs is affordable | DSS, combined with GPS sampling [arxiv:2303.11851 §3.4] | On VIGOR SAME, R@1 is 77.86 with GPS plus DSS, against 65.23 with random sampling and 74.69 with GPS alone [arxiv:2303.11851 tab. 5]. | | Self-supervised training with no labelled masks, and a dense similarity map inside each frame | Soft thresholds on the same-frame map, with an ignored band [arxiv:2104.02691 §3.4] | Needs no labels, but the thresholds must be chosen. Without the band, a negative threshold of 0.6 scored below no negatives (cIoU 0.667 against 0.675) [arxiv:2104.02691 tab. 4]. | | A ranked test needs hard distractors, and there is no trained model to mine with | Distractors from question similarity, plus popular and random answers [arxiv:1901.09107 §4] | Needs only question features and an answer pool, with no training pass. The paper reports no comparison of distractor rules, so the gain over other distractor sets is unknown [arxiv:1901.09107 tab. 3]. | | Triplet loss with hard negatives is unstable in your setup | A loss that scores all in-batch negatives at once, such as symmetric InfoNCE [arxiv:2303.11851 §3.1] | In the paper's loss comparison on CVUSA, triplet loss reached R@1 0.00 and InfoNCE reached 98.68 [arxiv:2303.11851 supp. "Loss comparison"]. The triplet setup there is not stated, so this does not show that triplet loss fails in general. | ## Implementation sketch The first two pieces are batch-hard selection inside one batch, and DSS candidate selection for one epoch. The soft-threshold variant keeps the cosine map but replaces the mining step with per-location weights as described above; its card gives the loss in pseudocode [arxiv:2104.02691 §3.4]. The distractor variant builds a fixed candidate list per query, and its card gives that construction in pseudocode [arxiv:1901.09107 §4]. ```pseudocode # Batch-hard triplet selection inside one batch [arxiv:2011.04530 §3.2] # Input: emb (n, D), label (n,); a positive shares the anchor's label # Output: one hinge value per anchor that has a positive and a negative def batch_hard_hinge(emb, label, margin=0.2): # Pairwise Euclidean distances, shape (n, n) [arxiv:2011.04530 §3.2] dist = pairwise_l2(emb) same = label[:, None] == label[None, :] # Hardest positive: farthest item with the same label, not the anchor d_ap = max_where(dist, same & not_self) # Hardest negative: closest item with a different label d_an = min_where(dist, not same) # Hinge per anchor, margin m [arxiv:2011.04530 §3.2] return relu(d_ap - d_an + margin) ``` ```pseudocode # Dynamic Similarity Sampling for one epoch [arxiv:2303.11851 §3.4] # Input: train_emb (N, D); k = 64 kept per query, K = 128 in the pool # Output: chosen[q] = k training indices for each query q def dss_candidates(train_emb, k=64, K=128): # Full inference pass; recomputed only every e = 4 epochs [§3.4] sims = cosine_matrix(train_emb) for q in range(N): # Top K most similar items to q, most similar first [§3.4] pool = top_k_indices(sims[q], K) # First k/2 by similarity, then k/2 drawn at random from the rest [§3.4] chosen[q] = pool[: k // 2] + sample(pool[k // 2 :], k // 2) # Packing into batches and de-duplication within an epoch are not stated [§3.4] return chosen ``` ## Contradictions - **Triplet loss and collapse.** Sample4Geo states that hard negatives with triplet loss can make a model collapse [arxiv:2303.11851 §1] [arxiv:2303.11851 §3]. Its triplet run on CVUSA reached R@1 0.00, with margin and mining not stated [arxiv:2303.11851 supp. "Loss comparison"]. MinkLoc3D trains a batch-hard triplet loss and reports AR@1% 97.9 on Oxford [arxiv:2011.04530 §3.2] [arxiv:2011.04530 tab. 4]. The tasks, data and triplet setups differ, and neither paper runs the other's setup. - **Starting batch size.** MinkLoc3D's Table 3 gives an initial batch size of 32 for baseline training, while the text of §3.2 says "say 16" [arxiv:2011.04530 tab. 3] [arxiv:2011.04530 §3.2]. ## Papers - [Audio-Visual Scene-Aware Dialog](/papers/1901.09107): question-similarity distractors, with popular and random answers, for ranking 100 candidate answers in a dialog benchmark. - [MinkLoc3D: Point Cloud Based Large-Scale Place Recognition](/papers/2011.04530): batch-hard triplet mining, with a batch size that grows during training, for LiDAR place recognition. - [Localizing Visual Sounds the Hard Way](/papers/2104.02691): same-frame background negatives chosen by soft thresholds on a cosine map, with an ignored band, for self-supervised audio-visual localization. - [Sample4Geo: Hard Negative Sampling For Cross-View Geo-Localisation](/papers/2303.11851): GPS-neighbour batches, then Dynamic Similarity Sampling, trained with symmetric InfoNCE.