---
kind: concept
slug: weakly-supervised-learning
title: Weakly-supervised learning
summary: >-
  Learn from sparse, indirect or noisy supervision instead of clean full labels. Four cards: label-overlap loss for MR-to-ultrasound registration, mined web text pairs for embeddings, one label per slide for pathology, and a survey's taxonomy for 3D detection. Trade-off: cheap labels, but thin evidence, mostly author-reported.
tags: [weak-supervision, image-registration, medical-imaging, multimodal, deformable-registration, text-embedding, data-curation, multiple-instance-learning, pathology, object-detection]
papers: ["1807.03361", "2212.03533", "2106.00908", "2206.09474"]
related: [image-registration, convolutional-neural-network, semi-supervised-learning, noisy-labels, contrastive-learning, dataset-curation, text-embedding, knowledge-distillation, multiple-instance-learning, object-detection]
status: draft
updated_at: 2026-11-01
---

## Overview

Here "weakly supervised" means the training signal is sparse, indirect or
noisy, rather than clean and complete. The four cards on this page use the term
for different signals:

- **Dense registration.** The ideal target, voxel-to-voxel correspondence, is
  not available for patient data. Anatomical label pairs are easier to collect,
  yet they are sparse and may be missing on some image pairs
  [arxiv:1807.03361 §1] [arxiv:1807.03361 §3.1].
- **Text embeddings.** Text pairs scraped from the web, such as a post and its
  comment, are plentiful but noisy. A filter keeps the more reliable ones
  [arxiv:2212.03533 §1] [arxiv:2212.03533 §3].
- **Slide classification.** Only the whole slide has a label, not its patches.
  Each bag is one slide, and its instance labels are unknown
  [arxiv:2106.00908 §3.1] [arxiv:2106.00908 eq. 1].
- **Taxonomy.** A 3D detection survey places weak supervision among its
  label-efficient settings. It is a survey, so it describes methods rather than
  proposing one [arxiv:2206.09474 §8] [arxiv:2206.09474 §8.2].

The shared idea is that the loss rewards agreement between paired signals, not
a fully labelled target. The registration loss scores how well corresponding
labels overlap after the predicted warp. It never uses voxel-level
correspondences [arxiv:1807.03361 §2.1]. The embedding loss pulls each pair
together and pushes apart the other texts in the batch [arxiv:2212.03533 §4.1].
In the bag-level variant, one label per bag is the only signal, and the model
must decide which patches drive it [arxiv:2106.00908 §3.1].

## Variants

### Label-overlap loss on warped labels (dense registration)

The network predicts a DDF u from the two images. The moving image's labels are
warped by u and compared with the fixed image's labels [arxiv:1807.03361 §2.1].
Training maximises the average label similarity over image pairs and label
pairs, minus a weighted deformation penalty [arxiv:1807.03361 eq. 1]
[arxiv:1807.03361 eq. 2].

Each minibatch draws K image pairs, then one label pair from each. The paper
shows the expected minibatch gradient equals the full gradient for this
two-stage sampler. It does not analyse other samplers [arxiv:1807.03361 eq. 3]
[arxiv:1807.03361 §2.1].

The label similarity is a soft Dice blurred at several scales. Each label is
Gaussian-filtered at σ in {0, 1, 2, 4, 8, 16, 32}, and the Dice is averaged
over these seven scales. The σ = 0 term keeps the unfiltered binary label
[arxiv:1807.03361 eq. 4] [arxiv:1807.03361 eq. 5] [arxiv:1807.03361 §2.2]. The
paper's stated reason is that binary overlap ignores distance when objects do
not overlap and saturates, so the blurred score grades near misses
[arxiv:1807.03361 §2.2].

The regulariser is the bending energy, with weight α = 0.5 in the baseline
[arxiv:1807.03361 §3.2.1]. The network is a 3D CNN with four down- and four
up-sampling blocks, and it sums displacement outputs from several resolution
levels [arxiv:1807.03361 §2.3]. [Card: Weakly-Supervised Convolutional Neural
Networks for Multimodal Image Registration](/papers/1807.03361).

### Alternatives to the registration loss (ablated)

The paper also defines a multiscale cross-entropy label score [arxiv:1807.03361
eq. 6]. Replacing the Dice with it made the median TRE worse, p < 0.001
[arxiv:1807.03361 §4.2] [arxiv:1807.03361 tab. 1]. Replacing bending energy
with an L2 norm of displacement gradients made TRE worse, p = 0.049, and Dice
worse, p < 0.001 [arxiv:1807.03361 §4.2].

### Mined text pairs with a consistency filter (text embeddings)

The weak signal here is text pairs from forums, encyclopedias, papers and web
pages, for example a question and its upvoted answer, or a title and its
abstract. The authors start from about 1.3 billion candidate pairs, built with
heuristic filters, and keep about 270 million after a consistency filter
[arxiv:2212.03533 §3] [arxiv:2212.03533 tab. 9].

The consistency filter trains a model on the candidates. It ranks each pair's
passage against a pool of one million random passages, and keeps the pair only
if its passage ranks in the top two [arxiv:2212.03533 §3]. The top-two cut-off
came from manual inspection, and iterated filtering is left to future work
[arxiv:2212.03533 §3]. On a six-dataset BEIR subset with base models, the filter
raised the average nDCG@10 from 34.9 to 40.7 on 1M pairs, and from 50.0 to 51.6
on all pairs [arxiv:2212.03533 tab. 7]. The gain is much smaller at full scale.

Pre-training is contrastive, with one shared encoder, cosine similarity at
temperature 0.01, and the other passages in the batch as negatives. The batch
is 32,768 [arxiv:2212.03533 §4.1] [arxiv:2212.03533 tab. 11]. On the same
subset, in-batch negatives beat pre-batch negatives and MoCo [arxiv:2212.03533
tab. 8]. An optional second stage fine-tunes on labelled NLI, MS-MARCO and NQ
data, with hard negatives and a distillation term [arxiv:2212.03533 §4.2].
[Card: Text Embeddings by Weakly-Supervised Contrastive
Pre-training](/papers/2212.03533).

### One label per bag, correlated patches (whole-slide classification)

A whole-slide image is a bag of patches, and the bag has one label. A binary
bag is positive when at least one patch is positive, and the patch labels are
unknown [arxiv:2106.00908 §3.1] [arxiv:2106.00908 eq. 1]. The card records
about 8,800 patches per bag on average for CAMELYON16 at x20
[arxiv:2106.00908 §4].

The model is TransMIL. Instead of scoring each patch alone, it runs two
Transformer layers over all patch features. Self-attention uses a Nystrom
approximation with landmark tokens [arxiv:2106.00908 §3.3] [arxiv:2106.00908
eq. 9]. A convolutional position encoder (PPEG) sits between the two layers and
reshapes the patch sequence into a square grid in input order [arxiv:2106.00908
alg. 2] [arxiv:2106.00908 alg. 3]. The bag label is read from a class token
[arxiv:2106.00908 alg. 2]. The bag-level variant is trained with cross-entropy
on the bag label [arxiv:2106.00908 §4].

Author-reported AUC: 0.9309 on CAMELYON16, 0.9603 on TCGA-NSCLC and 0.9882 on
TCGA-RCC, each the highest in the paper's table [arxiv:2106.00908 tab. 1].
Removing the position encoder drops CAMELYON16 AUC to 0.8416, and a sinusoidal
encoding gives 0.8941 [arxiv:2106.00908 tab. 2]. Training used Lookahead, a
learning rate of 2e-4, weight decay 1e-5 and batch size 1 on one RTX 3090
[arxiv:2106.00908 §4]. [Card: TransMIL](/papers/2106.00908). The official code
is GPLv3 and non-commercial, so read the card's Paper vs code section before
reusing it.

### Label-efficient settings in a 3D detection survey

The survey does not propose a method. It groups label-efficient 3D detection
into domain adaptation, weak, semi-supervised and self-supervised learning
[arxiv:2206.09474 §8] [arxiv:2206.09474 §8.2]. This wiki does not yet hold the
methods in those sections, so the survey is cited here for the taxonomy only.
[Card: 3D Object Detection for Autonomous Driving](/papers/2206.09474).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Dense voxel correspondences exist for training | Not the registration variant | The registration paper's premise is that dense ground truth is unavailable for patient data [arxiv:1807.03361 §1] |
| Sparse, corresponding labels on some training pairs | Label-overlap loss | Labels are used only in training, and may be missing on some pairs [arxiv:1807.03361 §1] [arxiv:1807.03361 §2.1] |
| Need warps at inference without labels or manual initialisation | Trained network | Inference takes only the image pair and runs at about 4 registrations per second on the authors' GPU [arxiv:1807.03361 §4.1] |
| Limited GPU memory | Baseline network | Baseline trains on a 12 GB card; the Composite-Net needed 24 GB for the same minibatch [arxiv:1807.03361 §3.2] [arxiv:1807.03361 §3.2.2] |
| Need folding-free warps | Keep α high | With α = 0.5 the baseline gave no negative Jacobian determinants; the illustrative α = 0.01 network did [arxiv:1807.03361 §4.3]. The DDF is not topology-preserving by construction [arxiv:1807.03361 §5] |
| Choosing the registration label score | Multiscale soft Dice | Multiscale cross-entropy gave worse median TRE, p < 0.001 [arxiv:1807.03361 §4.2] |
| Choosing between registration network designs | Baseline over Global-Net or Composite-Net | Baseline median TRE 3.6 mm and Dice 0.87, against 5.8 mm and 0.77 for Global-Net and 4.7 mm and 0.82 for Composite-Net [arxiv:1807.03361 tab. 1] |
| Registration with no labels at all | Not covered by this approach | The paper's own intensity-based comparison is weak on this data. Its pairwise FFD runs had median TRE 20.6 to 25.6 mm after centroid initialisation, and the paper calls these simplified runs [arxiv:1807.03361 §4.4] [arxiv:1807.03361 tab. 2] |
| Only unlabelled text, no pairs | Mined pairs with consistency filter, then contrastive pre-training | Author-reported: E5-PT-base scores 42.9 nDCG@10 on BEIR, against 41.7 for BM25 [arxiv:2212.03533 tab. 1] [arxiv:2212.03533 §5.3] |
| Labelled pairs exist for the target task | Add the fine-tuning stage | Author-reported MTEB average for the base model rises from 55.6 without fine-tuning to 60.4 with the full mix [arxiv:2212.03533 tab. 6] |
| Compute or memory is tight | Smaller encoder, or a smaller batch with a quality drop | E5-small has 33M parameters against 110M for base [arxiv:2212.03533 tab. 10]. Batch 1k averages 45.8 against 51.6 for 32k on the six-dataset subset [arxiv:2212.03533 tab. 5]. Pre-training took 1 to 2 days on 16 to 64 V100 GPUs [arxiv:2212.03533 tab. 11] |
| One label per bag, instance labels unknown, bags of thousands of patches | Bag-level MIL with correlated attention (TransMIL) | Bag-level labels only [arxiv:2106.00908 §3.1] [arxiv:2106.00908 eq. 1]. The paper says the Nystrom approximation reduces attention cost from O(n²) to O(n) [arxiv:2106.00908 §3.3] |
| Patch context matters for the bag label | Include the position encoder | Position encoding raised CAMELYON16 AUC from 0.8416 with none to 0.9309 with PPEG [arxiv:2106.00908 tab. 2] |
| Faster convergence is needed | Check it yourself | Authors claim 2 to 3 times fewer epochs than ABMIL, DSMIL and CLAM, with no epoch counts given [arxiv:2106.00908 §4.4] |
| Need a taxonomy of weak settings for 3D detection | Survey only | It names the settings, but this wiki does not yet hold the methods [arxiv:2206.09474 §8] [arxiv:2206.09474 §8.2] |

Data limits, registration: the results come from one trial, 108 image pairs
from 76 patients, under 12-fold patient-level cross-validation. No independent
test set was available, so generalisation to other centres is untested
[arxiv:1807.03361 §3.1] [arxiv:1807.03361 §3.3] [arxiv:1807.03361 §5].

Data limits, embeddings: the benchmark numbers are author-reported. The
pipeline that builds CCPairs is described only in outline, and the exact
contamination check is an exact string match [arxiv:2212.03533 §3]
[arxiv:2212.03533 app. A].

Data limits, slide classification: CAMELYON16 is scored on one official split.
The TCGA numbers are 4-fold averages, and no standard deviations are reported
[arxiv:2106.00908 §4].

Data limits, survey: its scores are copied from original papers. The tables mix
KITTI test, nuScenes test and Waymo validation results, and inference times
were measured on different hardware [arxiv:2206.09474 tab. 16].

## Implementation sketch

Own-words pseudocode for the two variants. Where the paper leaves a choice
open, the comment says so.

Registration variant, one training step:

```pseudocode
# One training step  [arxiv:1807.03361 §2.1, eq. 1-3]
# Input:  image pairs, label pairs per image pair, K, alpha
# Output: updated network parameters theta
def train_step(theta, image_pairs, labels_per_pair, K, alpha):
    # Stage 1: draw K image pairs uniformly  [arxiv:1807.03361 §2.1]
    # Stage 2: draw one label pair from each chosen image pair  [§2.1]
    pairs, label_pairs = sample_minibatch(image_pairs, labels_per_pair, K)
    loss = 0
    for (x_a, x_b), (l_a, l_b) in zip(pairs, label_pairs):
        # predict a dense displacement field u from the image pair  [§2.3]
        u = network(x_a, x_b; theta)
        # warp the moving label with u  [§2.1]
        y_a = warp(l_a, u)
        # reward label overlap, so subtract it from the loss  [§2.1, eq. 2]
        loss -= multiscale_soft_dice(l_b, y_a)  # [§2.2, eq. 4-5]
        # penalise non-smooth displacement  [§2.1, eq. 2]
        loss += alpha * bending_energy(u)  # formula not stated in paper
    # minibatch averaging is not stated; mean over K assumed here
    theta = adam_step(theta, grad(loss / K))  # Adam [§3.2.1]
    return theta

# Soft Dice averaged over blur scales  [arxiv:1807.03361 §2.2, eq. 5]
def multiscale_soft_dice(a, b):
    scores = []
    for sigma in [0, 1, 2, 4, 8, 16, 32]:  # sigma units not stated  [§2.2]
        # sigma = 0 keeps the unfiltered binary label  [§2.2]
        fa, fb = gaussian_filter(a, sigma), gaussian_filter(b, sigma)
        scores.append(soft_dice(fa, fb))  # [§2.2, eq. 4]
    return sum(scores) / len(scores)
```

Embedding variant, the consistency filter:

```pseudocode
# Consistency filter for mined text pairs  [arxiv:2212.03533 §3]
# Input:  candidate (query, passage) pairs, a pool of 1M random passages, k
# Output: kept pairs
def consistency_filter(pairs, pool, k=2):
    # Train a ranker on all candidates first; its model is not named  [§3]
    ranker = train_ranker(pairs)
    kept = []
    for query, passage in pairs:
        # Rank this pair's passage against the pool  [§3]
        # whether the pool includes the pair's own passage is not stated
        rank = rank_of(ranker, query, passage, pool)
        # Keep the pair only if its passage is in the top k  [§3]
        if rank <= k:
            kept.append((query, passage))
    return kept
```

Where variants differ: within the registration variant, the regulariser
(bending energy or L2 gradient norm) and the label score (soft Dice or
cross-entropy) are both ablated [arxiv:1807.03361 §4.2]. The cross-entropy sign
is not settled by the paper, so check that the loss decreases [arxiv:1807.03361
eq. 6] [arxiv:1807.03361 §2.2]. The bag-level variant has no sketch here. It
trains with cross-entropy on one label per bag [arxiv:2106.00908 §4], and its
model is the TPT module [arxiv:2106.00908 alg. 2], which the card gives in full.

## Contradictions

No paper here reports a result that contradicts another. The four cards use
"weakly supervised" for different signals, so their numbers are not comparable:
registration overlap, retrieval and benchmark scores, slide-level AUC, and
detection AP [arxiv:1807.03361 §1] [arxiv:2212.03533 §5.2]
[arxiv:2106.00908 §4] [arxiv:2206.09474 tab. 16].

Within the registration card, §2.3 describes the displacement skip layers as
randomly initialised, but §3.2.1 says the final displacement layers were
zero-initialised for all reported results. The reported numbers used
zero-initialisation [arxiv:1807.03361 §2.3] [arxiv:1807.03361 §3.2.1].

Within the survey card, the text and tables disagree on several values. For
example, §10.1.1 gives PV-RCNN++ an easy AP3D of 90.90, while the table gives
PV-RCNN++ 90.14 and Voxel R-CNN 90.90 [arxiv:2206.09474 §10.1.1]
[arxiv:2206.09474 tab. 16]. The card says to use the table rows.

## Papers

- [Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration](/papers/1807.03361): label-overlap loss, multiscale soft Dice and a 3D CNN for prostate MR-to-ultrasound dense displacement fields.
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](/papers/2212.03533): E5 text embeddings from about 270M consistency-filtered web pairs, with contrastive pre-training and optional fine-tuning.
- [TransMIL](/papers/2106.00908): two-layer Transformer over slide patches, with Nystrom attention and PPEG position encoding, trained from one label per slide.
- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](/papers/2206.09474): survey whose taxonomy places weak supervision among the label-efficient 3D detection settings.