Weakly-supervised learning

Overview

Here "weakly supervised" means the training signal is sparse, indirect or noisy, rather than clean and complete. The four cards on this page use the term for different signals:

• Dense registration. The ideal target, voxel-to-voxel correspondence, is not available for patient data. Anatomical label pairs are easier to collect, yet they are sparse and may be missing on some image pairs 1807.03361 §1 1807.03361 §3.1.
• Text embeddings. Text pairs scraped from the web, such as a post and its comment, are plentiful but noisy. A filter keeps the more reliable ones 2212.03533 §1 2212.03533 §3.
• Slide classification. Only the whole slide has a label, not its patches. Each bag is one slide, and its instance labels are unknown 2106.00908 §3.1 2106.00908 eq. 1.
• Taxonomy. A 3D detection survey places weak supervision among its label-efficient settings. It is a survey, so it describes methods rather than proposing one 2206.09474 §8 2206.09474 §8.2.

The shared idea is that the loss rewards agreement between paired signals, not a fully labelled target. The registration loss scores how well corresponding labels overlap after the predicted warp. It never uses voxel-level correspondences 1807.03361 §2.1. The embedding loss pulls each pair together and pushes apart the other texts in the batch 2212.03533 §4.1. In the bag-level variant, one label per bag is the only signal, and the model must decide which patches drive it 2106.00908 §3.1.

Variants

Label-overlap loss on warped labels (dense registration)

The network predicts a DDF u from the two images. The moving image's labels are warped by u and compared with the fixed image's labels 1807.03361 §2.1. Training maximises the average label similarity over image pairs and label pairs, minus a weighted deformation penalty 1807.03361 eq. 1 1807.03361 eq. 2.

Each minibatch draws K image pairs, then one label pair from each. The paper shows the expected minibatch gradient equals the full gradient for this two-stage sampler. It does not analyse other samplers 1807.03361 eq. 3 1807.03361 §2.1.

The label similarity is a soft Dice blurred at several scales. Each label is Gaussian-filtered at σ in {0, 1, 2, 4, 8, 16, 32}, and the Dice is averaged over these seven scales. The σ = 0 term keeps the unfiltered binary label 1807.03361 eq. 4 1807.03361 eq. 5 1807.03361 §2.2. The paper's stated reason is that binary overlap ignores distance when objects do not overlap and saturates, so the blurred score grades near misses 1807.03361 §2.2.

The regulariser is the bending energy, with weight α = 0.5 in the baseline 1807.03361 §3.2.1. The network is a 3D CNN with four down- and four up-sampling blocks, and it sums displacement outputs from several resolution levels 1807.03361 §2.3. Card: Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration.

Alternatives to the registration loss (ablated)

The paper also defines a multiscale cross-entropy label score [arxiv:1807.03361 eq. 6]. Replacing the Dice with it made the median TRE worse, p < 0.001 1807.03361 §4.2 1807.03361 tab. 1. Replacing bending energy with an L2 norm of displacement gradients made TRE worse, p = 0.049, and Dice worse, p < 0.001 1807.03361 §4.2.

Mined text pairs with a consistency filter (text embeddings)

The weak signal here is text pairs from forums, encyclopedias, papers and web pages, for example a question and its upvoted answer, or a title and its abstract. The authors start from about 1.3 billion candidate pairs, built with heuristic filters, and keep about 270 million after a consistency filter 2212.03533 §3 2212.03533 tab. 9.

The consistency filter trains a model on the candidates. It ranks each pair's passage against a pool of one million random passages, and keeps the pair only if its passage ranks in the top two 2212.03533 §3. The top-two cut-off came from manual inspection, and iterated filtering is left to future work 2212.03533 §3. On a six-dataset BEIR subset with base models, the filter raised the average nDCG@10 from 34.9 to 40.7 on 1M pairs, and from 50.0 to 51.6 on all pairs 2212.03533 tab. 7. The gain is much smaller at full scale.

Pre-training is contrastive, with one shared encoder, cosine similarity at temperature 0.01, and the other passages in the batch as negatives. The batch is 32,768 2212.03533 §4.1 2212.03533 tab. 11. On the same subset, in-batch negatives beat pre-batch negatives and MoCo [arxiv:2212.03533 tab. 8]. An optional second stage fine-tunes on labelled NLI, MS-MARCO and NQ data, with hard negatives and a distillation term 2212.03533 §4.2. Card: Text Embeddings by Weakly-Supervised Contrastive Pre-training.

One label per bag, correlated patches (whole-slide classification)

A whole-slide image is a bag of patches, and the bag has one label. A binary bag is positive when at least one patch is positive, and the patch labels are unknown 2106.00908 §3.1 2106.00908 eq. 1. The card records about 8,800 patches per bag on average for CAMELYON16 at x20 2106.00908 §4.

The model is TransMIL. Instead of scoring each patch alone, it runs two Transformer layers over all patch features. Self-attention uses a Nystrom approximation with landmark tokens 2106.00908 §3.3 [arxiv:2106.00908 eq. 9]. A convolutional position encoder (PPEG) sits between the two layers and reshapes the patch sequence into a square grid in input order [arxiv:2106.00908 alg. 2] 2106.00908 alg. 3. The bag label is read from a class token 2106.00908 alg. 2. The bag-level variant is trained with cross-entropy on the bag label 2106.00908 §4.

Author-reported AUC: 0.9309 on CAMELYON16, 0.9603 on TCGA-NSCLC and 0.9882 on TCGA-RCC, each the highest in the paper's table 2106.00908 tab. 1. Removing the position encoder drops CAMELYON16 AUC to 0.8416, and a sinusoidal encoding gives 0.8941 2106.00908 tab. 2. Training used Lookahead, a learning rate of 2e-4, weight decay 1e-5 and batch size 1 on one RTX 3090 2106.00908 §4. Card: TransMIL. The official code is GPLv3 and non-commercial, so read the card's Paper vs code section before reusing it.

Label-efficient settings in a 3D detection survey

The survey does not propose a method. It groups label-efficient 3D detection into domain adaptation, weak, semi-supervised and self-supervised learning 2206.09474 §8 2206.09474 §8.2. This wiki does not yet hold the methods in those sections, so the survey is cited here for the taxonomy only. Card: 3D Object Detection for Autonomous Driving.

When to use which

Data limits, registration: the results come from one trial, 108 image pairs from 76 patients, under 12-fold patient-level cross-validation. No independent test set was available, so generalisation to other centres is untested 1807.03361 §3.1 1807.03361 §3.3 1807.03361 §5.

Data limits, embeddings: the benchmark numbers are author-reported. The pipeline that builds CCPairs is described only in outline, and the exact contamination check is an exact string match 2212.03533 §3 2212.03533 app. A.

Data limits, slide classification: CAMELYON16 is scored on one official split. The TCGA numbers are 4-fold averages, and no standard deviations are reported 2106.00908 §4.

Data limits, survey: its scores are copied from original papers. The tables mix KITTI test, nuScenes test and Waymo validation results, and inference times were measured on different hardware 2206.09474 tab. 16.

Implementation sketch

Own-words pseudocode for the two variants. Where the paper leaves a choice open, the comment says so.

Registration variant, one training step:

# One training step  [arxiv:1807.03361 §2.1, eq. 1-3]
# Input:  image pairs, label pairs per image pair, K, alpha
# Output: updated network parameters theta
def train_step(theta, image_pairs, labels_per_pair, K, alpha):
    # Stage 1: draw K image pairs uniformly  [arxiv:1807.03361 §2.1]
    # Stage 2: draw one label pair from each chosen image pair  [§2.1]
    pairs, label_pairs = sample_minibatch(image_pairs, labels_per_pair, K)
    loss = 0
    for (x_a, x_b), (l_a, l_b) in zip(pairs, label_pairs):
        # predict a dense displacement field u from the image pair  [§2.3]
        u = network(x_a, x_b; theta)
        # warp the moving label with u  [§2.1]
        y_a = warp(l_a, u)
        # reward label overlap, so subtract it from the loss  [§2.1, eq. 2]
        loss -= multiscale_soft_dice(l_b, y_a)  # [§2.2, eq. 4-5]
        # penalise non-smooth displacement  [§2.1, eq. 2]
        loss += alpha * bending_energy(u)  # formula not stated in paper
    # minibatch averaging is not stated; mean over K assumed here
    theta = adam_step(theta, grad(loss / K))  # Adam [§3.2.1]
    return theta

# Soft Dice averaged over blur scales  [arxiv:1807.03361 §2.2, eq. 5]
def multiscale_soft_dice(a, b):
    scores = []
    for sigma in [0, 1, 2, 4, 8, 16, 32]:  # sigma units not stated  [§2.2]
        # sigma = 0 keeps the unfiltered binary label  [§2.2]
        fa, fb = gaussian_filter(a, sigma), gaussian_filter(b, sigma)
        scores.append(soft_dice(fa, fb))  # [§2.2, eq. 4]
    return sum(scores) / len(scores)

Embedding variant, the consistency filter:

# Consistency filter for mined text pairs  [arxiv:2212.03533 §3]
# Input:  candidate (query, passage) pairs, a pool of 1M random passages, k
# Output: kept pairs
def consistency_filter(pairs, pool, k=2):
    # Train a ranker on all candidates first; its model is not named  [§3]
    ranker = train_ranker(pairs)
    kept = []
    for query, passage in pairs:
        # Rank this pair's passage against the pool  [§3]
        # whether the pool includes the pair's own passage is not stated
        rank = rank_of(ranker, query, passage, pool)
        # Keep the pair only if its passage is in the top k  [§3]
        if rank <= k:
            kept.append((query, passage))
    return kept

Where variants differ: within the registration variant, the regulariser (bending energy or L2 gradient norm) and the label score (soft Dice or cross-entropy) are both ablated 1807.03361 §4.2. The cross-entropy sign is not settled by the paper, so check that the loss decreases [arxiv:1807.03361 eq. 6] 1807.03361 §2.2. The bag-level variant has no sketch here. It trains with cross-entropy on one label per bag 2106.00908 §4, and its model is the TPT module 2106.00908 alg. 2, which the card gives in full.

Contradictions

No paper here reports a result that contradicts another. The four cards use "weakly supervised" for different signals, so their numbers are not comparable: registration overlap, retrieval and benchmark scores, slide-level AUC, and detection AP 1807.03361 §1 2212.03533 §5.2 2106.00908 §4 2206.09474 tab. 16.

Within the registration card, §2.3 describes the displacement skip layers as randomly initialised, but §3.2.1 says the final displacement layers were zero-initialised for all reported results. The reported numbers used zero-initialisation 1807.03361 §2.3 1807.03361 §3.2.1.

Within the survey card, the text and tables disagree on several values. For example, §10.1.1 gives PV-RCNN++ an easy AP3D of 90.90, while the table gives PV-RCNN++ 90.14 and Voxel R-CNN 90.90 2206.09474 §10.1.1 2206.09474 tab. 16. The card says to use the table rows.

Papers

• Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration: label-overlap loss, multiscale soft Dice and a 3D CNN for prostate MR-to-ultrasound dense displacement fields.
• Text Embeddings by Weakly-Supervised Contrastive Pre-training: E5 text embeddings from about 270M consistency-filtered web pairs, with contrastive pre-training and optional fine-tuning.
• TransMIL: two-layer Transformer over slide patches, with Nystrom attention and PPEG position encoding, trained from one label per slide.
• 3D Object Detection for Autonomous Driving: A Comprehensive Survey: survey whose taxonomy places weak supervision among the label-efficient 3D detection settings.