Multiple instance learning

Overview

In multiple instance learning (MIL), the label belongs to a bag, a set of instances. Here a bag is one whole-slide image and its instances are patches. A binary bag is 0 only if every instance is 0, and 1 otherwise. Instance labels are not known 2106.00908 §3.1 2106.00908 eq. 1. The shared rule is that the model must give one bag-level prediction from instance features. The variants differ in how they combine instances before that prediction.

A 2017 review of histopathology methods lists MIL as one answer to scarce labels. It uses the same bag rule: a slide is the bag, and a cancer slide contains at least one cancer patch 1709.00786 §4.2.2. The review reports no experiments, so this page takes from it the framing and terminology only, and no result.

Many MIL methods treat the instances in a bag as independent and identically distributed. TransMIL argues that this often fails in pathology, because a pathologist uses both the context around a region and the relations between regions 2106.00908 §1. The size of the input is part of the problem: CAMELYON16 has about 8,800 patches per bag on average at x20 magnification 2106.00908 §1 2106.00908 §4.

Variants

Independent instances with pooling

Each patch is scored on its own, and a fixed rule combines the patch outputs into one bag score. The paper uses mean-pooling and max-pooling as baselines 2106.00908 tab. 1. On CAMELYON16 their AUCs are 0.4647 and 0.8569. On TCGA-RCC they are 0.9786 and 0.9879 2106.00908 tab. 1. The choice of rule matters on CAMELYON16 and barely on TCGA-RCC, so a pooling rule should be checked on the task at hand. The baselines are reported in TransMIL.

Correlated instances with a Transformer (TransMIL)

Instances in one bag interact through self-attention. A two-layer Transformer runs over the patch features with a class token, and a convolutional position encoder (PPEG) sits between the two layers 2106.00908 §3.3 2106.00908 alg. 2. Attention is approximated with the Nystrom method, which the authors say cuts the cost from quadratic to linear in the number of instances 2106.00908 §3.3. In the paper's ablation on CAMELYON16, removing position encoding gives AUC 0.8416 and PPEG gives 0.9309, at 2.625M and 2.669M parameters 2106.00908 tab. 2. The full method is in TransMIL.

The same table names other MIL methods, ABMIL, DSMIL, CLAM-SB, CLAM-MB, MIL-RNN and PT-MTA 2106.00908 tab. 1. No cards exist for them yet, so this page does not describe how they work.

When to use which

The author-reported gains come from single splits and report no standard deviations 2106.00908 §4. Treat the margins as indicative only.

Implementation sketch

Pooling and correlated variants share the instance encoder and the bag head. Only the aggregation step differs.

# Shared MIL core: instance features -> one bag prediction  [alg. 1]
# Input:  bag of n patch features, shape (n, d); one label per bag  [§3.1]
# Output: bag-level probability

def encode(bag):
    # Project each patch to a shared width, e.g. 1024 -> 512  [§4]
    return linear(bag)                        # (n, 512)

def pooled_mil(bag):
    # Independent instances: one fixed rule over patch scores  [tab. 1]
    h = encode(bag)
    pooled = mean_or_max(h, axis=0)           # (512,)
    return softmax(head(pooled))

def correlated_mil(bag):
    # Class token first, then two attention layers with PPEG between  [alg. 2]
    h = concat(class_token, encode(bag))      # (n+1, 512)
    h = h + attention(layer_norm(h))          # correlate instances
    h = ppeg(h)                               # position encoding  [alg. 3]
    h = h + attention(layer_norm(h))          # second correlation step
    return softmax(head(layer_norm(h)[0]))    # read the class token

Train with cross-entropy on the bag label §4. The paper's training schedule and the optimizer details are in the card's Hyperparameters section, where values the paper leaves unstated are marked.

Contradictions

None known. The review 1709.00786 §4.2.2 gives no numbers, so it cannot conflict with the TransMIL results. Its bag rule matches the one in 2106.00908 eq. 1.

Papers

• Machine learning methods for histopathological image analysis (Komura and Ishikawa, 2017): a mini-review that names MIL as one way to handle scarce slide-level labels. No experiments.
• TransMIL (Shao et al., 2021): a two-layer Transformer over all patch features of a slide, with Nystrom attention and a convolutional position encoder.