Interactive segmentation

Overview

Interactive segmentation takes a few user prompts, usually clicks inside the target or a box around it, and returns a mask. More prompts can refine the mask. The two SAM-style papers below use a Segment Anything-style promptable model on medical images 2304.10517 §1 2310.15161 §1. The third, SegDiff, is a diffusion model that takes no clicks. The paper still calls its per-object setting interactive, because it aims to speed up object annotation 2112.00390 §5, Datasets.

The invariant the three share is that the mask is scored by overlap with a ground-truth mask: IoU 2304.10517 §2.2.3, Dice 2310.15161 tab. 3 or mIoU 2112.00390 tab. 1. In the two SAM-style papers, the prompts are simulated from the ground truth. The first prompt is an interior point of the target, and later prompts are placed where the prediction and the ground truth disagree 2304.10517 alg. 1 2310.15161 §4.3. SegDiff's per-object crops are also taken from ground-truth boxes [arxiv:2112.00390 §5, Datasets]. A real user prompts differently, so these numbers are upper-bound-style estimates for a product.

Variants

Zero-shot 2D SAM with prompt modes

The released 2D model is used without training, one image or slice at a time. The paper compares five non-iterative prompt modes: one point in the largest part, one point per part, one box around the largest part, one box per part, and one box around the whole object 2304.10517 §2.2.1. Across 28 segmentation tasks, one box per part has the best reported mean IoU, 0.6542 2304.10517 §3.1. Single-prompt IoU ranges from 0.1136 to 0.9118 across tasks, so the outcome depends strongly on the target 2304.10517 §3.1. Card: Segment Anything Model for Medical Image Analysis.

With one point, the same paper reports a mean IoU of 0.4595 for SAM, against 0.2240 for the best of three interactive baselines (RITM, SimpleClick, FocalClick) 2304.10517 §3.2.

Iterative point refinement with SAM

The same paper also tests iterative points. Each new point goes into the largest region of error between the current prediction and the ground truth 2304.10517 alg. 1. The authors report that SAM gains little after two or three points, and that two of the baselines surpass it at five or more points 2304.10517 §3.3.

Promptable 3D model for volumes

SAM-Med3D is trained from scratch for volumetric data. It has a 3D image encoder, a 3D point-prompt encoder and a 3D mask decoder 2310.15161 §4.1. It is trained on SA-Med3D-140K, which the authors describe as 22K volumes with 143K 3D masks 2310.15161 §3.1. The authors report a Dice of 76.27% overall with one 3D point, against 36.83% for SAM-Med2D with one point per slice. With ten points, Dice reaches 80.71% 2310.15161 tab. 3. Card: SAM-Med3D.

The paper's own pre-experiment compares three ways of adapting SAM to 3D. A 3D adapter on frozen SAM scores 43.84 overall Dice, fine-tuning with SAM weights 43.78, and training from scratch 45.21. On unseen targets, fine-tuning with SAM weights scores lowest at 18.43 2310.15161 tab. 2. The pre-experiment used AMOS training data and is smaller than the main study.

Diffusion segmenter on per-object crops, no clicks

SegDiff adapts a DDPM to binary masks. The noise predictor adds features of the input image to the features of the noisy mask, and the whole model is trained end-to-end without a pretrained backbone 2112.00390 §4. At test time, several independent reverse chains are averaged and thresholded 2112.00390 §4.1.

The setting is per object. Each Cityscapes sample is one object's crop, taken from ground-truth boxes, and the model returns that object's mask 2112.00390 §5, Datasets. The model takes no clicks and refines nothing iteratively, so it does not fit the prompt loop in the next section. Card: SegDiff.

The authors report 76.44 mIoU on the Cityscapes expansion setting, against 73.70 for the best listed baseline, and 76.19 on tight boxes 2112.00390 tab. 1. The same paper reports Vaihingen and MoNuSeg results in 2112.00390 tab. 2 and 2112.00390 tab. 3. The baselines mostly use ImageNet, PASCAL or COCO pretraining, and this model starts from random weights 2112.00390 §5. Compare architectures with that in mind. Inference time grows linearly with the number of chains, and the paper reports it as slower than Segformer and STDC [arxiv:2112.00390 §5, Results].

When to use which

Implementation sketch

The loop is the same in both SAM-style papers: encode the image once, then add a prompt and decode, repeat. The papers differ in how the next prompt is placed, so the choice is marked in the code. SegDiff takes no clicks, so it uses a sampling loop of its own, not this one.

# Interactive loop, shared core  [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]
# Input:  image, gt_mask (binary), num_clicks
# Output: binary mask

def interactive_segment(model, image, gt_mask, num_clicks):
    # First prompt: an interior point of the target
    # [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]
    points = [interior_point(gt_mask)]
    labels = [1]
    # Encode the image once, then decode for each prompt set
    embedding = model.encode(image)
    mask = model.decode(embedding, points, labels)
    for step in range(num_clicks - 1):
        # Error region: where the prediction and ground truth disagree
        # [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]
        error = xor(gt_mask, mask)
        # Next prompt: 2304.10517 uses the interior point of the error
        # region; 2310.15161 uses a random error voxel
        # [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]
        point = pick_click(error)
        # Positive if the ground truth is set at that point
        # [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]
        label = 1 if gt_mask[point] else 0
        points.append(point)
        labels.append(label)
        mask = model.decode(embedding, points, labels)
    return mask

For box prompts, the 2D study makes one box per part and then merges the outputs. The paper does not say how the outputs are merged [arxiv:2304.10517 §2.2.1]. Consider the merge rule an open choice, and state it in your own code.

Contradictions

• Gain from additional points. One study reports that SAM gains little after two or three points, and that two baselines overtake it at five or more 2304.10517 §3.3. The other reports SAM at 16.15% Dice with N points, 32.24% with 3N and 45.89% with 5N, where N is the number of slices containing the target 2310.15161 tab. 3. The studies differ in unit (per-volume against per-slice), metric (IoU against Dice) and data. Neither paper resolves the difference.
• Upper end of the single-prompt range. The abstract gives a maximum of 0.8650, for a hip X-ray 2304.10517 abstract. Section 3.1 gives 0.9118 as the maximum across tasks 2304.10517 §3.1. The two statements come from the same paper; check the section before quoting either value.

Papers

• Segment Anything Model for Medical Image Analysis: an Experimental Study: zero-shot evaluation of SAM on 28 medical segmentation tasks under five prompt modes, with iterative points and three interactive baselines.
• SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images: a promptable 3D segmentation model trained from scratch on a large volumetric dataset, with 1 to 10 point prompts.
• SegDiff: Image Segmentation with Diffusion Probabilistic Models: a diffusion model that segments per-object crops with no clicks, trained from scratch and averaged over several sampled chains.