---
kind: concept
slug: interactive-segmentation
title: Interactive segmentation
summary: >-
  Clicks or boxes refine a mask. Zero-shot SAM with boxes is the simplest option for 2D medical images; a promptable 3D model suits volumes, where per-slice prompting is costly. A diffusion model segments one object crop with no clicks. Results are author-reported and simulated from ground truth.
tags: [segmentation, prompting, medical-imaging, diffusion, interactive-segmentation]
papers: ["2304.10517", "2310.15161", "2112.00390"]
related: [promptable-segmentation, diffusion-models, foundation-model, segmentation-evaluation, vision-transformer]
status: draft
updated_at: 2026-10-31
---

## Overview

Interactive segmentation takes a few user prompts, usually clicks inside the
target or a box around it, and returns a mask. More prompts can refine the mask.
The two SAM-style papers below use a Segment Anything-style promptable model on
medical images [arxiv:2304.10517 §1] [arxiv:2310.15161 §1]. The third, SegDiff,
is a diffusion model that takes no clicks. The paper still calls its
per-object setting interactive, because it aims to speed up object annotation
[arxiv:2112.00390 §5, Datasets].

The invariant the three share is that the mask is scored by overlap with a
ground-truth mask: IoU [arxiv:2304.10517 §2.2.3], Dice [arxiv:2310.15161 tab. 3]
or mIoU [arxiv:2112.00390 tab. 1]. In the two SAM-style papers, the prompts are
simulated from the ground truth. The first prompt is an interior point of the
target, and later prompts are placed where the prediction and the ground truth
disagree [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]. SegDiff's
per-object crops are also taken from ground-truth boxes [arxiv:2112.00390 §5,
Datasets]. A real user prompts differently, so these numbers are
upper-bound-style estimates for a product.

## Variants

### Zero-shot 2D SAM with prompt modes

The released 2D model is used without training, one image or slice at a time.
The paper compares five non-iterative prompt modes: one point in the largest
part, one point per part, one box around the largest part, one box per part,
and one box around the whole object [arxiv:2304.10517 §2.2.1]. Across 28
segmentation tasks, one box per part has the best reported mean IoU, 0.6542
[arxiv:2304.10517 §3.1]. Single-prompt IoU ranges from 0.1136 to 0.9118 across
tasks, so the outcome depends strongly on the target [arxiv:2304.10517 §3.1].
Card: [Segment Anything Model for Medical Image Analysis](/papers/2304.10517).

With one point, the same paper reports a mean IoU of 0.4595 for SAM, against
0.2240 for the best of three interactive baselines (RITM, SimpleClick,
FocalClick) [arxiv:2304.10517 §3.2].

### Iterative point refinement with SAM

The same paper also tests iterative points. Each new point goes into the
largest region of error between the current prediction and the ground truth
[arxiv:2304.10517 alg. 1]. The authors report that SAM gains little after two
or three points, and that two of the baselines surpass it at five or more points
[arxiv:2304.10517 §3.3].

### Promptable 3D model for volumes

SAM-Med3D is trained from scratch for volumetric data. It has a 3D image encoder,
a 3D point-prompt encoder and a 3D mask decoder [arxiv:2310.15161 §4.1]. It is
trained on SA-Med3D-140K, which the authors describe as 22K volumes with 143K 3D
masks [arxiv:2310.15161 §3.1]. The authors report a Dice of 76.27% overall with
one 3D point, against 36.83% for SAM-Med2D with one point per slice. With ten
points, Dice reaches 80.71% [arxiv:2310.15161 tab. 3]. Card:
[SAM-Med3D](/papers/2310.15161).

The paper's own pre-experiment compares three ways of adapting SAM to 3D. A
3D adapter on frozen SAM scores 43.84 overall Dice, fine-tuning with SAM weights
43.78, and training from scratch 45.21. On unseen targets, fine-tuning with SAM
weights scores lowest at 18.43 [arxiv:2310.15161 tab. 2]. The pre-experiment
used AMOS training data and is smaller than the main study.

### Diffusion segmenter on per-object crops, no clicks

SegDiff adapts a DDPM to binary masks. The noise predictor adds features of the
input image to the features of the noisy mask, and the whole model is trained
end-to-end without a pretrained backbone [arxiv:2112.00390 §4]. At test time,
several independent reverse chains are averaged and thresholded
[arxiv:2112.00390 §4.1].

The setting is per object. Each Cityscapes sample is one object's crop, taken
from ground-truth boxes, and the model returns that object's mask
[arxiv:2112.00390 §5, Datasets]. The model takes no clicks and refines nothing
iteratively, so it does not fit the prompt loop in the next section. Card:
[SegDiff](/papers/2112.00390).

The authors report 76.44 mIoU on the Cityscapes expansion setting, against 73.70
for the best listed baseline, and 76.19 on tight boxes [arxiv:2112.00390 tab. 1].
The same paper reports Vaihingen and MoNuSeg results in [arxiv:2112.00390 tab. 2] and [arxiv:2112.00390 tab. 3].
The baselines mostly use ImageNet, PASCAL or COCO pretraining, and this model
starts from random weights [arxiv:2112.00390 §5]. Compare architectures with
that in mind. Inference time grows linearly with the number of chains, and the
paper reports it as slower than Segformer and STDC [arxiv:2112.00390 §5,
Results].

## When to use which

| Situation | Use | Why |
|---|---|---|
| 2D images, a model with no training budget, and a sweep of prompt styles | Zero-shot SAM with box prompts | Box prompts beat point prompts in the reported results [arxiv:2304.10517 §3.1]. The method involves no training [arxiv:2304.10517 abstract]. |
| Only one click per target is affordable | A box prompt in 2D; a 3D point model for volumes | One point gives a mean IoU of 0.4595 for SAM across 2D tasks [arxiv:2304.10517 §3.2]. SAM-Med3D reaches 76.27% Dice with one 3D point [arxiv:2310.15161 tab. 3]. The two numbers use different metrics and setups. |
| 3D volumes where a prompt per slice is costly | A volumetric model such as SAM-Med3D | The paper's motivation is that slice-by-slice methods need a prompt per slice [arxiv:2310.15161 §1]. Its per-slice SAM-Med2D baseline reaches 36.83% Dice with one point per slice [arxiv:2310.15161 tab. 3]. |
| Many clicks are affordable | Measure first; the evidence disagrees on the gain per click | SAM gains little after 2–3 points in one study [arxiv:2304.10517 §3.3]. SAM-Med3D rises from 76.27% to 80.71% Dice between one and ten points [arxiv:2310.15161 tab. 3]. |
| Own labelled 3D data, enough to train from scratch | Training from scratch, as SAM-Med3D did | Scratch training scored best overall in the pre-experiment [arxiv:2310.15161 tab. 2]. The paper's data scale is 22K volumes [arxiv:2310.15161 §3.1]. |
| Little labelled data and mostly unseen targets | Do not rely on fine-tuning SAM weights | In the pre-experiment, fine-tuning with SAM weights scored lowest on unseen targets [arxiv:2310.15161 tab. 2]. |
| Object boxes are known, no user is in the loop, and labelled data suffices to train from scratch | A diffusion segmenter such as SegDiff | It is trained end-to-end without a pretrained backbone and reports 76.44 mIoU on Cityscapes object crops [arxiv:2112.00390 tab. 1]. Averaging chains improves mIoU [arxiv:2112.00390 §5]. Each mask needs many reverse steps per chain and several chains [arxiv:2112.00390 §4.1] [arxiv:2112.00390 §5, Training details]. It does not answer how a user refines a mask. |

## Implementation sketch

The loop is the same in both SAM-style papers: encode the image once, then add
a prompt and decode, repeat. The papers differ in how the next prompt is placed,
so the choice is marked in the code. SegDiff takes no clicks, so it uses a
sampling loop of its own, not this one.

```pseudocode
# Interactive loop, shared core  [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]
# Input:  image, gt_mask (binary), num_clicks
# Output: binary mask

def interactive_segment(model, image, gt_mask, num_clicks):
    # First prompt: an interior point of the target
    # [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]
    points = [interior_point(gt_mask)]
    labels = [1]
    # Encode the image once, then decode for each prompt set
    embedding = model.encode(image)
    mask = model.decode(embedding, points, labels)
    for step in range(num_clicks - 1):
        # Error region: where the prediction and ground truth disagree
        # [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]
        error = xor(gt_mask, mask)
        # Next prompt: 2304.10517 uses the interior point of the error
        # region; 2310.15161 uses a random error voxel
        # [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]
        point = pick_click(error)
        # Positive if the ground truth is set at that point
        # [arxiv:2304.10517 alg. 1] [arxiv:2310.15161 §4.3]
        label = 1 if gt_mask[point] else 0
        points.append(point)
        labels.append(label)
        mask = model.decode(embedding, points, labels)
    return mask
```

For box prompts, the 2D study makes one box per part and then merges the
outputs. The paper does not say how the outputs are merged [arxiv:2304.10517
§2.2.1]. Consider the merge rule an open choice, and state it in your own code.

## Contradictions

- **Gain from additional points.** One study reports that SAM gains little after
  two or three points, and that two baselines overtake it at five or more
  [arxiv:2304.10517 §3.3]. The other reports SAM at 16.15% Dice with N points,
  32.24% with 3N and 45.89% with 5N, where N is the number of slices containing
  the target [arxiv:2310.15161 tab. 3]. The studies differ in unit (per-volume
  against per-slice), metric (IoU against Dice) and data. Neither paper resolves
  the difference.
- **Upper end of the single-prompt range.** The abstract gives a maximum of
  0.8650, for a hip X-ray [arxiv:2304.10517 abstract]. Section 3.1 gives 0.9118
  as the maximum across tasks [arxiv:2304.10517 §3.1]. The two statements come
  from the same paper; check the section before quoting either value.

## Papers

- [Segment Anything Model for Medical Image Analysis: an Experimental Study](/papers/2304.10517): zero-shot evaluation of SAM on 28 medical segmentation tasks under five prompt modes, with iterative points and three interactive baselines.
- [SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Images](/papers/2310.15161): a promptable 3D segmentation model trained from scratch on a large volumetric dataset, with 1 to 10 point prompts.
- [SegDiff: Image Segmentation with Diffusion Probabilistic Models](/papers/2112.00390): a diffusion model that segments per-object crops with no clicks, trained from scratch and averaged over several sampled chains.