Segment Anything Model for Medical Image Analysis: an Experimental Study

In plain words

Segment Anything (SAM) is a model that cuts out whatever object a person points at: you click on it, or draw a box around it. This paper tests how well it works on medical scans such as X-rays, MRI, CT, ultrasound and PET, rather than on everyday photos. Medical images are costly to label, so a model that needs no extra training for each new kind of scan would be useful.

The answer is mixed. On well-defined objects, a single prompt can give a result close to the expert outline. On other targets, such as brain tumours, the result is poor. Boxes work much better than clicks. When the user keeps adding clicks, SAM improves only a little, while the other interactive tools it was compared with catch up and then pass it. The authors conclude that SAM is promising for medical images, but that the prompt style and the dataset decide how well it works.

TL;DR

• Evaluates Meta's SAM zero-shot on 19 public medical datasets, split into 28 segmentation tasks, under five prompt modes, scored by IoU §2.1 §2.2.1.
• Box prompts beat point prompts. The best mode, one box per part, averages 0.6542 IoU; single-prompt IoU ranges from 0.1136 to 0.9118 across tasks §3.1.
• With one point, SAM beats RITM, SimpleClick and FocalClick (mean 0.4595 IoU against 0.2240 for the best baseline). With more iterative points, SAM gains little and two baselines pass it at five or more points §3.2 §3.3.
• All numbers are author-reported. The released code covers the evaluation, but the datasets must be assembled separately.

Problem

Segmentation models for medical images need expert mask annotations, which are expensive to collect §1. Foundation models and zero-shot use are proposed as a way around this §1. SAM, however, takes prompts rather than an image alone, so it cannot replace a standard segmentation network as is §1.2. The paper asks how well SAM segments medical images when prompts are simulated the way a user would give them §2, using the released model with no training described [abstract].

Method

Data. The authors compiled 19 public datasets covering planar X-ray, MRI, CT, ultrasound and PET §2.1 tab. 1 tab. 2. Where one dataset has several kinds of object, each object is a separate task, giving 28 tasks §2.1 §3.1. 3D volumes are fed to SAM as 2D slices tab. 1 caption.

Prompt modes. For each image and each class, the ground-truth object is split into connected parts. Five non-iterative modes build prompts from those parts §2.2.1 fig. 1:

1. One point at the interior point of the largest part.
2. One point per part, for up to three parts.
3. One box tightly around the largest part.
4. One box per part, for up to three parts.
5. One box tightly around the whole object.

Modes 1 and 2 are the same when the object has one part, and modes 3, 4 and 5 are the same in that case too §2.2.1. The interior point is the pixel farthest from the part's edge; ties are broken at random §2.2.1 [code: prompt_gen_and_exec_v2_allmode.py#L263-L266].

Scoring. SAM returns three candidate masks per prompt. The reported non-oracle score uses SAM's highest-confidence mask §2.2.3. The oracle score uses whichever of the three candidates has the best IoU with the ground truth §2.2. IoU is the overlap of prediction and ground truth divided by their union §2.2.3 prompt_gen_and_exec_v2_allmode.py.

Iterative prompts. Algorithm 1 generates points one at a time from the current prediction alg. 1 §2.2.1. Each new point is labelled by the ground truth at that pixel. The error region is the set of pixels where prediction and ground truth disagree.

Competitors (§2.2.2). RITM uses an HRNet-18 backbone with iterative mask correction. SimpleClick uses a plain ViT with a click-embedding layer. FocalClick uses a SegFormer backbone with local updates and progressive merging §2.2.2.

# Five non-iterative prompt modes for one class  [§2.2.1] [fig. 1]
# Input:  mask (H, W), binary, one class
# Output: prompt sets for modes 1 to 5

def make_prompts(mask):
    # Split the object into connected parts (8-neighbour in 2D)
    # [code: prompt_gen_and_exec_v2_allmode.py#L237]
    parts = connected_parts(mask, connectivity=8)
    # Sort parts by pixel count, largest first
    # [code: prompt_gen_and_exec_v2_allmode.py#L250-L251]
    parts = sorted_by_size(parts, largest_first=True)
    # Mode 1: one point in the largest part
    # [§2.2.1] [code: prompt_gen_and_exec_v2_allmode.py#L263-L266]
    points_mode_1 = [interior_point(parts[0])]
    # Mode 2: one point per part, at most three
    # [§2.2.1] [code: prompt_gen_and_exec_v2_allmode.py#L270-L281]
    points_mode_2 = [interior_point(part) for part in parts[:3]]
    # Mode 3: one box around the largest part
    # [§2.2.1] [code: prompt_gen_and_exec_v2_allmode.py#L283-L286]
    boxes_mode_3 = [tight_box(parts[0])]
    # Mode 4: one box per part, at most three
    # [§2.2.1] [code: prompt_gen_and_exec_v2_allmode.py#L288-L294]
    boxes_mode_4 = [tight_box(part) for part in parts[:3]]
    # Mode 5: one box around the whole mask
    # [§2.2.1] [code: prompt_gen_and_exec_v2_allmode.py#L296-L298]
    boxes_mode_5 = [tight_box(mask)]
    return (points_mode_1, points_mode_2, boxes_mode_3,
            boxes_mode_4, boxes_mode_5)

# Interior point and tight box helpers  [§2.2.1]
# Input:  part or mask (H, W), binary
# Output: one (x, y) point, or one box (x0, y0, x1, y1)

def interior_point(part):
    # Distance to the nearest background pixel, L2, 5x5 mask
    # [code: prompt_gen_and_exec_v2_allmode.py#L263]
    distance = distance_transform(part, mask_size=5)
    # Random choice among pixels at the maximum distance
    # [code: prompt_gen_and_exec_v2_allmode.py#L265]
    return random_choice(pixels_where(distance == max(distance)))

def tight_box(mask):
    # Smallest axis-aligned box containing every foreground pixel
    # [code: prompt_gen_and_exec_v2_allmode.py#L60-L68]
    rows, cols = foreground_coordinates(mask)
    return (min(cols), min(rows), max(cols), max(rows))

# Iterative point prompts  [alg. 1] [§2.2.1]
# Input:  image, gt_mask (H, W) binary, num_prompts N
# Output: points [(x, y)], labels [1 foreground, 0 background]

def iterative_points(image, gt_mask, num_prompts):
    # First point: interior point of the object
    # [alg. 1] [code: prompt_gen_and_exec_v1.py#L341-L347]
    points = [interior_point(gt_mask)]
    labels = [1]
    # Predict from all points so far, keep the first candidate
    # [code: prompt_gen_and_exec_v1.py#L355, #L391]
    predicted = sam(image, points, labels)[0]
    for step in range(num_prompts - 1):
        # Error region: false positives and false negatives
        # [alg. 1] [code: prompt_gen_and_exec_v1.py#L402]
        error = xor(gt_mask, predicted)
        # Next point: interior point of the error region
        # [alg. 1] [code: prompt_gen_and_exec_v1.py#L404-L406]
        point = interior_point(error)
        # Label from the ground truth at that pixel
        # [code: prompt_gen_and_exec_v1.py#L409-L413]
        label = 1 if gt_mask[point] else 0
        points.append(point)
        labels.append(label)
        predicted = sam(image, points, labels)[0]
    return points, labels

# Score one prompt set for one class  [§2.2.3]
# Input:  image (H, W, 3) uint8, prompts, mask (H, W) binary
# Output: IoU (float)

def score(image, prompts, mask):
    # Points: one SAM call with all points
    # [code: prompt_gen_and_exec_v2_allmode.py#L306]
    if prompts_are_points(prompts):
        candidates = sam(image, points=prompts)
    # Boxes: one SAM call per box, candidates OR-ed index by index
    # [code: prompt_gen_and_exec_v2_allmode.py#L309-L317]
    else:
        candidates = pixelwise_or(
            [sam(image, box=box) for box in prompts])
    # First candidate, not the top-scored one
    # [code: prompt_gen_and_exec_v2_allmode.py#L331]
    predicted = candidates[0]
    return intersection(predicted, mask) / union(predicted, mask)

Hyperparameters

Results

Author-reported. Setup: 19 datasets, 28 tasks, SAM ViT-H (per code), IoU against ground truth §2.1 §2.2.3. The paper gives no runtime or hardware figures.

The paper states that with a single box (mode 3) SAM outperformed all other methods on all but one task, with an average of 0.5891 IoU §3.2.

Paper vs code

The official code is at raw/code/2304.10517 (commit above). It has two evaluation scripts: prompt_gen_and_exec_v2_allmode.py for the five modes and prompt_gen_and_exec_v1.py for iterative points and the baselines. The released result tables are in experimental_results_tables/ README.md.

1. Mode numbering. The paper numbers modes 1 to 5 §2.2.1. The code numbers them 0 to 4 prompt_gen_and_exec_v2_allmode.py. Paper mode 2 is code mode 1, and so on.
2. "Each part, up to three" means three largest. The paper says one point or box per part, up to three §2.2.1. The code takes the three largest parts, sorted by pixel share, and ignores smaller ones [code: prompt_gen_and_exec_v2_allmode.py#L250-L294].
3. Multi-box outputs are combined by OR. The paper does not say how the outputs of several box prompts are merged. The code runs one SAM call per box and OR-s the binary outputs index by index [code: prompt_gen_and_exec_v2_allmode.py#L309-L317].
4. Mask choice. The paper says the score uses SAM's most confident mask §2.2.3. The code discards SAM's score output and uses candidate index 0 prompt_gen_and_exec_v2_allmode.py. Whether index 0 is always the most confident mask is not checked in this card.
5. Iterative point choice. The text describes the next point as "the central location of the largest component of the error mask" §2.2.1. Algorithm 1 and the code use the point farthest from the error region's boundary, with random tie-breaking alg. 1 prompt_gen_and_exec_v1.py. The paper is inconsistent here; this card follows Algorithm 1 and the code.
6. Iterative labels. The paper says the new point's label depends on whether it lies in the object §2.2.1. The code reads the label from the ground-truth mask at that pixel prompt_gen_and_exec_v1.py, which matches.
7. Prompt history. The code passes all points so far to SAM on each step and does not pass the previous mask back in prompt_gen_and_exec_v1.py. The paper does not describe this.
8. Data preparation is not released. The README lists "Obtaining datasets from our paper" as TODO README.md. The code expects image and mask PNG folders per dataset. How 3D slices were chosen, and how the PET/CT data were packed into PNG channels app. B, is not in the code.
9. Abstract vs results. The abstract gives 0.1135 for spine MRI [abstract]; Section 3.1 gives 0.1136 §3.1. The released table gives 0.11365 for the MRI-Spine GM task under mode 1 [experimental_results_tables/Fig2-Performance of SAM for 5 modes of Use.csv].

Pitfalls

• Prompts come from the ground truth. Boxes are tight to the ground-truth mask, points are the most central pixels of the ground truth, and iterative labels come from the ground truth. The paper calls this a simulation of how a user would prompt §2.2.1. It does not measure real user prompts, so expect different numbers in an interactive product.
• Oracle numbers use the answer key. The oracle picks the candidate with the best IoU against ground truth §2.2. Read it as an upper bound, not as a result a user can get.
• "Mode 3" wording. Section 3.2 calls mode 3 "single box" §3.2. Mode 5 is also a single box, covering the whole object §2.2.1. Check the mode number against §2.2.1 before you use 0.5891.
• Random tie-breaking. Interior points are chosen at random among ties, with seed 1 prompt_gen_and_exec_v2_allmode.py. The code notes that this shifts the score by about 1e-4 prompt_gen_and_exec_v1.py.
• Approximate distance transform. OpenCV's L2 distance with a 5x5 mask is an approximation of Euclidean distance. Points are chosen from it, so ties and near-ties can differ from an exact computation [code: prompt_gen_and_exec_v2_allmode.py#L263].
• Empty error region. If the error region is empty, every pixel has distance zero, so the code picks a random pixel anywhere in the image [code: prompt_gen_and_exec_v1.py#L402-L406]. Not stated in the paper.
• Intensity preprocessing matters. The code rescales each image by its own maximum to 0 to 255 and converts it to RGB [code: prompt_gen_and_exec_v2_allmode.py#L178] [code: prompt_gen_and_exec_v2_allmode.py#L184]. For CT and PET converted to PNG, this changes the contrast SAM sees. The paper does not state it.
• Small parts are kept. The code's loop over parts has a comment about cleaning small regions, but it applies no minimum size [code: prompt_gen_and_exec_v2_allmode.py#L240-L248]. Tiny parts can therefore become prompts in modes 2 and 4.
• Averaging is not specified. The code averages IoU over images for each class, skipping NaN entries prompt_gen_and_exec_v1.py. The paper does not say how the 28 task values are combined into its headline averages.
• Cost. SAM runs once per image for the image embedding, then once per prompt set, and once per box in box modes prompt_gen_and_exec_v2_allmode.py prompt_gen_and_exec_v2_allmode.py. The paper reports no timings.
• Dataset licences. This card covers the paper only. The 19 datasets have their own terms, which the paper does not restate. Check each source before redistributing data or derived images.

Lineage

• Builds on: Segment Anything (SAM), the model evaluated here without changes §1.1. Its card is not in the wiki yet; arXiv:2304.02643 (https://arxiv.org/abs/2304.02643).
• Compared against: RITM, arXiv:2102.06583 (https://arxiv.org/abs/2102.06583), named in the code README README.md. SimpleClick and FocalClick are compared by name only; this card gives no arXiv ids for them §2.2.2.
• Iterative prompt strategy: adapted from an iterative-prompting method cited as Mahadevan et al., 2018 §2.2.1. No arXiv id in the source.
• Superseded by: not stated.