--- kind: concept slug: promptable-segmentation title: Promptable segmentation summary: >- Segment the structure a user points at, given clicks or boxes. Carded medical models: zero-shot SAM, where one box per part averages 0.6542 IoU, and SAM-Med3D, trained from scratch, at 76.27% Dice from one click. Feature 3DGS runs SAM's decoder on 3D scenes. Author-reported. Trade-off: fewer user prompts against training cost and prompt-style sensitivity. tags: [computer-vision, segmentation, interactive-segmentation, medical-imaging, vision-transformer, zero-shot] papers: ["2304.10517", "2310.15161", "2312.00732", "2312.03203", "2404.17569"] related: [semantic-segmentation, vision-transformer, referring-segmentation, instance-segmentation, segmentation-evaluation, open-vocabulary-segmentation, interactive-segmentation, foundation-model, gaussian-splatting] status: draft updated_at: 2026-10-19 --- ## Overview A promptable segmenter takes an input and a prompt, such as a click or a box on the target, and returns the mask of what the prompt points at. Two carded models do this for medical images, and a third does it on 3D scenes. They differ in how much they are adapted: - A 2D model, SAM, used as released. The authors evaluate it zero-shot on 19 medical datasets with point and box prompts, with no training described [arxiv:2304.10517 §2.1] [arxiv:2304.10517 §2.2]. - A 3D model, SAM-Med3D, trained from scratch on 3D scans. Its architecture has a 3D image encoder, a point-prompt encoder and a mask decoder [arxiv:2310.15161 §4.1]. The shipped code runs the encoder once per volume and reruns only the prompt encoder and mask decoder for each click ([card: Paper vs code and Pitfalls](/papers/2310.15161)). - A 3D feature field, Feature 3DGS, in which SAM's decoder reads features rendered from a trained scene, so new camera views skip SAM's encoder [arxiv:2312.03203 §4.2]. Two more cards are listed under this concept, and neither is a prompted segmenter. [Gaussian Grouping](/papers/2312.00732) runs SAM in automatic mode to supervise its 3D labels. Its groups are anonymous IDs, and text queries need a separate detector [arxiv:2312.00732 §3.2] [arxiv:2312.00732 §4.3]. [MaPa](/papers/2404.17569) splits a mesh into segments by geometry and uses the segment mask as conditioning for a ControlNet-style diffusion model [arxiv:2404.17569 §3.1]. Neither is counted as a variant here. ## Variants ### Zero-shot 2D promptable model (SAM) SAM is prompted with one point, one point per part, one box, or one box per part. Each prompt set is made for one image and one class [arxiv:2304.10517 §2.2.1]. Box prompts do best. One box per part, up to three parts, averages 0.6542 mean IoU across 28 tasks, and one box around the largest part averages 0.5891 [arxiv:2304.10517 §3.1] [arxiv:2304.10517 §3.2]. With one point, the mean is 0.4595. That beats RITM (0.1322), SimpleClick (0.1910) and FocalClick (0.2240), and SAM is ahead of all three on 24 of 28 tasks [arxiv:2304.10517 §3.2]. Single-prompt IoU ranges from 0.1136 to 0.9118 across tasks [arxiv:2304.10517 §3.1]. Card: [Segment Anything Model for Medical Image Analysis](/papers/2304.10517). ### 3D promptable model for volumes A 3D Vision Transformer encodes the whole volume, and the user's clicks steer a 3D mask decoder. The first click is a random foreground voxel. Each later click is a random voxel in the error region, marked positive where the mask misses the target and negative where it marks extra tissue [arxiv:2310.15161 §4.3]. The model is trained from scratch on a labelled collection of 3D scans, not adapted from a 2D model [arxiv:2310.15161 tab. 2] [arxiv:2310.15161 §4.2]. Author-reported overall Dice is 76.27% with one point and 80.71% with ten [arxiv:2310.15161 tab. 3]. Card: [SAM-Med3D](/papers/2310.15161). Volumes wider than 128 voxels per axis are handled, according to the paper, by a sliding window that overlaps by half [arxiv:2310.15161 §4.3]. The released code does not include that window, so this step is paper-only [card: Paper vs code](/papers/2310.15161). ### Prompted segmentation in a 3D feature field Feature 3DGS gives each 3D Gaussian a feature vector and trains it by distillation from a 2D foundation model, using the same splatting that renders colour [arxiv:2312.03203 §3.1] [arxiv:2312.03203 §3.2]. For SAM, the rendered feature goes to the SAM decoder alone, so novel views skip the SAM encoder [arxiv:2312.03203 §4.2]. The authors report this path as up to 1.7 times faster in total than rendering a view and running SAM's encoder and decoder, with nearly equivalent masks [arxiv:2312.03203 fig. 4]. The card records no accuracy for prompts in this mode. The paper's segmentation table uses the LSeg teacher and label queries, not SAM prompts [arxiv:2312.03203 §4] [arxiv:2312.03203 tab. 2]. Card: [Feature 3DGS](/papers/2312.03203). ### Pre-trained 3D encoder for semantic segmentation The same encoder can serve as the pre-trained start of a conventional segmenter. In the paper's UNETR experiment, using the encoder raised the average Dice across four datasets from 83.09 to 85.79, with the largest gain of 5.63 points on AMOS [arxiv:2310.15161 tab. 6] [arxiv:2310.15161 §5.1.6]. This variant needs labelled data for the target task, because it still trains a decoder for that task. Card: [SAM-Med3D](/papers/2310.15161). ## When to use which | Situation | Use | Why | |---|---|---| | 2D medical image, a user who can draw a box, no training budget | Zero-shot SAM with box prompts | One box per part gives 0.6542 mean IoU and one box for the largest part 0.5891, against 0.4595 for one point [arxiv:2304.10517 §3.1] [arxiv:2304.10517 §3.2]. These are simulated prompts from the ground truth, so a live user may do worse. | | 2D medical image, many clicks planned | Do not expect much from extra points | SAM gains little after two or three points, and both SimpleClick and RITM surpass it at five or more [arxiv:2304.10517 §3.3]. | | 2D target that is hard to isolate by one prompt | Check each task before relying on SAM | Single-prompt IoU runs from 0.1136 to 0.9118 across tasks [arxiv:2304.10517 §3.1]. Check the per-task figure before you use the mean. | | Volumetric scan, one target, a user who can click a few times | 3D promptable model, 1 to 10 clicks | Overall Dice 76.27 with one click and 80.71 with ten, against 36.83 for a 2D model with one click per slice [arxiv:2310.15161 tab. 3]. | | Each click costs the user real time | 1 click | The authors' time model gives τ+2 seconds for one click and 10τ+6 for ten, where τ is the user's interaction time [arxiv:2310.15161 tab. 3]. The gain from ten clicks is 4.44 points overall [arxiv:2310.15161 tab. 3]. | | Fixed benchmark with a task-specific model that can be trained | Task-specific model (nnU-Net in the paper) | On AMOS-CT and AMOS-MR the paper's nnU-Net beats its 10-point model, 88.87 vs 83.99 and 86.92 vs 81.13 [arxiv:2310.15161 tab. 4]. This is a benchmark result, not a workflow with prompts. | | Targets may be organs or lesions the training data did not contain | Broad training, not narrow fine-tuning | Fine-tuning on AMOS alone raised unseen-organ Dice from 27.80 to 38.30 but lowered unseen-lesion Dice from 48.44 to 37.43 [arxiv:2310.15161 tab. 7]. | | Little 3D labelled data | Not decided by the cited results | The pre-experiment trains on AMOS only, and the paper does not test small datasets [arxiv:2310.15161 tab. 2]. | | Need a pre-trained 3D encoder for a semantic segmenter | Pre-trained encoder variant | Average Dice 85.79 with the encoder against 83.09 without [arxiv:2310.15161 tab. 6]. One paper, four datasets. | | Prompted masks from new camera views of a 3D scene, research use only | Feature 3DGS, SAM decoder on rendered features | Up to 1.7 times faster in total than running SAM's encoder for each view, with nearly equivalent masks [arxiv:2312.03203 fig. 4]. The official code is under a research-only licence [card: Paper vs code](/papers/2312.03203), so this is not a commercial option as released. | | Every object in a 3D scene needs a mask, with no prompt | Not a prompted segmenter: Gaussian Grouping | SAM runs in automatic mode to supervise per-Gaussian labels [arxiv:2312.00732 §3.2]. Groups are anonymous IDs, so a name needs a separate text query [arxiv:2312.00732 §4.3]. | | Parts of a 3D mesh are needed without any user prompt | Mesh decomposition, not a prompted segmenter | MaPa takes connected components, or graph-cut parts for watertight meshes, as its segments [arxiv:2404.17569 §3.1]. Its segmenter is not prompted, so it does not show a prompt's effect on the mask. | The one-click and per-slice comparison is the SAM-Med3D paper's own headline and pits a single 3D click against a click for every slice. Expect the gap to be smaller or larger on other data, and the baseline weights are not stated [arxiv:2310.15161 §4.3]. ## Implementation sketch The loop below follows the click rules and the decoder call of the 3D model [arxiv:2310.15161 §4.1] [arxiv:2310.15161 §4.3]. It is our sketch, not the paper's algorithm, which the paper does not give. The 2D SAM evaluation uses a different prompt construction, described under Variants and in [arxiv:2304.10517 §2.2.1]. ```pseudocode # Interactive segmentation of one ROI [arxiv:2310.15161 §4.3] # Input: volume (D, H, W), z-scored; gt (D, H, W), used only to simulate clicks # Output: binary mask (D, H, W) def segment_interactive(model, volume, gt, num_clicks): # Encode the volume once; every click reuses these embeddings emb = model.image_encoder(volume) points = [] prev_low_res = None mask = zeros_like(gt) for k in range(num_clicks): # First click: random foreground voxel. Later: random error voxel p, label = next_click(mask, gt, first=(k == 0)) points.append((p, label)) # Decoder reruns with every click so far and the previous logits low_res = model.mask_decoder(emb, points, prev_low_res) prev_low_res = low_res # Logit above 0 is probability above 0.5; this feeds the next click mask = upsample(low_res, gt.shape) > 0 return mask ``` Points where the variants differ or can break: - Use the array axis order of your volume for click coordinates. The card's Pitfalls record that the released code takes (D, H, W) indices, not NIfTI (x, y, z) order. - The 3D encoder's position embedding is fixed to the 128-voxel ROI. Another ROI size needs the embedding resized [card: Pitfalls](/papers/2310.15161). - The paper does not give the click rule used during training. Keep one rule for training and evaluation (see Contradictions). - The 2D SAM prompts are tight boxes or interior points taken from the ground truth. The paper says it simulates a user and does not measure real prompts [arxiv:2304.10517 §2.2.1]. - The paper does not say how the outputs of several box prompts are merged. The released code combines them by a pixel-wise OR [card: Paper vs code](/papers/2304.10517). - The 2D SAM code rescales each image by its own maximum and converts it to RGB. The paper does not state this step, and it changes the contrast SAM sees on CT and PET converted to PNG [card: Pitfalls](/papers/2304.10517). - In a feature field, the SAM teacher runs on a padded 1024 by 1024 input, so the content of a non-square image fills only part of its 64 by 64 map. Pad the rendered map back to 64 by 64 before the SAM decoder runs [arxiv:2312.03203 app. C] [card: Pitfalls](/papers/2312.03203). ## Contradictions - **Click rule in training and evaluation.** The paper does not say which rule its reported numbers use [arxiv:2310.15161 §4.3]. The released code keeps every click at evaluation, but during training it sends only the newest click unless a flag is set [card: Pitfalls](/papers/2310.15161). Both sides are reported; neither is resolved here. - **Training length.** The paper reports 800 epochs for its first training stage [arxiv:2310.15161 §4.2]. The training default in the release is 200, and the release has no first-stage config [card: Paper vs code](/papers/2310.15161). - **Number of clicks evaluated.** The paper reports 1, 3, 5 and 10 points [arxiv:2310.15161 tab. 3]. The released evaluation scripts run one click [card: Paper vs code](/papers/2310.15161). - **Brain Dice.** The 10-point model scores 61.27 on the Brain column of one table [arxiv:2310.15161 tab. 5] and 61.91 in the "all" row of another [arxiv:2310.15161 tab. 7]. The two tables use different validation sets, so do not treat them as one series. - **Next point for iterative prompts.** The SAM paper's text places the next point at the central location of the largest error component [arxiv:2304.10517 §2.2.1]. Its Algorithm 1 uses the point farthest from the error region's boundary, with random ties [arxiv:2304.10517 alg. 1]. The card follows Algorithm 1 and the code. Both sides are reported here. - **Single-prompt SAM on spine MRI.** The SAM paper's abstract gives 0.1135 and its §3.1 gives 0.1136 for the same task [arxiv:2304.10517 §3.1]. - **Zero-shot SAM across the two papers.** SAM-Med3D reports SAM at 16.15 Dice with N points per volume on its 16 validation datasets [arxiv:2310.15161 tab. 3]. The SAM paper reports 0.4595 mean IoU with one point on 28 tasks [arxiv:2304.10517 §3.2]. The data, metric and prompt counts all differ, so the two numbers are not comparable. - **Source of all numbers.** Every number here is author-reported [arxiv:2310.15161 §5.1] [arxiv:2304.10517 §3]. No independent reproduction is recorded in this wiki. ## Papers - [Segment Anything Model for Medical Image Analysis: an Experimental Study](/papers/2304.10517): zero-shot evaluation of SAM under five prompt modes on 19 medical datasets. - [SAM-Med3D](/papers/2310.15161): a 3D promptable model for volumetric medical scans, with a pre-trained encoder shown on semantic segmentation. - [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](/papers/2312.03203): a feature field distilled from SAM and LSeg, with a speed-up for rendering the features. - [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](/papers/2312.00732): per-Gaussian identity labels from SAM masks tracked across views. It is listed here for those labels, not as a prompted segmenter. - [MaPa](/papers/2404.17569): text-driven material painting for 3D meshes. It is listed here for its segment masks, not as a prompted segmenter.