Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications

In plain words

Meta released Segment Anything (SAM), a model that cuts out an object in a picture when you point at it, draw a box around it, or ask it to find everything. It was trained on a very large collection of images, but the real world has many cases that such a collection may not cover well: dark scenes, see-through objects, objects that blend into their background, or pictures taken from above.

This paper runs SAM on those kinds of problems: natural photos, crops and pests, factory defects, aerial images of roads and buildings, and medical images. It looks at the pictures by eye and also measures how far SAM's best output is from specialist models on eight public benchmarks.

The result is mixed. SAM finds the location of prominent objects well and handles regular shapes. It struggles with fine detail, with hidden or low-contrast objects, with shadows, and with specialist images such as retinal scans, where the right prompt needs expert knowledge. The authors read this as a call for domain-specific models and for more study.

TL;DR

• SAM is a promptable segmentation model. This paper is an evaluation study, not a new method.
• Qualitatively, SAM works on prominent objects and regular shapes. It misses camouflaged objects and shadows, struggles with small or irregular objects, and needs expert prompts for medical and industrial images. Transparent objects are located well but with weak detail §2.1 §2.3 §2.4 §2.5.
• Quantitatively, the best automatic candidate, chosen with ground truth, has a higher MAE than dedicated models on all eight benchmarks. With ViT-H the gap is 0.8 to 17.6 points tab. 1.
• Use the numbers as an oracle upper bound for SAM's candidate masks, not as a deployable result. The code's selection rules and metrics also differ from the paper in places (see Paper vs code).

Problem

Pre-training on a single dataset cannot cover every real-world condition, such as low light, fog, rain, bird's-eye views, or non-RGB inputs like depth, infrared, event, point cloud, CT and MRI §1. SAM was pre-trained on SA-1B, which the paper describes as more than 11 million images and one billion masks §1. The paper asks how well SAM generalises to such scenarios and applications, without training anything new §1.

The paper does not propose a method. It assumes that SAM is used with prompts: foreground and background points, a box, a mask, or text §1. Its qualitative runs use the authors' prompts in SAM's online demo, and the paper says these prompts may not be optimal because excessive interaction is not desirable §1, footnote.

Method

The "method" is an evaluation protocol with two parts.

Qualitative study §2. SAM is run on images from several applications (natural scenes, agriculture, manufacture, remote sensing, healthcare). Each application is shown in three prompt modes: click (SAM^1), box (SAM^2), and everything (SAM^3) §1 fig. 2–fig. 14. Results are judged visually and described in words. The runs use the online demo, not the released code [§1 footnote].

Quantitative study §3. For each benchmark image the authors run SAM to produce N candidate object masks. They then choose the candidate that best matches the ground truth and report the MAE of that candidate §3.2. The paper says the choice is based on IoU with the ground truth, and that the mask with the highest IoU is selected §3.2. The metric is the mean absolute error, M, where lower is better §3.1.

The protocol, as implemented in the official code (see Paper vs code):

# Oracle best-candidate evaluation  [§3.2]
# Code: scripts/sam_dice_f1_mae.py#L106-L139
# Input:  candidate masks for one image, binary ground truth gt
# Output: MAE of the selected candidate

def evaluate_image(candidates, gt):
    # Score each candidate against the ground truth  [§3.2]
    # The paper says IoU; the default script ranks by Dice, which orders
    # candidates the same way.
    best_score = 0
    for mask in candidates:
        score = overlap_score(mask, gt)
        if score > best_score:
            best_score = score
            best_mask = mask
    # Mean absolute error of the chosen mask, pixels in [0, 1]  [§3.1]
    return mean(abs(best_mask - gt))

# Dataset score  [code: scripts/sam_dice_f1_mae.py#L170-L172]
# Input:  per-image MAE values for one benchmark
# Output: average MAE over images

def dataset_score(per_image_mae):
    # Plain mean over images; weighting not stated  [§3.2]
    return mean(per_image_mae)

Key equations: none are numbered in the paper. The IoU is described in words §3.2.

Hyperparameters

The paper states few settings. Candidate-generation values below come from the code's library defaults, since the code's runner passes none of them unless flags are set amg.py.

Results

Author-reported. Setup: eight quantitative benchmarks §3.1; SAM ViT-B, ViT-L and ViT-H with the oracle best-candidate protocol above §3.2; MAE, lower is better. Baseline numbers are cited to their source papers in the table; the paper does not say whether they were re-run tab. 1.

Source: tab. 1. The "gap" is the paper's Δ, shown as printed. For every row it equals the absolute MAE difference between SAM ViT-H and the best non-SAM value (e.g. 0.265 − 0.089 = 0.176 for CDS2K).

Qualitative findings, from the text and figures (no numerical scores):

• Salient objects (ReDWeb-S): strong at finding prominent objects, weaker on fine detail §2.1 fig. 2.
• Camouflaged objects (COD10K): often misses whole objects in cluttered or low-contrast scenes; false positives remain even with SAM^1 §2.1 fig. 3.
• Transparent objects (Trans10K-v2): locates objects well; finer detail is weak §2.1 fig. 4.
• Shadows (shadow detection datasets): fails to detect shadows, even with many clicks §2.1 fig. 5.
• Crops: relatively satisfactory on the first case shown, less so on the second; the authors suggest crops and land are usually labelled as background in general segmentation data, so positive examples are scarce §2.2 fig. 6.
• Pests and leaf disease: strong on the easier case; the whole pest body is hard in the harder case §2.2 fig. 7.
• Anomaly detection (MVTec AD): described as showing powerful recognition §2.3 fig. 8.
• Surface defects: SAM^1 acceptable with reasonable prompts; SAM^3 gives no meaningful result §2.3 fig. 9.
• Buildings (remote sensing): good on regular shapes, weak on small or hard-to-see buildings §2.4 fig. 10.
• Roads: diverse shapes, sizes and textures make accurate segmentation hard §2.4 fig. 11.
• Optic disc and cup (RIGA): severe shortcomings; SAM^3 gives no result in some cases §2.5 fig. 12.
• Polyps: SAM^1 impressive with adequate prompts; SAM^2 and SAM^3 struggle §2.5 fig. 13.
• Skin lesions: satisfactory with enough prompts; considerable room for SAM^2 and SAM^3 §2.5 fig. 14.

Paper vs code

Official repository: LiuTingWed/SAM-Not-Perfect, linked from the paper's abstract and introduction, at commit 28d442f. The repository contains evaluation scripts and a copy of the segment_anything package README.md. There is no LICENSE file. The vendored package's header refers to "the LICENSE file in the root directory", which is absent automatic_mask_generator.py.

• Selection rule. The paper selects by IoU §3.2. sam_dice_f1_mae.py selects by Dice sam_dice_f1_mae.py. For a given image Dice is 2·IoU/(1+IoU), an increasing function of IoU, so the chosen candidate is the same. sam_f1_dice_mae.py selects by its F1 value sam_f1_dice_mae.py. That can choose a different candidate. Which script produced Table 1 is not stated.
• F1 formula. calculate_f_measure returns β²·P·R/(P+R) and calls it F1 sam_dice_f1_mae.py sam_dice_f1_mae.py. At β=1 this is PR/(P+R), which is half the standard F1 (2PR/(P+R)). The paper does not report F1, so the table is not affected, but any F1 number from this code is halved.
• Unused IoU helper. cal_iou computes I/(|G|+|P|), which is neither IoU nor Dice, and it is never called sam_dice_f1_mae.py.
• Candidate pool. The paper says "N potential object masks" §3.2 but gives no N. The runner uses the library's automatic generator with defaults unless flags are set amg.py automatic_mask_generator.py. Each grid point yields three multimask candidates automatic_mask_generator.py, filtered by the thresholds in the Hyperparameters table.
• Hard-coded paths. amg.py defaults point to author-machine paths for the checkpoint and output, and it builds the input path as ../datasets/<name>/test_images amg.py amg.py amg.py amg.py. These must be edited before a run.
• Baseline script bug. In other_methods_dice_mae.py, the "F1" average is total_mae / num_images other_methods_dice_mae.py. Any F1 column from that script is actually MAE. Ground-truth names are also forced to .jpg other_methods_dice_mae.py.
• Qualitative visuals. The figures come from the online demo with human prompts §1 footnote. The released code does not reproduce them; it has no interactive click or box runner.
• Version. This card covers arXiv v3 (updated 2023-05-22). The README also mentions a longer version in Machine Intelligence Research README.md. That version was not read and may differ.

Searched: the repository is linked by the paper's abstract and by its introduction footnote, so no further search was needed. The code was read, not run. The discrepancies above come from reading the scripts and the candidate generator defaults, and were not confirmed by executing them.

Pitfalls

• The "17.6% worse" statement is an absolute gap, not a relative one. The paper says SAM ViT-H is "17.6% worse" than DGNet on CDS2K §3.2 tab. 1. The values are 0.265 vs 0.089, so the difference is 0.176 MAE points. Relative to DGNet, SAM's MAE is about 198% higher. Quote the points, not a percentage of error.
• The SAM numbers are an oracle upper bound. The best candidate is picked using ground truth §3.2. In deployment no ground truth exists, and the paper gives no label-free rule for choosing among candidates. Expect worse numbers from any real selection rule.
• Protocols may differ across rows. SAM's rows use the authors' oracle selection. Baselines are copied from other papers tab. 1. The paper does not say whether they used the same splits, input sizes or metrics.
• Numbers depend on undocumented generator settings. N and the candidate pool follow from the library defaults in the code. Changing points-per-side, the IoU or stability thresholds, or crop layers changes the candidates and so the scores.
• Prompts for the qualitative runs are not reproducible from the paper. The authors say the prompts are effective but not optimal, and the effort spent is not measured §1 footnote.
• Only MAE is reported. Dice and F1 are computed in the code but are not in the paper. Comparisons with other papers that report only F-measure, or only Dice, need care.
• Mode names are inconsistent across figures. The quantitative table does not name the prompt mode. The text's "N potential object masks" matches the everything mode, which is the code's only quantitative path amg.py.
• Medical and industrial results depend on expert prompts. The section recaps for healthcare and manufacture say expert prior knowledge is needed in click mode, and that box and everything modes fall short §2.5 §2.3.
• Figure and table numbers are in source order. The paper has one table (tab. 1) and fourteen figures; figure numbers here follow the order of the LaTeX source fig. 1–fig. 14.

Lineage

• Builds on: SAM, cited in the paper as [SAM]. The paper gives no arXiv id for it §1.
• Compared against: the dedicated models listed per benchmark in tab. 1: VST, ICONet, EDNet, Gate, PFNet, IS-Net, SegMaR, PFNet+, ZoomNet, DSC, DSDNet, MTMT, SINetV2, HitNet, DGNet, FAPNet, CFA-Net and HSNet.
• Related follow-up named in the repository: Medical SAM Adapter, arXiv 2304.12620, for medical images README.md. Not covered by this card.
• Superseded by: not known from content already in this wiki.