The Segment Anything Model (SAM) has revolutionized interactive segmentation through spatial prompting. While existing work primarily focuses on automating prompts in various settings, real-world annotation workflows involve iterative refinement where annotators observe model outputs and strategically place prompts to resolve ambiguities. Current pipelines typically rely on the annotator's visual assessment of the predicted mask quality. We postulate that a principled approach for automated interactive prompting is to use a model-derived criterion to identify the most informative region for the next prompt. In this work, we establish active prompting: a spatial active learning approach where locations within images constitute an unlabeled pool and prompts serve as queries to prioritize information-rich regions, increasing the utility of each interaction. We further present BALD-SAM: a principled framework adapting Bayesian Active Learning by Disagreement (BALD) to spatial prompt selection by quantifying epistemic uncertainty. To do so, we freeze the entire model and apply Bayesian uncertainty modeling only to a small learned prediction head, making intractable uncertainty estimation practical for large multi-million parameter foundation models. Across 16 datasets spanning natural, medical, underwater, and seismic domains, BALD-SAM demonstrates strong cross-domain performance, ranking first or second on 14 of 16 benchmarks. We validate these gains through a comprehensive ablation suite covering 3 SAM backbones and 35 Laplace posterior configurations, amounting to 38 distinct ablation settings. Beyond strong average performance, BALD-SAM surpasses human prompting and, in several categories, even oracle prompting, while consistently outperforming one-shot baselines in final segmentation quality, particularly on thin and structurally complex objects.
Figures & tables
Fig. 1 : Iterative prompt-based interactive segmentation using SAM. (a) In the interactive loop, SAM receives an input image and a set of user-provided point prompts (positive/inclusion and negative/exclusion) and returns a segmentation mask. A human expert compares the predicted mask against the desired target segmentation and provides additional corrective prompts, which are fed back to SAM in the next iteration. (b) Prompt accumulation and mask evolution across iterations: the left panels show the prompt set at iterations t=0,1,2 , and the right panels show the corresponding SAM outputs, demonstrating error correction and progressive convergence to the desired object mask.
Fig. 2 : BALD-SAM active prompt sampling. At iteration t , the image I and current prompt set St are processed by frozen SAM components and a Bayesian head sampled from a Laplace posterior. Multiple posterior samples produce an ensemble of mask probability maps, from which we compute a disagreement (mutual-information) map. The location with the highest BALD score is queried next, the user returns its label, and the prompt set is updated.
Fig. 3 : Strategy comparison across datasets using Δ IoU over iterative prompting. Each subplot corresponds to one dataset (arranged in a 4 × 4 grid) and shows Δ IoU versus interaction iteration for HUMAN , BALD-SAM (ours) , ENTROPY , RANDOM , and ORACLE strategies, averaged across seeds for a 15-iteration run. To enable within-dataset comparison of trend dynamics, Δ IoU values are min–max normalized separately for each data source. The grid spans diverse domains, including natural images, medical images, underwater images, and seismic images, highlighting the robustness and cross-domain consistency of BALD-SAM under a unified evaluation protocol.
Backbone
Params (M)
Best Val IoU
Worst Val IoU
Mean All Cfgs
ViT-H
632
0.149
0.120
0.141
ViT-B
86
0.121
0.100
0.112
ViT-Tiny
5.7
0.068
0.048
0.059
TABLE I : ViT backbone comparison averaged across the same hyperparameter configurations. We ablated on different inference posterior samples for only 1000 Laplace subsets since they consistently give the best baselines. ViT-H achieves superior performance in all settings, establishing it as the backbone of choice for subsequent ablation studies.
Laplace
Posterior Samples
Training
Subset
10
20
30
40
50
75
100
Overhead
100
0.120
0.124
0.128
0.129
0.130
0.131
0.132
0.92×
0.0195
0.0188
0.0185
0.0183
0.0182
0.0180
0.0178
0.33 ×
0.67 ×
1.00 ×
1.33 ×
1.67 ×
2.50 ×
3.33 ×
300
0.132
0.136
0.140
0.141
0.142
0.142
0.143
0.96×
0.0148
0.0138
0.0135
0.0133
0.0132
0.0131
0.0130
TABLE II : Laplace subset × posterior samples ablation (ViT-H backbone). Each cell shows: Val IoU / ECE / Relative inference cost. Inference baseline (30 samples). & Pareto-optimal configurations are highlighted.
Dataset
Strategy
Peak Normalized Δ IoU
Mean Normalized Δ IoU/Iter
AUC
Baseball bat
BALD-SAM (ours)
0.6570±0.1261
0.3462±0.0575
0.3581
ORACLE
0.6094±0.1169
0.2896±0.0464
0.2933
ENTROPY
0.3891±0.0915
0.1965±0.0492
0.2041
HUMAN
0.5100±0.1544
0.3772±0.1440
0.3873
RANDOM
0.2174±0.0059
0.1429±0.0216
0.1429
Bird
BALD-SAM (ours)
0.6189±0.1584
0.3257±0.1051
0.3212
TABLE III : Performance comparison of active prompting strategies on MS COCO natural images. Best and second-best entries are highlighted, and BALD-SAM (ours) is emphasized in bold.
Dataset
Strategy
Peak Normalized Δ IoU
Mean Normalized Δ IoU/Iter
AUC
Breast
BALD-SAM (ours)
0.3012±0.0167
0.2636±0.0111
0.2668
ORACLE
0.2313±0.0145
0.2212±0.0134
0.2229
ENTROPY
0.2556±0.0095
0.2301±0.0092
0.2327
HUMAN
0.2121±0.0292
0.1921±0.0215
0.1927
RANDOM
0.2719±0.0227
0.2184±0.0169
0.2225
Polyp
BALD-SAM (ours)
0.4535±0.0287
0.3937±0.0248
0.3997
TABLE IV : Performance comparison of active prompting strategies on MS COCO natural images. Best and second-best entries are highlighted, and BALD-SAM (ours) is emphasized in bold.
Dataset
Strategy
Peak Normalized Δ IoU
Mean Normalized Δ IoU/Iter
AUC
Dolphin above
BALD-SAM (ours)
0.9013±0.0715
0.4089±0.0380
0.4273
ORACLE
0.2339±0.0420
0.1784±0.0237
0.1748
ENTROPY
0.2190±0.0022
0.1349±0.0406
0.1435
HUMAN
0.2555±0.2719
0.1360±0.0566
0.1365
RANDOM
0.2160±0.0597
0.1646±0.0761
0.1719
Dolphin below
BALD-SAM (ours)
0.5531±0.3085
0.3385±0.0793
0.3425
TABLE V : Performance comparison of active prompting strategies on MS COCO natural images. Best and second-best entries are highlighted, and BALD-SAM (ours) is emphasized in bold.
Dataset
Strategy
Peak Normalized Δ IoU
Mean Normalized Δ IoU/Iter
AUC
Salt dome
BALD
0.6254±0.0155
0.4855±0.0180
0.4929
ORACLE
0.7713±0.0128
0.5497±0.0194
0.5556
ENTROPY
0.4284±0.0233
0.3371±0.0195
0.3400
HUMAN
0.1642±0.0097
0.1642±0.0097
0.1642
RANDOM
0.5255±0.0952
0.4311±0.0703
0.4380
Chalk group
BALD
0.5534±0.0431
0.4376±0.0340
0.4433
TABLE VI : Performance comparison of active prompting strategies on MS COCO natural images. Best and second-best entries are highlighted, and BALD-SAM (ours) is emphasized in bold
Category
Human
Random
Saliency
K-Medoids
Entropy
Max Dist
Shi-Tomasi
BALD-SAM
Baseball bat
0.747 ± 0.152
0.684 ± 0.198
0.422 ± 0.323
0.724 ± 0.172
0.653 ± 0.217
0.632 ± 0.249
0.701 ± 0.178
0.743 ± 0.175
Bird
0.677 ± 0.231
0.615 ± 0.222
0.308 ± 0.300
0.645 ± 0.212
0.483 ± 0.267
0.456 ± 0.296
0.620 ± 0.216
0.795 ± 0.167
Bus
0.803 ± 0.144
0.593 ± 0.196
0.158 ± 0.190
0.636 ± 0.172
0.359 ± 0.260
0.289 ± 0.277
0.548 ± 0.204
0.855 ± 0.190
Cat
0.887 ± 0.079
0.771 ± 0.149
0.487 ± 0.329
0.825 ± 0.108
0.583 ± 0.284
0.508 ± 0.342
0.795 ± 0.140
0.885 ± 0.105
Clock
0.814 ± 0.181
0.745 ± 0.209
0.432 ± 0.338
0.735 ± 0.205
0.680 ± 0.262
0.692 ± 0.265
0.715 ± 0.223
0.803 ± 0.226
Cow
0.808 ± 0.130
0.675 ± 0.189
0.321 ± 0.294
0.660 ± 0.185
0.423 ± 0.280
0.343 ± 0.314
0.646 ± 0.187
0.805 ± 0.198
TABLE VII : Mean final IoU comparison across prompting strategies across datasets. Best and second-best performance per dataset are highlighted. BALD results are reported after at most 15 prompting rounds, or earlier when the mutual information (MI) reaches the stopping criteria.
Promptable foundation models such as the Segment Anything Model (SAM) produce high-quality masks but remain semantically blind, relying on external prompts to specify categories. Existing vision-language approaches address this limitation by using external prompt coupling, where a vision-language model generates spatial prompts for SAM as a separate stage. We propose CLIP-Guided SAM, a parameter-efficient segmentation framework built on internal semantic conditioning. Instead of using semantic signals only to generate prompts, we inject CLIP-derived text, vision, and similarity features directly into SAM's image encoder through lightweight multi-modal semantic adapters. These adapters condition SAM's internal feature representations, allowing semantic information to influence mask prediction while preserving SAM's original promptable interface. Our framework is designed for low labeled-data settings and applies to both general-domain benchmarks and specialized downstream tasks. It supports two operating modes: Manual mode, for interactive segmentation with both text and spatial prompts, and Semi-Automatic text-only mode, for applications that require concept-specific segmentation using only textual input. We show that robustness depends on aligning training with the type of prompts used at inference, making train-test prompt consistency an important design principle. Through extensive experiments and ablations, we evaluate our method against SAM+PEFT baselines without semantic conditioning, vision-language + SAM pipelines, SAM 3, and strong semi-supervised segmentation methods that rely on large amounts of unlabeled data. Across these settings, CLIP-Guided SAM consistently achieves superior or competitive performance while remaining parameter-efficient in both training and deployment.
Prompt-conditioned foundation segmenters have emerged as a dominant paradigm for image segmentation, where explicit spatial prompts(e.g., points, boxes, masks) guide mask decoding. However, many real-world deployments require fully automatic segmentation, creating a structural mismatch: the decoder expects prompts that are unavailable at inference. Existing adaptations typically modify intermediate features, inadvertently bypassing the model's native prompt interface and weakening prompt-conditioned decoding. We propose IP-SAM, which revisits adaptation from a prompt-space perspective through prompt-space conditioning. Specifically, a Self-Prompt Generator (SPG) distills image context into complementary intrinsic prompts that serve as coarse regional anchors. These cues are projected through SAM2's frozen prompt encoder, restoring prompt-guided decoding without external intervention. To suppress background-induced false positives, Prompt-Space Gating (PSG) leverages the intrinsic background prompt as an asymmetric suppressive constraint prior to decoding. Under a deterministic no-external-prompt protocol, IP-SAM achieves state-of-the-art performance across four camouflaged object detection benchmarks with only 21.26M trainable parameters. Furthermore, the proposed conditioning strategy generalizes beyond COD to medical polyp segmentation.
Huiyao Zhang, Jin Bai, Rui Guo +3
University of Chinese Academy of Sciences · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences
Promptable segmentation models provide a reusable interface, but direct transfer to automatic infrared small-target segmentation (IRSTD) exposes a mismatch between spatial prompts and target-domain mask responses. In a diagnostic using target-covering loose-box prompts deterministically derived from test reference masks, the best official SAM2.1 results are only 4.69%, 1.64%, and 2.28% IoU on NUAA-SIRST, NUDT-SIRST, and IRSTD-1K. We introduce SPARK-SAM (Self-Prompt Adaptation with Response Knowledge for SAM), which learns target-domain response knowledge and conditions the decoder through an image-conditioned joint self-prompt state. Training combines benchmark-mask supervision with reliability-aware response guidance. SPARK-SAM achieves 75.78%, 86.49%, and 68.34% IoU with 0.726M additional parameters, ranking first on two benchmarks among 14 retrained SAM variants and adaptations evaluated as automatic image-to-mask methods. The staged IRSTD-1K diagnostic shows that response adaptation reaches most of the final IoU before the predicted points acquire reliable target grounding. Prompt supervision aligns the predicted prompt candidates with target locations, and frozen-weight interventions measure output sensitivity to the joint self-prompt state. Matched ablations show consistent accuracy gains from response guidance and high-resolution prompt refinement across all three datasets. Code is available at https://github.com/Sakauma/SPARK-SAM.
Aji Mao, Zhenming Peng, Bailin Mu +1
School of Information and Communication Engineering, University of Electronic Science and Technology of China