Segment Anything Models provide reusable priors, yet they require user prompts and cannot support fully automatic instrument segmentation. Automatic prompting is difficult for thin, articulated, reflective, and partly occluded tools, where several configurations can be valid. We formulate automatic prompting as lightweight point-set planning and isolate the point source under a frozen pathway. To this end, we present SetPlanner, a 1.52M-parameter plug-in point-set planner for frozen SAM. The plug-in preserves SAM's point-prompt interface and enables reuse across backbones. SetPlanner plans complete unordered K-point sets from geometry-aware targets with a permutation-aware conditional flow. SAM decodes eight candidates; their consensus readout yields a ground-truth-free prediction. Across three endoscopic datasets, SetPlanner wins all six transfer routes over a LoRA-adapted system. Under our frozen-pathway protocol, SetPlanner reaches 0.934 Dice on Kvasir-Instrument and recovers 96% of a 44.4-point localization gap, while candidate disagreement ranks low-Dice cases at AUROC 0.969.
Figures & tables
Figure 1: Point-source contrast. Left: a single score map with local jitter. Right: one flow over complete point sets. Each polygon links one K=4 point set. The qualitative pair uses an official Kvasir-Instrument test frame.
Figure 2: SetPlanner, left to right. Targets x1 come from the eroded foreground under a Mahalanobis floor in the foreground’s second-moment frame; AR = axis ratio. In model coordinates [−1,1] , the optimal assignment π⋆ pairs a Gaussian draw with the target set. M=8 decoded candidates reduce to one by a consensus readout; disagreement d ranks failures without GT.
Point source
Dice ↑
Mask IoU ↑
Hit ↑
Zero-training rules
Uniform random
0.091
0.060
0.114
Center Gaussian
0.121
0.091
0.131
Color rule
0.509
0.436
0.564
Learned sources ( M=1 , control), frozen decoder
Logistic-1
0.922
0.873
0.970
Table 1: Automatic point sources on frozen SAM. Kvasir. Hit is the fraction of points inside GT. Best scores are bold.
Kvasir
Endoscapes
EndoVis
Kvasir (SAM1)
rule floor
0.509
0.181
0.198
0.594
GT-ref
0.953
0.901
0.929
0.937
Δloc
44.4
72.1
73.0
34.3
Table 2: SetPlanner against Logistic and LoRA, all sharing a frozen image encoder. All three systems decode an M=8 candidate pool and reduce it with the same consensus readout. Top: per-target reference levels. Bottom: Dice by training set; each block tests on one target, so its first row is in-domain and the rows below it transfer. Bold marks the best system in a row. Δm is the eight-candidate gain over one candidate, in the frozen/LoRA order.
Variant
M=1
M=8
Oracle
Pair IoU
Points per set
K=1
0.855
0.911
0.932
0.865
K=2
0.912
0.928
0.941
0.936
K=4
0.925
0.934
0.947
0.951
K=8
0.909
0.918
0.932
0.949
Candidate source
Table 3: Candidate generation and selection ablations. Kvasir. M=1 is one candidate, M=8 the medoid-selected pool, and Oracle the best; all report Dice. The candidate-source rows perturb deterministic slots; the last group varies the selector on one pool. TTA = test-time augmentation. Predicted IoU selects by SAM’s own mask-quality score.
Efficient surgical segmentation empowers clinical diagnosis, intraoperative monitoring, and downstream robotic pipelines for reconstruction and simulation. Although prompt-driven foundation models like Segment Anything Model 3 (SAM3) achieve strong segmentation performance on natural images, surgical data exhibits domain gaps against its pre-training data, resulting in degraded segmentation accuracy. Furthermore, existing medical SAM methods require full-parameter fine-tuning, incurring heavy computational consumption and low efficiency. To address these limitations, this work proposes a parameter-efficient Low-Rank Adaptation (LoRA) adaptation of SAM3 for surgical concept segmentation. We inject low-rank adapters into the prompt encoder, detector and tracker while fully freezing the vision backbone, which only optimizes 0.98% of the total model parameters and supports training on a single consumer GPU. Comprehensive experiments demonstrate that our method consistently outperforms zero-shot SAM3 and other mainstream baselines, and the generated segmentation results can be directly deployed to support downstream robotic surgical scene reconstruction and physical simulation pipelines.
Changjing Liu, Yiming Huang, Beilei Cui +5
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · XGEN Labs
Promptable segmentation models provide a reusable interface, but direct transfer to automatic infrared small-target segmentation (IRSTD) exposes a mismatch between spatial prompts and target-domain mask responses. In a diagnostic using target-covering loose-box prompts deterministically derived from test reference masks, the best official SAM2.1 results are only 4.69%, 1.64%, and 2.28% IoU on NUAA-SIRST, NUDT-SIRST, and IRSTD-1K. We introduce SPARK-SAM (Self-Prompt Adaptation with Response Knowledge for SAM), which learns target-domain response knowledge and conditions the decoder through an image-conditioned joint self-prompt state. Training combines benchmark-mask supervision with reliability-aware response guidance. SPARK-SAM achieves 75.78%, 86.49%, and 68.34% IoU with 0.726M additional parameters, ranking first on two benchmarks among 14 retrained SAM variants and adaptations evaluated as automatic image-to-mask methods. The staged IRSTD-1K diagnostic shows that response adaptation reaches most of the final IoU before the predicted points acquire reliable target grounding. Prompt supervision aligns the predicted prompt candidates with target locations, and frozen-weight interventions measure output sensitivity to the joint self-prompt state. Matched ablations show consistent accuracy gains from response guidance and high-resolution prompt refinement across all three datasets. Code is available at https://github.com/Sakauma/SPARK-SAM.
Aji Mao, Zhenming Peng, Bailin Mu +1
School of Information and Communication Engineering, University of Electronic Science and Technology of China
The Segment Anything Model (SAM) has revolutionized interactive segmentation through spatial prompting. While existing work primarily focuses on automating prompts in various settings, real-world annotation workflows involve iterative refinement where annotators observe model outputs and strategically place prompts to resolve ambiguities. Current pipelines typically rely on the annotator's visual assessment of the predicted mask quality. We postulate that a principled approach for automated interactive prompting is to use a model-derived criterion to identify the most informative region for the next prompt. In this work, we establish active prompting: a spatial active learning approach where locations within images constitute an unlabeled pool and prompts serve as queries to prioritize information-rich regions, increasing the utility of each interaction. We further present BALD-SAM: a principled framework adapting Bayesian Active Learning by Disagreement (BALD) to spatial prompt selection by quantifying epistemic uncertainty. To do so, we freeze the entire model and apply Bayesian uncertainty modeling only to a small learned prediction head, making intractable uncertainty estimation practical for large multi-million parameter foundation models. Across 16 datasets spanning natural, medical, underwater, and seismic domains, BALD-SAM demonstrates strong cross-domain performance, ranking first or second on 14 of 16 benchmarks. We validate these gains through a comprehensive ablation suite covering 3 SAM backbones and 35 Laplace posterior configurations, amounting to 38 distinct ablation settings. Beyond strong average performance, BALD-SAM surpasses human prompting and, in several categories, even oracle prompting, while consistently outperforming one-shot baselines in final segmentation quality, particularly on thin and structurally complex objects.