SetPlanner: A Lightweight Plug-in Point-Set Planner for Frozen SAM

Dawei Yan; Yuezhe Yang; Menglan Ruan; Chunfeng Yang; Yudong Zhang

cs.CVSep 16, 2026

SetPlanner: A Lightweight Plug-in Point-Set Planner for Frozen SAM

Authors: Dawei Yan, Yuezhe Yang, Menglan Ruan, Chunfeng Yang, Yudong Zhang

Organizations: School of Computer Science and Engineering, Southeast Univ., China · School of Computer Science, Univ. of Sydney, Australia

Abstract

Segment Anything Models provide reusable priors, yet they require user prompts and cannot support fully automatic instrument segmentation. Automatic prompting is difficult for thin, articulated, reflective, and partly occluded tools, where several configurations can be valid. We formulate automatic prompting as lightweight point-set planning and isolate the point source under a frozen pathway. To this end, we present SetPlanner, a 1.52M-parameter plug-in point-set planner for frozen SAM. The plug-in preserves SAM's point-prompt interface and enables reuse across backbones. SetPlanner plans complete unordered K-point sets from geometry-aware targets with a permutation-aware conditional flow. SAM decodes eight candidates; their consensus readout yields a ground-truth-free prediction. Across three endoscopic datasets, SetPlanner wins all six transfer routes over a LoRA-adapted system. Under our frozen-pathway protocol, SetPlanner reaches 0.934 Dice on Kvasir-Instrument and recovers 96% of a 44.4-point localization gap, while candidate disagreement ranks low-Dice cases at AUROC 0.969.

Figures & tables

Figure 1: Point-source contrast. Left: a single score map with local jitter. Right: one flow over complete point sets. Each polygon links one $K\!=\!4$ point set. The qualitative pair uses an official Kvasir-Instrument test frame.

Figure 2: SetPlanner, left to right. Targets $x_{1}$ come from the eroded foreground under a Mahalanobis floor in the foreground’s second-moment frame; AR = axis ratio. In model coordinates $[-1,1]$ , the optimal assignment $\pi^{\star}$ pairs a Gaussian draw with the target set. $M{=}8$ decoded candidates reduce to one by a consensus readout; disagreement $d$ ranks failures without GT.

Zero-training rules
Point source	Dice $\uparrow$	Mask IoU $\uparrow$	Hit $\uparrow$
Uniform random	0.091	0.060	0.114
Center Gaussian	0.121	0.091	0.131
Color rule	0.509	0.436	0.564
Learned sources ( $M{=}1$ , control), frozen decoder
Logistic-1	0.922	0.873	0.970

Table 1: Automatic point sources on frozen SAM. Kvasir. Hit is the fraction of points inside GT. Best scores are bold.

	Kvasir	Endoscapes	EndoVis	Kvasir (SAM1)
rule floor	0.509	0.181	0.198	0.594
GT-ref	0.953	0.901	0.929	0.937
$\Delta_{\mathrm{loc}}$	44.4	72.1	73.0	34.3

Table 2: SetPlanner against Logistic and LoRA, all sharing a frozen image encoder. All three systems decode an

M{=}8

candidate pool and reduce it with the same consensus readout. Top: per-target reference levels. Bottom: Dice by training set; each block tests on one target, so its first row is in-domain and the rows below it transfer. Bold marks the best system in a row.

\Delta_{\mathrm{m}}

is the eight-candidate gain over one candidate, in the frozen/LoRA order.

Points per set
Variant	$M{=}1$	$M{=}8$	Oracle	Pair IoU
$K{=}1$	0.855	0.911	0.932	0.865
$K{=}2$	0.912	0.928	0.941	0.936
$K{=}4$	0.925	0.934	0.947	0.951
$K{=}8$	0.909	0.918	0.932	0.949
Candidate source

Table 3: Candidate generation and selection ablations. Kvasir.

M{=}1

is one candidate,

M{=}8

the medoid-selected pool, and Oracle the best; all report Dice. The candidate-source rows perturb deterministic slots; the last group varies the selector on one pool. TTA = test-time augmentation. Predicted IoU selects by SAM’s own mask-quality score.

Promptable Image Segmentation

Promptable image segmentation uses points, boxes, text, or masks to specify objects and generate pixel-level regions.

See all 152 papers

Parameter-Efficient Adaptation of SAM3 for Prompt-Driven Surgical Concept Segmentation
Jul 26, 2026Changjing Liu, Yiming Huang, Beilei Cui +5Promptable Image Segmentation Vision Foundation Model Adaptation
Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · XGEN Labs
SPARK-SAM: Learning How to Prompt and Respond for Infrared Small Target Segmentation
Aug 21, 2026Aji Mao, Zhenming Peng, Bailin Mu +1Image Segmentation Promptable Image Segmentation
School of Information and Communication Engineering, University of Electronic Science and Technology of China
BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation
Mar 11, 2026Prithwijit Chowdhury, Mohit Prabhushankar, Ghassan AlRegibInteractive Image Segmentation Promptable Image Segmentation
OLIVES at the Georgia Institute of Technology

Jul 26, 2026J/K move · Enter open · S save

Parameter-Efficient Adaptation of SAM3 for Prompt-Driven Surgical Concept Segmentation

Changjing Liu, Yiming Huang, Beilei Cui, Liangjing Shao+4

Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China · XGEN Labs

Promptable Image Segmentation Vision Foundation Model Adaptation Medical Image Segmentation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Efficient surgical segmentation empowers clinical diagnosis, intraoperative monitoring, and downstream robotic pipelines for reconstruction and simulation. Although prompt-driven foundation models like Segment Anything Model 3 (SAM3) achieve strong segmentation performance on natural images, surgical data exhibits domain gaps against its pre-training data, resulting in degraded segmentation accuracy. Furthermore, existing medical SAM methods require full-parameter fine-tuning, incurring heavy computational consumption and low efficiency. To address these limitations, this work proposes a parameter-efficient Low-Rank Adaptation (LoRA) adaptation of SAM3 for surgical concept segmentation. We inject low-rank adapters into the prompt encoder, detector and tracker while fully freezing the vision backbone, which only optimizes 0.98% of the total model parameters and supports training on a single consumer GPU. Comprehensive experiments demonstrate that our method consistently outperforms zero-shot SAM3 and other mainstream baselines, and the generated segmentation results can be directly deployed to support downstream robotic surgical scene reconstruction and physical simulation pipelines.

Abstract

Promptable Image Segmentation

Explore similar work

Parameter-Efficient Adaptation of SAM3 for Prompt-Driven Surgical Concept Segmentation

SPARK-SAM: Learning How to Prompt and Respond for Infrared Small Target Segmentation

BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation