IP-SAM: Rethinking Prompt-Conditioned Segmentation for Prompt-Absent Deployment

Huiyao Zhang; Jin Bai; Rui Guo; JianWen Tan; HongFei Wang; Ye Li

cs.CVMar 28, 2026

IP-SAM: Rethinking Prompt-Conditioned Segmentation for Prompt-Absent Deployment

Authors: Huiyao Zhang, Jin Bai, Rui Guo, JianWen Tan, HongFei Wang, Ye Li

Organizations: University of Chinese Academy of Sciences · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences

Abstract

Prompt-conditioned foundation segmenters have emerged as a dominant paradigm for image segmentation, where explicit spatial prompts(e.g., points, boxes, masks) guide mask decoding. However, many real-world deployments require fully automatic segmentation, creating a structural mismatch: the decoder expects prompts that are unavailable at inference. Existing adaptations typically modify intermediate features, inadvertently bypassing the model's native prompt interface and weakening prompt-conditioned decoding. We propose IP-SAM, which revisits adaptation from a prompt-space perspective through prompt-space conditioning. Specifically, a Self-Prompt Generator (SPG) distills image context into complementary intrinsic prompts that serve as coarse regional anchors. These cues are projected through SAM2's frozen prompt encoder, restoring prompt-guided decoding without external intervention. To suppress background-induced false positives, Prompt-Space Gating (PSG) leverages the intrinsic background prompt as an asymmetric suppressive constraint prior to decoding. Under a deterministic no-external-prompt protocol, IP-SAM achieves state-of-the-art performance across four camouflaged object detection benchmarks with only 21.26M trainable parameters. Furthermore, the proposed conditioning strategy generalizes beyond COD to medical polyp segmentation.

Figures & tables

Figure 1: Overall architecture of IP-SAM. Shifting the paradigm to prompt-space conditioning, IP-SAM utilizes a Self-Prompt Generator (SPG) to synthesize complementary intrinsic prompts ( $P^{\pm}$ ). These are projected into SAM2’s native manifold via the frozen prompt encoder. Prior to decoding, Prompt-Space Gating (PSG) leverages the negative embedding ( $Z^{-}$ ) to asymmetrically suppress deceptive cues in the positive embedding ( $Z^{+}$ ). The purified condition ( $Z$ ) explicitly steers the mask decoder. For visual clarity, the optional auxiliary structural regularizer (ablated in $\lx@sectionsign~$ 4.3 ) is omitted.

Figure 2: Detailed schematic of Prompt-Space Gating (PSG). PSG isolates and eliminates background-induced false positives before they enter the decoder. It formulates a localized gate decision map from the negative prompt embedding ( $Z^{-}$ ) to physically filter deceptive cues from the positive counterpart ( $Z^{+}$ ). The suppressed feature ( $\widetilde{Z}^{+}$ ) is then fused with unactivated background cues and compensated via an anchored residual connection to yield the robust Conditioned Prompt Embedding ( $Z$ ).

Specialist COD Methods (CNN & Transformer-based)
Method	Param	T-Param	CAMO				CHAMELEON				COD10K				NC4K
Method	(M)	(M)	MAE $\downarrow$	$F^{\omega}\uparrow$	$S_{m}\uparrow$	$E_{\phi}\uparrow$	MAE $\downarrow$	$F^{\omega}\uparrow$	$S_{m}\uparrow$	$E_{\phi}\uparrow$	MAE $\downarrow$	$F^{\omega}\uparrow$	$S_{m}\uparrow$	$E_{\phi}\uparrow$	MAE $\downarrow$	$F^{\omega}\uparrow$	$S_{m}\uparrow$	$E_{\phi}\uparrow$
FSPNet [ 15 ]	274.2	274.2	.060	.761	.864	.880	.036	.766	.887	.889	.034	.671	.843	.858	.043	.772	.878	.891
ZoomNet [ 32 ]	33.4	33.4	.070	.731	.809	.856	.030	.812	.881	.928	.031	.719	.833	.882	.045	.774	.847	.887
DGNet [ 16 ]	8.30	8.30	.057	.769	.839	.901	.029	.816	.890	.934	.033	.693	.822	.877	.042	.784	.857	.907
CamoFormer [ 48 ]	71.3	71.3	.048	.824	.868	.923	.022	.860	.901	.942	.023	.777	.868	.928	.031	.841	.888	.935

Table 1: Quantitative comparison (no-external-prompt). Best results are boldfaced and IP-SAM is highlighted in gray.

SPG	PSG	Lat.	Ref.	T-Param	COD10K				CAMO
SPG	PSG	Lat.	Ref.	(M)	MAE $\downarrow$	$F^{\omega}\uparrow$	$S_{m}\uparrow$	$E_{\phi}\uparrow$	MAE $\downarrow$	$F^{\omega}\uparrow$	$S_{m}\uparrow$	$E_{\phi}\uparrow$
✗	✗	✗	✗	9.15	0.024	0.788	0.857	0.902	0.043	0.818	0.859	0.893
✓	✗	✗	✗	17.00	0.022	0.802	0.872	0.916	0.041	0.826	0.875	0.915
✓	✓	✗	✗	18.98	0.019	0.826	0.893	0.938	0.034	0.868	0.902	0.938
✓	✓	✓	✗	19.13	0.018	0.835	0.899	0.941	0.032	0.871	0.907	0.942
✓	✓	✓	✓	21.26	0.017	0.838	0.906	0.947	0.032	0.878	0.912	0.946

Table 2: Ablation on core components. Incrementally adding SPG, PSG, Lateral, and Refine to the baseline.

(I) Conditioning paradigm & SPG Capacity
Setting	T-Param (M)	COD10K				CAMO
Setting	T-Param (M)	MAE $\downarrow$	$F^{\omega}\uparrow$	$S_{m}\uparrow$	$E_{\phi}\uparrow$	MAE $\downarrow$	$F^{\omega}\uparrow$	$S_{m}\uparrow$	$E_{\phi}\uparrow$
Minimal-SPG (Direct)	13.85	0.051	0.802	0.887	0.896	0.058	0.727	0.873	0.885
Minimal-SPG + PromptSpace	13.85	0.040	0.851	0.894	0.928	0.049	0.818	0.893	0.936
Full-SPG (Direct)	21.26	0.025	0.765	0.883	0.899	0.043	0.829	0.897	0.913
FeatAdd	19.02	0.020	0.825	0.896	0.940	0.039	0.864	0.904	0.936

Table 3: Ablation on design choices. (I) Conditioning paradigm and SPG capacity, (II) PSG operator and residual anchoring, (III) PSG and token replacement, and (IV) LoRA injection strategy.

Figure 3: Qualitative comparison on challenging camouflaged scenarios. Visual predictions of our IP-SAM against representative specialist COD models and SAM-based adaptations under the prompt-absent setting.

Figure 4: Internal feature visualization of the Prompt-Space Gating (PSG) mechanism. Feature evolution from the baseline’s leakage (Col 2) to IP-SAM’s clean prediction (Col 3). Intermediate columns depict suppressed false positives (Col 4), complementary intrinsic prompts (Cols 5-6), gate decision map (Col 7), and feature energy adjustments (Col 8).

Method	Kvasir-SEG (In-domain)				CVC-ClinicDB				ETIS
Method	mDice $\uparrow$	mIoU $\uparrow$	$S_{m}\uparrow$	MAE $\downarrow$	mDice $\uparrow$	mIoU $\uparrow$	$S_{m}\uparrow$	MAE $\downarrow$	mDice $\uparrow$	mIoU $\uparrow$	$S_{m}\uparrow$	MAE $\downarrow$
SEPNet [ 40 ]	.896	.839	.917	.028	.815	.744	.874	.028	.741	.663	.847	.018
LACFormer [ 38 ]	.906	.851	.922	.029	.826	.758	.889	.026	.798	.722	.884	.016
ASPS [ 21 ]	.788	.704	.844	.063	.670	.580	.786	.069	.362	.313	.660	.039
ZoomNeXt [ 33 ]	.892	.831	.909	.031	.805	.731	.863	.031	.693	.625	.841	.023
SAM-Adapter [ 4 ]	.624	.514	.724	.117	.450	.366	.671	.093	.223	.189	.596	.070

Table 4: Cross-dataset medical polyp segmentation. Models are trained on Kvasir-SEG and evaluated on CVC-ClinicDB and ETIS. Best overall are underlined and best SAM-based are boldfaced (IP-SAM belongs to both categories).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

CNN-Based Methods
Method	DUTS-TE				DUT-OMRON				HKU-IS				ECSSD				PASCAL-S
Method	MAE $\downarrow$	$F_{\beta}^{\max}\uparrow$	$S_{m}\uparrow$	$E_{m}^{\max}\uparrow$	MAE $\downarrow$	$F_{\beta}^{\max}\uparrow$	$S_{m}\uparrow$	$E_{m}^{\max}\uparrow$	MAE $\downarrow$	$F_{\beta}^{\max}\uparrow$	$S_{m}\uparrow$	$E_{m}^{\max}\uparrow$	MAE $\downarrow$	$F_{\beta}^{\max}\uparrow$	$S_{m}\uparrow$	$E_{m}^{\max}\uparrow$	MAE $\downarrow$	$F_{\beta}^{\max}\uparrow$	$S_{m}\uparrow$	$E_{m}^{\max}\uparrow$
F3Net [ 42 ]	.035	.905	.888	.920	.053	.841	.838	.864	.028	.943	.917	.952	.033	.957	.924	.948	.061	.892	.861	.898
GateNet-X [ 54 ]	.035	.908	.897	.916	.051	.847	.849	.865	.029	.946	.925	.947	.035	.957	.929	.944	.064	.892	.865	.895
MINet-R [ 34 ]	.037	.884	.884	.917	.056	.831	.833	.860	.029	.942	.919	.952	.033	.954	.925	.950	.064	.881	.856	.896
MENet [ 41 ]	.028	.918	.905	.938	.045	.845	.850	.871	.023	.951	.927	.960	.021	.957	.928	.951	.053	.897	.872	.910

Appendix

Table 5: Quantitative comparison on standard RGB Salient Object Detection (SOD) benchmarks. Results of competing methods are quoted from their original papers when available. “-” indicates the metric is not reported.

F_{\beta}^{\max}

and

E_{m}^{\max}

denote the maximum F-measure and maximum E-measure, respectively. The best results are highlighted in bold .

Figure 5: Qualitative comparison on Salient Object Detection (SOD). Visual results demonstrate IP-SAM’s precise localization and resistance to over-segmentation. A detailed analysis is provided in Section B .

Figure 6: Qualitative comparison on medical polyp segmentation. IP-SAM demonstrates robust boundary preservation and effectively mitigates mucosa-induced false positives. See Section C for detailed observations.

Figure 7: Additional qualitative results on extreme camouflaged scenarios. Visual comparison of IP-SAM against representative COD models and SAM adaptations under the prompt-absent setting. IP-SAM demonstrates robust disambiguation across near-zero contrast (top rows), deceptive distractors (middle rows), and fragile topologies (bottom rows).

(I) Prompt Conditioning Pathway
Setting	T-Param (M)	COD10K				CAMO
Setting	T-Param (M)	MAE $\downarrow$	$F^{\omega}\uparrow$	$S_{m}\uparrow$	$E_{\phi}\uparrow$	MAE $\downarrow$	$F^{\omega}\uparrow$	$S_{m}\uparrow$	$E_{\phi}\uparrow$
No Prompt Conditioning	13.40	0.020	0.811	0.890	0.934	0.040	0.849	0.896	0.930
Bypass Prompt Encoder	21.26	0.020	0.817	0.893	0.935	0.040	0.850	0.897	0.928
Prompt-Space Cond. (Ours)	21.26	0.017	0.838	0.906	0.947	0.032	0.878	0.912	0.946
(II) Input Logit Formats for Prompt Encoder

Appendix

Table 6: Ablation study on prompt conditioning pathways and input logit formats. All variants share an identical downstream MaskDecoderV2 architecture to strictly isolate the source of performance gains. The results indicate that the observed gain is closely tied to routing intrinsic prompts through SAM2’s frozen prompt encoder, especially when using raw continuous logits, rather than to decoder replacement alone.

(a) End-to-end Inference Speed (Wall-clock Protocol)
Method	Latency (ms)	FPS
SAM2-L (zero-prompt baseline)	$\sim$ 28.57	35
IP-SAM-L (Ours)	$\sim$ 34.48	29
Extra Overhead	+5.91	+20.7%

Appendix

Table 7: Computational efficiency analysis of IP-SAM. (a) End-to-end inference speed measured with the same wall-clock protocol as the main paper. (b) Internal component-wise profiling measured with synchronized CUDA Events (averaged over 200 runs after 50 warm-up iterations), used solely to illustrate the relative computational distribution inside the IP-SAM forward pass. Most computation remains in the SAM2 backbone, while the proposed prompt-space modules contribute only a modest fraction of the internal runtime.

Figure 8: Mechanistic visualization of IP-SAM’s failure modes. By unboxing the internal feature evolution, we illustrate how extreme visual ambiguity triggers a coupled error mechanism: the negative prompt ( $P^{-}$ , Column 8) erroneously encroaches upon the target regions, establishing an aggressive suppressive gate, which further diminishes the already weakened intrinsic foreground prompt ( $P^{+}$ , Column 7). This propagates through the Prompt-Space Gating (PSG) module, ultimately resulting in severe under-segmentation.

Explore similar work

CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation
May 24, 2026Shayan Jalilian, Abdul BaisVLM Adaptation Text-Guided Image Segmentation
University of Regina, Regina, SK, Canada
SPARK-SAM: Learning How to Prompt and Respond for Infrared Small Target Segmentation
Aug 21, 2026Aji Mao, Zhenming Peng, Bailin Mu +1Image Segmentation Promptable Image Segmentation
School of Information and Communication Engineering, University of Electronic Science and Technology of China
PR-MaGIC: Prompt Refinement Via Mask Decoder Gradient Flow For In-Context Segmentation
Apr 13, 2026Minjae Lee, Sungwoo Hur, Soojin Hwang +1Image Segmentation Promptable Image Segmentation
Pohang University of Science and Technology, Pohang, South Korea

May 24, 2026J/K move · Enter open · S save

CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation

Shayan Jalilian, Abdul Bais

University of Regina, Regina, SK, Canada

VLM Adaptation Text-Guided Image Segmentation Promptable Image Segmentation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Promptable foundation models such as the Segment Anything Model (SAM) produce high-quality masks but remain semantically blind, relying on external prompts to specify categories. Existing vision-language approaches address this limitation by using external prompt coupling, where a vision-language model generates spatial prompts for SAM as a separate stage. We propose CLIP-Guided SAM, a parameter-efficient segmentation framework built on internal semantic conditioning. Instead of using semantic signals only to generate prompts, we inject CLIP-derived text, vision, and similarity features directly into SAM's image encoder through lightweight multi-modal semantic adapters. These adapters condition SAM's internal feature representations, allowing semantic information to influence mask prediction while preserving SAM's original promptable interface. Our framework is designed for low labeled-data settings and applies to both general-domain benchmarks and specialized downstream tasks. It supports two operating modes: Manual mode, for interactive segmentation with both text and spatial prompts, and Semi-Automatic text-only mode, for applications that require concept-specific segmentation using only textual input. We show that robustness depends on aligning training with the type of prompts used at inference, making train-test prompt consistency an important design principle. Through extensive experiments and ablations, we evaluate our method against SAM+PEFT baselines without semantic conditioning, vision-language + SAM pipelines, SAM 3, and strong semi-supervised segmentation methods that rely on large amounts of unlabeled data. Across these settings, CLIP-Guided SAM consistently achieves superior or competitive performance while remaining parameter-efficient in both training and deployment.

Abstract

Explore similar work

CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation

SPARK-SAM: Learning How to Prompt and Respond for Infrared Small Target Segmentation

PR-MaGIC: Prompt Refinement Via Mask Decoder Gradient Flow For In-Context Segmentation