SPARK-SAM: Learning How to Prompt and Respond for Infrared Small Target Segmentation

Aji Mao; Zhenming Peng; Bailin Mu; Tian Pu

cs.CVAug 21, 2026

SPARK-SAM: Learning How to Prompt and Respond for Infrared Small Target Segmentation

Authors: Aji Mao, Zhenming Peng, Bailin Mu, Tian Pu

Organizations: School of Information and Communication Engineering, University of Electronic Science and Technology of China

Abstract

Promptable segmentation models provide a reusable interface, but direct transfer to automatic infrared small-target segmentation (IRSTD) exposes a mismatch between spatial prompts and target-domain mask responses. In a diagnostic using target-covering loose-box prompts deterministically derived from test reference masks, the best official SAM2.1 results are only 4.69%, 1.64%, and 2.28% IoU on NUAA-SIRST, NUDT-SIRST, and IRSTD-1K. We introduce SPARK-SAM (Self-Prompt Adaptation with Response Knowledge for SAM), which learns target-domain response knowledge and conditions the decoder through an image-conditioned joint self-prompt state. Training combines benchmark-mask supervision with reliability-aware response guidance. SPARK-SAM achieves 75.78%, 86.49%, and 68.34% IoU with 0.726M additional parameters, ranking first on two benchmarks among 14 retrained SAM variants and adaptations evaluated as automatic image-to-mask methods. The staged IRSTD-1K diagnostic shows that response adaptation reaches most of the final IoU before the predicted points acquire reliable target grounding. Prompt supervision aligns the predicted prompt candidates with target locations, and frozen-weight interventions measure output sensitivity to the joint self-prompt state. Matched ablations show consistent accuracy gains from response guidance and high-resolution prompt refinement across all three datasets. Code is available at https://github.com/Sakauma/SPARK-SAM.

Figures & tables

Figure 1: Prompt–response gap in direct SAM2.1 transfer. A loose-box prompt deterministically derived from the test reference mask covers the target, yet the unadapted response contains extensive false positives.

Figure 2: Overview of SPARK-SAM. Response-knowledge adaptation learns target-domain conditional prompt-to-mask behavior, and self-prompt adaptation conditions the decoder through a joint self-prompt state containing encoded coordinates, location-sampled tokens, residual prompt context, and a compact box. Reliability-aware response guidance supports training, and high-resolution encoder maps provide an additional prompt candidate.

Figure 3: Frozen-weight dependence on the joint self-prompt state. Bars report variant-minus-full-model

\Delta\mathrm{nIoU}

in percentage points; panel-specific symmetric-log horizontal scales expand the near-zero box effects. Complete removal deletes

S_{I}

; spatial interventions preserve

T_{\mathrm{res}}

and the box, while box interventions preserve

T_{\mathrm{res}}

and the point/local-token branch. Post-adaptation mask-derived box replacement measures residual box-conditioned headroom.

Figure 4: Mean IoU across NUAA-SIRST, NUDT-SIRST, and IRSTD-1K versus audited batch-1 FP32 latency. The broken vertical axis separates the low-IoU Tiny-core anchor from the adapted methods. Marker shapes distinguish method families, marker sizes denote model complexity, and labels indicate native input resolution.

Official SAM2.1 models with mask-derived box prompts
	NUAA-SIRST						NUDT-SIRST						IRSTD-1K
Method	IoU $\uparrow$	nIoU $\uparrow$	F1 $\uparrow$	Pr $\uparrow$	Re $\uparrow$	FA $\downarrow$	IoU $\uparrow$	nIoU $\uparrow$	F1 $\uparrow$	Pr $\uparrow$	Re $\uparrow$	FA $\downarrow$	IoU $\uparrow$	nIoU $\uparrow$	F1 $\uparrow$	Pr $\uparrow$	Re $\uparrow$	FA $\downarrow$
SAM2.1 Tiny (mask-derived box) ( Ravi et al. 2025 )	2.99	37.24	5.80	3.00	90.08	18991.00	1.40	32.82	2.76	1.40	76.37	35925.50	1.57	32.55	3.09	1.58	76.55	11996.90
SAM2.1 Small (mask-derived box) ( Ravi et al. 2025 )	4.69	37.58	8.96	4.71	92.47	13898.30	1.41	31.48	2.78	1.42	79.65	37098.30	1.70	31.57	3.35	1.71	80.55	11646.80
SAM2.1 Base+ (mask-derived box) ( Ravi et al. 2025 )	4.00	38.09	7.68	4.01	93.73	15652.80	1.64	33.22	3.23	1.64	86.97	34824.00	1.85	31.43	3.64	1.86	90.24	11980.10
SAM2.1 Large (mask-derived box) ( Ravi et al. 2025 )	2.93	38.59	5.69	2.94	91.85	20560.50	1.16	33.76	2.28	1.16	80.16	45778.70	2.28	33.18	4.47	2.29	87.63	9388.80

Table 1: Segmentation comparison with official and retrained SAM variants. IoU, nIoU, F1, precision (Pr), and recall (Re) are percentages; FA is in px/MP. Official rows form a diagnostic with mask-derived box prompts: each frozen SAM2.1 model receives a target-covering loose box deterministically computed from the corresponding test reference mask and predicts only the mask. Retrained rows receive only infrared images at inference and use automatic image-to-mask pathways. Cyan and orange denote the best and second-best results.

Official SAM2.1 models with mask-derived box prompts
Method	Input	Params	VRAM	GFLOPs	Mean	Median	P95
SAM2.1 Tiny (mask-derived box)	$1024^{2}$	38.962	595.02	209.855	31.400	30.020	37.614
SAM2.1 Small (mask-derived box)	$1024^{2}$	46.060	622.10	271.644	35.286	34.167	39.915
SAM2.1 Base+ (mask-derived box)	$1024^{2}$	80.850	802.84	532.233	51.094	47.803	67.576
SAM2.1 Large (mask-derived box)	$1024^{2}$	224.447	1446.67	1622.795	119.189	96.632	236.362
Retrained SAM variants and adaptations

Table 2: Computational efficiency of official SAM2 models and retrained SAM variants/adaptations. Input lists the internal square resolution; Native denotes model-defined dynamic preprocessing. Parameters are in millions, peak VRAM in MiB, computational cost in GFLOPs, and latency in ms. Lower is better.

NUAA-SIRST
Method	IoU $\uparrow$	nIoU $\uparrow$	F1 $\uparrow$	Pr $\uparrow$	Re $\uparrow$	FA $\downarrow$
DNANet ( Li et al. 2023 )	76.82	77.12	86.89	88.06	85.75	70.67
ACM ( Dai et al. 2021 )	64.11	64.00	78.13	76.61	79.71	147.83
SCTransNet ( Yuan et al. 2024 )	78.46	80.04	87.93	94.83	81.96	27.14
RPCANet++ ( Wu et al. 2026 )	63.48	62.96	77.66	78.00	77.33	132.46
L2SKNet ( Wu et al. 2024a )	55.27	53.47	71.19	82.41	62.66	81.25

Table 3: Comparison with specialized IRSTD networks. IoU, nIoU, F1, precision (Pr), and recall (Re) are percentages; FA is in px/MP. Cyan and orange denote the best and second-best results.

(a) Prompt interventions: change from learned prompts
	NUAA		NUDT		IRSTD
Intervention	$\Delta$ IoU	$\Delta$ nIoU	$\Delta$ IoU	$\Delta$ nIoU	$\Delta$ IoU	$\Delta$ nIoU
No prompt state	$-74.70$	$-28.27$	$-41.74$	$-8.84$	$-47.09$	$-18.68$
Random spatial branch	$-74.84$	$-33.63$	$-14.26$	$-5.45$	$-18.65$	$-0.65$
Shuffled spatial branch	$-74.95$	$-29.41$	$-5.39$	$-5.15$	$-0.11$	$-0.14$
Box-only shuffle	$-0.49$	$-0.11$	$-0.45$	$-0.18$	$-0.20$	$-0.20$

Table 4: Mechanism analysis. All changes use variant-minus-full-model percentage points. (a) applies frozen-weight interventions to the joint self-prompt state: spatial rows preserve

T_{\mathrm{res}}

and box context, box rows preserve

T_{\mathrm{res}}

and the spatial branch, and mask-derived box replacement substitutes, after adaptation, a box computed from the test reference mask. (b) reports matched training ablations that separately remove the complete response-guidance pipeline and high-resolution prompt refinement.

Figure 5: Qualitative comparison and failures. Columns: input+ground truth (GT), SAM2.1-Large, SAMamba, and SPARK-SAM. Colors: true/false positives and false negatives (TP/FP/FN). Bottom: dim targets, hot clutter, target-like distractors, and diffuse responses.

Explore similar work

IP-SAM: Rethinking Prompt-Conditioned Segmentation for Prompt-Absent Deployment
Mar 28, 2026Huiyao Zhang, Jin Bai, Rui Guo +3Image Segmentation Promptable Image Segmentation
University of Chinese Academy of Sciences · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences
BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation
Mar 11, 2026Prithwijit Chowdhury, Mohit Prabhushankar, Ghassan AlRegibPromptable Image Segmentation Active Learning
OLIVES at the Georgia Institute of Technology
Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM
Apr 25, 2026Jingxuan Kang, Ziqi Zhang, Shaoming Zheng +9Medical Imaging Foundation Models Promptable Image Segmentation
Imperial College London · Beihang University · National Health Service +1

Mar 28, 2026J/K move · Enter open · S save

IP-SAM: Rethinking Prompt-Conditioned Segmentation for Prompt-Absent Deployment

Huiyao Zhang, Jin Bai, Rui Guo, JianWen Tan+2

University of Chinese Academy of Sciences · Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences

Image Segmentation Promptable Image Segmentation Visual Prompt Learning Camouflaged Object Detection

Open paper AskPDF arXiv

Prompt-conditioned foundation segmenters have emerged as a dominant paradigm for image segmentation, where explicit spatial prompts(e.g., points, boxes, masks) guide mask decoding. However, many real-world deployments require fully automatic segmentation, creating a structural mismatch: the decoder expects prompts that are unavailable at inference. Existing adaptations typically modify intermediate features, inadvertently bypassing the model's native prompt interface and weakening prompt-conditioned decoding. We propose IP-SAM, which revisits adaptation from a prompt-space perspective through prompt-space conditioning. Specifically, a Self-Prompt Generator (SPG) distills image context into complementary intrinsic prompts that serve as coarse regional anchors. These cues are projected through SAM2's frozen prompt encoder, restoring prompt-guided decoding without external intervention. To suppress background-induced false positives, Prompt-Space Gating (PSG) leverages the intrinsic background prompt as an asymmetric suppressive constraint prior to decoding. Under a deterministic no-external-prompt protocol, IP-SAM achieves state-of-the-art performance across four camouflaged object detection benchmarks with only 21.26M trainable parameters. Furthermore, the proposed conditioning strategy generalizes beyond COD to medical polyp segmentation.

Abstract

Explore similar work

IP-SAM: Rethinking Prompt-Conditioned Segmentation for Prompt-Absent Deployment

BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation

Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM