Modality-Agnostic Prompt Learning for Multi-Modal Camouflaged Object Detection

Hao Wang; Jiqing Zhang; Xin Yang; Baocai Yin; Lu Jiang; Zetian Mi; Huibing Wang

cs.CVApr 14, 2026

Modality-Agnostic Prompt Learning for Multi-Modal Camouflaged Object Detection

Authors: Hao Wang, Jiqing Zhang, Xin Yang, Baocai Yin, Lu Jiang, Zetian Mi, Huibing Wang

Organizations: Information Science and Technology College, Dalian Maritime University, Liaoning, 116026, China · Dalian University of Technology, Liaoning, 116024, China · Beijing University of Technology, Beijing, 100124, China

Abstract

Camouflaged Object Detection (COD) aims to segment objects that blend seamlessly into complex backgrounds, with growing interest in exploiting additional visual modalities to enhance robustness through complementary information. However, most existing approaches generally rely on modality-specific architectures or customized fusion strategies, which limit scalability and cross-modal generalization. To address this, we propose a novel framework that generates modality-agnostic multi-modal prompts for the Segment Anything Model (SAM), enabling parameter-efficient adaptation to arbitrary auxiliary modalities and significantly improving overall performance on COD tasks. Specifically, we model multi-modal learning through interactions between a data-driven content domain and a knowledge-driven prompt domain, distilling task-relevant cues into unified prompts for SAM decoding. We further introduce a lightweight Mask Refine Module to calibrate coarse predictions by incorporating fine-grained prompt cues, leading to more accurate camouflaged object boundaries. Extensive experiments on RGB-Depth, RGB-Thermal, and RGB-Polarization benchmarks validate the effectiveness and generalization of our modality-agnostic framework.

Figures & tables

Fig. 1: The

F_{\beta}^{w}

scores with respect to the trainable parameters of different methods on COD10K (RGB-Depth), PCOD-1200 (RGB-Polarization), and VIAC (RGB-Thermal) datasets.

Panel of Figure 2 — Fig. 2: Overview of our proposed framework. The framework adopts a dual-stream architecture where RGB and auxiliary modalities are processed via a frozen SAM image encoder and a PVT encoder, respectively, to construct the content domain through feature summation. In Content-Prompt Interaction Module (CPIM), a set of learnable parameters serves as the prompt domain prototype, interacting with the content domain to achieve synergy between data-driven and knowledge-driven representations. Finally, the enhanced content features, along with learnable auxi-tokens and SAM’s original tokens, are fed into the mask decoder, followed by a Mask Refine Module (MRM) to produce the final precise segmentation.

Methods	Pub. Year	CAMO				COD10K				NC4K				CHAMELEON
		$S_{\alpha}\uparrow$	$F_{\beta}^{w}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$	$S_{\alpha}\uparrow$	$F_{\beta}^{w}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$	$S_{\alpha}\uparrow$	$F_{\beta}^{w}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$	$S_{\alpha}\uparrow$	$F_{\beta}^{w}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$
RGB-based Methods
SINet Fan et al. [2020]	CVPR’20	0.751	0.606	0.771	0.100	0.771	0.551	0.806	0.051	0.808	0.723	0.883	0.058	0.869	0.740	0.891	0.044
PFNet Mei et al. [2021]	CVPR’21	0.782	0.695	0.845	0.085	0.800	0.660	0.880	0.040	0.829	0.745	0.891	0.053	0.882	0.810	0.927	0.053
FSPNet Huang et al. [2023]	CVPR’23	0.856	0.799	0.899	0.050	0.851	0.735	0.895	0.026	0.879	0.816	0.915	0.035	-	-	-	-
FGANet Zhai et al. [2022]	NeurIPS’23	0.803	0.748	0.871	0.068	0.821	0.678	0.895	0.031	-	-	-	-	0.902	0.840	0.947	0.030

TABLE I: Comparison of SOTA methods on multiple RGB-Depth datasets in terms of

S_{\alpha}

F_{\beta}^{w}

E_{\phi}

, and

M

. The best results are highlighted in bold . * represents the SAM-based methods.

Panel of Figure 4 — Fig. 3: Qualitative comparison of detection results on RGB-D and RGB-P camouflaged object detection datasets.

Methods	Pub.’Year	$S_{\alpha}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$	$F_{\beta}^{w}\uparrow$
RGB-based Methods
PFNet Mei et al. [2021]	CVPR’21	0.876	0.946	0.013	0.788
SINet-V2 Fan et al. [2021]	TPAMI’22	0.882	0.941	0.013	-
ZoomNet Pang et al. [2022]	CVPR’22	0.897	0.922	0.010	-
BSANet Zhu et al. [2022]	AAAI’22	0.903	0.945	0.011	-
HQSAM* Ke et al. [2023]	NeurIPS’23	0.936	0.973	0.008	0.909

TABLE II: Quantitative comparisons with SOTAs on PCOD-1200. The best results are highlighted in bold . * represents the SAM-based methods.

Methods	Pub.’Year	$S_{\alpha}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$	$F_{\beta}^{w}\uparrow$
RGB-based Methods
HQSAM* Ke et al. [2023]	NeurIPS’23	0.929	0.981	0.011	0.925
ZoomNetxt Pang et al. [2024]	TPAMI’24	0.934	0.981	0.011	0.920
UTNet Sun et al. [2025]	TMM’25	0.903	0.969	0.016	0.876
PlantCamo Yang et al. [2024]	AIR’25	0.926	0.982	0.010	0.918
ESCNet Ye et al. [2025]	ICCV’25	0.928	0.979	0.011	0.919

TABLE III: Quantitative comparisons with SOTAs on the VIAC (RGB-T) dataset. The best results are highlighted in bold . * represents the SAM-based methods.

Setting	#	Method	RGB-D (COD10K)				RGB-P (PCOD-1200)				RGB-T (VIAC)
Setting	#	Method	$S_{\alpha}\uparrow$	$F_{\beta}^{w}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$	$S_{\alpha}\uparrow$	$F_{\beta}^{w}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$	$S_{\alpha}\uparrow$	$F_{\beta}^{w}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$
Standard Train	A	w/o WF	0.897	0.860	0.958	0.019	0.939	0.916	0.979	0.006	0.928	0.926	0.984	0.010
	B	w/o CPIM	0.887	0.843	0.950	0.023	0.934	0.910	0.974	0.007	0.921	0.917	0.981	0.012
	C	CPIM w/o CA	0.887	0.845	0.953	0.021	0.935	0.909	0.914	0.006	0.919	0.911	0.979	0.012
	D	w/o MRM	0.895	0.855	0.958	0.020	0.944	0.923	0.981	0.006	0.931	0.929	0.984	0.010
	E	w/o Auxi-Tokens	0.891	0.846	0.952	0.022	0.943	0.921	0.982	0.006	0.931	0.928	0.983	0.010

TABLE IV: Unified ablation study under standard and joint training settings.

Panel of Figure 8 — Fig. 4: Qualitative comparison of detection results on the VIAC (RGB-T) dataset.

Panel of Figure 9 — Fig. 5: Visual ablation comparison of different variants.

Panel of Figure 10 — Fig. 6: Attentional visualization of the transposed convolution branch within the SAM mask decoder.

Method	Params (M) $\downarrow$	$F_{\beta}^{w}\uparrow$
Method	Params (M) $\downarrow$	COD10K	PCOD-1200	VIAC
HQSAM	5.10	0.846	0.909	0.925
MM-SAM	93.73	0.808	0.801	0.906
COMPrompter	94.85	0.821	0.795	0.894
DSAM	121.93	0.760	0.839	0.440
SAM2-UNet	4.38	0.798	0.871	0.915

TABLE V: Comparison of trainable parameters and performance across three multi-modal COD benchmarks.

Methods	VT5000				VT1000				VT821
	$S_{\alpha}\uparrow$	$F_{\beta}^{w}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$	$S_{\alpha}\uparrow$	$F_{\beta}^{w}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$	$S_{\alpha}\uparrow$	$F_{\beta}^{w}\uparrow$	$E_{\phi}\uparrow$	$M\downarrow$
HRTTransNet Tang et al. [2022]	0.945	0.912	0.870	0.025	0.945	0.938	0.913	0.017	0.929	0.906	0.849	0.026
WaveNet Zhou et al. [2023]	0.940	0.911	0.864	0.026	0.952	0.945	0.921	0.015	0.929	0.912	0.863	0.024
SACNet Wang et al. [2024a]	0.933	0.892	0.838	0.030	0.949	0.932	0.907	0.018	0.917	0.883	0.817	0.033
VSCode-v2 Luo et al. [2024]	0.931	-	0.962	-	0.952	-	0.984	-	0.932	-	0.958	-
UniSOD Wang et al. [2025b]	0.919	0.891	0.955	0.021	0.946	0.931	0.956	0.013	0.916	0.876	0.937	0.024

TABLE VI: Comparison of SOTA methods on multiple RGB-T SOD datasets. The best results are highlighted in bold .

Panel of Figure 13 — Fig. 7: Qualitative comparison of detection results on RGB-T SOD datasets.

Train Modality	Test Modality	$S_{\alpha}\uparrow$	$E_{\phi}\uparrow$	$F_{\beta}^{w}\uparrow$	$M\downarrow$
RGB+D	RGB+P	0.924 (0.945)	0.968 (0.987)	0.890 (0.924)	0.010 (0.005)
RGB+D	RGB+T	0.916 (0.930)	0.976 (0.983)	0.908 (0.929)	0.013 (0.010)
RGB+DT	RGB+P	0.923 (0.945)	0.975 (0.987)	0.889 (0.924)	0.009 (0.005)
RGB+DP	RGB+T	0.920 (0.930)	0.978 (0.983)	0.915 (0.929)	0.012 (0.010)

TABLE VII: Quantitative comparison of cross-modality transfer experiments (parentheses denote matched-modality settings).

Panel of Figure 15 — Fig. 8: Comparison under different challenging scenarios.

Panel of Figure 16 — Fig. 9: The case of failure.

Explore similar work

Language-Guided Structure-Aware Network for Camouflaged Object Detection
Mar 25, 2026Min ZhangCamouflaged Object Detection Boundary-Aware Segmentation
School of Artificial Intelligence, Chongqing University of Technology, Chongqing, 401120, China
Phantom-Insight: Adaptive Multi-cue Fusion for Video Camouflaged Object Detection with Multimodal LLM
Sep 8, 2025Hua Zhang, Changjiang LuoVideo Object Segmentation Vision Foundation Model Adaptation
Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences
VCP-DCN: Beyond Visual Concealed Property via Depth Collaborative Network for Camouflaged Object Detection
Jul 30, 2026Songsong Duan, Xi Yang, Nannan WangCross-Modal Alignment Multimodal Fusion
the State Key Laboratory of Integrated Services Networks, School of Telecommunications Engineering, Xidian University, Xi’an, China

Mar 25, 2026J/K move · Enter open · S save

Language-Guided Structure-Aware Network for Camouflaged Object Detection

Min Zhang

School of Artificial Intelligence, Chongqing University of Technology, Chongqing, 401120, China

Camouflaged Object Detection Boundary-Aware Segmentation

Open paper AskPDF arXiv

Camouflaged Object Detection (COD) aims to segment objects that are highly integrated with the background in terms of color, texture, and structure, making it a highly challenging task in computer vision. Although existing methods introduce multi-scale fusion and attention mechanisms to alleviate the above issues, they generally lack the guidance of textual semantic priors, which limits the model's ability to focus on camouflaged regions in complex scenes. To address this issue, this paper proposes a Language-Guided Structure-Aware Network (LGSAN). Specifically, based on the visual backbone PVT-v2, we introduce CLIP to generate masks from text prompts and RGB images, thereby guiding the multi-scale features extracted by PVT-v2 to focus on potential target regions. On this foundation, we further design a Fourier Edge Enhancement Module (FEEM), which integrates multi-scale features with high-frequency information in the frequency domain to extract edge enhancement features. Furthermore, we propose a Structure-Aware Attention Module (SAAM) to effectively enhance the model's perception of object structures and boundaries. Finally, we introduce a Coarse-Guided Local Refinement Module (CGLRM) to enhance fine-grained reconstruction and boundary integrity of camouflaged object regions. Extensive experiments demonstrate that our method consistently achieves highly competitive performance across multiple COD datasets, validating its effectiveness and robustness.

Abstract

Explore similar work

Language-Guided Structure-Aware Network for Camouflaged Object Detection

Phantom-Insight: Adaptive Multi-cue Fusion for Video Camouflaged Object Detection with Multimodal LLM

VCP-DCN: Beyond Visual Concealed Property via Depth Collaborative Network for Camouflaged Object Detection