S$^3$AM: A Single-Stream SAM with Reliability-Calibrated Frequency Adapter for Multi-modal Salient Object Detection

Ruichao Hou; Boyue Xu; Tongwei Ren; Dongming Zhou; Gangshan Wu; Jinde Cao

cs.CVAug 18, 2026

S $^3$ AM: A Single-Stream SAM with Reliability-Calibrated Frequency Adapter for Multi-modal Salient Object Detection

Authors: Ruichao Hou, Boyue Xu, Tongwei Ren, Dongming Zhou, Gangshan Wu, Jinde Cao

Organizations: School of Elite Biomedical Engineers and the Institute for Interdisciplinary Intelligent Pharmacy, China Pharmaceutical University, Nanjing 211198, China · State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210008, China · School of Information Science and Engineering, Yunnan University, Kunming 650091, China · School of Mathematics, Southeast University, Nanjing 211189, China ·

Abstract

Vision foundation models have recently advanced multi-modal salient object detection (MSOD) through parameter-efficient tuning and prompt learning. However, existing Segment Anything Model (SAM)-adapted MSOD methods often rely on dual-stream encoders or auxiliary prompt generators, leading to redundant computation. Although a single-stream alternative can reduce this cost, early fusion may also propagate noisy or misaligned auxiliary high-frequency cues through the backbone. In this paper, we propose a novel single-stream framework that integrates reliability-calibrated frequency adaptation into the adopted SAM backbone for MSOD. It avoids duplicated foundation backbones while explicitly controlling auxiliary frequency injection. Specifically, we design a mixture of frequency experts module, which uses the stationary wavelet transform to decompose each modality and aggregate cross-modal frequency information. We further introduce a reliability-calibrated frequency adapter with a dual-gate calibration mechanism, which selectively propagates the calibrated residual across transformer stages while jointly controlling its injection strength and cross-modal reliability. A hypernetwork-guided semantic-structural decoder then combines semantic mask features from the adopted backbone with Mamba-based structural detail recovery. Comprehensive experiments on RGB-D, RGB-T, and RGB-NIR salient object detection benchmarks validate that the proposed framework achieves competitive performance with only 12.20M trainable parameters, accounting for 5.4% of the total parameters. The code will be available at https://github.com/xuboyue1999/SSSAM.

Figures & tables

Fig. 1: Comparison of SAM adaptation paradigms for multi-modal salient object detection. (a) Dual-stream SAM encodes RGB and auxiliary inputs with two separate SAM encoders, resulting in redundant foundation-backbone computation. (b) Prompt-driven SAM converts auxiliary cues into extra prompts through an additional prompt network. (c) Our S 3 AM performs early frequency-aware fusion and adapts a single frozen SAM backbone with reliability-calibrated frequency adapters.

Fig. 2: Framework of the proposed S 3 AM. Given RGB and auxiliary features $\mathcal{F}^{\mathrm{rgb}}$ and $\mathcal{F}^{\mathrm{aux}}$ , MoFE performs SWT-based frequency decomposition and cross-modal expert aggregation to obtain $\mathcal{F}^{\mathrm{fused}}$ , which is added to $\mathcal{F}^{\mathrm{rgb}}$ before the first block of the adopted SAM backbone, and the initial candidate frequency prior $\mathcal{P}_{0}$ . At RCFA stage $i$ , the current encoder feature $\mathcal{F}_{i}$ and incoming frequency prior $\mathcal{P}_{i-1}$ produce a calibrated residual $\mathcal{F}^{\mathrm{res}}_{i}$ , which is injected to form $\mathcal{F}^{\mathrm{out}}_{i}$ and propagated as the calibrated frequency prior $\mathcal{P}_{i}$ . The resulting multi-scale encoder features are decoded by HSSD, which combines the adopted SAM backbone’s hypernetwork-guided semantic prediction with Mamba-based structural detail recovery.

Fig. 3: Motivation for reliability calibration. Raw RGB and depth inputs exhibit complementary yet locally inconsistent high-frequency responses. In the cross-modal relation map, cyan and magenta denote RGB-only and depth-only responses, respectively, and their coexistence reveals local disagreement.

	Metric	CAVER [ 21 ]	CPNet [ 10 ]	MAGNet [ 44 ]	KAN-SAM [ 14 ]	BTNet [ 25 ]	LESOD [ 45 ]	Ours
NJU2K	$F_{\beta}\uparrow$	0.901	0.923	0.912	0.935	0.917	0.898	0.939
	$\mathcal{M}\downarrow$	0.031	0.025	0.027	0.022	0.025	0.032	0.019
	$E_{m}\uparrow$	0.923	0.935	0.928	0.942	0.932	0.925	0.939
	$S_{m}\uparrow$	0.921	0.935	0.929	0.939	0.932	0.922	0.939
NLPR	$F_{\beta}\uparrow$	0.895	0.918	0.910	0.925	0.917	0.897	0.934
	$\mathcal{M}\downarrow$	0.020	0.016	0.017	0.017	0.016	0.020	0.014

TABLE I: Comparison with different methods on RGB-D datasets. Best in bold.

Fig. 4: Precision-recall curves of competitive RGB-D salient object detection methods on the NJU2K, NLPR, SSD, and STERE datasets.

	Metric	CMDBIF [ 39 ]	CAVER [ 21 ]	SMR-Net [ 38 ]	LAFB [ 34 ]	UMINet [ 7 ]	LESOD [ 45 ]	Ours
VT5000	$F_{\beta}\uparrow$	0.846	0.849	0.859	0.841	0.820	0.839	0.898
	$\mathcal{M}\downarrow$	0.032	0.028	0.030	0.030	0.035	0.032	0.020
	$E_{m}\uparrow$	0.933	0.935	0.935	0.931	0.915	0.926	0.950
	$S_{m}\uparrow$	0.886	0.899	0.891	0.893	0.882	0.893	0.921
VT1000	$F_{\beta}\uparrow$	0.909	0.912	0.899	0.905	0.896	0.897	0.934
	$\mathcal{M}\downarrow$	0.019	0.016	0.020	0.018	0.021	0.020	0.013

TABLE II: Comparison with different methods on RGB-T datasets. Best in bold.

Methods	$F_{\mathrm{avg}}\uparrow$	$F_{\max}\uparrow$	$F_{\beta}\uparrow$	$\mathcal{M}\downarrow$	$E_{m}\uparrow$	$S_{m}\uparrow$
RC [ 3 ]	0.664	0.736	0.442	0.148	0.810	0.724
DCL [ 13 ]	0.779	0.838	0.660	0.076	0.881	0.796
SOD8s+ [ 26 ]	0.803	0.850	0.745	0.061	0.894	0.828
Ours	0.928	0.941	0.921	0.020	0.964	0.930

TABLE III: Comparison with representative RGB-NIR methods. Best in bold.

Fig. 5: Qualitative comparison with competitive methods on RGB-D datasets.

Fig. 6: Qualitative comparison with competitive methods on RGB-T datasets.

Fig. 7: Qualitative comparison with representative RGB-NIR methods.

Base	MoFE	RCFA	HSSD	NJU2K				NLPR
Base	MoFE	RCFA	HSSD	$F_{\beta}\uparrow$	$\mathcal{M}\downarrow$	$E_{m}\uparrow$	$S_{m}\uparrow$	$F_{\beta}\uparrow$	$\mathcal{M}\downarrow$	$E_{m}\uparrow$	$S_{m}\uparrow$
✓				0.919	0.028	0.928	0.926	0.901	0.020	0.96	0.926
✓	✓			0.931	0.023	0.929	0.935	0.917	0.017	0.967	0.936
✓		✓	✓	0.931	0.021	0.934	0.938	0.918	0.017	0.965	0.936
✓	✓	✓		0.932	0.022	0.937	0.937	0.923	0.016	0.970	0.938
✓	✓	✓	✓	0.939	0.019	0.939	0.939	0.934	0.014	0.974	0.944

TABLE IV: Component analysis on NJU2K and NLPR. Best in bold.

Variant	NJU2K				NLPR
Variant	$F_{\beta}\uparrow$	$\mathcal{M}\downarrow$	$E_{m}\uparrow$	$S_{m}\uparrow$	$F_{\beta}\uparrow$	$\mathcal{M}\downarrow$	$E_{m}\uparrow$	$S_{m}\uparrow$
w/o Context Gate	0.938	0.020	0.937	0.939	0.932	0.014	0.974	0.943
w/o Reliability Gate	0.938	0.019	0.935	0.939	0.931	0.014	0.973	0.943
w/o Both Gates	0.937	0.019	0.931	0.939	0.931	0.014	0.974	0.943
Full model	0.939	0.019	0.939	0.939	0.934	0.014	0.974	0.944

TABLE V: Dual-gate calibration analysis on NJU2K and NLPR.

Strategy	NJU2K				NLPR				Params. (M)	FLOPs (G)
Strategy	$F_{\beta}\uparrow$	$\mathcal{M}\downarrow$	$E_{m}\uparrow$	$S_{m}\uparrow$	$F_{\beta}\uparrow$	$\mathcal{M}\downarrow$	$E_{m}\uparrow$	$S_{m}\uparrow$	Params. (M)	FLOPs (G)
Early	0.931	0.021	0.934	0.938	0.918	0.017	0.965	0.936	223.8	420.1
Middle	0.937	0.020	0.935	0.941	0.923	0.016	0.969	0.940	438.7	816.7
Late	0.937	0.019	0.939	0.941	0.920	0.016	0.966	0.937	438.6	816.7

TABLE VI: Comparison of fusion strategies on NJU2K and NLPR.

Fig. 8: Qualitative ablation results of the proposed components.

Panel of Figure 15 — Fig. 9: t-SNE visualization of token features from a representative RGB-D sample. Red and green markers denote background and foreground token features, respectively. The proposed components progressively improve inter-class separation and intra-class compactness.

Base	MoFE	RCFA	HSSD	Model Params. (M)	Tunable Params. (M)	FLOPs (G)	FPS
✓				218.89	6.73	412.5	52.4
✓	✓			219.19	7.03	420.6	49.2
✓	✓	✓		224.06	11.90	427.0	47.6
✓	✓	✓	✓	224.36	12.20	428.1	45.8

TABLE VII: Comparison of model complexity and inference efficiency.

Salient Object Detection

Salient Object Detection finds visually prominent objects in images or video and predicts their pixel-level foreground masks.

See all 37 papers

M $^4$ -SAM: Multi-Modal Mixture-of-Experts with Memory-Augmented SAM for RGB-D Video Salient Object Detection
May 12, 2026Jiyuan Liu, Jia Lin, Xiaofei Zhou +3Video Object Segmentation Vision Foundation Models
Hangzhou Dianzi University, China · Shandong University, China · Anqing Normal University, China +1
Global-Local Feature Decoding with Adapter-Guided SAMv2 for Salient Object Detection
May 4, 2026Morteza Moradi, Mohammad Moradi, Simone Palazzo +2Salient Object Detection Vision Foundation Model Adaptation
University of Catania, Catania, Italy · Quintic AI, San Francisco, CA, USA
TF-SSD: A Strong Pipeline via Synergic Mask Filter for Training-free Co-salient Object Detection
Apr 1, 2026Zhijin He, Shuo Jin, Siyue Yu +4Salient Object Detection Object Detection
XJTLU · University of Liverpool · China University of Petroleum (East China) +1

May 12, 2026J/K move · Enter open · S save

M $^4$ -SAM: Multi-Modal Mixture-of-Experts with Memory-Augmented SAM for RGB-D Video Salient Object Detection

Jiyuan Liu, Jia Lin, Xiaofei Zhou, Runmin Cong+2

Hangzhou Dianzi University, China · Shandong University, China · Anqing Normal University, China+1

Video Object Segmentation Vision Foundation Models Mixture of Experts Salient Object Detection Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

The Segment Anything Model 2 (SAM2) has emerged as a foundation model for universal segmentation. Owing to its generalizable visual representations, SAM2 has been successfully applied to various downstream tasks. However, extending SAM2 to the RGB-D video salient object detection (RGB-D VSOD) task encounters three challenges including limited spatial modeling of linear LoRA, insufficient employment of SAM's multi-scale features, and dependence of initialization on explicit prompts. To address the issues, we present Multi-Modal Mixture-of-Experts with Memory-Augmented SAM (M

^4

-SAM), which equips SAM2 with modality-related PEFT, hierarchical feature fusion, and prompt-free memory initialization. Firstly, we inject Modality-Aware MoE-LORA, which employs convolutional experts to encode local spatial priors and introduces a modality dispatcher for efficient multi-modal fine-tuning, into SAM2's encoder. Secondly, we deploy Gated Multi-Level Feature Fusion, which hierarchically aggregates multi-scale encoder features with an adaptive gating mechanism, to balance spatial details and semantic context. Finally, to conduct zero-shot VSOD without manual prompts, we utilize a Pseudo-Guided Initialization, where a coarse mask is regarded as a pseudo prior and used to bootstrap the memory bank. Extensive experiments demonstrate that M

^4

-SAM achieves the state-of-the-art performance across all evaluation metrics on three public RGB-D VSOD datasets.

Abstract

Salient Object Detection

Explore similar work

M4^44-SAM: Multi-Modal Mixture-of-Experts with Memory-Augmented SAM for RGB-D Video Salient Object Detection

Global-Local Feature Decoding with Adapter-Guided SAMv2 for Salient Object Detection

TF-SSD: A Strong Pipeline via Synergic Mask Filter for Training-free Co-salient Object Detection

M $^4$ -SAM: Multi-Modal Mixture-of-Experts with Memory-Augmented SAM for RGB-D Video Salient Object Detection