Organizations: School of Elite Biomedical Engineers and the Institute for Interdisciplinary Intelligent Pharmacy, China Pharmaceutical University, Nanjing 211198, China · State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210008, China · School of Information Science and Engineering, Yunnan University, Kunming 650091, China · School of Mathematics, Southeast University, Nanjing 211189, China · Purple Mountain Laboratories, Nanjing 211111, China
Vision foundation models have recently advanced multi-modal salient object detection (MSOD) through parameter-efficient tuning and prompt learning. However, existing Segment Anything Model (SAM)-adapted MSOD methods often rely on dual-stream encoders or auxiliary prompt generators, leading to redundant computation. Although a single-stream alternative can reduce this cost, early fusion may also propagate noisy or misaligned auxiliary high-frequency cues through the backbone. In this paper, we propose a novel single-stream framework that integrates reliability-calibrated frequency adaptation into the adopted SAM backbone for MSOD. It avoids duplicated foundation backbones while explicitly controlling auxiliary frequency injection. Specifically, we design a mixture of frequency experts module, which uses the stationary wavelet transform to decompose each modality and aggregate cross-modal frequency information. We further introduce a reliability-calibrated frequency adapter with a dual-gate calibration mechanism, which selectively propagates the calibrated residual across transformer stages while jointly controlling its injection strength and cross-modal reliability. A hypernetwork-guided semantic-structural decoder then combines semantic mask features from the adopted backbone with Mamba-based structural detail recovery. Comprehensive experiments on RGB-D, RGB-T, and RGB-NIR salient object detection benchmarks validate that the proposed framework achieves competitive performance with only 12.20M trainable parameters, accounting for 5.4% of the total parameters. The code will be available at https://github.com/xuboyue1999/SSSAM.
Figures & tables
Fig. 1: Comparison of SAM adaptation paradigms for multi-modal salient object detection. (a) Dual-stream SAM encodes RGB and auxiliary inputs with two separate SAM encoders, resulting in redundant foundation-backbone computation. (b) Prompt-driven SAM converts auxiliary cues into extra prompts through an additional prompt network. (c) Our S 3 AM performs early frequency-aware fusion and adapts a single frozen SAM backbone with reliability-calibrated frequency adapters.
Fig. 2: Framework of the proposed S 3 AM. Given RGB and auxiliary features Frgb and Faux , MoFE performs SWT-based frequency decomposition and cross-modal expert aggregation to obtain Ffused , which is added to Frgb before the first block of the adopted SAM backbone, and the initial candidate frequency prior P0 . At RCFA stage i , the current encoder feature Fi and incoming frequency prior Pi−1 produce a calibrated residual Fires , which is injected to form Fiout and propagated as the calibrated frequency prior Pi . The resulting multi-scale encoder features are decoded by HSSD, which combines the adopted SAM backbone’s hypernetwork-guided semantic prediction with Mamba-based structural detail recovery.
Fig. 3: Motivation for reliability calibration. Raw RGB and depth inputs exhibit complementary yet locally inconsistent high-frequency responses. In the cross-modal relation map, cyan and magenta denote RGB-only and depth-only responses, respectively, and their coexistence reveals local disagreement.
Metric
CAVER [ 21 ]
CPNet [ 10 ]
MAGNet [ 44 ]
KAN-SAM [ 14 ]
BTNet [ 25 ]
LESOD [ 45 ]
Ours
NJU2K
Fβ↑
0.901
0.923
0.912
0.935
0.917
0.898
0.939
M↓
0.031
0.025
0.027
0.022
0.025
0.032
0.019
Em↑
0.923
0.935
0.928
0.942
0.932
0.925
0.939
Sm↑
0.921
0.935
0.929
0.939
0.932
0.922
0.939
NLPR
Fβ↑
0.895
0.918
0.910
0.925
0.917
0.897
0.934
M↓
0.020
0.016
0.017
0.017
0.016
0.020
0.014
TABLE I: Comparison with different methods on RGB-D datasets. Best in bold.
Fig. 4: Precision-recall curves of competitive RGB-D salient object detection methods on the NJU2K, NLPR, SSD, and STERE datasets.
Metric
CMDBIF [ 39 ]
CAVER [ 21 ]
SMR-Net [ 38 ]
LAFB [ 34 ]
UMINet [ 7 ]
LESOD [ 45 ]
Ours
VT5000
Fβ↑
0.846
0.849
0.859
0.841
0.820
0.839
0.898
M↓
0.032
0.028
0.030
0.030
0.035
0.032
0.020
Em↑
0.933
0.935
0.935
0.931
0.915
0.926
0.950
Sm↑
0.886
0.899
0.891
0.893
0.882
0.893
0.921
VT1000
Fβ↑
0.909
0.912
0.899
0.905
0.896
0.897
0.934
M↓
0.019
0.016
0.020
0.018
0.021
0.020
0.013
TABLE II: Comparison with different methods on RGB-T datasets. Best in bold.
Methods
Favg↑
Fmax↑
Fβ↑
M↓
Em↑
Sm↑
RC [ 3 ]
0.664
0.736
0.442
0.148
0.810
0.724
DCL [ 13 ]
0.779
0.838
0.660
0.076
0.881
0.796
SOD8s+ [ 26 ]
0.803
0.850
0.745
0.061
0.894
0.828
Ours
0.928
0.941
0.921
0.020
0.964
0.930
TABLE III: Comparison with representative RGB-NIR methods. Best in bold.
Fig. 5: Qualitative comparison with competitive methods on RGB-D datasets.
Fig. 6: Qualitative comparison with competitive methods on RGB-T datasets.
Fig. 7: Qualitative comparison with representative RGB-NIR methods.
Base
MoFE
RCFA
HSSD
NJU2K
NLPR
Fβ↑
M↓
Em↑
Sm↑
Fβ↑
M↓
Em↑
Sm↑
✓
0.919
0.028
0.928
0.926
0.901
0.020
0.96
0.926
✓
✓
0.931
0.023
0.929
0.935
0.917
0.017
0.967
0.936
✓
✓
✓
0.931
0.021
0.934
0.938
0.918
0.017
0.965
0.936
✓
✓
✓
0.932
0.022
0.937
0.937
0.923
0.016
0.970
0.938
✓
✓
✓
✓
0.939
0.019
0.939
0.939
0.934
0.014
0.974
0.944
TABLE IV: Component analysis on NJU2K and NLPR. Best in bold.
Variant
NJU2K
NLPR
Fβ↑
M↓
Em↑
Sm↑
Fβ↑
M↓
Em↑
Sm↑
w/o Context Gate
0.938
0.020
0.937
0.939
0.932
0.014
0.974
0.943
w/o Reliability Gate
0.938
0.019
0.935
0.939
0.931
0.014
0.973
0.943
w/o Both Gates
0.937
0.019
0.931
0.939
0.931
0.014
0.974
0.943
Full model
0.939
0.019
0.939
0.939
0.934
0.014
0.974
0.944
TABLE V: Dual-gate calibration analysis on NJU2K and NLPR.
Strategy
NJU2K
NLPR
Params. (M)
FLOPs (G)
Fβ↑
M↓
Em↑
Sm↑
Fβ↑
M↓
Em↑
Sm↑
Early
0.931
0.021
0.934
0.938
0.918
0.017
0.965
0.936
223.8
420.1
Middle
0.937
0.020
0.935
0.941
0.923
0.016
0.969
0.940
438.7
816.7
Late
0.937
0.019
0.939
0.941
0.920
0.016
0.966
0.937
438.6
816.7
TABLE VI: Comparison of fusion strategies on NJU2K and NLPR.
Fig. 8: Qualitative ablation results of the proposed components.
Fig. 9: t-SNE visualization of token features from a representative RGB-D sample. Red and green markers denote background and foreground token features, respectively. The proposed components progressively improve inter-class separation and intra-class compactness.
Base
MoFE
RCFA
HSSD
Model Params. (M)
Tunable Params. (M)
FLOPs (G)
FPS
✓
218.89
6.73
412.5
52.4
✓
✓
219.19
7.03
420.6
49.2
✓
✓
✓
224.06
11.90
427.0
47.6
✓
✓
✓
✓
224.36
12.20
428.1
45.8
TABLE VII: Comparison of model complexity and inference efficiency.
The Segment Anything Model 2 (SAM2) has emerged as a foundation model for universal segmentation. Owing to its generalizable visual representations, SAM2 has been successfully applied to various downstream tasks. However, extending SAM2 to the RGB-D video salient object detection (RGB-D VSOD) task encounters three challenges including limited spatial modeling of linear LoRA, insufficient employment of SAM's multi-scale features, and dependence of initialization on explicit prompts. To address the issues, we present Multi-Modal Mixture-of-Experts with Memory-Augmented SAM (M4-SAM), which equips SAM2 with modality-related PEFT, hierarchical feature fusion, and prompt-free memory initialization. Firstly, we inject Modality-Aware MoE-LORA, which employs convolutional experts to encode local spatial priors and introduces a modality dispatcher for efficient multi-modal fine-tuning, into SAM2's encoder. Secondly, we deploy Gated Multi-Level Feature Fusion, which hierarchically aggregates multi-scale encoder features with an adaptive gating mechanism, to balance spatial details and semantic context. Finally, to conduct zero-shot VSOD without manual prompts, we utilize a Pseudo-Guided Initialization, where a coarse mask is regarded as a pseudo prior and used to bootstrap the memory bank. Extensive experiments demonstrate that M4-SAM achieves the state-of-the-art performance across all evaluation metrics on three public RGB-D VSOD datasets.
Jiyuan Liu, Jia Lin, Xiaofei Zhou +3
Hangzhou Dianzi University, China · Shandong University, China · Anqing Normal University, China +1
Salient Object Detection (SOD) remains an essential yet underexplored task in the era of large-scale vision models. Although foundation models like SAM exhibit strong generalization, their potential for SOD is not fully realized, and training or fully fine-tuning them is computationally expensive and prone to overfitting under limited data. To overcome these challenges, we introduce GLASSNet, a Global-Local feature decoding framework that uses SAMv2 as a frozen encoder paired with a lightweight, spatially aware convolutional adapter-reducing learnable encoder parameters by over 97%. To enhance saliency quality, GLASSNet employs a dual-decoder architecture: one decoder captures global, long-range semantics with an expanded receptive field, while the other captures fine local details such as edges and textures. Fusing these complementary cues yields saliency maps that combine global coherence with local precision, producing accurate final masks. Extensive experiments on standard SOD and camouflaged object detection benchmarks show that GLASSNet surpasses state-of-the-art methods, demonstrating the power of frozen foundation models combined with targeted adaptation and global-local decoding.
Morteza Moradi, Mohammad Moradi, Simone Palazzo +2
University of Catania, Catania, Italy · Quintic AI, San Francisco, CA, USA
Co-salient Object Detection (CoSOD) aims to segment salient objects that consistently appear across a group of related images. Despite the notable progress achieved by recent training-based approaches, they still remain constrained by the closed-set datasets and exhibit limited generalization. However, few studies explore the potential of Vision Foundation Models (VFMs) to address CoSOD, which demonstrate a strong generalized ability and robust saliency understanding. In this paper, we investigate and leverage VFMs for CoSOD, and further propose a novel training-free method, TF-SSD, through the synergy between SAM and DINO. Specifically, we first utilize SAM to generate comprehensive raw proposals, which serve as a candidate mask pool. Then, we introduce a quality mask generator to filter out redundant masks, thereby acquiring a refined mask set. Since this generator is built upon SAM, it inherently lacks semantic understanding of saliency. To this end, we adopt an intra-image saliency filter that employs DINO's attention maps to identify visually salient masks within individual images. Moreover, to extend saliency understanding across group images, we propose an inter-image prototype selector, which computes similarity scores among cross-image prototypes to select masks with the highest score. These selected masks serve as final predictions for CoSOD. Extensive experiments show that our TF-SSD outperforms existing methods (e.g., 13.7% gains over the recent training-free method). Codes are available at https://github.com/hzz-yy/TF-SSD.
Zhijin He, Shuo Jin, Siyue Yu +4
XJTLU · University of Liverpool · China University of Petroleum (East China) +1