Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking

Yabin Zhu; Jianqi Li; Chenglong Li; Jiaxiang Wang; Chengjie Gu; Jin Tang

cs.CVMar 14, 2026

Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking

Authors: Yabin Zhu, Jianqi Li, Chenglong Li, Jiaxiang Wang, Chengjie Gu, Jin Tang

Organizations: School of Public Security and Emergency Management, Anhui University of Science and Technology, Hefei 231131, China. · Information Materials and Intelligent Sensing Laboratory of Anhui Province, Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Artificial Intelligence, Anhui University, Hefei 230601, China. · School of Artificial Intelligence, Anhui University of Science and Technology, Hefei 231131, China. · Information Materials and Intelligent Sensing Laboratory of Anhui Province, Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei 230601, China.

Abstract

Parameter-efficient fine-tuning (PEFT) techniques, such as prompts and adapters, are widely used in multi-modal tracking because they alleviate issues of full-model fine-tuning, including time inefficiency, high resource consumption, parameter storage burden, and catastrophic forgetting. However, due to cross-modal heterogeneity, most existing PEFT-based methods struggle to effectively represent multi-modal features within a unified framework with shared parameters. To address this problem, we propose a novel Sparse-Dense Mixture of Experts Adapter (SDMoEA) framework for PEFT-based multi-modal tracking under a unified model structure. Specifically, we design an SDMoE module as the multi-modal adapter to model modality-specific and shared information efficiently. SDMoE consists of a sparse MoE and a dense-shared MoE: the former captures modality-specific information, while the latter models shared cross-modal information. Furthermore, to overcome limitations of existing tracking methods in modeling high-order correlations during multi-level multi-modal fusion, we introduce a Gram-based Semantic Alignment Hypergraph Fusion (GSAHF) module. It first employs Gram matrices for cross-modal semantic alignment, ensuring that the constructed hypergraph accurately reflects semantic similarity and high-order dependencies between modalities. The aligned features are then integrated into the hypergraph structure to exploit its ability to model high-order relationships, enabling deep fusion of multi-level multi-modal information. Extensive experiments demonstrate that the proposed method achieves superior performance compared with other PEFT approaches on several multi-modal tracking benchmarks, including LasHeR, RGBT234, VTUAV, VisEvent, COESOT, DepthTrack, and VOT-RGBD2022.

Figures & tables

Fig. 1: Comparison with different mixture of experts structures.

Fig. 2: Overview architecture of our proposed method. Here, we embed the Spare-Dense Mixture of Experts (SDMoE) module as the multi-modal adapter to effectively model modality-specific and shared information of fused modalities. Moreover, we also embed the Gram-based Semantic Alignment Hypergraph Fusion (GSAHF) module to address the modeling of higher-order correlations in multi-level and multi-modal feature fusion, achieving more robust feature representations.

Fig. 3: Detailed design of the proposed SDMoE module. The SDMoE consists of a spare MoE and a dense-shared MoE. Spare MoE has a router and

N

specific experts. The details of specific expert is shown on the right of the figure. Correspondingly, details about dense-shared MoE are shown on the left of the figure. Specific-E and FFN-E denote specific expert and the feedforward expert network of dense-shared MoE, respectively. The SiLU is a activation function, called Sigmoid Linear Unit.

Fig. 4: The detailed design of the proposed GSAHF module. The $\mathbf{LW}$ denotes learnable weights.

Method Type		Method	Source	LasHeR		RGBT234
Method Type		Method	Source	PR	AUC	MPR	MSR
Multi-modal Trackers	PEFT	SDMoEA-L	Ours	77.7	60.9	92.6	68.3
		SDMoEA-B	Ours	76.5	60.3	92.2	68.7
		SeqTrackv2-L [ 11 ]	arXiv’23	76.7	61.0	91.3	68.0
		SeqTrackv2-B [ 11 ]	arXiv’23	71.5	56.2	90.0	66.3
		XTrack-L [ 12 ]	ICCV’25	73.1	58.7	87.8	65.4

TABLE I: Overall performance on LasHeR and RGBT234. Results are reported in percentage (%). PEFT and FFT represent parameter-efficient fine-tuning and full fine-tuning, respectively. The

*

indicates that our method performs full fine-tuning.

Method Type		Method	Source	VisEvent
Method Type		Method	Source	PR	AUC
Multi-modal Trackers	PEFT	SDMoEA-L	Ours	81.3	61.9
		SDMoEA-B	Ours	80.3	61.2
		SeqTrackv2-L	arXiv’23	80.0	63.4
		SeqTrackv2-B	arXiv’23	79.3	62.2
		XTrack-L	ICCV’25	80.5	63.3

TABLE II: Overall performance on VisEvent. Results are reported in percentage (%). PEFT and FFT represent parameter-efficient fine-tuning and full fine-tuning, respectively. The

*

indicates that our method performs full fine-tuning.

Method	Source	COESOT
Method	Source	PR	AUC
SDMoEA*-L	Ours	85.1	70.3
SDMoEA*-B	Ours	84.4	69.3
SDMoEA-L	Ours	83.2	68.8
SDMoEA-B	Ours	83.6	68.4
DS-MESA [ 55 ]	PRCV’24	77.5	69.1

TABLE III: Overall performance on COESOT. Results are reported in percentage (%). The

*

indicates that our method performs full fine-tuning.

Method Type		Method	Source	VOT-RGBD2022			DepthTrack
Method Type		Method	Source	EAO	Acc.	Rob.	F-score	Re	Pr
Multi-modal Trackers	PEFT	SDMoEA-L	Ours	78.4	83.0	94.2	69.1	68.4	68.8
		SDMoEA-B	Ours	78.3	81.9	94.9	64.7	65.2	65.0
		XTrack-L [ 12 ]	ICCV’25	74.0	82.8	88.9	64.8	64.3	65.4
		XTrack-B [ 12 ]	ICCV’25	74.0	82.1	88.8	61.5	62.0	61.8
		SeqTrackv2-L [ 11 ]	arXiv’23	74.8	82.6	91.0	62.3	62.6	62.5

TABLE IV: Overall performance on VOT-RGBD2022 and DepthTrack. Results are reported in percentage (%). PEFT and FFT represent parameter-efficient fine-tuning and full fine-tuning, respectively. The

*

indicates that our method performs full fine-tuning.

Method	SDMoE	MFF	GSAHF-MHG	GSAHF-Gram	Params	GFLOPs	LasHeR		VisEvent
							PR	SR	PR	SR
I					97.43 M	170.17	60.9	47.9	66.9	45.0
II	✓				105.88 M	173.15	74.3	59.0	79.3	60.4
III		✓			100.97 M	174.27	65.6	51.3	77.4	57.6
IV			✓		102.15 M	175.63	68.0	53.8	78.5	59.0
V				✓	101.56 M	176.99	70.8	56.1	78.9	59.1

TABLE V: Ablation studies of different components of the proposed method on the testing sets of LasHeR and VisEvent. Results are reported in percentage (%).

Method	LasHeR			VisEvent
Method	PR	NPR	SR	PR	NPR	SR
SDMoEA-odd	75.3	71.3	59.3	80.1	75.7	60.9
SDMoEA-even	71.7	68.2	56.7	80.4	75.1	60.8
SDMoEA	76.5	72.6	60.3	80.3	75.5	61.2

TABLE VI: Performance comparison of embedding SDMOE in odd and even layers of the backbone network on LasHeR and VisEvent. Results are reported in percentage (%).

Method	LasHeR			VisEvent
Method	PR	NPR	SR	PR	NPR	SR
SDMoEA-spare	74.0	70.4	58.5	79.2	74.9	60.4
SDMoEA-dense	74.9	71.5	59.1	80.5	75.9	61.2
SDMoEA	76.5	72.6	60.3	80.3	75.5	61.2

TABLE VII: Performance comparison of sparse MoE and dense-shared of SDMoE on LasHeR and VisEvent. Results are reported in percentage (%).

Fig. 5: Visualization results on multi-modal tracking datasets.

Fig. 6: Visualization of feature maps. The first to sixth columns are the RGB input, Thermal or Event input, the feature map of our method, class activation mappingand of our method, the feature map of baseline method, class activation mapping of baseline method, respectively.

Multimodal Visual Object Tracking

See all 19 papers

Apr 14, 2026

SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker

Parameter-efficient fine-tuning (PEFT) in multimodal tracking reveals a concerning trend where recent performance gains are often achieved at the cost of inflated parameter budgets, which fundamentally erodes PEFT's efficiency promise. In this work, we introduce SEATrack, a Simple, Efficient, and Adaptive two-stream multimodal tracker that tackles this performance-efficiency dilemma from two complementary perspectives. We first prioritize cross-modal alignment of matching responses, an underexplored yet pivotal factor that we argue is essential for breaking the trade-off. Specifically, we observe that modality-specific biases in existing two-stream methods generate conflicting matching attention maps, thereby hindering effective joint representation learning. To mitigate this, we propose AMG-LoRA, which seamlessly integrates Low-Rank Adaptation (LoRA) for domain adaptation with Adaptive Mutual Guidance (AMG) to dynamically refine and align attention maps across modalities. We then depart from conventional local fusion approaches by introducing a Hierarchical Mixture of Experts (HMoE) that enables efficient global relation modeling, effectively balancing expressiveness and computational efficiency in cross-modal fusion. Equipped with these innovations, SEATrack advances notable progress over state-of-the-art methods in balancing performance with efficiency across RGB-T, RGB-D, and RGB-E tracking tasks. cyan{Code is available}.

Junbin Su, Ziteng Xue, Shihui Zhang +3

School of Artificial Intelligence (School of Software), Yanshan University · School of Software, Beihang University · Hebei Key Laboratory of Computer Virtual Technology and System Integration +3

Multimodal Visual Object Tracking Efficient Multimodal Inference Multimodal Fusion Cross-Modal Alignment Parameter-Efficient Fine-Tuning

Ask PDF

Mar 22, 2026

Learning Progressive Adaptation for Multi-Modal Tracking

Due to the limited availability of paired multi-modal data, multi-modal trackers are typically built by adopting pre-trained RGB models with parameter-efficient fine-tuning modules. However, these fine-tuning methods overlook advanced adaptations for applying RGB pre-trained models and fail to modulate a single specific modality, cross-modal interactions, and the prediction head. To address the issues, we propose to perform Progressive Adaptation for Multi-Modal Tracking (PATrack). This innovative approach incorporates modality-dependent, modality-entangled, and task-level adapters, effectively bridging the gap in adapting RGB pre-trained networks to multi-modal data through a progressive strategy. Specifically, modality-specific information is enhanced through the modality-dependent adapter, decomposing the high- and low-frequency components, which ensures a more robust feature representation within each modality. The inter-modal interactions are introduced in the modality-entangled adapter, which implements a cross-attention operation guided by inter-modal shared information, ensuring the reliability of features conveyed between modalities. Additionally, recognising that the strong inductive bias of the prediction head does not adapt to the fused information, a task-level adapter specific to the prediction head is introduced. In summary, our design integrates intra-modal, inter-modal, and task-level adapters into a unified framework. Extensive experiments on RGB+Thermal, RGB+Depth, and RGB+Event tracking tasks demonstrate that our method shows impressive performance against state-of-the-art methods. Code is available at https://github.com/ouha1998/Learning-Progressive-Adaptation-for-Multi-Modal-Tracking.

He Wang, Tianyang Xu, Zhangyong Tang +2

School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China · Centre for Vision, Speech and Signal Processing, University of Surrey, GU2 7XH Guildford, U.K.

Multimodal Visual Object Tracking Multimodal Fusion Parameter-Efficient Fine-Tuning

Ask PDF

Mar 10, 2026

Exploring Modality-Aware Fusion and Decoupled Temporal Propagation for Multi-Modal Object Tracking

Most existing multimodal trackers adopt uniform fusion strategies, overlooking the inherent differences between modalities. Moreover, they propagate temporal information through mixed tokens, leading to entangled and less discriminative temporal representations. To address these limitations, we propose MDTrack, a novel framework for modality aware fusion and decoupled temporal propagation in multimodal object tracking. Specifically, for modality aware fusion, we allocate dedicated experts to each modality, including infrared, event, depth, and RGB, to process their respective representations. The gating mechanism within the Mixture of Experts dynamically selects the optimal experts based on the input features, enabling adaptive and modality specific fusion. For decoupled temporal propagation, we introduce two separate State Space Model structures to independently store and update the hidden states of the RGB and X modal streams, effectively capturing their distinct temporal information. To ensure synergy between the two temporal representations, we incorporate a set of cross attention modules between the input features of the two SSMs, facilitating implicit information exchange. The resulting temporally enriched features are then integrated into the backbone through another set of cross attention modules, enhancing MDTrack's ability to leverage temporal information. Extensive experiments demonstrate the effectiveness of our proposed method. Both MDTrack S and MDTrack U achieve state of the art performance across five multimodal tracking benchmarks.

Shilei Wang, Pujian Lai, Dong Gao +2

School of Automation, Northwestern Polytechnical University · College of Information Engineering, Northwest A&F University

Multimodal Visual Object Tracking Multimodal Fusion

Ask PDF

Abstract

Multimodal Visual Object Tracking

Explore similar work

SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker

Learning Progressive Adaptation for Multi-Modal Tracking

Exploring Modality-Aware Fusion and Decoupled Temporal Propagation for Multi-Modal Object Tracking