SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker

Junbin Su; Ziteng Xue; Shihui Zhang; Kun Chen; Weiming Hu; Zhipeng Zhang

cs.CVApr 14, 2026

SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker

Authors: Junbin Su, Ziteng Xue, Shihui Zhang, Kun Chen, Weiming Hu, Zhipeng Zhang

Organizations: School of Artificial Intelligence (School of Software), Yanshan University · School of Software, Beihang University · Hebei Key Laboratory of Computer Virtual Technology and System Integration · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA ·

Abstract

Parameter-efficient fine-tuning (PEFT) in multimodal tracking reveals a concerning trend where recent performance gains are often achieved at the cost of inflated parameter budgets, which fundamentally erodes PEFT's efficiency promise. In this work, we introduce SEATrack, a Simple, Efficient, and Adaptive two-stream multimodal tracker that tackles this performance-efficiency dilemma from two complementary perspectives. We first prioritize cross-modal alignment of matching responses, an underexplored yet pivotal factor that we argue is essential for breaking the trade-off. Specifically, we observe that modality-specific biases in existing two-stream methods generate conflicting matching attention maps, thereby hindering effective joint representation learning. To mitigate this, we propose AMG-LoRA, which seamlessly integrates Low-Rank Adaptation (LoRA) for domain adaptation with Adaptive Mutual Guidance (AMG) to dynamically refine and align attention maps across modalities. We then depart from conventional local fusion approaches by introducing a Hierarchical Mixture of Experts (HMoE) that enables efficient global relation modeling, effectively balancing expressiveness and computational efficiency in cross-modal fusion. Equipped with these innovations, SEATrack advances notable progress over state-of-the-art methods in balancing performance with efficiency across RGB-T, RGB-D, and RGB-E tracking tasks. cyan{Code is available}.

Figures & tables

Figure 1: Previous frameworks v.s. SEATrack . (a) The previous one-stream method 55 suffers from attention shifting when performing intra-modal matching on mixed inputs. (b) Similarly, domain gaps cause attention maps’ inconsistency in the two-stream method 11 . (c) Our method is able to produce aligned and refined attention maps, which facilitate cross-modal fusion.

Figure 2: Overall pipeline of SEATrack. Input tokens from each modality are processed by stacked shared ViT encoders for intra-modal target matching and feature extraction. To enable cross-modal interaction, the proposed AMG-LoRA and HMoE are embedded into the ViT encoder every 2 layers to perform attention alignment and information fusion, respectively. AMG-LoRA’s mutual guidance mechanism is exemplified through the “RGB refines X” pathway (Eq. 4 ), with the reverse direction behaving similarly.

Figure 3: Architecture details of HMoE configured with $e=4$ experts, each containing $h=2$ heads.

			LasHeR			RGBT234		DepthTrack			VOT-RGBD2022			VisEvent
Type	Method	Source	PR↑	NPR↑	SR↑	MPR↑	MSR↑	PR↑	RE↑	F-score↑	EAO↑	Acc.↑	Rob.↑	PR↑	SR↑	L.P. ↓	FPS ↑
FFT	TBSI 20	CVPR’23	69.2	65.7	55.6	87.1	63.7	-	-	-	-	-	-	-	-	203M	36.2
	IIMF 4	MM’24	72.4	68.4	58.1	86.8	64.4	-	-	-	-	-	-	-	-	182M	-
	CAFormer 44	AAAI’25	70.0	66.1	55.6	88.3	66.4	-	-	-	-	-	-	-	-	-	83.6
	DeT 45	CVPR’21	-	-	-	-	-	56.0	50.6	53.2	65.7	76.0	84.5	-	-	-	-
	TABBTrack 49	PR’25	-	-	-	-	-	62.2	61.5	61.8	72.2	82.1	87.4	-	-	-	27

Table 1: Main performance comparison across RGB-T, RGB-D, and RGB-E datasets. SEATrack delivers strong overall results on five benchmarks, with outstanding parameter efficiency. L.P. denotes Learnable Parameters. * indicates reproduced results.

AMG-LoRA	HMoE	LasHeR		DepthTrack			VisEvent		Param.
AMG-LoRA	HMoE	PR	SR	PR	RE	F-score	PR	SR	Param.
-	-	51.5	41.2	53.6	52.2	52.9	69.5	53.4	0
✓	-	69.8	55.7	58.1	57.2	57.6	75.6	59.1	0.14M
-	✓	67.4	54.2	60.9	61.3	61.1	76.5	59.9	0.46M
✓	✓	71.6	57.3	62.9	63.5	63.2	77.1	60.3	0.6M

Table 2: Component-wise Ablation Studies.

$w_{\text{X}}$ & $w_{\text{rgb}}$	LasHeR		DepthTrack			VisEvent
$w_{\text{X}}$ & $w_{\text{rgb}}$	PR	SR	PR	RE	F-score	PR	SR
0	70.4	56.2	57.5	57.6	57.5	75.9	59.6
0.5	69.7	55.7	58.4	59.5	58.9	76.8	60.2
1	71.6	57.3	62.9	63.5	63.2	77.1	60.3

Table 3: Initialization Choices of AMG-LoRA’s Scaling Factor.

Figure 4: LoRA v.s. AMG-LoRA across 19 challenging attributes on LasHeR 25 .

Heads	Dim.	Param.	LasHeR		DepthTrack			VisEvent
Heads	Dim.	Param.	PR	SR	PR	RE	F-score	PR	SR
1	768	0.9M	69.7	55.8	58.1	59.7	58.9	76.7	60.2
2	384	0.6M	71.6	57.3	62.9	63.5	63.2	77.1	60.3
4	192	0.5M	70.6	56.4	60.7	58.5	59.6	76.6	59.9
8	96	0.4M	70.6	56.5	61.0	61.7.	61.3	76.3	59.7

Table 4: Different Numbers of Heads per Expert in HMoE.

Attn.	FFN.	Param.	LasHeR		DepthTrack			VisEvent
Attn.	FFN.	Param.	PR	SR	PR	RE	F-score	PR	SR
4	4	0.5M	70.3	56.0	60.9	61.6	61.2	76.8	60.2
8	4	0.6M	70.7	56.6	59.7	60.0	59.8	76.6	59.9
4	8	0.6M	71.6	57.3	62.9	63.5	63.2	77.1	60.3
8	16	0.9M	70.9	56.7	59.3	59.0	59.1	76.1	59.7

Table 5: Different Numbers of Experts in HMoE.

Figure 5: LoRA (left) vs. AMG-LoRA (right) under a modality-missing scenario. (a) Missing RGB frame. (b) RGB attention map. (c) Thermal frame. (d) Thermal attention map.

Figure 6: Comprehensive visualization of AMG-LoRA’s adaptability. The results of “Pre-train” row are directly inferred from the frozen foundation tracker 47 .

Variants	LasHeR		DepthTrack			VisEvent		MACs (10 9 )	FPS
Variants	PR	SR	PR	RE	F-score	PR	SR	MACs (10 9 )	FPS
Crs_Attn	67.6	54.4	61.2	61.4	61.3	75.8	59.3	56.4	41.5
+AMG-LoRA	69.9	55.7	62.7	60.4	61.5	76.4	59.9	56.4	41.2
MCP	68.4	54.9	60.1	59.2	59.6	75.6	59.3	56.5	54.7
+AMG-LoRA	70.3	56.1	61.2	60.7	61.0	75.9	59.5	56.5	54.2
HMoE	67.4	54.2	60.9	61.3	61.1	76.5	59.9	56.4	63.8

Table 6: Comparison of Different Fusion Strategies.

Figure 7: Prediction-level comparison of SEATrack with two well-established PEFT-based multimodal trackers.

Table 7: Rank Choices of AMG-LoRA and HMoE.

Weight Type	LasHeR		DepthTrack			VisEvent
Weight Type	PR	SR	PR	RE	F-score	PR	SR
$W_{[q,k]}$	69.8	56.1	56.8	56.4	56.6	75.8	59.4
$W_{[q,v]}$	71.3	56.8	60.3	59.6	60.0	76.0	59.4
$W_{[k,v]}$	71.6	57.3	62.9	63.5	63.2	77.1	60.3

Table 8: Types of Adaptation Weights in AMG-LoRA

Method	LasHeR		DepthTrack		VisEvent
Method	Cos( $\uparrow$ )	SKL( $\downarrow$ )	Cos( $\uparrow$ )	SKL( $\downarrow$ )	Cos( $\uparrow$ )	SKL( $\downarrow$ )
LoRA	0.51	0.99	0.44	1.2	0.51	0.92
AMG-LoRA	0.99	0.04	0.97	0.09	0.91	0.16

Table 9: Statistical results of alignment. Cos denotes cosine similarity, and SKL denotes symmetric KL divergence (scaled by 10 4 for clarity).

Figure 8: Additional visualizations of the adaptability of AMG-LoRA under diverse real-world scenarios. For each case, the columns present the ground-truth target region ( left ), the corresponding RGB branch attention map ( middle ), and the corresponding X-modal attention map ( right ). While the rows correspond to the outputs of the Foundation model 47 , LoRA 13 , and the proposed AMG-LoRA.

Table 10: Efficiency under Different HMoE Configurations.

Figure 9: (a): the learned scalars across layers. (b) and (c): layer-wise attention maps (top: RGB; bottom: Thermal).

Multimodal Visual Object Tracking

See all 19 papers

Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking
Mar 14, 2026Yabin Zhu, Jianqi Li, Chenglong Li +3Multimodal Visual Object Tracking Parameter-Efficient Fine-Tuning
School of Public Security and Emergency Management, Anhui University of Science and Technology, Hefei 231131, China. · Information Materials and Intelligent Sensing Laboratory of Anhui Province, Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Artificial Intelligence, Anhui University, Hefei 230601, China. · School of Artificial Intelligence, Anhui University of Science and Technology, Hefei 231131, China. +1
Learning Progressive Adaptation for Multi-Modal Tracking
Mar 22, 2026He Wang, Tianyang Xu, Zhangyong Tang +2Multimodal Visual Object Tracking Multimodal Fusion
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China · Centre for Vision, Speech and Signal Processing, University of Surrey, GU2 7XH Guildford, U.K.
Unified Multimodal Visual Tracking with Dual Mixture-of-Experts
May 5, 2026Lingyi Hong, Jinglun Li, Xinyu Zhou +6Cross-Modal Learning Multimodal Visual Object Tracking
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University

Mar 14, 2026J/K move · Enter open · S save

Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking

Yabin Zhu, Jianqi Li, Chenglong Li, Jiaxiang Wang+2

School of Public Security and Emergency Management, Anhui University of Science and Technology, Hefei 231131, China. · Information Materials and Intelligent Sensing Laboratory of Anhui Province, Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Artificial Intelligence, Anhui University, Hefei 230601, China. · School of Artificial Intelligence, Anhui University of Science and Technology, Hefei 231131, China.+1

Multimodal Visual Object Tracking Parameter-Efficient Fine-Tuning Multimodal Fusion Mixture-of-Experts Models

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) techniques, such as prompts and adapters, are widely used in multi-modal tracking because they alleviate issues of full-model fine-tuning, including time inefficiency, high resource consumption, parameter storage burden, and catastrophic forgetting. However, due to cross-modal heterogeneity, most existing PEFT-based methods struggle to effectively represent multi-modal features within a unified framework with shared parameters. To address this problem, we propose a novel Sparse-Dense Mixture of Experts Adapter (SDMoEA) framework for PEFT-based multi-modal tracking under a unified model structure. Specifically, we design an SDMoE module as the multi-modal adapter to model modality-specific and shared information efficiently. SDMoE consists of a sparse MoE and a dense-shared MoE: the former captures modality-specific information, while the latter models shared cross-modal information. Furthermore, to overcome limitations of existing tracking methods in modeling high-order correlations during multi-level multi-modal fusion, we introduce a Gram-based Semantic Alignment Hypergraph Fusion (GSAHF) module. It first employs Gram matrices for cross-modal semantic alignment, ensuring that the constructed hypergraph accurately reflects semantic similarity and high-order dependencies between modalities. The aligned features are then integrated into the hypergraph structure to exploit its ability to model high-order relationships, enabling deep fusion of multi-level multi-modal information. Extensive experiments demonstrate that the proposed method achieves superior performance compared with other PEFT approaches on several multi-modal tracking benchmarks, including LasHeR, RGBT234, VTUAV, VisEvent, COESOT, DepthTrack, and VOT-RGBD2022.

Abstract

Multimodal Visual Object Tracking

Explore similar work

Sparse-Dense Mixture of Experts Adapter for Multi-Modal Tracking

Learning Progressive Adaptation for Multi-Modal Tracking

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts