UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Tracking

Shiao Wang; Xiao Wang; Yuhe Ding; Jiarui Xu; Zikang Yan; Bo Jiang; Yonghong Tian; Jin Tang

cs.CVOct 9, 2026

UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Tracking

Authors: Shiao Wang, Xiao Wang, Yuhe Ding, Jiarui Xu, Zikang Yan, Bo Jiang, Yonghong Tian, Jin Tang

Organizations: School of Computer Science and Technology, Anhui University, Hefei 230601, China · Peng Cheng Laboratory, Shenzhen, China · School of Computer Science, Peking University, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, China

Abstract

RGB-X tracking leverages complementary modalities, such as thermal infrared, event, or depth, to improve robustness under challenging conditions. Existing general-purpose RGB-X trackers typically follow two paradigms: independent task-specific training and joint training across modality combinations. However, the former suffers from knowledge isolation and model redundancy, while the latter introduces optimization interference across heterogeneous tasks. In this work, we present UniMergeTrack, a new model merging paradigm that preserves task-specific optimization while enabling effective cross-task knowledge integration. UniMergeTrack first trains modality-specific trackers independently and then performs hierarchical model merging. Specifically, transferable backbones are merged across RGB-X tasks via high-pass-filtered merging, and complemented by lightweight task-specific experts that preserve modality-specific cues. Tracking heads are merged within each task using optimal transport-guided channel alignment to mitigate hidden channel mismatch. A subsequent distribution alignment fine-tuning stage further improves compatibility between the unified backbone and task-specific heads. Extensive experiments on the LasHeR, RGBT234, VisEvent, VOT-RGBD22, COESOT, and CDTB benchmarks demonstrate that UniMergeTrack achieves state-of-the-art or competitive performance across RGB-Thermal, RGB-Event, and RGB-Depth tracking tasks, while exhibiting strong generalization to unseen datasets. The source code will be released upon acceptance. The source code of this paper will be released on https://github.com/Event-AHU/OpenEvTracking.

Figures & tables

Figure 1: Comparison of three paradigms for RGB-X tracking. (a) Independent training retains task specialization but suffers from knowledge isolation. (b) Joint training enables knowledge sharing but introduces optimization interference. (c) UniMergeTrack achieves unified tracking via hierarchical model merging with cross-task backbone and intra-task head merging.

Figure 2: Overview of the proposed model merging framework for RGB-X tracking. We first train RGB-E, RGB-T, and RGB-D trackers independently, then perform hierarchical model merging by merging high-pass-filtered backbone task vectors across tasks and OT-aligned tracking heads within each task, while extracting lightweight modality experts. Finally, distribution alignment fine-tuning improves the compatibility between the unified backbone and task-specific tracking heads.

Method	Source	LasHeR		RGBT234		VisEvent		VOT-RGBD22
Method	Source	AUC	PR	MSR	MPR	AUC	PR	EAO	Acc.	Rob.
OSTrack ( Ye et al., 2022 )	ECCV22	41.2	51.5	54.9	72.9	53.4	69.5	67.6	80.3	83.3
ProTrack ( Yang et al., 2022 )	ACM MM22	42.0	53.8	59.9	79.5	47.1	63.2	65.1	80.1	80.2
ViPT ( Zhu et al., 2023 )	CVPR23	52.5	65.1	61.7	83.5	59.2	75.8	72.1	81.5	87.1
SeqTrackv2 ( Chen et al., 2023a )	ArXiv23	55.8	70.4	64.7	88.0	61.2	78.2	74.4	81.5	91.0
OneTracker ( Hong et al., 2024 )	CVPR24	53.8	67.2	64.2	85.7	60.8	76.7	72.7	81.9	87.2

Table 1: Comparison results on LasHeR, RGBT234, VisEvent, and VOT-RGBD22 datasets. The best and second-best results are highlighted in bold and underlined , respectively.

Dataset	Metric	SMSTracker	XTrack	SUTrack	SEATrack	UETrack	UTPTrack	UniMergeTrack
COESOT (RGB-Event)	SR	63.6	65.2	64.6	64.8	61.5	64.7	66.3
COESOT (RGB-Event)	PR	76.8	78.8	77.7	78.9	74.7	78.2	80.5
CDTB (RGB-Depth)	F-score	69.7	69.8	70.8	68.5	67.7	71.5	72.0
CDTB (RGB-Depth)	Precision	70.5	68.5	69.8	69.4	68.2	71.4	72.0

Table 2: Generalization comparison on COESOT and CDTB datasets.

No.	Variant	LasHeR		VisEvent		Avg. Change
No.	Variant	AUC	PR	AUC	PR	$\bm{\Delta}$ AUC	$\bm{\Delta}$ PR
1	UniMergeTrack (Ours)	61.8	77.2	64.1	81.5	–	–
2	w/o Model Merging (Baseline)	58.8	73.4	62.2	79.2	-2.5	-3.1
3	w/o Distribution Alignment Fine-Tuning	57.7	73.2	60.9	78.6	-3.7	-3.5
4	w/o High-pass Filtering	60.1	75.2	62.9	80.0	-1.5	-1.8
5	w/o Lightweight Modality Expert	60.3	75.3	63.0	80.4	-1.3	-1.5

Table 3: Component Analysis of UniMergeTrack on LasHeR and VisEvent.

\Delta

denotes the average performance change relative to the full model.

Table 6

Figure 3: Visualization of (a) tracking results and (b) attention maps produced by UniMergeTrack.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Table 8

Figure 4: Attribute evaluation on the VisEvent and LasHeR test sets.

Hyperparameter	RGB-Thermal	RGB-Event	RGB-Depth
Expert sparsity ( $d$ )	0.01	0.02	0.02
Reference head contribution ( $\alpha$ )	0.50	0.70	0.50

Appendix

Table 8: Hyperparameter settings for different tasks.

Visual Object Tracking

Visual object tracking follows objects across video frames, maintaining their locations and identities despite motion, occlusion, and changing appearance.

See all 174 papers

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts
May 5, 2026Lingyi Hong, Jinglun Li, Xinyu Zhou +6Cross-Modal Learning Multimodal Visual Object Tracking
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University
Fed-MAPTrack: Federated Modality-aware Prompt Tuning for RGB-X Tracking
Oct 26, 2026Xiangyu Liu, Yijie Shen, Yue Yin +3Multimodal Visual Object Tracking Multimodal Federated Learning
Beijing Normal-Hong Kong Baptist University, 519087, Zhuhai, Guangdong, China · Zhejiang University, 310058, Hangzhou, Zhejiang, China · Peking University, 100871, Beijing, China +1
Learning Progressive Adaptation for Multi-Modal Tracking
Mar 22, 2026He Wang, Tianyang Xu, Zhangyong Tang +2Multimodal Visual Object Tracking Multimodal Fusion
School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China · Centre for Vision, Speech and Signal Processing, University of Surrey, GU2 7XH Guildford, U.K.

May 5, 2026J/K move · Enter open · S save

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

Lingyi Hong, Jinglun Li, Xinyu Zhou, Kaixun Jiang+5

Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University

Cross-Modal Learning Multimodal Visual Object Tracking Multimodal Robustness Unified Multimodal Models Mixture of Experts Visual Object Tracking+2

Open paper AskPDF arXiv

Multimodal visual object tracking can be divided into to several kinds of tasks (e.g. RGB and RGB+X tracking), based on the input modality. Existing methods often train separate models for each modality or rely on pretrained models to adapt to new modalities, which limits efficiency, scalability, and usability. Thus, we introduce OneTrackerV2, a unified multi-modal tracking framework that enables end-to-end training for any modality. We propose Meta Merger to embed multi-modal information into a unified space, allowing flexible modality fusion and robustness. We further introduce Dual Mixture-of-Experts (DMoE): T-MoE models spatio-temporal relations for tracking, while M-MoE embeds multi-modal knowledge, disentangling cross-modal dependencies and reducing feature conflicts. With a shared architecture, unified parameters, and a single end-to-end training, OneTrackerV2 achieves state-of-the-art performance across five RGB and RGB+X tracking tasks and 12 benchmarks, while maintaining high inference efficiency. Notably, even after model compression, OneTrackerV2 retains strong performance. Moreover, OneTrackerV2 demonstrates remarkable robustness under modality-missing scenarios.

Abstract

Visual Object Tracking

Explore similar work

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

Fed-MAPTrack: Federated Modality-aware Prompt Tuning for RGB-X Tracking

Learning Progressive Adaptation for Multi-Modal Tracking