Most existing multimodal trackers adopt uniform fusion strategies, overlooking the inherent differences between modalities. Moreover, they propagate temporal information through mixed tokens, leading to entangled and less discriminative temporal representations. To address these limitations, we propose MDTrack, a novel framework for modality aware fusion and decoupled temporal propagation in multimodal object tracking. Specifically, for modality aware fusion, we allocate dedicated experts to each modality, including infrared, event, depth, and RGB, to process their respective representations. The gating mechanism within the Mixture of Experts dynamically selects the optimal experts based on the input features, enabling adaptive and modality specific fusion. For decoupled temporal propagation, we introduce two separate State Space Model structures to independently store and update the hidden states of the RGB and X modal streams, effectively capturing their distinct temporal information. To ensure synergy between the two temporal representations, we incorporate a set of cross attention modules between the input features of the two SSMs, facilitating implicit information exchange. The resulting temporally enriched features are then integrated into the backbone through another set of cross attention modules, enhancing MDTrack's ability to leverage temporal information. Extensive experiments demonstrate the effectiveness of our proposed method. Both MDTrack S and MDTrack U achieve state of the art performance across five multimodal tracking benchmarks.
Figures & tables
Figure 1: Overview of multi-modal tracking frameworks (a) and (b), with performance comparison (c) from left to right: STTrack, SUTrack, MDTrack-S, and MDTrack-U.
Figure 2: Overall tracking framework of MDTrack. Template and search region tokens from different modalities are concatenated and fed into the backbone. The search region tokens are then decoupled and stored separately for temporal propagation. Finally, the resulting features are fused by the modality-aware fusion module and passed to the tracking head for prediction.
LasHeR
RGBT234
DepthTrack
VOT-RGBD2022
Modality-Specific Training
Method
Publication
Pr ( ↑ )
AUC ( ↑ )
MPR ( ↑ )
MSR ( ↑ )
Pr ( ↑ )
Re ( ↑ )
F-score ( ↑ )
EAO ( ↑ )
Acc. ( ↑ )
Rob. ( ↑ )
ViPT ( Zhu et al. 2023a )
CVPR’23
65.1
52.5
83.5
61.7
59.2
59.6
59.4
72.1
81.5
87.1
SPT ( Zhu et al. 2023b )
AAAI’23
-
-
-
-
52.7
54.9
53.8
65.1
79.8
85.1
Un-Track ( Wu et al. 2024 )
CVPR’24
66.7
53.6
83.7
61.8
61.3
61.0
61.2
72.1
81.5
87.1
SDSTrack ( Hou et al. 2024 )
CVPR’24
66.5
53.1
84.8
62.5
61.9
60.9
61.4
72.8
81.2
88.3
OneTracker ( Hong et al. 2024 )
CVPR’24
67.2
53.8
85.7
64.2
60.7
60.4
60.9
72.7
81.9
87.2
Table 1: Comparisons with state-of-the-arts on LasHeR, RGBT234, DepthTrack, and VOT-RGBD2022. The best results are highlighted with bold underlines, and the second-best results are shown in bold fonts.
Method
Modality-Specific Train
Unified-Modality Train
ViPT
Un-Track
SDSTrack
OneTrack
STTrack
MDTrack-S
SeqTrack
ViPT
Un-Track
SUTrack
XTrack
MDTrack-U
Precision ( ↑ )
75.8
76.3
76.7
76.7
78.6
82.2
66.5
74.0
75.5
79.9
80.5
81.3
Success ( ↑ )
59.2
59.7
59.7
60.8
61.9
65.3
50.4
57.9
58.9
62.7
63.3
63.9
Table 2: Precision and Success comparisons between Modality-Specific and Unified-Modality training on the VisEvent dataset. The best results are highlighted with bold underlines, and the second-best results are shown in bold fonts.
Configuration
LasHeR
DepthTrack
VisEvent
Mean
Baseline
58.5
65.8
62.2
62.2
+ Token-based Temporal Module
59.4 (+0.9)
66.3 (+0.5)
62.3 (+0.1)
62.7 (+0.5)
+ Temporal Module
59.3 (+0.8)
66.6 (+0.8)
62.6 (+0.4)
62.8 (+0.6)
+ Decoupled Temporal Module
60.2 (+1.7)
67.6 (+1.8)
63.3 (+1.1)
63.7 (+1.5)
+ Fusion Module
58.8 (+0.3)
65.9 (+0.1)
62.4 (+0.2)
62.4 (+0.2)
+ Modality-Aware Fusion Module
59.6 (+1.1)
66.3 (+0.5)
62.8 (+0.6)
62.9 (+0.7)
Table 3: Ablation study of MDTrack on the LasHeR, DepthTrack, and VisEvent datasets. Each row illustrates a different module added to the baseline.
Figure 3: Visual comparisons of MDTrack-S and MDTrack-U with other multimodal trackers on the LasHeR, DepthTrack, and VisEvent datasets.
Multimodal visual object tracking can be divided into to several kinds of tasks (e.g. RGB and RGB+X tracking), based on the input modality. Existing methods often train separate models for each modality or rely on pretrained models to adapt to new modalities, which limits efficiency, scalability, and usability. Thus, we introduce OneTrackerV2, a unified multi-modal tracking framework that enables end-to-end training for any modality. We propose Meta Merger to embed multi-modal information into a unified space, allowing flexible modality fusion and robustness. We further introduce Dual Mixture-of-Experts (DMoE): T-MoE models spatio-temporal relations for tracking, while M-MoE embeds multi-modal knowledge, disentangling cross-modal dependencies and reducing feature conflicts. With a shared architecture, unified parameters, and a single end-to-end training, OneTrackerV2 achieves state-of-the-art performance across five RGB and RGB+X tracking tasks and 12 benchmarks, while maintaining high inference efficiency. Notably, even after model compression, OneTrackerV2 retains strong performance. Moreover, OneTrackerV2 demonstrates remarkable robustness under modality-missing scenarios.
Lingyi Hong, Jinglun Li, Xinyu Zhou +6
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University
Visual object tracking aims to continuously locate specific targets within sequential frames, evolving from single-modal methods to multi-modal ones. However, existing multi-modal trackers are typically designed for fixed modality combinations, requiring separate models for different inputs. This leads to a poor adaptability to missing or imperfect modalities, and limited generalization. To address these issues, we propose a novel unified framework called AnyTrack for object tracking with any modalities. Specifically, we design a Modality-aware Interaction Module (MIM) to facilitate dynamic interaction across diverse modalities. This module bridges modality discrepancies and aggregates temporal cues to maintain spatio-temporal consistency during cross-modal interaction. Furthermore, we introduce a Context Understanding Module (CUM) to establish spatial correspondence between visual features and target locations via global-local prompts. This module employs target-aware context modeling to enhance foreground-background discrimination for precise localization. Finally, to support the training and evaluation under diverse modalities, we extend existing multi-modal object tracking benchmarks by incorporating grayscale images, language descriptions, and audio clips. Extensive experiments with both complete and missing modality settings demonstrate that our AnyTrack achieves state-of-the-art performance, validating its effectiveness and flexibility. The source code is available at https://github.com/IdolLab/AnyTrack.
Hao Li, Yunzhi Zhuge, Wenning Hao +4
Army Engineering University of PLA Nanjing, China · Dalian University of Technology Dalian, China · National University of Defense Technology Nanjing, China
With growing real-world demands, efficient tracking has received increasing attention. However, most existing methods are limited to RGB inputs and struggle in multi-modal scenarios. Moreover, current multi-modal tracking approaches typically use complex designs, making them too heavy and slow for resource-constrained deployment. To tackle these limitations, we propose UETrack, an efficient framework for single object tracking. UETrack demonstrates high practicality and versatility, efficiently handling multiple modalities including RGB, Depth, Thermal, Event, and Language, and addresses the gap in efficient multi-modal tracking. It introduces two key components: a Token-Pooling-based Mixture-of-Experts mechanism that enhances modeling capacity through feature aggregation and expert specialization, and a Target-aware Adaptive Distillation strategy that selectively performs distillation based on sample characteristics, reducing redundant supervision and improving performance. Extensive experiments on 12 benchmarks across 3 hardware platforms show that UETrack achieves a superior speed-accuracy trade-off compared to previous methods. For instance, UETrack-B achieves 69.2% AUC on LaSOT and runs at 163/56/60 FPS on GPU/CPU/AGX, demonstrating strong practicality and versatility. Code is available at https://github.com/kangben258/UETrack.
Ben Kang, Jie Zhao, Xin Chen +5
Dalian University of Technology · City University of Hong Kong