Organizations: Beijing Normal-Hong Kong Baptist University, 519087, Zhuhai, Guangdong, China · Zhejiang University, 310058, Hangzhou, Zhejiang, China · Peking University, 100871, Beijing, China · Military Management College, National Defense University PLA China, 100091, Beijing, China
RGB-X tracking complements RGB appearance with thermal, depth, or event observations, but centralized training requires collecting multimodal videos at a single location. Federated training keeps raw videos local, but sensor heterogeneity can create a mismatch between parameter-sharing domains and the clients contributing to them. This mismatch is particularly relevant when RGB-only clients coexist with sensor-equipped clients: their RGB observations remain useful, but they provide no auxiliary measurements for learning sensor-dependent interactions. We propose Fed-MAPTrack, a modality-aware federated prompt tuning framework for RGB-X tracking. The pretrained tracking backbone and prediction head are frozen, and clients optimize and exchange only prompt-related parameters. The server aggregates RGB-shared parameters globally and maintains separate thermal, event, and depth banks for auxiliary-specific, fusion, and normalization parameters. Selective participation retains RGB-only clients' contributions to the global RGB bank while excluding their fusion and normalization updates from modality-specific aggregation. Across three matched training repetitions under a fixed client partition, Fed-MAPTrack improves the mean FedAvg scores by 2.35 AUC points on LasHeR, 3.45 SR points on VisEvent, and 2.11 F-score points on DepthTrack. Among the nine evaluated methods, it achieves the highest mean thermal AUC and depth F-score, while its mean event SR is 0.07 points below that of Independent FL. Controlled ablations show that extending the bucketing scope without a compatible participation rule can reduce performance. These results highlight the need to jointly define the sharing domain and eligible contributors for each parameter group.
Figures & tables
Figure 1: Motivation of federated RGB-X tracking. (a) Centralized RGB-X tracking pools aligned data from all sensors for one training run. (b) In the federated setting, clients with different sensors (RGB, depth, thermal, event) train locally, and only the tuned prompt parameters are aggregated on top of a frozen ViT backbone.
Figure 2: Overall framework of Fed-MAPTrack. Clients keep training videos local and adapt prompt-related parameters on a frozen tracking backbone and prediction head. RGB-shared parameters use one global bank, while auxiliary-specific, fusion, and normalization parameters use modality banks. Server aggregation combines this sharing structure with sensor-dependent contributor eligibility.
Figure 3: Unified aggregation design. RGB-shared parameters use the global bank, while auxiliary-specific, fusion, and normalization parameters use thermal, depth, and event banks. The participation gate retains valid RGB updates from RGB-only clients but excludes their modality-related contributions from server bucket averages. Dispatch and local trainability are distinct from server eligibility.
Method
Category
LasHeR (RGB-T) AUC (%)
VisEvent (RGB-E) SR (%)
DepthTrack (RGB-D) F-score (%)
Local
Local training
47.34±0.31
52.27±0.25
49.91±1.46
FedAvg
Standard FL
46.55±0.99
52.93±0.59
53.85±0.73
FedProx
Standard FL
46.78±0.67
53.11±0.66
54.06±0.64
FedPer
Personalized FL
47.35±0.54
54.47±0.36
50.81±0.68
Ditto
Personalized FL
46.46±0.86
54.14±0.42
53.90±0.58
FedMSplit-P
Multimodal FL
46.43±0.33
54.85±0.35
54.24±0.89
Table 1: Main results on RGB-X tracking benchmarks at communication round 120. Scores are mean ± sample standard deviation across three matched training repetitions under one fixed client partition. For Local, FedPer, Ditto, FedMSplit-P, and FED-PRIME, each repetition first averages the scores of the benchmark’s three sensor-equipped client models. Bold indicates the highest mean in each metric.
Figure 4: Success and precision curves on LasHeR at communication round 120 using the same 245 test sequences. Curves show means across three training repetitions, and shaded bands indicate sample standard deviations. Legend values are success AUC in (a) and precision at a 20-pixel threshold in (b), expressed as percentages. Auxiliary-only bucketing and FNA are included as controlled aggregation variants.
Variant
Aggregation component
Tracking performance
Auxiliary bucket
Fusion/Norm bucket
Selective participation
RGB-T AUC (%)
RGB-E SR (%)
RGB-D F-score (%)
FedAvg
×
×
×
46.55±0.99
52.93±0.59
53.85±0.73
Auxiliary-only bucketing
✓
×
×
47.47±1.24
56.46±0.37
56.02±0.32
Extended bucketing without S-A (FNA)
✓
✓
×
43.70±0.95
54.21±0.27
55.31±0.57
Fed-MAPTrack
✓
✓
✓
48.90±0.63
56.38±0.41
55.96±0.50
Table 2: Ablation of the aggregation design. All scores are mean ± sample standard deviation over three matched training repetitions. Checkmarks indicate enabled components. RGB-shared parameters remain globally aggregated in all four configurations. FNA and Fed-MAPTrack have identical bucketing scopes and differ only in RGB-only eligibility for fusion and normalization aggregation. Bold indicates the highest mean in each metric.
Figure 5: Effect of selective participation with the parameter-group bucketing scope held fixed. Blue squares and red circles show the mean scores of FNA and Fed-MAPTrack, respectively, while error bars indicate sample standard deviations across three training repetitions. The mean gains are 5.20 AUC points on LasHeR, 2.17 SR points on VisEvent, and 0.65 F-score points on DepthTrack. Each vertical axis spans ten percentage points. Connecting lines indicate changes between aggregate means, rather than individual-run trajectories.
Figure 6: Training-sample composition of sensor-equipped and RGB-only clients under the evaluated partition. Totals are shown above the bars. RGB-only samples account for 48.0%, 84.9%, and 3.0% of LasHeR, VisEvent, and DepthTrack, respectively. Each benchmark has three sensor-equipped clients and one RGB-only client, while the bars show the proportions of training samples assigned to these two client types.
Figure 7: Empirical final weight distributions of FNA and Fed-MAPTrack at communication round 120. RGB-shared weights come from the global bank, while the other groups pool weights from the three active modality banks. Curves average three per-run densities using common histogram bins covering the full observed range. Biases, scalar fovea parameters, stored initial banks, and inactive global modality banks are excluded. Panel (d) shows normalization scale weights. These are final-state distributions, not changes from initialization.
Figure 8: Layer-wise separation of final thermal, event, and depth parameter banks, measured by the relative dispersion in Eq. 14 . All parameters, including biases, are included within each group and Transformer block. Bars and error bars show the mean and sample standard deviation across three training repetitions. The RGB-shared group has only one bank and is omitted. Each panel starts at zero but uses its own labeled range. The statistic describes final bank separation, not tracking accuracy or changes from initialization.
Figure 9: Recorded local training loss over 120 communication rounds. Within each repetition, the twelve clients’ mean local losses are averaged equally in each round. Curves and shaded bands then show the mean and sample standard deviation across three repetitions. All recorded rounds are plotted without smoothing or interpolation. The figure describes local optimization, not intermediate-round benchmark performance.
Figure 10: Qualitative RGB-X tracking comparisons on (a) LasHeR, (b) VisEvent, and (c) DepthTrack. Green, blue, purple, and red solid boxes indicate ground truth, FedAvg, Independent FL, and Fed-MAPTrack, respectively. Each row shows temporally ordered frames from one sequence. Insets magnify a common region containing all displayed predictions and the ground truth, with gray corner markers identifying the corresponding regions in the full images. The examples include both accurate localization and tracking failures.
Figure 11: Box-derived localization comparison between FNA and Fed-MAPTrack on basketboy. The first column shows reference RGB crops, while the remaining columns visualize saved box predictions using the same elliptical Gaussian encoding. Each Gaussian is centered on the predicted box, with horizontal and vertical standard deviations equal to one quarter of its width and height, respectively. Green boxes indicate ground truth in the common crop coordinates.
RGB-X tracking leverages complementary modalities, such as thermal infrared, event, or depth, to improve robustness under challenging conditions. Existing general-purpose RGB-X trackers typically follow two paradigms: independent task-specific training and joint training across modality combinations. However, the former suffers from knowledge isolation and model redundancy, while the latter introduces optimization interference across heterogeneous tasks. In this work, we present UniMergeTrack, a new model merging paradigm that preserves task-specific optimization while enabling effective cross-task knowledge integration. UniMergeTrack first trains modality-specific trackers independently and then performs hierarchical model merging. Specifically, transferable backbones are merged across RGB-X tasks via high-pass-filtered merging, and complemented by lightweight task-specific experts that preserve modality-specific cues. Tracking heads are merged within each task using optimal transport-guided channel alignment to mitigate hidden channel mismatch. A subsequent distribution alignment fine-tuning stage further improves compatibility between the unified backbone and task-specific heads. Extensive experiments on the LasHeR, RGBT234, VisEvent, VOT-RGBD22, COESOT, and CDTB benchmarks demonstrate that UniMergeTrack achieves state-of-the-art or competitive performance across RGB-Thermal, RGB-Event, and RGB-Depth tracking tasks, while exhibiting strong generalization to unseen datasets. The source code will be released upon acceptance. The source code of this paper will be released on https://github.com/Event-AHU/OpenEvTracking.
Shiao Wang, Xiao Wang, Yuhe Ding +5
School of Computer Science and Technology, Anhui University, Hefei 230601, China · Peng Cheng Laboratory, Shenzhen, China · School of Computer Science, Peking University, China +1
Multimodal visual object tracking can be divided into to several kinds of tasks (e.g. RGB and RGB+X tracking), based on the input modality. Existing methods often train separate models for each modality or rely on pretrained models to adapt to new modalities, which limits efficiency, scalability, and usability. Thus, we introduce OneTrackerV2, a unified multi-modal tracking framework that enables end-to-end training for any modality. We propose Meta Merger to embed multi-modal information into a unified space, allowing flexible modality fusion and robustness. We further introduce Dual Mixture-of-Experts (DMoE): T-MoE models spatio-temporal relations for tracking, while M-MoE embeds multi-modal knowledge, disentangling cross-modal dependencies and reducing feature conflicts. With a shared architecture, unified parameters, and a single end-to-end training, OneTrackerV2 achieves state-of-the-art performance across five RGB and RGB+X tracking tasks and 12 benchmarks, while maintaining high inference efficiency. Notably, even after model compression, OneTrackerV2 retains strong performance. Moreover, OneTrackerV2 demonstrates remarkable robustness under modality-missing scenarios.
Lingyi Hong, Jinglun Li, Xinyu Zhou +6
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University
Parameter-efficient fine-tuning (PEFT) in multimodal tracking reveals a concerning trend where recent performance gains are often achieved at the cost of inflated parameter budgets, which fundamentally erodes PEFT's efficiency promise. In this work, we introduce SEATrack, a Simple, Efficient, and Adaptive two-stream multimodal tracker that tackles this performance-efficiency dilemma from two complementary perspectives. We first prioritize cross-modal alignment of matching responses, an underexplored yet pivotal factor that we argue is essential for breaking the trade-off. Specifically, we observe that modality-specific biases in existing two-stream methods generate conflicting matching attention maps, thereby hindering effective joint representation learning. To mitigate this, we propose AMG-LoRA, which seamlessly integrates Low-Rank Adaptation (LoRA) for domain adaptation with Adaptive Mutual Guidance (AMG) to dynamically refine and align attention maps across modalities. We then depart from conventional local fusion approaches by introducing a Hierarchical Mixture of Experts (HMoE) that enables efficient global relation modeling, effectively balancing expressiveness and computational efficiency in cross-modal fusion. Equipped with these innovations, SEATrack advances notable progress over state-of-the-art methods in balancing performance with efficiency across RGB-T, RGB-D, and RGB-E tracking tasks. cyan{Code is available}.
Junbin Su, Ziteng Xue, Shihui Zhang +3
School of Artificial Intelligence (School of Software), Yanshan University · School of Software, Beihang University · Hebei Key Laboratory of Computer Virtual Technology and System Integration +3