Fed-MAPTrack: Federated Modality-aware Prompt Tuning for RGB-X Tracking

Xiangyu Liu; Yijie Shen; Yue Yin; Zhitong Yin; Yongqi Pan; Wentao Fan

cs.CVOct 26, 2026

Fed-MAPTrack: Federated Modality-aware Prompt Tuning for RGB-X Tracking

Authors: Xiangyu Liu, Yijie Shen, Yue Yin, Zhitong Yin, Yongqi Pan, Wentao Fan

Organizations: Beijing Normal-Hong Kong Baptist University, 519087, Zhuhai, Guangdong, China · Zhejiang University, 310058, Hangzhou, Zhejiang, China · Peking University, 100871, Beijing, China · Military Management College, National Defense University PLA China, 100091, Beijing, China

Abstract

RGB-X tracking complements RGB appearance with thermal, depth, or event observations, but centralized training requires collecting multimodal videos at a single location. Federated training keeps raw videos local, but sensor heterogeneity can create a mismatch between parameter-sharing domains and the clients contributing to them. This mismatch is particularly relevant when RGB-only clients coexist with sensor-equipped clients: their RGB observations remain useful, but they provide no auxiliary measurements for learning sensor-dependent interactions. We propose Fed-MAPTrack, a modality-aware federated prompt tuning framework for RGB-X tracking. The pretrained tracking backbone and prediction head are frozen, and clients optimize and exchange only prompt-related parameters. The server aggregates RGB-shared parameters globally and maintains separate thermal, event, and depth banks for auxiliary-specific, fusion, and normalization parameters. Selective participation retains RGB-only clients' contributions to the global RGB bank while excluding their fusion and normalization updates from modality-specific aggregation. Across three matched training repetitions under a fixed client partition, Fed-MAPTrack improves the mean FedAvg scores by 2.35 AUC points on LasHeR, 3.45 SR points on VisEvent, and 2.11 F-score points on DepthTrack. Among the nine evaluated methods, it achieves the highest mean thermal AUC and depth F-score, while its mean event SR is 0.07 points below that of Independent FL. Controlled ablations show that extending the bucketing scope without a compatible participation rule can reduce performance. These results highlight the need to jointly define the sharing domain and eligible contributors for each parameter group.

Figures & tables

Figure 1: Motivation of federated RGB-X tracking. (a) Centralized RGB-X tracking pools aligned data from all sensors for one training run. (b) In the federated setting, clients with different sensors (RGB, depth, thermal, event) train locally, and only the tuned prompt parameters are aggregated on top of a frozen ViT backbone.

Figure 2: Overall framework of Fed-MAPTrack. Clients keep training videos local and adapt prompt-related parameters on a frozen tracking backbone and prediction head. RGB-shared parameters use one global bank, while auxiliary-specific, fusion, and normalization parameters use modality banks. Server aggregation combines this sharing structure with sensor-dependent contributor eligibility.

Figure 3: Unified aggregation design. RGB-shared parameters use the global bank, while auxiliary-specific, fusion, and normalization parameters use thermal, depth, and event banks. The participation gate retains valid RGB updates from RGB-only clients but excludes their modality-related contributions from server bucket averages. Dispatch and local trainability are distinct from server eligibility.

Method	Category	LasHeR (RGB-T) AUC (%)	VisEvent (RGB-E) SR (%)	DepthTrack (RGB-D) F-score (%)
Local	Local training	$47.34\pm 0.31$	$52.27\pm 0.25$	$49.91\pm 1.46$
FedAvg	Standard FL	$46.55\pm 0.99$	$52.93\pm 0.59$	$53.85\pm 0.73$
FedProx	Standard FL	$46.78\pm 0.67$	$53.11\pm 0.66$	$54.06\pm 0.64$
FedPer	Personalized FL	$47.35\pm 0.54$	$54.47\pm 0.36$	$50.81\pm 0.68$
Ditto	Personalized FL	$46.46\pm 0.86$	$54.14\pm 0.42$	$53.90\pm 0.58$
FedMSplit-P	Multimodal FL	$46.43\pm 0.33$	$54.85\pm 0.35$	$54.24\pm 0.89$

Table 1: Main results on RGB-X tracking benchmarks at communication round 120. Scores are mean

\pm

sample standard deviation across three matched training repetitions under one fixed client partition. For Local, FedPer, Ditto, FedMSplit-P, and FED-PRIME, each repetition first averages the scores of the benchmark’s three sensor-equipped client models. Bold indicates the highest mean in each metric.

Figure 4: Success and precision curves on LasHeR at communication round 120 using the same 245 test sequences. Curves show means across three training repetitions, and shaded bands indicate sample standard deviations. Legend values are success AUC in (a) and precision at a 20-pixel threshold in (b), expressed as percentages. Auxiliary-only bucketing and FNA are included as controlled aggregation variants.

Variant	Aggregation component			Tracking performance
Variant	Auxiliary bucket	Fusion/Norm bucket	Selective participation	RGB-T AUC (%)	RGB-E SR (%)	RGB-D F-score (%)
FedAvg	$\times$	$\times$	$\times$	$46.55\pm 0.99$	$52.93\pm 0.59$	$53.85\pm 0.73$
Auxiliary-only bucketing	$\checkmark$	$\times$	$\times$	$47.47\pm 1.24$	$\mathbf{56.46\pm 0.37}$	$\mathbf{56.02\pm 0.32}$
Extended bucketing without S-A (FNA)	$\checkmark$	$\checkmark$	$\times$	$43.70\pm 0.95$	$54.21\pm 0.27$	$55.31\pm 0.57$
Fed-MAPTrack	$\checkmark$	$\checkmark$	$\checkmark$	$\mathbf{48.90\pm 0.63}$	$56.38\pm 0.41$	$55.96\pm 0.50$

Table 2: Ablation of the aggregation design. All scores are mean

\pm

sample standard deviation over three matched training repetitions. Checkmarks indicate enabled components. RGB-shared parameters remain globally aggregated in all four configurations. FNA and Fed-MAPTrack have identical bucketing scopes and differ only in RGB-only eligibility for fusion and normalization aggregation. Bold indicates the highest mean in each metric.

Figure 5: Effect of selective participation with the parameter-group bucketing scope held fixed. Blue squares and red circles show the mean scores of FNA and Fed-MAPTrack, respectively, while error bars indicate sample standard deviations across three training repetitions. The mean gains are 5.20 AUC points on LasHeR, 2.17 SR points on VisEvent, and 0.65 F-score points on DepthTrack. Each vertical axis spans ten percentage points. Connecting lines indicate changes between aggregate means, rather than individual-run trajectories.

Figure 6: Training-sample composition of sensor-equipped and RGB-only clients under the evaluated partition. Totals are shown above the bars. RGB-only samples account for 48.0%, 84.9%, and 3.0% of LasHeR, VisEvent, and DepthTrack, respectively. Each benchmark has three sensor-equipped clients and one RGB-only client, while the bars show the proportions of training samples assigned to these two client types.

Figure 7: Empirical final weight distributions of FNA and Fed-MAPTrack at communication round 120. RGB-shared weights come from the global bank, while the other groups pool weights from the three active modality banks. Curves average three per-run densities using common histogram bins covering the full observed range. Biases, scalar fovea parameters, stored initial banks, and inactive global modality banks are excluded. Panel (d) shows normalization scale weights. These are final-state distributions, not changes from initialization.

Figure 8: Layer-wise separation of final thermal, event, and depth parameter banks, measured by the relative dispersion in Eq. 14 . All parameters, including biases, are included within each group and Transformer block. Bars and error bars show the mean and sample standard deviation across three training repetitions. The RGB-shared group has only one bank and is omitted. Each panel starts at zero but uses its own labeled range. The statistic describes final bank separation, not tracking accuracy or changes from initialization.

Figure 9: Recorded local training loss over 120 communication rounds. Within each repetition, the twelve clients’ mean local losses are averaged equally in each round. Curves and shaded bands then show the mean and sample standard deviation across three repetitions. All recorded rounds are plotted without smoothing or interpolation. The figure describes local optimization, not intermediate-round benchmark performance.

Figure 10: Qualitative RGB-X tracking comparisons on (a) LasHeR, (b) VisEvent, and (c) DepthTrack. Green, blue, purple, and red solid boxes indicate ground truth, FedAvg, Independent FL, and Fed-MAPTrack, respectively. Each row shows temporally ordered frames from one sequence. Insets magnify a common region containing all displayed predictions and the ground truth, with gray corner markers identifying the corresponding regions in the full images. The examples include both accurate localization and tracking failures.

Figure 11: Box-derived localization comparison between FNA and Fed-MAPTrack on basketboy. The first column shows reference RGB crops, while the remaining columns visualize saved box predictions using the same elliptical Gaussian encoding. Each Gaussian is centered on the predicted box, with horizontal and vertical standard deviations equal to one quarter of its width and height, respectively. Green boxes indicate ground truth in the common crop coordinates.

Multimodal Visual Object Tracking

See all 21 papers

UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Tracking
Oct 9, 2026Shiao Wang, Xiao Wang, Yuhe Ding +5Visual Object Tracking Model Merging
School of Computer Science and Technology, Anhui University, Hefei 230601, China · Peng Cheng Laboratory, Shenzhen, China · School of Computer Science, Peking University, China +1
Unified Multimodal Visual Tracking with Dual Mixture-of-Experts
May 5, 2026Lingyi Hong, Jinglun Li, Xinyu Zhou +6Cross-Modal Learning Multimodal Visual Object Tracking
Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University
SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker
Apr 14, 2026Junbin Su, Ziteng Xue, Shihui Zhang +3Multimodal Visual Object Tracking Efficient Multimodal Inference
School of Artificial Intelligence (School of Software), Yanshan University · School of Software, Beihang University · Hebei Key Laboratory of Computer Virtual Technology and System Integration +3

Oct 9, 2026J/K move · Enter open · S save

UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Tracking

Shiao Wang, Xiao Wang, Yuhe Ding, Jiarui Xu+4

School of Computer Science and Technology, Anhui University, Hefei 230601, China · Peng Cheng Laboratory, Shenzhen, China · School of Computer Science, Peking University, China+1

Visual Object Tracking Model Merging Multimodal Sensor Fusion

Open paper AskPDF arXiv

RGB-X tracking leverages complementary modalities, such as thermal infrared, event, or depth, to improve robustness under challenging conditions. Existing general-purpose RGB-X trackers typically follow two paradigms: independent task-specific training and joint training across modality combinations. However, the former suffers from knowledge isolation and model redundancy, while the latter introduces optimization interference across heterogeneous tasks. In this work, we present UniMergeTrack, a new model merging paradigm that preserves task-specific optimization while enabling effective cross-task knowledge integration. UniMergeTrack first trains modality-specific trackers independently and then performs hierarchical model merging. Specifically, transferable backbones are merged across RGB-X tasks via high-pass-filtered merging, and complemented by lightweight task-specific experts that preserve modality-specific cues. Tracking heads are merged within each task using optimal transport-guided channel alignment to mitigate hidden channel mismatch. A subsequent distribution alignment fine-tuning stage further improves compatibility between the unified backbone and task-specific heads. Extensive experiments on the LasHeR, RGBT234, VisEvent, VOT-RGBD22, COESOT, and CDTB benchmarks demonstrate that UniMergeTrack achieves state-of-the-art or competitive performance across RGB-Thermal, RGB-Event, and RGB-Depth tracking tasks, while exhibiting strong generalization to unseen datasets. The source code will be released upon acceptance. The source code of this paper will be released on https://github.com/Event-AHU/OpenEvTracking.

Abstract

Multimodal Visual Object Tracking

Explore similar work

UniMergeTrack: Hierarchical Model Merging for General RGB-X Visual Object Tracking

Unified Multimodal Visual Tracking with Dual Mixture-of-Experts

SEATrack: Simple, Efficient, and Adaptive Multimodal Tracker