UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity

Pengyu Wang; Baochen Xiong; Xiaoshan Yang; Yifan Xu; Zhang Qimeng; Haifeng Chen; Changsheng Xu

cs.LGAug 16, 2026

UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity

Authors: Pengyu Wang, Baochen Xiong, Xiaoshan Yang, Yifan Xu, Zhang Qimeng, Haifeng Chen, Changsheng Xu

Abstract

Vision-Language Models (VLMs) have demonstrated strong performance in multimodal understanding and generation. However, fine-tuning of VLMs typically relies on centralized data, which raises privacy concerns in certain domains (e.g. healthcare). Federated Learning (FL) provides a natural solution by enabling model training without sharing raw data. However, applying FL to VLM instruction tuning is highly challenging. VLMs have substantial parameter scales, and in real-world scenarios, clients exhibit significant heterogeneity in tasks, modalities, and model architectures. Existing methods mainly focus on simplified settings and are unable to handle such multi-dimensional heterogeneous scenarios. In this work, we study federated instruction tuning under joint heterogeneity in tasks, modalities, and model architectures. We propose UniFed-VLM, a unified federated instruction tuning framework for VLMs that addresses multiple types of heterogeneity. It consists of two key components: 1) Federated Compensated Subspace Aggregation (FedCSA), which performs subspace-aligned aggregation of parameter-efficient adapters with dynamic weighting and compensation to mitigate heterogeneity-induced conflicts; 2) Two-stage Collaborative Distillation (TCoD), which enables effective knowledge transfer across heterogeneous models via a Mutual Distillation Adapter (MDA) and a mixture-of-experts-based distillation strategy. We conduct experiments on multiple benchmark datasets, and the results show that UniFed-VLM achieves stronger average performance across diverse tasks compared with existing FL methods. The source code is available at: https://github.com/wangpengyu2004/UniFed-VLM.

Figures & tables

Figure 1: An overview of the federated setting with intertwined heterogeneities across models, tasks, and modalities.

Figure 2: Overview of UniFed-VLM. Each client consists of a local VLM and an MDA. The training involves two steps. In Step 1, homogeneous LoRA parameters from the same model group are aggregated via FedCSA with SVD-based subspace alignment, dynamic weighting, and compensation optimization. In Step 2, the aggregated parameters are broadcast to clients, where the local VLM and MDA perform two-stage knowledge distillation with different MDA architectures across stages.

Scenario	Method	Fed-Nature				Fed-Crossdomain
		VQA	Caption	VG	CLS	VQA	Caption	VG	CLS
		Acc $\uparrow$	CIDEr/R-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	CIDEr/R-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
Scenario 1	LOCAL	0.712	0.705/0.348	0.390	0.891	0.509	2.168/0.569	0.311	0.213
	FedAdam	0.709	0.763/0.361	0.499	0.910	0.541	1.201/0.527	0.319	0.222
	FedAvg	0.734	0.744/0.349	0.465	0.912	0.542	2.501/0.624	0.378	0.226
	FedProx	0.702	0.753/0.357	0.457	0.906	0.550	1.748/0.579	0.367	0.234

Table 1: Comparison of LOCAL, Centralized, and federated baselines with UniFed-VLM on the Fed-Nature and Fed-Crossdomain datasets under different heterogeneous settings.

Method	VQA	Caption	VG	CLS
Method	Acc $\uparrow$	CIDEr/R-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
UniFed-VLM	0.743	0.784/0.358	0.519	0.918
w/o FedCSA (w/ FedAvg)	0.708	0.746/0.361	0.445	0.907
w/o FedCSA (w/ FedProx)	0.711	0.781/0.365	0.484	0.901
w/o TCoD	0.707	0.766/0.361	0.458	0.911
w/o Compensation	0.726	0.774/0.357	0.503	0.915

Table 2: Ablation study on UniFed-VLM. ”w/” indicates replacing FedCSA with an alternative aggregation method.

Panel of Figure 3 — Figure 3: Convergence and performance comparisons on Fed-Nature under Scenario 1. (a) Average training loss across all tasks (including the MDA module). (b)–(d) Performance on classification, VQA, and visual grounding, respectively.

LoRA Rank	VQA	Caption	VG	CLS
LoRA Rank	Acc $\uparrow$	CIDEr/R-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
8 / 8	0.743	0.784/0.358	0.519	0.918
16 / 16	0.732	0.805/0.365	0.533	0.915
32 / 32	0.736	0.824/0.371	0.522	0.922
16 / 8	0.717	0.792/0.367	0.492	0.915
8 / 16	0.712	0.756/0.360	0.487	0.913

Table 3: Effect of homogeneous and heterogeneous LoRA ranks on the Fed-Nature dataset (Scenario 1).

Panel of Figure 4 — Figure 4: Loss curves and performance trends of FedAdam on caption generation (a) and VQA (b) under Scenario 2 (Fed-Nature).

$d_{h}$	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
1024	0.726	0.783/0.363	0.499	0.910
2048	0.743	0.784/0.358	0.519	0.918
2560	0.718	0.778/0.366	0.478	0.910

Table 4: Impact of hidden dimension on downstream tasks.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure B.1: Loss curves of the supplementary parameter optimization process under different numbers of iterations for the Show-O model on the Fed-Nature dataset.

Iter.	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
100	0.724	0.782/0.363	0.504	0.912
200	0.743	0.784/0.358	0.519	0.918
300	0.737	0.792/0.368	0.497	0.910

Appendix

Table C.1: Effect of different numbers of iterations for supplementary parameters on performance on Fed-Nature.

$\lambda_{\text{task}}^{MoE}$	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
0.1	0.730	0.778/0.355	0.520	0.919
0.2	0.710	0.787/0.363	0.535	0.911
0.3	0.743	0.784/0.358	0.519	0.918
0.4	0.706	0.783/0.364	0.524	0.906

Appendix

Table C.2: Effect of the MoE task loss weight

\lambda_{\text{task}}^{MoE}

$\alpha$	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
0.3	0.738	0.762/0.358	0.526	0.912
0.4	0.743	0.784/0.358	0.519	0.918
0.5	0.726	0.782/0.360	0.529	0.916
0.6	0.724	0.781/0.359	0.519	0.911

Appendix

Table C.3: Effect of coefficient

\alpha

Layers	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
2	0.694	0.776/0.363	0.513	0.912
3	0.743	0.784/0.358	0.519	0.918
4	0.721	0.847/0.374	0.516	0.907

Appendix

Table C.4: Effect of the number of projection layers in MDA.

$\lambda_{S1}$ - $\lambda_{S2}$	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
0.7-0.1	0.729	0.777/0.355	0.513	0.912
0.7-0.3	0.739	0.729/0.352	0.515	0.918
0.7-0.2	0.743	0.784/0.358	0.519	0.918
0.6-0.2	0.727	0.788/0.362	0.520	0.914
0.8-0.2	0.722	0.821/0.371	0.517	0.905

Appendix

Table D.1: Effect of distillation weights

\lambda_{S1}

and

\lambda_{S2}

Method	VQA	Caption Generation	VG	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
FedAvg	0.725	0.777/0.348	0.347	0.892
FedProx	0.716	0.738/0.344	0.355	0.888
HetLoRA	0.717	0.766/0.342	0.342	0.891
pFedLoRA	0.571	0.665/0.324	0.307	0.837
Ours	0.738	0.765/0.346	0.365	0.899

Appendix

Table E.1: Evaluation under another heterogeneous task-model scenario on Fed-Nature.

Method	VQA	Caption Generation	VG	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
Ours	0.743	0.784/0.358	0.519	0.918
Ours (w/ DP)	0.724	0.781/0.360	0.521	0.918

Appendix

Table E.2: Privacy-preserving evaluation of UniFed-VLM on Fed-Nature under Scenario 1. Differential privacy (DP) is applied to LoRA updates before federated aggregation.

Explore similar work

Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models
May 5, 2026Shule Lu, Yujing Wang, Hainan Zhang +5Reward Modeling Preference Optimization
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing · Institute of Artificial Intelligence, Beihang University, China · MAIS, Institute of Automation, Chinese Academy of Sciences, China +2
Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning
May 27, 2026Yuting Ma, Lechao Cheng, Xiaohua XuVLM Adaptation Vision-Language Alignment
School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui, China.
On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing
Aug 5, 2026Simon Lösche, Barış Büyüktaş, Mathis Adler +3Remote Sensing Image Understanding Prompt Learning
BIFOLD - Berlin Institute for the Foundations of Learning and Data, 10587 Berlin, Germany · Technische Universität Berlin, 10623 Berlin, Germany · Orion Lab, School of Rural, Surveying and Geoinformatics Engineering, National Technical University of Athens, 15772 Athens, Greece +2

May 5, 2026, cs.AIJ/K move · Enter open · S save

Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models

Shule Lu, Yujing Wang, Hainan Zhang, Xiaoshan Yang+4

Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing · Institute of Artificial Intelligence, Beihang University, China · MAIS, Institute of Automation, Chinese Academy of Sciences, China+2

Reward Modeling Preference Optimization Vision-Language Alignment RL for VLMs Heterogeneous Federated Learning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have broad potential in privacy-sensitive domains such as healthcare and finance, yet strict data-sharing constraints render centralized training infeasible. Federated Learning mitigates this issue by enabling decentralized training, but practical deployments face challenges due to client heterogeneity in computational resources, application requirements, and model architectures. Under extreme model and data heterogeneity, replacing parameter aggregation with preference-based collaboration offers a more suitable interface, as it eliminates the need for direct parameter or data exchange. Motivated by this, we propose MoR, a federated alignment framework that combines GRPO with Mixture-of-Rewards for heterogeneous VLMs. In MoR, each client locally trains a reward model from local preference annotations, capturing specific evaluation signals without exposing raw data. To combine these heterogeneous supervision signals, MoR introduces a Mixture-of-Rewards mechanism with learned routing, which adaptively fuses client reward models according to the input and alignment objective. The server then optimizes a base VLM using GRPO with a KL penalty to a reference model, enabling preference alignment without requiring client models to share architectures or parameters. Experiments on diverse public vision-language benchmarks demonstrate that MoR consistently outperforms federated alignment baselines in generalization and cross-client adaptability. Our approach provides a scalable solution for privacy-preserving alignment of heterogeneous VLMs under federated settings.

Abstract

Explore similar work

Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning

On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing