UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity

Pengyu Wang; Baochen Xiong; Xiaoshan Yang; Yifan Xu; Zhang Qimeng; Haifeng Chen; Changsheng Xu

cs.LGAug 16, 2026

UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity

Authors: Pengyu Wang, Baochen Xiong, Xiaoshan Yang, Yifan Xu, Zhang Qimeng, Haifeng Chen, Changsheng Xu

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Pengcheng Laboratory, Shenzhen, China · King Abdullah University of Science and Technology, Thuwal, Saudi Arabia · Data Science and Artificial Intelligence Research Institute, China United Network Communications Group Co., Ltd., Beijing, China

Abstract

Vision-Language Models (VLMs) have demonstrated strong performance in multimodal understanding and generation. However, fine-tuning of VLMs typically relies on centralized data, which raises privacy concerns in certain domains (e.g. healthcare). Federated Learning (FL) provides a natural solution by enabling model training without sharing raw data. However, applying FL to VLM instruction tuning is highly challenging. VLMs have substantial parameter scales, and in real-world scenarios, clients exhibit significant heterogeneity in tasks, modalities, and model architectures. Existing methods mainly focus on simplified settings and are unable to handle such multi-dimensional heterogeneous scenarios. In this work, we study federated instruction tuning under joint heterogeneity in tasks, modalities, and model architectures. We propose UniFed-VLM, a unified federated instruction tuning framework for VLMs that addresses multiple types of heterogeneity. It consists of two key components: 1) Federated Compensated Subspace Aggregation (FedCSA), which performs subspace-aligned aggregation of parameter-efficient adapters with dynamic weighting and compensation to mitigate heterogeneity-induced conflicts; 2) Two-stage Collaborative Distillation (TCoD), which enables effective knowledge transfer across heterogeneous models via a Mutual Distillation Adapter (MDA) and a mixture-of-experts-based distillation strategy. We conduct experiments on multiple benchmark datasets, and the results show that UniFed-VLM achieves stronger average performance across diverse tasks compared with existing FL methods. The source code is available at: https://github.com/wangpengyu2004/UniFed-VLM.

Figures & tables

Figure 1: An overview of the federated setting with intertwined heterogeneities across models, tasks, and modalities.

Figure 2: Overview of UniFed-VLM. Each client consists of a local VLM and an MDA. The training involves two steps. In Step 1, homogeneous LoRA parameters from the same model group are aggregated via FedCSA with SVD-based subspace alignment, dynamic weighting, and compensation optimization. In Step 2, the aggregated parameters are broadcast to clients, where the local VLM and MDA perform two-stage knowledge distillation with different MDA architectures across stages.

Scenario	Method	Fed-Nature				Fed-Crossdomain
		VQA	Caption	VG	CLS	VQA	Caption	VG	CLS
		Acc $\uparrow$	CIDEr/R-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$	Acc $\uparrow$	CIDEr/R-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
Scenario 1	LOCAL	0.712	0.705/0.348	0.390	0.891	0.509	2.168/0.569	0.311	0.213
	FedAdam	0.709	0.763/0.361	0.499	0.910	0.541	1.201/0.527	0.319	0.222
	FedAvg	0.734	0.744/0.349	0.465	0.912	0.542	2.501/0.624	0.378	0.226
	FedProx	0.702	0.753/0.357	0.457	0.906	0.550	1.748/0.579	0.367	0.234

Table 1: Comparison of LOCAL, Centralized, and federated baselines with UniFed-VLM on the Fed-Nature and Fed-Crossdomain datasets under different heterogeneous settings.

Method	VQA	Caption	VG	CLS
Method	Acc $\uparrow$	CIDEr/R-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
UniFed-VLM	0.743	0.784/0.358	0.519	0.918
w/o FedCSA (w/ FedAvg)	0.708	0.746/0.361	0.445	0.907
w/o FedCSA (w/ FedProx)	0.711	0.781/0.365	0.484	0.901
w/o TCoD	0.707	0.766/0.361	0.458	0.911
w/o Compensation	0.726	0.774/0.357	0.503	0.915

Table 2: Ablation study on UniFed-VLM. ”w/” indicates replacing FedCSA with an alternative aggregation method.

Panel of Figure 3 — Figure 3: Convergence and performance comparisons on Fed-Nature under Scenario 1. (a) Average training loss across all tasks (including the MDA module). (b)–(d) Performance on classification, VQA, and visual grounding, respectively.

LoRA Rank	VQA	Caption	VG	CLS
LoRA Rank	Acc $\uparrow$	CIDEr/R-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
8 / 8	0.743	0.784/0.358	0.519	0.918
16 / 16	0.732	0.805/0.365	0.533	0.915
32 / 32	0.736	0.824/0.371	0.522	0.922
16 / 8	0.717	0.792/0.367	0.492	0.915
8 / 16	0.712	0.756/0.360	0.487	0.913

Table 3: Effect of homogeneous and heterogeneous LoRA ranks on the Fed-Nature dataset (Scenario 1).

Panel of Figure 4 — Figure 4: Loss curves and performance trends of FedAdam on caption generation (a) and VQA (b) under Scenario 2 (Fed-Nature).

$d_{h}$	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
1024	0.726	0.783/0.363	0.499	0.910
2048	0.743	0.784/0.358	0.519	0.918
2560	0.718	0.778/0.366	0.478	0.910

Table 4: Impact of hidden dimension on downstream tasks.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure B.1: Loss curves of the supplementary parameter optimization process under different numbers of iterations for the Show-O model on the Fed-Nature dataset.

Iter.	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
100	0.724	0.782/0.363	0.504	0.912
200	0.743	0.784/0.358	0.519	0.918
300	0.737	0.792/0.368	0.497	0.910

Appendix

Table C.1: Effect of different numbers of iterations for supplementary parameters on performance on Fed-Nature.

$\lambda_{\text{task}}^{MoE}$	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
0.1	0.730	0.778/0.355	0.520	0.919
0.2	0.710	0.787/0.363	0.535	0.911
0.3	0.743	0.784/0.358	0.519	0.918
0.4	0.706	0.783/0.364	0.524	0.906

Appendix

Table C.2: Effect of the MoE task loss weight

\lambda_{\text{task}}^{MoE}

$\alpha$	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
0.3	0.738	0.762/0.358	0.526	0.912
0.4	0.743	0.784/0.358	0.519	0.918
0.5	0.726	0.782/0.360	0.529	0.916
0.6	0.724	0.781/0.359	0.519	0.911

Appendix

Table C.3: Effect of coefficient

\alpha

Layers	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
2	0.694	0.776/0.363	0.513	0.912
3	0.743	0.784/0.358	0.519	0.918
4	0.721	0.847/0.374	0.516	0.907

Appendix

Table C.4: Effect of the number of projection layers in MDA.

$\lambda_{S1}$ - $\lambda_{S2}$	VQA	Caption Generation	Visual Grounding	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
0.7-0.1	0.729	0.777/0.355	0.513	0.912
0.7-0.3	0.739	0.729/0.352	0.515	0.918
0.7-0.2	0.743	0.784/0.358	0.519	0.918
0.6-0.2	0.727	0.788/0.362	0.520	0.914
0.8-0.2	0.722	0.821/0.371	0.517	0.905

Appendix

Table D.1: Effect of distillation weights

\lambda_{S1}

and

\lambda_{S2}

Method	VQA	Caption Generation	VG	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
FedAvg	0.725	0.777/0.348	0.347	0.892
FedProx	0.716	0.738/0.344	0.355	0.888
HetLoRA	0.717	0.766/0.342	0.342	0.891
pFedLoRA	0.571	0.665/0.324	0.307	0.837
Ours	0.738	0.765/0.346	0.365	0.899

Appendix

Table E.1: Evaluation under another heterogeneous task-model scenario on Fed-Nature.

Method	VQA	Caption Generation	VG	CLS
	Acc $\uparrow$	CIDEr/ROUGE-L $\uparrow$	IoU $\uparrow$	Acc $\uparrow$
Ours	0.743	0.784/0.358	0.519	0.918
Ours (w/ DP)	0.724	0.781/0.360	0.521	0.918

Appendix

Table E.2: Privacy-preserving evaluation of UniFed-VLM on Fed-Nature under Scenario 1. Differential privacy (DP) is applied to LoRA updates before federated aggregation.

Explore similar work

Oct 9, 2026cs.CV

BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models

Federated vision-language models (FedVLMs) represent one of the emerging frontiers of machine learning, aiming to integrate federated learning within the fine-tuning pipelines of vision-language models (VLMs). In this work, we introduce BLEND, a new framework designed to jointly enhance the personalization and generalization capabilities of FedVLMs. BLEND proposes a selective VLM parameter fine-tuning and aggregation strategy that consists of (i) global vision and text adapters shared across clients, and (ii) a personalized vision projection adapter tailored to each client. BLEND further introduces a novel loss function that blends the outputs of the global and personalized adapters to achieve personalization, while simultaneously promoting generalization through a principled anchor design implemented via the projection heads of the zero-shot VLM. Extensive experiments demonstrate that BLEND consistently outperforms state-of-the-art baselines, highlighting its ability to effectively balance personalization and generalization across diverse system settings. (Code is available at: https://github.com/payamsiabd/BLEND.git)

Payam Abdisarabshali, Fardis Nadimi, Kasra Borazjani +3

University at Buffalo–SUNY · Adobe · Zhejiang University

VLM Adaptation Federated Fine-Tuning Federated Learning Parameter-Efficient Fine-Tuning

Ask PDF

May 5, 2026cs.AI

Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models

Vision-Language Models (VLMs) have broad potential in privacy-sensitive domains such as healthcare and finance, yet strict data-sharing constraints render centralized training infeasible. Federated Learning mitigates this issue by enabling decentralized training, but practical deployments face challenges due to client heterogeneity in computational resources, application requirements, and model architectures. Under extreme model and data heterogeneity, replacing parameter aggregation with preference-based collaboration offers a more suitable interface, as it eliminates the need for direct parameter or data exchange. Motivated by this, we propose MoR, a federated alignment framework that combines GRPO with Mixture-of-Rewards for heterogeneous VLMs. In MoR, each client locally trains a reward model from local preference annotations, capturing specific evaluation signals without exposing raw data. To combine these heterogeneous supervision signals, MoR introduces a Mixture-of-Rewards mechanism with learned routing, which adaptively fuses client reward models according to the input and alignment objective. The server then optimizes a base VLM using GRPO with a KL penalty to a reference model, enabling preference alignment without requiring client models to share architectures or parameters. Experiments on diverse public vision-language benchmarks demonstrate that MoR consistently outperforms federated alignment baselines in generalization and cross-client adaptability. Our approach provides a scalable solution for privacy-preserving alignment of heterogeneous VLMs under federated settings.

Shule Lu, Yujing Wang, Hainan Zhang +5

Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing · Institute of Artificial Intelligence, Beihang University, China · MAIS, Institute of Automation, Chinese Academy of Sciences, China +2

Reward Modeling Preference Optimization Vision-Language Alignment RL for VLMs Heterogeneous Federated Learning

Ask PDF

May 27, 2026cs.CV

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning

Federated Learning (FL) with pre-trained Vision-Language Models (VLMs) has emerged as a promising paradigm for various downstream tasks. By leveraging its strong representations, recent studies improve task adaptation under insufficient local data while preserving generalization. However, these methods emphasize fully local optimization with simple parameter aggregation,which can amplify inter-client optimization inconsistency and intra-client over-specialization under heterogeneous and full-data FL settings, making it difficult to balance global task adaptation and generalization. To address these challenges, we propose FedDTL, a novel federated VLM framework that decouples the image encoder and text encoder across clients and the server. Through decoupled encoder training with server-client modality alignment, FedDTL promotes coherent global semantic update and reduces inter-client optimization inconsistency, improving global task adaptation.To further mitigate intra-client over-specialization,we introduce a two-stage local fine-tuning, where a supervised fine-tuning stage enables rapid and reliable warm-start, followed by a reinforcement learning stage that enhances generalization. Extensive experiments on multiple benchmarks, including label skew and feature shift, demonstrate that FedDTL achieves an effective balance between global task adaptation and generalization under various FL data distributions in both few-shot and full-data regimes.

Yuting Ma, Lechao Cheng, Xiaohua Xu

School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui, China.

VLM Adaptation Vision-Language Alignment Federated Fine-Tuning Federated Learning Heterogeneous Federated Learning

Ask PDF