BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models

Payam Abdisarabshali; Fardis Nadimi; Kasra Borazjani; Naji Khosravan; Richeng Jin; Seyyedali Hosseinalipour

cs.CVOct 9, 2026

BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Models

Authors: Payam Abdisarabshali, Fardis Nadimi, Kasra Borazjani, Naji Khosravan, Richeng Jin, Seyyedali Hosseinalipour

Organizations: University at Buffalo–SUNY · Adobe · Zhejiang University

Abstract

Federated vision-language models (FedVLMs) represent one of the emerging frontiers of machine learning, aiming to integrate federated learning within the fine-tuning pipelines of vision-language models (VLMs). In this work, we introduce BLEND, a new framework designed to jointly enhance the personalization and generalization capabilities of FedVLMs. BLEND proposes a selective VLM parameter fine-tuning and aggregation strategy that consists of (i) global vision and text adapters shared across clients, and (ii) a personalized vision projection adapter tailored to each client. BLEND further introduces a novel loss function that blends the outputs of the global and personalized adapters to achieve personalization, while simultaneously promoting generalization through a principled anchor design implemented via the projection heads of the zero-shot VLM. Extensive experiments demonstrate that BLEND consistently outperforms state-of-the-art baselines, highlighting its ability to effectively balance personalization and generalization across diverse system settings. (Code is available at: https://github.com/payamsiabd/BLEND.git)

Figures & tables

Figure 1: Left: A FedVLM system with $N$ clients and a central server. Right: Schematic of the model introduced by BLEND for each client $n$ , encompassing (i) a set of trainable and aggregatable global vision and text adapters, which are residually added to the transformer layers (i.e., $\{\delta_{n,\ell}^{\mathsf{Vis}},\delta_{n,\ell}^{\mathsf{Txt}}\}_{\ell=1}^{L}$ ), (ii) a trainable personalized vision projection adapter, which is integrated in a new branch and kept local during the training (i.e., $\delta_{n}^{\mathsf{Proj}}$ ), and (iii) dual loss functions (i.e., $\mathcal{L}^{\mathsf{Fus}}_{n}$ and $\mathcal{L}^{\mathsf{Anch}}_{n}$ ), which are integrated into the final loss $\mathcal{L}_{n}$ .

Figure 2: Comparison of the personalization (Per.) and generalization (Gen.) of various fine-tuning strategies along with their corresponding harmonic mean (H-Mean). Results are averaged over the 10 datasets in Sec. 4 , where the system configuration is explained.

	UCF101			Oxford Pets			Stanford Cars			Flowers			Imagenet			FGVC Aircraft
	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean
Zero-shot CLIP	68.10 $\pm$ 0.00	75.18 $\pm$ 0.00	71.46 $\pm$ 0.00	89.47 $\pm$ 0.00	96.81 $\pm$ 0.00	93.00 $\pm$ 0.00	63.77 $\pm$ 0.00	74.94 $\pm$ 0.00	68.91 $\pm$ 0.00	69.42 $\pm$ 0.00	76.74 $\pm$ 0.00	72.90 $\pm$ 0.00	72.40 $\pm$ 0.00	68.08 $\pm$ 0.00	70.17 $\pm$ 0.00	27.49 $\pm$ 0.00	33.05 $\pm$ 0.00	30.02 $\pm$ 0.00
MMA+FedAvg	85.40 $\pm$ 0.93	75.95 $\pm$ 1.20	80.40 $\pm$ 1.08	95.12 $\pm$ 0.07	97.65 $\pm$ 0.08	96.37 $\pm$ 0.05	76.23 $\pm$ 1.63	69.60 $\pm$ 0.40	72.75 $\pm$ 0.82	96.93 $\pm$ 0.22	70.87 $\pm$ 1.19	81.87 $\pm$ 0.87	75.52 $\pm$ 2.03	69.18 $\pm$ 0.98	72.21 $\pm$ 0.32	38.92 $\pm$ 2.03	33.77 $\pm$ 0.98	36.10 $\pm$ 0.32
pFedMMA	89.95 $\pm$ 1.16	70.51 $\pm$ 0.66	79.04 $\pm$ 0.22	96.81 $\pm$ 0.81	95.08 $\pm$ 0.25	95.93 $\pm$ 0.33	67.45 $\pm$ 2.01	64.52 $\pm$ 0.72	65.95 $\pm$ 1.33	84.61 $\pm$ 2.42	69.89 $\pm$ 0.73	76.52 $\pm$ 0.62	49.15 $\pm$ 1.42	50.93 $\pm$ 1.02	50.03 $\pm$ 1.22	40.37 $\pm$ 5.66	29.14 $\pm$ 1.69	33.78 $\pm$ 3.04
pFedMoAP	93.36 $\pm$ 0.12	67.27 $\pm$ 0.02	78.19 $\pm$ 0.04	99.18 $\pm$ 0.02	84.84 $\pm$ 0.10	91.45 $\pm$ 0.01	92.00 $\pm$ 0.20	68.50 $\pm$ 0.05	78.53 $\pm$ 0.07	98.83 $\pm$ 0.15	71.49 $\pm$ 0.00	82.96 $\pm$ 0.05	86.65 $\pm$ 0.00	62.56 $\pm$ 0.03	72.66 $\pm$ 0.02	65.51 $\pm$ 0.08	18.58 $\pm$ 0.11	28.95 $\pm$ 0.01
FedPHA	92.61 $\pm$ 0.70	50.29 $\pm$ 2.31	65.14 $\pm$ 1.82	98.81 $\pm$ 0.63	70.06 $\pm$ 2.75	81.95 $\pm$ 1.89	91.90 $\pm$ 0.55	44.52 $\pm$ 1.31	59.96 $\pm$ 1.07	98.73 $\pm$ 0.21	49.77 $\pm$ 2.86	66.12 $\pm$ 2.53	90.49 $\pm$ 0.14	39.71 $\pm$ 2.26	55.20 $\pm$ 2.33	66.62 $\pm$ 0.44	13.00 $\pm$ 2.97	21.59 $\pm$ 4.12

Table 1: Performance results (accuracy, %). The row

\Delta

denotes the performance gap between BLEND and the best-performing baseline. Standard deviations across three random seeds are shown via sub-indices. In the bottom rows, statistical significance is assessed using a two-sided paired

t

-test on seed-level H-Mean scores between BLEND and the strongest baseline, with improvements statistically significant at

p<0.05

Rank ( $r$ )	Per.	Gen.	H-Mean
No adapter	89.50	72.31	79.95
4	92.25	74.69	82.05
8	92.45	74.70	82.10
16	92.91	74.62	82.15
32	92.93	74.39	81.97
64	92.91	74.09	81.74

Table 2: Effect of the rank

r{=}r_{x}{=}r_{w}

on performance.

Rank ( $r_{p}$ )	Per.	Gen.	H-Mean
4	89.99	75.00	81.44
8	90.45	74.83	81.73
16	91.00	74.86	81.73
32	91.59	74.91	81.96
64	92.12	74.64	81.92
128	92.57	74.71	82.12

Table 3: Effect of the rank

r_{p}

on performance.

Panel of Figure 3 — Figure 3: Effect of injecting adapters at various transformer layers (left plot). Effect of varying $\gamma$ (middle plot) and $\lambda$ (right plot), which are used in ( 17 ) and ( 22 ), respectively, on the performance.

Agg. Strategy	Per.	Gen.	H-Mean
BLEND	92.91	74.62	82.15
Full	83.35	75.41	79.01
NoAgg	89.12	69.71	77.71
Only Vision	74.64	70.02	71.82
Only Text	93.84	67.31	77.04

Table 4: Comparison of different aggregation strategies.

Appendix figures & tables85 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Total Number of Exchanged Params.	Communication Overhead (MB)	Per.	Gen.	H-Mean
MMA+FedAVG	675K	64.33	82.69	73.14	77.34
BLEND	507K	48.38	92.92	74.61	82.15

Appendix

Table 5: Comparable communicated parameters (high budget). Communication overhead is measured as the total number of bits transmitted to the server over 50 global rounds, reported in MB.

Method	Total Number of Exchanged Params.	Communication Overhead (MB)	Per.	Gen.	H-Mean
pFedMMA	8.4K	0.81	77.99	66.69	71.48
pFedMOAP	8.2K	0.78	92.03	66.48	75.98
FedPHA	8.2K	0.78	92.35	47.90	61.15
FedOTP	8.2K	0.78	85.92	32.57	45.39
PromptFL	8.2K	0.78	78.75	69.95	73.61
PromptFolio	8.2K	0.78	85.75	50.84	61.85

Appendix

Table 6: Comparable communicated parameters (low budget). Communication overhead is measured as the total number of bits transmitted to the server over 50 global rounds, reported in MB.

Method	Total Number of Trainable Params.	Per.	Gen.	H-Mean
MMA+FedAVG	675K	82.69	73.14	77.34
pFedMMA	675K	77.99	66.69	71.48
BLEND	536K	92.67	74.15	81.79

Appendix

Table 7: Comparable trainable parameters (high budget).

Method	Total Number of Trainable Params.	Per.	Gen.	H-Mean
pFedMOAP	74.2K	92.03	66.48	75.98
FedPHA	16.4K	92.35	47.90	61.15
FedOTP	16.4K	85.92	32.57	45.39
PromptFL	8.2K	78.75	69.95	73.61
PromptFolio	8.2K	85.75	50.84	61.85
FedPGP	20.6K	87.88	64.38	72.86

Appendix

Table 8: Comparable trainable parameters (low budget).

Metric	BLEND	pFedMMA	MMA+FedAvg	FedPHA	FedOPT	FedPGP	PromptFL	pFedMOAP	PromptFolio
Time (s)	11.18	11.53	11.59	13.35	9.70	15.19	10.30	10.01	10.11
Mean GPU (GB)	0.62	0.60	0.61	0.63	0.59	1.28	0.71	0.76	0.74

Appendix

Table 9: Comparison of per-round wall-clock time and average GPU memory usage across FL methods.

Figure 4: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) averaged across all datasets .

Figure 5: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) for the UCF101 dataset.

Figure 6: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) for the OxfordPets dataset.

Figure 7: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) for the Stanford-Cars dataset.

Figure 8: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) for the Flowers dataset.

Figure 9: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) for the FGVC Aircraft dataset.

Figure 10: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) for the Food101 dataset.

Figure 11: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) for the DTD dataset.

Figure 12: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) for the EuroSAT dataset.

Figure 13: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) for the Caltech dataset.

Figure 14: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of

\alpha^{\mathsf{Dir}}

(data heterogeneity) for the ImageNet dataset.

	UCF101			Oxford Pets			Stanford Cars			Flowers			Imagenet			FGVC Aircraft
	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean
Zero-shot CLIP	68.10 $\pm$ 0.00	75.18 $\pm$ 0.00	71.46 $\pm$ 0.00	89.47 $\pm$ 0.00	96.81 $\pm$ 0.00	93.00 $\pm$ 0.00	63.77 $\pm$ 0.00	74.94 $\pm$ 0.00	68.91 $\pm$ 0.00	69.42 $\pm$ 0.00	76.74 $\pm$ 0.00	72.90 $\pm$ 0.00	72.40 $\pm$ 0.00	68.08 $\pm$ 0.00	70.17 $\pm$ 0.00	27.49 $\pm$ 0.00	33.05 $\pm$ 0.00	30.02 $\pm$ 0.00
MMA+FedAvg	79.22 $\pm$ 1.17	77.27 $\pm$ 0.98	78.23 $\pm$ 0.83	95.96 $\pm$ 0.10	97.50 $\pm$ 0.11	96.73 $\pm$ 0.04	75.86 $\pm$ 0.71	69.71 $\pm$ 0.79	72.66 $\pm$ 0.74	94.31 $\pm$ 0.29	72.29 $\pm$ 1.72	81.83 $\pm$ 1.02	77.15 $\pm$ 0.22	69.10 $\pm$ 0.30	72.90 $\pm$ 0.07	35.34 $\pm$ 0.13	34.67 $\pm$ 0.45	35.00 $\pm$ 0.26
pFedMMA	92.85 $\pm$ 0.51	69.35 $\pm$ 0.70	79.40 $\pm$ 0.53	97.19 $\pm$ 0.96	94.97 $\pm$ 0.24	96.06 $\pm$ 0.37	77.73 $\pm$ 2.77	62.93 $\pm$ 0.54	69.52 $\pm$ 0.92	86.38 $\pm$ 1.12	72.15 $\pm$ 0.29	78.62 $\pm$ 0.46	70.39 $\pm$ 2.26	58.47 $\pm$ 0.85	63.87 $\pm$ 1.44	40.40 $\pm$ 1.40	28.13 $\pm$ 0.45	33.16 $\pm$ 0.60
pFedMoAP	93.94 $\pm$ 0.35	67.22 $\pm$ 0.01	78.36 $\pm$ 0.12	99.63 $\pm$ 0.09	84.78 $\pm$ 0.01	91.61 $\pm$ 0.04	92.49 $\pm$ 0.54	68.45 $\pm$ 0.01	78.67 $\pm$ 0.20	96.73 $\pm$ 0.63	71.44 $\pm$ 0.01	82.18 $\pm$ 0.22	90.21 $\pm$ 0.00	62.49 $\pm$ 0.00	73.84 $\pm$ 0.00	73.23 $\pm$ 2.16	18.53 $\pm$ 0.00	29.58 $\pm$ 0.18
FedPHA	93.37 $\pm$ 0.56	51.90 $\pm$ 3.30	66.64 $\pm$ 2.58	99.82 $\pm$ 0.06	56.42 $\pm$ 3.71	72.02 $\pm$ 3.02	93.80 $\pm$ 0.70	41.06 $\pm$ 0.10	57.12 $\pm$ 0.03	95.07 $\pm$ 0.46	43.03 $\pm$ 1.25	59.23 $\pm$ 1.28	91.60 $\pm$ 0.01	34.09 $\pm$ 0.50	49.69 $\pm$ 0.53	71.86 $\pm$ 0.00	8.03 $\pm$ 0.00	14.44 $\pm$ 0.00

Appendix

Table 10: Results for a scenario with

N=20

clients. Performance comparisons (measured in accuracy, %) between BLEND and baselines. The row

\Delta

denotes the performance gap between BLEND and the best-performing baseline. Standard deviations across three random seeds are shown through sub-indices. Statistical significance is assessed using a two-sided paired

t

-test on seed-level H-Mean scores between BLEND and the strongest baseline, with improvements statistically significant at

p<0.05

	UCF101			Oxford Pets			Stanford Cars			Flowers			Imagenet			FGVC Aircraft
	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean	Per.	Gen.	H-Mean
Zero-shot CLIP	68.10 $\pm$ 0.00	75.18 $\pm$ 0.00	71.46 $\pm$ 0.00	89.47 $\pm$ 0.00	96.81 $\pm$ 0.00	93.00 $\pm$ 0.00	63.77 $\pm$ 0.00	74.94 $\pm$ 0.00	68.91 $\pm$ 0.00	69.42 $\pm$ 0.00	76.74 $\pm$ 0.00	72.90 $\pm$ 0.00	72.40 $\pm$ 0.00	68.08 $\pm$ 0.00	70.17 $\pm$ 0.00	27.49 $\pm$ 0.00	33.05 $\pm$ 0.00	30.02 $\pm$ 0.00
MMA+FedAvg	77.84 $\pm$ 0.74	77.54 $\pm$ 1.06	77.68 $\pm$ 0.55	94.78 $\pm$ 0.23	97.50 $\pm$ 0.05	96.12 $\pm$ 0.10	72.45 $\pm$ 1.07	71.57 $\pm$ 0.40	72.00 $\pm$ 0.49	92.89 $\pm$ 0.49	73.19 $\pm$ 1.61	81.86 $\pm$ 0.82	77.57 $\pm$ 0.25	69.20 $\pm$ 0.30	73.15 $\pm$ 0.28	32.25 $\pm$ 0.96	35.15 $\pm$ 1.04	33.64 $\pm$ 0.98
pFedMMA	95.70 $\pm$ 0.45	71.16 $\pm$ 0.56	81.63 $\pm$ 0.39	92.72 $\pm$ 2.16	95.14 $\pm$ 0.22	93.90 $\pm$ 1.02	78.41 $\pm$ 2.96	58.46 $\pm$ 1.80	66.94 $\pm$ 1.61	92.62 $\pm$ 1.16	71.28 $\pm$ 0.81	80.55 $\pm$ 0.28	71.74 $\pm$ 1.02	55.29 $\pm$ 1.04	62.45 $\pm$ 1.05	64.05 $\pm$ 3.77	27.65 $\pm$ 0.43	38.61 $\pm$ 1.10
pFedMoAP	94.82 $\pm$ 0.09	67.21 $\pm$ 0.00	78.66 $\pm$ 0.03	98.66 $\pm$ 0.11	84.78 $\pm$ 0.00	91.20 $\pm$ 0.05	90.56 $\pm$ 0.63	68.44 $\pm$ 0.00	77.96 $\pm$ 0.23	98.46 $\pm$ 0.66	71.43 $\pm$ 0.00	82.80 $\pm$ 0.24	90.83 $\pm$ 0.00	62.50 $\pm$ 0.00	74.05 $\pm$ 0.00	81.52 $\pm$ 1.10	18.53 $\pm$ 0.00	30.19 $\pm$ 0.08
FedPHA	96.38 $\pm$ 0.56	47.72 $\pm$ 2.62	63.78 $\pm$ 2.22	98.86 $\pm$ 0.20	49.07 $\pm$ 2.02	65.56 $\pm$ 1.76	94.59 $\pm$ 0.18	39.27 $\pm$ 0.35	55.50 $\pm$ 0.38	98.83 $\pm$ 0.57	33.60 $\pm$ 0.66	50.15 $\pm$ 0.66	91.68 $\pm$ 0.02	36.76 $\pm$ 0.36	52.48 $\pm$ 0.37	83.14 $\pm$ 0.65	7.11 $\pm$ 1.30	13.06 $\pm$ 2.20

Appendix

Table 11: Results for a scenario with

N=30

clients. Performance comparisons (measured in accuracy, %) between BLEND and baselines. The row

\Delta

t

-test on seed-level H-Mean scores between BLEND and the strongest baseline, with improvements statistically significant at

p<0.05

Table 26

Figure 15: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class averaged across all datasets .

Figure 16: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class for the UCF101 dataset.

Figure 17: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class for the OxfordPets dataset.

Figure 18: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class for the Stanford-Cars dataset.

Figure 19: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class for the Flowers dataset.

Figure 20: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class for the FGVC Aircraft dataset.

Figure 21: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class for the Food101 dataset.

Figure 22: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class for the DTD dataset.

Figure 23: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class for the EuroSAT dataset.

Figure 24: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class for the Caltech dataset.

Figure 25: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing number of shots per class for the ImageNet dataset.

Table 38

Table 39

Table 40

Table 41

Table 42

Table 43

Table 44

Table 45

Figure 26: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings averaged across all datasets .

Figure 27: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings for the UCF101 dataset.

Figure 28: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings for the OxfordPets dataset.

Figure 29: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings for the Stanford-Cars dataset.

Figure 30: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings for the Flowers dataset.

Figure 31: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings for the FGVC Aircraft dataset.

Figure 32: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings for the Food101 dataset.

Figure 33: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings for the DTD dataset.

Figure 34: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings for the EuroSAT dataset.

Figure 35: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings for the Caltech dataset.

Figure 36: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing adapter injection settings for the ImageNet dataset.

Figure 37: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

averaged across all datasets .

Figure 38: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

for the UCF101 dataset.

Figure 39: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

for the Oxford Pets dataset.

Figure 40: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

for the Stanford Cars dataset.

Figure 41: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

for the Flowers dataset.

Figure 42: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

for the FGVC Aircraft dataset.

Figure 43: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

for the Food101 dataset.

Figure 44: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

for the DTD dataset.

Figure 45: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

for the EuroSAT dataset.

Figure 46: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

for the Caltech dataset.

Figure 47: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\gamma

for the ImageNet dataset.

Figure 48: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

averaged across all datasets .

Figure 49: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

for the UCF101 dataset.

Figure 50: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

for the OxfordPets dataset.

Figure 51: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

for the Stanford-Cars dataset.

Figure 52: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

for the Flowers dataset.

Figure 53: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

for the FGVC Aircraft dataset.

Figure 54: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

for the Food101 dataset.

Figure 55: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

for the DTD dataset.

Figure 56: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

for the EuroSAT dataset.

Figure 57: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

for the Caltech dataset.

Figure 58: Personalization and generalization test accuracies alongside their harmonic mean (H-Mean) under changing values of parameter

\lambda

for the ImageNet dataset.

Table 79

Table 80

Table 81

Table 82

Table 83

Table 84

Table 85

Table 86

Setting	Increasing Difficulty	H-Mean Gain vs. Best Baseline
Data heterogeneity (across 10 datasets in Sec. 4 )	$\alpha^{\mathrm{Dir}}:10\rightarrow 0.1\rightarrow 0.01$	$+0.9\rightarrow+2.9\rightarrow+4.8$
Number of clients (across 10 datasets in Sec. 4 )	$N:10\rightarrow 20\rightarrow 30$	$+4.8\rightarrow+5.6\rightarrow+6.1$
Real-World iNaturalist-User-120k Dataset (Appendix D )	$N:100\rightarrow 200$	$+4.9\rightarrow+5.6$
Local data scarcity (across 10 datasets in Sec. 4 )	#Shots: $32\rightarrow 16\rightarrow 8\rightarrow 4\rightarrow 2$	$+3.0\rightarrow+4.8\rightarrow+5.2\rightarrow+5.3\rightarrow+5.7$

Appendix

Table 58: Performance variation of BLEND across different settings. H-mean gains are reported relative to the best baseline.

Method	Per.	Gen.	H-Mean
BLEND with Zero-Shot G-Head	89.50	72.31	79.95
BLEND with Zero-Shot Distilled G-Head	92.83	72.35	80.54
BLEND with Running Anchor Loss	92.91	74.62	82.15

Appendix

Table 59: Comparison of BLEND’s running anchor with alternative zero-shot CLIP regularization strategies.

$\alpha^{\mathrm{Dir}}$	Optimal $\gamma$	H-Mean
10	0.03	79.66
0.1	0.06	80.86
0.01	0.09	82.15

Appendix

Table 60: Effect of fusion weight

\gamma

under varying levels of data heterogeneity and different numbers of clients.

Method	Per.	Gen.	H-Mean
BLEND (Sep-Per)	93.56	70.96	79.78
BLEND (Original)	92.91	74.62	82.15

Appendix

Table 61: Effect of explicitly supervising the personalized branch with a separate cross-entropy loss.

$\gamma$	0.0	0.06	0.08	0.09	0.1	0.3	1
H-Mean	78.86	82.25	82.38	82.39	82.36	79.47	73.47
$\lambda$	0.0	0.1	0.19	0.2	0.21	0.5	1
H-Mean	77.32	82.08	82.24	82.39	82.36	82.21	81.16

Appendix

Table 62: Hyperparameter sensitivity of the fusion weight

\gamma

and regularization coefficient

\lambda

on the held-out validation split. The highest H-mean for each hyperparameter is highlighted in bold.

Global adapter dim.	0	4	8	16	32	64	128	256	512
H-Mean	80.84	82.11	82.18	82.39	82.06	81.97	81.75	81.31	81.32

Appendix

Table 63: Sensitivity to the global and projection adapter dimensions on the held-out validation split. The highest H-mean for each adapter is highlighted in bold.

Explore similar work

UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity
Aug 16, 2026Pengyu Wang, Baochen Xiong, Xiaoshan Yang +4Multimodal Federated Learning Heterogeneous Federated Learning
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Pengcheng Laboratory, Shenzhen, China +2
Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning
May 27, 2026Yuting Ma, Lechao Cheng, Xiaohua XuVLM Adaptation Vision-Language Alignment
School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui, China.
Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models
May 5, 2026Shule Lu, Yujing Wang, Hainan Zhang +5Reward Modeling Preference Optimization
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing · Institute of Artificial Intelligence, Beihang University, China · MAIS, Institute of Automation, Chinese Academy of Sciences, China +2

Aug 16, 2026, cs.LGJ/K move · Enter open · S save

UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity

Pengyu Wang, Baochen Xiong, Xiaoshan Yang, Yifan Xu+3

State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Pengcheng Laboratory, Shenzhen, China+2

Multimodal Federated Learning Heterogeneous Federated Learning Federated Knowledge Distillation

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have demonstrated strong performance in multimodal understanding and generation. However, fine-tuning of VLMs typically relies on centralized data, which raises privacy concerns in certain domains (e.g. healthcare). Federated Learning (FL) provides a natural solution by enabling model training without sharing raw data. However, applying FL to VLM instruction tuning is highly challenging. VLMs have substantial parameter scales, and in real-world scenarios, clients exhibit significant heterogeneity in tasks, modalities, and model architectures. Existing methods mainly focus on simplified settings and are unable to handle such multi-dimensional heterogeneous scenarios. In this work, we study federated instruction tuning under joint heterogeneity in tasks, modalities, and model architectures. We propose UniFed-VLM, a unified federated instruction tuning framework for VLMs that addresses multiple types of heterogeneity. It consists of two key components: 1) Federated Compensated Subspace Aggregation (FedCSA), which performs subspace-aligned aggregation of parameter-efficient adapters with dynamic weighting and compensation to mitigate heterogeneity-induced conflicts; 2) Two-stage Collaborative Distillation (TCoD), which enables effective knowledge transfer across heterogeneous models via a Mutual Distillation Adapter (MDA) and a mixture-of-experts-based distillation strategy. We conduct experiments on multiple benchmark datasets, and the results show that UniFed-VLM achieves stronger average performance across diverse tasks compared with existing FL methods. The source code is available at: https://github.com/wangpengyu2004/UniFed-VLM.

Abstract

Explore similar work

UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning

Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models