FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints

Lishan Yang; Wei Emma Zhang; Nam Kha Nguygen; Po Hu; Yanjun Shu; Weitong Chen; Mong Yuan Sim

cs.LGSep 1, 2025

FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints

Authors: Lishan Yang, Wei Emma Zhang, Nam Kha Nguygen, Po Hu, Yanjun Shu, Weitong Chen, Mong Yuan Sim

Organizations: Adelaide University · Central China Normal University · Harbin Institute of Technology

Abstract

Federated Learning with LoRA fine-tuning offers an efficient and privacy-aware solution for institutions to collaboratively leverage their large datasets to train VLLMs. However, participating institutions often possess heterogeneous computational resources, resulting in imbalanced LoRA ranks, which pose a major challenge for effective collaboration. In addition, real-world applications in domains such as healthcare and transportation frequently suffer from missing modalities due to user mistakes or device failures, which significantly degrade global model performance in federated settings. To the best of our knowledge, no prior work has addressed these two challenges simultaneously in federated VLLMs. To tackle these issues, we propose FediLoRA, a lightweight federated LoRA aggregation framework that effectively mitigates the impact of missing modalities in heterogeneous environment. FediLoRA is explicitly motivated by the observation that simple averaging and structured editing can jointly benefit both global and personalized models. Our approach achieves strong performance across multiple general-domain and medical-domain benchmark datasets. Additional experiments on healthcare data further demonstrate that FediLoRA is well-suited for practical, real-world deployment scenarios. Our code is released at https://github.com/gotobcn8/FediLoRA.

Figures & tables

Figure 1: Two challenges for current Federated VLLMs. (1) Left: Inconsistencies in clients’ local model configuration (heterogeneous rank), causing unstable and suboptimal aggregation. (2) Right: Missing modalities in different clients leads to poor multimodal representation learning and generalization.

Panel of Figure 2 — Figure 2: (a) The performance gap among global models and personalized models. The bidirectional arrow represents the percentage of the smaller value relative to the larger one. (b) The experiment investigates the impact of different LoRA editing strategies on local client performance. The blue curve represents the global performance used as a reference while there is a huge gap compared with None editing.

Figure 3: Visualization of the Aggregation Process for HetLoRA and Dimension-wise re-weighting of FediLoRA. $p_{i}=\frac{|\mathcal{D}_{k}|}{\sum_{j=1}^{K}|\mathcal{D}_{j}|}$ denotes the weights in FedAvg.

Figure 4: Layer-wise model editing. $A_{g,t-1}^{y}$ represents the $t-1$ -th epoch, $y$ -th global LoRA A matrix. LoRA-Editing happens at each ending of local fine-tuning and before aggregation.

Global Evaluation	Recaps-118K $\uparrow$				SAM-LLaVA $\uparrow$				Next-preference $\uparrow$
	Missing(30%)		Missing(60%)		Missing(30%)		Missing(60%)		Missing(30%)		Missing(60%)
	BLEU	RSUM	BLEU	RSUM	BLEU	RSUM	BLEU	RSUM	BLEU	RSUM	BLEU	RSUM
HetLoRA	9.97	21.60	15.36	25.14	13.16	25.39	2.01	7.37	10.29	32.96	2.33	9.34
FLoRA	9.21	30.91	13.81	25.13	13.16	24.57	12.41	24.47	9.96	32.70	10.13	28.40
FediLoRA	11.29	33.84	15.53	25.47	13.34	25.15	12.64	24.94	11.21	33.52	11.85	30.07
Personalized Evaluation

Table 1: Global and personalized performance comparisons. Bold font indicates the best performance. Higher values mean better results.

Panel of Figure 5 — Figure 5: Comparing editing methods. (a), (b) and (c) present the performance comparison between full-editing and FediLoRA on three datasets respectively. (d) shows the comparison of half-editing and FediLoRA.

Edited Matrix	Recaps-118K $\uparrow$	SAM-LLaVA $\uparrow$	Next-Preference $\uparrow$
LoRA-A	30.91	21.77	28.24
LoRA-B	29.03	18.06	26.84
Both	30.22	16.94	26.11
None	28.45	19.87	24.68

Table 2: Comparison of editing different LoRA matrices in three datasets. We set the missing ratio as 60%. All results are evaluated at the global model using the RSUM metric.

Panel of Figure 6 — Figure 6: Comparison of the Global Adapter L2 Norm between HetLoRA and FediLoRA under 30% and 60% missing ratios.

Figure 7: Performance of FediLoRA and baseline methods on the SLAKE dataset. The two bar clusters on the left show the global model accuracy, while the two clusters on the right present the client-side model accuracy.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

	Recaps-118K		SAM-LLaVA		Next-Preference
	BLEU	RSUM	BLEU	RSUM	BLEU	RSUM
Homogenous	15.96	27.16	8.08	16.95	11.19	31.34
Heterougenous	15.53	25.47	12.64	24.94	11.85	30.07

Appendix

Table 3: Global performance of FediLoRA on both homogeneous and heterogeneous rank configurations (60% missing ratio).

Distribution	Uniform	Long-tail-heavy	Long-tail-light	Normal
RSUM	19.66	26.64	18.87	27.51

Appendix

Table 4: Performance comparison in different distributions of heterogeneous ranks.

Panel of Figure 8 — Figure 8: Comparison of editing more layers in LoRA-A. (a) The global performance. (b) The client performance. Min K means editing the layers that have the top- K minimum similarities between global and client layer parameters. The evaluation is conducted on SAM-LLaVA dataset with 60% missing ratio.

Method	HetLoRA	FloRA	FediLoRA
Time per Epoch (s)	434.17	401.20	415.33

Appendix

Table 5: Time consumption per epoch (seconds), including both local training and global aggregation time.

Method	FediLoRA	CreamFL	CACMRN
Extra Storage (MiB)	16	$>$ 500	$>$ 2000

Appendix

Table 6: Comparison of extra memory storage requirements per client per epoch (in MiB) among different federated learning frameworks. CreamFL and CACMRN are representative approaches for handling missing modalities.

Panel of Figure 9 — Figure 9: Visualization of the edited layer. The height of each bar corresponds to the transformer layer index of the edited LoRA module in the LLaVA model, which contains a total of 32 Transformer blocks. Taller bars indicate layers closer to the output layer. The color gradient from red to blue reflects the similarity between the edited local module and its corresponding global counterpart, with red indicating lower similarity and blue indicating higher similarity.

Federated Fine-Tuning

Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.

See all 117 papers

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning
Sep 9, 2024Jianyi Zhang, Hao Frank Yang, Ang Li +5Multimodal Federated Learning Non-IID Federated Learning
Duke University · Johns Hopkins University · University of Maryland College Park +1
CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
Sep 27, 2026Yanan Ma, Qiyuan Chen, Zihan Fang +2Federated Fine-Tuning Federated Learning
City University of Hong Kong · The University of Hong Kong
FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
Aug 15, 2026Juseok Jeon, Ramy E. Ali, Doyun Kwon +3Federated Fine-Tuning Federated Learning
DGIST Daegu, Republic of Korea · Samsung San Diego, USA

Sep 9, 2024J/K move · Enter open · S save

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning

Jianyi Zhang, Hao Frank Yang, Ang Li, Xin Guo+4

Duke University · Johns Hopkins University · University of Maryland College Park+1

Multimodal Federated Learning Non-IID Federated Learning Heterogeneous Federated Learning

Open paper AskPDF arXiv

Previous studies on federated learning (FL) often encounter performance degradation due to data heterogeneity among different clients. In light of the recent advances in multimodal large language models (MLLMs), such as GPT-4v and LLaVA, which demonstrate their exceptional proficiency in multimodal tasks, such as image captioning and multimodal question answering. We introduce a novel federated learning framework, named Multimodal Large Language Model Assisted Federated Learning (MLLM-LLaVA-FL), which employs powerful MLLMs at the server end to address the heterogeneous and long-tailed challenges. Owing to the advanced cross-modality representation capabilities and the extensive open-vocabulary prior knowledge of MLLMs, our framework is adept at harnessing the extensive, yet previously underexploited, open-source data accessible from websites and powerful server-side computational resources. Hence, the MLLM-LLaVA-FL not only enhances the performance but also avoids increasing the risk of privacy leakage and the computational burden on local devices, distinguishing it from prior methodologies. Our framework has three key stages. Initially, we conduct global visual-text pretraining of the model. This pretraining is facilitated by utilizing the extensive open-source data available online, with the assistance of MLLMs. Subsequently, the pretrained model is distributed among various clients for local training. Finally, once the locally trained models are transmitted back to the server, a global alignment is carried out under the supervision of MLLMs to further enhance the performance. Experimental evaluations on established benchmarks, show that our framework delivers promising performance in the typical scenarios with data heterogeneity and long-tail distribution across different clients in FL.

Abstract

Federated Fine-Tuning

Explore similar work

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA