A Step Toward Federated Pretraining of Multimodal Large Language Models

Baochen Xiong; Yifan Xu; Xiaoshan Yang; Yaguang Song; Yaowei Wang; Changsheng Xu

cs.LGMar 25, 2026

A Step Toward Federated Pretraining of Multimodal Large Language Models

Authors: Baochen Xiong, Yifan Xu, Xiaoshan Yang, Yaguang Song, Yaowei Wang, Changsheng Xu

Organizations: MAIS, Institute of Automation, Chinese Academy of Sciences · Pengcheng Laboratory · School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS) · King Abdullah University of Science and Technology (KAUST)

Abstract

The rapid evolution of Multimodal Large Language Models (MLLMs) is bottlenecked by the saturation of high-quality public data, while vast amounts of diverse multimodal data remain inaccessible in privacy-sensitive silos. Federated Learning (FL) offers a promising solution to unlock these distributed resources, but existing research focuses predominantly on fine-tuning, leaving the foundational pre-training phase largely unexplored. In this paper, we formally introduce the Federated MLLM Alignment (Fed-MA) task, a lightweight pre-training paradigm that freezes the vision encoder and LLM while collaboratively training the cross-modal projector. We identify two critical challenges in this setting: (i) parameter interference in aggregating local projectors; and (ii) gradient oscillations in one-pass collaborative SGD. To address these challenges, we propose Fed-CMP, a pioneering framework for federated MLLM pre-training. Fed-CMP employs Canonical Reliability-Aware Aggregation, which constructs a canonical space to decompose client projectors into a shared alignment basis and client-specific coefficients, then performs reliability-weighted fusion to suppress parameter interference. Furthermore, Fed-CMP introduces Orthogonality-Preserved Momentum, which applies momentum to the shared alignment basis via orthogonal projection, accumulating historical optimization directions while preserving geometric structure. We construct four federated pre-training scenarios based on public datasets, and extensive experiments validate that Fed-CMP significantly outperforms existing baselines.

Figures & tables

Figure 1: A step toward Federated MLLMs: Comparison among conventional Federated Learning, centralized MLLMs training, and the proposed Federated MLLMs Alignment (Fed-MA) task.

Figure 2: Overview of the proposed Federated MLLM pretraining framework (Fed-CMP).

Figure 3: t-SNE visualization of heterogeneous clustering on the CC12M dataset, utilizing 5 distinct clusters across 4 similarity metrics.

Method	MM-Vet	MMBench	SEED	LLaVABench	POPE	MME	MMVP
Local Training	26.9	25.8	24.5	35.9	60.9	856.4	29.9
FedAvg	20.5	23.7	26.6	40.0	74.3	1056.4	33.6
FedAdam	19.4	24.6	25.3	39.2	69.6	1002.5	32.4
FedProx	20.0	21.4	21.9	38.4	68.4	991.6	30.3
MOON	22.5	25.5	26.9	41.2	73.5	937.6	31.9
FedAvg-TA	23.6	24.7	24.6	40.7	68.6	915.5	33.9

Table 1: Evaluation of MLLMs on multimodal benchmarks under the

\mathcal{M}_{I-I}

setting.

Method	MM-Vet	MMBench	SEED	LLaVABench	POPE	MME	MMVP
Local Training	25.4	30.1	19.6	38.5	72.8	958.9	30.5
FedAvg	25.8	33.9	20.3	39.6	74.9	1059.4	35.2
FedAdam	24.6	34.6	20.2	39.2	72.5	968.1	33.9
FedProx	22.9	29.1	18.4	36.9	70.8	871.6	28.2
MOON	25.6	34.5	22.7	40.1	72.0	989.3	36.4
FedAvg-TA	26.3	31.0	21.2	40.0	73.1	982.6	36.6

Table 2: Evaluation of MLLMs on multimodal benchmarks under the

\mathcal{M}_{T-T}

setting.

Ablation Modules			Evaluation Benchmarks
CSC	RWF	OPM	LLaVA-Bench	MMVet	MMBench
$\times$	$\times$	$\times$	40.0	20.5	23.7
$\checkmark$	$\times$	$\times$	43.1	26.2	28.2
$\times$	$\checkmark$	$\times$	43.2	28.7	29.0
$\times$	$\times$	$\checkmark$	40.2	21.3	25.0
$\checkmark$	$\checkmark$	$\times$	45.3	30.8	31.6

Table 3: Ablation studies under the

\mathcal{M}_{I-I}

setting.

Figure 4: Performance scores across communication rounds under the

\mathcal{M}_{I-I}

setting.

Experimental Settings			Evaluation Benchmarks
Mag	Qua	$\alpha$	MM-Vet	MMBench
$\times$	$\times$	$-$	30.4	31.0
$\checkmark$	$\times$	$-$	31.2	31.5
$\times$	$\checkmark$	$-$	31.6	31.7
$\checkmark$	$\checkmark$	0.5	32.0	32.1
$\checkmark$	$\checkmark$	1.0	32.5	32.7

Table 4: Experimental analysis of RWF components and sensitivity analysis of

\alpha

under the

\mathcal{M}_{I-I}

setting.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Datasets	Client 1	Client 2	Client 3	Client 4	Client 5	Total	Test Set	Metric
GQA	185K	198K	176K	192K	169K	0.92M	Test-balanced (12.6k)	EM $\uparrow$
OCRVQA	162K	178K	145K	168K	147K	0.80M	Test (100k)	Accuracy $\uparrow$

Appendix

Table 5: Data statistics for instruction tuning experiments.

Method	GQA					OCRVQA
Method	C1	C2	C3	C4	C5	C1	C2	C3	C4	C5
FedAvg	47.8	52.4	50.9	50.2	49.6	46.5	42.1	39.8	40.6	45.9
FedAvg-TIEs	48.2	53.5	51.3	51.4	50.0	47.8	42.5	40.9	41.0	47.2
Fed-CMP (Ours)	49.6	54.3	52.1	52.5	50.8	48.9	44.2	41.7	42.8	48.1

Appendix

Table 6: Instruction tuning performance across clients.

Method	MM-Vet	MMBench	SEED	LLaVABench	POPE	MME	MMVP
Local Training	27.2	28.9	23.6	37.3	66.5	1000.8	31.6
FedAvg	24.3	29.8	24.4	39.8	74.6	1052.0	34.2
FedAdam	22.5	29.3	22.6	39.1	71.9	995.4	33.1
FedProx	20.6	25.6	20.2	37.5	69.5	931.2	29.4
MOON	26.1	32.2	24.8	40.6	72.8	962.9	33.8
FedAvg-TA	26.4	28.5	22.8	40.2	71.0	949.6	35.2

Appendix

Table 7: Evaluation of MLLMs on multimodal benchmarks under the

\mathcal{M}_{Joint}

setting.

Method	MM-Vet	MMBench	SEED	LLaVABench	POPE	MME	MMVP
Local Training	27.8	24.9	25.1	36.7	62.1	999.5	31.2
FedAvg	22.4	26.9	27.8	41.3	75.6	1084.9	34.8
FedAdam	20.1	23.2	26.5	38.5	67.9	1033.6	33.1
FedProx	19.3	22.6	20.7	39.8	71.2	1010.7	31.4
MOON	23.7	27.4	28.2	42.1	74.1	956.3	33.8
FedAvg-TA	24.9	26.1	23.5	41.9	70.3	902.4	35.6

Appendix

Table 8: Evaluation of MLLMs on multimodal benchmarks under the

\mathcal{M}_{Cross}

setting.

Method	Metric	Cluster 0	Cluster 1	Cluster 2	Cluster 3	Cluster 4
Image-Image	Count	1M	1M	1M	1M	1M
Image-Image	Percentage	20.00%	20.00%	20.00%	20.00%	20.00%
Text-Text	Count	368K	1M	556K	1M	680K
Text-Text	Percentage	10.22%	27.73%	15.44%	27.73%	18.88%
Joint-Modal	Count	1M	615K	1M	1M	979K
Joint-Modal	Percentage	21.77%	13.39%	21.77%	21.77%	21.31%

Appendix

Table 9: Distribution Statistics for pretraining data clustering.

Figure 5: t-SNE visualization of clustering results using four distinct similarity metrics.

Figure 6: Randomly sampled image-text pairs from the image-image clustering results. Each row represents a single cluster.

Figure 7: Randomly sampled image-text pairs from the text-text clustering results. Each row represents a single cluster.

Figure 8: Randomly sampled image-text pairs from the joint image and text clustering results. Each row represents a single cluster.

Figure 9: Randomly sampled image-text pairs from the cross-modal image-text clustering results. Each row represents a single cluster.

Multimodal Federated Learning

Multimodal federated learning trains models across data silos with different data types while keeping raw data local.

See all 51 papers

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning
Sep 9, 2024Jianyi Zhang, Hao Frank Yang, Ang Li +5Multimodal Federated Learning Non-IID Federated Learning
Duke University · Johns Hopkins University · University of Maryland College Park +1
Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
Jul 13, 2026Jing Liu, Chenxuanyin Zou, Jiayang Ren +5Multimodal Federated Learning Generative Replay
Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada · College of Future Information Technology, Fudan University, Shanghai 200433, China · Department of Chemical and Biological Engineering, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada +3
Boosting Multimodal Federated Learning via Chained Modality Optimization
Jun 1, 2026Zixin Zhang, Fan Qi, Shuai Li +2Multimodal Federated Learning Federated Learning Aggregation
College of Computer Science, Inner Mongolia University, Hohhot, Inner Mongolia, China · School of Computer Science and Engineering, Tianjin University of Technology, Tianjin, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China

Sep 9, 2024J/K move · Enter open · S save

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning

Jianyi Zhang, Hao Frank Yang, Ang Li, Xin Guo+4

Duke University · Johns Hopkins University · University of Maryland College Park+1

Multimodal Federated Learning Non-IID Federated Learning Heterogeneous Federated Learning

Open paper AskPDF arXiv

Previous studies on federated learning (FL) often encounter performance degradation due to data heterogeneity among different clients. In light of the recent advances in multimodal large language models (MLLMs), such as GPT-4v and LLaVA, which demonstrate their exceptional proficiency in multimodal tasks, such as image captioning and multimodal question answering. We introduce a novel federated learning framework, named Multimodal Large Language Model Assisted Federated Learning (MLLM-LLaVA-FL), which employs powerful MLLMs at the server end to address the heterogeneous and long-tailed challenges. Owing to the advanced cross-modality representation capabilities and the extensive open-vocabulary prior knowledge of MLLMs, our framework is adept at harnessing the extensive, yet previously underexploited, open-source data accessible from websites and powerful server-side computational resources. Hence, the MLLM-LLaVA-FL not only enhances the performance but also avoids increasing the risk of privacy leakage and the computational burden on local devices, distinguishing it from prior methodologies. Our framework has three key stages. Initially, we conduct global visual-text pretraining of the model. This pretraining is facilitated by utilizing the extensive open-source data available online, with the assistance of MLLMs. Subsequently, the pretrained model is distributed among various clients for local training. Finally, once the locally trained models are transmitted back to the server, a global alignment is carried out under the supervision of MLLMs to further enhance the performance. Experimental evaluations on established benchmarks, show that our framework delivers promising performance in the typical scenarios with data heterogeneity and long-tail distribution across different clients in FL.

Abstract

Multimodal Federated Learning

Explore similar work

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning

Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning

Boosting Multimodal Federated Learning via Chained Modality Optimization