Can LoRA Fusion Support Cross-Domain Tasks in Cloud-Edge Collaboration?

Yatong Wang; Fali Wang; Naibin Gu; Zheng Lin; Zhengxiao Liu; Dingyu Yao; Zhiwei Zhang; Jianxin Shi

cs.DCApr 9, 2026

Can LoRA Fusion Support Cross-Domain Tasks in Cloud-Edge Collaboration?

Authors: Yatong Wang, Fali Wang, Naibin Gu, Zheng Lin, Zhengxiao Liu, Dingyu Yao, Zhiwei Zhang, Jianxin Shi,

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · The Pennsylvania State University, University Park, USA · Beihang University, Beijing, China

Abstract

Cloud-hosted large language models (LLMs) commonly rely on LoRA for domain adaptation, yet domain data are distributed across multiple edge devices and cannot be uploaded due to privacy constraints. This raises a fundamental question: how can knowledge from multiple private edges be integrated into a cloud LLM for cross-domain problem solving? A natural solution is to train LoRA adapters locally and fuse them in the cloud; however, existing pipelines rely on unrealistic assumptions that edge devices can host cloud-scale LLMs and are evaluated mainly on single-domain tasks. To address these limitations, we propose a prune-train-recover framework that enables local LoRA training on pruned models and privacy-preserving cloud integration. We further introduce MMLU-CD, a cross-domain benchmark that composes multiple domain samples into a single instance, enabling explicit evaluation of cross-domain problem solving. This allows us to ask a concrete question: Can existing LoRA fusion methods support cross-domain tasks in cloud-edge collaboration? Our empirical answer is negative. Existing LoRA fusion methods perform poorly on MMLU-CD, often underperforming the base LLM, revealing their inability to support cross-domain problem solving. We attribute this failure to parameter conflicts among LoRA adapters and propose a simple conflict-resolution module, LoRA-CR, which mitigates conflicting updates and improves LoRA fusion performance by up to 3.8%. These results identify conflict mitigation as a critical yet largely overlooked factor in cloud-edge LoRA fusion, warranting further investigation in future research.

Figures & tables

Figure 1: Cloud-edge collaboration for cross-domain tasks.

Figure 2: The Prune-Train-Recover Framework.

Split	Subset	Amount
In-Domain (Training Set)	Humanities	4,178
	Social Sciences	2,731
	STEM	2,790
	Total	9,699
Cross-Domain (Test Set)	Hum & SC	500
	SC & STEM	500

Table 1: Statistics of the MMLU-CD dataset, including in-domain training set and cross-domain test sets.

Figure 3: Comparison of LoRA fusion methods on three MMLU-CD cross-domain test subsets using base model LLaMA-3-8B.

Method	SC&STEM	$\Delta$	Hum&STEM	$\Delta$	Hum&SC	$\Delta$	Average	Avg. $\Delta$
Base Model	33.80	–	37.63	–	44.03	–	38.49	–
FedAvg	33.67	–	37.82	–	43.26	–	38.25	–
+ LoRA-CR	33.24	$-0.43$	39.03	$+1.21$	45.83	$+2.57$	39.37	$+1.12$
FedSA-LoRA	31.00	–	34.91	–	44.14	–	36.68	–
+ LoRA-CR	33.67	$+2.67$	38.41	$+3.50$	47.97	$+3.83$	40.02	$+3.34$
FFA-LoRA	33.14	–	37.44	–	43.60	–	38.06	–

Table 2: Accuracy (%) on cross-domain MMLU-CD test subsets before and after conflict resolution.

Figure 4: Conflict scores

\bar{c}

on cross-domain MMLU-CD test subsets before and after conflict resolution using LLaMA-3-8B.

Figure 5: Effect of pruning ratio on the Hum&SC cross-domain subtask using FedSA-LoRA with LLaMA-3-8B.

Figure 6: An illustration of a cross-domain example and FFA-LoRA response before and after LoRA-CR . The query requires jointly using philosophical and biological knowledge.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Ratio	Exact Ratio	Param Before	Param After
0.2	0.16	8B (8,030,261,248)	6.7B (6,734,745,600)
0.4	0.34		5.2B (5,271,851,008)
0.6	0.50		3.9B (3,976,728,576)
0.8	0.69		2.5B (2,513,833,984)

Appendix

Table 3: Parameter Reduction under Different Ratios.

Domain	Subjects
Humanities	Formal Logic; High School European History; High School US History; High School World History; International Law; Jurisprudence; Logical Fallacies; Moral Disputes; Moral Scenarios; Philosophy; Prehistory; Professional Law; World Religions
Social Sciences	Econometrics; High School Geography; High School Gov’t and Politics; High School Macroeconomics; High School Microeconomics; High School Psychology; Human Sexuality; Professional Psychology; Public Relations; Security Studies; Sociology; US Foreign Policy
STEM	Abstract Algebra; Anatomy; Astronomy; College Biology; College Chemistry; College Computer Science; College Mathematics; College Physics; Computer Security; Conceptual Physics; Electrical Engineering; Elementary Mathematics; High School Biology; High School Chemistry; High School Computer Science; High School Mathematics; High School Physics; High School Statistics; Machine Learning

Appendix

Table 4: Subject composition of the three domains in MMLU.

Step 1: Mergeability Assessment

Instruction: Evaluate whether the following two QA pairs are suitable for creating a cross-domain question. Determine if the concepts naturally integrate or if they are disjoint.

Criteria:

\bullet

If concepts from both pairs can be combined naturally

\rightarrow

"Yes"

\bullet

If they are too unrelated

\rightarrow

"No"

Example (Positive):

Q1 (Finance): How do banks define high-risk transaction patterns?

Q2 (ML): How does anomaly detection identify unusual behaviors?

Appendix

Table 5: Prompts designed for the three-stage cross-domain data synthesis pipeline.

Domain	Train	Val	Test	Total
Humanities	4,178	522	523	5,223
Social Sciences	2,731	341	342	3,414
STEM	2,790	348	350	3,488
In-Domain Total	9,699	1,211	1,215	12,125

Appendix

Table 6: Dataset statistics of MMLU-CD. The training data follows the original MMLU subject taxonomy with three disjoint domain groups. Cross-domain test sets are newly constructed and do not overlap with in-domain test splits.

Model Merging

Model merging combines trained model weights or adapters to reuse capabilities while reducing the cost of joint training or ensembles.

See all 192 papers

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
Sep 27, 2026Yanan Ma, Qiyuan Chen, Zihan Fang +2Federated Fine-Tuning Federated Learning
City University of Hong Kong · The University of Hong Kong
Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment
May 20, 2026Shuaida He, Liwen Chen, Long FengLLM Fine-Tuning Low-Rank Adaptation
School of Computing & Data Science, The University of Hong Kong
CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimization
Sep 14, 2026Victor H. Chen, Hairui Yu, Stella K. Chung +1Bilevel Optimization Federated Fine-Tuning

Sep 27, 2026J/K move · Enter open · S save

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Yanan Ma, Qiyuan Chen, Zihan Fang, Xianhao Chen+1

City University of Hong Kong · The University of Hong Kong

Federated Fine-Tuning Federated Learning Low-Rank Adaptation Federated Learning Aggregation

Open paper AskPDF arXiv

Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present CFLoRA, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates CFLoRA achieves

\mathcal{O}(1/\sqrt{T})

convergence rate of the original LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that CFLoRA achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.

Abstract

Model Merging

Explore similar work

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment

CIDERS: Cloud-Edge LLM Collaborative Learning via Accelerating Personalized Bilevel Optimization