A Measured Communication-Quality Frontier for Federated LoRA Fine-Tuning with Adaptive Phase-Switching

Jerry Adams Franklin

cs.LGSep 11, 2026

A Measured Communication-Quality Frontier for Federated LoRA Fine-Tuning with Adaptive Phase-Switching

Authors: Jerry Adams Franklin

Organizations: Independent Researcher, Fairport, NY, 14450, USA.

Abstract

Federated fine-tuning of large language models with low-rank adaptation (LoRA) reduces the number of trainable parameters, but communication remains the dominant cost, and protocols are usually compared by parameter-count ratios rather than by measured bytes. This paper measures per-round upload and download bytes for five federated LoRA protocols, three of them from prior work, and places them on a single communication-quality frontier scored by held-out instruction-following loss. The frontier has a knee. ReverseAdaptive, which learns both LoRA factors before freezing one once the relative improvement in training loss falls below a dimensionless threshold, sits at that knee: it cuts measured round-trip communication by 40.5% relative to FLoRA at a held-out loss cost of 0.0063, and beats FFA-LoRA, which freezes that factor at initialization, by 0.0182 in held-out loss, more than twenty times the largest per-method seed standard deviation. The same threshold carries to LLaMA-3.2-3B without retuning, where it saves 30.0%.

Figures & tables

Parameter	Value
Base models	TinyLlama-1.1B-Chat-v1.0 Zhang et al. (2024) ; LLaMA-3.2-3B Dubey et al. (2024)
Datasets	Alpaca Taori et al. (2023) ; Dolly-15k Conover et al. (2023) ; 3000 samples each
Data partitions	IID; Dirichlet $\alpha{=}0.5$ ; Dirichlet $\alpha{=}0.1$
Clients / round	10 / 10 (full participation)
Rounds	15
Local epochs	1

Table 1: Experimental setup

Method	MB	Savings	Final loss	Held-out $\Delta$	$p$ vs. FLoRA
FLoRA	2578.13	–	$1.2608\pm 0.0005$	$-0.5992\pm 0.0001$	–
FedIT	2578.13	0.00%	$1.2602\pm 0.0016$	$-0.5990\pm 0.0002$	$0.588$
Two-Phase $K{=}8$	1863.13	27.73%	$1.2705\pm 0.0006$	$-0.5958\pm{<}0.0001$	$4.2\times 10^{-5}$
ReverseAdaptive ( $\tau{=}0.01$ )	1533.13	40.53%	$1.2749\pm 0.0006$	$-0.5929\pm 0.0008$	$1.1\times 10^{-5}$
FFA-LoRA	983.13	61.87%	$1.3031\pm 0.0012$	$-0.5746\pm 0.0004$	$1.7\times 10^{-4}$

Table 2: Measured communication-quality frontier. TinyLlama-1.1B, Alpaca-3k, IID, CUDA corpus, three seeds, mean

\pm

one sample standard deviation. Communication is protocol-deterministic and identical across seeds. Held-out

\Delta

loss is tuned minus base on the 500-example slice of Section 4.2 ; more negative is better. Differences quoted in the text are computed from unrounded values and may differ from the displayed cells in the final digit.

p

-values are paired

t

-tests against FLoRA on final loss; the complete set of paired tests on both metrics is in Appendix 10 . Methods: FLoRA Wang et al. (2024) , FedIT Zhang et al. (2024) , FFA-LoRA Sun et al. (2024) . The MB column is total round-trip communication over 15 rounds

Figure 1: Measured communication-quality frontier on Alpaca-3k, TinyLlama-1.1B, IID, CUDA corpus. Five protocols, three from prior work, at four distinct operating points; error bars are one sample standard deviation over three seeds, and communication is identical across seeds. Held-out

\Delta

loss is tuned minus base on the 500-example slice defined in Section 4.2 ; more negative is better. FLoRA (open circle) and FedIT (filled square) coincide at lower right. The segment from ReverseAdaptive to FFA-LoRA is markedly steeper than the segment preceding it, which Section 5.3 quantifies

Method	MB	Final loss	Held-out $\Delta$	Gap (Dolly)	Gap (Alpaca)
FLoRA	2578.13	$1.6544\pm 0.0021$	$-0.5330\pm 0.0010$	–	–
Two-Phase $K{=}8$	1863.13	$1.6643\pm 0.0021$	$-0.5287\pm 0.0010$	0.0043	0.0034
ReverseAdaptive ( $\tau{=}0.01$ )	1533.13	$1.6686\pm 0.0022$	$-0.5268\pm 0.0005$	0.0061	0.0063

Table 3: Dolly-15k replication. TinyLlama-1.1B, IID, CUDA corpus, three seeds, mean

\pm

one sample standard deviation. Gap is the held-out

\Delta

loss difference against the same dataset’s FLoRA baseline, which is the quantity comparable across datasets; held-out levels are not, since base-model losses differ (1.9352 on Alpaca, 2.2608 on Dolly-15k). Differences quoted in the text are computed from unrounded values and may differ from the displayed cells in the final digit. Method: FLoRA Wang et al. (2024) . Two-Phase and ReverseAdaptive are defined in Section 3 . The MB column is total round-trip communication over 15 rounds

Figure 2: Left: training loss trajectories on TinyLlama-1.1B IID, MPS corpus (mean

\pm

1 standard deviation over three seeds). Vertical lines mark ReverseAdaptive’s switch (round 6) and Two-Phase

K{=}8

’s phase boundary (round 8; the transition round, with full upload, is round 9). Right: cumulative communication on TinyLlama-1.1B (seed 42, deterministic across seeds). Slope changes mark phase transitions; look-ahead download accounting moves the broadcast slope one round before the upload slope

Method	ARC-Easy	BoolQ	HellaSwag	$\Delta$ vs. base (BoolQ)
Base (no adapter)	0.274	0.626	0.448	–
FLoRA	0.246	0.562	0.420	$-6.4$ pp
Two-Phase $K{=}8$	0.248	0.560	0.420	$-6.6$ pp
ReverseAdaptive	0.252	0.552	0.420	$-7.4$ pp
Spread (max–min)	0.006	0.010	0.000	–

Table 4: Downstream zero-shot accuracy, TinyLlama-1.1B, single training seed (42), MPS corpus, 500 examples per benchmark. MMLU omitted: base accuracy is at chance for 4-way multiple choice

Method	MMLU	ARC-Easy	BoolQ	HellaSwag
Base (no adapter)	0.544	0.832	0.706	0.540
FLoRA	$0.558\pm 0.007$	$0.836\pm 0.007$	$0.736\pm 0.007$	$0.535\pm 0.003$
Two-Phase $K{=}8$	$0.558\pm 0.002$	$0.837\pm 0.008$	$0.738\pm 0.010$	$0.539\pm 0.004$
ReverseAdaptive	$0.557\pm 0.003$	$0.839\pm 0.008$	$0.739\pm 0.004$	$0.538\pm 0.004$

Table 5: Downstream zero-shot accuracy, LLaMA-3.2-3B. Mean

\pm

1 standard deviation across three random seeds (42, 123, 456). 500 examples per benchmark. Cross-method spread is at most 0.5 percentage points on any benchmark and the per-method error bars overlap

$\tau$	Switch round	Final loss	Total comm. (MB)	Savings
0.001	11	1.2638	2083.13	19.20%
0.002	9	1.2675	1863.13	27.73%
0.005–0.2	6	1.2736	1533.13	40.53%

Table 6: ReverseAdaptive threshold sensitivity (TinyLlama IID, seed 42, MPS corpus)

Figure 3: ReverseAdaptive threshold sensitivity (TinyLlama IID, seed 42, MPS corpus). Loss values are not comparable with the CUDA results in Table 2 , and the

\tau

=0.001 and

\tau

=0.002 settings switch one round earlier on CUDA (Appendix 11 )

Method	Final loss	MB	Savings
FLoRA	$1.3724\pm 0.0009$	2625.0	0%
Two-Phase $K{=}8$	$1.3938\pm 0.0011$	1942.5	26.0%
ReverseAdaptive	$1.3938\pm 0.0036$	$1837.5\pm 105.0$	$30.0\pm 4.0$ %

Table 7: LLaMA-3.2-3B IID results. ReverseAdaptive uses

\tau{=}0.01

. The MB column is total round-trip communication over 15 rounds and savings are against FLoRA. Mean

\pm

1 standard deviation across three random seeds (42, 123, 456). ReverseAdaptive switch rounds: 7, 8, 9. FLoRA and Two-Phase

K{=}8

communication is protocol-deterministic; ReverseAdaptive communication varies with switch round

Figure 4: Scale validation. Left: TinyLlama-1.1B, CUDA corpus (mean

\pm

1 standard deviation over three seeds). Right: LLaMA-3.2-3B, MPS corpus (mean

\pm

1 standard deviation over three seeds). Both panels show the three federated methods on the same axes. Dotted polyline connects the three method points on each panel. The Pareto structure (downward slope from FLoRA to ReverseAdaptive) is preserved across scales. Y-axis ranges differ between panels because absolute training loss is model-dependent; the relative Pareto shape is the invariant being validated. ReverseAdaptive’s larger error bar at LLaMA-3.2-3B reflects across-seed variance in switch round (7, 8, 9)

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Seed	Final loss	Total MB	Switch / boundary
FLoRA	42	1.261408	2578.13	–
FLoRA	123	1.260479	2578.13	–
FLoRA	456	1.260500	2578.13	–
FedIT	42	1.260767	2578.13	–
FedIT	123	1.258420	2578.13	–
FedIT	456	1.261503	2578.13	–

Appendix

Table 1: Per-seed TinyLlama-1.1B IID results, CUDA corpus, backing Table 2 . Communication is deterministic across seeds. For Two-Phase the Switch/boundary column shows the predetermined phase boundary

K

, not an adaptive switch event; for FFA-LoRA,

A

is frozen from initialization (shown as “init”); for ReverseAdaptive it shows the round at which the plateau signal triggered

Setting	Seed	Final loss	Total MB	Switch round	Active clients
ReverseAdaptive, $\alpha{=}0.5$	42	1.285933	1533.13	6	10
ReverseAdaptive, $\alpha{=}0.5$	123	1.282625	1533.13	6	10
ReverseAdaptive, $\alpha{=}0.5$	456	1.252133	1533.13	6	10
ReverseAdaptive, $\alpha{=}0.1$	42	1.268353	1379.81	6	9
ReverseAdaptive, $\alpha{=}0.1$	123	1.288985	1533.13	6	10
ReverseAdaptive, $\alpha{=}0.1$	456	1.348083	1533.13	6	10

Appendix

Table 2: Non-IID results, MPS corpus, five seeds at

\alpha{=}0.1

. This table documents partition sensitivity rather than comparing methods, so the larger MPS seed set is retained; Table 1 follows the CUDA corpus because it backs a method comparison. Seeds 42 and 1000 at

\alpha{=}0.1

had 9 and 8 active clients due to empty Dirichlet partitions, which scales communication to

9/10

and

8/10

of the full value (1379.81 and 1226.50 MB against 1533.13). Cross-backend verification (Appendix 11 ) covers seeds 42, 123, and 456; seeds 789 and 1000 were executed on MPS only

Method	Seed	Final loss	Total MB	Switch
FLoRA	42	1.371683	2625.0	–
FLoRA	123	1.373402	2625.0	–
FLoRA	456	1.371998	2625.0	–
Two-Phase $K{=}8$	42	1.393776	1942.5	$K{=}8$
Two-Phase $K{=}8$	123	1.394877	1942.5	$K{=}8$
Two-Phase $K{=}8$	456	1.392772	1942.5	$K{=}8$

Appendix

Table 3: Per-seed LLaMA-3.2-3B IID results, MPS corpus, backing Table 7 . ReverseAdaptive uses

\tau{=}0.01

. Communication is deterministic given the switch round; the three ReverseAdaptive seeds occupy three adjacent lattice points

Method	Seed	Final loss	Held-out $\Delta$	Total MB
FLoRA	42	1.656762	$-0.531848$	2578.13
FLoRA	123	1.652934	$-0.533376$	2578.13
FLoRA	456	1.653426	$-0.533664$	2578.13
Two-Phase $K{=}8$	42	1.666616	$-0.527571$	1863.13
Two-Phase $K{=}8$	123	1.662685	$-0.529113$	1863.13
Two-Phase $K{=}8$	456	1.663475	$-0.529410$	1863.13

Appendix

Table 4: Per-seed TinyLlama-1.1B Dolly-15k IID results, CUDA corpus, backing Table 3 . Held-out

\Delta

is tuned minus base on the Dolly-15k held-out slice; more negative is better

TinyLlama-1.1B, Alpaca, IID
Comparison	$\Delta$ final loss	$p$	$\Delta$ held-out	$p$
Two-Phase $K{=}8$ vs. FLoRA	$+0.0098$	$4.2\times 10^{-5}$	$+0.0034$	$7.9\times 10^{-5}$
ReverseAdaptive vs. FLoRA	$+0.0141$	$1.1\times 10^{-5}$	$+0.0063$	$6.2\times 10^{-3}$
ReverseAdaptive vs. Two-Phase $K{=}8$	$+0.0043$	$3.4\times 10^{-4}$	$+0.0029$	$2.6\times 10^{-2}$
FFA-LoRA vs. ReverseAdaptive	$+0.0282$	$4.4\times 10^{-4}$	$+0.0182$	$3.8\times 10^{-4}$
FFA-LoRA vs. FLoRA	$+0.0423$	$1.7\times 10^{-4}$	$+0.0245$	$1.1\times 10^{-4}$

Appendix

Table 1: Paired

t

-tests, three seeds. TinyLlama-1.1B rows use the CUDA corpus; LLaMA-3.2-3B rows use MPS. Positive

\Delta

indicates the first method has higher loss. Held-out instruction-following loss was not computed for the LLaMA-3.2-3B corpus; quality at that scale is reported as final training loss and downstream zero-shot accuracy

Category	Runs
Agreed within setting-aware tolerance	27
Disagreed	4
Comparable pairs	31
No valid MPS reference (CUDA reported)	3
Total	34

Appendix

Table 1: Cross-backend verification summary, MPS against CUDA, 34 run pairs

Run	Config.	MPS	CUDA	Note
11	$\tau{=}0.001$ , IID	2083.13	1973.13	$s$ : 11 to 10
12	$\tau{=}0.002$ , IID	1863.13	1753.13	$s$ : 9 to 8
16	$\alpha{=}0.1$ , 123	1533.13	1533.13	spike rule
17	$\alpha{=}0.1$ , 456	1533.13	1643.13	$s$ : 6 to 7

Appendix

Table 2: Cross-backend disagreements. Every communication difference is one 110 MB lattice step. Run 16 failed only the consecutive-spike rule

Federated Fine-Tuning

Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.

See all 115 papers

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
Sep 27, 2026Yanan Ma, Qiyuan Chen, Zihan Fang +2Federated Fine-Tuning Federated Learning
City University of Hong Kong · The University of Hong Kong
FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation
Aug 4, 2026Shenghui Li, Thiemo VoigtFederated Learning Aggregation Communication-Efficient Distributed Training
Uppsala University, Uppsala, Sweden · Research Institutes of Sweden, Stockholm, Sweden
FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization
Oct 1, 2026Hang Zou, Chao Zhang, Yuzhi Yang +3Federated Learning Aggregation Low-Rank Adaptation
6G Research Center, Khalifa University, 127788 Abu Dhabi, UAE · Department of Computer Science, Central South University, 410083, Changsha, China · Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France

Sep 27, 2026J/K move · Enter open · S save

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Yanan Ma, Qiyuan Chen, Zihan Fang, Xianhao Chen+1

City University of Hong Kong · The University of Hong Kong

Federated Fine-Tuning Federated Learning Low-Rank Adaptation Federated Learning Aggregation

Open paper AskPDF arXiv

Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present CFLoRA, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates CFLoRA achieves

\mathcal{O}(1/\sqrt{T})

convergence rate of the original LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that CFLoRA achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.

Abstract

Federated Fine-Tuning

Explore similar work

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

FraQ: Efficient Coordinate-Space Recompression for Federated Low-Rank Adaptation

FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization