FedMIX-P: Mixing Local and Global Preconditioners for Federated Vision and Language Model Training

Junkang Liu

cs.LGOct 1, 2026

FedMIX-P: Mixing Local and Global Preconditioners for Federated Vision and Language Model Training

Authors: Junkang Liu

Organizations: Tianjin University

Abstract

Adaptive preconditioners accelerate model training, but heterogeneous client geometries can bias federated updates even when gradients are evaluated at the same model. Round-start synchronization alone cannot prevent this mismatch from reappearing during local training. We propose \texttt{FedMIX-P}, which mixes shared and local preconditioners at every local step, retaining local adaptation while reducing mean-squared operator mismatch by a factor of $λ^2$ . For smooth nonconvex objectives with stochastic gradients and partial participation, we establish an $O(R^{-1/2})$ stationarity bound using suitable stepsizes and a horizon-dependent mixing weight, without requiring local preconditioners to converge to one another. A two-client counterexample shows that fixed positive mixing can preserve a nonstationary fixed point. The theory covers bounded linear symmetric positive-definite preconditioners. Experiments with SOAP, Sophia, and Muon variants across vision and language tasks show improvements over corresponding local optimizers, including accuracy gains of up to $19.47$ percentage points and lower validation loss for 60M--350M language models. Full nonlinear and momentum-based updates require separate analysis.

Figures & tables

Method	ResNet-18				ViT-Tiny
	CIFAR-100		Tiny-ImageNet		CIFAR-100		Tiny-ImageNet
	Dir-0.1	Dir-0.05	Dir-0.1	Dir-0.05	Dir-0.1	Dir-0.05	Dir-0.1	Dir-0.05
FedAvg	60.17	56.75	47.48	43.80	27.24	23.42	15.68	14.05
SCAFFOLD	60.69	56.43	47.76	43.92	26.86	23.23	15.70	14.21
FedCM	66.61	62.65	41.16	36.00	28.23	25.74	18.88	18.15
Local AdamW	59.23	55.24	44.01	40.00	37.57	36.06	24.31	21.35

Table 1: Test accuracy (%) on CIFAR-100 and Tiny-ImageNet using ResNet-18 and ViT-Tiny over 300 communication rounds under Dir-0.1 and Dir-0.05. Experiments are conducted with 100 clients, 10% participation, batch size 50, and

K=50

. Best results are highlighted in

\mathbf{bold}

, and second-best results are underlined .

Method	ViT-Base				LLaMA
	CIFAR-100		Tiny-ImageNet		LLaMA 60M		LLaMA 130M		LLaMA 350M
	Acc. $\uparrow$	Loss $\downarrow$	Acc. $\uparrow$	Loss $\downarrow$	Loss $\downarrow$	Val PPL $\downarrow$	Loss $\downarrow$	Val PPL $\downarrow$	Loss $\downarrow$	Val PPL $\downarrow$
FedAvg	$89.28$	$0.332$	$86.97$	$0.421$	$5.156$	$173.47$	$5.258$	$192.10$	$5.354$	$211.45$
SCAFFOLD	$89.35$	$0.328$	$86.85$	$0.432$	$5.028$	$152.63$	$5.238$	$188.29$	$5.365$	$213.79$
FedCM	$87.44$	$0.592$	$84.24$	$0.708$	$5.231$	$186.98$	$5.356$	$211.88$	$5.426$	$227.24$
Local AdamW	$90.10$	$0.281$	$86.65$	$0.252$	$4.164$	$64.33$	$4.056$	$57.74$	$3.798$	$44.61$

Table 2: Comparison of different methods on vision and language tasks. On ViT-Base , results are reported under the Dir-0.1 partition over 100 communication rounds with 50 clients, 10% participation, batch size 16, and

K=50

local steps. On LLaMA , we report validation loss and validation perplexity ( Val PPL ) over 100 communication rounds with 20 clients, 20% participation, batch size 16, and

K=50

local steps.

Model / $\lambda$	0.0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
ResNet-18	$54.25$	$62.58$	$63.26$	$63.22$	$63.49$	$\mathbf{65.18}$	$\mathbf{65.18}$	$64.26$	$62.52$	$58.94$	$61.26$
ViT-Tiny	$38.65$	$45.24$	$46.11$	$46.22$	$47.01$	$\mathbf{48.45}$	$47.05$	$46.15$	$45.74$	$45.60$	$45.86$
$D_{P}$ of ViT-Tiny $\downarrow$	$16.98$	$15.78$	$13.55$	$11.52$	$9.55$	$8.20$	$7.05$	$6.19$	$4.40$	$2.27$	–

Table 3: Sensitivity to the mixing weight

\lambda

for FedMIX-P SOAP on CIFAR-100 under Dir-

0.05

after 300 communication rounds. Accuracies are reported in percentages. The results at

\lambda=0

0.5

, and

1

match Table 4 . Best accuracies are highlighted in bold, and second-best accuracies are underlined. A dash indicates an unreported drift measurement.

Method	Applied preconditioner	ResNet-18 Acc. (%)	ViT-Tiny Acc. (%)	Comm. (MB)	Time (s)	Comm. ratio
Local SOAP	$P_{i}^{r,k}$ ; retained local state	$58.16$	$41.68$	$22.80$	$5.56$	$1.0\times$
Shared only	$P^{r}$ ( $\lambda=0$ )	$54.25$	$38.65$	$45.60$	$5.12$	$2.0\times$
Local only after synchronization	$P_{i}^{r,k}$ ( $\lambda=1$ )	$61.26$	$45.86$	$45.60$	$5.56$	$2.0\times$
FedMIX-P SOAP	$(1-\lambda)P^{r}+\lambda P_{i}^{r,k}$ ( $\lambda=0.5$ )	$\mathbf{65.18}$	$\mathbf{48.45}$	$45.60$	$5.67$	$2.0\times$
FedMIX-P SOAP-light	Mixing with compressed shared state	$64.86$	$47.89$	$25.08$	$5.67$	$1.1\times$

Table 4: Ablation of shared and local preconditioning on CIFAR-100 under Dir-

0.05

after 300 communication rounds. Local SOAP and FedMIX-P SOAP results match Table 1 . Communication volume and computation time are measured per round on ViT-Tiny; communication ratios are relative to Local SOAP. Best accuracies are highlighted in bold.

Explore similar work

FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection
Oct 1, 2026Junkang LiuFederated Learning Fedavg
Tianjin University
FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning
Sep 29, 2026Mengjun Yi, Huaian Gu, Yinghao Ai +2Fedavg Model Fine-Tuning
State Key Laboratory for Novel Software Technology · School of Artificial Intelligence · School of Computer Science +1
CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning
Sep 29, 2026Mengjun Yi, Langxing Yang, Suhan Guo +2Hip-Lora Parameter-Efficient Adaptation
State Key Laboratory for Novel Software Technology · School of Artificial Intelligence School of Electronic Science and Engineering Nanjing University

Oct 1, 2026, cs.AIJ/K move · Enter open · S save

FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Junkang Liu

Tianjin University

Federated Learning Fedavg Low-Rank Structure Subspace Fragmentation Projection

Open paper AskPDF arXiv

Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create a problem we term \emph{subspace fragmentation}: local projections interact with data heterogeneity to bias aggregated directions, while aggregation can increase update rank and communication cost. Thus, accurate local gradient compression need not preserve global descent. We propose \texttt{FedLore}, which shares a low-rank optimization basis within each round and refreshes it across rounds. The shared basis enables exact aggregation in low-rank coordinates and eliminates the identified projection bias. Subspace refresh allows the accumulated model update to exceed the per-round rank budget. We characterize the aggregation bias and establish an

O(T^{-1/2})

stationarity bound for the projected-SGD variant under a global-gradient coverage condition and standard smoothness and variance assumptions, with bounded gradient heterogeneity. Experiments on vision and language tasks, including federated pre-training, show that \texttt{FedLore} outperforms the evaluated low-rank adapter baselines and matches or exceeds full-parameter training, while reducing communication and optimizer-state memory.

Abstract

Explore similar work

FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning