Fractional-Order Federated Learning

Mohammad Partohaghighi; Roummel Marcia; YangQuan Chen

cs.LGFeb 17, 2026

Fractional-Order Federated Learning

Authors: Mohammad Partohaghighi, Roummel Marcia, YangQuan Chen

Organizations: Electrical Engineering and Computer Science, University of California at Merced, USA · Department of Applied Mathematics, University of California Merced, Merced, CA, USA · Mechatronics, Embedded Systems and Automation (MESA) Lab, Department of Mechanical Engineering, School of Engineering, University of California, Merced, CA 95343, USA

Abstract

Federated learning (FL) allows remote clients to train a global model collaboratively while protecting client privacy. Despite its privacy-preserving benefits, FL has significant drawbacks, including slow convergence, high communication cost, and non-independent-and-identically-distributed (non-IID) data. In this work, we present a novel FedAvg variation called Fractional-Order Federated Averaging (FOFedAvg), which incorporates Fractional-Order Stochastic Gradient Descent (FOSGD) to capture long-range relationships and deeper historical information. By introducing memory-aware fractional-order updates, FOFedAvg improves communication efficiency and accelerates convergence while mitigating instability caused by heterogeneous, non-IID client data. We compare FOFedAvg against a broad set of established federated optimization algorithms on benchmark datasets including MNIST, FEMNIST, CIFAR-10, CIFAR-100, EMNIST, the Cleveland heart disease dataset, Sent140, PneumoniaMNIST, and Edge-IIoTset. Across a range of non-IID partitioning schemes, FOFedAvg is competitive with, and often outperforms, these baselines in terms of test performance and convergence speed. On the theoretical side, we prove that FOFedAvg converges to a stationary point under standard smoothness and bounded-variance assumptions for fractional order $0<α\le 1$ . Together, these results show that fractional-order, memory-aware updates can substantially improve the robustness and effectiveness of federated learning, offering a practical path toward distributed training on heterogeneous data.

Figures & tables

\Theta_{1}=\Theta_{0}-\mu_{0}\nabla f(\Theta_{0});

Algorithm 1 Fractional Order SGD (FOSGD) [ 10 ]

\Theta_{t+1}=\sum_{k\in S_{t}}\frac{n_{k}}{n}\Theta_{t+1}^{(k)},

Algorithm 2 Fractional Order Federated Averaging Algorithm (FOFedAvg)

Symbol	Description
$\Gamma(z)$	Gamma function, generalizing the factorial
$\alpha$	Fractional order, $0<\alpha\leq 1$
$\mu_{t}$	Learning rate, $\mu_{t}=\mu_{0}/\sqrt{t+1}$
$\mu_{0}$	Initial learning rate (FOSGD algorithm)
$\delta$	Regularization constant, $\delta>0$
$\Theta_{t}$	Model parameters at iteration $t$

TABLE I : Key Mathematical Symbols Used in FOFedAvg

Fig. 1 : Comparison of federated algorithms on the MNIST dataset under a non-IID setting with 10 clients.

Fig. 2 : Comparison of federated algorithms on the CIFAR-10 dataset under a non-IID setting.

Fig. 3 : Comparison of federated learning algorithms on the EMNIST dataset.

Fig. 4 : Comparison of federated learning algorithms on the Cleveland heart disease dataset.

Fig. 5 : Test accuracy of federated learning algorithms on the Sent140 dataset.

Fig. 6 : Test accuracy of federated learning algorithms on the CIFAR-100 dataset.

Fig. 7 : Test accuracy of federated learning algorithms on the FEMNIST dataset.

Fig. 8 : Test accuracy comparison of FL algorithms on the PneumoniaMNIST dataset.

Fig. 9 : Test accuracy comparison of FL algorithms on the Edge-IIoTset dataset.

Fig. 10 : Test accuracies of federated learning methods under mild non-IID conditions.

Panel of Figure 14 — Fig. 11 : Test accuracy visualization for the MNIST dataset ( $B=50$ , $E=20$ ) across fractional orders and communication rounds in a non-IID setting.

Panel of Figure 15 — Fig. 12 : Left: 3D line visualization of test accuracy for MNIST ( $B=50$ , $E=20$ ) across fractional orders and communication rounds in a non-IID setting. Right: test accuracy for CIFAR-10 across fractional orders and communication rounds in an IID setting.

Panel of Figure 16 — Fig. 13 : 3D and 2D visualizations of test accuracy trends on CIFAR-10 across fractional orders and communication rounds.

Algorithm	Mean	Variance	Std. Dev.
FOFedAvg	0.9812	0.0016	0.0400
MOON	0.9577	0.0017	0.0412
FedAdam	0.9278	0.0080	0.0894
FedNova	0.8952	0.0228	0.1510
FedAvg	0.8953	0.0228	0.1510
FedProx	0.8949	0.0230	0.1517

TABLE II : Summary Statistics of Test Accuracy on MNIST with IID setting.

Algorithm	Accuracy CI (95%)
FOFedAvg	[0.9639, 0.9985]
MOON	[0.9398, 0.9756]
FedAdam	[0.8880, 0.9676]
FedNova	[0.8373, 0.9531]
FedAvg	[0.8374, 0.9532]
FedProx	[0.8368, 0.9530]

TABLE III : 95% Confidence Intervals for Test Accuracy on MNIST with IID setting.

Fig. 14 : Test accuracy progression over rounds for batch size 32.

Fig. 15 : Average test accuracy across batch sizes, grouped by client fraction and parallelism.

Fig. 16 : Average test accuracy across parallelism levels, grouped by client fraction and batch size.

Fig. 17 : Average test accuracy across client fractions, grouped by parallelism and batch size.

Fig. 18 : Mean test accuracy for FOFedAvg with different $\alpha$ values on the MNIST dataset (IID). The plot compares $\alpha=0.5$ , $\alpha=0.6$ , $\alpha=0.97$ , and $\alpha=1.5$ , showing the impact of $\alpha$ on model convergence and performance.

$\alpha$	Accuracy
	Mean	Std	CI
0.5	0.9673	0.0059	[0.9622, 0.9725]
0.6	0.9722	0.0059	[0.9670, 0.9774]
0.97	0.9673	0.0059	[0.9622, 0.9725]
1.5	0.9908	0.0006	[0.9903, 0.9913]

TABLE IV : Final Round Accuracy of FOFedAvg

Fig. 19 : Rounds to 75% accuracy for FOFedAvg and FedAvg on MNIST (IID setting).

Fig. 20 : Communication cost (MB) requiblack to reach 75% accuracy for FOFedAvg and FedAvg on MNIST (IID setting).

Fig. 21 : Cumulative communication cost (MB) for FOFedAvg and FedAvg on MNIST (IID setting).

Algorithm	10 Clients	100 Clients	1000 Clients
FOFedAvg	4	25	60
MOON	9	27	90
FedAdam	17	35	102
FedNova	8	19	189
FedAvg	17	39	215
FedProx	12	30	200

TABLE V : Communication Rounds to Achieve 60% Test Accuracy on MNIST (Severe Non-IID)

Algorithm	10 Clients	100 Clients	1000 Clients
FOFedAvg	24	662	36002
MOON	54	1381	64200
FedAdam	191	1502	68410
FedNova	126	6550	138590
FedAvg	249	7480	147100
FedProx	251	5381	146450

TABLE VI : Communication Cost to Achieve 60% Test Accuracy on MNIST (Severe Non-IID, MB)

Fig. 22 : Test accuracies of federated learning methods under Severe Non-IID conditions.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 921 papers

Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data
Mar 30, 2026Yuanqiao Zhang, Tiantian He, Yuan Gao +5Federated Learning Second-Order Optimization
Key Laboratory of Collaborative Intelligence Systems of Ministry of Education, Xidian University, Xi’an, Shaanxi, China · Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · State Key Laboratory of ISN, School of Cyber Engineering, Xidian University, Xi’an, Shaanxi, China +2
FedeRage: Provably Convergent Agnostic Federated Learning under General Client Drift
Sep 17, 2026Herlock Rahimi, Dionysis KalogeriasFederated Learning Federated Learning Aggregation
HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation
Mar 16, 2026Qiyuan Chen, Xian Wu, Yi Wang +1Communication-Efficient Distributed Training Zeroth-Order Optimization
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China.

Mar 30, 2026J/K move · Enter open · S save

Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data

Yuanqiao Zhang, Tiantian He, Yuan Gao, Yixin Wang+4

Key Laboratory of Collaborative Intelligence Systems of Ministry of Education, Xidian University, Xi’an, Shaanxi, China · Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · State Key Laboratory of ISN, School of Cyber Engineering, Xidian University, Xi’an, Shaanxi, China+2

Federated Learning Second-Order Optimization Non-IID Federated Learning Federated Learning Aggregation

Open paper AskPDF arXiv

In this paper, we present Federated Robust Curvature Optimization (FedRCO), a novel second-order optimization framework designed to improve convergence speed and reduce communication cost in Federated Learning systems under statistical heterogeneity. Existing second-order optimization methods are often computationally expensive and numerically unstable in distributed settings. In contrast, FedRCO addresses these challenges by integrating an efficient approximate curvature optimizer with a provable stability mechanism. Specifically, FedRCO incorporates three key components: (1) a Gradient Anomaly Monitor that detects and mitigates exploding gradients in real-time, (2) a Fail-Safe Resilience protocol that resets optimization states upon numerical instability, and (3) a Curvature-Preserving Adaptive Aggregation strategy that safely integrates global knowledge without erasing the local curvature geometry. Theoretical analysis shows that FedRCO can effectively mitigate instability and prevent unbounded updates while preserving optimization efficiency. Extensive experiments show that FedRCO achieves superior robustness against diverse non-IID scenarios while achieving higher accuracy and faster convergence than both state-of-the-art first-order and second-order methods.

Abstract

Federated Learning

Explore similar work

Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data

FedeRage: Provably Convergent Agnostic Federated Learning under General Client Drift

HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation