Fractional-Order Federated Learning

Mohammad Partohaghighi; Roummel Marcia; YangQuan Chen

cs.LGFeb 17, 2026

Fractional-Order Federated Learning

Authors: Mohammad Partohaghighi, Roummel Marcia, YangQuan Chen

Organizations: Electrical Engineering and Computer Science, University of California at Merced, USA · Department of Applied Mathematics, University of California Merced, Merced, CA, USA · Mechatronics, Embedded Systems and Automation (MESA) Lab, Department of Mechanical Engineering, School of Engineering, University of California, Merced, CA 95343, USA

Abstract

Federated learning (FL) allows remote clients to train a global model collaboratively while protecting client privacy. Despite its privacy-preserving benefits, FL has significant drawbacks, including slow convergence, high communication cost, and non-independent-and-identically-distributed (non-IID) data. In this work, we present a novel FedAvg variation called Fractional-Order Federated Averaging (FOFedAvg), which incorporates Fractional-Order Stochastic Gradient Descent (FOSGD) to capture long-range relationships and deeper historical information. By introducing memory-aware fractional-order updates, FOFedAvg improves communication efficiency and accelerates convergence while mitigating instability caused by heterogeneous, non-IID client data. We compare FOFedAvg against a broad set of established federated optimization algorithms on benchmark datasets including MNIST, FEMNIST, CIFAR-10, CIFAR-100, EMNIST, the Cleveland heart disease dataset, Sent140, PneumoniaMNIST, and Edge-IIoTset. Across a range of non-IID partitioning schemes, FOFedAvg is competitive with, and often outperforms, these baselines in terms of test performance and convergence speed. On the theoretical side, we prove that FOFedAvg converges to a stationary point under standard smoothness and bounded-variance assumptions for fractional order $0<α\le 1$ . Together, these results show that fractional-order, memory-aware updates can substantially improve the robustness and effectiveness of federated learning, offering a practical path toward distributed training on heterogeneous data.

Figures & tables

\Theta_{1}=\Theta_{0}-\mu_{0}\nabla f(\Theta_{0});

Algorithm 1 Fractional Order SGD (FOSGD) [ 10 ]

\Theta_{t+1}=\sum_{k\in S_{t}}\frac{n_{k}}{n}\Theta_{t+1}^{(k)},

Algorithm 2 Fractional Order Federated Averaging Algorithm (FOFedAvg)

Symbol	Description
$\Gamma(z)$	Gamma function, generalizing the factorial
$\alpha$	Fractional order, $0<\alpha\leq 1$
$\mu_{t}$	Learning rate, $\mu_{t}=\mu_{0}/\sqrt{t+1}$
$\mu_{0}$	Initial learning rate (FOSGD algorithm)
$\delta$	Regularization constant, $\delta>0$
$\Theta_{t}$	Model parameters at iteration $t$

TABLE I : Key Mathematical Symbols Used in FOFedAvg

Fig. 1 : Comparison of federated algorithms on the MNIST dataset under a non-IID setting with 10 clients.

Fig. 2 : Comparison of federated algorithms on the CIFAR-10 dataset under a non-IID setting.

Fig. 3 : Comparison of federated learning algorithms on the EMNIST dataset.

Fig. 4 : Comparison of federated learning algorithms on the Cleveland heart disease dataset.

Fig. 5 : Test accuracy of federated learning algorithms on the Sent140 dataset.

Fig. 6 : Test accuracy of federated learning algorithms on the CIFAR-100 dataset.

Fig. 7 : Test accuracy of federated learning algorithms on the FEMNIST dataset.

Fig. 8 : Test accuracy comparison of FL algorithms on the PneumoniaMNIST dataset.

Fig. 9 : Test accuracy comparison of FL algorithms on the Edge-IIoTset dataset.

Fig. 10 : Test accuracies of federated learning methods under mild non-IID conditions.

Panel of Figure 14 — Fig. 11 : Test accuracy visualization for the MNIST dataset ( $B=50$ , $E=20$ ) across fractional orders and communication rounds in a non-IID setting.

Panel of Figure 15 — Fig. 12 : Left: 3D line visualization of test accuracy for MNIST ( $B=50$ , $E=20$ ) across fractional orders and communication rounds in a non-IID setting. Right: test accuracy for CIFAR-10 across fractional orders and communication rounds in an IID setting.

Panel of Figure 16 — Fig. 13 : 3D and 2D visualizations of test accuracy trends on CIFAR-10 across fractional orders and communication rounds.

Algorithm	Mean	Variance	Std. Dev.
FOFedAvg	0.9812	0.0016	0.0400
MOON	0.9577	0.0017	0.0412
FedAdam	0.9278	0.0080	0.0894
FedNova	0.8952	0.0228	0.1510
FedAvg	0.8953	0.0228	0.1510
FedProx	0.8949	0.0230	0.1517

TABLE II : Summary Statistics of Test Accuracy on MNIST with IID setting.

Algorithm	Accuracy CI (95%)
FOFedAvg	[0.9639, 0.9985]
MOON	[0.9398, 0.9756]
FedAdam	[0.8880, 0.9676]
FedNova	[0.8373, 0.9531]
FedAvg	[0.8374, 0.9532]
FedProx	[0.8368, 0.9530]

TABLE III : 95% Confidence Intervals for Test Accuracy on MNIST with IID setting.

Fig. 14 : Test accuracy progression over rounds for batch size 32.

Fig. 15 : Average test accuracy across batch sizes, grouped by client fraction and parallelism.

Fig. 16 : Average test accuracy across parallelism levels, grouped by client fraction and batch size.

Fig. 17 : Average test accuracy across client fractions, grouped by parallelism and batch size.

Fig. 18 : Mean test accuracy for FOFedAvg with different $\alpha$ values on the MNIST dataset (IID). The plot compares $\alpha=0.5$ , $\alpha=0.6$ , $\alpha=0.97$ , and $\alpha=1.5$ , showing the impact of $\alpha$ on model convergence and performance.

$\alpha$	Accuracy
	Mean	Std	CI
0.5	0.9673	0.0059	[0.9622, 0.9725]
0.6	0.9722	0.0059	[0.9670, 0.9774]
0.97	0.9673	0.0059	[0.9622, 0.9725]
1.5	0.9908	0.0006	[0.9903, 0.9913]

TABLE IV : Final Round Accuracy of FOFedAvg

Fig. 19 : Rounds to 75% accuracy for FOFedAvg and FedAvg on MNIST (IID setting).

Fig. 20 : Communication cost (MB) requiblack to reach 75% accuracy for FOFedAvg and FedAvg on MNIST (IID setting).

Fig. 21 : Cumulative communication cost (MB) for FOFedAvg and FedAvg on MNIST (IID setting).

Algorithm	10 Clients	100 Clients	1000 Clients
FOFedAvg	4	25	60
MOON	9	27	90
FedAdam	17	35	102
FedNova	8	19	189
FedAvg	17	39	215
FedProx	12	30	200

TABLE V : Communication Rounds to Achieve 60% Test Accuracy on MNIST (Severe Non-IID)

Algorithm	10 Clients	100 Clients	1000 Clients
FOFedAvg	24	662	36002
MOON	54	1381	64200
FedAdam	191	1502	68410
FedNova	126	6550	138590
FedAvg	249	7480	147100
FedProx	251	5381	146450

TABLE VI : Communication Cost to Achieve 60% Test Accuracy on MNIST (Severe Non-IID, MB)

Fig. 22 : Test accuracies of federated learning methods under Severe Non-IID conditions.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 921 papers

Mar 30, 2026

Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data

In this paper, we present Federated Robust Curvature Optimization (FedRCO), a novel second-order optimization framework designed to improve convergence speed and reduce communication cost in Federated Learning systems under statistical heterogeneity. Existing second-order optimization methods are often computationally expensive and numerically unstable in distributed settings. In contrast, FedRCO addresses these challenges by integrating an efficient approximate curvature optimizer with a provable stability mechanism. Specifically, FedRCO incorporates three key components: (1) a Gradient Anomaly Monitor that detects and mitigates exploding gradients in real-time, (2) a Fail-Safe Resilience protocol that resets optimization states upon numerical instability, and (3) a Curvature-Preserving Adaptive Aggregation strategy that safely integrates global knowledge without erasing the local curvature geometry. Theoretical analysis shows that FedRCO can effectively mitigate instability and prevent unbounded updates while preserving optimization efficiency. Extensive experiments show that FedRCO achieves superior robustness against diverse non-IID scenarios while achieving higher accuracy and faster convergence than both state-of-the-art first-order and second-order methods.

Yuanqiao Zhang, Tiantian He, Yuan Gao +5

Key Laboratory of Collaborative Intelligence Systems of Ministry of Education, Xidian University, Xi’an, Shaanxi, China · Centre for Frontier AI Research, Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · State Key Laboratory of ISN, School of Cyber Engineering, Xidian University, Xi’an, Shaanxi, China +2

Federated Learning Second-Order Optimization Non-IID Federated Learning Federated Learning Aggregation

Ask PDF

Sep 17, 2026

FedeRage: Provably Convergent Agnostic Federated Learning under General Client Drift

Federated learning (FL) enables collaborative model training without sharing raw data, but its performance degrades under non-IID data and stochastic client participation. Remedies built on classical Federated Averaging (FedAvg) typically presuppose that client participation probabilities are known to the server, which is rarely the case in deployed systems. We first discuss and then characterize the optimization problem that distributionally agnostic FedAvg actually solves when participation is entirely unknown, possibly highly skewed, and of variable size across rounds: uniform aggregation is shown to minimize a well-defined stochastic objective, weighted by the participation-induced marginal, at a standard

\mathcal{O}(1/\sqrt{T})

rate for convex and possibly nonsmooth losses. Building on this characterization, we propose Federated Risk-Averse Averaging (FedeRage), a risk-averse extension of FedAvg that embeds the Conditional Value-at-Risk (CVaR) into the local objective within a natural distributionally robust optimization (DRO) framework. FedeRage implicitly upweights high-loss and infrequently participating clients while adding only a single scalar per-client, and admits an

\mathcal{O}(κ/\sqrt{T})

rate in which the factor

κ

is the upper bound on the ``price" of risk aversion. In contrast with aggregation-alignment schemes based on optimal transport, which require the availability distribution as an input, FedeRage remains agnostic to it. Several experiments on three heterogeneous benchmarks indicate consistent improvements over state-of-the-art methods in accuracy, fairness, and convergence speed.

Herlock Rahimi, Dionysis Kalogerias

Federated Learning Federated Learning Aggregation Non-IID Federated Learning Distributionally Robust Optimization

Ask PDF

Mar 16, 2026

HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation

Fine-tuning large models on edge devices is severely hindered by the memory-intensive backpropagation (BP) in standard frameworks like federated learning and split learning. While substituting BP with zeroth-order optimization can significantly reduce memory footprints, it typically suffers from prohibitively degraded convergence speed. To resolve this dilemma, we propose Hybrid-Order Split Federated Learning (HO-SFL). By reformulating the split learning process within a Lagrangian framework, HO-SFL decouples the optimization landscape: The server performs precise first-order updates (i.e., BP), whereas clients conduct memory-efficient zeroth-order optimization. This hybrid design not only eliminates the need for client-side BP but also enables dimension-free model aggregation, drastically lowering communication costs. Crucially, we provide a theoretical convergence analysis, demonstrating that HO-SFL mitigates the dimension-dependent convergence slowdown of zeroth-order optimization, achieving a convergence rate comparable to first-order methods. Extensive experiments on tasks across vision and language modalities validate that HO-SFL achieves convergence speeds comparable to first-order baselines while significantly reducing communication costs and client memory footprints.

Qiyuan Chen, Xian Wu, Yi Wang +1

Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China.

Communication-Efficient Distributed Training Zeroth-Order Optimization Split Federated Learning Efficient Language Model Training Memory-Efficient Training Federated Learning

Ask PDF

Abstract

Federated Learning

Explore similar work

Taming the Instability: A Robust Second-Order Optimizer for Federated Learning over Non-IID Data

FedeRage: Provably Convergent Agnostic Federated Learning under General Client Drift

HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation