Fuzzy-FedAvg: Server-Aware Client Selection for Federated Speech Emotion Recognition in Heterogeneous Environments

Farrukh Pervez; Siddique Latif; Moid Sandhu; Raja Jurdak

cs.SDNov 6, 2026

Fuzzy-FedAvg: Server-Aware Client Selection for Federated Speech Emotion Recognition in Heterogeneous Environments

Authors: Farrukh Pervez, Siddique Latif, Moid Sandhu, Raja Jurdak

Organizations: College of Aeronautical Engineering, National University of Sciences & Technology (NUST), Islamabad 24090, Pakistan · Queensland University of Technology (QUT), Brisbane QLD 4000, Australia · Commonwealth Scientific and Industrial Research Organisation (CSIRO), Brisbane, QLD 4029, Australia

Abstract

Federated speech emotion recognition (SER) is challenged by speaker-dependent, class-imbalanced data and by heterogeneous client devices. This paper proposes Fuzzy-FedAvg, a server-aware framework in which the server learns Client Selection Weights (CSWs) through fuzzy Q-learning from global-model feedback. The same CSWs weight client accuracy, training time, and \divkpi{} in both client selection and aggregation, while a dedicated PCAS node manages stragglers through staleness-discounted deferred updates. On IEMOCAP and CREMA-D, Fuzzy-FedAvg achieves higher unweighted average recall (UAR) than FedAvg in all configurations under severe non-IID conditions, and in 22 of 24 participation-level and 10 of 12 training-horizon configurations overall. It also outperforms FedProx and Oort, while SCAFFOLD and Power-of-Choice remain competitive, and it reduces cumulative training time by 81.2% in a FedScale-based evaluation. The benefit is greatest under strong statistical heterogeneity and narrows as data become more balanced.

Figures & tables

Paper	Year	Dataset(s)	Feature Space	Client Selection Scheme	Federated Environment	Heterogeneity Level
Federated SER
Latif et al. [ 29 ]	2020	IEMOCAP	Mel Filterbank	None	Homogeneous	N/A
Somandepalli et al. [ 30 ]	2022	IEMOCAP	Utterance-level acoustic features	None	Homogeneous	N/A
Tsouvalas et al. [ 31 ]	2022	IEMOCAP	Mel	None	Heterogeneous	Statistical heterogeneity (speaker variability)
Feng et al. [ 32 ]	2022	IEMOCAP, MSP-Improv	EmoBase feature set, APC, Tera, DeCoAR 2.0, DistilHuBERT & MFCC	None	Heterogeneous	Statistical heterogeneity (non-IID)
Feng et al. [ 33 ]	2022	IEMOCAP, CREMA-D, MSP-Improv	EmoBase and APC feature set	None	Homogeneous	N/A

TABLE I: Comparison of Fuzzy-FedAvg with related federated SER, general FL, and decentralised FL studies.

Fig. 1: Server-Aware Client Selection Scheme for SER in Heterogeneous Federated Environments. \tiny{1}⃝ Clients sharing $O$ KPIs with PCAS. \tiny{2}⃝ Server sharing global model weights $\omega_{i,\text{glob}}^{t}$ along with $O$ CSWs with PCAS. \tiny{3}⃝ Ranking of $K$ clients by PCAS. \tiny{4}⃝ Selection and sharing of $\omega_{i,\text{glob}}^{t}$ with top-ranked $N$ clients by PCAS. \tiny{5}⃝ PCAS threshold timer to manage clients participation and stragglers. \tiny{6}⃝ Local training by $N$ clients using $\omega_{i,\text{glob}}^{t}$ . \tiny{7}⃝ Sharing of updated weights $\omega_{i,k}^{t+1}$ and $O$ KPIs by $N$ clients with PCAS. \tiny{8}⃝ Update of global weights $\omega_{i,\text{glob}}^{t+1}$ by PCAS. \tiny{9}⃝ Sharing of updated global weights $\omega_{i,\text{glob}}^{t+1}$ with server for global model training. \tiny{10}⃝ Server updating global model with weights $\omega_{i,\text{glob}}^{t+1}$ , obtaining immediate reward $R^{t}$ and updating quality values.

Aim - To learn optimal client selection weights corresponding to articulated fuzzy rules in a heterogeneous FL environment.

RL Agent - The central server hosting and updating the global model.

Fuzzified Inputs -

Acc_{\text{glob}}

T_{\text{glob}}

and

\zeta_{\text{glob}}

Fuzzy Rule Base

Acc_{\text{glob}}

is BELOW AND

T_{\text{glob}}

is LESS AND

\zeta_{\text{glob}}

is LOW THEN conc is

\dots

Acc_{\text{glob}}

is BELOW AND

T_{\text{glob}}

is LESS AND

\zeta_{\text{glob}}

is HIGH THEN conc is

\dots

TABLE II: Salient Design Details of Adaptive Learning of CSWs by the Server Using Fuzzy Q-learning

Parameter	Value / Description
Datasets	IEMOCAP, CREMA-D
Acoustic Features	40 MFCCs; 128-band log-Mel-spectrogram (16 kHz, 25 ms window, 10 ms hop)
Number of Clients, $K$	30
Clients per Round, $N$	8, 10, 12
Dirichlet Parameter, $\alpha$	0.1, 0.3, 0.5, 1.0
Global Rounds, $G$	60, 80, 120

TABLE III: Simulation parameters used in the experimental evaluation.

Dataset	Method	UAR $\uparrow$	Time (s) $\downarrow$	PF $\uparrow$	AF $\uparrow$	AD $\downarrow$	WCA $\uparrow$
IEMOCAP	FedAvg	$0.394\pm 0.084$	$669.05\pm 127.59$	$0.964\pm 0.005$	$0.929\pm 0.009$	$0.704\pm 0.085$	$0.241\pm 0.077$
	FedProx	$0.412\pm 0.044$	$481.94\pm 82.47$	$\mathbf{0.967\pm 0.006}$	$0.914\pm 0.018$	$0.739\pm 0.082$	$0.215\pm 0.079$
	SCAFFOLD	$0.501\pm 0.029$	$432.49\pm 80.45$	$\mathbf{0.967\pm 0.006}$	$0.935\pm 0.006$	$0.617\pm 0.030$	$0.209\pm 0.067$
	Power-of-Choice	$\mathbf{0.503\pm 0.038}$	$\mathbf{373.98\pm 8.31}$	$0.562\pm 0.044$	$\mathbf{0.959\pm 0.008}$	$\mathbf{0.549\pm 0.050}$	$\mathbf{0.386\pm 0.064}$
	Oort	$0.365\pm 0.042$	$467.89\pm 127.56$	$0.889\pm 0.017$	$0.928\pm 0.029$	$0.699\pm 0.090$	$0.238\pm 0.078$
	Fuzzy-FedAvg	$0.437\pm 0.060$	$505.14\pm 25.19$	$0.935\pm 0.013$	$0.915\pm 0.007$	$0.732\pm 0.067$	$0.191\pm 0.063$

TABLE IV: Comprehensive comparison with stronger federated learning and client-selection baselines under severe statistical heterogeneity (

\alpha=0.1

), with

K=30

N=8

, and

G=80

, using MFCC features. Results are reported as mean

\pm

standard deviation over three independent runs using seeds 42, 137, and 911. Cumulative training time is obtained under the common simulation timing setup used for all methods.

\uparrow

indicates higher is better and

\downarrow

indicates lower is better. PF (Participation Fairness), AF (Accuracy Fairness), AD (Accuracy Disparity), WCA (Worst Case Accuracy).

Fig. 2: Client-level fairness of FedAvg and Fuzzy-FedAvg under severe statistical heterogeneity (

\alpha=0.1

K=30

N=8

G=80

): accuracy disparity (AD, lower is better) and worst-case accuracy (WCA, higher is better). Bars show the mean and error bars the standard deviation over seeds 42, 137, and 911.

Fig. 3: UAR of FedAvg and Fuzzy-FedAvg under varying statistical heterogeneity

\alpha

(

K=30

N=8

G=80

). Markers show the mean and error bars the standard deviation over seeds 42, 137, and 911. Fuzzy-FedAvg provides the clearest gains under severe non-IID conditions (

\alpha=0.1

), while the advantage becomes configuration-dependent as the data approach the IID regime.

Fig. 4: UAR of FedAvg and Fuzzy-FedAvg for (top row) varying numbers of participating clients per round

N

\alpha\in\{0.1,0.3\}

with

G=80

, and (bottom row) varying numbers of global rounds

G

\alpha=0.1

with

N=8

;

K=30

throughout. Markers show the mean and error bars the standard deviation over seeds 42, 137, and 911.

Fig. 5: Cumulative training time under FedScale-based heterogeneous client timing profiles (IEMOCAP, MFCC,

\alpha=0.1

K=30

N=8

G=80

) for each seed and on average (mean

\pm

standard deviation). Labels give the reduction achieved by Fuzzy-FedAvg relative to FedAvg.

Fig. 6: Ablation study of the main components of Fuzzy-FedAvg under severe statistical heterogeneity (

\alpha=0.1

K=30

N=8

G=80

). Bars show the mean UAR and error bars the standard deviation over seeds 42, 137, and 911.

Parameter	Setting	UAR $\uparrow$	Time (s) $\downarrow$
Conclusions	$\{0,2,4\}$	$\mathbf{0.3920}$	$387.55$
	$\{0,3,6\}$	$0.3399$	$367.22$
	$\{0,4,8\}$	$0.3695$	$\mathbf{366.79}$
Reward scale	$0.8$	$0.3579$	$357.46$
	$1.0$	$\mathbf{0.4369}$	$\mathbf{356.96}$
	$1.2$	$0.4229$	$384.53$

TABLE V: Sensitivity analysis of key Fuzzy-FedAvg design parameters on IEMOCAP using MFCC features under severe statistical heterogeneity (

\alpha=0.1

), with

K=30

N=8

, and

G=80

. Each parameter is varied while the remaining settings are held fixed at their nominal values. Results correspond to seed 42.

Client Selection in Federated Learning

Client selection in federated learning chooses which devices train each round to balance model quality, efficiency, fairness, and privacy.

See all 40 papers

May 23, 2026

Hardware-Aware Federated Learning for Speech Emotion Recognition

Federated learning (FL) enables privacy-preserving collaborative training across distributed edge devices, but real deployments involve heterogeneous clients with different processing power, memory capacity, and communication latency, which often increase round duration and system cost. This paper proposes a hardware-aware federated learning framework for emotion recognition on session-partitioned IEMOCAP that integrates hardware profiling, top-K client selection, and adaptive local epochs within a unified training loop. We compare the method against FedAvg, FedProx, and random top-K selection under a non-IID setup and show that, across 50 federated rounds and 5 independent trials, the proposed approach achieves competitive validation accuracy (0.352), reduces total training time by about 36.5% compared to FedAvg, and lowers cumulative communication cost by 40%.

Beyazit Bestami Yuksel, Emrah Dikbiyik

Computer Engineering Istanbul Technical University Istanbul, Turkey · Department of Computer Technologies Istanbul University-Cerrahpaşa Istanbul, Turkey

Communication-Efficient Distributed Training Client Selection in Federated Learning Non-IID Federated Learning Speech Emotion Recognition Federated Learning Heterogeneous Federated Learning

Ask PDF

Jun 11, 2026

Towards Personalized Federated Learning for Dysarthric Speech Recognition

Speech recognition is challenging for dysarthric speakers. While federated learning (FL)-based ASR can be an effective tool for protecting privacy, it suffers from heterogeneity issues caused by speaker variability. Forcing all speakers to share the same model components can be suboptimal under such heterogeneity, making personalization a promising direction; however, related research on dysarthric speech remains limited. To this end, this paper explores two aggregation strategies to achieve personalization, including the parameter-based averaging strategy and the embedding-based averaging strategy. Experiments on UASpeech and TORGO show that the proposed methods outperform the baseline regularized FedAvg by statistically significant WER reductions of up to 0.99% absolute (3.15% relative) on UASpeech and 0.56% absolute (4.73% relative) on TORGO, respectively.

Tao Zhong, Mengzhe Geng, Jiajun Deng +2

The Chinese University of Hong Kong, China · National Research Council Canada, Canada

Federated Learning Aggregation Automatic Speech Recognition Federated Learning Heterogeneous Federated Learning Dysarthric Speech Recognition

Ask PDF

Mar 9, 2026

FedLECC: Cluster- and Loss-Guided Client Selection for Federated Learning under Non-IID Data

Federated Learning (FL) enables distributed Artificial Intelligence (AI) across cloud-edge environments by allowing collaborative model training without centralizing data. In cross-device deployments, FL systems face strict communication and participation constraints, as well as strong non-independent and identically distributed (non-IID) data that degrades convergence and model quality. Since only a subset of devices (a.k.a clients) can participate per training round, intelligent client selection becomes a key systems challenge. This paper proposes FedLECC (Federated Learning with Enhanced Cluster Choice), a lightweight, cluster-aware, and loss-guided client selection strategy for cross-device FL. FedLECC groups clients by label-distribution similarity and prioritizes clusters and clients with higher local loss, enabling the selection of a small yet informative and diverse set of clients. Experimental results under severe label skew show that FedLECC improves test accuracy by up to 12%, while reducing communication rounds by approximately 22% and overall communication overhead by up to 50% compared to strong baselines. These results demonstrate that informed client selection improves the efficiency and scalability of FL workloads in cloud-edge systems.

Daniel M. Jimenez-Gutierrez, Giovanni Giunta, Mehrdad Hassanzadeh +3

Sapienza University of Rome Via Ariosto 25 Rome, 00185, Italy

Client Selection in Federated Learning Non-IID Federated Learning Communication-Efficient Distributed Training Clustered Federated Learning

Ask PDF

Abstract

Client Selection in Federated Learning

Explore similar work

Hardware-Aware Federated Learning for Speech Emotion Recognition

Towards Personalized Federated Learning for Dysarthric Speech Recognition

FedLECC: Cluster- and Loss-Guided Client Selection for Federated Learning under Non-IID Data