Fuzzy-FedAvg: Server-Aware Client Selection for Federated Speech Emotion Recognition in Heterogeneous Environments

Farrukh Pervez; Siddique Latif; Moid Sandhu; Raja Jurdak

cs.SDNov 6, 2026

Fuzzy-FedAvg: Server-Aware Client Selection for Federated Speech Emotion Recognition in Heterogeneous Environments

Authors: Farrukh Pervez, Siddique Latif, Moid Sandhu, Raja Jurdak

Organizations: College of Aeronautical Engineering, National University of Sciences & Technology (NUST), Islamabad 24090, Pakistan · Queensland University of Technology (QUT), Brisbane QLD 4000, Australia · Commonwealth Scientific and Industrial Research Organisation (CSIRO), Brisbane, QLD 4029, Australia

Abstract

Federated speech emotion recognition (SER) is challenged by speaker-dependent, class-imbalanced data and by heterogeneous client devices. This paper proposes Fuzzy-FedAvg, a server-aware framework in which the server learns Client Selection Weights (CSWs) through fuzzy Q-learning from global-model feedback. The same CSWs weight client accuracy, training time, and \divkpi{} in both client selection and aggregation, while a dedicated PCAS node manages stragglers through staleness-discounted deferred updates. On IEMOCAP and CREMA-D, Fuzzy-FedAvg achieves higher unweighted average recall (UAR) than FedAvg in all configurations under severe non-IID conditions, and in 22 of 24 participation-level and 10 of 12 training-horizon configurations overall. It also outperforms FedProx and Oort, while SCAFFOLD and Power-of-Choice remain competitive, and it reduces cumulative training time by 81.2% in a FedScale-based evaluation. The benefit is greatest under strong statistical heterogeneity and narrows as data become more balanced.

Figures & tables

Paper	Year	Dataset(s)	Feature Space	Client Selection Scheme	Federated Environment	Heterogeneity Level
Federated SER
Latif et al. [ 29 ]	2020	IEMOCAP	Mel Filterbank	None	Homogeneous	N/A
Somandepalli et al. [ 30 ]	2022	IEMOCAP	Utterance-level acoustic features	None	Homogeneous	N/A
Tsouvalas et al. [ 31 ]	2022	IEMOCAP	Mel	None	Heterogeneous	Statistical heterogeneity (speaker variability)
Feng et al. [ 32 ]	2022	IEMOCAP, MSP-Improv	EmoBase feature set, APC, Tera, DeCoAR 2.0, DistilHuBERT & MFCC	None	Heterogeneous	Statistical heterogeneity (non-IID)
Feng et al. [ 33 ]	2022	IEMOCAP, CREMA-D, MSP-Improv	EmoBase and APC feature set	None	Homogeneous	N/A

TABLE I: Comparison of Fuzzy-FedAvg with related federated SER, general FL, and decentralised FL studies.

Fig. 1: Server-Aware Client Selection Scheme for SER in Heterogeneous Federated Environments. \tiny{1}⃝ Clients sharing $O$ KPIs with PCAS. \tiny{2}⃝ Server sharing global model weights $\omega_{i,\text{glob}}^{t}$ along with $O$ CSWs with PCAS. \tiny{3}⃝ Ranking of $K$ clients by PCAS. \tiny{4}⃝ Selection and sharing of $\omega_{i,\text{glob}}^{t}$ with top-ranked $N$ clients by PCAS. \tiny{5}⃝ PCAS threshold timer to manage clients participation and stragglers. \tiny{6}⃝ Local training by $N$ clients using $\omega_{i,\text{glob}}^{t}$ . \tiny{7}⃝ Sharing of updated weights $\omega_{i,k}^{t+1}$ and $O$ KPIs by $N$ clients with PCAS. \tiny{8}⃝ Update of global weights $\omega_{i,\text{glob}}^{t+1}$ by PCAS. \tiny{9}⃝ Sharing of updated global weights $\omega_{i,\text{glob}}^{t+1}$ with server for global model training. \tiny{10}⃝ Server updating global model with weights $\omega_{i,\text{glob}}^{t+1}$ , obtaining immediate reward $R^{t}$ and updating quality values.

Aim - To learn optimal client selection weights corresponding to articulated fuzzy rules in a heterogeneous FL environment.

RL Agent - The central server hosting and updating the global model.

Fuzzified Inputs -

Acc_{\text{glob}}

T_{\text{glob}}

and

\zeta_{\text{glob}}

Fuzzy Rule Base

Acc_{\text{glob}}

is BELOW AND

T_{\text{glob}}

is LESS AND

\zeta_{\text{glob}}

is LOW THEN conc is

\dots

Acc_{\text{glob}}

is BELOW AND

T_{\text{glob}}

is LESS AND

\zeta_{\text{glob}}

is HIGH THEN conc is

\dots

TABLE II: Salient Design Details of Adaptive Learning of CSWs by the Server Using Fuzzy Q-learning

Parameter	Value / Description
Datasets	IEMOCAP, CREMA-D
Acoustic Features	40 MFCCs; 128-band log-Mel-spectrogram (16 kHz, 25 ms window, 10 ms hop)
Number of Clients, $K$	30
Clients per Round, $N$	8, 10, 12
Dirichlet Parameter, $\alpha$	0.1, 0.3, 0.5, 1.0
Global Rounds, $G$	60, 80, 120

TABLE III: Simulation parameters used in the experimental evaluation.

Dataset	Method	UAR $\uparrow$	Time (s) $\downarrow$	PF $\uparrow$	AF $\uparrow$	AD $\downarrow$	WCA $\uparrow$
IEMOCAP	FedAvg	$0.394\pm 0.084$	$669.05\pm 127.59$	$0.964\pm 0.005$	$0.929\pm 0.009$	$0.704\pm 0.085$	$0.241\pm 0.077$
	FedProx	$0.412\pm 0.044$	$481.94\pm 82.47$	$\mathbf{0.967\pm 0.006}$	$0.914\pm 0.018$	$0.739\pm 0.082$	$0.215\pm 0.079$
	SCAFFOLD	$0.501\pm 0.029$	$432.49\pm 80.45$	$\mathbf{0.967\pm 0.006}$	$0.935\pm 0.006$	$0.617\pm 0.030$	$0.209\pm 0.067$
	Power-of-Choice	$\mathbf{0.503\pm 0.038}$	$\mathbf{373.98\pm 8.31}$	$0.562\pm 0.044$	$\mathbf{0.959\pm 0.008}$	$\mathbf{0.549\pm 0.050}$	$\mathbf{0.386\pm 0.064}$
	Oort	$0.365\pm 0.042$	$467.89\pm 127.56$	$0.889\pm 0.017$	$0.928\pm 0.029$	$0.699\pm 0.090$	$0.238\pm 0.078$
	Fuzzy-FedAvg	$0.437\pm 0.060$	$505.14\pm 25.19$	$0.935\pm 0.013$	$0.915\pm 0.007$	$0.732\pm 0.067$	$0.191\pm 0.063$

TABLE IV: Comprehensive comparison with stronger federated learning and client-selection baselines under severe statistical heterogeneity (

\alpha=0.1

), with

K=30

N=8

, and

G=80

, using MFCC features. Results are reported as mean

\pm

standard deviation over three independent runs using seeds 42, 137, and 911. Cumulative training time is obtained under the common simulation timing setup used for all methods.

\uparrow

indicates higher is better and

\downarrow

indicates lower is better. PF (Participation Fairness), AF (Accuracy Fairness), AD (Accuracy Disparity), WCA (Worst Case Accuracy).

Fig. 2: Client-level fairness of FedAvg and Fuzzy-FedAvg under severe statistical heterogeneity (

\alpha=0.1

K=30

N=8

G=80

): accuracy disparity (AD, lower is better) and worst-case accuracy (WCA, higher is better). Bars show the mean and error bars the standard deviation over seeds 42, 137, and 911.

Fig. 3: UAR of FedAvg and Fuzzy-FedAvg under varying statistical heterogeneity

\alpha

(

K=30

N=8

G=80

). Markers show the mean and error bars the standard deviation over seeds 42, 137, and 911. Fuzzy-FedAvg provides the clearest gains under severe non-IID conditions (

\alpha=0.1

), while the advantage becomes configuration-dependent as the data approach the IID regime.

Fig. 4: UAR of FedAvg and Fuzzy-FedAvg for (top row) varying numbers of participating clients per round

N

\alpha\in\{0.1,0.3\}

with

G=80

, and (bottom row) varying numbers of global rounds

G

\alpha=0.1

with

N=8

;

K=30

throughout. Markers show the mean and error bars the standard deviation over seeds 42, 137, and 911.

Fig. 5: Cumulative training time under FedScale-based heterogeneous client timing profiles (IEMOCAP, MFCC,

\alpha=0.1

K=30

N=8

G=80

) for each seed and on average (mean

\pm

standard deviation). Labels give the reduction achieved by Fuzzy-FedAvg relative to FedAvg.

Fig. 6: Ablation study of the main components of Fuzzy-FedAvg under severe statistical heterogeneity (

\alpha=0.1

K=30

N=8

G=80

). Bars show the mean UAR and error bars the standard deviation over seeds 42, 137, and 911.

Parameter	Setting	UAR $\uparrow$	Time (s) $\downarrow$
Conclusions	$\{0,2,4\}$	$\mathbf{0.3920}$	$387.55$
	$\{0,3,6\}$	$0.3399$	$367.22$
	$\{0,4,8\}$	$0.3695$	$\mathbf{366.79}$
Reward scale	$0.8$	$0.3579$	$357.46$
	$1.0$	$\mathbf{0.4369}$	$\mathbf{356.96}$
	$1.2$	$0.4229$	$384.53$

TABLE V: Sensitivity analysis of key Fuzzy-FedAvg design parameters on IEMOCAP using MFCC features under severe statistical heterogeneity (

\alpha=0.1

), with

K=30

N=8

, and

G=80

. Each parameter is varied while the remaining settings are held fixed at their nominal values. Results correspond to seed 42.

Client Selection in Federated Learning

Client selection in federated learning chooses which devices train each round to balance model quality, efficiency, fairness, and privacy.

See all 40 papers

Hardware-Aware Federated Learning for Speech Emotion Recognition
May 23, 2026Beyazit Bestami Yuksel, Emrah DikbiyikCommunication-Efficient Distributed Training Client Selection in Federated Learning
Computer Engineering Istanbul Technical University Istanbul, Turkey · Department of Computer Technologies Istanbul University-Cerrahpaşa Istanbul, Turkey
Towards Personalized Federated Learning for Dysarthric Speech Recognition
Jun 11, 2026Tao Zhong, Mengzhe Geng, Jiajun Deng +2Federated Learning Aggregation Automatic Speech Recognition
The Chinese University of Hong Kong, China · National Research Council Canada, Canada
FedLECC: Cluster- and Loss-Guided Client Selection for Federated Learning under Non-IID Data
Mar 9, 2026Daniel M. Jimenez-Gutierrez, Giovanni Giunta, Mehrdad Hassanzadeh +3Client Selection in Federated Learning Non-IID Federated Learning
Sapienza University of Rome Via Ariosto 25 Rome, 00185, Italy

May 23, 2026J/K move · Enter open · S save

Hardware-Aware Federated Learning for Speech Emotion Recognition

Beyazit Bestami Yuksel, Emrah Dikbiyik

Computer Engineering Istanbul Technical University Istanbul, Turkey · Department of Computer Technologies Istanbul University-Cerrahpaşa Istanbul, Turkey

Communication-Efficient Distributed Training Client Selection in Federated Learning Non-IID Federated Learning Speech Emotion Recognition Federated Learning Heterogeneous Federated Learning

Open paper AskPDF arXiv

Federated learning (FL) enables privacy-preserving collaborative training across distributed edge devices, but real deployments involve heterogeneous clients with different processing power, memory capacity, and communication latency, which often increase round duration and system cost. This paper proposes a hardware-aware federated learning framework for emotion recognition on session-partitioned IEMOCAP that integrates hardware profiling, top-K client selection, and adaptive local epochs within a unified training loop. We compare the method against FedAvg, FedProx, and random top-K selection under a non-IID setup and show that, across 50 federated rounds and 5 independent trials, the proposed approach achieves competitive validation accuracy (0.352), reduces total training time by about 36.5% compared to FedAvg, and lowers cumulative communication cost by 40%.

Abstract

Client Selection in Federated Learning

Explore similar work

Hardware-Aware Federated Learning for Speech Emotion Recognition

Towards Personalized Federated Learning for Dysarthric Speech Recognition

FedLECC: Cluster- and Loss-Guided Client Selection for Federated Learning under Non-IID Data