Sparse Federated Echo State Networks for Time-Series Classification via Smooth-L1 Regularization

Gengcheng Lyu; Manuel Röder; Frank-Michael Schleif

cs.LGOct 15, 2026

Sparse Federated Echo State Networks for Time-Series Classification via Smooth-L1 Regularization

Authors: Gengcheng Lyu, Manuel Röder, Frank-Michael Schleif

Organizations: Faculty of Computer Science and Business Information Systems Technical University of Applied Sciences Würzburg-Schweinfurt, Würzburg, Germany

Abstract

Echo State Networks (ESNs), as lightweight implementations of reservoir computing, provide efficient solutions for temporal data processing in resource-constrained environments. While standard ESNs offer computational efficiency, their dense readout layers inflate the stored model and the per-sample inference cost. This paper introduces a Smooth-L1 regularized ESN (SL1ESN) that promotes sparsity in the readout layer while preserving differentiability for gradient-based optimization. The method employs a twice-differentiable Smooth-L1 approximation compatible with Newton's method, which we exploit both for centralized training and for a second-order (Newton-step) federated aggregation scheme. Under a single fixed shrinkage threshold shared by every dataset and both settings, experiments on seven time-series classification benchmarks show that SL1ESN learns readouts that are $77$ -- $95\%$ sparse in the centralized setting and $75$ -- $95\%$ sparse in the federated one, while staying within $0.7$ accuracy points of the strongest dense baseline on five of the seven benchmarks in the centralized setting and within $1.6$ points on the same five in the federated one, and the second-order federated update converges in only a few rounds. The resulting sparsity compresses the stored model, the per-sample inference cost, and the server-to-client broadcast by a comparable factor. These results demonstrate the practical value of sparse readouts for edge computing and IoT deployments.

Figures & tables

Figure 1: FedSL1ESN training pipeline. The server broadcasts the fixed global reservoir once and a sparse global readout each round (CSR, minimal payload); clients evaluate the local gradient and Gauss-Newton Hessian of their Smooth-L1 loss on private data and return this curvature information for the server’s Newton-step aggregation.

Dataset	Train	Test	Length	Dim.	Classes
UCI HAR	7352	2947	128	9	6
Character	2286	572	205	3	20
JapaneseVowels	270	370	29	12	9
ECG5000	500	4500	140	1	5
Distal.	600	276	80	1	2
Yoga	300	3000	426	1	2

Table 1: Classification datasets. For ease of reference, the DistalPhalanxOutlineCorrect dataset is abbreviated as Distal., and Character denotes the Character Trajectories dataset from the UCI repository. All datasets are used with the train/test partition of their archive, except Character, for which the UCI repository provides none: its

2858

sequences are split

80/20

once, and the same split is reused by every method and seed. Length is the padded sequence length, and variable-length sequences (Character, JapaneseVowels) are zero-padded to the longest sequence in the dataset.

Parameter	Distribution
Spectral radius ( $\rho$ )	Uniform $(0.3,\ 3.0)$
Input scaling ( $a_{\text{in}}$ )	Log-uniform $[10^{-2},\ 10^{2}]$
Leaking rate ( $\gamma$ )	Log-uniform $[5\times 10^{-3},\ 1]$
Regularization ( $\lambda$ )	Log-uniform $[10^{-4},\ 10^{2}]$

Table 2: Hyperparameter search distributions (random search).

Dataset	none	L2-ESN	SL1ESN (ours)
Dataset	Acc. (%)	Acc. (%)	Acc. (%)	Sparsity (%)
UCI HAR	$87.37\pm 1.11$	$88.01\pm 0.43$	$\mathbf{88.05}\pm 0.90$	$77.6\pm 1.5$
Character	$\mathbf{99.58}\pm 0.12$	$98.88\pm 0.28$	$98.97\pm 0.25$	$92.7\pm 0.6$
JapaneseVowels	$\mathbf{98.24}\pm 0.63$	$97.57\pm 0.17$	$97.68\pm 0.56$	$94.6\pm 0.4$
ECG5000	$91.38\pm 0.64$	$\mathbf{92.37}\pm 0.08$	$91.91\pm 0.20$	$93.3\pm 0.4$
Distal.	$\mathbf{73.55}\pm 0.81$	$73.08\pm 1.23$	$73.51\pm 1.52$	$89.6\pm 3.3$

Table 3: Centralized classification test accuracy (%) and SL1ESN readout sparsity. All readouts are selected for predictive accuracy; bold marks the highest accuracy per dataset. Sparsity is the fraction of zero readout weights; the unregularized and L2 readouts have no sparsification mechanism and remain dense. All values are mean

\pm

std over ten seeds. A single fixed shrinkage threshold

\tau=0.1

is used for every dataset.

Figure 2: Centralized setting. Test accuracy versus readout sparsity for the accuracy-selected SL1ESN configuration, trained on the pooled data by a single learner (mean over ten seeds), obtained by sweeping the shrinkage threshold

\tau

; shaded bands are

\pm

1 std across seeds. Accuracy stays essentially flat up to

80

–

95\%

sparsity on every dataset, and collapses only once the readout is driven to near-total sparsity. Figure 4 repeats this sweep for the federated global readout; the two are plotted on the same axes so that the curves can be compared directly, and their similarity is itself the result, indicating that federation does not degrade the sparsity–accuracy trade-off.

Dataset	none	L2-ESN	FedSL1ESN (ours)
Dataset	Acc. (%)	Acc. (%)	Acc. (%)	Sparsity (%)
UCI HAR	$\mathbf{89.41}\pm 1.15$	$87.79\pm 0.72$	$87.86\pm 0.79$	$74.9\pm 0.5$
Character	$98.48\pm 0.35$	$\mathbf{98.74}\pm 0.26$	$98.20\pm 0.57$	$84.8\pm 2.2$
JapaneseVowels	$\mathbf{96.19}\pm 0.96$	$81.27\pm 2.12$	$94.84\pm 1.33$	$95.2\pm 0.3$
ECG5000	$90.32\pm 0.32$	$\mathbf{91.38}\pm 0.77$	$91.30\pm 0.34$	$91.6\pm 1.0$
Distal.	$67.10\pm 2.13$	$71.93\pm 1.33$	$\mathbf{73.77}\pm 2.56$	$86.7\pm 2.4$

Table 4: Federated learning results (five clients,

50

rounds, IID split, mean

\pm

std over ten seeds; test accuracy) and FedSL1ESN readout sparsity, all reported at the final round. All readouts are selected for predictive accuracy; bold marks the highest accuracy per dataset. The unregularized and L2 readouts have no sparsification mechanism and remain dense (

<2\%

zeros). The same fixed shrinkage threshold

\tau=0.1

as in the centralized setting is used for every dataset.

Figure 3: FedSL1ESN convergence over communication rounds for each dataset (five clients, IID split,

50

rounds, mean over ten seeds; shaded bands are

\pm

1 std). (a) Test accuracy reaches near-peak within a few rounds on most datasets and is stable thereafter (Strawberry is the slowest, climbing until round

\sim 20

). (b) Readout sparsity accrues more gradually over the same rounds, as the continuation schedule sharpens the penalty toward L1, and has largely levelled off by round

50

; this is why we fix the budget at

50

rounds and report there. The two panels separate the two timescales of the method, since the cross-entropy fit is reached almost immediately, while the sparsification unfolds over the full round budget. The bands in (b) are wide at early rounds, where seeds sit at different points on the steep part of the curve, and narrow by round

50

, which is why the round-

50

sparsity is reproducible across seeds.

Figure 4: Federated setting; the counterpart of Figure 2 . Test accuracy versus readout sparsity for the accuracy-selected global readout obtained by FedSL1ESN, i.e., the model aggregated from five clients that never share data (

50

rounds, IID partition, mean over ten seeds, reported at the final round), under the same sweep of the shrinkage threshold

\tau

; shaded bands are

\pm

1 std across seeds. As in the centralized case, accuracy stays essentially flat out to

75

–

95\%

sparsity on most datasets, and collapses only once the readout is driven to near-total sparsity.

Figure 5: One-at-a-time ablation on UCI HAR (mean over ten seeds; shaded bands are

\pm

1 std). Left axis (blue): validation accuracy; right axis (orange): readout sparsity. (a) The shrinkage threshold

\tau

trades sparsity for accuracy cleanly. (b) The penalty strength

\lambda

also sparsifies, but sacrifices more accuracy and saturates. (c) Reservoir size

N_{R}

governs accuracy and saturates around

500

. (d) The continuation factor

\kappa

leaves accuracy unchanged.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$N_{I},N_{R},N_{O}$	Input, reservoir, and output dimensions
$t,\;T$	Discrete time step; total training steps
$\mathbf{u}(t),\mathbf{x}(t)$	Input vector and reservoir state
$\tilde{\mathbf{x}}(t)$	Candidate (pre-leak) reservoir state
$\mathbf{X}$	Matrix of collected reservoir states
$\hat{\mathbf{y}}(t),\mathbf{y}(t)$	Predicted and target output vectors

Appendix

Table 5: Notation summary.

Dataset	$\rho$	$a_{\text{in}}$	$\gamma$	$\lambda$
Mackey-Glass	1.26	1.76	0.54	$1.5\times 10^{-3}$
Lorenz63	1.10	0.23	0.45	$1.0\times 10^{-4}$
UCI HAR	1.48	7.12	0.016	$6.1\times 10^{-3}$
Character	1.22	0.014	0.16	$7.4\times 10^{-3}$
JapaneseVowels	0.50	0.60	0.032	$5.7\times 10^{-4}$
ECG5000	2.04	10.3	0.033	$1.0\times 10^{-2}$

Appendix

Table 6: Selected SL1ESN hyperparameters per dataset (random search over the ranges in Table 2 ; see Section 5 ). All runs use the reservoir size

N_{R}=500

Dataset	Metric	none	L2	L1	SL1ESN
Mackey-Glass	RMSE	$\mathbf{0.085}$	$0.108$	$0.113$	$0.118$
	VPT	$511$	$176$	$230$	$377$
	Spars. (%)	$0$	$0$	$91.0$	$46.4$
Lorenz63	RMSE	$\mathbf{9.90}$	$10.70$	$11.74$	$10.14$
	VPT	$277$	$270$	$109$	$148$
	Spars. (%)	$0$	$0$	$92.9$	$0.2$

Appendix

Table 7: Centralized regression results (best of five runs). RMSE lower is better; VPT (valid prediction time, in steps) higher is better; sparsity is the fraction of zero readout weights. All readouts are selected for accuracy. Bold : best RMSE per dataset.

Panel of Figure 6 — Figure 6: Autonomous Mackey-Glass rollout (dashed) against the true test trajectory (solid) under the four readout variants, at the accuracy-selected configuration. The unregularized readout tracks the true trajectory longest (VPT $511$ ; Table 7 ), followed by the Smooth-L1 readout (VPT $377$ ), whereas the L2 and L1 rollouts lose phase alignment earlier (VPT $176$ and $230$ ).

Panel of Figure 7 — Figure 7: Autonomous Lorenz rollout (dashed) against the true attractor (solid) under the four readout variants, at the accuracy-selected configuration. The unregularized, L2, and Smooth-L1 readouts all reconstruct the butterfly and track its two lobes before slowly drifting outward (VPT $277$ , $270$ , and $148$ ; Table 7 ), whereas the L1 readout leaves the attractor earliest, collapsing away from the far lobe (VPT $109$ ). The Smooth-L1 rollout attains the lowest RMSE of the three regularized readouts while, unlike the coordinate-descent L1 solver, remaining compatible with second-order optimization.

Figure 8: Federated test accuracy (left axis, solid) and readout sparsity (right axis, dashed) as the number of clients

K

is swept from

2

50

(

50

rounds, IID partition, shared threshold

\tau=0.1

, mean

\pm

1 std over ten seeds). The dotted vertical line marks the

K=5

setting used in the main-text experiments. Accuracy is essentially flat in

K

on every dataset, and sparsity stays in the band observed at

K=5

Sparse Neural Networks

Sparse neural networks use a subset of weights, connections, activations, or experts to reduce model complexity and computation.

See all 66 papers

Fast Linear Reservoirs via Diagonalization
Feb 23, 2026Romain de Coudenhove, Yannis Bendi-Ouis, Anthony Strock +1Reservoir Computing Efficient Neural Network Inference
Mnemosyne, DI-ENS · ENS PSL, Inria Center of Bordeaux University, LaBRI, IMN · Mnemosyne +2
Federated Learning of Spiking Neural Networks under Heterogeneous Temporal Resolutions
May 14, 2026Sanja Karilanova, Subhrakanti Dey, Ayça ÖzçelikkaleSpiking Neural Networks Federated Learning
Department of Electrical Engineering, Uppsala University, Sweden
Centrality-Based Pruning for Efficient Echo State Networks
Mar 21, 2026Sudip LaudariReservoir Computing Neural Network Pruning
Independent Researcher.

Feb 23, 2026J/K move · Enter open · S save

Fast Linear Reservoirs via Diagonalization

Romain de Coudenhove, Yannis Bendi-Ouis, Anthony Strock, Xavier Hinaut

Mnemosyne, DI-ENS · ENS PSL, Inria Center of Bordeaux University, LaBRI, IMN · Mnemosyne+2

Reservoir Computing Efficient Neural Network Inference

Open paper AskPDF arXiv

We introduce a diagonalization-based optimization for Linear Echo State Networks (ESNs) that reduces the per-step computational complexity of reservoir state updates from quadratic to linear. By reformulating reservoir dynamics in the eigenbasis of the recurrent matrix, the recurrent update becomes a set of independent element-wise operations, eliminating the matrix multiplication. We further propose three methods to use our optimization depending on the situation: (i) Eigenbasis Weight Transformation (EWT), which preserves the dynamics of standard and trained Linear ESNs, (ii) End-to-End Eigenbasis Training (EET), which directly optimizes readout weights in the transformed space and (iii) Direct Parameter Generation (DPG), that bypasses matrix diagonalization by directly sampling eigenvalues and eigenvectors, achieving comparable performance to standard Linear ESNs. Across all experiments, both our methods preserve predictive accuracy while offering significant computational speedups, making them a replacement for standard Linear ESNs computations and training, and suggesting a shift of paradigm in linear ESN towards the direct selection of eigenvalues.

Abstract

Sparse Neural Networks

Explore similar work

Fast Linear Reservoirs via Diagonalization

Federated Learning of Spiking Neural Networks under Heterogeneous Temporal Resolutions

Centrality-Based Pruning for Efficient Echo State Networks