Sparse Federated Echo State Networks for Time-Series Classification via Smooth-L1 Regularization
Authors: Gengcheng Lyu, Manuel Röder, Frank-Michael Schleif
Organizations: Faculty of Computer Science and Business Information Systems Technical University of Applied Sciences Würzburg-Schweinfurt, Würzburg, Germany
Echo State Networks (ESNs), as lightweight implementations of reservoir computing, provide efficient solutions for temporal data processing in resource-constrained environments. While standard ESNs offer computational efficiency, their dense readout layers inflate the stored model and the per-sample inference cost. This paper introduces a Smooth-L1 regularized ESN (SL1ESN) that promotes sparsity in the readout layer while preserving differentiability for gradient-based optimization. The method employs a twice-differentiable Smooth-L1 approximation compatible with Newton's method, which we exploit both for centralized training and for a second-order (Newton-step) federated aggregation scheme. Under a single fixed shrinkage threshold shared by every dataset and both settings, experiments on seven time-series classification benchmarks show that SL1ESN learns readouts that are 77--95% sparse in the centralized setting and 75--95% sparse in the federated one, while staying within 0.7 accuracy points of the strongest dense baseline on five of the seven benchmarks in the centralized setting and within 1.6 points on the same five in the federated one, and the second-order federated update converges in only a few rounds. The resulting sparsity compresses the stored model, the per-sample inference cost, and the server-to-client broadcast by a comparable factor. These results demonstrate the practical value of sparse readouts for edge computing and IoT deployments.
Figures & tables
Figure 1: FedSL1ESN training pipeline. The server broadcasts the fixed global reservoir once and a sparse global readout each round (CSR, minimal payload); clients evaluate the local gradient and Gauss-Newton Hessian of their Smooth-L1 loss on private data and return this curvature information for the server’s Newton-step aggregation.
Dataset
Train
Test
Length
Dim.
Classes
UCI HAR
7352
2947
128
9
6
Character
2286
572
205
3
20
JapaneseVowels
270
370
29
12
9
ECG5000
500
4500
140
1
5
Distal.
600
276
80
1
2
Yoga
300
3000
426
1
2
Table 1: Classification datasets. For ease of reference, the DistalPhalanxOutlineCorrect dataset is abbreviated as Distal., and Character denotes the Character Trajectories dataset from the UCI repository. All datasets are used with the train/test partition of their archive, except Character, for which the UCI repository provides none: its 2858 sequences are split 80/20 once, and the same split is reused by every method and seed. Length is the padded sequence length, and variable-length sequences (Character, JapaneseVowels) are zero-padded to the longest sequence in the dataset.
Table 3: Centralized classification test accuracy (%) and SL1ESN readout sparsity. All readouts are selected for predictive accuracy; bold marks the highest accuracy per dataset. Sparsity is the fraction of zero readout weights; the unregularized and L2 readouts have no sparsification mechanism and remain dense. All values are mean ± std over ten seeds. A single fixed shrinkage threshold τ=0.1 is used for every dataset.
Figure 2: Centralized setting. Test accuracy versus readout sparsity for the accuracy-selected SL1ESN configuration, trained on the pooled data by a single learner (mean over ten seeds), obtained by sweeping the shrinkage threshold τ ; shaded bands are ± 1 std across seeds. Accuracy stays essentially flat up to 80 – 95% sparsity on every dataset, and collapses only once the readout is driven to near-total sparsity. Figure 4 repeats this sweep for the federated global readout; the two are plotted on the same axes so that the curves can be compared directly, and their similarity is itself the result, indicating that federation does not degrade the sparsity–accuracy trade-off.
Dataset
none
L2-ESN
FedSL1ESN (ours)
Acc. (%)
Acc. (%)
Acc. (%)
Sparsity (%)
UCI HAR
89.41±1.15
87.79±0.72
87.86±0.79
74.9±0.5
Character
98.48±0.35
98.74±0.26
98.20±0.57
84.8±2.2
JapaneseVowels
96.19±0.96
81.27±2.12
94.84±1.33
95.2±0.3
ECG5000
90.32±0.32
91.38±0.77
91.30±0.34
91.6±1.0
Distal.
67.10±2.13
71.93±1.33
73.77±2.56
86.7±2.4
Table 4: Federated learning results (five clients, 50 rounds, IID split, mean ± std over ten seeds; test accuracy) and FedSL1ESN readout sparsity, all reported at the final round. All readouts are selected for predictive accuracy; bold marks the highest accuracy per dataset. The unregularized and L2 readouts have no sparsification mechanism and remain dense ( <2% zeros). The same fixed shrinkage threshold τ=0.1 as in the centralized setting is used for every dataset.
Figure 3: FedSL1ESN convergence over communication rounds for each dataset (five clients, IID split, 50 rounds, mean over ten seeds; shaded bands are ± 1 std). (a) Test accuracy reaches near-peak within a few rounds on most datasets and is stable thereafter (Strawberry is the slowest, climbing until round ∼20 ). (b) Readout sparsity accrues more gradually over the same rounds, as the continuation schedule sharpens the penalty toward L1, and has largely levelled off by round 50 ; this is why we fix the budget at 50 rounds and report there. The two panels separate the two timescales of the method, since the cross-entropy fit is reached almost immediately, while the sparsification unfolds over the full round budget. The bands in (b) are wide at early rounds, where seeds sit at different points on the steep part of the curve, and narrow by round 50 , which is why the round- 50 sparsity is reproducible across seeds.
Figure 4: Federated setting; the counterpart of Figure 2 . Test accuracy versus readout sparsity for the accuracy-selected global readout obtained by FedSL1ESN, i.e., the model aggregated from five clients that never share data ( 50 rounds, IID partition, mean over ten seeds, reported at the final round), under the same sweep of the shrinkage threshold τ ; shaded bands are ± 1 std across seeds. As in the centralized case, accuracy stays essentially flat out to 75 – 95% sparsity on most datasets, and collapses only once the readout is driven to near-total sparsity.
Figure 5: One-at-a-time ablation on UCI HAR (mean over ten seeds; shaded bands are ± 1 std). Left axis (blue): validation accuracy; right axis (orange): readout sparsity. (a) The shrinkage threshold τ trades sparsity for accuracy cleanly. (b) The penalty strength λ also sparsifies, but sacrifices more accuracy and saturates. (c) Reservoir size NR governs accuracy and saturates around 500 . (d) The continuation factor κ leaves accuracy unchanged.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Description
NI,NR,NO
Input, reservoir, and output dimensions
t,T
Discrete time step; total training steps
u(t),x(t)
Input vector and reservoir state
x~(t)
Candidate (pre-leak) reservoir state
X
Matrix of collected reservoir states
y^(t),y(t)
Predicted and target output vectors
Appendix
Table 5: Notation summary.
Dataset
ρ
ain
γ
λ
Mackey-Glass
1.26
1.76
0.54
1.5×10−3
Lorenz63
1.10
0.23
0.45
1.0×10−4
UCI HAR
1.48
7.12
0.016
6.1×10−3
Character
1.22
0.014
0.16
7.4×10−3
JapaneseVowels
0.50
0.60
0.032
5.7×10−4
ECG5000
2.04
10.3
0.033
1.0×10−2
Appendix
Table 6: Selected SL1ESN hyperparameters per dataset (random search over the ranges in Table 2 ; see Section 5 ). All runs use the reservoir size NR=500 .
Dataset
Metric
none
L2
L1
SL1ESN
Mackey-Glass
RMSE
0.085
0.108
0.113
0.118
VPT
511
176
230
377
Spars. (%)
0
0
91.0
46.4
Lorenz63
RMSE
9.90
10.70
11.74
10.14
VPT
277
270
109
148
Spars. (%)
0
0
92.9
0.2
Appendix
Table 7: Centralized regression results (best of five runs). RMSE lower is better; VPT (valid prediction time, in steps) higher is better; sparsity is the fraction of zero readout weights. All readouts are selected for accuracy. Bold : best RMSE per dataset.
Figure 6: Autonomous Mackey-Glass rollout (dashed) against the true test trajectory (solid) under the four readout variants, at the accuracy-selected configuration. The unregularized readout tracks the true trajectory longest (VPT 511 ; Table 7 ), followed by the Smooth-L1 readout (VPT 377 ), whereas the L2 and L1 rollouts lose phase alignment earlier (VPT 176 and 230 ).
Figure 7: Autonomous Lorenz rollout (dashed) against the true attractor (solid) under the four readout variants, at the accuracy-selected configuration. The unregularized, L2, and Smooth-L1 readouts all reconstruct the butterfly and track its two lobes before slowly drifting outward (VPT 277 , 270 , and 148 ; Table 7 ), whereas the L1 readout leaves the attractor earliest, collapsing away from the far lobe (VPT 109 ). The Smooth-L1 rollout attains the lowest RMSE of the three regularized readouts while, unlike the coordinate-descent L1 solver, remaining compatible with second-order optimization.
Figure 8: Federated test accuracy (left axis, solid) and readout sparsity (right axis, dashed) as the number of clients K is swept from 2 to 50 ( 50 rounds, IID partition, shared threshold τ=0.1 , mean ± 1 std over ten seeds). The dotted vertical line marks the K=5 setting used in the main-text experiments. Accuracy is essentially flat in K on every dataset, and sparsity stays in the band observed at K=5 .
We introduce a diagonalization-based optimization for Linear Echo State Networks (ESNs) that reduces the per-step computational complexity of reservoir state updates from quadratic to linear. By reformulating reservoir dynamics in the eigenbasis of the recurrent matrix, the recurrent update becomes a set of independent element-wise operations, eliminating the matrix multiplication. We further propose three methods to use our optimization depending on the situation: (i) Eigenbasis Weight Transformation (EWT), which preserves the dynamics of standard and trained Linear ESNs, (ii) End-to-End Eigenbasis Training (EET), which directly optimizes readout weights in the transformed space and (iii) Direct Parameter Generation (DPG), that bypasses matrix diagonalization by directly sampling eigenvalues and eigenvectors, achieving comparable performance to standard Linear ESNs. Across all experiments, both our methods preserve predictive accuracy while offering significant computational speedups, making them a replacement for standard Linear ESNs computations and training, and suggesting a shift of paradigm in linear ESN towards the direct selection of eigenvalues.
Romain de Coudenhove, Yannis Bendi-Ouis, Anthony Strock +1
Mnemosyne, DI-ENS · ENS PSL, Inria Center of Bordeaux University, LaBRI, IMN · Mnemosyne +2
Spiking neural networks (SNNs) are biologically inspired energy-efficient models that use sparse binary spike-based communication between neurons, making them attractive for resource-constrained edge devices. Federated learning enables such devices to train collaboratively without sharing raw data. In time-series applications, edge devices often collect data at different time resolutions due to hardware and energy constraints. This temporal heterogeneity poses a fundamental challenge for federated learning: parameters learned at one temporal resolution do not necessarily transfer directly to another, which might result in the naive federated averaging being ineffective. Targeting SNNs and, more broadly, deep networks with stateful neurons, we propose a federated learning framework that addresses this temporal resolution mismatch. We investigate how neuron parameters learned from data at different temporal resolutions and model aggregation should be integrated. We evaluate the proposed framework across two SNN-native benchmark datasets (SHD and DVS-Gesture) under a range of resolution heterogeneity scenarios. Our results show that the proposed adaptation methods can substantially recover accuracy lost due to temporal mismatch, hence enabling each client to train at their local temporal resolution while remaining compatible with the global model.
Echo State Networks (ESNs) are a reservoir computing framework widely used for nonlinear time-series prediction. However, despite their effectiveness, randomly initialized reservoirs often contain redundant nodes, leading to unnecessary computational overhead and reduced efficiency. In this work, we propose a graph centrality-based pruning approach that interprets the reservoir as a weighted directed graph and removes structurally less important nodes using centrality measures. Experiments on Mackey-Glass time-series prediction and electric load forecasting demonstrate that the proposed method can significantly reduce reservoir size while maintaining, and in some cases improving, prediction accuracy.