Empirical Boundaries of Reservoir Computing for Few-Shot Sequence Pattern Tasks

Mario Mallea; Angela Nebot; Franscisco Mugica

cs.LGOct 14, 2026

Empirical Boundaries of Reservoir Computing for Few-Shot Sequence Pattern Tasks

Authors: Mario Mallea, Angela Nebot, Franscisco Mugica

Organizations: Universitat Politècnica de Catalunya - BarcelonaTech, Soft Computing Research Group at the Intelligent Data Science and Artificial Intelligence Research Center, Barcelona, Spain.

Abstract

Efficient sequence modeling remains a fundamental challenge in machine learning. Despite the success of Transformers, recurrent neural networks (RNNs) have recently re-emerged as efficient alternatives. Particularly, Echo State Networks (ESNs) offer a compelling framework of randomized RNNs that have been extensively studied for time-series forecasting and dynamical systems. This paper presents an empirical analysis of distributional, architectural, and optimization variants of ESNs evaluated on few-shot sequence-to-sequence (seq2seq) pattern tasks. These synthetic tasks utilize discrete symbol sequences, characterized by delays, inactive phases, and triggered markers to isolate and evaluate memory and pattern recognition capabilities. The minimality of these tasks is crucial to understand the strengths and limitations of machine learning models. Furthermore, when the input signal utilizes a one-hot encoding, ESN dynamics are governed strictly by their weight distributions, offering an ideal lens through which to study reservoir computing. Our findings map the performance boundaries of ESNs relative to state-of-the-art (SOTA) models. We show that preserving a linear state regime is vital for memory tasks, such as pattern completion or simple copy. We propose a data-dependent modulation mechanism for nonlinear input injection into the reservoir (G-ESN). G-ESN functions as a statistical filter capable of solving tasks that conventional ESNs cannot (the adding problem and bracket matching). Finally, for Deep ESNs, we evaluate a shared readout mechanism across layers that outperforms conventional concatenated readouts on structural tasks, including selective copy, sorting problem, and bracket matching.

Figures & tables

Model	Hyperparameter	Grid Search	Results (Appendix)
ESN and LESN	$\lambda$	$[0.0001,0.001,0.01,0.1,1,10,100]$	Tab. 2
Deep ESN variants	$\lambda$	$[0.0001,0.001,0.01,0.1,1,10,100]$	Tab. 3
LeakyESN	$\alpha$	$[0.01,0.1,0.3,,.05,0.7,0.9]$	Tab. 4
G-ESN	$b$	$[0,-1,-3,-5,-7,-10]$	Tab. 5

Table 1: Validation experiments for ESN variants. Best

\lambda

found for ESN was transferred to LeakyESN and G-ESN.

Model	Init	Sinus	Chaotic	Disc. post.	Cont. post.
LESN	IID	0.001 $\pm$ 0.000 (0.001)	0.042 $\pm$ 0.026 (0.021)	0.000 $\pm$ 0.000 (0.000)	0.000 $\pm$ 0.000 (0.000)
LESN	ON	0.001 $\pm$ 0.001 (0.001)	0.044 $\pm$ 0.025 (0.020)	0.000 $\pm$ 0.000 (0.000)	0.000 $\pm$ 0.000 (0.000)
ESN	IID	0.000 $\pm$ 0.000 (0.000)	0.011 $\pm$ 0.000 (0.011)	0.000 $\pm$ 0.000 (0.000)	0.000 $\pm$ 0.000 (0.000)
ESN	ON	0.001 $\pm$ 0.001 (0.001)	0.044 $\pm$ 0.025 (0.020)	0.000 $\pm$ 0.000 (0.000)	0.000 ∗ $\pm$ 0.000 (0.000)
G-ESN	IID	0.001 $\pm$ 0.000 (0.001)	0.368 $\pm$ 0.175 (0.024)	0.443 $\pm$ 0.175 (0.092)	0.096 $\pm$ 0.002 (0.094)
G-ESN	ON	0.003 $\pm$ 0.003 (0.001)	0.117 ∗ $\pm$ 0.101 (0.036)	0.433 $\pm$ 0.049 (0.337)	0.101 $\pm$ 0.004 (0.098)

Table 2: Reservoir computing performance for the tasks: Sinus, Chaotic, Disc. post., and Cont. post.

Model	Init	Disc. patt.	Cont. patt.	Bracket	Copy
LESN	IID	0.056 $\pm$ 0.006 (0.050)	0.050 $\pm$ 0.003 (0.044)	0.400 $\pm$ 0.102 (0.200)	0.000 $\pm$ 0.000 (0.000)
LESN	ON	0.056 $\pm$ 0.009 (0.046)	0.050 $\pm$ 0.004 (0.044)	0.388 $\pm$ 0.085 (0.220)	0.000 $\pm$ 0.000 (0.000)
ESN	IID	0.076 $\pm$ 0.008 (0.069)	0.023 ∗ $\pm$ 0.002 (0.021)	0.392 $\pm$ 0.028 (0.340)	0.400 $\pm$ 0.005 (0.394)
ESN	ON	0.057 ∗ $\pm$ 0.009 (0.046)	0.048 $\pm$ 0.004 (0.044)	0.396 $\pm$ 0.066 (0.270)	0.000 ∗ $\pm$ 0.000 (0.000)
G-ESN	IID	0.131 $\pm$ 0.016 (0.111)	0.042 ∗ $\pm$ 0.004 (0.037)	0.256 $\pm$ 0.254 (0.040)	0.471 $\pm$ 0.007 (0.465)
G-ESN	ON	0.124 $\pm$ 0.050 (0.071)	0.049 $\pm$ 0.003 (0.043)	0.052 $\pm$ 0.015 (0.040)	0.430 ∗ $\pm$ 0.012 (0.418)

Table 3: Reservoir computing performance for the tasks: Disc. patt., Cont. patt., Bracket, and Copy.

Model	Init	S-Copy	Adding	Sorting	SMNIST
LESN	IID	0.645 $\pm$ 0.021 (0.622)	0.722 $\pm$ 0.025 (0.690)	0.578 $\pm$ 0.012 (0.560)	0.436 $\pm$ 0.081 (0.290)
LESN	ON	0.645 $\pm$ 0.029 (0.616)	0.720 $\pm$ 0.018 (0.690)	0.568 $\pm$ 0.008 (0.557)	0.452 $\pm$ 0.067 (0.320)
ESN	IID	0.634 $\pm$ 0.026 (0.594)	0.762 $\pm$ 0.075 (0.670)	0.601 $\pm$ 0.019 (0.579)	0.622 $\pm$ 0.028 (0.590)
ESN	ON	0.642 $\pm$ 0.031 (0.612)	0.726 $\pm$ 0.015 (0.710)	0.568 ∗ $\pm$ 0.009 (0.555)	0.466 ∗ $\pm$ 0.050 (0.370)
G-ESN	IID	0.429 $\pm$ 0.014 (0.410)	0.006 ∗ $\pm$ 0.005 (0.000)	0.518 $\pm$ 0.034 (0.487)	0.448 $\pm$ 0.020 (0.420)
G-ESN	ON	0.436 $\pm$ 0.015 (0.414)	0.452 $\pm$ 0.186 (0.250)	0.504 $\pm$ 0.011 (0.491)	0.348 ∗ $\pm$ 0.043 (0.280)

Table 4: Reservoir computing performance for the tasks: S-copy, Adding, Sorting, and SMNIST.

Figure 1: We represent a sample of the adding problem, which is a sequence of $12$ elements: a sequence of $10$ time steps with three symbols (first three rows of Data Features), two markers (fourth row of Data Features) in yellow (at time steps $2$ and $7$ ), a trigger (last row of Data Features), and a last state expecting the result (marked by the red box). We must learn that the sum of $1$ (at $t=2$ ) plus $0$ (at $t=7$ ) equals $1$ (at $t=11$ ). Note that $\alpha_{t}$ has $d_{r}=2000$ as $d_{o}=5$ possible results.

		SOTA		ESN
Task	S6	Best [ 3 ]	Model [ 3 ]	Best	Model
S-Copy	0.348	0.426	EST / 10k	0.429	G-ESN / IID
Sorting	0.358	0.473	EST / 10k	0.477	LeakyESN / IID
Sinus	0.009	0.007	Transformer / 100k	0.000	ESN / IID
Chaotic	0.028	0.011	Transformer / 1M	0.011	ESN / IID
Disc. post.	0.432	0.000	EST & GRU / 10k	0.000	1

Table 5: Performance comparison on the Stream benchmark. Rows are color-coded to group tasks where the gap between SOTA and ESN is less than

0.05

Figure 2: Accuracy comparison between two groups of task (high is better). Colors differentiate IID and ON initialization methods. Identity denotes the function $y=x$ .

Sequence-to-Sequence Models

See all 11 papers

Jun 23, 2026

Frequency Domain Reservoir Computing

While the quadratic sequence-length bottleneck of transformers has fueled a resurgence in recurrent models, effectively capturing complex dynamics requires architectures that balance efficient training with highly expressive latent states. Echo State Networks (ESNs) offer a compelling approach by utilizing fixed recurrent weights to circumvent backpropagation through time, enabling a closed-form training solution. However, achieving the expressivity needed for complex tasks demands large reservoirs, exposing an

\mathcal{O}(N^2)

state-update bottleneck that prevents ESNs from matching the scale of contemporary recurrent models. To address this limitation, we introduce Frequency Domain Reservoir Computing (FRESCO), an ESN architecture operating entirely in the frequency domain while avoiding domain-shift overheads to achieve

\mathcal{O}(N)

complexity for dense, non-linear recurrent updates. By employing a novel dimensional zero-padding input embedding, a packed \FDh readout, and a natively applied frequency-domain non-linearity, FRESCO drastically reduces computational costs and energy consumption of training and inference. Furthermore, FRESCO matches the state-of-the-art predictive performance on memory benchmarks, sequential classification, and multivariate long-horizon forecasting, offering a scalable path forward for dense recurrent architectures.

Klaus Schertler, Xiomara Runge, Andrea Ceni +2

Airbus Central Research & Technology Taufkirchen, Germany · Department of Computer Science University of Pisa, Italy · Faculty of Technology Bielefeld University, Germany

Time Series Forecasting Recurrent Neural Networks Reservoir Computing

Ask PDF

May 27, 2026

Adaptive Reservoir Computing for Multi-Scenario Chaotic System Forecasting

We present an adaptive reservoir computing framework for the CTF-4-Science Lorenz benchmark, which evaluates machine learning models across twelve distinct tasks spanning five qualitatively different scenarios: baseline forecasting, noisy signal reconstruction, forecasting under noise, few-shot learning, and parametric generalization. Rather than applying a uniform inference strategy, we tailor the training and prediction procedure of Echo State Networks (ESNs) to the specific demands of each evaluation scenario. Our key contributions are fourfold: (1) exact reservoir state synchronization that eliminates warmup approximation error in short-time prediction; (2) histogram-guided candidate selection that directly optimizes the long-time ergodic evaluation metric; (3) multi-seed reservoir search for few-shot regimes with severely limited training data; and (4) sequential multi-sequence training that resolves state-distribution mismatch in parametric generalization tasks. The proposed framework achieves a score of 74.91 on the public benchmark leaderboard, demonstrating that carefully adapted reservoir computing constitutes a competitive and computationally efficient approach for diverse chaotic system modeling challenges.

Shadmehr Zaregarizi, Khashayar Yavari

Politecnico di Torino Turin, Italy

Dynamic Neural Networks Forecasting Benchmarks Reservoir Computing Chaotic Time-Series Prediction

Ask PDF

Aug 5, 2026

Rethinking Reservoir Pruning: A Dynamical Perspective for Echo State Networks

Echo State Networks (ESNs) offer an efficient framework for temporal prediction, but their randomly initialized reservoirs are often over-parameterized and dynamically redundant. Existing pruning methods largely rely on static connectivity or activation statistics, which may overlook neurons that shape input-driven state transitions. We propose Dynamical Mode Pruning (DMP), a reservoir pruning method that ranks neurons by their contribution to dominant transition modes obtained from a trajectory-averaged Jacobian Gramian. DMP removes low-impact units and retrains only the readout. Experiments on chaotic and real-world time-series benchmarks show that DMP improves or preserves forecasting accuracy while reducing redundant reservoir components. Our results suggest that dynamical influence is a useful criterion for reservoir refinement beyond static structural importance alone.

Sudip Laudari, Puspa Raj Adhikari

Independent Researcher. · Advanced Materials Science and Engineering, Sungkyunkwan University.

Dynamic Neural Networks Time Series Forecasting Reservoir Computing Neural Network Pruning

Ask PDF

Abstract

Sequence-to-Sequence Models

Explore similar work

Frequency Domain Reservoir Computing

Adaptive Reservoir Computing for Multi-Scenario Chaotic System Forecasting

Rethinking Reservoir Pruning: A Dynamical Perspective for Echo State Networks