Recurrent Neural Networks

Also known as RNN

Recurrent neural networks study how a model can process ordered inputs by repeatedly updating an internal state, which is useful for streaming data, long-term memory, and decisions under partial observability. Common approaches include gated LSTMs and GRUs, linear and state-space recurrences, continuous-time and spiking networks, and weight-shared loops for iterative reasoning; training ranges from backpropagation through time to online learning and parallel scans. Researchers test these models on state-tracking and recall tasks, language modeling, time-series forecasting, control, and physical-system prediction. A central tension is how to preserve useful information and stable learning over long sequences while keeping computation efficient and matching the accuracy and flexibility of Transformers.

Latest papers

302 papers47 papers in the last four weeks, up 15% on the four weeks before. 0.3% of all new papers.

ELiSe: Efficient Learning of Sequences in Structured Recurrent Networks
Feb 26, 2024Laura Kriener, Ben von Hünerbein, Kristin Spicher +5Neural Processes Recurrent Neural Networks
Department of Physiology, University of Bern, 3012 Bern, Switzerland. · Institute of Neuroinformatics, University of Zurich and ETH Zurich, 8057 Zurich, Switzerland. · Catlab Engineering GmbH, 82284 Grafrath, Germany.
On Orthogonal Recurrent Neural Networks Quantization
Feb 5, 2024Armand Foucault, Franck Mamalet, François MalgouyresRecurrent Neural Networks Neural Network Quantization
IMT · Institut de Mathématiques de Toulouse, UMR5219. Université de Toulouse, CNRS. UPS IMT, F-31062 Toulouse Cedex 9, France · Comue de Toulouse +1

Feb 26, 2024J/K move · Enter open · S save

ELiSe: Efficient Learning of Sequences in Structured Recurrent Networks

Laura Kriener, Ben von Hünerbein, Kristin Spicher, Federico Benitez+4

Department of Physiology, University of Bern, 3012 Bern, Switzerland. · Institute of Neuroinformatics, University of Zurich and ETH Zurich, 8057 Zurich, Switzerland. · Catlab Engineering GmbH, 82284 Grafrath, Germany.

Neural Processes Recurrent Neural Networks Biologically Plausible Learning

Open paper AskPDF arXiv

Behavior can be described as a temporal sequence of actions driven by neural activity. To learn complex sequential patterns in neural networks, memories of past activities need to persist on significantly longer timescales than the relaxation times of single-neuron activity. While recurrent networks can produce such long transients, training these networks is a challenge. Learning via error propagation confers models such as FORCE, RTRL or BPTT a significant functional advantage, but at the expense of biological plausibility. While reservoir computing circumvents this issue by learning only the readout weights, it does not scale well with problem complexity. We propose that two prominent structural features of cortical networks can alleviate these issues: the presence of a certain network scaffold at the onset of learning and the existence of dendritic compartments for enhancing neuronal information storage and computation. Our resulting model for Efficient Learning of Sequences (ELiSe) builds on these features to acquire and replay complex non-Markovian spatio-temporal patterns using only local, always-on and phase-free synaptic plasticity. We showcase the capabilities of ELiSe in a mock-up of birdsong learning, and demonstrate its flexibility with respect to parametrization, as well as its robustness to external disturbances.

Latest papers

ELiSe: Efficient Learning of Sequences in Structured Recurrent Networks

On Orthogonal Recurrent Neural Networks Quantization