Parallel Training of Recurrent Networks

Latest papers

11 papers1 paper in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

LoopFold: Parallelizing Recurrent Depth in Looped Transformers
Oct 26, 2026Feichen Song, Boyi Zeng, Yiqin Hao +7Parallel Training of Recurrent Networks Recurrent Transformers
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute
ADPTNet: Adaptive with Prescriptive Timescales Non-Linear SSM for Sequence Modelling
Sep 27, 2026Matei-Ioan Stan, Oliver RhodesParallel Training of Recurrent Networks Recurrent Neural Networks
International Centre for Neuromorphic Systems The University of Manchester Manchester, United Kingdom
Phase State Space Models: Parallel, Surrogate-Free Training of Spiking Networks
Aug 7, 2026Wilkie Olin-AmmentorpSpiking Neural Networks State Space Models
Argonne National Laboratory, Lemont, Illinois, USA
Pretraining Recurrent Networks without Recurrence
Jun 4, 2026Akarsh Kumar, Phillip IsolaParallel Training of Recurrent Networks Recurrent Neural Networks
MIT
Parallel Recursive LSTM
May 16, 2026Tristan Gaudreault, Yongyi MaoParallel Training of Recurrent Networks Recurrent Neural Networks
School of Electrical Engineering and Computer Science University of Ottawa Ottawa, ON
Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction
May 12, 2026Florian Hess, Florian Götz, Daniel DurstewitzNonlinear System Identification Recurrent Neural Networks
1Dept. of Theoretical Neuroscience, Central Institute of Mental Health,2026 Mannheim, Germany · Faculty of Mathematics and Computer Science, Heidelberg University, Germany · Faculty of Physics and Astronomy, Heidelberg University, Germany +1
Improving the Performance and Learning Stability of Parallelizable RNNs Designed for Ultra-Low Power Applications
May 12, 2026Julien Brandoit, Arthur Fyon, Damien Ernst +1Energy-Efficient ML Parallel Training of Recurrent Networks
Department of Electrical Engineering and Computer Science, University of Liège, Liège, Belgium.
ParaRNN: An Interpretable and Parallelizable Recurrent Neural Network for Time-Dependent Data
May 4, 2026Yuxi Cai, Lan Li, Feiqing Huang +1Parallel Training of Recurrent Networks Interpretable ML
Department of Statistics and Actuarial Science, University of Hong Kong
State Stream Transformer (SST) V2: Parallel Training of Nonlinear Recurrence for Latent Space Reasoning
Apr 30, 2026Thea AvissParallel Training of Recurrent Networks Recurrent Neural Networks
Fifth Dimension
Why Are Linear RNNs More Parallelizable?
Mar 4, 2026William Merrill, Hongjian Jiang, Yanhong Li +2Parallel Training of Recurrent Networks Circuit Complexity
Allen Institute for AI · Rheinland-Pfälzische Technische Universität · Max-Planck Institute for Software Systems
Parallelizable memory recurrent units
Jan 14, 2026Florent De Geeter, Gaspard Lambrechts, Damien Ernst +1Recurrent Neural Networks Parallel Training of Recurrent Networks
Montefiore Institute, University of Liege, Liege, Belgium

Oct 26, 2026J/K move · Enter open · S save

LoopFold: Parallelizing Recurrent Depth in Looped Transformers

Feichen Song, Boyi Zeng, Yiqin Hao, Shixiang Song+6

Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute

Parallel Training of Recurrent Networks Recurrent Transformers Efficient Transformer Inference LLM Inference Acceleration

Open paper AskPDF arXiv

Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream task performance at a fixed backbone parameter count. However, each loop depends on the preceding loop, so increasing recurrent depth also increases decoding latency. Existing approaches include Parallel Loop Transformers, which reduce latency by overlapping loop computation across token positions, but yield rapidly diminishing marginal accuracy gains at larger depths. To address this challenge, we introduce LoopFold, which folds computation at multiple recurrent depths into parallel batches for the same token prefix. A lightweight predictor estimates inputs to later loops so they can start before earlier loops finish. A corrector then refines the outputs of these later loops using newly available results from earlier loops. All components are trained jointly using only the final next-token prediction loss. In experiments with 1.4B models, deeper LoopFold configurations achieve progressively higher average zero-shot accuracy across eight tasks, rising from 53.41% to 56.38%. Vanilla looped models and PLT show limited gains over the same depth range. Compared with a nine-loop vanilla model, LoopFold uses five sequential stages and additional parallel computation to reduce decoding latency by 46%.