RNN T Training

Training Recurrent Neural Networks (RNNs) remains a significant challenge due to inherent instability in gradient calculations, particularly for long sequences. Current research focuses on improving training stability and efficiency through techniques like gradient flossing (controlling Lyapunov exponents), alternative training objectives beyond maximum likelihood estimation (e.g., learning to search), and architectural modifications such as incorporating acoustic lookahead in RNN-Transducers for speech recognition. These advancements aim to enhance RNN performance in various applications, including machine translation, speech recognition, and time-series forecasting, particularly in low-resource settings where data scarcity is a major limitation.

Papers

May 20, 2024

Beyond MLE: Investigating SEARNN for Low-Resourced Neural Machine Translation
Chris Emezue
Machine Translation Recurrent Neural Network Maximum Likelihood Low Resource Neural Machine Translation RNN T Training

December 28, 2023

Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians
Rainer Engelken
Gradient Descent Recurrent Neural Network Dynamic Control Continual Training RNN Architecture Jacobian Matrix Lyapunov Exponent RNN T Training

October 26, 2023

Bifurcations and loss jumps in RNN training
Lukas Eisenmann, Zahra Monfared, Niclas Alexander Göring, Daniel Durstewitz
Recurrent Neural Network \Sigma}{\Delta}$ Low Pas RNN Bifurcation Diagram RNN T Training Loss Spike

July 11, 2023

Improving RNN-Transducers with Acoustic LookAhead
Vinit S. Unni, Ashish Mittal, Preethi Jyothi, Sunita Sarawagi
Text Representation Mt RNN RNN Transducer RNN T Training Lookahead Decoding

June 2, 2023

Streaming Speech-to-Confusion Network Speech Recognition
Denis Filimonov, Prabhat Pandey, Ariya Rastrow, Ankur Gandhe, Andreas Stolcke
Automatic Speech Recognition RNN T Training Streaming End to End

March 18, 2023

Powerful and Extensible WFST Framework for RNN-Transducer Losses
Aleksandr Laptev, Vladimir Bataev, Igor Gitman, Boris Ginsburg
Mt RNN RNN Transducer Finite State Transducer RNN T Training RNN T Loss

October 29, 2022

Accelerating RNN-T Training and Inference Using CTC guidance
Yongqiang Wang, Zhehuai Chen, Chengjian Zheng, Yu Zhang, Wei Han, Parisa Haghani
Scientific Inference Connectionist Temporal Classification Mt RNN Recurrent Neural Network Transducer RNN T Training RNN Inference

January 25, 2022

Improving the fusion of acoustic and text representations in RNN-T
Chao Zhang, Bo Li, Zhiyun Lu, Tara N. Sainath, Shuo-yiin Chang
Hybrid Fusion Text Representation Multilingual Automatic Speech Recognition Speech Encoder Mt RNN Recurrent Neural Network Transducer RNN T Training

January 13, 2022

Recognizing semantic relation in sentence pairs using Tree-RNNs and Typed dependencies
Jeena Kleenankandy, K A Abdul Nazeer
Mt RNN Sentence Pair Semantic Textual Relatedness RNN T Training Tree LSTMs

December 27, 2021

Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies
Paul Vicol, Luke Metz, Jascha Sohl-Dickstein
Evolution Strategy Unbiased Gradient Computation Graph RNN T Training Unrolled Optimization

December 1, 2021

Investigation of Training Label Error Impact on RNN-T
I-Fan Chen, Brian King, Jasha Droppo
Comprehensive Investigation Mt RNN Label Error RNN T Training