Gate Initialisation versus Recurrent Dropout and Zoneout in Learning Long Dependencies: A Controlled Study of LSTM and GRU Networks on Synthetic Tasks

Oleksandr Kholodniak; Oleksandr Prokhorov

cs.LGOct 21, 2026

Gate Initialisation versus Recurrent Dropout and Zoneout in Learning Long Dependencies: A Controlled Study of LSTM and GRU Networks on Synthetic Tasks

Authors: Oleksandr Kholodniak, Oleksandr Prokhorov

Organizations: National Aerospace University “Kharkiv Aviation Institute”, Kharkiv, Ukraine

Abstract

Dropout and zoneout are routinely added to long short-term memory (LSTM) and gated recurrent unit (GRU) networks, but, to our knowledge, their placements have not been crossed with gate initialisation in a factorial design, and it is not known whether early gradient propagation tells which runs will learn long dependencies. We trained one-layer LSTMs and GRUs on the adding problem and on delayed recall in a factorial grid of 720 runs, crossing conventional and chrono gate initialisation with up to six recurrent regulariser placements at one rate each, with pre-specified follow-ups, and recorded how sensitive the final hidden state was to the input at every lag. Because both tasks supply fresh or exhaustive training data, the study measures the regularisers' effects on optimisation and does not assess generalisation from a finite training set. At lengths chosen to make conventional initialisation difficult, all 360 chrono-initialised runs reached the validation criterion within 5000 updates, as did 119 of 120 unregularised runs with a horizon about four times too short or too long, whereas conventionally initialised networks learned slowly or did not reach the criterion within the budget at the longer lengths. Under conventional initialisation, dropout generally delayed learning and zoneout helped only in some conditions. The early input sensitivity added little to predicting which conditions learn once the early validation loss was modelled per task, architecture and initialisation. Within conditions, a flatter early decay went with the seeds that learned, but in a pre-specified test the association was modest (concordance 0.61), condition-dependent and not robust to resampling seeds. The results support chrono initialisation, even with a rough horizon, on such tasks.

Figures & tables

Figure 1: Chrono initialisation reached the criterion in every main-grid run; conventional initialisation often did not at the longer lengths, where the regulariser could change the outcome. Each cell gives the runs that reached the validation criterion within 5000 updates (of five seeds) and the restricted mean number of updates to the criterion (runs that did not reach it count as 5000); darker shading means more runs reached it. Blank cells: cell-state and both-state zoneout, not defined for the GRU. Columns combine architecture, initialisation (conv. = conventional) and length.

Plan	Content	Seeds	Runs	Budget
Calibration	unregularised; adding $T\in\{50,\dots,400\}$ , recall $T\in\{10,\dots,500\}$	0–1	104	5000
Main grid	2 architectures $\times$ 2 initialisations $\times$ 5 or 7 regularisers $\times$ 6 lengths	0–4	720	5000
More seeds	the 13 main-grid conditions with mixed outcomes	5–19	195	5000
Long budget	32 conventional conditions below 5/5 at shorter lengths	0–4	160	20 000
Horizon mismatch	unregularised, chrono with $T_{\max}\approx T/4$ or $4T$	0–4	120	5000
PyTorch replication	the 24 unregularised main-grid conditions, PyTorch nn.LSTM / nn.GRU ; outcomes only	0–4	120	5000

Table 1: Experimental plans. Conditions combine task, length, architecture, initialisation and regulariser; the GRU has five regulariser settings (no cell-state zoneout), the LSTM seven.

∗ Already 5 of 5 within 5000 updates in the main grid; not rerun.
		none	rec.	var.	upd.	zo. cell	zo. hid.	zo. both
LSTM adding $T=200$	by 5000 $\to$ 20 000	0 $\to$ 5	0 $\to$ 5	0 $\to$ 5	0 $\to$ 5	0 $\to$ 5	0 $\to$ 5	0 $\to$ 5
	median	8100	9000	11 200	10 100	8500	8700	9100
GRU recall $T=30$	by 5000 $\to$ 20 000	2 $\to$ 5	0 $\to$ 5	0 $\to$ 5	0 $\to$ 2	–	5 $\to$ 5 ∗	–
GRU recall $T=45$	by 5000 $\to$ 20 000	0 $\to$ 5	0 $\to$ 0	0 $\to$ 5	0 $\to$ 0	–	0 $\to$ 2	–
LSTM recall $T=45$	by 5000 $\to$ 20 000	0 $\to$ 3	0 $\to$ 1	1 $\to$ 1	1 $\to$ 1	0 $\to$ 2	1 $\to$ 3	1 $\to$ 3

Table 2: Long budget: conventionally initialised conditions below 5/5 at 5000 updates, rerun with 20 000. Runs (of five) that reached the criterion by 5000

\to

by 20 000 updates, and median updates among those that reached it. Selected conditions; all 32 are tabulated in the repository. rec. = recurrent, var. = variational, upd. = update dropout; zo. = zoneout of the cell, hidden or both states.

Figure 2: Under conventional initialisation the input sensitivity decays roughly exponentially with lag; chrono networks attenuate far less. Input sensitivity of the final hidden state,

G(k)/\ln 10

, against lag

k

for recall at

T=45

after 100 updates, evaluation mode, mean over five seeds.

	Conventional		Chrono
Regulariser	LSTM	GRU	LSTM	GRU
rec. dropout	$-0.006$ [ $-0.023$ , $0.011$ ]	$-0.017$ [ $-0.022$ , $-0.012$ ]	$-0.001$ [ $-0.002$ , $0.001$ ]	$-0.001$ [ $-0.002$ , $0.000$ ]
var. dropout	$-0.038$ [ $-0.057$ , $-0.019$ ]	$-0.060$ [ $-0.068$ , $-0.052$ ]	$-0.006$ [ $-0.012$ , $-0.001$ ]	$-0.007$ [ $-0.013$ , $-0.002$ ]
upd. dropout	$-0.008$ [ $-0.032$ , $0.016$ ]	$-0.034$ [ $-0.043$ , $-0.024$ ]	$-0.001$ [ $-0.004$ , $0.001$ ]	$-0.002$ [ $-0.007$ , $0.002$ ]
zoneout, cell	$0.033$ [ $0.015$ , $0.050$ ]	–	$0.000$ [ $-0.006$ , $0.006$ ]	–
zoneout, hidden	$0.003$ [ $-0.009$ , $0.016$ ]	$0.066$ [ $0.062$ , $0.070$ ]	$0.000$ [ $-0.001$ , $0.001$ ]	$-0.003$ [ $-0.011$ , $0.004$ ]

Table 3: Change of the slope

s

at update 100 (evaluation mode) against the unregularised network with the same seed, task and length, averaged over lengths within each seed; mean and pointwise 95 %

t

interval over five seeds (exploratory). Positive values mean slower decay; rec., var., upd. = recurrent, variational, update. Zoneout values include the direct effect of its expected mask in evaluation mode, present before training (see text).

Baseline	Held out	Baseline	$+\,s$	Reduction	Interval
additive	conditions (10 folds)	0.239	0.165	0.074	[0.044, 0.104]
	crossed interval				[0.022, 0.120]
stratified	conditions (10 folds)	0.162	0.154	0.008	[ $-0.002$ , 0.018]
	crossed interval				[ $-0.009$ , 0.025]
stratified	seeds	0.141	0.130	0.011	[0.002, 0.022]
stratified	regulariser families	0.301	0.270	0.030	[0.006, 0.059]

Table 4: Landmark prediction of reaching the criterion from update 100 (720 runs, 480 reached it; seeds 2–4 only: 432 runs, 288 reached it). Log loss of the gradient-free baseline and of the baseline plus the slope

s

, out of sample, and the reduction with a 95 % interval from resampling conditions; for held-out conditions also from resampling conditions and seeds crossed. Intervals hold the out-of-sample predictions fixed; reductions are computed before rounding.

Figure 3: Seeds that learned tended to have flatter early slopes in the LSTM recall conditions, but not in LSTM adding at

T=100

. Confirmatory runs: slope

s

at update 100 for every run with seeds 5–19 in the 13 conditions with mixed outcomes in the main grid. Filled markers reached the criterion within 5000 updates.

Recurrent Neural Networks

Recurrent neural networks process sequences by updating a persistent hidden state, enabling temporal modeling, memory, and streaming computation.

See all 302 papers

Oct 9, 2026

The RNN Homunculus: Learning to Gate Long-Term Memory

Learning long-range dependencies with recurrent neural networks (RNNs) is difficult because of the vanishing gradient problem. Gated recurrent neural networks like GRUs and LSTMs address this problem and are commonly explained by their ability to create additive copy paths through time. This explanation is correct, but incomplete: useful RNN gate values must themselves be learned, which is typically done with another RNN. We study this gate-learning problem through carefully controlled sequence classification tasks. Our analysis shows that gated architectures struggle when all training samples require long-range memory and thus equally suffer from the vanishing gradient problem. We intervene by adding short-dependency samples which let gated models learn solutions that capture long-dependency samples, but also generalize across different class labels. When properly learned, gated RNNs match or even exceed the test accuracy of modern architectures like Mamba and xLSTM. Overall, our results highlight gate learning as a central part of learning long-range memory and show how training data can support this process by exposing the same task over shorter dependencies.

Zeyu Fan, Alexandru-Dragos Manolache, Jan van Gemert

Delft University of Technology, Mekelweg 5, 2628 CD Delft, The Netherlands

Long-Range Dependency Modeling Recurrent Neural Networks Sequence Modeling Gated Recurrent Unit Long Short-Term Memory Networks Deep Learning Optimization

Ask PDF

May 27, 2026

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

Randomly initialized neural networks induce a prior over functions, but the predictor used in practice is produced only after training. We ask how much of this initial bias survives the training pipeline. To make the question measurable, we introduce initialization memory: the dependence of the validation-selected predictor on the scale of the random initialization. We perform controlled CIFAR-10 experiments on ResNets where initialization memory already sharply separates training regimes. Low-learning-rate SGD can interpolate while still remembering its initialization: on ResNet-9 with batch size

b=128

, test accuracy varies by

26.5

percentage points across initialization scales despite

\ge99.5\%

training accuracy. This is not undertraining: extending the same low-learning-rate regime to

5{,}000

epochs leaves the spread essentially unchanged. In contrast, Adam-family methods largely erase the dependence. SGD can also be made to forget when larger learning rates are paired with explicit

L_2

norm control. We interpret these findings in terms of the time scale of forgetting: gradient-flow-like dynamics can preserve initialization memory, whereas stochastic finite-step effects, explicit norm decay, and adaptive preconditioning erase it on scales governed by the size of explicit or implicit regularization. The practical inductive bias of a trained network is therefore not the architectural prior alone, but the architectural prior after being filtered by the forgetting dynamics of the training pipeline; and the same regularizers that improve generalization are precisely those that erase memory of initialization.

Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey +2

MIT · Northwestern University

Neural Network Generalization Neural Network Initialization Neural Network Training Dynamics Feedforward Neural Networks Stochastic Gradient Descent

Ask PDF

Dec 5, 2025

Learnability Window in Gated Recurrent Neural Networks

We develop a statistical theory of temporal learnability in recurrent neural networks, quantifying the maximal temporal horizon

\mathcal{H}_N

over which gradient-based learning can recover lag-dependent structure at finite sample size

N

. The theory is built on the effective learning rate envelope

f(\ell)

, a function that captures how gating mechanisms and adaptive optimizers jointly shape the coupling between state-space dynamics and parameter updates during Backpropagation Through Time. Under heavy-tailed (

α

-stable) fluctuations, where empirical averages concentrate at rate

N^{-1/κ_α}

with

κ_α= α/(α-1)

, the interplay between envelope decay and statistical concentration yields explicit scaling laws for the growth of

\mathcal{H}_N

: logarithmic, polynomial, and exponential temporal learning regimes emerge according to the decay law of

f(\ell)

. These results identify envelope decay as the key determinant of temporal learnability. Slower attenuation of

f(\ell)

enlarges

\mathcal{H}_N

, while heavy-tailed fluctuations compress it by weakening statistical concentration. Moreover, envelope geometry outweighs dataset size: slowing the envelope's decay enlarges

\mathcal{H}_N

more than adding data, so more complex architectures that realize slower-decaying envelopes can be more data-efficient than simpler ones. Experiments across multiple gated architectures and optimizers corroborate these structural predictions.

Lorenzo Livi

OPIT – Open Institute of Technology

Gated Recurrent Unit Recurrent Neural Networks Neural Network Training Dynamics Statistical Learning Theory

Ask PDF

Abstract

Recurrent Neural Networks

Explore similar work

The RNN Homunculus: Learning to Gate Long-Term Memory

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

Learnability Window in Gated Recurrent Neural Networks