Recurrent sequence models, such as recurrent neural networks (RNNs) and state space models (SSMs), are often trained to emit an end-of-sequence (EOS) token when their output sequence is complete. During training, EOS is treated as another token without a special loss term or incentive. Although this training paradigm is widely used, it is not understood how models decide to output EOS. We study a minimal version of the EOS halting problem in which networks are trained to emit a given input sequence an instructed number of times, followed by EOS. We train tanh RNNs and linear SSMs on this task, finding that the RNNs encode the number of tokens left to copy in a linear subspace, of which two principal components capture most of the variation. Hidden states with the same remaining count converge to a shared representation regardless of their instructed repeat count. By modulating the hidden state using an affine transformation at the boundary between copied sequences, we shift the RNN's state to the representation shared by states with one fewer copy remaining. The unchanged network emits EOS one copy earlier while reproducing the shortened target sequence in 94.4% of new sequences, revealing that this representation causally determines the stopping time. In the SSMs, the difference between trajectories with different repeat counts is separated across two-dimensional rotation planes whose contributions to the EOS comparison cancel during copying and align near the stopping time. Rotating the planes independently removes most of the intended stopping while preserving the copied content, which shows that the stopping time depends on the planes' relative phases. These results identify two candidate primitives of EOS halting in recurrent models: a shared encoding of the number of remaining tokens in tanh RNNs and phase alignment across fixed rotations in linear SSMs.
Figures & tables
Figure 1: The count instruction changes the EOS time while the stored sequence remains fixed. A , RNN trajectory differences on held-out sequences projected onto two principal components obtained from separate sequences. Open circles mark the state after the count instruction, stars mark the earlier target EOS, and arrows follow the autonomous recurrence. B , mean contributions of the 64 SSM rotation blocks to the limiting EOS comparison for count change 4→2 . Between copy boundaries the contributions have mixed signs. At the target EOS, most contributions have the same sign and the EOS margin crosses zero. C , repeat-copy examples ending after two or four copies of S . RNN seed 5, the model with the median energy in the two plotted components, and SSM seed 1 are shown.
Secants used in the transport
Continuation accuracy
Exact
1.000
Identity, DtJK=I
0.000
Permuted across hidden units
0.010
Mean across hidden units, same sequence and step
0.091
Circularly shifted in time
0.316
Taken from other sequences
0.669
Table 1: Replacements for the tanh secants DtJK in Equation 4 , evaluated across the five RNNs and the three count changes. Each replacement propagates ΔtJK over the final copy of the shorter target. We add the result to the trajectory of the longer count and decode it through Wy , without rerunning the network. Continuation accuracy is the fraction of these reconstructions whose output matches the output of the unmodified network. The exact row is a numerical check.
Figure 2: Transport of the RNN trajectory difference. A , mean tanh secants for model seed 5 and count change 4→2 , with hidden units ordered by the step of strongest attenuation on separate sequences. B , multiplication by Wh expands the trajectory difference before tanh attenuates it. Faint lines denote model means and solid lines denote the cohort mean. C , EOS margins along the two trajectories. D , continuation accuracy under the exact secants and their controlled replacements.
Figure 3: RNN states indexed by the number of output steps remaining. A , distances between states that share (n,p) and S under different count instructions, between states one copy apart, and between states storing other content, each relative to the distance between states one copy apart. Gray lines denote models and the blue line denotes their mean. B , cycle centers projected onto two principal components obtained from separate sequences. Arrows follow one complete copy. C , evaluation states of model seed 3, the model with the median R2 , after subtracting the fitting-set mean state of each emitted symbol, projected onto the first two principal components of the fitting states and colored by τ . D , stopping and content accuracy after the affine state transformation of Equation 7 as a function of its rank. E , EOS margins after the native recurrence advances a state by two copies.
Intervention
Stopping
Content
Complete
RNN, affine state transformation T128 from n to n−1 copies remaining
Fitted transformation, new sequences
0.995
0.994
0.944
Mean change, same count and cycle
0.829
0.976
0.728
Targets shuffled across sequences
0.456
0.049
0.000
h replaced by mean target state
0.520
0.046
0.000
SSM, amplitude-preserving phase rotations
Table 2: Interventions on the stopping mechanism, evaluated on held-out sequences. All accuracies are computed over every rollout. RNN entries are means over the five models, and SSM entries are ranges across the five models. The fixed-direction rows pool the targets r=1,2,4,5 , starting from a rollout of three copies. Since the SSM rotations preserve almost all of the copied content, their complete-sequence accuracy rounds to the same range as their stopping accuracy.
Figure 4: Coherence of the rotation-plane contributions at EOS in each SSM, compared with independent amplitude-preserving rotations of the planes. Stopping and content accuracy under these rotations are given in Table 2 .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Magnitude of ΔtJK and its visibility through the EOS comparisons. A , magnitude of ΔtJK relative to the start of the last copy. B , visibility through all EOS comparisons. C , cosine with the weakest EOS comparison at the target state. D , coherence of the SSM plane contributions. Faint lines denote models and dark lines denote their means.
Figure 6: Propagation of the RNN trajectory difference from the instruction to the EOS comparisons for model seed 5 and count change 4→2 . A , the preactivation response Wx(cJ−cK) and resulting hidden response at the instruction. Hidden units are ordered by the step of their largest response on separate sequences. B , the hidden response during autonomous output, with horizontal lines at copy boundaries. C , the norm of ΔtJK , its effect on the EOS comparisons, and the difference between the two EOS margins, each scaled by its largest absolute value.
Figure 7: Fixed short-copy dynamics under reduced recurrences. A , agreement of the rank three system across five models. B , EOS margins from the full and reduced systems on the mixed sequences. C , contributions of output position and symbols to the exhaustive state table. D , output agreement for the reduced system, random subspaces, and the linearized RNN. Rank three preserves every output in four of the five RNNs and the stopping time in four of the five SSMs.
Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length.
Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations. Standard backpropagation through time (BPTT) addresses this problem poorly: it is sequential in time, limiting parallelism, and suffers from vanishing or exploding gradients, making long-range associations difficult to learn. We propose Supervised Memory Training (SMT), a method for training nonlinear RNNs that sidesteps recurrent credit propagation entirely by reducing RNN training to supervised learning on one-step memory transition labels (mt,xt+1)→mt+1. SMT acquires these memory labels by training a Transformer-based encoder on a predictive state objective--retaining only information from the past necessary to predict the future. By decoupling what to remember from how to update memory, SMT enables time-parallel RNN training with a stable O(1) length gradient path between any two tokens--without ever unrolling the RNN. We find that SMT outperforms BPTT when pretraining various RNN architectures on tasks like language modeling and pixel sequence modeling. SMT enables nonlinear RNNs to better capture long-range dependencies and train in parallel, potentially unlocking the scaling of models that build temporal abstractions of past experience.
Linear recurrent neural networks, such as State Space Models (SSMs) and Linear Recurrent Units (LRUs), have recently shown state-of-the-art performance on long sequence modelling benchmarks. Despite their success, their empirical performance is not well understood and they come with a number of drawbacks, most notably their complex initialisation and normalisation schemes. In this work, we address some of these issues by proposing RotRNN -- a linear recurrent model which utilises the convenient properties of rotation matrices. We show that RotRNN provides a simple and efficient model with a robust normalisation procedure, and a practical implementation that remains faithful to its theoretical derivation. RotRNN also achieves competitive performance to state-of-the-art linear recurrent models on several long sequence modelling datasets.
Kai Biegun, Rares Dolga, Jake Cunningham +1
UCL AI Centre London, UK · UCL AI Centre & UiPath London, UK