ADPTNet: Adaptive with Prescriptive Timescales Non-Linear SSM for Sequence Modelling

Matei-Ioan Stan; Oliver Rhodes

Figure 1: Trade-Offs in RNNs and SSMs This figure shows the four qualities required for state-of-the-art and versatile sequence modelling. Three of them are staples of Transformer performance: adaptability (ii), long-range sequence modelling (iii), and parallelisability (iv). The fourth, non-linear recurrence (i), is thought to enable performance unattainable by vanilla Transformers. A competitive RNN or SSM should possess all four. However, as the red arrows highlight, they are fundamentally at odds with each other.

Figure 2: Difference in predictability between LTI and Non-Linear Systems . Subfigures 2(a) and 2(b) show the vector fields of unstable/stable 2-dimensional LTI systems (

\dot{x}=A_{11}x+A_{12}y,\dot{y}=A_{21}x+A_{22}y

), where

A

has positive and negative eigenvalues, respectively. Subfigure 2(c) shows the vector field of a non-linear dynamical system:

\dot{x}=-2x-3xy,\dot{y}=3y-y^{2}

[ 87 ] . The system has two critical points: A is a stable point (sink) that attracts trajectories in its vicinity, while B is a saddle point that attracts and repels trajectories depending on their location.

Figure 3: Riemannian Gradient Descent over the Orthogonal Manifold Given an initial orthonormal matrix

Q

and a Euclidean gradient

\nabla f(Q)

, Riemannian Gradient Descent is performed by first projecting the Euclidean gradient to the tangent space to

O(n)

at

Q

, i.e.,

\mathcal{T}_{Q}

[ 3 ] . The updated

Q

from the tangent space is then projected back to the manifold using a retraction function

R

Figure 4: Topological Conjugates in Linear Dynamics . Subfigure 4(a) shows trajectories from two linear dynamical systems, one governed by

\frac{dx}{dt}=Ax

and the other by a topologically conjugate equation

\frac{dx}{dt}=C^{-1}AC

. Notably, they share the same oscillation frequency and decay back to the origin at the same rate, as they share the same eigenvalues. However, the direction of travel, i.e., eigenmodes, differs. Subfigure 4(b) shows a toy example of how one can chain together (or "stitch") topological conjugates, changing eigenvector direction every 450ms, while keeping certain dynamical properties such as oscillation frequency constant (Eq. 44 ).

Figure 5: Effect of

\beta

on off-diagonal interactions Intuitively, as one takes a larger step away from the identity

I

on the orthogonal manifold (Eq. 48 ), the resulting matrix becomes less and less dominated by its main diagonal, strengthening interactions between the neurons in the network recurrent state

x

.

Figure 6: Patterns of Image Reconstruction Error Each row represents a matrix factorisation method, each producing different patterns of distortion when trained with gradient descent to reconstruct a target image on a low parameter budget. The reconstruction loss reported is Mean Squared Error (MSE). On one hand, Low-Rank matrices are shown to oversmooth the image on very low ranks

\in\{2,4\}

. On the other hand, Kronecker products have much more accurate reconstructions on lower parameter budgets, but the performance varies massively depending on the ratio between

N_{a}

and

N_{b}

. The proposed LowParam heuristic is shown to partially mitigate the extremes of Kronecker products with low-rank corrections.

Figure 7: Pre-Activation Distribution Comparison between LowParam and Kaiming Initialisation The plot shows the output distribution of the proposed LowParam heuristic compared to a Kaiming-Uniform initialised dense matrix. The inputs are 1024 random vectors

\in\mathbb{R}^{1024}

with entries sampled from

\mathcal{N}(0,1)

. The LowParam layer uses

rank=16

for the low-rank component and

N_{a},N_{b}=128,8

for the Kronecker product. While less smooth than the baseline, the initialisation scheme closely reproduces the mean and variance of the original, with a KL-divergence of

\approx 10^{-2}

between the two distributions.

	Conv-DEER	Forward-DEER
Vanilla DEER	$\epsilon_{k}^{sim}\leq\lambda_{max}-\lambda_{min}$ $\epsilon_{k}^{der}\leq 8\beta\lambda_{max}$	N/A
Quasi-DEER	$\epsilon_{k}^{fwd}=\left\\|2\Lambda(l_{1}\odot r_{1}+l_{2}\odot r_{2})+(\hat{l}_{1}l_{1}+\hat{l}_{2}l_{2})\right\\|_{2}\propto\lambda_{max},\beta$ $\epsilon_{k}^{der}\leq 8\beta\lambda_{max}$	$\epsilon_{k}^{der}\leq 8\beta\lambda_{max}$
Forward-DEER	$\epsilon_{k}^{fwd}=\left\\|2\Lambda(l_{1}\odot r_{1}+l_{2}\odot r_{2})+(\hat{l}_{1}l_{1}+\hat{l}_{2}l_{2})\right\\|_{2}\propto\lambda_{max},\beta$	N/A

Table 1: Summary of the different local approximation error terms that make up

\epsilon_{k}^{approx}

depending on the baseline established method (DEER and Quasi-DEER) and the approximation methods proposed here (Conv and Forward-DEER). The error terms listed are also multiplied by the norm of the state

s_{k-1}

, to obtain

\epsilon_{k}^{approx}=(\epsilon+\dots)\left\|s_{k-1}\right\|_{2}

.

Figure 8: Saving memory and compute with Block-Wise DEER execution . Subfigure 8(a) shows how DEER uses compute and, especially, memory to store intermediary copies of already converged states (the red shaded area). Subfigure 8(b) shows how splitting the sequence into 4 equally split sequential blocks, as an example, results in roughly

4\times

reduction in memory required for storing intermediary Newton iterates.

Figure 9: Lyapunov Spectrum of Orthogonal RNN at Initialisation Each subfigure shows the estimated Lyapunov spectrum of a single-layer untrained Orthogonal RNN (Eq. 115 ) at initialisation with hidden size

=128

for a given random input of length

\in\{64,4096,16384\}

. The blue line shows the logarithm of the eigenspectrum of the orthogonal recurrent matrix, which

=\mathbf{1}

, and is only partially aligned with the Lyapunov spectrum of the system.

Rot.-Aware	Neg. Eigvals.	$\gamma$	RoPE	D	Exp.	KV	$(\lambda_{\min},\lambda_{\max})$	$(\Delta t_{\min},\Delta t_{\max})$
✗	✗	$\sqrt{1-\bar{\lambda}^{2}}$	✓	16	Low Rank	Ortho.	$(2,100)$	$(10^{-4},10^{-4})$

Table 2: Baseline Untrained ADPTNet Configuration Rotation-Awareness refers to the correction for the element-wise mismatch from the similarity transform (Eq. 105 ). Negative Eigenvalues can be included as mentioned in Section 4.2 . The RoPE column indicates whether the custom RoPE embeddings with S4D-Inv parametrisation are included. Otherwise, no positional embeddings are used. D refers to the hidden state dimension. Matrix Exponential (Exp.) refers to whether the low-rank structure-aware formula (Section 4.4 ) or the Cayley Map is used (Section 2.5 ).

\gamma

refers to the input normalisation scheme used, EMA or LRU (Section 4.7 ). KV refers to the matrix parametrisation used for the

K

and

V

linear projections: baseline unconstrained (Kaiming initialisation), orthogonal, or LowParam (Section 4.3 ). Using the ZOH discretisation scheme (Section 4.2 ) eigenvalues

\bar{\lambda_{i}}=e^{-\lambda_{i}\Delta t_{i}}

, where

\lambda_{i}

is

\sim\mathcal{U}(\lambda_{\min},\lambda_{\max})

and

\Delta t_{i}

is obtained with

\mathtt{linspace}(\Delta t_{\min},\Delta t_{\max},D)

.

Figure 10: Effect of Time Steps and Hidden Size on Lyapunov Spectrum Each subfigure in the grid represents the recurrent eigenvalues

\Lambda

and Lyapunov spectrum of untrained ADPTNet layers driven by random input of different sequence lengths

T\in\{64,4096,16384\}

. Each subfigure is obtained using a different randomly initialised network. Both D configurations have

\beta=0.125

. It should be observed how all Lyapunov exponents

\alpha_{i}\approx\ln(\lambda_{i})

over long sequences, and are universally contained within the bounds of the recurrent eigenspectrum as shown by the yellow-shaded areas.

Figure 11: Interaction of KV Weight Parametrisation and Choice of

\beta

Each subfigure shows the Lyapunov spectrum and

\ln(\bar{\Lambda})

for random ADPTNet networks at initialisation with different parametrisation schemes. The LowParam setting uses

\text{rank}=2

and

N_{a}=N_{b}=4

. Inputs are driven by

T=4096

random inputs. Notably, only for very large

\beta=1000

do the dynamics of orthogonal constraints start to diverge significantly from the baseline dense linear with Kaiming initialisation and LowParam.

Figure 12: Effect of Matrix Exponential Function Choice The two rows of the grid show the two different matrix exponential functions tested: the Cayley map and the low-rank structure-aware exponential, as described in Section 4.4 . All randomly initialised networks in the grid are driven by

T=4096

-long random inputs. It is apparent that for any scale of

\beta

, both exponentiation methods produce very similar behaviour in terms of Lyapunov spectra.

Figure 13: Effect of Recurrent Weights Condition Number Each subfigure shows the effect of different ratios

\frac{\bar{\lambda}_{\max}}{\bar{\lambda}_{\min}}

on the aligment between the eigenvalue and Lyapunov spectra. Each random network with

D=128

takes in

T=4096

long random inputs. There are three condition number configurations, obtained by setting

\lambda_{\min}=\lambda_{\max}=1

and

(\Delta t_{\min},\Delta t_{\max})\in\{(10^{-2},2.3),(0.7,2.3),(10^{-3},10^{-2})\}

. It should be noted that alignment is driven more by the choice of

\beta

than by the condition number.

Neg. Eigvals.	$\gamma$	RoPE	D	Exp.	KV	$(\lambda_{\min},\lambda_{\max})$	$(\Delta t_{\min},\Delta t_{\max})$	$\beta$
✗	$1-\|\bar{\lambda}\|$	✓	128	Low Rank	Dense	$0.5$	$(10^{-3},10^{-1})$	0.0125

Table 3: Baseline Trained ADPTNet Configuration Column headers follow the same conventions as in Table 2 . Notably,

\gamma

is parametrised as a function of

\bar{\lambda}

throughout training, not just initialised to the value. While Rotational-awareness is omitted from the table, it is assumed to be absent from all models tested.

KV

Dense refers to unconstrained dense linear layers with Kaiming initialisation.

Figure 14: Copy Memory Task All models in this section are trained on the Copy Memory Task, which measures long-term recall ability. Target tokens (integers

\in\{1,2,\dots,8\}

) need to be retrieved in the correct order. The loss is then computed on the final recall window and backpropagated through the distraction steps.

Table 4: LLEs and their Maximum Theoretical Bounds Each cell in the table contains the LLE associated with each KV-parametrisation scheme along with the theoretical bound on it obtained with Lemma 2 . While the bounds

\ln(\bar{\lambda}_{\max}+8\beta\bar{\lambda}_{\max})

are technically specific to orthogonal weights, it can be observed that all parametrisations are well within them.

Figure 15: Effect of

\beta

over ADPTNet Trained Dynamics Each row represents a trained ADPTNet model with a given

\beta

, and the columns are snapshots from different epochs (epoch 50 is the final one). The networks are trained on the Copy Memory task with 44 distraction tokens for a total sequence length of

T=64

. The initial learning rate is

\eta=0.005

. It should be observed how the bounds from Lemma 2 hold up over training and how lower

\beta

induces a higher eigenvalue spread.

Figure 16: Effect of KV-parametrisation over Lyapunov Spectrum During Training Each subfigure in the grid shows the Lyapunov spectrum and the

ln(\bar{\Lambda})

for each KV parametrisation option. The networks are trained on the Copy Memory task with 108 distraction tokens and a total of

T=128

time steps. Baseline denotes dense and unconstrained linear layers, and for the LowParam configuration,

\text{rank}=32

,

N_{a}=32

and

N_{b}=4

. As also reinforced in Table 4 , all configurations have Lyapunov spectra within the bounds from Lemma 2 .

Figure 17: Effect of Including Negative Eigenvalues of Trained ADPTNet Spectra . Subfigure 17(a) shows the different distributions of log-singular values and Lyapunov exponents for a network with a fully positive eigenspectrum and a second network with 32 eigenvalues constrained to be negative throughout training. Both networks are trained on the Copy Memory task with 1004 distraction tokens (total sequence length

T=1024

). The "positive" model reaches

95\%

test accuracy at Epoch 100, while the "negative" model reaches

90\%

. The distribution snapshots are taken at the end of training (Epoch 100). Subfigure 17(b) shows the evolution of the eigenvalue spread for the two model configurations throughout training. It should be noted that the Lyapunov spectra are close to the log-singular values for both settings. However, the distribution has a lower minimum for the "positive" case. It can also be observed that both spreads tend to their respective maximums over training.

Figure 18:

\mathbf{\Delta_{i}^{(t)}}

Distribution Throughout Training

\Delta_{i}^{(t)}

perturbations are computed at each time step using as the difference

\sigma_{i}(J_{t})-\bar{\lambda}_{i}=\tilde{\sigma}_{i}^{(t)}-\bar{\lambda}_{i}

, and averaged over sequence length for a Copy Memory task sample.

\mu

denotes the mean over time steps, while

\sigma

represents the standard deviation. The perturbations are collected from the "positive" model in Figure 17 . The key trends to observe are that for both Epoch 5 and Epoch 100 (final epoch) the mean of the perturbation distribution is 0. In addition,

\Delta_{i}^{(t)}

variance increases between the two epochs visualised.

DEER Approx.	MACs
Conv	0
Forward	$20*D$

Table 5: Computational Overhead for Jacobian-Free DEER Variations Conv-DEER does not require any additional MACs to obtain its

A_{t}

terms, while Forward DEER scales linearly. The MAC count estimate for Forward-DEER is obtained using the torchprofile library and the implementation of Eq. 95 in Appendix B . Here,

D

refers to the model hidden state size.

Neg. Eigvals.	$\gamma$	RoPE	D	Exp.	KV	$(\lambda_{\min},\lambda_{\max})$	T
✗	$1-\|\bar{\lambda}\|$	✓	64	Low Rank	Dense	$0.5$	1024

Table 6: Baseline Untrained ADPTNet Configuration for DEER Parallelisation Ablations Overall, the baseline configuration here is close to the trained model configuration in Section 5.1.3 .

T

refers to the total number of time steps in the random sequences fed into the networks as inputs. To collect convergence metrics, the networks are then simulated with these random drivers following Gonzalez et al. [69] .

Figure 19: Eingspectrum Effects on DEER Convergence . All Subfigures are produced using 15 random samples for each configuration. For Subfigure 19(a) ,

\Delta t_{\min}

controls the longest time scale of the layer through

\bar{\lambda}_{\max}

and takes values

\in\{10^{-2},10^{-3}\}

.

\Delta t_{\max}

controls the shortest timescale through

\bar{\lambda}_{\min}

and is constant at

10^{-1}

. The subfigure highlights how increasing the eigenvalue spread and

\beta

leads to more DEER iterations across all algorithms tested. Subfigure 19(b) holds

\Delta t_{\min}

fixed at

10^{-3}

, while varying

\Delta t_{\max}\in\{0.01,0.005,0.001\}

. This shows that DEER iterations for long-range, slowly decaying dynamics can be reduced by reducing the eigenvalue spread along with

\beta

. Subfigure 19(c) uses the same samples and models as Subfigure 19(a) and shows that Forward, and to a great extent Conv DEER, suffer from more high-iteration-count outliers.

Figure 20: Scaling with Sequence Length This figure shows the averaged number of iterations required for convergence over 15 random samples of varying length

T\in\{512,1204,4096,8192,16384\}

.

\beta=0.125

,

\Delta t_{\min}=10^{-4}

, and

\Delta t_{\max}=10^{-1}

for all sequence length configurations. The main takeaway is that, unsurprisingly, the number of Newton iterations needed for convergence grows with sequence length. However, importantly, the average number of convergence iterations is the same between Conv, Forward, and Quasi DEER.

Figure 21: Block-Wise DEER Trade-Offs Subfigures 21(a) and 21(b) show the wall clock time and peak GPU memory utilisation depending on the different number of sequential blocks (

B\in\{1,2,4,8,16,32\}

) used for Conv/Forward DEER simulation. A random input is sampled for each sequence length

T\in\{4096,8192\}

, along with a randomly initialised ADPTNet layer. For all configurations,

\beta=0.125

,

\Delta t_{\min}=10^{-4}

, and

\Delta t_{\max}=10^{-1}

. Subfigure 21(c) shows the number of DEER iterations used for each block/ sequence length configuration

(B,T)

. For

B>1

, the number of DEER iterations per sequential block may vary. Therefore, the iteration counts in Subfigure 21(c) are the median number of iterations to convergence across blocks in a

(B,T)

simulation. This median is then used in all blocks in each

(B,T)

setting to obtain the time and memory metrics. Subfigure 21(d) shows the efficiency gains in simulation time and peak memory usage of Conv DEER compared to Forward DEER. Note that the Conv DEER implementation here is suboptimal because it recomputes

\bar{\Lambda}

for

A_{t}

instead of caching it from the forward pass. In contrast, Forward DEER reuses the same intermediary results from the forward pass where possible, minimising overhead in accordance with Table 5 . Peak memory utilisation is obtained with torch.cuda_max_memory_allocated() , while simulation time is obtained as an average of 7 trials after 2 warm-up runs. The recordings are taken without saving gradients for the backwards pass.

Figure 22: Effect of Damping on DEER Convergence . Subfigure 22(a) shows the effect of Scale-ELK damping on the absolute number of iterations required for convergence. The 30 dots in each column represent the convergence iterations for different random input samples, while the column shows their median. Each sample is also obtained with its own randomly initialised ADPTNet layers, with 16 eigenvalues

\bar{\Lambda}

set to negative, leading to a higher median count for

T=1024

than reported in Figure 20 . All models tested have

\beta=0.125

,

\Delta t_{\min}=10^{-4}

, and

\Delta t_{\max}=10^{-1}

. Subfigure 22(b) shows the relative difference in iterations to convergence between Conv/Forward DEER and Quasi DEER. The smooth curve shows the probability density. It can be noted that increasing the damping effect quickly reduces the number of outliers.

Figure 23: DEER Convergence over Training Epochs . The convergence data in this figure are obtained from the same fully positive

\bar{\Lambda}

model snapshots used to produce the spectral analysis in Figure 17 (Table 3 ). That also means that the training iterations correspond to learning the

T=1024

Copy Memory Task. Consequently, Subfigure 23(c) is a reproduction of the data in Subfigure 17(b) included here to aid convergence data analysis. For the block-wise simulations in Subfigure 23(b) , the number of sequential blocks is

B=8

. The shown convergence iteration counts are obtained from the same randomly selected Copy Memory test sample. The main observations are that convergence iterations for all DEER variants increase over epochs in line with increasing

\bar{\Lambda}

spread. Additionally, in this particular case, Conv-DEER appears unstable for full DEER parallelism in Subfigure 23(a) , while block-wise execution in Subfigure 23(b) appears to mitigate the convergence pathology.

Model	Seq.Len.	N. Layers Required	Extrapolation
Mamba	8	4	×
Mamba	32	16	×
RNN	8	1	✓
RNN	32	1	✓

Table 7: Mamba / RNN State Tracking Performance Summarised from Schöne et al. [154] This table summarises the number of layers required for Mamba and non-linear RNNs to converge on the

S_{5}

task, as reported in Schöne et al. [154] . Of note, Mamba requires more layers as sequence length grows, while not being able to extrapolate to longer sequences at test time (i.e., 2x or 4x the training length). In contrast, a single layer RNN can not only solve the task, but also extrapolate to longer input sequences.

Eig. Init.	Model	Width	Sparsity	Neg. Eig.	Accuracy (%)
Eig. Init.	Model	Width	Sparsity	Neg. Eig.	$L{=}8$	$L{=}16$	$L{=}32$
					$1\times$	$2\times$	$4\times$
–	ReLU RNN	512	–	–	100.0	100.0	99.6
–	ReLU RNN	864	–	–	100.0	100.0	100.0
Linspace	ADPTNet ( $\beta=0$ )	512	0%	✓	31.2	1.1	0.9
	ADPTNet ( $\beta=0$ )	624	0%	✓	31.2	1.0	0.9

Table 8: Baseline

\mathbf{S_{5}}

Accuracy Results are obtained by averaging over three seeds for each configuration and following the training setup from Appendix C . One can observe the strong alignment with the results in Table 7 with respect to the stark gap in accuracy between linear and non-linear recurrence.

Eig. Init.	Sparsity	$\beta$	Neg. Eig.	Accuracy (%)
Eig. Init.	Sparsity	$\beta$	Neg. Eig.	$L{=}8$	$L{=}16$	$L{=}32$
				$1\times$	$2\times$	$4\times$
Linspace	0%	4	✓	43.4	1.7	0.8
	0%	4	×	51.5	1.3	0.9
	75%	0.00125	✓	28.8	0.8	0.8
		0.0125	✓	36.2	1.2	1.1

Table 9: Effect of

\mathbf{\beta}

and Eigenspectrum on ADPTNet State Tracking Performance Each result in this table is obtained by averaging over three seeds, as in Table 8 . In addition, the training configuration is in Appendix C . It can be seen that the most reliable indicator of

S_{5}

word problem accuracy is the choice of

\beta

. To maximise performance, however, eigenvalue initialisation and parametrisation must also be considered.

LN	ReLU	$\langle k,v\rangle$	Mean Accuracy (%)			Best (%)
			$L{=}32$	$L{=}64$	$L{=}128$	$L{=}128$
			$1\times$	$2\times$	$4\times$	$4\times$
×	×	×	49.2	26.3	9.9	10.3
✓	×	×	5.3	3.6	2.5	2.9
×	✓	×	100.0	97.0	5.1	36.8
×	×	✓	43.4	21.8	8.7	8.9

Table 10: Ablation on L=32

\mathbf{S_{5}}

As with Tables 8 and 9 , results are obtained by averaging over 3 seeds and following the training configuration from Appendix C . LN refers to whether LayerNorm is applied to

Kx

and

Vx

before unit normalisation. ReLU refers to including a ReLU activation in

K\mathtt{ReLU}(x)

and

V\mathtt{ReLU}(x)

.

<k,v>

denotes whether the dot product between

k

and

v

is constrained to 0.5 using the methods from Section 4.12 . Overall, ReLU can be observed to significantly benefit state tracking performance, mitigating even the effects of adding LN and

<k,v>

constraints, which otherwise can severely impact accuracy.

Figure 24: Selective Copy Task The Selective Copying task inherits the same general formulation as the Copy Memory task (Fig. 14 ). The difference is that here the target tokens are spread out randomly among the distractors.

Table 11: Effect of

\mathbf{\beta}

on Selective Copy Accuracy Details of the experimental setup are available in Appendix D . It should be mentioned that each baseline in Tables 11(a) and 11(b) is obtained by taking the maximum accuracy over three seeds across six learning rate

\times

weight decay configurations. The main observation from both tables is that using a larger

\beta

improves Selective Copying accuracy. Table 11(a) is obtained using sequence length

L=1024

, while Table 11(b) uses

L=128

. In addition, Table 11(a) uses a model hidden size of

h=128

, and an state expansion of

n_{\text{state}=2}

(see Section 4.6 ). For Table 11(b) , hidden size is

h=64

and state expansion is

n_{\text{state}}=4

. It is important to highlight that the specific hyperparameter selection here is not optimised for producing the absolute highest Selective Copy accuracy. Instead, it purposefully underparametrises models to the point where differences in expressivity can be isolated. Evidently, drastically scaling the number of parameters could allow non-selective architectures to simply memorise all possible target token positions.

Figure 25: Lyapunov Spectrum learning for Matching Copying Tasks . Each subfigure here shows the Lyapunov spectrum compared to the eigenspectrum for each layer in both networks trained on Selective Copying and Copy Memory tasks with matching training configurations and model hyperparameters (Appendix D ). Note that the empirical Lyapunov exponents are bounded and closely match

\ln(|\bar{\Lambda}|)

across all layers examined. To obtain the spectra, a random sample is chosen from both datasets for each corresponding model.

Task	Layer	Learned $\beta$	Acc. Diff.
Copy Memory	0	0.026	-1.5%
Copy Memory	1	0.029	-1.5%
Selective Copy	0	0.027	+2.4%
Selective Copy	1	0.027	+2.4%

Table 12: Trained

\mathbf{\beta}

across Tasks and Layers Both networks are initialised with

\beta=0.0125

, and it can be observed that training pushes it to saturation of the upper clipping bound

0.025

. The Acc. Diff. column shows the change in final accuracy compared to the fixed

\beta

models used for Figure 25 . In absolute terms, fixed

\beta

resulted in

89.69\%

accuracy for Copy Memory and

63.70\%

for Selective Copy. For trainable

\beta

, the accuracies are

88.20\%

for Copy Memory and

66.07\%

for Selective Copying.

Figure 26: Recurrent Matrix

\mathbf{M}

Materialisation during Training - First Layers . Figures 26 and 27 show how the data-dependent recurrent matrices

M

adapt depending on the task, Copy Memory or Selective Copy. Here, the same models and training experiments are used as in Fig. 25 . On the left, the matrices are extracted at initialisation (before training) from their respective networks, while on the right the matrices are materialised at the end of training. Each entry in the matrices is the coupling strength between two recurrent neurons, so each matrix also implicitly represents a network topology. The main differences pertain to how evenly-distributed the off-diagonal entries are, as quantified by the normalised Shannon entropy (Eq. 118 ). It can be observed that Selective Copying leads to higher overall entropy, and thus more evenly distributed all-to-all recurrent connectivity.

Figure 27: Recurrent Matrix

\mathbf{M}

Materialisation during Training - Second Layers . This figure accompanies Figure 26 , showing the effect of training on different tasks on the materialised recurrent matrix

M

. While the effect on the second layers is less dramatic than on the first, there is still a visiable difference in entropy, with more evenly spread out connectivity for Selective Copying than Copy Memory.

Figure 28: Training Effect on DEER Convergence . This figure uses the models from Table 12 and Figure 25 , i.e., the trained and fixed

\beta

models trained on either Selective Copying or Copy Memory. Using Conv-DEER, it can be seen that deeper layers are slower to converge across all settings. Notably, Selective Copy appear to converge more quickly than Copy Memory, despite the higher spread eigenspectrum (see Fig. 25 ).

Model	Memorisation	Selective Copy	Compression
Mamba2 [ 35 ]	42.0	95.4	41.3
GLA [ 193 ]	82.5	96.1	42.3
xLSTM [ 12 ]	79.8	95.4	43.4
DeltaNet [ 194 ]	40.8	98.8	43.3
Gated DeltaNet [ 192 ]	81.7	95.7	45.0
MesaNet [ 181 ]	77.2	99.2	45.4

Table 13: Accuracies on MAD Tasks The ADPTNet network tested here is situated in a perturbation regime, where off-diagonal couplings are scaled by

\beta=0.00125

. The baseline results included in this table are reported from von Oswald et al. [181] . GLA stands for Gated Linear Attention [ 193 ] . All networks besides Hawk, the Transformer, and ADPTNet, compress information in a matrix-valued state. The Transformer evidently keeps the entire sequence as its state, while ADPTNet and Hawk both have vector-valued states. The most notable results are that ADPTNet performs competitively against models with much larger states on Compression, while outperforming its closest equivalent alternative, Hawk, on Selective Copying.

Table 14: sCifar Accuracy Details of the experimental setup are available in Appendix F . The baseline accuracies in Table 14(a) are reported from Beck et al. [12] , while Table 14(b) collects baseline results from Gu et al. [74] . It should be observed that both the 2-layer non-linear ADPTNet and the 4-layer LinearADPTNet outperform all of the larger selective architectures in Table 14(a) . Most importantly, however, the non-linear ADPTNet configuration matches and even slightly outperforms the baseline S4D-Inv model (

\beta=0

) of the same size.

Model	Acc(%)	State Size	N. Layers	N. Param.
RSNN [ 31 ]	$51.1\pm 1.1$	–	–	–
Adaptive RSNN [ 196 ]	$74.2$	–	–	–
EventProp [ 130 ]	$76.1\pm 1.0$	–	–	–
RSNN with Adaptation [ 18 ]	$77.4$	–	–	–
d-cAdLIF [ 41 ]	$80.23\pm 0.07$	–	–	0.35M
SE-adLIF [ 11 ]	$80.4\pm 0.3$	–	–	1.6M

Table 15: Accuracies on SSC The ranking of prior state-of-the-art work reported in this table is reported from the official SSC leaderboard (Available at https://zenkelab.org/resources/spiking-heidelberg-datasets-shd/). As shown, the smaller SpikingADPTNet configuration performs on par with the current state-of-the-art, while the larger one sets a new state-of-the-art by over

1

point. (*) denotes that the state size in practice is larger due to axonal/synaptic delays requiring explicit buffers of past outputs.

	$L=8$ sweep	$L=32$ sweep
	s5_L8_*_beta_sparsity	s5_L32_ln_relu_cdot
Task and data
Group	$S_{5}$ , 120 elements, non-solvable; identity monoid, $p=0$
Training sequences	$8{,}388{,}608$ ( $512\times 16384$ ), uint8
Test sequences	$4{,}096$ per evaluation length
Training length	8	32

	Baseline	ADPTNet
Vocabulary	$16$
Tokens to copy	$16$
Training samples	$10{,}000$ per epoch, regenerated
Test samples	$1{,}000$
Layers	$2$
Eig. Init.	S4D-Real

ADPTNet: Adaptive with Prescriptive Timescales Non-Linear SSM for Sequence Modelling

Abstract

Figures & tables

Appendix

Explore similar work

Pretraining Recurrent Networks without Recurrence

ADSEQ: A delay-aware autograd-compatible framework for spike-event delivery in SNNs

Towards transferable lightweight neuromorphic computing through a model-free temporal-switch framework

$d_{\text{model}}$	$64$
$n_{\text{state}}$	$4$
Layers	$2$
Eig. Init.	S4D-Inv
$\beta$	0.0125
Epochs	$100$
LR	$5\times 10^{-4}$