cond-mat.dis-nnFeb 16, 2026
SaveDrift-Diffusion Matching: Embedding dynamics in latent manifolds of asymmetric neural networks
Organizations: Mathematical Institute, University of Oxford, Woodstock Road, Oxford, OX2 6GG, United Kingdom · Centre for Eudaimonia and Human Flourishing, University of Oxford, 7 Stoke Pl, Oxford, OX3 9BX, United Kingdom · Complexity Science Hub, Metternichgasse 8, Vienna, 1030, Austria
Abstract
Recurrent neural networks (RNNs) provide a theoretical framework for understanding computation in biological neural circuits, yet classical results, such as Hopfield's model of associative memory, rely on symmetric connectivity that restricts network dynamics to gradient-like flows. In contrast, biological networks support rich time-dependent behaviour facilitated by their asymmetry. Here we introduce a general framework, which we term drift-diffusion matching, for training continuous-time RNNs to represent arbitrary, nonlinear stochastic differential equations (SDEs), with given drift and diffusion coefficients, within a low-dimensional latent subspace. Allowing asymmetric connectivity, we show that RNNs can faithfully embed the drift and diffusion of a given SDE, including nonlinear and nonequilibrium dynamics such as chaotic attractors. As an application, we construct RNN realisations of stochastic systems that transiently explore various attractors through both input-driven switching and autonomous transitions driven by nonequilibrium currents, which we interpret as models of associative and sequential (episodic) memory. To elucidate how these dynamics are encoded in the network, we introduce decompositions of the RNN based on its asymmetric connectivity and its time-irreversibility. Our results extend attractor neural network theory beyond equilibrium, showing that asymmetric neural populations can implement a broad class of dynamical computations within low-dimensional manifolds, unifying ideas from associative memory, nonequilibrium statistical mechanics, and neural computation.
Figures & tables
Figure 1: Embedding dynamics in latent manifolds of RNNs. Our framework focuses on RNNs with low-rank connectivity of the form , where with . We train these autonomous systems to encode arbitrary target SDEs in a learnt affine subspace. The training of the RNN can be reformulated as the training of a two-layer perceptron to approximate a target vector field. Once the parameters of the RNN have been fitted, it can be integrated over time as an autonomous circuit which produces stochastic dynamics. Here we plot traces from 5 of 64 neurons in a network trained to approximate the van der Pol oscillator. When we project the dynamics into the learnt subspace, we recover a trajectory from the stochastic van der Pol system, which has been embedded into the RNN.
Figure 2: Embedding attractors in neural networks. Traces from 5 randomly selected neurons in a 64-neuron RNN trained to embed a stochastic VDP oscillator in a latent subspace. In the subspace, we see the characteristic VDP limit-cycle. Traces from a 512-neuron RNN trained to embed the LS, and a 1024-neuron RNN trained to embed the DS.
Figure 3: Symmetric-asymmetric decomposition of RNNs. Decomposition of RNN trained to encode the VDP. The symmetric dynamics descend the contoured energy function. This appears to keep the process near the limit cycle, whilst the asymmetric component provides rotational dynamics. Decompositions of RNNs trained to encode the LS and DS. For the DS, the symmetric dynamics keep the process near the attractor, whilst the asymmetric component provides rotational dynamics. For the LS, it appears the symmetric component pushes the dynamics away from the origin, and the asymmetric component provides a restoring force. This is confirmed by plotting isopotentials of the energy functions for the LS and DS respectively.
Figure 4: Reconstructing latent subspaces from trajectories . Trajectories from a low-rank RNN trained to embed a dimensional system only have non-zero principal components. With PCA, we can identify the affine subspace in which the dynamics are constrained. The PCA representation is equivalent to original coordinates up to an affine linear transformation. Given can solve for this transformation to recover the original coordinates.
Figure 5: ‘Tilting’ the energy landscape. Our approach uses network inputs to ‘tilt’ the energy landscape, encouraging the process to push a trajectory into a particular energy minimum, by making it a global minimum. This is a soft version of the child’s ‘labyrinth’ game, where the board can be tilted such that the ball rolls into a specific hole – though in the actual game, one tries rather hard to avoid them.
Figure 6: Input-driven attractor-switching . We consider a four-well energy potential in the plane. In the absence of inputs, the trajectories converge to a minima at random, with a preference for the closest one. We can then tilt the potential with an input, which prioritises the relevant minima, and causes the trajectories to tend towards it. We train a 256-neuron RNN to embed the potential-gradient dynamics. Using a one-hot coding for the input, we learn the coupling matrix input connectivity and bias, then add the input to the RNN during integration. We start with the input that drives the process to the top-right well, then switch it halfway through to the bottom-right well. We plot traces from 10 random neurons and see the network dynamics react to the input and converge to the other minima. Projecting the RNN trajectory onto the latent space, we see the attractor-switching that occurs due to the change in input.
Figure 7: Constructing rotational diffusions. To construct an SDE with rotation between energy minima, we begin with the gradient of the energy potential. We then draw a closed curve, , through the minima in the direction of rotation. From this, we can define a rotational drift which decays with distance from the curve due to the weighting kernel .
Figure 8: Attractor-cycling diffusions. We construct example diffusion processes that cycle 3, 4, and 5 attractors with irreversible currents (Panels ). These are composed of a gradient flow (first column) plus a rotational component. The rotational component is constructed by taking a smooth curve through the minima, defining a potential as the distance from the curl, multiplying by a decaying weight kernel (contour in second column), and dividing by the stationary density (second column). Combined, this defines a vector field that both converges to, and cycles between attractors (third and fourth column).
Figure 9: Attractor-cycling in RNNs. We train 128-neuron RNNs to approximate the planar diffusion processes from Fig. 8 for three, four, and five minima respectively. Trajectories from the RNN can then be sampled autonomously, and the learnt drift can be projected onto the learnt affine subspace. We find that the RNNs are able to learn the target diffusion processes. For each RNN, we plot traces from ten random neurons. We see that the attractor cycling dynamics in the latent subspace are visible in the raw trajectories.
Figure 10: The HHD of a latent, embedded process. When we embed a process with a known NESS, we can perform the HHD. In this example with three minima, we see that the reversible dynamics balance the diffusion and maintain the process on the stationary distribution, whilst the rotational dynamics create rotation around the curve . We train an RNN with 64 neurons to approximate the process in a latent affine subspace. Using the HHD and Eq. ( 45 ), we can define the irreversible, and reversible, components of the high-dimensional dynamics. We see that the reversible dynamics in the ambient space keep the process near the minima, balancing the noise, whilst the irreversible dynamics show limit-cycle behaviour in the ambient space.
Figure 11: The HHD of an RNN. Following Eq. ( 46 ), we train two coupled RNNs to approximate the reversible and irreversible components of the HHD respectively. Their projected drift shows that they were able to learn the components effectively. We can decompose the learnt connectivity matrix, , using both the HHD, , or using the symmetric-asymmetric decomposition from Sec. II.3 , . We find that for this three minima example, the magnitude of the irreversible and asymmetric components are far greater than the reversible component suggesting that encoding rotational or cycling dynamics requires more network complexity. We can also examine the decomposition of the input current vector, . As with the connectivity, more of the input current is used to encode the irreversible component than the reversible component.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Input-driven attractor switching with random Gaussian codes. Instead of a ‘one-hot’ encoding, we can also take random Gaussian vectors to represent each memory. We add Gaussian noise to this input signal and repeat the experiment from before, switching the input signal from minimum 1 to minimum 2 halfway along the trajectory. As the input noise intensity is smaller than the difference between the codes, the network stills switches between the ‘memories’ correctly, which can be seen at the level of the neural manifold.
Figure 13: Cycling between pixelated images in high-dimensions . We can modify our approach from Sec. III to design processes which cycle between energy minima in high-dimensions. We illustrate this using pixelated images of a dog and a chick, which are points in . As before, we design a drift field that encodes these points as the minima of a Gaussian potential, but drives rotation between them without disrupting the stationary density. Integrating this SDE, we can sample paths which cycle between (noisy) instances of the two images in the high-dimensional space.
Figure 14: Cycling between non-coplanar minima. We can also design SDEs which cycle between non-coplanar minima in 3D. In this example, we consider four minima on a saddle, which we interpolate with a cubic spline. We design a rotational field which pushes trajectories around this curve, but is DFSD. This produces a combined field which has both attracting and rotational dynamics. Sampling from this process, we see that it rotates transiently between attracting states.
Figure 15: Network asymmetry for increasing EPR. For a range of values we train an ensemble of 100 RNNs. We then compute the relative size of each component in the decomposition with Eq. ( 86 ) for , , and . In we also plot from the network HHD in Eq. ( 46 ).
Figure 16: A universal representation for nonlinear dynamics. The DDM framework allows us to obtain a universal representation for nonlinear dynamics. By studying the inferred parameters, we can define measures of ‘complexity’ for the dynamics and their representations. We illustrate this with chaotic attractors, where we compute the spectral radius, mean participation ratio, and participation ratio of the dominant mode, for the connectivity, , of 100 RNNs with 1024 neurons each, trained to encode the Rössler, Dadras, Halvorsen, and Lorenz attractors. We find that the Lorenz attractor has, by far, the highest spectral radius, whilst the Dadras attractor has the highest participation ratio.