Organizations: Mathematical Institute, University of Oxford, Woodstock Road, Oxford, OX2 6GG, United Kingdom · Centre for Eudaimonia and Human Flourishing, University of Oxford, 7 Stoke Pl, Oxford, OX3 9BX, United Kingdom · Complexity Science Hub, Metternichgasse 8, Vienna, 1030, Austria
Recurrent neural networks (RNNs) provide a theoretical framework for understanding computation in biological neural circuits, yet classical results, such as Hopfield's model of associative memory, rely on symmetric connectivity that restricts network dynamics to gradient-like flows. In contrast, biological networks support rich time-dependent behaviour facilitated by their asymmetry. Here we introduce a general framework, which we term drift-diffusion matching, for training continuous-time RNNs to represent arbitrary, nonlinear stochastic differential equations (SDEs), with given drift and diffusion coefficients, within a low-dimensional latent subspace. Allowing asymmetric connectivity, we show that RNNs can faithfully embed the drift and diffusion of a given SDE, including nonlinear and nonequilibrium dynamics such as chaotic attractors. As an application, we construct RNN realisations of stochastic systems that transiently explore various attractors through both input-driven switching and autonomous transitions driven by nonequilibrium currents, which we interpret as models of associative and sequential (episodic) memory. To elucidate how these dynamics are encoded in the network, we introduce decompositions of the RNN based on its asymmetric connectivity and its time-irreversibility. Our results extend attractor neural network theory beyond equilibrium, showing that asymmetric neural populations can implement a broad class of dynamical computations within low-dimensional manifolds, unifying ideas from associative memory, nonequilibrium statistical mechanics, and neural computation.
Figures & tables
Figure 1: Embedding dynamics in latent manifolds of RNNs. a) Our framework focuses on RNNs with low-rank connectivity of the form W=ΓWs , where Γ∈Rn×k with k≪n . We train these autonomous systems to encode arbitrary target SDEs in a learnt affine subspace. b) The training of the RNN can be reformulated as the training of a two-layer perceptron to approximate a target vector field. c) Once the parameters of the RNN have been fitted, it can be integrated over time as an autonomous circuit which produces stochastic dynamics. Here we plot traces from 5 of 64 neurons in a network trained to approximate the van der Pol oscillator. When we project the dynamics into the learnt subspace, we recover a trajectory from the stochastic van der Pol system, which has been embedded into the RNN.
Figure 2: Embedding attractors in neural networks. a) Traces from 5 randomly selected neurons in a 64-neuron RNN trained to embed a stochastic VDP oscillator in a latent subspace. In the subspace, we see the characteristic VDP limit-cycle. b−c) Traces from a 512-neuron RNN trained to embed the LS, and a 1024-neuron RNN trained to embed the DS.
Figure 3: Symmetric-asymmetric decomposition of RNNs. a) Decomposition of RNN trained to encode the VDP. The symmetric dynamics descend the contoured energy function. This appears to keep the process near the limit cycle, whilst the asymmetric component provides rotational dynamics. b−c) Decompositions of RNNs trained to encode the LS and DS. For the DS, the symmetric dynamics keep the process near the attractor, whilst the asymmetric component provides rotational dynamics. For the LS, it appears the symmetric component pushes the dynamics away from the origin, and the asymmetric component provides a restoring force. d−e) This is confirmed by plotting isopotentials of the energy functions for the LS and DS respectively.
Figure 4: Reconstructing latent subspaces from trajectories . a) Trajectories from a low-rank RNN trained to embed a k− dimensional system only have k non-zero principal components. b) With PCA, we can identify the affine subspace in which the dynamics are constrained. c) The PCA representation is equivalent to original coordinates up to an affine linear transformation. Given (Γ,b) can solve for this transformation to recover the original coordinates.
Figure 5: ‘Tilting’ the energy landscape. Our approach uses network inputs to ‘tilt’ the energy landscape, encouraging the process to push a trajectory into a particular energy minimum, by making it a global minimum. This is a soft version of the child’s ‘labyrinth’ game, where the board can be tilted such that the ball rolls into a specific hole – though in the actual game, one tries rather hard to avoid them.
Figure 6: Input-driven attractor-switching . a) We consider a four-well energy potential in the plane. In the absence of inputs, the trajectories converge to a minima at random, with a preference for the closest one. We can then tilt the potential with an input, which prioritises the relevant minima, and causes the trajectories to tend towards it. b) We train a 256-neuron RNN to embed the potential-gradient dynamics. Using a one-hot coding for the input, we learn the coupling matrix input connectivity and bias, then add the input to the RNN during integration. We start with the input that drives the process to the top-right well, then switch it halfway through to the bottom-right well. We plot traces from 10 random neurons and see the network dynamics react to the input and converge to the other minima. c) Projecting the RNN trajectory onto the latent space, we see the attractor-switching that occurs due to the change in input.
Figure 7: Constructing rotational diffusions. To construct an SDE with rotation between energy minima, we begin with the gradient of the energy potential. We then draw a closed curve, γ , through the minima in the direction of rotation. From this, we can define a rotational drift which decays with distance from the curve due to the weighting kernel g(y) .
Figure 8: Attractor-cycling diffusions. We construct example diffusion processes that cycle 3, 4, and 5 attractors with irreversible currents (Panels a−c ). These are composed of a gradient flow (first column) plus a rotational component. The rotational component is constructed by taking a smooth curve through the minima, defining a potential as the distance from the curl, multiplying by a decaying weight kernel (contour in second column), and dividing by the stationary density (second column). Combined, this defines a vector field that both converges to, and cycles between attractors (third and fourth column).
Figure 9: Attractor-cycling in RNNs. We train 128-neuron RNNs to approximate the planar diffusion processes from Fig. 8 for a) three, b) four, and c) five minima respectively. Trajectories from the RNN can then be sampled autonomously, and the learnt drift can be projected onto the learnt affine subspace. We find that the RNNs are able to learn the target diffusion processes. For each RNN, we plot traces from ten random neurons. We see that the attractor cycling dynamics in the latent subspace are visible in the raw trajectories.
Figure 10: The HHD of a latent, embedded process. a) When we embed a process with a known NESS, we can perform the HHD. In this example with three minima, we see that the reversible dynamics balance the diffusion and maintain the process on the stationary distribution, whilst the rotational dynamics create rotation around the curve γ . b) We train an RNN with 64 neurons to approximate the process in a latent affine subspace. Using the HHD and Eq. ( 45 ), we can define the irreversible, Firr and reversible, Frev components of the high-dimensional dynamics. We see that the reversible dynamics in the ambient space keep the process near the minima, balancing the noise, whilst the irreversible dynamics show limit-cycle behaviour in the ambient space.
Figure 11: The HHD of an RNN. a) Following Eq. ( 46 ), we train two coupled RNNs to approximate the reversible and irreversible components of the HHD respectively. Their projected drift shows that they were able to learn the components effectively. b) We can decompose the learnt connectivity matrix, W , using both the HHD, W=Wrev+Wirr , or using the symmetric-asymmetric decomposition from Sec. II.3 , W=Wsym+Wasym . We find that for this three minima example, the magnitude of the irreversible and asymmetric components are far greater than the reversible component suggesting that encoding rotational or cycling dynamics requires more network complexity. c) We can also examine the decomposition of the input current vector, I=Iirr+Irev . As with the connectivity, more of the input current is used to encode the irreversible component than the reversible component.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Input-driven attractor switching with random Gaussian codes. a) Instead of a ‘one-hot’ encoding, we can also take random Gaussian vectors to represent each memory. We add Gaussian noise to this input signal and repeat the experiment from before, switching the input signal from minimum 1 to minimum 2 halfway along the trajectory. b) As the input noise intensity is smaller than the difference between the codes, the network stills switches between the ‘memories’ correctly, which can be seen at the level of the neural manifold.
Figure 13: Cycling between pixelated images in high-dimensions . We can modify our approach from Sec. III to design processes which cycle between energy minima in high-dimensions. a) We illustrate this using 32×32 pixelated images of a dog and a chick, which are points in R1024 . b) As before, we design a drift field that encodes these points as the minima of a Gaussian potential, but drives rotation between them without disrupting the stationary density. c) Integrating this SDE, we can sample paths which cycle between (noisy) instances of the two images in the high-dimensional space.
Figure 14: Cycling between non-coplanar minima. We can also design SDEs which cycle between non-coplanar minima in 3D. a) In this example, we consider four minima on a saddle, which we interpolate with a cubic spline. b) We design a rotational field which pushes trajectories around this curve, but is DFSD. c) This produces a combined field which has both attracting and rotational dynamics. d) Sampling from this process, we see that it rotates transiently between attracting states.
Figure 15: Network asymmetry for increasing EPR. For a range of values ω∈[0,5] we train an ensemble of 100 RNNs. We then compute the relative size of each component in the decomposition with Eq. ( 86 ) for a)(W−C,C) , b)(ΓΠ,ΓΩ) , and c)(Wirr,Wrev) . In d) we also plot ϱ from the network HHD in Eq. ( 46 ).
Figure 16: A universal representation for nonlinear dynamics. The DDM framework allows us to obtain a universal representation for nonlinear dynamics. By studying the inferred parameters, we can define measures of ‘complexity’ for the dynamics and their representations. We illustrate this with chaotic attractors, where we compute the a) spectral radius, b) mean participation ratio, and c) participation ratio of the dominant mode, for the connectivity, W , of 100 RNNs with 1024 neurons each, trained to encode the Rössler, Dadras, Halvorsen, and Lorenz attractors. We find that the Lorenz attractor has, by far, the highest spectral radius, whilst the Dadras attractor has the highest participation ratio.
Recurrent neural networks are canonical models of biological memory. In these models, memories are represented by distributed patterns of neural activity that are stored in the recurrent connections between neurons, such that they become attractors of the network's dynamics. During memory recall, network dynamics thus converge toward one of these memory patterns when started from a noisy or partial cue. Therefore, memory performance critically hinges on the dynamical stability of the stored patterns. However, previous theoretical approaches only studied dynamical stability under highly restrictive conditions that do not readily apply to biological neural circuits. Here, we develop a theory of the local stability of discrete fixed points in a broad class of networks with graded neural activities and in the presence of noise. Using methods from random matrix theory, we analyze the bulk and outliers of the eigenvalue spectra of the Jacobians that characterize network dynamics around fixed points. We show that either all fixed points are stable or all of them are unstable, depending on whether their number is below a ``critical load for stability'', which is distinct from the classical critical capacity that measures the maximal number of achievable fixed points regardless of their stability. We further analyze the dependence of this critical load for stability on experimentally measurable quantities characterizing the statistics of memory patterns and the activation functions of neurons. Our analysis highlights the computational benefits of sparse-like patterns and threshold-linear activation functions and offers testable predictions for neural circuits supporting memory.
Uri Cohen, Máté Lengyel
Computational and Biological Learning Lab, Department of Engineering, University of Cambridge, Cambridge, UK · Center for Cognitive Computation, Department of Cognitive Science, Central European University, Budapest, Hungary
Classical Hopfield networks are limited to static patterns due to symmetric weights, whereas asymmetric networks can encode temporal sequences via limit-cycle attractors. Achieving high-capacity storage of long sequences in classical synchronous asymmetric networks, however, has remained a challenge. We present a simple and robust construction within the classical asymmetric Hopfield model with binary neurons and synchronous updates, that allows n neurons to support exp(Ω(n/(logn)2)) distinct limit-cycle attractors, each with period exp(Ω(n/logn)) and robust to random noise with flip probability up to 21−o(1), yielding superpolynomial capacity in both the number and length of stored sequences. This is the first demonstration of such capacity for asymmetric Hopfield networks, which we obtain by combining results from combinatorics, number theory and the analysis of opinion dynamics. Our findings show that synchronous asymmetric Hopfield networks possess a sequence-memory capacity which is larger and more robust than previously recognized, demonstrating that, in both biological and artificial neural systems, robust sequence representation can be achieved through coarse architectural motifs rather than complex nonlinearities.
Aakash Kumar, Anatoly Khina, Frederik Mallmann-Trenn +1
COATI, CNRS, Inria, I3S, Université Côte d’Azur, France · School of Electrical and Computer Engineering, Tel Aviv University, Israel · Department of Informatics, King’s College London, UK +1
Learning in recurrent neural networks can fundamentally reshape their underlying dynamics, transforming initially chaotic activity into stable task-dependent behavior. We develop a non-equilibrium dynamical mean-field theory(DMFT) to describe this transition during learning. We show that a slow feedback-driven learning process generates an evolving effective feedback strength that drives the network through a transition from chaotic to stable dynamics defined by a bifurcation of the DMFT solution. By deriving the two-time correlation function throughout learning, we identify a critical feedback strength and a corresponding learning rate dependent critical time separating these regimes. The transition arises from the progressive deformation of an effective dynamical landscape by the growing learned feedback structure. Starting from the untrained state, the theory predicts the time evolution of the network output during training and shows quantitative agreement with numerical simulations.
Varun Vaidya
Department of Physics, University of South Dakota, Vermillion 57069, USA