Task-Restricted Symmetries in Recurrent Weight Space

Continuous-memory recurrent networks must preserve phase, position, or orientation despite model imperfections and state noise. We develop a geometric framework that separates three questions: how many memory coordinates are neutrally transported, how deterministic perturbations alter their finite-horizon stability, and how ambient noise is decoded along them. Exact per-input equivariance transports analytical group tangents pathwise and, on a compact nondegenerate orbit stratum, yields at least $q=\dim(G/H)$ zero group-tangent Lyapunov exponents under stationary ergodic driving. For imperfect dynamics, a four-block tangent/normal decomposition gives local and finite-horizon bounds on tangent growth and subspace rotation, distinguishing first-order direct damage from second-order leakage through contracting normal directions. For noisy dynamics, a specified decoder maps ambient covariance $Q$ to coordinate covariance $ZQZ^\top$; under isotropic noise and fixed tangent energy, least-squares decoding and scaled-isometric action geometry minimize local diffusion. Local and finite-horizon evaluations include cases both within and outside the sufficient conditions. In a fresh twenty-seed $T^2$ replication, a decoder-covariance objective improves noisy horizon-256 memory in every pair while meeting a prespecified clean-error equivalence margin. Direct noise training also improves noisy memory but incurs a clean-error tradeoff. In coupled $T^4/T^8$ integrators, the advantage of anisotropic-covariance over isotropic regularization reverses when evaluation noise becomes isotropic. These results connect continuous symmetry to measurable limits and design choices for recurrent memory under explicit dynamical, decoder, and noise assumptions.

Learning in neural systems arises from synaptic changes that reshape the representations underlying behavior. While low-rank recurrent neural networks (RNNs) have emerged as a powerful framework for linking connectivity to function, a theoretical understanding of their learning process remains elusive. Here, we extend the low-rank framework from activity to learning by deriving gradient-descent dynamics directly in a reduced overlap space. We formulate a closed-form, low-dimensional system of ODEs that governs learning in this space, exact for linear RNNs and asymptotically exact for nonlinear RNNs in the large-$N$ Gaussian limit. Central to our analysis is a distinction between two classes of overlaps: loss-visible overlaps, which fully determine network activity, output, and loss, and loss-invisible overlaps, which do not affect function but are required to describe learning. We illustrate the consequences of this decomposition through two phenomena. First, we show that learning can serve as a perturbation that exposes differences in connectivity between functionally equivalent networks. Second, we show that loss-invisible overlaps can act as memory variables that encode training history, and characterize the conditions under which this occurs. Finally, we present several testable predictions for biological learning experiments derived from our theory.

The pruning of network connections is key to brain function but, despite its importance, there exist few biologically-plausible pruning rules with demonstrated good performance. In this work we evaluate noise-prune, a recently introduced unsupervised local pruning rule for recurrent networks that uses noisy fluctuations to determine the importance of connections. Noise-prune has previously only been empirically tested on random networks without a specific computational function. We show that noise-prune preserves task-performance in task-trained recurrent neural networks, greatly outperforming a strategy that only uses the magnitude of connections and performing on par with or exceeding a non-local strategy that uses second-order information. Rather than deterministically removing connections that fall below a certain threshold importance, noise-prune samples connections to preserve based on their importance and strengthens retained connections to preserve average synaptic strength. We show that this sampling and rescaling is essential to good performance, but that the optimal empirical degree of rescaling is lower than that predicted by the original theoretical argument. Our work thus validates noise-prune as a biologically-plausible pruning rule for functional recurrent network architectures and characterizes its optimal parameter settings.