Compositional Dynamics in Learning and Mechanics

David I. Spivak

math.CTJun 27, 2026

Compositional Dynamics in Learning and Mechanics

Authors: David I. Spivak

Abstract

We give a single compositional setting in which gradient-based learning and Hamiltonian-style mechanics appear as functorial semantics. The syntax is an operad Arr whose objects are input-output interfaces (pairs of manifolds) and whose morphisms are smooth adaptive arrangements, which consist of a reactive parameter space, a lens given by smooth output and input maps, and a real-valued potential. The main technical result of the paper is what we call lens internalization, a lax symmetric monoidal functor Lens(C) $\to$ C associated to any symmetric monoidal closed category C. Using it, we provide two functors $Φ_\text{phase}$ , $Φ_\text{conf}$ : Arr $\to$ PC into the 2-category of polynomial coalgebras -- input-output discrete dynamical systems -- which we take as the semantics category. $Φ_\text{phase}$ stores both position and momentum, whereas $Φ_\text{conf}$ stores only position. When applied to a parameterized function, $Φ_\text{conf}$ recovers the gradient descent training algorithm, with backpropagation as the lens' backward pass. When applied to harmonic particles wired together -- in series, or according to any finite directed graph -- one diagram yields two different regimes, both of which are governed by the graph Laplacian: $Φ_\text{phase}$ gives the discrete wave equation, which is conservative and second-order, and $Φ_\text{conf}$ gives the discrete heat equation, which is dissipative and first-order. They are two semantics of one adaptive arrangement, e.g. with the same potential in each case. And because Arr is an operad, such diagrams nest -- larger systems wired from smaller ones -- and each semantics assembles a system's dynamics functorially from its parts. These dynamics are moreover executable: a parameterized neural network and a graph of particles both compile, by the same construction, to explicit state machines one can run.

Explore similar work

Feb 4, 2026cs.LG

Gradient Flow Through Diagram Expansions: Learning Regimes and Explicit Solutions

We develop a general mathematical framework to analyze scaling regimes and derive explicit analytic solutions for gradient flow (GF) in large learning problems. Our key innovation is a formal power series expansion of the loss evolution, with coefficients encoded by diagrams akin to Feynman diagrams. We show that this expansion has a well-defined large-size limit that can be used to reveal different learning phases and, in some cases, to obtain explicit solutions of the nonlinear GF. We focus on learning Canonical Polyadic (CP) decompositions of high-order tensors, and show that this model has several distinct extreme lazy and rich GF regimes such as free evolution, NTK and under- and over-parameterized mean-field. We show that these regimes depend on the parameter scaling, tensor order, and symmetry of the model in a specific and subtle way. Moreover, we propose a general approach to summing the formal loss expansion by reducing it to a PDE; in a wide range of scenarios, it turns out to be first-order and solvable by the method of characteristics. We observe a very good agreement of our theoretical predictions with experimental results.

Dmitry Yarotsky, Eugene Golikov, Yaroslav Gusev

Gradient Flow Wasserstein Gradient Flows Tensor Learning Rates Exact Neural Scaling Laws+1

Ask PDF

Jun 16, 2026cs.LG

A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks

We develop a mathematically explicit link between shock-wave theory and the symmetry-quotiented learning dynamics of stochastic gradient descent, drawing on differential geometry, Lie group theory, and fluid mechanics. Specifically, after quotienting parameter symmetries and applying local-entropy coarse-graining, the effective dynamics satisfy a viscous Hamilton--Jacobi equation on the quotient manifold. Moreover, under the assumption that the raw parameter dynamics can be summarized by a gradient field on the quotiented space, the gradient of the coarse-grained loss function obeys a Burgers-type equation, and shock formation can be established rigorously. We apply our theory to multilayer perceptrons, convolutional neural networks, Transformers, and mean-field networks, and show that they obey the Hamilton--Jacobi or Burgers-type equations. We conjecture that this framework also yields practical diagnostics for deep learning. In architectures such as Transformers, raw parameter norms are often distorted by symmetry redundancy and may therefore be misleading, whereas symmetry-corrected quotient observables provide a principled basis for monitoring, forecasting, and controlling training-phase transitions.

Taiki Miyagawa

Two-Layer Neural Networks Stochastic Gradient Descent Neural Network Symmetry Loss Function Multi-Material Shock Physics

Ask PDF

Sep 10, 2026cs.AI

A Function-Space Approach to the Statistical Mechanics of Learning Dynamics

Deep neural networks exhibit regular macroscopic behavior despite highly nonlinear dynamics in vast parameter spaces. We develop a statistical-mechanical description of learning directly in function space, treating parameter configurations as microscopic realizations and functions with their dynamical operators as macroscopic variables. For mean-squared loss, the exact error dynamics are governed by the learning operator

M=JJ^\ast

. Combining the dynamical Boltzmann weight of the conditional stochastic dynamics with the parameter-space density of states, whose local curvature defines a statistical operator

B

, and integrating over local fluctuations yields

\Phi_{\mathrm{fluc}}(M;B)=\frac{\sigma_\xi^2}{2}\log\det(M^{-1}+B)+\mathrm{const}.

At fixed spectrum, this term is rotationally stationary when

[M,B]=0

, is minimized by pairing large eigenvalues of

M

with small eigenvalues of

B

, and generates a local restoring contribution against rotational mismatch. For ReLU-type function spaces under mild stable statistical conditions,

B=\sigma_\xi^2L^\ast\mathcal K L

, where

L

measures coarse-grained second-order structure. Thus the low-

B

sector corresponds, up to bounded anisotropy of

\mathcal K

, to low structural curvature, implying a preference for faster relaxation along smooth, data-adaptive directions. These results identify function space as a natural macroscopic level for studying stable collective organization in learning.

Yizhou Zhang, Weichen Wu, Lun Du +1

Learning Dynamics Statistical Mechanics Weight-Space Boltzmann Distributions

Ask PDF