Large Language Models as Nonlinear Dynamical Systems: The Three Clocks of Depth, Training, and Generation
Organizations: Leo AI, Inc., 50 Milk Street, Boston, MA 02109, United States
Abstract
Language models are dynamical systems. A transformer is iterated over its layers, over the steps of its training and over the tokens it generates, and on each of these three clocks it shows behavior that nonlinear dynamics has studied for a century: fixed points, bifurcations, synchronization, chaos and noise-driven escape from potential wells. The two communities rarely meet, so this behavior is often rediscovered under new names. We build a bridge between them. To describe, we condense about 200 papers into the Three-Clock Dictionary, which maps fifteen concepts of nonlinear dynamics onto each clock and marks every correspondence as proved, measured or conjectured, and we show each concept in a mechanical system and in a language model side by side. To explain, we derive reduced-order models: the softmax is a Boltzmann distribution, gradient descent an Euler step on a nonlinear spring, a loss spike a ball thrown out of a stiff valley, attention on the circle a Kuramoto model, and a repetition loop a well that deepens while it is occupied. We then test these explanations on open language models, on a laptop CPU. Escape from repetition loops in GPT-2 and Pythia follows the Arrhenius law with the logit gap as the barrier, and the barrier deepens and then saturates, so the temperature needed to leave a loop falls like with the generation length and rises with model size. Coherent text lives in a narrow window of temperatures, which a small reasoning model shares at its recommended temperature. Training makes each Pythia layer more expansive, and a small transformer trains at a multi-mode edge of stability. The third step, solving problems of language models, is still ahead: controllers derived from the escape analysis remove loops but do not yet beat the best fixed temperature. We state twelve open problems where nonlinear dynamics could get there.
Figures & tables
| Concept | Clock 1: layer time | Clock 2: training time | Clock 3: generation time |
|---|---|---|---|
| Fixed point | Tokens collapse to a cluster; rank collapse P ( Geshkovski et al., 2023a ; Dong et al., 2021 ) . Massive activations nearly input independent M ( Sun et al., 2024 ) | Convergence to a minimum; neural collapse at the end of training M ( Papyan et al., 2020 ) | Period-one loops (one token repeated) as the simplest case of degenerate repetition M ( Holtzman et al., 2019 ) |
| Linear stability, Jacobian spectrum | Dynamical isometry; layer Jacobians of trained models P / M ( Pennington et al., 2017 ; Fernando and Guitchounts, 2026 ) | Sharpness vs. M ( Cohen et al., 2021 ) | Derivative of the feedback map factorizes through M ( Jones, 2026 ) |
| Finite-time Lyapunov exponent | Ridges in input space separate classes M ( Storm et al., 2023 ) ; per-layer profile over training M (Sec. 5.1 ) | Sensitivity of runs to minibatch noise early in training M ( Frankle et al., 2019 ) | Divergence of nearby generations M ( Jaca et al., 2026 ; Jones, 2026 ) |
| Edge of chaos, criticality | Order-chaos transition at initialization sets trainable depth P ( Poole et al., 2016 ; Schoenholz et al., 2016 ) | Edge of stability M ( Cohen et al., 2021 ) | Critical temperature with power-law text statistics M ( Nakaishi et al., 2024 ) |
| Limit cycle (periodic orbit) | ? | Period-2 orbits of GD P ( Chen and Bruna, 2022 ) ; SGD converges to cycles P ( Chaudhari and Soatto, 2017 ) | Repetition loops M ( Xu et al., 2022 ) ; paraphrase 2-cycles M ( Wang et al., 2025 ) ; loops converge in hidden-state space M (Exp. 3) |
| Period doubling, route to chaos | ? | Larger step sizes give period doubling and chaos in minimal models P ( Zhu et al., 2022 ; Kong and Tao, 2020 ) ; period-2 orbit, then irregular oscillation, in a transformer M (Exp. 2) | Feigenbaum scaling in embedding feedback M ( Jones, 2026 ) |
| Language model | Reduced model | Shared term | Status |
|---|---|---|---|
| Escape from a loop (clock 3) | Particle in a well under noise, Fig. 2 (b) | Boltzmann factor: and | Exact identity (Prop. 1 ) |
| Loops that hold for a while (clock 3) | Particle in a deepening well, Fig. 2 (d) | Barrier that grows with residence time | Fitted; within 9% to 25% |
| Regimes of generation (clock 3) | Noisy double-well Duffing oscillator, Fig. 2 (a) | Ratio of barrier to noise | Same ordering, measured |
| Period two at the edge of stability (clock 2) | Explicit Euler step on a nonlinear spring | Step multiplier crossing , cubic term of the loss | Derived; multi-mode edge not covered |
| Loss spike (clock 2) | Ball in a valley whose stiffness changes along it, Fig. 11 | Step multiplier below where the valley is stiff | Exact for the two-parameter model; untested at scale |
| Clustering over depth (clock 1) | Kuramoto oscillators, Fig. 2 (c) | Coupling | Exact for idealized attention; trained GPT-2 departs |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning |
|---|---|
| Language model and clock 1 (layer time) | |
| number of residual blocks (depth) | |
| block index; the layer time of clock 1 | |
| model width | |
| , | vocabulary; its size |
| , | hidden state (residual stream) of token after block ; all tokens together |