Stochastic Dynamics of Generative AI: From Autoregressive Models to Interacting LLM Networks and Neural SDEs
Organizations: School of Electrical and Computer Engineering, Cornell University, Ithaca, NY 14853, USA. · Department of Electrical Engineering, Yale University, New Haven, CT 06520, USA.
Abstract
This paper develops a stochastic dynamical-systems perspective on three classes of generative AI models. First, we formulate large language models (LLMs) as high-dimensional nonlinear autoregressive models with attention-based dependencies. This formulation connects pretraining by conditional maximum likelihood to sequential token generation and provides a probabilistic interpretation of LLM operation. Second, we consider networks of interacting LLMs and characterize their collective behavior through mean-field ordinary differential equations. These dynamics provide a framework for understanding emergent preferential attachment and persistent structural biases, including type-dependent glass-ceiling effects. Third, we consider neural stochastic differential equations (SDEs) as continuous-time generative models for stochastic trajectories, including those observed at irregular time intervals. We discuss their training with particular emphasis on Wiener-space cubature methods based on rough-path analysis as an efficient alternative to Monte Carlo estimation. Together, the three settings address probabilistic prediction, information exchange and emergent behavior in interacting networks, and path-space generative modeling.
Figures & tables
| Method | Time | Memory | Comment |
|---|---|---|---|
| Pathwise differentiation engines (per path) | |||
| AD through solver | exact relative to discretization | ||
| Stochastic adjoint | in | default for long trajectories | |
| Forward sensitivity | in | poor in high dimension | |
| Objective / expectation estimators | |||
| Variational MC | engine | engine | ELBO; SDE paths |
Autoregressive Language Modeling
Autoregressive language modeling predicts sequences one step at a time, using prior tokens to generate text, speech, code, and other data.
See all 142 papers