cs.LGNov 4, 2026

Stochastic Dynamics of Generative AI: From Autoregressive Models to Interacting LLM Networks and Neural SDEs

Authors: Vikram Krishnamurthy, Yiming Zhang, Luke Snow

Organizations: School of Electrical and Computer Engineering, Cornell University, Ithaca, NY 14853, USA. · Department of Electrical Engineering, Yale University, New Haven, CT 06520, USA.

Abstract

This paper develops a stochastic dynamical-systems perspective on three classes of generative AI models. First, we formulate large language models (LLMs) as high-dimensional nonlinear autoregressive models with attention-based dependencies. This formulation connects pretraining by conditional maximum likelihood to sequential token generation and provides a probabilistic interpretation of LLM operation. Second, we consider networks of interacting LLMs and characterize their collective behavior through mean-field ordinary differential equations. These dynamics provide a framework for understanding emergent preferential attachment and persistent structural biases, including type-dependent glass-ceiling effects. Third, we consider neural stochastic differential equations (SDEs) as continuous-time generative models for stochastic trajectories, including those observed at irregular time intervals. We discuss their training with particular emphasis on Wiener-space cubature methods based on rough-path analysis as an efficient alternative to Monte Carlo estimation. Together, the three settings address probabilistic prediction, information exchange and emergent behavior in interacting networks, and path-space generative modeling.

Figures & tables

Explore similar work

Feb 23, 2026

Generative Modeling via Kernelized Stochastic Interpolants

We develop a kernel method for generative modeling within the stochastic interpolant framework, replacing neural network training with linear systems. The drift of the generative SDE is b^t(x)=∇φ(x)⊤ηt\hat b_t(x) = \nablaφ(x)^\topη_t, where ηt∈RPη_t \in \mathbb{R}^P solves a P×PP\times P system computable from data, with PP independent of the data dimension dd. Since estimates are inexact, the diffusion coefficient DtD_t affects sample quality; the optimal Dt∗D_t^* from Girsanov diverges at t=0t=0, but this poses no difficulty and we develop an integrator that handles it seamlessly. The framework accommodates diverse feature maps: scattering transforms, pretrained generative models, etc, enabling generation and model combination without neural network training. We demonstrate the approach on financial time series, turbulence, and image generation.
Oct 12, 2026

Autoregressive Decoding as a Martingale: Characterizing the Stochastic Dynamics of LLM Generation

Large language models (LLMs) have achieved remarkable success across a wide range of language tasks. At their core, autoregressive LLMs are conditional probability models that learn to approximate the next-token distribution given the preceding context. Despite this probabilistic nature, the stochastic dynamics of autoregressive decoding have rarely been investigated from a rigorous probabilistic perspective. In this work, we define the per-step prediction error as the difference between the sampled token indicator and the model's conditional predictive distribution, and prove that the cumulative error process is a martingale with respect to the decoding filtration, implying that its conditional first-order moment is identically zero. Beyond this first-order property, we derive a closed-form expression for the second-order behavior of the martingale sequence: its predictable quadratic variation reduces to ∑t(1−p(yt∣y<t))\sum_t\left(1-p(y_t\mid y_{<t})\right), computable from realized token probabilities alone. This identity establishes a direct theoretical connection between sequence fluctuations and predictive uncertainty during autoregressive decoding. White-box experiments on Llama-3.x and Qwen2.5 families (1B to 72B parameters) verify that the empirically measured second-order moments closely match the theoretical predictions derived from our martingale analysis. These results provide a probabilistic framework for analyzing autoregressive LLM decoding, reveal intrinsic probabilistic fingerprints of different language models, and establish a rigorous mathematical foundation for LLM-generated text detection and uncertainty quantification.
Sep 20, 2026

The Probabilistic Structure of Large Language Models

This paper presents a probabilistic perspective on large language models (LLMs), developed with the aim of bringing together, in a single self-contained account, tools that are usually treated separately across the literature. LLMs are described through probability measures on the set of sequences of tokens, specified via their autoregressive conditional distributions. Training is formulated as a maximum-likelihood estimation problem, addressed by stochastic gradient methods, while text generation is viewed as the sequential simulation of the resulting stochastic process. The role of the asymmetry of the Kullback--Leibler divergence in text generation is examined in relation with characteristic phenomena such as hallucination and the distinction between statistical plausibility and truth. As a complementary illustration of the same viewpoint, we also discuss diffusion models, built around the score function, which cast generation not as sequential token prediction but as the simulation of a reverse-time stochastic process transforming noise into data both in discrete and continuous time.