Detailed balance in large language model-driven agents

Zhuo-Yang Song; Qing-Hong Cao; Ming-xing Luo; Hua Xing Zhu

cs.LGDec 10, 2025

Detailed balance in large language model-driven agents

Authors: Zhuo-Yang Song, Qing-Hong Cao, Ming-xing Luo, Hua Xing Zhu

Organizations: School of Physics, Peking University, Beijing 100871, China · Center for High Energy Physics, Peking University, Beijing 100871, China · Beijing Computational Science Research Center, Beijing 100193, China

Abstract

Large language model (LLM)-driven agents are emerging as a powerful new paradigm for solving complex problems. Despite the empirical success of these practices, a theoretical framework to understand and unify their macroscopic dynamics remains lacking. This paper proposes a method based on the least action principle to estimate the underlying generative directionality of LLMs embedded within agents. By experimentally measuring the transition probabilities between LLM-generated states, we report evidence consistent with a tendency toward detailed balance in LLM-generated transitions, indicating that LLM generation may not be achieved by generally learning rule sets and strategies, but rather by implicitly learning a class of underlying potential functions that may transcend different LLM architectures and prompt templates. To our knowledge, this provides an effective macroscopic description of LLM generative dynamics that does not depend on specific model details. This work is an attempt to establish a macroscopic dynamics theory of complex AI systems, aiming to elevate the study of AI agents from a collection of engineering practices to a science built on effective measurements that are predictable and quantifiable.

Figures & tables

Figure 1: A schematic of a formalization framework for studying the directionality of LLM generation, illustrating the state space and possible transitions. LLMs are embedded within an agent, which transitions from state $f$ to state $g$ with probability $\mathcal{T}(g\leftarrow f)=P(g|f)$ . The underlying potential function $V_{\mathcal{T}}$ quantifies the agent’s global ordering of each state, satisfying the detailed balance condition at equilibrium.

Figure 2: In the conditioned word generation task, the Claude Sonnet 4.5 model exhibits directionality. Transition process of the Claude Sonnet 4.5 model in the prompt word state space, ordered by the potential function

V_{\mathcal{T}}

. Transitions tend to move toward states with lower potentials. States with

\beta V(f)\gg\log(20000)\sim 10

are those where the equilibrium condition cannot be strictly satisfied.

Figure 3: In the task of conditioned word generation without a reasoning chain, verification of detailed balance through closed paths in the state transition graph of the GPT-5 Nano model. Each point represents a triplet, with all different triplets found in the experimental data. The error bars directly arise from sampling errors.

\chi^{2}/\text{ndf}

is the chi-square per degree of freedom. Note that both axes are in natural logarithm (

\ln

), not

\log_{10}

states $f$	Potential
param1 * tanh(param2 * x + param3) + param4	5.70
param1 - (param2 / (x + param3))	0.88
param1 * x / (1 + param2 * log(x + 1))	-0.57
param1 * tanh(param2 * x) + param3	-1.57
param2 + param1 * (1 - exp(-x))	-3.30

Table 1: Examples of some states of the agent

\mathcal{T}_{\text{real}}

and their potentials. The independent variable log_v_k_nu in the fitting task is abbreviated as x .

Figure 4: In the task of symbolic fitting with a long reasoning chain, test of the detailed balance condition for the agent

\mathcal{T}_{\text{real}}

. The error estimation includes root mean square statistical errors, and the error of the potential function is estimated through bootstrapping.

r

is the Pearson correlation coefficient, and

\chi^{2}/\text{ndf}

is the chi-squared per degree of freedom. Measurement includes state pairs with at least one measured transition between the two states. Points with

|V_{\mathcal{T}}(f)-V_{\mathcal{T}}(g)|>\log{50288}

are excluded. One-tenth of the data points are displayed for clarity.

Figure 5: Illustration of the ability of the potential function discovered using IdeaSearch [ 18 ] to predict the directionality of state transitions. Each point represents a transition pair from state $f$ to state $g$ . The figure shows a subgraph composed of 70 states selected from the database, displaying transitions with high transition kernel $\mathcal{T}_{\text{real}}(g\leftarrow f)>0.05$ . Red and green lines represent transitions with increasing and decreasing potential functions, respectively. The horizontal axis represents the mean square error of the expression corresponding to the state in the symbolic fitting task, while the vertical axis represents the potential function.

Agent	Transition Samples	Unique States	Unique Transitions	States with Samples $>1$
IdeaSearchFitter	50228	7484	21697	2551
Conditioned Word Generation (GPT-5 nano)	19968	645	9473	620

Table S1: Database statistics for the two agents, including transition sampling counts, unique state counts, unique transition counts, and the number of states with sampling times greater than 1.

Figure S1: Comparison of

\text{Correct Rate}=1-\frac{\sum_{f}{N(\text{failed}\leftarrow f)}}{\sum_{f}{N_{0}(f)}}

for different LLMs in the conditioned word generation task. The Qwen series models are Qwen2.5-Instruct-3B, Qwen2.5-Instruct-7B, Qwen2.5-Instruct-14B, Qwen2.5-Instruct-32B, Qwen2.5-Instruct-72B and Qwen3-Max. The Claude series models are Claude Sonnet 3.7, Claude Sonnet 4, and Claude Sonnet 4.5. The Gemini series models are Gemini 2.0 Flash and Gemini 2.5 Flash. The GPT series models are GPT-4.1 nano and GPT-5 nano.

Figure S2: Test of the detailed balance condition for the Conditioned Word Generation Agent using GPT-5 nano model. The error estimation includes root mean square statistical errors, and the error of the potential function is estimated using the bootstrap method.

r

is the Pearson correlation coefficient, and

\chi^{2}/\text{ndf}

is the chi-squared per degree of freedom. Measurement includes state pairs with at least one measured transition between the two states. The agent’s underlying potential function is consistent with the detailed balance condition, with systematic deviations from detailed balance observed at higher potential function values. Points with

|V_{\mathcal{T}}(f)-V_{\mathcal{T}}(g)|>\log{20000}

are excluded. One-fifth of the data points are displayed for clarity.

Figure S3: Architecture of the IdeaSearchFitter agent (reproduced from Ref. [ 45 ] ). Input data and variable descriptions are expanded into a structured prompt once at initialization, after which the system iterates: an ansatz agent (LLM) generates candidate expressions informed by the best-performing entries in a database, an evaluator scores them, and the results are stored in an island-based database for selection and mutation.

Parameter	Value
Parallel LLM calls per generation ( samplers_num )	3
Best examples shown in prompt ( examples_num )	1
Candidates per call ( generate_num )	3
Sampling temperature	1000
Number of LLM generators	10
Number of islands	8

Table S2: Key hyperparameters of the IdeaSearchFitter agent used in this work, taken from the GitHub repository.

Figure S4: Test of detailed balance through closed paths in the state transition graph of the complex agent

\mathcal{T}_{\text{real}}

. Each point represents a triplet, with a total of 620 different triplets found in the experimental data, where each transition was detected at least twice. This indicates that within the error range, the sums of the logarithms of the forward and backward transition kernels are roughly equal, consistent with the detailed balance condition. To clearly display the figure, only 1/5 of the data points are shown.

Figure S5: Test of the detailed balance condition for both agents using

K(x)=\log{(1+e^{-x})}

, with all settings the same as in the main text.

r

is the Pearson correlation coefficient, and

\chi^{2}/\text{ndf}

is the chi-squared per degree of freedom. (a) Results for the IdeaSearchFitter agent. (b) Results for the Conditioned Word Generation Agent.

Figure S6: For pairs in the IdeaSearchFitter and Conditioned Word Generation Agent where the transition

f\leftarrow g

was measured but the transition

g\leftarrow f

was not, we compare the differences in their potential functions and the logarithm of the transition kernel ratios. The figure shows points corresponding to 500 such pairs. The black dashed line represents the 90th percentile line. It is mostly above the red dashed line representing detailed balance, indicating that the inequality Eq. ( S8 ) is basically satisfied, with those points at larger

|\beta V_{\mathcal{T}}(f)-\beta V_{\mathcal{T}}(g)|

possibly arising from systematic overestimation of the potential function differences. (a) Results for the IdeaSearchFitter agent with a total of 18,935 such pairs. (b) Results for the Conditioned Word Generation Agent with a total of 8,805 such pairs. From this figure, it can be directly seen that most of such transition pairs come from cases where

N(f\leftarrow g)=1

Figure S7: Results of the detailed balance test for the random agent

\mathcal{T}_{\text{rand}}

r

is the Pearson correlation coefficient, and

\chi^{2}/\text{ndf}

is the chi-squared per degree of freedom. (a) Examination of the detailed balance condition by comparing the differences in potential functions and the logarithm of transition kernel ratios. (b) Examination of the detailed balance condition through closed paths in the state transition graph. It can be seen that the state transitions of this agent do not satisfy the detailed balance condition.

Task	States	Trans	$N_{\rm pairs}$	$r$	$r_{\rm floor}$	$r_{\rm ceil}$
Semantic agents (Qwen2.5-72B):
5-Letter-S	206	4232	240	0.58	0.17	0.88
Digit Sum $=$ 15	41	196	15	0.79	0.34	1.00
Sum $\div$ 10	69	242	7	0.84	0.52	0.99
Reference models from main text:
GPT-5 nano word gen.	645	9473	329	0.77	0.15	0.90

Table S3: Multi-task test of detailed balance, with matched floor and ceiling controls.

r

is the measured Pearson correlation between

\ln(\mathcal{T}(g\leftarrow f)/\mathcal{T}(f\leftarrow g))

and

V(f)-V(g)

for bidirectional pairs with both counts

>1

r_{\rm floor}

and

r_{\rm ceil}

are matched controls processed through the identical pipeline, quoted as medians over repeated randomizations; their construction and interpretation are described in the text.

Figure S8: Two overlaid pairs of distributions of the consistency measure

r

for the tasks of Table S3 ; the construction of the controls is described in the text. Left: the matched floor control

r_{\rm floor}

(gray) overlaid with the measured random-agent controls (blue). Right: the matched ceiling control

r_{\rm ceil}

(gray) overlaid with the measured semantic agents (red). Each box spans the min–max range across tasks within its group; the black bar marks the cross-task mean of the measured

r

in each group.

Figure S9: Sensitivity of the detailed-balance test to the minimum sampling threshold

N_{0}^{\min}

for IdeaSearchFitter.

N_{0}(f)

is the total number of outgoing transitions from state

f

, and

N_{\mathrm{pairs}}

is the number of state pairs with bidirectional transitions passing the threshold. Left: Pearson

r

between

\ln[\mathcal{T}(g\!\leftarrow\!f)/\mathcal{T}(f\!\leftarrow\!g)]

and

\beta(V(f)-V(g))

remains high (

r\simeq 0.82

–

0.92

) across all thresholds, with only a modest decline even when the dataset is restricted to the

{\sim}\,600

best-sampled pairs. Right:

\chi^{2}/\mathrm{ndf}

increases from

{\sim}\,0.85

N_{0}^{\min}=1

, crossing unity near

N_{0}^{\min}=10

and reaching

{\sim}\,1.2

N_{0}^{\min}=50

; the rise reflects a slight mismatch between the globally fitted potential and the high-threshold subset, rather than a systematic violation of detailed balance.

Figure S10: Comparison of PageRank potential $V_{\text{PR}}=-\ln\pi(f)$ and variational potential $V_{\text{var}}$ for the GPT-5 nano word task with damping factor $\alpha=0.99$ . Light blue points are per-state values; dark red points are binned means with $1\sigma$ error bars. The dashed line has slope 1; the solid line is a linear fit (slope $=1.30$ ). The binned Pearson correlation is $r=0.97$ .

From $\backslash$ To	ATT.	TUR.	PER.	PRO.	DIS.	failed
ATT.	0	0	31	27	4	3480
TUR.	3521	0	0	0	0	74
PER.	3500	0	0	0	0	0
PRO.	3401	0	0	0	0	0
BUZ.	2055	1254	0	0	0	218
DIS.	2435	0	0	0	0	0

Table S4: Transition kernel

\mathcal{T}(g\leftarrow f)

for the Claude Sonnet 4.5 model, where ATT., TUR., PER., PRO., BUZ., and DIS. represent the states ATTITUDE, TURKEY, PERSONAL, PROBLEM, BUZZY, and DISCIPLINE respectively. Each row represents the number of transitions starting from state

f

, and each column represents the number of transitions to state

g

. The reason for “failed” is that some transitions are rejected because they are words but the sum of letters is not 100, or the generated word is still the prompt word, especially for transitions starting from the ATTITUDE state.

Figure S11: Transition process of the Gemini 2.5 Flash model in the prompt word state space, sorted by the potential function

V_{\mathcal{T}}

. Transitions tend to move towards states with lower potential functions. States with

\beta V(f)\gg log(20000)\sim 10

are those where the equilibrium condition cannot be strictly satisfied. Thick lines represent high-frequency transitions, while thin lines represent low-frequency transitions. Each horizontal line represents a state, arranged in order of increasing potential function.

Metric	Claude Sonnet 4.5	Gemini 2.5 Flash
Chains converging to ATTITUDE	89/100 (89%)	99/100 (99%)
Mean steps to first ATTITUDE	2.3	1.0
Median steps to first ATTITUDE	2	1
ATTITUDE fraction of valid outputs	68.8%	54.1%
Second most common word	TURKEY (16.6%)	DISCIPLINE (45.9%)
Overall validity rate	15.7%	75.3%

Table S5: Chain experiment: 100 independent Markov chains from diverse starting words, each running 10 steps. Starting words include common English words, rare words, and nonsensical strings satisfying the letter-sum-equals-100 constraint.

Figure S12: (a) Distribution of potential functions for all states sampled at least twice in the IdeaSearchFitter agent. It exhibits a localized structure and can be fitted with a Gaussian distribution

\mathcal{N}(\mu=-0.56,\sigma=4.37)

. (b) Distribution of potential functions for all states sampled at least twice in the Conditioned Word Generation Agent. It exhibits a localized structure and can be fitted with a Gaussian distribution

\mathcal{N}(\mu=-0.93,\sigma=2.30)

Agent	$\sigma$	Expected min. action	Actual min. action
IdeaSearchFitter	4.38	0.129	0.150
Conditioned Word Generation(GPT-5 nano)	2.30	0.245	0.195

Table S6: Comparison of expected minimum action size through the potential function and actual minimized action for two agents.

Figure S13: Numerical comparison of both sides of Eq. ( S22 ). The comparison is performed for

M=10

, with different colors representing different values of

\mathcal{T}(g\leftarrow f)

. The three subplots represent the cases of

n=5,7,9

, respectively.

Figure S14: Resolution contrast between the microscopic architectural energy and the macroscopic state-level potential. (a) In an energy-based model such as EBT,

E_{\theta}(x,\hat{y})

is a per-token energy; inference proceeds by gradient-descent energy minimization, and the full sequence is still generated autoregressively. (b) Our potential

V(f)

is a macroscopic free energy defined on the semantically closed states

f

(blue) and

g

(red), obtained by coarse-graining over all token sequences that realize a given state. The intermediate chain-of-thought reasoning tokens (gray) carry no

V

Statistical Physics of Learning

Statistical physics of learning uses tools from physics to explain how data, model structure, and training dynamics shape learning and generalization.

See all 53 papers

Mechanism Plausibility in Generative Agent-Based Modeling
May 12, 2026Patrick Zhao, David Huu Pham, Nicholas VincentLLM Agent Evaluation Agent-Based Modeling
Simon Fraser University, Burnaby, BC, Canada
Conversable Complexity: Agentic LLM Collectives as Interpretable Substrates
Jul 1, 2026Elias Najarro, Ane Espeseth, Eleni Nisioti +2Artificial Life LLM Interpretability
IT University of Copenhagen, Denmark · University of Oslo, Norway · Sakana AI, Japan +1
Building LLM Agent Systems the Deep Learning Way: From Modular Design to Architecture Search
Oct 4, 2026Tao Feng, Pengrui Han, Zhongjie Dai +1LLM Agents AI Agent Architecture
University of Illinois Urbana-Champaign

May 12, 2026J/K move · Enter open · S save

Mechanism Plausibility in Generative Agent-Based Modeling

Patrick Zhao, David Huu Pham, Nicholas Vincent

Simon Fraser University, Burnaby, BC, Canada

LLM Agent Evaluation Agent-Based Modeling Large Language Model-Based Social Simulation Causal Explanation Explainability Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) can generate high-level diverse phenomena without explicitly programmed rules. This capability has led to their adoption within different agent-based models (ABMs) and social simulations. Recent studies investigate their ability to generate different phenomena of interest, for example, human behavior on social media platforms or alien behavior in game-theoretic scenarios. However, capability, prediction, and explanation are different--drawing from the philosophy of science and mechanisms literature, explanation requires showing, to some degree, how a phenomenon is produced by related organized entities and activities. For modelers, describing the characteristics of an experiment or whether a simulation provides progress in capability (or explanation), can be difficult without being grounded in potentially distant research areas. We integrate recent work on LLM-ABMs with contemporary philosophy of science literature and use it to operationalize a definition of 'plausibility' in a four-level scale. Our scale separates the evaluation of a model's generative sufficiency (ability to reproduce a phenomenon) from its mechanistic plausibility (how the phenomenon could be produced), and clarifies the distinct roles of different models, such as predictive and explanatory ones. We introduce this as the Mechanism Plausibility Scale.

Abstract

Statistical Physics of Learning

Explore similar work

Mechanism Plausibility in Generative Agent-Based Modeling

Conversable Complexity: Agentic LLM Collectives as Interpretable Substrates

Building LLM Agent Systems the Deep Learning Way: From Modular Design to Architecture Search