Large language model (LLM) agents can be organized into multi-agent markets that act as world models, mapping macro information to emergent macro dynamics while the evaluator observes only macro inputs and outputs. This paper separates a model's capability into two stages that a single aggregate score cannot separate: information simulation, from macro information to micro behavior, and macro emergence, from micro behavior to the macro outcome. We introduce a soft macro-control method that announces a public reference path day by day without touching individual decisions, orders, or matching, and a two-regime evaluation framework that measures both stages from macro observations alone. In the reference-only regime the symmetric true and flipped references share a common level R, which measures realization under the announced reference, while their paired gap B measures structural bias; in the information-injection regime external information dilutes the reference, so only the true-flipped gap Delta remains interpretable. On five single-index LLM-agent markets, announcing a target path shifts the realized path toward the announced target on both announced references under both information conditions. Relative to the no-reference path, the tracking score is higher by 0.306 and 0.415 without news and by 0.183 and 0.329 with news; within the stock/cash scan structural bias is lowest at 70/30 (B=0.042); realization peaks at a 3:7 technical share (R=0.894 versus 0.358 for technical-only); and Delta is 0.0341 with fundamental data and 0.0487 with news. Both stages use the same two observable scores, each read in the regime where it is interpretable.
Figures & tables
Figure 1: Overview of the framework. Top: the evaluator sees only the black-box map W from the public macro information to a macro output path; micro behavior is internal and unobserved. The evaluator announces the public macro reference level at the start of each trading day. (a) In the reference-only regime the model is run on a true reference and on its flipped mirror, so the common level R measures realization capability and the root-mean-square paired gap B measures structural bias. (b) In the information-injection regime external information is added; absolute scores are no longer comparable, and only Δ=TSR−FSR is interpreted.
Family
Manipulation
Held fixed
Control
true target with vs. without a public reference; news off vs. news on
composition 3:7, stock/cash 70/30, fundamental data on
C1
stock/cash = 30/70, 50/50, 70/30, 90/10
composition 3:7, news off, fundamental off
C2
technical/fundamental = 0/10, 3/7, 5/5, 10/0
stock/cash 70/30, news off, fundamental off
C3
fundamental information off vs. on
composition 3:7, stock/cash 70/30, news off
News
news off vs. on
composition 3:7, stock/cash 70/30
Table 1: Experimental conditions. Every condition is run on the five indices under both a true and a flipped macro reference. C1 varies the stock/cash split (structure), C2 the technical/fundamental composition (realization), and C3 whether fundamental data is injected (information).
Condition
Reference
Qno
Qref
Gain
News off
true
0.6115
0.9180
0.3065
flipped
0.4688
0.8839
0.4152
News on
true
0.6473
0.8307
0.1834
flipped
0.4533
0.7820
0.3287
Table 2: Control effectiveness for the four groups (true and flipped reference × news off and on). Both news conditions include the fundamental data and therefore lie in the information-injection regime; the reference-only baseline of Tables 3 – 4 is a different cell. Qno is the no-reference score on the announced reference, Qref the score with the public reference, and Gain their difference (from unrounded values).
Figure 2: Per-asset control gains for the four groups (true and flipped reference, news off and on). Bars are group means and dots the five individual assets; solid bars are true references and hatched bars flipped references. Every per-asset gain is positive, and the group means are 0.306 and 0.415 without news and 0.183 and 0.329 with news.
Stock/cash
TSR
FSR
R
Δˉ
Brms
30/70
0.7802
0.7258
0.7530
+0.0544
0.2277
50/50
0.8527
0.8222
0.8375
+0.0305
0.1184
70/30
0.8917
0.8970
0.8943
−0.0052
0.0418
90/10
0.6890
0.7884
0.7387
−0.0994
0.1831
Table 3: Reference-only stock/cash experiment. TSR and FSR are means over the five assets; R is their average; Δˉ is the mean paired gap and Brms is its root mean square; unrounded values.
Tech./fund.
TSR
FSR
R
Δˉ
Brms
0/10
0.8829
0.8917
0.8873
−0.0089
0.0169
3/7
0.8917
0.8970
0.8943
−0.0052
0.0418
5/5
0.8359
0.7908
0.8134
+0.0451
0.0871
10/0
0.2764
0.4398
0.3581
−0.1633
0.2347
Table 4: Reference-only composition experiment. R=(TSR+FSR)/2 ; Δˉ and Brms are the mean and root-mean-square paired gaps; unrounded values.
Figure 3: Information effects measured by Δ=TSR−FSR . The first bar is the reference-only symmetry check; the second adds fundamental data and the third adds news to the injected information. Both information-injection bars are positive, and news raises Δ from 0.0341 to 0.0487 . Absolute scores are not interpreted once information is injected.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Component
Ref.-only
News off
News on
Level, true arm
0.8645
0.8906
0.8008
Shape, true arm
0.9189
0.9453
0.8605
TSR
0.8917
0.9180
0.8307
Level, flipped arm
0.8658
0.8443
0.7863
Shape, flipped arm
0.9281
0.9235
0.7777
FSR
0.8970
0.8839
0.7820
Appendix
Table 5: Component decomposition of the headline cells (means over the five assets). The true arm is scored against MT and the flipped arm against MF ; TSR and FSR are the corresponding means of Q , equal to the average of the two components up to rounding; the reference-only column has fundamental data off, and both news columns have it on.
Symbol
Meaning
Min , Bmicro , Mout
macro information, micro behavior, macro output
W , P=W(M)
black-box world model and its macro output path
Mref
public macro reference path announced by the evaluator
MT , MF
true target path and flipped target path
Q(P,M)
path realization score of output P against target M
et , σM
daily relative error (%); volatility of target log returns (% per day)
Appendix
Table 6: Notation.
Regime
Quantities
Interpretation
Reference-only
TSR, FSR, R , B
Common level measures realization capability; paired gap measures structural bias; B is read jointly with R .
Information-injection
Δ=TSR−FSR
Positive or increasing Δ indicates effective information; absolute TSR/FSR are not interpreted.
Appendix
Table 7: Interpretation rules for the same two observable scores.
Code
Name
Example constituents
Const.
Platform
CGEI
consumer goods
Kweichow Moutai (69.2%)
5
18.2%
EREI
energy & resources
China Yangtze Power (33.5%)
6
13.2%
FSEI
financial services
Ping An (22.9%)
11
33.0%
REEI
real estate
Poly Developments (100%)
1
1.0%
TTEI
technology & telecom
SMIC (18.1%)
10
13.0%
Appendix
Table 8: The five evaluated composite indices. “Constituents” is the number of A-share stocks in the fixed-weight basket; the parenthesised weight is the largest constituent’s share within the index. The last column is the sector’s weight in the platform’s aggregate ten-sector total-market index; it is not used in the single-index runs and is listed only for provenance.
News off
News on
Asset
Qno
Qref
Gain
Qno
Qref
Gain
CGEI
0.5344
0.9132
+0.3788
0.7085
0.9055
+0.1970
EREI
0.6205
0.9259
+0.3055
0.7318
0.8548
+0.1230
FSEI
0.7134
0.8785
+0.1650
0.6905
0.7529
+0.0623
REEI
0.6541
0.9382
+0.2841
0.7368
0.8686
+0.1318
TTEI
0.5351
0.9340
+0.3989
0.3688
0.7716
+0.4028
Appendix
Table 9: Per-asset control effectiveness on the true target. Both news conditions include the fundamental data and therefore lie in the information-injection regime. Qno and Qref are the no-reference and reference realization scores against MT ; Gain =Qref−Qno computed from unrounded values. The mean rows are the aggregates (Table 2 ).
News off
News on
Asset
Qno
FSR
FSR gain
Qno
FSR
FSR gain
CGEI
0.6382
0.8896
+0.2514
0.3887
0.7608
+0.3722
EREI
0.5989
0.9157
+0.3167
0.3777
0.8325
+0.4548
FSEI
0.3510
0.8392
+0.4882
0.3059
0.5168
+0.2109
REEI
0.5645
0.9330
+0.3684
0.4680
0.9260
+0.4580
TTEI
0.1911
0.8421
+0.6510
0.7263
0.8739
+0.1476
Appendix
Table 10: Per-asset flipped-arm gain, in the information-injection regime. Qno is the no-reference output scored against MF ; FSR is the reference arm’s flipped-arm score against MF ; FSR gain is their difference (computed from unrounded values). Means are over the five assets.
Condition
CGEI
EREI
FSEI
REEI
TTEI
Stock/cash 30/70
0.8072/0.3702
0.7532/0.7143
0.8076/0.7733
0.9061/0.8886
0.6271/0.8826
Stock/cash 50/50
0.8524/0.6547
0.8253/0.8038
0.8810/0.8119
0.9413/0.9180
0.7638/0.9227
Stock/cash 70/30
0.9316/0.8688
0.8947/0.9119
0.8850/0.8919
0.8992/0.8975
0.8481/0.9148
Stock/cash 90/10
0.9169/0.8955
0.6791/0.6852
0.8005/0.9249
0.2202/0.6096
0.8282/0.8267
Composition 0/10
0.8955/0.8786
0.8704/0.8876
0.9115/0.9148
0.8861/0.9044
0.8507/0.8732
Composition 3/7 (shared)
0.9316/0.8688
0.8947/0.9119
0.8850/0.8919
0.8992/0.8975
0.8481/0.9148
Appendix
Table 11: Per-asset TSR / FSR by condition. Each cell is TSR/FSR for that asset. Reference-only rows use that condition’s true and flipped arms; the last two rows are the information-injection cells of the fundamental-data and news experiments (their news-off row is the fundamental-data-on cell). The composition 3/7 row is the shared baseline reused by the other experiments; it is the same cell as stock/cash 70/30 and is listed here for completeness.
News off
News on
Asset
QT
QF
Δno
QT
QF
Δno
CGEI
0.5344
0.6382
−0.1038
0.7085
0.3887
+0.3198
EREI
0.6205
0.5989
+0.0215
0.7318
0.3777
+0.3541
FSEI
0.7134
0.3510
+0.3624
0.6905
0.3059
+0.3846
REEI
0.6541
0.5645
+0.0896
0.7368
0.4680
+0.2688
TTEI
0.5351
0.1911
+0.3440
0.3688
0.7263
−0.3575
Appendix
Table 12: Per-asset no-reference baseline for the news experiment. The same no-reference output is scored against MT ( QT ) and MF ( QF ); Δno=QT−QF . Means are over the five assets.
Path / model
Qˉ
Level
Shape
Percentile
Constant
0.6104
0.7207
0.5000
56
Linear interpolation
0.8210
0.8121
0.8299
96
Matched-vol. RW
0.5650
0.6306
0.4994
–
Shuffled returns
0.6821
0.7308
0.6333
69
No-reference, news off
0.6115
0.5979
0.6251
58
Reference, news off
0.9180
0.8906
0.9453
100
Appendix
Table 13: Zero-information nulls and model scores (means over the five true targets). Percentile columns locate the model score in the per-asset distribution of the matched-volatility random-walk null. “–” means not applicable.
Q definition
R
B
Δ off
Δ on
News eff.
Sign
Paper Q
0.8943
0.0418
+0.0341
+0.0487
+0.0146
+
Common σ
0.8938
0.0444
+0.0342
+0.0503
+0.0161
+
Shape on returns
0.7602
0.0476
+0.0386
−0.0029
−0.0415
−
Shape only
0.9235
0.0356
+0.0218
+0.0828
+0.0610
+
Level only
0.8652
0.0506
+0.0463
+0.0146
−0.0318
−
Appendix
Table 14: Score sensitivity. R and B are the reference-only 70/30 cell; Δ is the news experiment. “News effect” is Δon−Δoff (or its sign); these are the values discussed in Section 7.6 .
Estimand
Point
95% interval
Gain (true), news off
0.3065
[0.2316,0.3722]
Gain (flipped), news off
0.4152
[0.3009,0.5516]
Gain (true), news on
0.1834
[0.1005,0.2944]
Gain (flipped), news on
0.3287
[0.2178,0.4395]
R , reference-only 70/30
0.8943
[0.8866,0.9011]
B , reference-only 70/30
0.0418
[0.0089,0.0589]
Appendix
Table 15: Asset-level bootstrap intervals (percentile, 20,000 draws over the five asset pairs; intervals are discrete and descriptive). Δ is TSR−FSR , and the gains are reported separately for the true and flipped references under each news condition.
Asset
Reported Δ
Repeat Δ
Difference
CGEI
+0.1447
−0.0022
−0.1469
EREI
+0.0223
+0.0214
−0.0009
FSEI
+0.2361
+0.3451
+0.1090
REEI
−0.0574
−0.0333
+0.0242
TTEI
−0.1023
−0.0148
+0.0875
Mean
+0.0487
+0.0632
+0.0146
Appendix
Table 16: Replicate batch of the news-on condition: per-asset Δ=TSR−FSR . “Reported” is the batch used in the tables; “repeat” is the earlier independent batch; the last column is repeat minus reported.
Figure 4: Structural scans in the reference-only regime. (a) Stock/cash allocation versus TSR and FSR (lines, left axis) and structural bias Brms (bars, right axis); bias is lowest at 70/30 ( Brms=0.042 ). (b) Technical-agent share versus realization capability R ; an intermediate share (3:7) realizes best ( R=0.894 ), whereas a technical-only composition collapses ( R=0.358 ).
Figure 5: Null baselines for the realization score. Constant path at the normalization level, straight-line interpolation between the first and last target level, a random walk matched to the target’s volatility, and the no-reference and reference arms, all scored with the paper’s Q (mean over the five true targets). Dots are the five per-asset values; the whisker on the random-walk bar is the 5th–95th percentile over 5,000 seeded paths (averaged over assets), not an error bar of the mean. The reference arm under news is only 0.010 above the linear-interpolation null.
News
TSR
FSR
Δref
QT
QF
Δno
Off
0.9180
0.8839
+0.0341
0.6115
0.4688
+0.1427
On
0.8307
0.7820
+0.0487
0.6473
0.4533
+0.1940
Appendix
Table 17: News experiment. Both conditions include the fundamental data and therefore lie in the information-injection regime: TSR, FSR, and Δref are computed under the public reference, and Δref=TSR−FSR is the interpretable quantity. The last three columns report an independent no-reference baseline that is outside the two-regime framework: the same no-reference output is scored against the true target ( QT ) and the flipped target ( QF ), and Δno=QT−QF .
Figure 6: Score sensitivity. R and B on the 70/30, 3:7, news-off cell and Δ on the news experiment, under the five definitions of the realization score in Table 14 . The news increment changes sign under shape-on-returns and level-only.
An LLM agent shown a professional-looking market panel commits to a directional call on a provably unpredictable question far more often than one asked the bare question: across 12 frontier models, commitment rises from 6.5% to 54.0% as evidence is escalated. It commits just as readily when every number on the panel is invented: fabricating the entire display, so nothing the model can see is true except the question itself, still lifts commitment from 24.5% to 36.8%, statistically indistinguishable from the 37.6% produced by genuine market data. What unlocks confident action is not information but the authority of its packaging. The failure is narrow and locatable. Incapacity is not the answer: on matched answerable questions attached to the same panels, the same models answer essentially always, at near-perfect accuracy. Nor is it belief - stated probabilities barely move across the gradient that swings action by 48 points, and score worse than a climatological baseline. Missing judgment isn't it either: asked to classify a question's knowability before acting, models call it irreducible 90% of the time and then commit on just 0.4% of those. The act/don't-act gate is what fails, and the effect is concentrated in a few models rather than universal. Because the gate is separable, it can be trained. Supervised fine-tuning of a 3B model on 540 synthetic cases, predominantly dice, coins, jars and timers, drives commitment to 0.0% on the original cases and transfers to three unseen domains. It does not survive everything: the gate holds exactly when the response format leaves room to reason, and rigid formats that remove that room leave the model confident and wrong on questions it otherwise answers correctly. The gate is trainable and context-fragile, and deployment needs both halves of that sentence.
Simulating societies of many large language model (LLM) agents is expensive, yet the questions asked of such simulations are usually macroscopic: phase behaviour, stylised facts, and scaling with the number of agents N, not the cognition of any single agent. We turn a statistical-physics observation into a method: replace each LLM agent by a low-parameter model fitted from a few hundred to a few thousand cheap queries, then run the society at any N on a laptop. Whether this works is decided before the simulation runs, chiefly by what each agent perceives. We introduce an [interaction order x memory] taxonomy that maps perception and memory to an effective theory and a predicted N-trend of the surrogate error. We validate it on a faithful reimplementation of the LLM macroeconomy EconAgent and seven further named LLM simulations, with agent decisions cloned from genuine LLM elicitations (primarily DeepSeek) for a few dollars; the predicted error trends hold cell by cell, and the two refuted predictions, both on a strongly saturating response and traced to its curvature, are themselves matched quantitatively by the theory with no free parameters.
Evaluating agents by outcomes alone can obscure the capabilities that produce them. This problem is especially pronounced in evolving environments, where outcomes reflect a closed-loop interaction between agent behavior and changing external conditions. We introduce LiveMACEBench, a process-aware benchmark that uses live financial markets as a naturally evolving testbed for persistent LLM agents. Five frontier LLMs operate along continuous trajectories under matched Tool Use, Persistent Memory, Rule Following, and Multi-Agent Collaboration configurations. We evaluate them through both realized outcomes and mechanism-specific diagnostics derived from complete decision traces. Across 30 days of live evaluation, we find a pronounced outcome-capability gap: realized returns often diverge from capability-specific measurements, and similar outcomes can arise from markedly different patterns of mechanism use. Trace-level diagnostics further expose distinct bottlenecks across capabilities, demonstrating that mechanism access, effective mechanism use, and downstream performance are not interchangeable measures of agent capability. LiveMACEBench makes this distinction measurable, turning live markets from a performance leaderboard into a diagnostic environment for agent capability
Jun Zhao, Leiming Fu, Yanbo Wen +9
National University of Singapore · Fudan University · The University of Sydney