CivWorldBench: Evaluating Language World Models in Structured Multi-Agent Sandbox Worlds

Ang Li; Haofei Yu; Zhengwei Tao; Liang Chen; Kuan Li; Miaomiao Li; Chenran Cai; Bin Liang

cs.CLOct 12, 2026

CivWorldBench: Evaluating Language World Models in Structured Multi-Agent Sandbox Worlds

Authors: Ang Li, Haofei Yu, Zhengwei Tao, Liang Chen, Kuan Li, Miaomiao Li, Chenran Cai, Bin Liang,

Organizations: The Chinese University of Hong Kong · University of Illinois at Urbana-Champaign · UniPat AI · Peking University ·

Abstract

The growing use of language models for forecasting motivates a broader evaluation of their world-model capabilities. Such evaluation must go beyond predictive accuracy to assess hidden-state inference and intervention prediction. Real-world forecasting benchmarks offer limited access to hidden-state ground truth and controlled experiments. We introduce CivWorldBench, a benchmark built on a multi-agent strategy game engine with recorded internal states, controlled observations, and paired intervention rollouts. Its 1,900 decision points from 475 worlds support three task levels: association forecasting, belief inference, and intervention prediction. Reference predictors support attributable evaluation of capability gaps, with an expert harness and empirical ceiling estimates contextualizing scores. Stored trajectories make intermediate predictions verifiable, while agent-play experiments test whether world-model capabilities support decision-making. The main panel comprises eleven frontier language models. Composite benchmark scores correlate with real-world forecasting leaderboards on matched model panels, more closely than with a general-capability index, supporting its use as a proxy for real-world forecasting capability. Strong association forecasts do not consistently extend to belief inference or intervention prediction, and effect magnitudes remain poorly calibrated despite informative direction predictions. Process verification reveals errors missed by final-answer evaluation, and structured reasoning improves intervention prediction for the strongest model in the process study. Agent-play experiments identify useful assisted plans and motivate evaluating decision-relevant capabilities beyond association forecasting. These findings identify hidden-state inference, intermediate-state prediction, and effect calibration as priorities for world-model training.

Figures & tables

Figure 1: Overview of CivWorldBench . Textual observation histories support association forecasting, belief inference, and intervention prediction. Predictions are evaluated against recorded futures, hidden engine states, and paired engine rollouts, respectively.

Set	Class	Worlds	Points	Agents	Map	Turns
Basic	small	75	300	3	$24{\times}32$	400
	medium	75	300	6	$32{\times}64$	800
	large	75	300	8	$48{\times}80$	1200
Gen.	joint	250	1,000	3–10	$24{\times}32$ – $64{\times}85$	400–1200

Table 1: Corpus composition across 475 worlds and 1,900 decision points, with 561,301 per-turn state records and 7,104 collected arms.

Level	Task	Prediction target	Ground truth	Metric (reference $\to$ ceiling)	Prompts
L1	Quantity	Five quantiles for 11 quantities at two horizons	Mainline	CRPS skill over trend ( $0\to+0.68$ )	3,600
L1	Event	Probabilities for 11 event templates: ten binary with supplied base rates, one leader	Mainline	Brier gain over reference ( $0\to+0.11$ )	3,600
L2	Trait	Rank civilizations on four traits	Recorded $\theta_{w}$	Adjusted pairwise accuracy ( $0\to 0.66$ )	3,360
L2	Rules	Infer three hidden world rules from observable evidence	Recorded $\theta_{w}$	Adjusted accuracy ( $0\to 0.93$ )	3,360
L3	Intervene	Quantity forecasts for 11 edits and matched placebos	Paired rollouts	Capture, adjusted direction ( $0\to 0.92$ ) , specificity	7,939

Table 2: Task definitions and evaluation metrics. Prompt counts include all settings and ablation conditions. Reference ceilings are for Omni-ID .

			L1 association		L2 belief		L3 intervention
Model	AA	SWFI	Quantity $\uparrow$	Event $\uparrow$	Trait $\uparrow$	Rules $\uparrow$	Capture $\uparrow$	Direction $\uparrow$	Specificity $\uparrow$
Floor	—	0.00	+0.00	+0.000	0.00	0.00	$-$ 1.00	0.00	+0.00

gpt-oss-120b	12	$-$ 3.89	+0.03 (4%)	$-$ 0.031 (-29%)	0.01 (2%)	0.04 (5%)	$-$ 0.65 (25%)	0.15 (16%)	+0.05 (5%)
Gemma 4 31B	15	16.52	+0.21 (31%)	+0.012 (11%)	0.08 (12%)	0.11 (12%)	$-$ 0.61 (28%)	0.30 (33%)	+0.19 (19%)
Nemotron 3 Ultra	23	$-$ 0.98	+0.08 (12%)	+0.003 (3%)	0.08 (11%)	0.12 (13%)	$-$ 0.63 (27%)	$-$ 0.16 (-17%)	$-$ 0.33 (-33%)

Table 3: Model performance on CivWorldBench : three-setting SWFI and Omni-ID diagnostics. Bold denotes the best model score, excluding the expert reference. Percentages indicate reference-normalized completion. Full three-setting results are reported in Table 14 .

Figure 4: Direction accuracy and effect magnitude for nine models in Omni-ID . Section H.4 adds the rest.

Figure 5: Reasoning protocols and prediction quality. (a) Above-reference answers, with hatching for intermediate forecasts at or below reference. (b) L3 process skill and Capture completion, averaged over Table and Code. (c) Capture completion by protocol. Scores average Omni-ID and Omni-OOD equally, and Appendix I details scoring and coverage.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Patch	Modification
Trait initialization	Corrects trait arithmetic and initializes diplomatic contact at peace.
Diplomatic thresholds	Revises ceasefire, peace, and war decision thresholds.
Alliance dynamics	Reduces persistent alliance-bloc warfare.
Aggression response	Connects aggression to ceasefire acceptance and war persistence.
Production preference	Connects building preference to the active production decision.

Appendix

Table 5: Source patches used to construct the benchmark engine.

Panel of Figure 7 — Figure 7: Four branches from a shared checkpoint provide replay checks and intervention controls.

Intervention	Scored quantities	Band
Treasury change	Treasury	5–30
Research points	Technologies learned	5–60
City destruction	Total city size, population	5–150
Government-enabling technology	Score increment	5–60
Map reveal	City count, score increment	30–300
Aggression trait	Enemy count, military units	50–300

Appendix

Table 6: Benchmark intervention families and numerical scoring targets. Bands are in turns.

Line	Trait accuracy	Rule accuracy	Direction accuracy
Raw reference	0.50 / 0.50 / 0.50	0.40 / 0.40 / 0.41	0.50 / 0.50 / 0.50
gpt-oss-120b	0.50 / 0.49 / 0.52	0.43 / 0.44 / 0.46	0.57 / 0.56 / 0.54
Gemma 4 31B	0.52 / 0.51 / 0.54	0.47 / 0.46 / 0.46	0.64 / 0.63 / 0.59
Nemotron 3 Ultra	0.52 / 0.50 / 0.51	0.47 / 0.46 / 0.40	0.43 / 0.42 / 0.41
Inkling	0.53 / 0.51 / 0.55	0.47 / 0.48 / 0.44	0.54 / 0.56 / 0.48
Qwen3.8-27B	0.52 / 0.51 / 0.54	0.34 / 0.30 / 0.29	0.61 / 0.61 / 0.57

Appendix

Table 7: Underlying accuracies before the main table’s reference adjustment. Triples are Omni-ID / Ego / Omni-OOD . Raw references are 0.50 for trait and direction, and the equal-parameter majority accuracy for rules.

Check	Observation	Interpretation
Repeated futures	Quantity $\simeq+0.68$ , Event $\simeq+0.11$	Predictable signal remains under engine randomness with full-checkpoint information.
Trait observer	0.59 on all pairs, 0.63 at gap $\geq$ 40, 0.66 at gap $\geq$ 60	Privileged behavioral features provide partial recovery of trait orderings.
Rule observer	0.93 / 0.86 / 0.90	Formula inversion and disaster statistics support parameter-specific identifiability.
Direction reference	all moved cells 0.71 / 0.71 / 0.68	The displayed 0.92 concerns only effects above estimated placebo divergence.
Capture reference	+0.39 / +0.39 / +0.30	A reference derived under a Gaussian noise decomposition.
Specificity endpoint	1.00	An algebraic endpoint for real-only responses.

Appendix

Table 8: Validation evidence and reference assumptions. Triples denote Omni-ID / Ego / Omni-OOD . Numerical references differ in information and estimation procedure.

External reference	$n$	Pearson $r$	Spearman $\rho$
ForecastBench Tournament (Overall)	4	0.864–0.879	0.800
ForecastBench Preliminary (Dataset)	5	0.930	0.900
Metaculus FutureEval (Avg Score)	5	0.954	0.975
Prophet Fixed Context (SKILL)	6	0.786	0.886
AA Intelligence Index v4.3 (reference)	11	0.756	0.795

Appendix

Table 9: Associations between fixed-weight SWFI and external model scores.

n

counts matched models. Ranges span eligible external configurations. The two ForecastBench tracks overlap. AA provides a general-capability reference. Section G.3 describes the protocols and sensitivity analyses.

Model	SWFI	FB T	FB P	FE	SKILL	AA
GPT-6 Astra	36.59	–	–	–	-0.98	–
Claude Opus 5	29.47	66.10–66.30	61.70	15.78	–	45
Gemini 3.7 Flash	28.75	–	62.00	–	-0.91	–
GPT-5.6 Sol	27.68	65.40–65.50	61.50	12.93	-1.05	39
GLM-5.3 Flash	22.57	–	–	–	–	–
Kimi K3	21.94	65.60	60.90	12.93	-1.14	44

Appendix

Table 10: SWFI and external scores for exact model matches. T/P denotes ForecastBench Tournament/Preliminary and FE denotes FutureEval. SKILL uses Prophet’s Fixed Context track. Dashes indicate unavailable matches. External scores retain their native units.

External reference	Drop-one $r$	Permutation $p$
ForecastBench Tournament (Overall)	0.455–1.000	0.0833–0.1250
ForecastBench Preliminary (Dataset)	0.917–0.994	0.0250
Metaculus FutureEval (Avg Score)	0.937–0.987	0.0167
Prophet Fixed Context (SKILL)	0.673–0.913	0.0667

Appendix

Table 11: External-validity sensitivity. Drop-one ranges omit each model in turn and include all eligible configurations. Exact two-sided permutation tests are exploratory and unadjusted.

External reference	$n$	SWFI $r$	AA $r$
ForecastBench Tournament (Overall)	4	0.864–0.879	0.991–0.994
ForecastBench Preliminary (Dataset)	4	0.917	0.964
Metaculus FutureEval (Avg Score)	5	0.954	0.987
Prophet Fixed Context (SKILL)	3	0.510	-0.645

Appendix

Table 12: Same-model comparison with AA: each predictor is correlated with the external outcome. This differs from the SWFI–AA association in Figure 3 . Only panels with at least three recorded AA values are included.

Metric	Omni-ID	Ego	Omni-OOD
Quantity	$+0.84$	$+0.67$	$+0.92$
Event	$+0.75$	$+0.86$	$+0.77$
Trait	$+0.07$	$-0.03$	$+0.11$
Rules	$+0.15$	$+0.31$	$+0.23$
Capture	$+0.76$	$+0.75$	$+0.58$
Direction	$+0.34$	$+0.39$	$+0.34$

Appendix

Table 13: Spearman rank correlations between the external capability index and the unrounded main-table scores across 11 unassisted model lines in each setting. All seven metrics are included; tied values receive average ranks. The assisted Expert line is excluded. These are descriptive model-level associations, not matched-anchor comparisons or significance tests.

		L1 association		L2 belief		L3 intervention
Line	AA Index	Quantity $\uparrow$	Event $\uparrow$	Trait $\uparrow$	Rules $\uparrow$	Capture $\uparrow$	Direction $\uparrow$	Specificity $\uparrow$
Omni-ID
Floor	—	+0.00	+0.000	0.00	0.00	$-$ 1.00	0.00	+0.00

gpt-oss-120b	12	+0.03 (4%)	$-$ 0.031 (-29%)	0.01 (2%)	0.04 (5%)	$-$ 0.65 (25%)	0.15 (16%)	+0.05 (5%)
Gemma 4 31B	15	+0.21 (31%)	+0.012 (11%)	0.08 (12%)	0.11 (12%)	$-$ 0.61 (28%)	0.30 (33%)	+0.19 (19%)

Appendix

Table 14: Complete results across three settings. Bold denotes the best model score, excluding the expert reference. Percentages report reference-normalized completion. Missing or unparseable answers receive the failure penalties in Appendix F . Section G.2 specifies the numerical reference assumptions.

Figure 8: Quantity-family skill across three settings. All panels share a color scale centered on the trend reference at zero. Variables are equally weighted within each family.

Model	Aggressive	Expansionist	Trader	Builder
gpt-oss-120b	0.51 / 0.49 / 0.52	0.51 / 0.51 / 0.56	0.49 / 0.49 / 0.49	0.50 / 0.49 / 0.49
Gemma 4 31B	0.52 / 0.51 / 0.54	0.53 / 0.53 / 0.57	0.52 / 0.51 / 0.52	0.52 / 0.51 / 0.52
Nemotron 3 Ultra	0.51 / 0.48 / 0.50	0.52 / 0.52 / 0.55	0.53 / 0.50 / 0.52	0.51 / 0.49 / 0.48
Inkling	0.53 / 0.50 / 0.55	0.54 / 0.53 / 0.58	0.54 / 0.50 / 0.54	0.52 / 0.50 / 0.52
Qwen3.8-27B	0.52 / 0.50 / 0.53	0.53 / 0.53 / 0.57	0.52 / 0.50 / 0.52	0.53 / 0.51 / 0.52
GPT-5.6 Sol	0.54 / 0.52 / 0.54	0.55 / 0.49 / 0.61	0.59 / 0.54 / 0.58	0.55 / 0.53 / 0.56

Appendix

Table 15: Pairwise ranking accuracy by personality trait. Triples are Omni-ID / Ego / Omni-OOD, using the main-table samples and failure penalties.

Model	Disaster frequency	Growth cost	Research cost
gpt-oss-120b	0.63 / 0.60 / 0.61	0.34 / 0.35 / 0.37	0.32 / 0.36 / 0.41
Gemma 4 31B	0.65 / 0.64 / 0.62	0.39 / 0.38 / 0.34	0.36 / 0.37 / 0.41
Nemotron 3 Ultra	0.63 / 0.62 / 0.62	0.39 / 0.36 / 0.30	0.40 / 0.41 / 0.28
Inkling	0.65 / 0.64 / 0.62	0.36 / 0.39 / 0.35	0.39 / 0.40 / 0.34
Qwen3.8-27B	0.42 / 0.38 / 0.40	0.29 / 0.25 / 0.24	0.31 / 0.27 / 0.21
GPT-5.6 Sol	0.73 / 0.66 / 0.66	0.32 / 0.37 / 0.44	0.43 / 0.43 / 0.42

Appendix

Table 16: Classification accuracy by world rule. Triples are Omni-ID / Ego / Omni-OOD, using the main-table samples and failure penalties.

Figure 9: Selected model profiles across three settings. Axes show reference-normalized scores clipped to

[0,1]

for display. The intervention axis uses direction, and radar area is not an aggregate score.

Figure 10: Event-template Brier skill ratios in Omni-ID and Omni-OOD , with leave-one-out rerun references in black. This diagnostic uses

1-\sum\mathrm{Brier}/\sum\mathrm{Brier}_{\rm ref}

, not the headline Brier gain.

Condition	Task	Worlds	Cells	Base	Added/ablated	Change	95% interval
One frame	Quantity	60	27251	+0.269	+0.137	-0.132	[-0.163, -0.103]
Future narrative	Quantity	60	23085	+0.285	+0.346	+0.061	[+0.031, +0.091]
Rules primer	Quantity	40	18027	+0.266	+0.247	-0.019	[-0.050, +0.011]
Study notes	Quantity	60	26408	+0.268	+0.354	+0.086	[+0.061, +0.111]
Thinking off	Quantity	60	27564	+0.270	+0.217	-0.053	[-0.078, -0.029]
Study notes	Event	60	22018	-0.034	-0.015	+0.019	[+0.014, +0.025]

Appendix

Table 17: Paired information ablations of Flash-Next in Omni-ID. Each row uses its own common valid cells; intervals are paired world bootstraps. Event comparisons here use a no-base-rate baseline.

Arm	Cells	Worlds	Ops	Base	Notes	Change	95% interval
real	433	60	11	-0.514	-0.277	+0.237	[+0.070, +0.343]
placebo	341	58	11	-0.585	-0.335	+0.250	[+0.053, +0.357]

Appendix

Table 18: Study notes: paired CAPTURE on each arm’s common valid moved cells in Omni-ID. Both arm and no-intervention predictions are note-conditioned. Intervals use paired world Bayesian bootstrap weights.

Model	Counts	Direction	$<0.5\times$	$0.5$ – $2\times$	$>2\times$	$S_{+}$
gpt-oss-120b	1749/57/105/937	59 [56, 62]	21	45	34 [30, 38]	-0.34 [-0.44, -0.16]
Gemma 4 31B	1759/1/145/1033	64 [61, 67]	18	48	34 [31, 38]	-0.31 [-0.41, -0.20]
Nemotron 3 Ultra	1767/545/93/717	64 [60, 67]	15	52	33 [29, 37]	-0.07 [-0.20, +0.03]
Inkling	1772/169/178/871	61 [58, 65]	27	51	21 [18, 25]	+0.20 [+0.10, +0.28]
Qwen3.8-27B	1764/56/100/1017	63 [60, 66]	23	50	27 [24, 30]	+0.03 [-0.07, +0.13]
GPT-5.6 Sol	176/0/16/103	64 [54, 74]	9	57	34 [25, 44]	-0.07 [-0.30, +0.18]

Appendix

Table 19: Direction-conditioned magnitude diagnostics in omni-ID. Counts give eligible, missing, zero-response, and correct-sign cells. Direction is the percentage correct among valid nonzero responses. Magnitude-bin percentages condition on a correct sign. Brackets give 95% world-bootstrap intervals.

S_{+}

is selected-subset capture, not a replacement main score. An asterisk marks an omitted interval due to a singleton-world operator.

Model	Counts	Direction	$<0.5\times$	$0.5$ – $2\times$	$>2\times$	$S_{+}$
gpt-oss-120b	1753/25/132/917	57 [54, 61]	20	48	32 [28, 36]	-0.33 [-0.44, -0.20]
Gemma 4 31B	1767/2/185/995	63 [60, 66]	24	48	27 [24, 30]	-0.10 [-0.20, +0.02]
Nemotron 3 Ultra	1767/480/104/700	59 [55, 63]	16	47	37 [33, 41]	-0.24 [-0.36, -0.13]
Inkling	1774/135/218/885	62 [59, 66]	27	49	24 [20, 27]	+0.12 [+0.04, +0.23]
Qwen3.8-27B	1770/11/116/1014	62 [59, 65]	22	50	29 [25, 32]	+0.04 [-0.07, +0.16]
GPT-5.6 Sol	176/0/14/107	66 [56, 75]	17	60	23 [15, 35]	+0.11 [-0.10, +0.28]

Appendix

Table 20: Direction-conditioned magnitude diagnostics in ego. Counts give eligible, missing, zero-response, and correct-sign cells. Direction is the percentage correct among valid nonzero responses. Magnitude-bin percentages condition on a correct sign. Brackets give 95% world-bootstrap intervals.

S_{+}

is selected-subset capture, not a replacement main score. An asterisk marks an omitted interval due to a singleton-world operator.

Model	Counts	Direction	$<0.5\times$	$0.5$ – $2\times$	$>2\times$	$S_{+}$
gpt-oss-120b	1781/118/111/903	58 [55, 61]	16	42	42 [38, 46]	-0.42 [-0.52, -0.28]
Gemma 4 31B	1793/2/163/971	60 [57, 62]	16	40	44 [41, 48]	-0.31 [-0.40, -0.24]
Nemotron 3 Ultra	1802/592/66/716	63 [59, 67]	15	41	44 [40, 49]	-0.29 [-0.44, -0.15]
Inkling	1804/228/137/812	56 [53, 59]	21	52	28 [24, 31]	+0.04 [-0.09, +0.16]
Qwen3.8-27B	1802/80/87/991	61 [58, 64]	16	46	39 [35, 42]	-0.13 [-0.22, -0.00]
GPT-5.6 Sol	169/0/12/108	69 [59, 79]	10	46	44 [33, 56]	-0.07 [-0.27, +0.11]

Appendix

Table 21: Direction-conditioned magnitude diagnostics in OOD. Counts give eligible, missing, zero-response, and correct-sign cells. Direction is the percentage correct among valid nonzero responses. Magnitude-bin percentages condition on a correct sign. Brackets give 95% world-bootstrap intervals.

S_{+}

is selected-subset capture, not a replacement main score. An asterisk marks an omitted interval due to a singleton-world operator.

Model	$<1\times$	$>1\times$	$>2\times$	Filtered $>2\times$	Filtered $n_{+}$	Excluded
gpt-oss-120b	38.1	52.7	33.7	27.6	646	25
Gemma 4 31B	38.7	54.1	34.1	25.6	738	15
Nemotron 3 Ultra	33.5	57.3	33.3	25.7	501	7
Inkling	50.9	40.4	21.4	12.5	615	2
Qwen3.8-27B	42.1	49.7	26.8	17.5	718	10
GPT-5.6 Sol	25.2	68.0	34.0	23.5	68	0

Appendix

Table 22: Magnitude sensitivity in Omni-ID . All percentages condition on a correct sign. The sensitivity filter removes effects at or below each operator’s lower quartile before selecting correct-sign predictions. Excluded counts refer instead to valid predictions omitted by the inherited main scoring mask.

Model	Omni-ID	Ego	Omni-OOD
gpt-oss-120b	120.0 [58.3, 300.0]	106.2 [55.7, 300.0]	158.0 [68.4, 455.4]
Gemma 4 31B	122.2 [62.5, 300.0]	100.0 [50.0, 229.7]	172.8 [82.1, 566.6]
Nemotron 3 Ultra	123.3 [73.8, 300.0]	140.0 [68.6, 327.5]	183.3 [83.3, 407.8]
Inkling	96.4 [43.4, 180.0]	100.0 [45.1, 200.0]	105.3 [53.1, 246.1]
Qwen3.8-27B	100.0 [50.0, 225.0]	108.5 [52.1, 244.0]	150.0 [82.0, 349.0]
GPT-5.6 Sol	160.0 [99.7, 272.1]	133.3 [69.8, 200.0]	176.0 [99.0, 406.6]

Appendix

Table 23: Relative effect magnitudes conditional on correct direction. Entries give the median and [25th, 75th] percentiles, all in percentages. The ratio uses absolute predicted and realized effects in the scoring space. 100% denotes a magnitude match. Brackets describe sample dispersion, not confidence intervals.

Line	capture real (placebo)	direction	shift real (placebo)
gpt-oss-120b	$-$ 0.65 ( $-$ 0.74) / $-$ 0.73 ( $-$ 0.77) / $-$ 0.69 ( $-$ 0.87)	0.57 / 0.56 / 0.54	0.56 (0.50) / 0.63 (1.15) / 0.58 (0.60)
Gemma 4 31B	$-$ 0.61 ( $-$ 0.62) / $-$ 0.51 ( $-$ 0.56) / $-$ 0.61 ( $-$ 0.69)	0.64 / 0.63 / 0.59	0.38 (0.30) / 0.29 (0.27) / 0.41 (0.40)
Nemotron 3 Ultra	$-$ 0.63 ( $-$ 0.74) / $-$ 0.68 ( $-$ 0.83) / $-$ 0.80 ( $-$ 0.87)	0.43 / 0.42 / 0.41	0.28 (0.28) / 0.31 (0.29) / 0.28 (0.24)
Inkling	$-$ 0.31 ( $-$ 0.43) / $-$ 0.31 ( $-$ 0.44) / $-$ 0.42 ( $-$ 0.55)	0.54 / 0.56 / 0.48	0.15 (0.15) / 0.18 (0.17) / 0.14 (0.14)
Qwen3.8-27B	$-$ 0.34 ( $-$ 0.50) / $-$ 0.33 ( $-$ 0.47) / $-$ 0.41 ( $-$ 0.66)	0.61 / 0.61 / 0.57	0.20 (0.20) / 0.27 (0.26) / 0.18 (0.17)
GPT-5.6 Sol	$-$ 0.60 ( $-$ 0.70) / $-$ 0.27 ( $-$ 0.60) / $-$ 0.24 ( $-$ 0.50)	0.63 / 0.65 / 0.69	0.17 (0.17) / 0.17 (0.15) / 0.15 (0.12)

Appendix

Table 24: Intervention diagnostics across three settings. Capture and shift are reported for real arms with matched placebos in parentheses. Shift is the mean absolute change from the model’s no-intervention forecast in the normalized units of Section 4.1 . Specificity applies the missing-response penalty to the clipped shift ratio defined in Equations 4 and F . Triples are Omni-ID / Ego / Omni-OOD .

Model	Setting	Arm	Cells/ops	Base	Card	Change
Flash-Next	omni-ID	real	48/7	-0.581	-0.678	-0.097
Flash-Next	omni-ID	placebo	44/8	-0.436	-0.626	-0.190
Flash-Next	ego	real	52/8	-0.406	-0.231	+0.175
Flash-Next	ego	placebo	44/8	-0.349	-0.237	+0.112
Flash-Next	OOD	real	39/9	-0.482	-0.592	-0.110
Flash-Next	OOD	placebo	15/4	-0.251	-0.372	-0.121

Appendix

Table 25: Paired Capture with mechanism cards and a shared no-intervention prediction. Counts denote cells/operators. Real and placebo arms use separate moved-cell sets.

Model	Anchors	Outputs F/T/C	State missing	Effect missing	Runnable
Qwen3.8-27B	190	1537/1537/1034	200/4272	142/2320	1034/1034
Gemma 4 31B	190	1537/1537/1034	205/4272	157/2320	1033/1034
Flash-Next	190	1537/1537/1034	205/4272	159/2320	1033/1034
Kimi K3	190	1537/1537/1034	238/4272	180/2320	1033/1034
GPT-5.6 Sol	30	269/269/169	48/728	29/400	169/169

Appendix

Table 26: Completed output census and diagnostic coverage. F/T/C counts include all executed settings. Missing state/effect units are counted before protocol matching.

Model	Protocol	L1 skill	L3-state skill	L1 A+P- (%)	L3-state A+P- (%)
Qwen3.8-27B	Table	$-0.027$	$+0.067$	10.55	5.66
Qwen3.8-27B	Code	$-0.006$	$-0.049$	5.84	4.82
Gemma 4 31B	Table	$-0.058$	$-0.036$	10.01	5.92
Gemma 4 31B	Code	$-0.013$	$-0.050$	7.38	4.49
Flash-Next	Table	$-0.023$	$+0.028$	7.63	9.16
Flash-Next	Code	$-0.065$	$-0.001$	6.49	4.54

Appendix

Table 27: Intermediate-state skill and joint answer-positive/process-negative shares. Percentages use all eligible matched units. L3-state evaluates intervention-branch states.

Model	Protocol	Quantity	Event	Capture
Qwen3.8-27B	Free	$+0.206$	$+0.012$	$-0.408$
Qwen3.8-27B	Table	$+0.171$	$+0.009$	$-0.456$
Qwen3.8-27B	Code	$+0.127$	$-0.087$	$-0.642$
Gemma 4 31B	Free	$+0.153$	$+0.009$	$-0.423$
Gemma 4 31B	Table	$+0.120$	$+0.007$	$-0.443$
Gemma 4 31B	Code	$+0.015$	$-0.129$	$-0.605$

Appendix

Table 28: Final-answer scores averaged equally over Omni-ID and OOD. Each model uses 190 anchors, and GPT-5.6 Sol uses 30. Protocols use the same anchors within each model.

Aggregation	Points	Pearson $r$	Spearman $\rho$
Model–protocol pairs	10	$+0.384$	$+0.539$
Table only	5	$+0.556$	$+0.300$
Code only	5	$+0.308$	$+0.800$
Model means, raw Capture	5	$+0.699$	$+0.800$
Model means, Capture completion	5	$+0.693$	$+0.800$
Model means, L1 process and L1 completion	5	$+0.872$	$+0.800$

Appendix

Table 29: Associations between process and final-answer scores. Rows use L3 state skill and Capture unless marked L1. Model means average Table and Code before computing correlations. Completion follows the frozen main-table normalization. These are descriptive associations.

World model	Mean score	$\Delta$ score	Wins / 76
None	157.49	—	—
Flash-Next	164.00	$+6.51\pm 5.39$	44
Kimi K3	160.80	$+3.32\pm 5.93$	39

Appendix

Table 30: Turn-210 values underlying Figure 6 . Differences are paired against the no-WM planning control.

\pm

denotes standard error across 76 worlds.

WM	First two executions	Next two executions	Four executions
Flash-Next	$+0.39\pm 1.17$	$-0.01\pm 0.78$	$+0.19\pm 0.79$
Kimi K3	$-0.15\pm 0.93$	$-1.17\pm 1.65$	$-0.66\pm 1.01$
Sol	$+1.82\pm 0.99$	$+0.97\pm 1.16$	$+1.40\pm 0.84$
Opus	$+0.96\pm 1.00$	$+1.61\pm 1.15$	$+1.28\pm 0.91$

Appendix

Table 31: Local gain relative to self-review, mean

\pm

SE. The 36-world replication re-executes the same finalized plans. The final column pools four executions per plan. The exploratory 12 worlds are excluded from these columns.

Model	L1 quantity	L2 trait	L3 capture	SWFI	$\Delta$ score
Flash-Next	0.26	0.10	$-0.57$	16.93	$+0.19\pm 0.79$
Kimi K3	0.34	0.00	$-0.35$	21.94	$-0.66\pm 1.01$
Sol	0.27	0.23	$-0.60$	27.68	$+1.40\pm 0.84$
Opus	0.33	0.17	$-0.08$	29.47	$+1.28\pm 0.91$

Appendix

Table 32: Component diagnostics and utility accompanying Table 4 . Diagnostics are Omni-ID scores from Table 3 , while SWFI spans three settings. Utility averages four executions per plan in 36 worlds and is reported relative to self-review, with world-level SE.

Plan	First pair	Second pair	Mean score	Mean cities
Original	106, 83	109, 111	102.25	1.75
Self-review	86, 147	84, 119	109.00	1.75
Flash-assisted	95, 95	117, 108	103.75	2.25
Kimi-assisted	96, 116	117, 106	108.75	1.50
Sol-assisted	118, 115	127, 139	124.75	3.50
Opus-assisted	134, 114	115, 134	124.25	3.75

Appendix

Table 33: Complete six-condition results for Figure 13 after 20 turns. Each pair independently executes the same finalized plan. The last column averages remaining city counts across all four executions.

LLM World Models

Large language model world models predict how environments change so agents can plan, learn, and evaluate actions before execution.

See all 34 papers

ForecastBench-Sim: A Simulated-World Forecasting Benchmark
Jun 17, 2026Jaeho Lee, Nick Merrill, Ezra KargerForecasting Benchmarks Probabilistic Forecasting
1Forecasting Research Institute · 2UC Berkeley
World Agent: Can Language Models Keep a World Running?
Sep 26, 2026Weixing Chen, Weipeng Zhang, Nan An +2World Model Evaluation World Models
Sun Yat-sen University · South China University of Technology · Guangdong Key Laboratory of Big Data Analysis and Processing +2
WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning
Jun 10, 2026Yizhou Chi, Eric Chamoun, Zifeng Ding +1Event Time Prediction LLM Agent Evaluation
Department of Computer Science and Technology University of Cambridge

Jun 17, 2026J/K move · Enter open · S save

ForecastBench-Sim: A Simulated-World Forecasting Benchmark

Jaeho Lee, Nick Merrill, Ezra Karger

1Forecasting Research Institute · 2UC Berkeley

Forecasting Benchmarks Probabilistic Forecasting Counterfactual Prediction Synthetic Benchmark Generation

Open paper AskPDF arXiv

Forecasting benchmarks for general-purpose AI systems usually inherit the constraints of the real world: outcomes resolve slowly, tail events are rare, and counterfactual questions are difficult to score. We introduce ForecastBench-Sim, a simulated-world forecasting benchmark built on game rollouts from Freeciv, a turn-based strategy game modelled on the Civilization series. Forecasters receive a fixed world report (a structured snapshot of the current game state) and answer questions about hidden future states; the benchmark then continues the simulation and scores forecasts. Because the world is simulated, the same setup can generate continuous or binary forecasting questions at arbitrary time horizons, paired intervention worlds for conditional or causal questions, and resolved examples of rare or disruptive outcomes. We describe the benchmark pipeline, question families, scoring protocol, and release artifacts, and report validation slices from model evaluations and an anonymized human pilot. ForecastBench-Sim is intended to complement real-world forecasting benchmarks by providing controlled, immediately resolvable tasks for studying probabilistic reasoning under dynamic world states.

Abstract

LLM World Models

Explore similar work

ForecastBench-Sim: A Simulated-World Forecasting Benchmark

World Agent: Can Language Models Keep a World Running?

WorldReasoner: Evaluating Whether Language Model Agents Forecast Events with Valid Reasoning