World Agent: Can Language Models Keep a World Running?

Weixing Chen; Weipeng Zhang; Nan An; Yang Liu; Liang Lin

cs.AISep 26, 2026

World Agent: Can Language Models Keep a World Running?

Authors: Weixing Chen, Weipeng Zhang, Nan An, Yang Liu, Liang Lin

Organizations: Sun Yat-sen University · South China University of Technology · Guangdong Key Laboratory of Big Data Analysis and Processing · X-Era AI Lab · Peng Cheng Laboratory

Abstract

World models are moving from generating realistic frames to generating playable worlds, yet whether a delivered world can keep running is not tested anywhere. Existing evaluations stop at generation, at delivery, or at single-step transitions, and each stops at a different point along the way. Correct local state transitions or intermediate outcomes do not guarantee a correctly organized causal event flow. We propose the world agent task, which moves the evaluation point of world generation from the moment of delivery to the continued operation that follows. In this task, a model is not asked to generate a world. It is held responsible for keeping the world running, which requires coordinating events and carrying forward their consequences to constrain subsequent evolution. We instantiate the task in WorldAgent-Benchmark with two complementary tracks. In the maintenance track, the model must ground the events of a continuous narrative into correct transitions of the explicit world state while respecting causal, temporal, and concurrency constraints. In the deduction track, the model must predict how the world will evolve under partial observations and act toward a goal. The maintenance track combines LLM-assisted semantic judgments with programmatic validation and scoring, while the deduction track is evaluated entirely programmatically. Individual judgments are auditable against world states and execution logs, and scores can be recomputed from the saved judgments and execution records. Across 8 models, scores decline steadily as pre-built structure is removed from the world, and causal-relation checking is the weakest component for every model. The benchmark makes the continued operation of a world measurable and distinguishes local completion from failures in event organization. Code and dataset will be released on https://github.com/HCPLab-SYSU/WorldAgent-Benchmark.

Figures & tables

Figure 1: Generative world models and one-shot delivered applications both suffer persistent-state failures that remain unmeasured. The world agent task holds a model responsible for keeping an explicit world running through a shared tool interface, with a maintenance track that grounds events into state updates and a deduction track that predicts evolution from partial observations.

Figure 2: The WorldAgent-Benchmark pipeline. Ground truth for both tracks is authored and frozen before any run, each track runs on the same runtime under one of two task conditions, and scoring reads only frozen artifacts. The maintenance track (left) is scored against per-round contracts by a language-model judge and deterministic programs. The deduction track (right) is scored field by field against the engine’s exact evolution, with no language-model judge. Badges at the bottom state the three design commitments.

	Maintenance		Deduction
Model	Compl.	Rel.	M1	M2	Lookahead	Counterfactual
GLM 5.3 Flash	0.752	0.574	0.392	0.494	0.143	0.323
Qwen-3.8 Flash	0.735	0.627	0.315	0.414	0.145	0.423
DeepSeek v4.1 Flash	0.744	0.580	0.395	0.522	0.159	0.466
DeepSeek v4.1 Flash †	0.747	0.584	0.431	0.550	0.151	0.458
GPT 5.6 Luna	0.734	0.543	0.182	0.352	0.156	0.405

Table 1: Main results on both tracks. The left panel shows the mean per-run score of the nine-case subset evaluated by all 8 maintenance models, with 27 runs per model. The right panel shows field-level F1 averaged over the open and naming conditions, with counterfactual modes merged. Bold entries mark the best result in each column within the upper block. † denote the model working on deepseek harness, while the other is working on PhyAgentOS.

Figure 3: Component strengths on the maintenance track . Per-component pass rates are averaged over the three difficulty tiers and normalized by the cross-model mean, with the dashed ring marking

1.0

. Per-model radars are given in Appendix G .

Figure 4: Maintenance track. Panel (a) shows per-round completion (solid lines) and relation scores (dashed lines) broken down by difficulty tier. Panel (b) reports component pass rates, where in each component the three markers denote the easy (circle), hard (triangle), and extreme (square) tiers, and short black ticks mark the cross-model mean within each tier.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Tool	Function
help	return the public protocol, data schemas, and construction rules
get_context	return the cumulative narrative, run coordinates, and remaining budget
read_data	read-only paginated queries over the state and the log
construct	create, update, or delete one type, entity, or capability
commit_step	submit one or more capability triggers as one atomic step
end_round	close the current round and freeze its snapshot

Appendix

Table 3: The six world-interface tools exposed to the maintenance agent. Construction permissions depend on difficulty tier (Section 4.2 ).

Item		What it checks	Settled by
$E$	event	a contracted event happened, bound to the accepted triggers that realized it	judge + audit
$A$	structure	a required structure was prepared and satisfied its availability and use constraints	judge + audit
$P$	probe	nothing forbidden occurred while the precondition held	judge + audit
$CA$	causal	all authored causes and the effect occurred, with each cause ending before the effect begins	program
$TM$	temporal	one event must precede another, immediately or not	program
$CC$	concurrency	two authored event intervals start together, overlap, or end together	program

Appendix

Table 4: The six contract item types of the maintenance track. Semantic items are judged item by item by an isolated language-model judge. Authored event relations and both scores are computed by programs from validated semantic bindings. A structure requirement enters the completion score only in tiers that assign its construction to the agent, and type-level requirements are audited but never scored.

	Easy						Hard						Extreme
Case	DS		QF		GF		DS		QF		GF		DS		QF		GF
	C	R	C	R	C	R	C	R	C	R	C	R	C	R	C	R	C	R
Ark_001	0.99	0.89	0.99	0.68	0.85	0.44	0.67	0.30	0.74	0.75	0.04 †	0.00	0.49	0.13	0.62	0.36	0.51	0.16
Ark_002	0.93	0.93	0.75	0.57	0.90	0.88	0.82	0.75	0.85	0.86	0.85	0.75	0.83	0.82	0.94	0.92	0.81	0.75
Ark_003	0.91	0.85	0.94	0.96	0.95	0.96	0.71	0.68	0.87	0.94	0.90	0.86	0.74	0.74	0.79	0.87	0.83	0.72
Ark_004	0.85	0.70	1.00	1.00	0.93	0.92	0.74	0.56	0.51	0.30	0.48	0.24	0.84	0.74	0.87	0.83	0.88	0.84

Appendix

Table 5: Per-case completion (C) and relation (R) scores on the full maintenance grid, 93 runs per model. DS = DeepSeek v4.1 Flash, QF = Qwen-3.8 Flash, GF = GLM 5.3 Flash. † marks the single full-crash run discussed in the text.

Case family	Mode	DeepSeek v4.1 Flash	Qwen-3.8 Flash	GLM 5.3 Flash
Ashspring	M1	1.00/0.20	0.67/0.04	0.88/0.16
Ashspring	M2	0.67/0.57	0.44/0.41	0.75/0.48
Orbit	M1	0.96/0.08	0.62/0.09	0.92/0.07
Orbit	M2	0.80/0.57	0.79/0.47	0.78/0.68
Manor	M1	0.66/0.03	0.46/0.05	0.79/0.04
Manor	M2	0.30/0.26	0.28/0.14	0.27/0.17

Appendix

Table 6: Field-level F1 per case family on the deduction track, reported as open/naming, five-fold episode means. M1 rows report pure-deduction segment F1; M2 rows report closed-loop action-interval F1. Each model ran its own scenario variant of every family (DeepSeek v4.1 Flash the alternate variants such as Ashspring-chip, the other two models the base variants); variants are merged within a family.

Stratum	Case	Checkpoints	Spread	Consistency
Small	Myth_004	725	0.365	2/3
Small	Ark_006	583	0.288	2/3
Small	Ark_002	472	0.167	3/3
Medium	Li_009	1325	0.299	3/3
Medium	Li_002	1195	0.136	2/3
Medium	Myth_002	1097	0.143	2/3

Appendix

Table 7: The nine selected cases. Checkpoints count the scored contract items of a case over all three tiers (E, A, P, CA, TM, CC). Spread is the mean over tiers of the cross-model range of the mix score (equal-weight mean of completion and relation) among the three reference models. Consistency counts how many of a case’s three tiers reproduce the reference models’ difficulty-level ordering (out of 3).

Figure 6: Per-model component radars on the maintenance track (nine-case common grid). Each panel shows one model’s per-component pass rate, averaged over the three difficulty tiers and normalized by the cross-model mean; the dashed ring marks the mean (1.0). Bold labels and filled markers denote the model’s three strongest components.

Explore similar work

Sep 21, 2026

World State Generator

Language agents solve complex tasks through plans and actions. A single step the world refuses puts the goal out of reach, and what the agent does next decides the task. Prompted planners fail at exactly this point, rewriting the refused step in new words, meeting the same refusal, and burning the attempt budget without moving. They fail because the plan was never tied to the world, so a refusal has nothing in the plan to attach to. A world is where a task runs, and it has its own rules, its own admissible actions, and its own constraints. We build synthetic worlds across 7 domains and extract training data from them. A program enforces each world's rules and grades its goal, and every world is admitted only if its goal is reachable from its initial state. Agents run inside and leave verified failures paired with repairs that carried the run to a state the world certified, a record of about 226K trajectories. On this record we train the World State Generator, a model that writes a plan as checkable states of the world and keeps that plan aligned with the world it runs in. That alignment is what a plan written in language lacks, since the world it runs in has physical limits, logical dependencies, and required orders the language never states, and the plan encounters these rules only when a state fails. WSG takes that failure as the rule the world has stated and rewrites the remaining states to obey it, so the plan bends to the world as the run goes on. Across 7 public benchmarks, WSG raises end-to-end success for two open models near 30B parameters over prompting and brings to the level of proprietary model.

Sungheon Jeong, Sanggeon Yun, Ryozo Masukawa +3

University of California, Irvine · Northeastern University

LLM Agent Planning LLM Planning World Model-Based Planning

Ask PDF

Sep 21, 2026

HappyWorld-Bench

Evaluating world models requires assessing both the quality of the worlds they generate and their consistency and responsiveness under exploration, interaction, and modification. We introduce HappyWorld-Bench, a comprehensive benchmark that evaluates whether generated worlds remain reliable as agents interact with them. Our design is built on a hierarchical capability framework of six world capabilities (W1-W6), from generative construction to unified world modeling, instantiated across three independent evaluation tracks: video world models, spatial world models, and embodied world models. HappyWorld-Bench comprises 1,138 video prompts, 300 spatial scenes, and 254 embodied test cases. Across all three tracks, we build and operate HappyWorld-Arena to organize human A/B comparisons and derive model-level Elo ratings, which complement newly designed automated metrics that capture behavioral correctness. We evaluate 14 video world models, 9 spatial systems, and 8 embodied candidates under this unified framework. Results reveal remaining reliability gaps across all three tracks: video models exhibit reduced consistency during extended rollouts and revisits, spatial models achieve at best 70.14% placement accuracy and 73.33% edit execution, and embodied models struggle to preserve state across multi-step actions and respond precisely to altered action conditions and physical rules. These findings highlight the need to evaluate world models not only by visual quality, but also by state consistency and the correctness of their responses to actions and interventions.

Zhiqi Bai, Junai Cai, Yixin Chen +33

Alibaba Token Hub, Alibaba Group · State Key Laboratory of General Artificial Intelligence, BIGAI · Tsinghua University +2

World Models World Model Evaluation Interactive World Models Embodied World Models Video World Models

Ask PDF

Aug 17, 2026

HarnessEval-W: Agentifying the Evaluation of Visual Worlds

A benchmark should deliver more than a scalar score: what makes an evaluation trustworthy is the reasoning that justifies the score. This is especially critical for world models, where judging a rollout requires understanding whether physics, causality, and world state evolve correctly. Humans spot such violations naturally, yet no existing benchmark automates this capability: metrics are computed brute-force, leaving no reasoning chain that can be examined or verified. We introduce HarnessEval-W, an agentified evaluation pipeline that brings the harness paradigm from the LLM ecosystem to world model benchmarking. Rather than applying a fixed rubric, HarnessEval-W interprets the context of each evaluation case, decomposes the evaluation question into measurable subproblems, and spawns specialized sub-agents, each equipped with tailored context and diagnostic tools to reason over its own subproblem. The parent agent then validates the gathered evidence and summarizes it into the final verdict. This hierarchical workflow turns every evaluation into a transparent evidence tree whose complete reasoning chain justifies the result. We apply HarnessEval-W to 18 representative world models over 330 evaluation cases. Its judgments closely align with human preferences while providing verifiable, fine-grained diagnoses of every generated rollout. We open-source the full pipeline as a live benchmark and invite the broad community to contribute to grow new skills and evaluation cases as world models evolve.

Weiliang Chen, Haowen Sun, Jun Gao +40

Agent Evaluation Automated Evaluation World Model Evaluation

Ask PDF