From Language Models to World-Acting Systems: Progress and Limits of Agentic AI across Digital, Social, Virtual, and Physical Environments

Linsen Zhu; Mengqing Cai

cs.AISep 4, 2026

From Language Models to World-Acting Systems: Progress and Limits of Agentic AI across Digital, Social, Virtual, and Physical Environments

Authors: Linsen Zhu, Mengqing Cai

Abstract

Large language models become consequential agents when surrounding systems let outputs change external state. Models now call tools, operate interfaces, delegate work, retain state, inhabit generated worlds, and control robots or laboratory equipment. Such advances are often narrated as one march toward autonomy, conflating model competence, system integration, persistence, and safe authority. This critical review synthesizes primary research and official technical specifications available by 31 August 2026. We organize the evidence along delegated authority, temporal persistence, and environmental coupling, while separating model, harness, and environment. Within the evidence examined, action-interface expansion is documented more convincingly than robust completion, recovery, authorization, or independent verification. Model Context Protocol and Agent2Agent improve interoperability but do not establish trustworthy delegation; multi-agent organization adds specialization alongside cost and correlated failure. Persistent simulations and world models support training and planning but do not themselves demonstrate agency; robotics and self-driving laboratories establish bounded feasibility rather than unattended open-world reliability. We propose justified delegation as an analytical and normative heuristic, not an observed law or certified score: expand action scope only where evidence supports provenance, bounded authority, failure detection, safe recovery, and calibrated human control. This framing yields a research agenda for coupled model-harness evaluation, capability-based permissions, durable state, cross-agent accountability, and staged physical validation.

Figures & tables

Source category	Strongest defensible use	Typical blind spot	Representative examples
Peer-reviewed benchmark or systems paper	Performance and failure under the published task, model, harness, budget, and evaluator	Distribution shift, live permissions, changing interfaces, and longitudinal incidents	WebArena, OSWorld, SWE-agent, $\tau$ -bench [ 56 , 50 , 51 , 52 ]
Peer-reviewed controlled physical experiment	Feasibility and measured behaviour in a disclosed apparatus and procedure	Scale, repeated unattended operation, rare hazards, and transfer to other instruments	RT-2, Coscientist, ChemCrow [ 57 , 8 , 9 ]
Preprint or institutional technical report	Timely architecture, released artifact, and provisional evaluation	Peer review, independent replication, stable model endpoints, and selective reporting	Voyager, V-JEPA 2, Gemini Robotics, Magentic-UI [ 47 , 6 , 12 , 24 ]
Open technical specification	Defined roles, messages, state transitions, discovery, and security requirements	Implementation quality, adoption, semantic correctness, and end-task reliability	MCP and A2A [ 22 , 1 ]
First-party preview, system card, or product record	Existence, stated design, first-party tests, deployment constraints, and disclosed failure modes	Independent replication, complete denominators, comparison fairness, and durability	OpenAI CUA, Genie 3, Project Eden, MHS [ 29 , 33 , 46 , 5 ]

Table 1: Evidence categories used in this review and the claims each can support.

Panel of Figure 1 — Figure 1: Agency is a property of a configured model–harness–environment system under authority granted by the delegator $U$ . Verification and recovery are drawn as an expanded function of the harness $H$ , not as a fifth element of $\mathcal{S}$ . An authorized human operator interacts through $H$ but is not equated with $U$ ; affected people and institutions need not have delegated or operated the system and are therefore shown outside its boundary. Their impacts, complaints, and remedies remain inputs to independent verification and recovery.

Interface	Primary relationship	State / continuation	Human-control affordance	What it does not establish
MCP 2026-07-28	AI host or client to tool/resource server	Stateless protocol core; explicit handles; task extension for long work	Multi-round-trip input or confirmation mediated by the client	Tool safety, least privilege, semantic success, or server trustworthiness [ 22 , 23 ]
A2A 1.0	Client agent to remote, internally opaque agent	Stateful task lifecycle, context identifiers, artifacts, streaming and push	input-required , cancellation, authenticated task continuation	Benefit of multiple agents, truthful capability claims, or correctness of completed artifacts [ 1 ]
Human-centred agent UI	Human delegator to one or more software agents	Plans, parallel sessions, checkpoints, and optional memory	Co-planning, co-tasking, action approval, intervention, and final verification	That people notice every hazard, understand approvals, or can supervise at scale [ 24 ]
Physical driver layer	Agent or program to heterogeneous programmable devices	Device state, commands, streamed measurements, and reusable scripts	Hardware limits, allowlists, supervision, and escalation depend on implementation	Physical reasoning, calibrated sensing, general fault recovery, or unattended safety [ 5 ]

Table 2: Different delegation interfaces expose complementary semantics and leave complementary gaps.

Environment	Typical evidence of success	Characteristic hidden state	Required control emphasis	Residual question
Tool/API sandbox	Returned object, state predicate, replayable trace	Service policy and effects outside the schema	Typed arguments, idempotency, allowlists, deterministic evaluators	Did the valid call satisfy the user’s actual intent?
Live web/desktop	Account state, receipt, artifact, independent query	Focus, overlays, remote policy, other users, indirect prompts	Isolated browser, credential scoping, confirmations, transaction reconciliation	What affected-party or account consequence was omitted?
Agent or human delegation	Signed artifact, task record, evidence package, acceptance decision	Contractor competence, incentives, private actions, copied context	Identity, bounded brief, budget, provenance, conflict and dispute mechanisms	Is “completion” truthful, lawful, and independently checkable?
Generated/simulated world	Instrumented state, causal probes, repeatable episode	Model shortcuts and mismatch with the target world	State access for evaluators, counterfactual tests, held-out dynamics, transfer validation	Did the agent learn the task or exploit the simulator?
Robot or laboratory	Independent sensors, calibrated measurement, physical inspection	Wear, contamination, occlusion, people, irreversible change	Interlocks, units, uncertainty, staged trials, emergency stop, expert accountability	Is failure bounded when the model or instrument is wrong?

Table 3: Assurance obligations grow with persistence and environmental coupling. The examples are controls to evaluate, not claims that current systems implement them completely.

Agentic AI

Agentic artificial intelligence studies systems that plan, use tools, learn from feedback, and act toward goals with varying degrees of autonomy.

See all 215 papers

From Traces to Agentic Worlds: Agentic Language World Models for Interactive Environment Simulation
Oct 5, 2026Quanyu Long, Xiao Chen, Jianda Chen +4LLM World Models Interactive World Models
Nanyang Technological University · The Hong Kong Polytechnic University
Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond
Apr 24, 2026Meng Chu, Xuan Billy Zhang, Kevin Qinghong Lin +47World Model Learning Action-Conditioned World Models
False Prophets: On the Security of World Models in Agentic Systems
Jul 25, 2026Erik Imgrund, Anna Wimbauer, Klim Kireev +1Adversarial Attacks on World Models Adversarial Attacks
BIFOLD & TU Berlin Berlin, Germany

Oct 5, 2026J/K move · Enter open · S save

From Traces to Agentic Worlds: Agentic Language World Models for Interactive Environment Simulation

Quanyu Long, Xiao Chen, Jianda Chen, Haozhen Zhang+3

Nanyang Technological University · The Hong Kong Polytechnic University

LLM World Models Interactive World Models

Open paper AskPDF arXiv

Realistic environment replicas are increasingly valuable for training and evaluating LLM agents, yet the original systems may be inaccessible or impractical to reproduce. We explore agentic language world modeling: rather than rebuilding an executable environment, a world model agent serves as the environment for a task agent and supports faithful and stateful simulation. We instantiate this paradigm with Trace2Env, a learning-free framework for settings where the original system is unavailable but historical interaction traces remain accessible. Trace2Env reconstructs these traces into a reusable environment worldbook containing environment schemas, grounded evidence, and induced behavioral knowledge. At runtime, the world model agent actively consults the worldbook together with persistent episodic state to infer each action's observation and lasting state effects. Across nine environments, Trace2Env improves both next-observation fidelity and long-horizon interaction consistency over conventional prompt-based LWMs. In multi-turn interaction, task agent actions generated against Trace2Env remain valid more often when replayed in the real environment, indicating that its simulated dynamics better preserve the consequences of earlier actions across successive turns. These results establish agentic language world modeling as an alternative direction for building realistic environment replicas without reconstructing the original executable system.

Abstract

Agentic AI

Explore similar work

From Traces to Agentic Worlds: Agentic Language World Models for Interactive Environment Simulation

Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond

False Prophets: On the Security of World Models in Agentic Systems