Organizations: Lightspeed Studios, Tencent · The Hong Kong University of Science and Technology · University of Science and Technology of China · Hunyuan Team, Tencent
Evaluating autonomous agents requires examining whether they can translate goals into coherent action across extended interactions. In persistent environments, this requires coordinating interdependent actions under incomplete observations, accounting for delayed effects, and adapting as earlier decisions shape subsequent opportunities. We introduce ASIWorld, a survival sandbox that combines persistent world and character state with character-scoped observations and a programmable asynchronous interface to study sustained goal execution. Agents pursue interdependent production and exploration tasks through direct actions or agent-authored programs that continue running between model calls. The evaluation assesses task success, the attainment of development achievements, and survival costs, complemented by an analysis of execution behavior. In an exploratory study of nine agent configurations using a common harness, eight complete the production task and six complete both tasks. Analyses of execution trajectories and performance across difficulty levels reveal different preparation, combat, and recovery behaviors among configurations with similar task outcomes. Incomplete preparation is associated with stalled progression, while configurations that complete the tasks differ in survival costs and the fulfillment of additional achievement criteria. These findings illustrate the value of assessing sustained goal execution through both task outcomes and the behaviors and costs associated with achieving them.
Figures & tables
Figure 1: ASIWorld evaluation framework. (a) Persistent character state connects production, construction, and exploration. (b) Agents interact through a model and execution harness with asynchronous hosted execution. (c) Execution records and world state support assessment of task outcomes, development achievements, behavior, and costs.
Figure 2: Environment and agent interaction. (a) Current character-scoped observations and last-seen information provide complementary views of a changing world; inventory, equipment, facilities, and health persist across interactions. (b) Agents issue direct actions or author programs, with both subject to world-rule validation. Feedback from ongoing operations and terminal outcomes, together with updated observations, informs subsequent decisions. Hosted execution extends across model calls along the world timeline.
Figure 3: Task objectives and qualification. The two paths connect preparation and task progression to the assigned objectives: acquiring a travel outfit and defeating the Shadow King. Production combines materials and an outfit design through tailoring; exploration progresses through dungeon tiers and altar activation to the hidden boss. The lower strip summarizes additional acquisition and expedition qualifications, assessed separately from primary task success.
Stage
Achievement
Requirements
1
Foothold
One continuous survival day, a usable bed, and self-crafted equipment.
2
Self-sufficiency
Cooking, equipment, and bed qualification; weapon and armor equipped together; a full night without damage.
3
Home
A locked home and at least nine tiles of completed building area.
4
Outer Court
Solo Tier 1 completion in a fresh instance, followed by leaving the dungeon alive.
5
Inner Temple
Solo Tier 2 completion in a fresh instance, followed by leaving the dungeon alive.
6
Steadiness
Qualifying solo Tier 2 or 3 completion with either damage at most 30 and supply consumption at most one, or no damage.
Table 1: Development achievements in the environment’s native curriculum. Alternatives are stated explicitly. Section 5.1 describes how achievement attainment is assessed.
Component
Setting
Evaluated systems
Nine agent configurations; one run per configuration
Environment
Newly initialized shared world; approximately synchronized entry
Evaluation horizon
300 simulation minutes
Primary tasks
Outfit production and hidden-boss defeat
Primary outcomes
Success on each task
Development measures
Eight assessed achievements; independent coverage and completed prefix
Table 2: Main evaluation setting. Additional configuration details and assessment limitations are given in the text.
Rank
Configuration
Outfit
Depth
Hero
Kills
Bosses
Crafts
Coins
Trial
Downs
Score
1
claude-opus-5-5
Y
3
Y
30
5
101
88
7/8
0
93.8
2
gpt-6-astra
Y
3
Y
23
5
87
92
8/8
0
90.9
3
claude-opus-5
Y
3
Y
24
5
37
92
8/8
0
78.5
4
deepseek-v4-flash
Y
3
Y
45
8
19
149
6/8
3
78.5
5
glm-5.3
Y
3
Y
31
6
33
126
1/8
1
75.0
6
kimi-k3
Y
3
Y
29
5
29
119
5/8
0
74.7
Table 3: Main evaluation: task success and activity. Outfit and Hero indicate success on the production and exploration tasks, respectively (Y/N). Depth is the deepest dungeon tier cleared. Kills counts all defeated enemies, Bosses the dungeon bosses defeated, Crafts the successful crafting operations, Coins the T-coins earned in expeditions, and Trial the stages passed under the environment’s native cumulative rule, in which a stage counts only after every earlier stage is passed (per-stage attainment is reported in Table 4 ). Downs counts knockdowns. Score is the composite defined in Equation ( 1 ); configurations are ranked by unrounded Score, with displayed values rounded to one decimal place.
Table 4: Development achievements attained by each configuration, assessed independently of one another; meals prepared by stove crafting count toward cooking.
Figure 4: Dungeon performance by difficulty level. (a) Share of attempts that defeat the level’s boss; (b) mean damage taken in those clears.
Figure 5: Composition of executed game actions by category in the main evaluation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Incident
claude-opus-5
Approximately 107 minutes of interaction interruption due to upstream service errors
gpt-6-luna
Approximately ten minutes waiting for an interactive approval
Appendix
Table 5: Service interruptions in E-1004b. No configuration-specific interruption occurred in the main evaluation.
j
Component xm,j
Interpretation
Weight wj
1
hm
Task outcome: Shadow King defeat
30
2
om
Task outcome: travel-outfit production
10
3
dm/3
Progression: dungeon depth
15
4
Bm/maxkBk
Activity: defeated dungeon bosses
15
5
Cm/maxkCk
Activity: successful crafts
25
6
Rm/8
Native Trial progress
5
Appendix
Table 6: Components and weights of the composite score in Equation ( 1 ); count-based components are normalized by the cohort maximum.
Configuration
Outfit
Tier 1
Tier 2
Tier 3
Hero
Homes
claude-opus-5
Y / Y
Y / Y
Y / Y
Y / Y
Y / Y
1 / 1
claude-opus-5-5
Y / Y
Y / Y
Y / Y
Y / Y
Y / Y
1 / 1
gpt-6-astra
Y / Y
Y / Y
Y / Y
Y / Y
Y / Y
1 / 1
glm-5.3
Y / Y
Y / Y
Y / –
Y / Y
Y / Y
1 / 1
gemini-3.8-flash
Y / Y
Y / Y
Y / Y
Y / Y
Y / Y
1 / 1
deepseek-v4-flash
Y / Y
Y / Y
Y / Y
– / Y
– / –
1 / 1
Appendix
Table 7: Achievements in the earlier evaluations. Each cell reports E1 / E2. Y denotes a recorded achievement; – denotes an absent achievement record.
Configuration
Stages
Progress
First unmet stage
Missing qualification
claude-opus-5
8/8
100.0%
Complete
–
deepseek-v4-flash
7/8
87.5%
Hero
Hidden-boss defeat
gemini-3.8-flash
6/8
75.0%
Royal Tomb
Solo Tier 3; five-day streak without knockdown
gpt-6-luna
4/8
50.0%
Inner Temple
Solo Tier 2
gpt-6-astra
2/8
25.0%
Home
Locked home
glm-5.3
2/8
25.0%
Home
Locked home
Appendix
Table 8: Curriculum completion in E2. Progress is the completed prefix; the first unmet stage identifies its limiting prerequisite.
Configuration
Days
Crafts
Kills
Bosses
Downs
claude-opus-5
9.9 / 8.4
56 / 57
46 / 36
9 / 8
0 / 1
claude-opus-5-5
9.9 / 8.4
33 / 61
24 / 17
5 / 4
1 / 0
gpt-6-astra
9.8 / 8.5
36 / 37
21 / 19
4 / 4
0 / 0
glm-5.3
9.7 / 8.4
19 / 31
23 / 22
4 / 3
4 / 3
gemini-3.8-flash
9.9 / 8.3
9 / 16
23 / 27
4 / 6
6 / 10
deepseek-v4-flash
9.8 / 9.8
29 / 13
39 / 53
3 / 13
2 / 0
Appendix
Table 9: Activity and survival outcomes in the earlier evaluations. Each cell reports E1 / E2.
Standard embodied evaluations do not independently score whether an agent correctly commits to task completion at episode closure, a capacity we call terminal commitment. Behaviorally distinct failures--never completing the task, completing it but failing to stop, and reporting success without sufficient evidence--collapse into the same benchmark failure. We introduce VIGIL, an evaluation framework that makes terminal commitment independently measurable. Under VIGIL's default protocol, agents observe only egocentric RGB, receive no action-success signals, and must end each episode with a semantic report checked deterministically against hidden world state. This yields two separate scores: world-state completion (W) and benchmark success (B), where B additionally requires a correct terminal report. This decoupling makes four outcome categories distinguishable: missed execution, post-attainment drift, unsupported commitment, and verified success. Across 20 models on 1,000 frozen episodes, systems with comparable W differ by up to 19.7 pp in B: one model converts achieved states into correct reports, while another with near-identical execution drifts past the goal without closing. An action-feedback intervention further tests the separation: execution-oriented signals improve W broadly, yet commitment failures persist in models that do not already ground terminal reports in the achieved state. VIGIL provides a protocol that makes terminal commitment independently visible and scorable.
Existing multi-agent benchmarks primarily test in competitive settings, short-horizon interactions under 20 steps, or simply aggregate individual performance, failing to isolate and highlight genuine collaboration capabilities of LLM-based agents. We introduce AgentWorld, a benchmark of 100 human-annotated tasks (with 100 augmented variants) for evaluating long-horizon, multi-agent collaboration. Tasks span 50+ interaction rounds across a rich MMORPG sandbox and require 3-20 agents with asymmetric roles and abilities to coordinate through communication, joint planning, and resource sharing under a blackbox setting where each agent acts independently without access to others' internal states. To quantify collaboration effectiveness in addition to conventional binary task success, we propose Causal Collaboration Effectiveness (CCE), a graph-based metric that traces causal dependencies between agent actions and measures what fraction of a team's effort actually contributed to the outcome. Experiments with Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini, and DeepSeek R1-70B show that even the best model achieves only 52.0% task success, with systematic failure modes including communication breakdowns, role confusion, and inability to maintain shared plans across rounds. AgentWorld is fully open-source.
Raphael Shu, Yusen Zhang, Young Min Cho +7
OpenAgents · Columbia University · University of Pennsylvania +2
Autonomous agents are increasingly adopted to complete complex, multi-tool workflows in real-world settings. However, existing benchmarks typically separate tasks by application or capability and evaluate agents in environments that are cleaner and more stable than those encountered in practice. We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions collected from a large-scale production agent platform. Each task preserves the relevant pre-solution interaction history, persistent configurations, and workspace state, and is then validated through human verification. The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories, with most tasks requiring multiple capability coordination. We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy, and assess performance using a hybrid deterministic and LLM-as-Judge evaluation protocol. Experiments across five representative autonomous-agent frameworks paired with four state-of-the-art LLMs reveal substantial gaps in strict task completion. Complementary robustness, efficiency, and diagnostic analyses further show that performance under environmental perturbations is jointly shaped by the capabilities of the LLM and the surrounding agent framework. The code and data are publicly available at https://dumatebench.com/.
Zechun Niu, Yukun Zhao, Jiaxin Zhang +12
Renmin University of China Beijing, China · Shandong University Jinan, China · Independent Researcher China +5