Organizations: Lightspeed Studios, Tencent · The Hong Kong University of Science and Technology · University of Science and Technology of China · Hunyuan Team, Tencent
Evaluating autonomous agents requires examining whether they can translate goals into coherent action across extended interactions. In persistent environments, this requires coordinating interdependent actions under incomplete observations, accounting for delayed effects, and adapting as earlier decisions shape subsequent opportunities. We introduce ASIWorld, a survival sandbox that combines persistent world and character state with character-scoped observations and a programmable asynchronous interface to study sustained goal execution. Agents pursue interdependent production and exploration tasks through direct actions or agent-authored programs that continue running between model calls. The evaluation assesses task success, the attainment of development achievements, and survival costs, complemented by an analysis of execution behavior. In an exploratory study of nine agent configurations using a common harness, eight complete the production task and six complete both tasks. Analyses of execution trajectories and performance across difficulty levels reveal different preparation, combat, and recovery behaviors among configurations with similar task outcomes. Incomplete preparation is associated with stalled progression, while configurations that complete the tasks differ in survival costs and the fulfillment of additional achievement criteria. These findings illustrate the value of assessing sustained goal execution through both task outcomes and the behaviors and costs associated with achieving them.
Figures & tables
Figure 1: ASIWorld evaluation framework. (a) Persistent character state connects production, construction, and exploration. (b) Agents interact through a model and execution harness with asynchronous hosted execution. (c) Execution records and world state support assessment of task outcomes, development achievements, behavior, and costs.
Figure 2: Environment and agent interaction. (a) Current character-scoped observations and last-seen information provide complementary views of a changing world; inventory, equipment, facilities, and health persist across interactions. (b) Agents issue direct actions or author programs, with both subject to world-rule validation. Feedback from ongoing operations and terminal outcomes, together with updated observations, informs subsequent decisions. Hosted execution extends across model calls along the world timeline.
Figure 3: Task objectives and qualification. The two paths connect preparation and task progression to the assigned objectives: acquiring a travel outfit and defeating the Shadow King. Production combines materials and an outfit design through tailoring; exploration progresses through dungeon tiers and altar activation to the hidden boss. The lower strip summarizes additional acquisition and expedition qualifications, assessed separately from primary task success.
Stage
Achievement
Requirements
1
Foothold
One continuous survival day, a usable bed, and self-crafted equipment.
2
Self-sufficiency
Cooking, equipment, and bed qualification; weapon and armor equipped together; a full night without damage.
3
Home
A locked home and at least nine tiles of completed building area.
4
Outer Court
Solo Tier 1 completion in a fresh instance, followed by leaving the dungeon alive.
5
Inner Temple
Solo Tier 2 completion in a fresh instance, followed by leaving the dungeon alive.
6
Steadiness
Qualifying solo Tier 2 or 3 completion with either damage at most 30 and supply consumption at most one, or no damage.
Table 1: Development achievements in the environment’s native curriculum. Alternatives are stated explicitly. Section 5.1 describes how achievement attainment is assessed.
Component
Setting
Evaluated systems
Nine agent configurations; one run per configuration
Environment
Newly initialized shared world; approximately synchronized entry
Evaluation horizon
300 simulation minutes
Primary tasks
Outfit production and hidden-boss defeat
Primary outcomes
Success on each task
Development measures
Eight assessed achievements; independent coverage and completed prefix
Table 2: Main evaluation setting. Additional configuration details and assessment limitations are given in the text.
Rank
Configuration
Outfit
Depth
Hero
Kills
Bosses
Crafts
Coins
Trial
Downs
Score
1
claude-opus-5-5
Y
3
Y
30
5
101
88
7/8
0
93.8
2
gpt-6-astra
Y
3
Y
23
5
87
92
8/8
0
90.9
3
claude-opus-5
Y
3
Y
24
5
37
92
8/8
0
78.5
4
deepseek-v4-flash
Y
3
Y
45
8
19
149
6/8
3
78.5
5
glm-5.3
Y
3
Y
31
6
33
126
1/8
1
75.0
6
kimi-k3
Y
3
Y
29
5
29
119
5/8
0
74.7
Table 3: Main evaluation: task success and activity. Outfit and Hero indicate success on the production and exploration tasks, respectively (Y/N). Depth is the deepest dungeon tier cleared. Kills counts all defeated enemies, Bosses the dungeon bosses defeated, Crafts the successful crafting operations, Coins the T-coins earned in expeditions, and Trial the stages passed under the environment’s native cumulative rule, in which a stage counts only after every earlier stage is passed (per-stage attainment is reported in Table 4 ). Downs counts knockdowns. Score is the composite defined in Equation ( 1 ); configurations are ranked by unrounded Score, with displayed values rounded to one decimal place.
Table 4: Development achievements attained by each configuration, assessed independently of one another; meals prepared by stove crafting count toward cooking.
Figure 4: Dungeon performance by difficulty level. (a) Share of attempts that defeat the level’s boss; (b) mean damage taken in those clears.
Figure 5: Composition of executed game actions by category in the main evaluation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
Incident
claude-opus-5
Approximately 107 minutes of interaction interruption due to upstream service errors
gpt-6-luna
Approximately ten minutes waiting for an interactive approval
Appendix
Table 5: Service interruptions in E-1004b. No configuration-specific interruption occurred in the main evaluation.
j
Component xm,j
Interpretation
Weight wj
1
hm
Task outcome: Shadow King defeat
30
2
om
Task outcome: travel-outfit production
10
3
dm/3
Progression: dungeon depth
15
4
Bm/maxkBk
Activity: defeated dungeon bosses
15
5
Cm/maxkCk
Activity: successful crafts
25
6
Rm/8
Native Trial progress
5
Appendix
Table 6: Components and weights of the composite score in Equation ( 1 ); count-based components are normalized by the cohort maximum.
Configuration
Outfit
Tier 1
Tier 2
Tier 3
Hero
Homes
claude-opus-5
Y / Y
Y / Y
Y / Y
Y / Y
Y / Y
1 / 1
claude-opus-5-5
Y / Y
Y / Y
Y / Y
Y / Y
Y / Y
1 / 1
gpt-6-astra
Y / Y
Y / Y
Y / Y
Y / Y
Y / Y
1 / 1
glm-5.3
Y / Y
Y / Y
Y / –
Y / Y
Y / Y
1 / 1
gemini-3.8-flash
Y / Y
Y / Y
Y / Y
Y / Y
Y / Y
1 / 1
deepseek-v4-flash
Y / Y
Y / Y
Y / Y
– / Y
– / –
1 / 1
Appendix
Table 7: Achievements in the earlier evaluations. Each cell reports E1 / E2. Y denotes a recorded achievement; – denotes an absent achievement record.
Configuration
Stages
Progress
First unmet stage
Missing qualification
claude-opus-5
8/8
100.0%
Complete
–
deepseek-v4-flash
7/8
87.5%
Hero
Hidden-boss defeat
gemini-3.8-flash
6/8
75.0%
Royal Tomb
Solo Tier 3; five-day streak without knockdown
gpt-6-luna
4/8
50.0%
Inner Temple
Solo Tier 2
gpt-6-astra
2/8
25.0%
Home
Locked home
glm-5.3
2/8
25.0%
Home
Locked home
Appendix
Table 8: Curriculum completion in E2. Progress is the completed prefix; the first unmet stage identifies its limiting prerequisite.
Configuration
Days
Crafts
Kills
Bosses
Downs
claude-opus-5
9.9 / 8.4
56 / 57
46 / 36
9 / 8
0 / 1
claude-opus-5-5
9.9 / 8.4
33 / 61
24 / 17
5 / 4
1 / 0
gpt-6-astra
9.8 / 8.5
36 / 37
21 / 19
4 / 4
0 / 0
glm-5.3
9.7 / 8.4
19 / 31
23 / 22
4 / 3
4 / 3
gemini-3.8-flash
9.9 / 8.3
9 / 16
23 / 27
4 / 6
6 / 10
deepseek-v4-flash
9.8 / 9.8
29 / 13
39 / 53
3 / 13
2 / 0
Appendix
Table 9: Activity and survival outcomes in the earlier evaluations. Each cell reports E1 / E2.