Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning

Qinchuan Cheng; Zhantao Gong; Pengzhan Sun; Angela Yao; Shijie Li

cs.AIOct 2, 2026

Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning

Authors: Qinchuan Cheng, Zhantao Gong, Pengzhan Sun, Angela Yao, Shijie Li

Organizations: Xi’an Jiaotong University (xjtu.edu.cn) · Nankai University (nankai.edu.cn) · National University of Singapore (nus.edu) · Institute for Infocomm Research, A*STAR (i2r.a-star.edu.sg)

Abstract

Egocentric videos capture how people carry out everyday activities, yet testing an agent requires evaluating the consequences of actions it chooses itself. We introduce Ego2World, a benchmark that turns annotated cooking activities into executable planning environments under partial observation. Its compiler links source steps and objects to symbolic action rules, persistent world states, and explicit task conditions, so researchers can execute an agent's proposed actions and check their outcomes. World state and agent belief are maintained separately, enabling controlled studies of planning and information reuse across continuing tasks. Evaluating six planners on 105 tasks shows that accepted operations often leave task goals unmet. Execution traces and condition checks distinguish interrupted runs, partial attainment, and completed execution without goal attainment. In a separate paired Qwen-Plus study, persistent belief improves action validity by 4.15 percentage points and reduces visual-query attempts by 90.27%, with higher token use and no detected completion gain. Ego2World provides a reusable testbed for tracing how planning and memory choices affect execution, observation demand, and task attainment, connecting recorded human activity to the development and evaluation of interactive agents.

Figures & tables

Figure 1: From activity evidence to interactive evaluation. The upper panel illustrates construction: its hidden graph $G_{h}$ is denoted $G^{\mathrm{w}}$ in the text, and skills correspond to action groups. The lower schematic shows how agents use the compiled environment. Source links, action records, and task checks make decisions inspectable; Table 1 follows a real task through these stages.

Stage	Recorded evidence and executable interpretation
Source activity	At 8.82–10.49 s, narration 6 reads: “Pour a small 1/20th measure of squash into the glass.”
Compiled operation	Steps 5–7 form group 3. Its reference pour binds the substance squash and the destination glass.
Task condition	The final glass contents must include squash; missing contents evidence is unknown .
Persistent belief	Pour accepted; glass contents include squash. Condition true, task succeeds.
Per-group reset	Squash pickups rejected; glass pickup accepted; no pour attempted. Contents evidence missing, task fails.

Table 1: A real task from source evidence to agent outcome: Add Squash , episode P02-20240209-231412 . The two attempts come from the paired memory study (Section 6.4 ); both finish all four groups. The frozen condition checks contents, not the quantity mentioned in the narration. Appendix A.4 gives the source mapping.

Figure 2: Goal attainment within each contract type and execution endpoints for all 105 tasks. (a) Terminal Strict-TSR and GCR; (b) graded attainment (GCR) for successful-event contracts; both use pointwise 95% episode-cluster intervals. Operational-TSR remains the fixed-denominator completion baseline in Table 6 . Percentage axes have different ranges. (c) Mutually exclusive recorded endpoints: partial attainment is

0<\mathrm{GCR}<1

in a completed run. Counts describe outcomes, not their causes. Tables 6 and 8 give aggregate and stratified values.

Figure 3: Paired memory study on 51 tasks. (a,b) Post-hoc resource profiles across four quarters of each episode’s group sequence; all 22 matched episodes contribute to every quarter. Rates pool groups, while (c) averages task metrics. Whiskers and shading show pointwise 95% episode-cluster intervals. The validity gain has paired CI

[+0.11,+7.78]

pp; no completion gain is detected. Full paired effects and aggregation details appear in Appendix C.6 .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Annotation-to-environment compilation pipeline. Raw HD-EPIC narrations are grouped into executable action groups, validated through deterministic and LLM-assisted quality control, aligned with task goals, and replayed in the simulator to produce graph-transition worlds.

Property	Count
Source recordings	41 hours; 9 kitchens; 69 recipes
Normalized executable action types	155
Controlled task set	105
Complete episodes in controlled set	18
Scenes in controlled set	4
Execution groups per planner	3,441

Appendix

Table 2: Source-data context and the two evaluation cohorts. HD-EPIC scale is not the evaluated scope. The six-planner cohort is a non-random prefix of complete episodes; the memory cohort is defined separately.

Component	Stored information
Objects	Label-keyed region, confidence, and evidence source; VLM entries can additionally store current state.
Regions	Flags recording local or VLM observation.
Containers	Known open/closed state and source.
Agent	Current region.

Appendix

Table 3: Belief-store fields in the aligned runtime.

Conditions per task	Terminal tasks	Event tasks
1	9	11
2	13	27
3	4	23
4–5	1	10
6–10	3	0
11–29	4	0

Appendix

Table 4: Number of required conditions per task in the frozen 105-task suite. Both types have median two. Counts describe contract structure, not observed success.

Final execution outcome	Cup in cupboard	Cupboard closed	(Strict-TSR, GCR)
Cup placed; cupboard left open	True	False	$(0,0.5)$
Cup placed; cupboard closed	True	True	$(1,1.0)$
Wrong cup placed; cupboard closed	False	True	$(0,0.5)$
Cup location unavailable; cupboard closed	Unknown	True	$(0,0.5)$

Appendix

Table 5: Worked terminal-goal example. The task requires a particular cup to be in the cupboard and the cupboard to be closed.

Planner	Completed	Op.-TSR	95% CI	Strict-TSR	GCR
Qwen-Plus	44/105	5.71	[2.78, 9.20]	17.65	9.29
GPT-5.5	59/105	5.71	[2.50, 9.89]	14.71	11.61
Gemini 3.1 Pro	69/105	6.67	[2.91, 11.76]	17.65	12.31
DeepSeek-V4-Flash	65/105	3.81	[0.00, 7.29]	8.82	11.48
GLM-5	56/105	3.81	[1.02, 7.50]	11.76	7.27
Claude Sonnet 4.6	49/105	5.71	[1.74, 11.70]	14.71	9.75

Appendix

Table 6: Protocol-aligned evaluation on the same 105 tasks (18 episodes; 3,441 groups per planner). Completed counts runs with terminal status completed . Operational-TSR combines 34 terminal-state and 71 successful-event contracts; Strict-TSR uses the 34 terminal tasks. Other than Completed, values are percentages. Intervals are 95% scene-stratified episode-cluster bootstrap intervals (10,000 samples).

Planner	GCR [95% CI]	Validity [95% CI]
Qwen-Plus	9.29 [5.90, 13.21]	57.44 [48.66, 67.86]
GPT-5.5	11.61 [7.13, 16.22]	59.34 [55.08, 63.21]
Gemini 3.1 Pro	12.31 [6.30, 20.50]	54.44 [47.10, 62.40]
DeepSeek-V4-Flash	11.48 [6.90, 16.15]	52.77 [46.52, 61.82]
GLM-5	7.27 [3.38, 12.48]	50.72 [46.90, 55.90]
Claude Sonnet 4.6	9.75 [4.05, 18.48]	56.91 [49.29, 66.79]

Appendix

Table 7: Primary task-macro diagnostic means and 95% scene-stratified episode-cluster bootstrap intervals on the 105-task set (percent). GCR measures graded contract attainment and validity measures accepted-action reliability; both are computed on the same task IDs and provide complementary capability axes. Reference coverage is retained as a process audit in the frozen analysis export.

	Terminal state ( $n=34$ )		Successful event ( $n=71$ )
Planner	Strict-TSR	GCR	GCR
Qwen-Plus	17.65 [8.70, 28.57]	23.53 [15.31, 34.45]	2.46 [0.00, 5.00]
GPT-5.5	14.71 [5.88, 25.00]	24.25 [11.90, 35.80]	5.56 [2.38, 9.50]
Gemini 3.1 Pro	17.65 [9.09, 27.27]	26.40 [14.71, 39.39]	5.56 [1.97, 11.01]
DeepSeek-V4-Flash	8.82 [0.00, 17.86]	22.37 [11.95, 31.13]	6.27 [3.13, 9.52]
GLM-5	11.76 [3.23, 22.73]	17.70 [6.52, 31.09]	2.28 [0.86, 4.43]

Appendix

Table 8: Outcome profile within each frozen contract type. Values are percentages; brackets give pointwise 95% episode-cluster intervals. Terminal contracts report Strict-TSR and GCR, while successful-event contracts report their graded GCR signal. The aggregate Operational-TSR remains the fixed-denominator completion baseline over all 105 tasks.

Check	Scope	Passed
Frozen reference record satisfies contract	105 tasks	105/105
Initial state correctly fails	105 tasks	105/105
Predicate-level negative suites pass	317 suites	317/317
Reference and action-only replay	12 tasks	12/12; 12/12
Partial and wrong-binding negatives	12 tasks each	12/12; 12/12
Missing state is unknown and fails	12 tasks	12/12

Appendix

Table 9: Frozen-contract and evaluator audits. The 105-task audit uses exactly the planner-evaluation IDs and no planner outcomes. The 12-task stress test contains 43 predicates and 45 replayed evidence changes.

Planner pair	Difference (pp)	95% CI (pp)
Qwen-Plus $-$ GPT-5.5	+0.00	[-4.12, +3.53]
Qwen-Plus $-$ Gemini 3.1 Pro	-0.95	[-6.06, +2.78]
Qwen-Plus $-$ DeepSeek-V4-Flash	+1.90	[-1.01, +5.43]
Qwen-Plus $-$ GLM-5	+1.90	[-2.67, +5.75]
Qwen-Plus $-$ Claude Sonnet 4.6	+0.00	[-6.41, +4.38]
GPT-5.5 $-$ Gemini 3.1 Pro	-0.95	[-5.04, +3.12]

Appendix

Table 10: Paired Operational-TSR baseline differences in percentage points (row model minus column model). Intervals use the same 10,000 scene-stratified episode-cluster samples; the capability profile in Tables 11 – 13 provides the primary model comparison.

Planner pair	Difference	Pointwise 95% CI	Simultaneous 95% CI
Qwen-Plus $-$ GPT-5.5	-2.33	[-6.12, +1.71]	[-8.86, +4.21]
Qwen-Plus $-$ Gemini 3.1 Pro	-3.02	[-9.89, +1.88]	[-12.85, +6.80]
Qwen-Plus $-$ DeepSeek-V4-Flash	-2.20	[-6.70, +2.44]	[-9.77, +5.37]
Qwen-Plus $-$ GLM-5	+2.01	[-2.30, +5.63]	[-4.56, +8.59]
Qwen-Plus $-$ Claude Sonnet 4.6	-0.47	[-7.52, +4.24]	[-10.43, +9.49]
GPT-5.5 $-$ Gemini 3.1 Pro	-0.70	[-7.03, +4.57]	[-10.30, +8.90]

Appendix

Table 11: Goal completion ratio: all 15 paired planner differences, in percentage points (first model minus second). Pointwise intervals use paired episode-cluster percentiles; simultaneous intervals cover the joint family of 45 diagnostic contrasts. Bold differences have simultaneous intervals excluding zero.

Planner pair	Difference	Pointwise 95% CI	Simultaneous 95% CI
Qwen-Plus $-$ GPT-5.5	-1.89	[-8.19, +5.84]	[-13.98, +10.19]
Qwen-Plus $-$ Gemini 3.1 Pro	+3.01	[-1.73, +7.06]	[-4.34, +10.35]
Qwen-Plus $-$ DeepSeek-V4-Flash	+4.67	[-1.32, +10.12]	[-5.06, +14.40]
Qwen-Plus $-$ GLM-5	+6.72	[-0.14, +13.60]	[-4.98, +18.42]
Qwen-Plus $-$ Claude Sonnet 4.6	+0.54	[-4.21, +4.49]	[-6.72, +7.79]
GPT-5.5 $-$ Gemini 3.1 Pro	+4.90	[-1.26, +10.22]	[-4.93, +14.73]

Appendix

Table 12: Action validity: all 15 paired planner differences, in percentage points (first model minus second). Pointwise intervals use paired episode-cluster percentiles; simultaneous intervals cover the joint family of 45 diagnostic contrasts. Bold differences have simultaneous intervals excluding zero.

Planner pair	Difference	Pointwise 95% CI	Simultaneous 95% CI
Qwen-Plus $-$ GPT-5.5	-14.29	[-20.78, -6.60]	[-26.17, -2.40]
Qwen-Plus $-$ Gemini 3.1 Pro	-23.81	[-33.05, -14.78]	[-39.06, -8.55]
Qwen-Plus $-$ DeepSeek-V4-Flash	-20.00	[-30.12, -10.08]	[-36.61, -3.39]
Qwen-Plus $-$ GLM-5	-11.43	[-18.12, -1.30]	[-25.56, +2.71]
Qwen-Plus $-$ Claude Sonnet 4.6	-4.76	[-12.15, +5.48]	[-19.40, +9.88]
GPT-5.5 $-$ Gemini 3.1 Pro	-9.52	[-17.81, -1.80]	[-22.76, +3.71]

Appendix

Table 13: Completed-run rate: all 15 paired planner differences, in percentage points (first model minus second). Pointwise intervals use paired episode-cluster percentiles; simultaneous intervals cover the joint family of 45 diagnostic contrasts. Bold differences have simultaneous intervals excluding zero.

Metric	Persistent	Reset	Difference	95% paired CI
Strict-TSR	11.76%	11.76%	+0.00 pp	[-6.38, +6.25] pp
Action validity	45.94%	41.79%	+4.15 pp	[+0.11, +7.78] pp
VLM query attempts	95	976	-881	[-1,106, -651]
Group-accounted tokens	27.14 M	23.58 M	+3.56 M	[+1.82, +5.45] M

Appendix

Table 14: Paired Qwen-Plus memory comparison on frozen Strict-51 (51 tasks, 22 episodes, 3,878 groups per condition). Differences are persistent minus per-group reset. Intervals use 10,000 scene-stratified episode-cluster bootstrap samples (seed 42).

Paper label	Requested model ID	Request setting
Qwen-Plus	qwen-plus	enable_thinking=false
GPT-5.5	gpt-5.5	reasoning_effort=none
Gemini 3.1 Pro	gemini-3.1-pro	reasoning_effort=none
DeepSeek-V4-Flash	deepseek-v4-flash	thinking.type=disabled
GLM-5	glm-5	reasoning_effort=none
Claude Sonnet 4.6	claude-sonnet-4-6	reasoning_effort=none

Appendix

Table 15: Planner request identities and optional-reasoning controls. Qwen uses the official DashScope compatible endpoint; the other planners use the same OpenAI-compatible gateway. The gateway resolved the Gemini request to gemini-3.1-pro-low .

Payload component	Contents and truncation
Local observation	Text, current region, up to 30 visible objects, 20 open storages, 10 active processes, and 50 reachable-action hints.
Agent memory	Current region, up to 30 remembered objects (label, region, confidence, source), and known container states.
Feedback	Last three execution-failure messages.
Completed prefix	Last eight successful actions in this group.
Goal and schema	Compiled group instruction, supported action types, and output schema.

Appendix

Table 16: Actual field structure assembled by the aligned planner-message function. Local symbolic observations and action hints supplement visual anchors.

Planner	Attempts	Accepted	Rejected	Repair attempts
Qwen-Plus	61,302	39,292	22,010	11,763
GPT-5.5	43,981	28,643	15,338	7,663
Gemini 3.1 Pro	46,410	28,602	17,808	8,904
DeepSeek-V4-Flash	58,739	34,075	24,664	13,421
GLM-5	54,948	30,536	24,412	13,874
Claude Sonnet 4.6	57,368	37,596	19,772	10,188

Appendix

Table 17: Primitive attempts in the frozen six-planner group traces. Repair counts are included in total attempts. These are pooled counts, not task-macro validity percentages. Planner/controller steps include controller-inserted navigation.

Task	Condition	Accepted/attempted	Queries	Success
Prepare mushrooms	P	85/122	0	0
	R	26/46	4	1
Add Squash	P	16/21	0	1
	R	13/17	0	0
Wrap tortilla leftovers	P	23/71	0	0
	R	20/30	6	1

Appendix

Table 18: All four discordant task outcomes in the paired memory study. P: persistent belief; R: per-group reset. Accepted/attempted counts and outcome scores come from task records. An execution block receives zero outcome score even when many attempts were accepted.

Group	Instruction and execution excerpt	Recorded belief evidence
1	Retrieve a mug: open(cupboard) accepted.	Mug region recorded as cupboard from local observation; no retrieval action in this group.
2	Place mug under nozzle: pickup, navigation, and placement accepted.	Mug region subsequently recorded as countertop, not a fine-grained nozzle relation.
3	Switch on machine and insert capsule: switch, pickup, and placement accepted.	Capsule region recorded as countertop after execution.
4	Retrieve milk from fridge: navigation accepted.	Only navigation is recorded; milk remains a retained countertop belief.
5	Move frother: placement rejected, subsequent move accepted.	Frother region recorded as countertop.
6	Open, pour, close milk: pickup and pouring accepted; four opening attempts and one closing attempt rejected.	Mixed acceptance despite completed runtime status.

Appendix

Table 19: Logged coffee-preparation excerpt. All seven group records have runtime status completed ; this status is not a claim of successful goal attainment. Accepted actions and stored beliefs are distinct evidence.

Component	Research use	Manuscript package
Worlds and action rules	Execute new proposals and inspect state changes	Method and interface specifications
Source and task mappings	Trace task objects and operations to activity evidence	Cohort identities and the Add Squash example
Contracts and scorer	Evaluate states and accepted events	Definitions, audits, and condition inventory
Agent and execution records	Examine decisions, observations, and results	Frozen metric exports and selected/minimal trace projections
Analysis scripts	Reproduce tables, figures, and uncertainty	Included; no provider access required
Video assets	Supply source appearance and context	Access through the source dataset

Appendix

Table 20: Benchmark components and the scope of the accompanying manuscript package. Interactive components belong to the separate benchmark release; the manuscript package reproduces analyses from recorded evidence.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,497 papers

Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning
May 13, 2026Qinchuan Cheng, Zhantao Gong, Pengzhan Sun +3Belief-Space Planning Long-Horizon Agent Evaluation
Xi’an Jiaotong University · Nankai University · National University of Singapore +1
Grounded World Model: Latent Planning with Language Goals
Apr 13, 2026Quanyi Li, Lan Feng, Haonan Zhang +4Language-Conditioned Robot Planning World Models for Robotics
EPFL · NUS
EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions
Oct 13, 2026Songruo Chen, Yuqian Fu, Yu Li +7Egocentric Video Understanding Egocentric Video QA
Fudan University · KAUST · Universiti Malaya +4

May 13, 2026J/K move · Enter open · S save

Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning

Qinchuan Cheng, Zhantao Gong, Pengzhan Sun, Angela Yao+2

Xi’an Jiaotong University · Nankai University · National University of Singapore+1

Belief-Space Planning Long-Horizon Agent Evaluation AI Agent Benchmarks

Open paper AskPDF arXiv

Embodied agents in household environments must plan under partial observation: they need to remember objects, track state changes, and recover when actions fail. Existing benchmarks only partially test this ability. Egocentric video datasets capture realistic human activities but remain passive, while interactive simulators support execution but rely on synthetic scenes and hand-crafted dynamics, introducing a sim-to-real gap and often assuming fully observable state. We introduce Ego2World, an executable benchmark that turns egocentric cooking videos into executable symbolic worlds governed by graph-transition rules. Built on HD-EPIC, Ego2World derives reusable transition rules from video annotations and executes them in a hidden symbolic world graph. During evaluation, the simulator maintains the hidden world graph, while the agent plans over its own partial belief graph using only local observations and execution feedback. This separation forces agents to update memory and replan without observing the true world state. Experiments show that action-overlap scores overestimate physical-state success, and that persistent belief memory improves task completion while reducing repeated visual exploration -- suggesting that belief maintenance should be a first-class target of embodied-agent evaluation.

Abstract

AI Agent Benchmarks

Explore similar work

Ego2World: Compiling Egocentric Cooking Videos into Executable Worlds for Belief-State Planning

Grounded World Model: Latent Planning with Language Goals

EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions