Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

Patrick Amadeus Irawan; Iskandar Muda Rizky Parlambang; Rava Maulana; Qinrong Cui; Erland Hilman Fuadi; Zayd M. K. Zuhri; Nanda Ryaas Absar; Ahmed Elshabrawy

cs.CVOct 1, 2026

Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation

Authors: Patrick Amadeus Irawan, Iskandar Muda Rizky Parlambang, Rava Maulana, Qinrong Cui, Erland Hilman Fuadi, Zayd M. K. Zuhri, Nanda Ryaas Absar, Ahmed Elshabrawy, +5 more

Organizations: Mohamed bin Zayed University of Artificial Intelligence · Independent Researcher · Nanyang Technological University · University of North Carolina at Chapel Hill

Abstract

Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must not only generate visually appealing frames, but also predict how environments dynamically evolve when executing goal-directed actions. While evaluating these capabilities is crucial, existing benchmarks focus mainly on single short actions or step-by-step instructions. This leaves multi-step physical reasoning underexplored, especially in egocentric video generation that requires planning to simulate proper execution to accomplish high-level goals by carrying out multiple real-world manipulations. We introduce Ego2Act, a goal-directed benchmark featuring 2,640 videos from 110 real-world tasks across day-to-day settings, varying object clutter and multi-step complexity. Given an initial scene image and a high-level goal, Ego2Act evaluates whether video generation models can produce realistic egocentric videos of a hand manipulating objects to carry out the task. To support scalable evaluation, we also introduce Ego2ActJudge, a reference-free evaluation pipeline that achieves better task completion and physics plausibility evaluation alignment with human consensus compared to relevant baselines. Our findings reveal that models' generated simulations often skip or partially execute steps, leaving later steps missing dependent states, which leads to unfulfilled goal. Furthermore, models consistently fail at fine-grained physical dynamics, particularly during complex object manipulation and persistent world modeling. We hope Ego2Act provides a rigorous testbed for advancing video models toward physically plausible, goal-directed simulation.

Figures & tables

Figure 1: Ego2Act assesses whether different attempts at the same goal complete the required actions through physically plausible interactions. The trajectories share an initial scene and goal, with highlighted task and physics errors along each execution.

Benchmarks	Real-World Egocentric	Multi-Step Manipulation	Open-Ended Goal Planning	Reference-Free Evaluation
PQSG ( Pothiraj et al., 2026 )	$\times$	$\times$	$\times$	$\checkmark$
WorldModelBench ( Li et al., 2025 )	$\times$	$\times$	$\checkmark$	$\checkmark$
WR-Arena ( PAN Team et al., 2026 )	$\times$	$\times$	$\checkmark$	$\checkmark$
RBench ( Deng et al., 2026 )	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
EgoWM ( Bagchi et al., 2026 )	$\checkmark$	$\checkmark$	$\times$	$\times$
EgoForge ( Shen et al., 2026 )	$\checkmark$	$\checkmark$	$\times$	$\times$

Table 1: Benchmark comparison between Ego2Act and representative video and world model benchmarks. Unlike other benchmarks that do not target egocentric, real-world manipulation (e.g., PQSG, WR-Arena) or egocentric benchmarks that depend on reference trajectories (e.g., EgoWM, EgoForge), Ego2Act evaluates egocentric real-world generated videos by prompting models to perform multi-step embodied manipulations scored with our reference-free evaluator.

Figure 2: Composition and descriptive statistics of Ego2Act . (a) Domain composition of the 110-case benchmark. Panels (b–e) summarize the feature distributions: (b) action-family co-occurrence and coverage, (c) human control videos duration, (d) involved solid-object counts, and (e) case-mean observed operations. Co-occurrence means joint presence within a case, and observed operations describe number of subaction per case. (f) unique action family count per case.

Figure 3: Overview of Ego2Act and Ego2ActJudge . The collection pipeline verifies task specifications and human recordings to produce cases containing an $f_{1}$ , $G$ , human controls, and metadata. Video models receive the $f_{1}$ and $G$ to generate task executions. Ego2ActJudge derives subgoals, examines $V$ , and applies the task and physics rubrics to produce per-component & joint scores. The rubric stages are the Task gates T1–T3 and Physics gates P1–P4 of Section 4 .

	Human Judge †		WRA	PQSG		RBench		WMBench		VideoScore		Ego2ActJudge
Gen. / Eval.	T / P	Agg.	Ori.	T / P	Ori.	T / P	Ori.	T / P	Ori.	T / P	Ori.	T / P	Agg.
Human (+)	100.0 / 100.0	100.0	95.6	95.5 / 98.6	97.3	76.7 / 80.4	75.7	63.4 / 78.7	76.2	83.3 / 84.3	87.5	93.9 / 98.1	95.4
Human ( $-$ )	59.6 / 94.1	73.4	59.7	61.7 / 80.6	76.2	47.4 / 73.8	43.6	58.5 / 80.5	75.8	83.9 / 85.3	88.2	57.8 / 91.6	70.1
Grok-1.5	56.6 / 47.9	49.9	78.7	82.5 / 95.1	88.6	67.4 / 78.2	64.3	73.7 / 73.1	75.9	88.2 / 87.9	90.6	67.8 / 79.4	69.5
Seedance-2.0	67.8 / 63.0	64.0	88.6	90.2 / 97.3	93.8	74.1 / 80.3	72.4	69.7 / 73.0	74.7	89.3 / 89.8	92.2	79.9 / 91.1	83.3
Kling-v3-Pro	59.3 / 64.2	59.6	77.0	77.1 / 94.2	86.4	65.0 / 78.9	61.9	67.3 / 74.1	74.6	90.5 / 91.2	93.4	64.0 / 88.9	72.5

Table 2: Cross-Benchmark Performance Leaderboard on Ego2Act . Sub-component Task / Physics splits are reported alongside average of per video geometric mean or benchmark official score, with human controls excluded from best and second-best model rankings.

Panel of Figure 4 — Figure 4: Component score correlation and performance consistency across trials on 25-cases subset

(a) Rubric Agreement
Metric $\uparrow$	Final	Task	Physics
ICC	0.863	0.831	0.834
Kripp.’s $\alpha$	0.745	0.691	0.697

Table 3: Inter-annotator Agreement and Alignment.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 5 — Table 5: Features of the Ego2Act data. Recording duration, trajectory description, and number of steps are recorded for individual recordings, and all other features describe a case as a whole.

Domain	Definition	Example objects
Kitchen and food preparation	Preparing, serving, or storing food and drink, and handling or cleaning the tableware and utensils used for them.	Spoons, forks, knives, plates, bowls, mugs, cups, jars, oil bottles, eggs, and food such as tofu or broccoli.
Household organization and storage	Tidying, storing, or rearranging everyday household items such as clothing, toys, and home accessories.	Keys, toy cars, shoe boxes, storage boxes, trash cans, pillows, garments and laundry bags, candles and lighters, clocks, remote controls, and light bulbs.
Office and study workspace	Setting up, organizing, or using items on a desk for work or study, including stationery, books, and electronic devices with their accessories.	Laptops, keyboards, mice, books, notebooks, pens, markers, highlighters, erasers, staplers, scissors, calculators, headphones, and chargers.
Personal care and utilities	Handling hygiene, grooming, and health items, and packing personal belongings into pouches or bags outside an office or study context.	Toothbrushes, toothpaste tubes, pouches and bags, eye drops, lipsticks, makeup jars, cotton pads, spray bottles, face wash tubes, and bandages.
Others	Everyday activities outside the four domains above, such as handling musical instruments, footwear, money, or games.	Guitars, shoes and sandals, backpacks, wallets, banknotes and coins, board-game pieces, matryoshka dolls, and tennis rackets.

Appendix

Table 6: The five domains of Ego2Act , with a definition and example objects from their cases.

ID	Action family	Definition
A1	Relocation	Moving an object to a new position or arranging objects relative to one another (place, move, stack, align, sort).
A2	Containment	Putting an object into a container or enclosure, or taking it out (insert, remove, pack, retrieve).
A3	Opening / closing	Changing whether a container, lid, cap, or latch is open or closed (open, close, cap, uncap, latch).
A4	Attachment	Joining two objects into a lasting connection, such as attaching a part or plugging in a cable, or undoing such a connection (attach, plug in).
A5	Shape / orientation	Changing the shape or orientation of an object (turn upright, fold, flip).
A6	Transfer / mixing	Moving a liquid or granular material from one place to another, or mixing materials (pour, dispense, scoop, stir).

Appendix

Table 7: Action families used to group the annotated action types. Examples in parentheses are action types that occur in the benchmark.

Observation	Scoring consequence
Correct-source reach, followed only by preparation	T2 fails: $t_{i}=1$ .
Requested transfer executed, but required destination relation fails	T3 fails: $t_{i}=2$ .
Subgoal levels $(0,3,3)$ , with the enabling first subgoal skipped	$T=2$ .
Relevant action is absent and no movement is judgeable	Task level 0, Physics unattempted, $S=0$ .
Observable entity unexpectedly duplicates	P1 fails: $p_{i}=0$ .
A physically coherent wrong placement remains stable	Physics may score 4 despite Task failure.

Appendix

Table 8: Illustrative rubric outcomes. Gate failures assume preceding gates passed.

Generator	Human, panel	Ego2ActJudge , panel	Ego2ActJudge , benchmark
Grok-1.5	49.9 (4)	72.9 (3)	69.5 (4)
Seedance-2.0	64.3 (1)	88.0 (1)	83.3 (1)
Kling-v3-Pro	59.6 (2)	76.7 (2)	72.5 (2)
Wan-2.7	44.3 (5)	68.6 (5)	71.4 (3)
MiniMax-H3	55.3 (3)	70.7 (4)	59.3 (5)
Cosmos-3	4.0 (6)	14.4 (6)	13.3 (6)

Appendix

Table 9: Mean Final score (rank) per generator. The first two columns use the same panel videos, and the last column is the Ego2ActJudge Agg. of Table 2 over all benchmark videos.

Group	Task	Physics	Unattempted ( $S=0$ )	Final
Grok-1.5	100.0	99.4	0.3	99.7
Seedance-2.0	100.0	98.8	0.0	98.8
Kling-v3-Pro	100.0	96.0	0.9	96.9
Wan-2.7	100.0	98.5	0.6	99.1
MiniMax-H3	100.0	95.7	2.8	98.5
Cosmos-3	100.0	61.5	36.4	97.9

Appendix

Table 10: Score coverage of Ego2ActJudge over all benchmark videos (%). Physics is missing when no attempted interaction can be inspected. Unattempted videos (Task 0, no Physics) receive

S=0

and count as scored. The remaining missing Final scores come from unresolved gate decisions or from videos with

T>0

but no inspectable interaction.

Gen. / Eval.	Human	WR-Arena	PQSG	RBench	WMBench	VideoScore	Ego2ActJudge
Human (+)	100.0	95.6 [93.5, 97.6]	95.5 [93.4, 97.5]	76.7 [73.9, 79.6]	63.4 [59.0, 67.9]	83.3 [80.4, 86.0]	93.9 [91.6, 95.8]
Human ( $-$ )	59.6 [54.7, 64.9]	59.7 [57.1, 62.4]	61.7 [58.1, 65.1]	47.4 [45.0, 50.0]	58.5 [54.4, 62.9]	83.9 [80.8, 86.6]	57.8 [54.6, 60.8]
Grok-1.5	56.6 [50.3, 62.9]	78.7 [74.8, 82.8]	82.5 [78.7, 85.9]	67.4 [63.9, 70.7]	73.7 [69.8, 77.8]	88.2 [85.4, 90.4]	67.8 [63.7, 72.2]
Seedance-2.0	67.8 [58.4, 76.0]	88.6 [86.0, 91.5]	90.2 [87.4, 93.0]	74.1 [70.7, 77.4]	69.7 [65.3, 74.0]	89.3 [87.2, 91.4]	79.9 [76.2, 83.4]
Kling-v3-Pro	59.3 [52.8, 66.1]	77.0 [73.1, 80.2]	77.1 [72.9, 81.3]	65.0 [61.9, 67.9]	67.3 [63.4, 71.4]	90.5 [88.1, 92.3]	64.0 [59.4, 68.5]
Wan-2.7	55.5 [48.7, 62.7]	80.5 [77.1, 84.0]	80.7 [77.0, 84.2]	67.1 [63.5, 70.8]	75.1 [70.8, 79.3]	92.2 [90.1, 93.9]	67.8 [63.7, 72.5]

Appendix

Table 11: Task scores (0–100) with 95% confidence intervals. Values match Table 2 , and subscripts give 95% case-level bootstrap intervals (1,000 resamples of cases, keeping all videos of a sampled case). Human uses the 600-video panel, while automated evaluators use every benchmark video they scored. Human (+) recordings are verified as successful executions during dataset construction (Section 3.2 ) and receive the rubric maximum.

Gen. / Eval.	Human	WR-Arena	PQSG	RBench	WMBench	VideoScore	Ego2ActJudge
Human (+)	100.0	–	98.6 [97.3, 99.5]	80.4 [77.8, 83.3]	78.7 [76.1, 81.1]	84.3 [80.9, 87.5]	98.1 [96.8, 99.2]
Human ( $-$ )	94.1 [88.7, 98.0]	–	80.6 [77.2, 84.2]	73.8 [71.2, 76.4]	80.5 [78.2, 82.7]	85.3 [82.0, 88.0]	91.6 [89.5, 93.8]
Grok-1.5	47.9 [41.3, 54.9]	–	95.1 [93.4, 96.6]	78.2 [75.4, 81.3]	73.1 [70.8, 75.3]	87.9 [85.0, 91.0]	79.4 [75.3, 83.4]
Seedance-2.0	63.0 [56.4, 69.7]	–	97.3 [96.0, 98.5]	80.3 [77.6, 83.0]	73.0 [70.9, 75.3]	89.8 [87.1, 92.3]	91.1 [88.4, 93.2]
Kling-v3-Pro	64.2 [58.1, 70.8]	–	94.2 [92.2, 95.9]	78.9 [76.2, 81.5]	74.1 [72.0, 76.4]	91.2 [88.9, 93.3]	88.9 [86.4, 91.0]
Wan-2.7	41.4 [34.2, 49.3]	–	93.4 [91.3, 95.3]	76.9 [74.0, 79.8]	75.9 [73.1, 78.5]	94.0 [92.0, 95.6]	80.5 [76.9, 84.0]

Appendix

Table 12: Physics scores (0–100) with 95% confidence intervals. Values match Table 2 , and subscripts give 95% case-level bootstrap intervals (1,000 resamples of cases, keeping all videos of a sampled case). Human uses the 600-video panel, while automated evaluators use every benchmark video they scored. Human (+) recordings are verified as successful executions during dataset construction (Section 3.2 ) and receive the rubric maximum. WR-Arena has no Physics component.

Generator	Household	Kitchen	Office	Personal care	Other
Grok-1.5	63.5	76.4	75.6	82.4	75.7
Seedance-2.0	84.7	86.2	78.4	86.8	93.6
Kling-v3-Pro	74.9	75.6	69.6	80.1	74.3
Wan-2.7	69.4	73.8	67.3	60.1	82.1
MiniMax-H3	64.2	62.1	74.3	69.6	70.6
Cosmos-3	13.0	18.8	21.9	17.9	7.6

Appendix

Table 13: Domain profiles of generated rollouts. Values are case-balanced Final scores under Ego2ActJudge , and each domain contains five cases.

Feature	Group	Cases	Score
Action	Relocation/arrangement	20	65.1
Action	Containment/retrieval	16	67.4
Action	Opening/closing	17	64.5
Action	Attachment/connection	7	64.3
Action	Shape/orientation change	6	64.1
Action	Material transfer/mixing	3	59.0

Appendix

Table 14: Lower scores in some interaction categories motivate targeted failure analysis, but the groups are small and overlap. Scores are descriptive case-balanced Final scores under Ego2ActJudge . Clutter labels cover all 25 cases.

Figure 6: Reference action count and duration have weak observed associations with execution scores. Each point is one of 25 cases, and the vertical axis is its case-balanced Final score under Ego2ActJudge . Spearman correlations are

0.06

for observed operations and

0.04

for reference duration. These associations do not establish which mechanism causes failure.

Figure 7: The tested feature associations do not survive Benjamini–Hochberg correction in this sample. (a) Associations between case features and generator scores. (b) Score differences when an interaction type is present versus absent. (c) Associations between case features and absolute judge error. Nonsignificant results do not establish equal difficulty or uniform evaluator reliability. Table 15 reports the displayed adjusted values. Intervals are 95% case bootstraps, and

q

-values are Benjamini–Hochberg adjusted within each panel.

Interaction-associated score differences
Case feature	Generator-score $q$	Judge-error $q$
Observed operations	0.848	0.702
Involved objects	0.529	0.702
Reference duration	0.848	0.086
Action-family diversity	0.529	0.702
Clutter (ordinal)	0.529	–

Appendix

Table 15: Adjusted

q

-values for the feature diagnostics. No displayed association meets

q<0.05

. The interaction tests shown here are those in Figure 7 b, which cover every action family with at least three cases on each side. Values are Benjamini–Hochberg adjusted within each panel of Figure 7 .

Figure 8: Subgoal-level Task and Physics outcome distributions by action type. Judgments are pooled over the six generators on 294 case–seed units and grouped by the primary action family of each subgoal. Families with fewer than 100 Task subgoals are omitted. T

k

and P

k

denote score levels, so T0 means the first Task gate failed and P0 means the first Physics gate failed. Physics shares are over judgeable subgoals.

Figure 9: Subgoal-level performance by action type and video generator. The heatmaps report Task completion (T3), skipped Task operations (T0), and physically valid transitions among judgeable Physics subgoals (P4), where T $k$ and P $k$ denote score levels. Cells with fewer than ten judgments are left blank.

	Cosmos-3		Wan-2.7
Case	Original	Expanded 1 / 2 / 3	Original	Expanded 1 / 2 / 3
aromatic_cube_leaf	17.6	11.1 / 37.9 / 0.0	54.4	51.5 / 0.0 / 66.4
double_shoe_box	20.8	42.2 / 14.1 / 34.8	69.8	100.0 / 96.5 / 100.0
egg_candy_air_fryer	37.0	54.2 / 60.7 / 53.7	94.3	98.3 / 96.1 / 98.9
paper_binder_book	7.5	0.0 / 8.6 / 34.5	55.7	89.0 / 86.5 / 95.8
remote_marker_stand	43.2	11.1 / 0.0 / 15.7	77.8	100.0 / 100.0 / 93.9

Appendix

Table 16: Prompt-expansion diagnostic. Mean Ego2ActJudge Final score (three seeds) with the original prompt and with each of three expanded prompts, scored against the original goal. Unattempted videos count as 0.

Figure 10: Subgoal-level score-level profiles across video generators. Bars show the distribution of Task and Physics levels for each generator on the same 294 case–seed units. Task and Physics subgoals are summarized independently, and Physics N/A is retained.

Figure 11: Representative errors in generated action rollouts. Task errors: omitted prerequisites (a), incomplete outcomes (b), and incomplete outcomes in domain-specific operations (c). Physics errors: object duplication or identity changes (d), penetration through closed lids (e), and motion incompatible with object mechanisms (f). Each example pairs an earlier frame with a frame showing the error, and red circles highlight the relevant regions.

Failure mode	Observable pattern	Representative case IDs
Task failures
Skipped prerequisite operations	A necessary enabling action is absent, while a dependent operation proceeds.	• cosmos_3__coffee_beans__seed_202 • grok_imagine_video_1_5__three_cups_and_water__seed_202 • kling_v3_pro__air_freshener_dispenser__seed_101 • seedance_2_0__staple_paper__seed_101 • wan_2_7__pour_milk__seed_101
Incomplete outcomes	The relevant operation is attempted, but the required final state or object relation is not established.	• grok_imagine_video_1_5__stack_cap_bottle__seed_101 • kling_v3_pro__bottle_cap__seed_101 • minimax_h3__toothbrush_case__seed_101 • seedance_2_0__headphone_mouse_book__seed_303 • seedance_2_0__desk_mat_tablet__seed_303
Physics failures: state
World-state inconsistency	Object identity, appearance, material, geometry, or assembly state is not preserved across frames.	• cosmos_3__eyeshadow_palette__seed_303 • grok_imagine_video_1_5__kendama_can__seed_202 • kling_v3_pro__coffee_beans__seed_202 • minimax_h3__screwdriver_wrench__seed_202 • seedance_2_0__dock_switch_game__seed_202
Object-count inconsistency	The scene inventory changes without a corresponding addition, removal, division, or assembly event.	• grok_imagine_video_1_5__router_laptop_cable__seed_202 • kling_v3_pro__lava_toast_microwave__seed_101 • minimax_h3__three_caps__seed_202 • wan_2_7__trophy_luggage__seed_101 • kling_v3_pro__stack_blocks__seed_202

Appendix

Table 17: Overview of recurring error categories used in our qualitative analysis. For each failure mode, the final column indexes representative cases presented in the following sections. The categories and examples are not exhaustive.

Panel of Figure 12 — Figure 12: Examples of skipped prerequisite operations.

Panel of Figure 13 — Figure 13: Examples of incomplete outcomes.

Panel of Figure 14 — Figure 14: Examples of world-state inconsistency.

Panel of Figure 15 — Figure 15: Examples of object-count inconsistency.

Panel of Figure 16 — Figure 16: Examples of causal and boundary violations.

Panel of Figure 17 — Figure 17: Examples of manipulation and coupling failures.

Panel of Figure 18 — Figure 18: Examples of mechanism violations.

Axis	Variant	MAE $\downarrow$	$\Delta$ MAE
Task	Full rubric	0.292	0.000
	No T1	0.497	+0.205
	No T2	0.324	+0.032
	No T3	0.331	+0.039
	No hierarchy	0.306	+0.014
Physics	Full rubric	0.432	0.000

Appendix

Table 18: Exact human-rubric ablation results. Task uses 28 pilot videos and a native

0

–

3

scale, and Physics uses 24 pilot videos and a native

0

–

4

scale. Deltas subtract the full-rubric MAE, so positive values indicate worse agreement. Removed gates automatically pass.

Axis	Scoring	$r$ $\uparrow$	$\tau$ $\uparrow$	CCC $\uparrow$	MAE $\downarrow$	Bias
Task	Sequential	0.64	0.47	0.61	19.7	+8.3
	Independent	0.64	0.47	0.59	20.6	+11.8
	$\Delta$	$r$ : $-0.00$ [ $-0.02$ , $+0.02$ ]			$+0.9$ [ $+0.2$ , $+1.6$ ]
Physics	Sequential	0.54	0.38	0.44	29.8	+23.2
	Independent	0.54	0.38	0.33	33.8	+31.2
	$\Delta$	$r$ : $-0.01$ [ $-0.03$ , $+0.02$ ]			$+4.0$ [ $+2.6$ , $+5.4$ ]

Appendix

Table 19: Judge-side gating ablation. Ego2ActJudge is rerun with every gate asked independently, and both rules are scored from the same answers. Rows compare each rule with the human consensus on the same videos.

\Delta

is independent minus sequential with 95% case-bootstrap intervals.

	Teleport	Swap	Ghost
Detection	6.9%	10.3%	55.2%
Physics change	$-2.9$ [ $-7.5$ , 0.0]	$-2.6$ [ $-6.7$ , 0.0]	$-29.8$ [ $-42.7$ , $-17.9$ ]
Task change	$-2.4$ [ $-9.1$ , 2.7]	$-1.9$ [ $-5.2$ , 1.1]	$-3.1$ [ $-6.3$ , $-0.6$ ]
Main failing gate	P1	P3	P1

Appendix

Table 20: Physics negative controls on 30 real recordings. Detection is the share of pairs whose Physics score falls below the unedited rerun (29 pairs with a Physics score). Changes are corrupted minus unedited, with 95% bootstrap intervals over recordings. False alarms on unedited videos: 0 of 29.

Evaluator	MAE $\downarrow$	Bias $\to 0$	CCC $\uparrow$	$r\uparrow$	$\tau\uparrow$
WR-Arena §	28.2	+20.0	0.34	0.42	0.32
PQSG	37.2	+32.8	0.16	0.31	0.20
RBench	28.0	+5.4	0.39	0.40	0.28
WMBench	31.4	+24.7	0.08	0.16	0.11
VideoScore †	45.2	+43.3	$-$ 0.02	$-$ 0.10	$-$ 0.15
Simple VQA	37.9	+29.2	0.22	0.32	0.21

Appendix

Table 21: Final-score alignment with the human consensus on the 600-video panel. Values match Table 3 . MAE and signed bias (evaluator minus human) are in

0

–

100

units. Bold marks the best automated point estimate, which does not establish statistical significance.

\dagger

denotes VideoScore’s five-aspect mean diagnostic. § WR-Arena has no Final score and is compared on Task. ‡ Human reference: each rating compared with the consensus of the other raters on the same video.

	Task		Physics
Evaluator	$r\uparrow$	$\tau_{b}\uparrow$	$r\uparrow$	$\tau_{b}\uparrow$
WR-Arena	0.42	0.32	–	–
PQSG	0.44	0.32	0.23	0.13
RBench	0.47	0.34	0.37	0.25
WMBench	0.18	0.14	0.15	0.12
VideoScore	$-$ 0.14	$-$ 0.15	$-$ 0.08	$-$ 0.15

Appendix

Table 22: Task and Physics alignment with the human consensus on the 600-video panel. Bold marks the highest automated point estimate. A dash indicates an unsupported axis.

Figure 19: Better correlation does not remove calibration error. The detailed alignment view separates MAE, signed bias, concordance, and Pearson correlation. Ego2ActJudge improves most point estimates relative to the evaluated baselines, but retains positive bias and a gap to the human reference. Table 21 gives exact values.

Quantity	Axis	SD	Test–retest $r$
Judge repeatability	Task	8.1	0.93
	Physics	15.7	0.76
	Final	10.4	0.89
Generation variability	Final	11.4–18.4	–

Appendix

Table 23: Judge repeatability and generation variability. Judge rows compare two independent runs of Ego2ActJudge on the same videos, with subgoals re-derived in each run. Single-run SD is

\sqrt{\overline{d^{2}}/2}

for run differences

d

. Generation variability is the SD of the Ego2ActJudge Final score across the three seeds of a case, averaged over cases, and it includes judge noise. Both are per-video quantities, not uncertainties of model means.

Explore similar work

Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?
Jun 3, 2026Rui Zhao, Kaiming Yang, Jifeng Zhu +6Generative Video Models Human Motion Generation
Show Lab, National University of Singapore · University of Oxford · Tencent.
EgoForge: Goal-Directed Egocentric World Simulator
Mar 20, 2026Yifan Shen, Jiateng Liu, Xinzhuo Li +9Egocentric Video Video World Models
University of Illinois Urbana-Champaign · University of California San Diego
EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
Jul 30, 2026Zexuan Yan, Yuzhou Wu, Yue Ma +9Egocentric Video Camera-Conditioned Video Generation
1Shanghai Jiao Tong University · 2Alibaba Group · 3Tianji KernalMind Co., Ltd. +4

Jun 3, 2026, cs.CVJ/K move · Enter open · S save

Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?

Rui Zhao, Kaiming Yang, Jifeng Zhu, Siyang Chen+5

Show Lab, National University of Singapore · University of Oxford · Tencent.

Generative Video Models Human Motion Generation Video Generation Robot Systems

Open paper AskPDF arXiv

Video generation models have made impressive strides in synthesizing visually compelling content, yet their outputs remain confined to the virtual domain. A natural question follows: how well do these models reflect the physical world when their generated videos leave the screen and enter reality? We propose robotic manipulation as a concrete, measurable window onto this question: if a model has truly internalized physical laws, the motion it depicts should translate into executable robot behavior. We introduce Dream

.

exe, an evaluation framework that operationalizes this criterion through a video-to-execution pipeline. Given a scene image and a task description, Dream

.

exe synthesizes a manipulation video, converts the generated motion into robot trajectories, and executes them in a physics simulator, yielding a grounding signal that purely visual metrics cannot offer. Using this pipeline, we evaluate 8 models spanning frontier closed-source generators, open-source generators, and robot-specific models. Our benchmark covers 101 manually curated manipulation tasks at three levels of physical complexity, measured across visual quality, trajectory fidelity, and execution success. Encouragingly, several models achieve measurable execution success, suggesting that generative priors learned from internet-scale data already encode meaningful physical knowledge. Yet visual quality proves a poor predictor of executability, exposing a dimension of model capability that standard visual evaluations do not capture. Dream

.

exe will be open-sourced at https://github.com/showlab/Dream.exe.

Abstract

Explore similar work

Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?

EgoForge: Goal-Directed Egocentric World Simulator

EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE