EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions

Songruo Chen; Yuqian Fu; Yu Li; Mingkang Dong; Sifan Zhou; Bin Ren; Xu Zheng; Tianwen Qian

cs.CVOct 13, 2026

EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions

Authors: Songruo Chen, Yuqian Fu, Yu Li, Mingkang Dong, Sifan Zhou, Bin Ren, Xu Zheng, Tianwen Qian, +2 more

Organizations: Fudan University · KAUST · Universiti Malaya · CMU · MBZUAI · Great Bay University · ECNU

Abstract

Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their tight coupling in real-world interaction. We propose the Ego-Action-World interaction loop as a unifying principle: the wearer perceives the world, forms goals, acts, changes the world, and uses feedback to guide subsequent behavior. Based on this view, we introduce EgoOmniWorld, a benchmark organized around four complementary interaction structures: Node captures ego, action, and world states; Edge captures directed relations among them; Chain captures linked dependencies; and Loop captures feedback-closed interactions. We develop an interaction-first, evidence-grounded data pipeline that extracts structured interactions from egocentric videos, grounds them in visual and auditory evidence, and generates questions according to their interaction structure. Required modalities and single- or multi-turn formats follow naturally from the supporting evidence and dependencies. EgoOmniWorld contains 24,166 QA groups, including 20,284 training and 3,882 test samples. Experiments on representative open- and closed-source multimodal large language models reveal substantial and uneven limitations across interaction structures. Models trained on EgoOmniWorld improve in-domain interaction understanding and generalize to existing egocentric benchmarks. EgoOmniWorld provides a unified foundation for evaluating and advancing egocentric video understanding. Data, code, and models will be released.

Figures & tables

Figure 1: EgoOmniWorld unifies fragmented egocentric evaluation around the Ego–Action–World interaction loop, organizing questions into Node, Edge, Chain, and Loop structures.

Benchmark	Input	Node	Edge	Chain	Loop	Dialogue
EgoSchema	Video	A, W	A $\rightarrow$ E	$\sim$	–	single-turn
EgoTempo	Video	A, W	E $\rightarrow$ A	$\sim$	–	single-turn
EgoTaskQA	Video	E, A, W	A $\rightarrow$ E; A $\rightarrow$ W; W $\rightarrow$ A	$\sim$	–	single-turn
EgoToM	Video	E, A	A $\rightarrow$ E	$\sim$	–	single-turn
MultiHop-EgoQA	Video	A, W	–	$\sim$	–	single-turn
EgoLifeQA	Video + Audio	E, A, W	W $\rightarrow$ E ; E $\rightarrow$ A	$\sim$	–	single-turn

Table 1: Representative coverage of existing egocentric benchmarks under the EgoOmniWorld taxonomy. E, A, and W denote Ego, Action, and World; arrows indicate Edge directions.

\checkmark

: explicit coverage;

\sim

: related but not explicitly structured.

Figure 2: The EgoOmniWorld taxonomy. Left: an egocentric interaction loop. Middle: the Ego–Action–World structure, comprising three node states and six directed edge relations. Right: four complementary interaction structures, from Node to Loop, with example questions.

Figure 3: Overview of the proposed data engine. Given source egocentric videos, the four stages extract local Ego–Action–World interactions, ground them in multimodal evidence, aggregate episodes and link dependencies, and generate and validate evidence-grounded QA.

Figure 4: Dataset statistics and representative QA examples in EgoOmniWorld. Top: distributions of (a) clip duration, (b) interaction scopes, (c) Node and Edge semantic categories, (d) dialogue length, and (e) required evidence modality. Bottom: representative questions, answers, supporting evidence, and video frames illustrating individual states, directed relations, linked dependencies, and feedback-closed interactions. Each complete multi-turn dialogue counts as one QA group.

Closed-source models
Model	Node	Edge	Chain	Loop	Overall
Gemini 3.8 Flash	2.86/54.74	3.03/60.26	3.34/66.41	3.54/69.29	3.14/61.51
Qwen3.5-Omni-Plus	2.18/37.31	2.64/50.76	2.80/56.25	2.90/57.89	2.59/49.15
Qwen3.8-Omni-Flash	2.69/48.62	2.91/56.09	3.06/57.48	3.03/58.27	2.89/54.58
Open-source models
MiniCPM-o 2.6 8B	2.09/30.28	2.01/34.19	2.25/38.28	2.21/37.83	2.11/34.38

Table 2: In-domain performance on EgoOmniWorld. We report mean score (0–5) and accuracy (%) across the four interaction structures and overall.

Model	EgoSound	EgoToM	EgoTempo
VideoLLaMA2 7B AV	20.5	–	–
SALMONN2+ 7B	36.0	–	23.60
MiniCPM-o 2.6-8B	40.4	57.51	20.60
Qwen2.5-Omni-7B (Base)	31.33	55.95	20.40
Qwen2.5-Omni-7B + EgoOmniWorld-LoRA	39.41	58.22	24.80
Improvement	+8.08	+2.27	+4.40

Table 3: Cross-benchmark transfer accuracy (%) on existing egocentric benchmarks. Bold highlights the improved results within our base–LoRA pair. See Appendix E.2 for evaluation details.

Figure 5: Fine-grained cross-benchmark transfer analysis under the EgoOmniWorld taxonomy. We compare Qwen2.5-Omni-7B before (solid bars) and after (hatched bars) EgoOmniWorld LoRA adaptation on EgoSound, EgoToM, and EgoTempo. Colors distinguish benchmarks, and accuracy is computed on matched questions within each dataset–category group. Empty categories indicate no evaluated questions, and groups with fewer than three questions are omitted.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Level	Category	Train	Test	Total
Node	Ego State	1,911	345	2,256
	Action Status	4,549	890	5,439
	World State	1,966	365	2,331
	Subtotal	8,426	1,600	10,026
Edge	Goal Planning	242	40	282
	Intent Attribution	644	127	771

Appendix

Table 4: Sample counts by task category and split. A multi-turn dialogue counts as one QA group.

Figure 6: Representative QA examples in EgoOmniWorld. The figure contains two examples for each of Ego State, Action Status, World State, Goal Planning, Intent Attribution, Action Effects, Action Grounding, Intent Formation, and World Perception, together with two Chain and two Loop dialogues. Frames come from the corresponding dataset clips; timestamps are clip-relative. Displayed questions and answers are shortened for readability, and Chinese multi-turn examples are translated into English. Each complete dialogue counts as one QA group. The examples illustrate the annotations rather than establish exhaustive validation of their assigned categories.

Statistic	Value
Videos in the original clustering	8,931
Original clusters	160
Selected source videos (deduplicated)	1,000
Selected videos matched to original clusters	1,000
Clusters covered	159
Cluster coverage	99.375%

Appendix

Table 5: Observed Ego4D source-video selection statistics.

ID	Topic	All	Selected	Original keywords (first six)
0	Crochet	10	5	yarn, fabric, hand, crochet, left, right
1	Kitchen weighing and container washing	11	4	hand, table, sink, jug, summary, kitchen
2	Paper crafts and gluing	44	6	paper, craft, table, hand, right, drops
3	Multiplayer card games	148	10	man, table, hand, cards, right, card
4	Reading and bookshelf organization	45	6	book, books, shelf, summary, floor, phone
5	Supermarket shopping	95	10	shopping, basket, man, woman, bag, paper

Appendix

Table 6: Original Ego4D clusters and selected source videos. Topic summaries are interpretive, not human-annotated activity classes.

Evidence type	Definition
Direct evidence	Evidence introduced and used in the current turn.
Inherited evidence	Evidence established in a preceding turn and still required by the current reasoning step.
New evidence	Additional evidence introduced in the current turn.

Appendix

Table 7: Evidence types in multi-turn QA.

Category	Query target and evidence requirements
Ego State	Characterize interaction-relevant attributes of the wearer.
Action Status	Describe actions that have been performed.
World State	Characterize object attributes, spatial configurations, locations, and world states.
Intent Attribution	Connect observed actions to evidence-supported goals or intentions; the expected answer must be supported by behavior, speech, task context, or other available evidence.
Action Effects	Connect performed actions to observed world changes; evidence must contain both the action and its resulting state.
Action Grounding	Characterize how the current environment or world state influences feasible behavior.

Appendix

Table 8: Semantic categories, query targets, and evidence requirements.

Level	Retention criteria	Failure cases
Node	A clearly identifiable target state or fact localized to an appropriate evidence interval. Attribute questions require explicit evidence for the queried attribute; historical questions require observations of the same tracked entity or ego trajectory.	Unclear targets, missing attribute evidence, or historical observations that cannot be associated with the same entity or ego trajectory.
Edge	Clearly defined relation endpoints, a specified task direction, and supporting evidence; forward causation is not required for every edge. Intent Attribution requires behavioral, linguistic, or contextual support for attributing intention from action; Action Effects requires observable state changes; Action Grounding requires specific environmental conditions affecting action feasibility; Intent Formation requires external conditions influencing later intentions or decisions; Goal Planning is evaluated using only evidence before the prediction cutoff; World Perception requires world-perception evidence available from the ego’s perspective.	Insufficient relation endpoints or category-specific supporting evidence.
Chain	Multiple necessary information-bearing components forming an open path without feedback closure. Dependency explanations and semantic review verify the contribution of earlier information.	Mere temporal adjacency, thematic similarity, or shared objects; earlier components do not contribute to the answer; a complete loop is actually required but the sample remains labeled Chain.
Loop	Evidence covers the original goal, relevant action, world change, and perceptual feedback, supporting the return of feedback to the ego to confirm, adjust, or update the original goal. The answer must integrate this closed connection.	Only action and outcome, or only feedback and a later action; missing goal linkage, outcome use, or confirmation evidence; success inferred from an action ending, or feedback adjustment inferred from temporal order alone.
Cross-loop	Satisfies closure requirements and includes necessary bridging facts and entity/state correspondences between loops. Earlier-loop information contributes to a later loop and the final answer.	Cross-stage state transfer without closure evidence; unclear inter-loop connections; removing an earlier loop still permits a complete answer.

Appendix

Table 9: Structural filtering criteria.

Level	Permitted repairs	Required constraints and rejection conditions
Node	Adjust overly specific state descriptions to the finest granularity supported by evidence; clarify references; correct modality labels or evidence bindings.	Reject the sample if the target attribute is entirely unobservable.
Edge	Intent Attribution: generalize unsupported specific intentions to evidence-supported goals. Action Effects: refocus on actual world-state changes. Action Grounding/Intent Formation: clarify external conditions and the behaviors or goals they influence. Goal Planning: restrict the answer to a next goal or action category supported by evidence before the prediction cutoff.	Preserve the original relation type and task direction. If the relation remains unsupported, reject rather than downgrade to a Node question.
Chain	Remove redundant components, clarify existing information transfer, and correct answer leakage or referential ambiguity.	Preserve an open path of multiple necessary components. Do not add unrelated events to lengthen the chain or invent feedback to turn it into a Loop. Reject or review if the repaired sample no longer meets its original category.
Loop	Narrow overstated claims and clarify existing evidence of goals, outcomes, checking, or feedback use.	Preserve goal–action–outcome–perceptual-feedback–goal-assessment closure. Do not invent confirmation, mental updates, or successful outcomes. Reject or review if closure evidence is missing.
Cross-loop	Clarify entity references, bridging facts, and existing feedback connections.	Preserve both closure and inter-loop dependencies. Reject or review when identity, state transfer, or closure cannot be verified; do not convert the sample into an open chain while retaining its label.

Appendix

Table 10: Permitted repairs and constraints.

Figure 7: Web interface for human review of EgoOmniWorld. The interface combines video playback with questions, reference answers, evidence intervals, and structural annotations, and provides controls for recording quality checks and suggested revisions. The screenshot illustrates the interface, not aggregate review outcomes.

Closed-source models
	Node			Edge
Model	Ego State	Action Status	World State	Goal Planning	Intent Attribution	Action Effects	Action Grounding	Intent Formation	World Perception
Gemini 3.8 Flash	3.55 / 73.13	2.41 / 44.09	3.35 / 64.86	3.27 / 70.00	4.00 / 82.14	2.06/42.86	3.75 / 74.63	3.18 / 64.37	2.65 / 49.61
Qwen3.5-Omni-Plus	2.99/58.21	1.76/25.81	2.53/47.30	1.47/33.33	3.86/78.57	2.61 / 44.90	3.21/65.67	2.99/59.78	2.11/36.72
Qwen3.8-Omni-Flash	3.49/70.15	2.35/39.25	2.80/52.70	3.10/63.33	3.96/78.57	2.33/38.78	3.57/71.64	3.14/61.96	2.35/43.75
Open-source models

Appendix

Table 11: Test-set performance across the nine semantic categories. Cells show mean score / accuracy (%). Best scores and accuracies within each model group are bolded independently; ties are retained.

EgoSound
Original task category	Base	LoRA	$\Delta$
Sound-source identification	15.58	36.92	+21.34
Spatial localization	25.88	37.00	+11.12
Sound characteristics	31.49	40.43	+8.94
Cross-modal reasoning	30.85	38.30	+7.45
Temporal information	28.44	32.24	+3.80

Appendix

Table 12: Transfer accuracy (%) by each benchmark’s original task categories for Qwen2.5-Omni-7B and its LoRA-adapted counterpart.

\Delta

denotes the difference in percentage points, computed from displayed accuracies.

Explore similar work

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning
Nov 23, 2025Yogesh Kulkarni, Pooyan FazliEgocentric Video Understanding Video Reasoning
Arizona State University
Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding
May 14, 2026Arsha Nagrani, Jasper Uijilings, Shyamal Buch +6Egocentric Video QA Egocentric Video Understanding
Google DeepMind
EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs
May 19, 2026Yang Dai, Dian Jiao, Tianwei Lin +1Egocentric Video QA CoT Reasoning
Zhejiang University · Hangzhou, Zhejiang, China

Nov 23, 2025J/K move · Enter open · S save

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

Yogesh Kulkarni, Pooyan Fazli

Arizona State University

Egocentric Video Understanding Video Reasoning Visually Grounded Reasoning LLM Reasoning

Open paper AskPDF arXiv

Egocentric video understanding requires procedural reasoning under partial observability and continuously shifting viewpoints. Current multimodal large language models (MLLMs) struggle with this setting, often generating plausible but visually inconsistent or weakly grounded responses. We introduce

\textbf{EgoVITA}

, a framework that decomposes egocentric video reasoning into a structured

\textit{plan-then-verify}

process. The model first generates an

\textbf{egocentric plan}

: a causal sequence of anticipated actions from a first-person perspective. This plan is then evaluated by an

\textbf{exocentric verification}

stage that uses third-person reasoning over the same video to verify its spatiotemporal and logical consistency, without exocentric video input. This decomposition enables cross-perspective feedback without requiring paired ego-exo supervision. To train this reasoning process, we adopt Group Relative Policy Optimization (GRPO) with two dense reward signals: one that grounds anticipated actions in subsequent visual observations and another that reinforces consistent third-person verification.

\textbf{EgoVITA}

achieves state-of-the-art performance on egocentric reasoning benchmarks, outperforming Qwen2.5-VL-7B by

\mathbf{+7.7}

on EgoBlind and

\mathbf{+4.4}

on EgoOrient, while maintaining strong generalization on exocentric video tasks with only

52k

training samples.

Abstract

Explore similar work

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding

EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs