EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environment

Qiance Tang; Ziqi Wang; Jieyu Lin; Ziyun Li; Barbara De Salvo; Sai Qian Zhang

cs.LGApr 9, 2026

EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environment

Authors: Qiance Tang, Ziqi Wang, Jieyu Lin, Ziyun Li, Barbara De Salvo, Sai Qian Zhang

Organizations: New York University, USA · Meta Reality Labs, USA

Abstract

Long context egocentric video understanding has recently attracted significant research attention, with augmented reality (AR) highlighted as one of its most important application domains. Nevertheless, the task remains highly challenging due to the need for reasoning over extended temporal contexts and diverse, unstructured activities. Although several benchmarks exist, most egocentric datasets rely on human worn cameras and focus mainly on visual content, with limited consideration of underlying user behavior when forming video-related queries. EgoEverything is a benchmark that explicitly considers human behavior by leveraging human attention signals, abstracted from gaze data, when generating questions. It comprises over 5,000 multiple choice question answer pairs, spanning more than 100 hours of video. By integrating human attention signals during question generation, it more faithfully captures natural human behavior and offers a realistic evaluation setting for long-context egocentric video understanding in AR.

Figures & tables

Figure 1: (a) Real-life AR LEU scenario. (b) User attention variation.

Figure 2: (a) Front and (b) inner views of the Meta Quest Pro headset, which functions as both an AR and VR (Virtual Reality) device. (c) Meta Aria glasses equipped with multiple cameras.

Figure 3: Data generation pipeline for EgoEverything. Step 1, Video Stream Summary and Clustering (VSSC), combines clustering, summary generation, and manual inspection to obtain high-quality video descriptions. Step 2, Gaze-Oriented Target Sampling, detects objects in each frame, computes their distance to gaze, applies ReID to group object instances, and samples target objects by distance. Step 3, Question Generation and Manual Curation, uses a self-feedback Synthesizer Agent and Validator Agent to generate and curate MCQs for each target object.

Panel of Figure 4 — Figure 4: Video frames and keywords in the MCQ are highlighted in different colors. Orange indicates the target objects, Blue highlights contextual entities mentioned in the question or answer, and Purple denotes regions where the target objects cannot be located. The GT label refers to the ground truth option, and the VLM label refers to the option selected by the model.

Figure 5: The middle ring shows the proportion of each category and the number of MCQs in it. The outer ring shows the most frequent interrogative words in that category. The inner circle shows the average accuracy of all VLM models from Table 1 for each category.

Figure 6: Distribution of target object categories. The x-axis represents object categories, the y-axis counts the number of MCQs generated using the target object in a given category. The length of each colored bar segment indicates the number of questions of that type, revealing the reasoning associated with each target object.

Model	FR	AD	GC	GM	VMP	AMEGO
Videollama3-7b [ 57 ]	49.1	46.1	42.4	35.2	20.2	19.5
Videollama3-2b	46.5	44.9	40.5	34.5	21.3	19.4
Gemini 1.5 pro [ 48 ]	63.1	58.4	52.7	37.7	33.2	18.3
LongVA [ 58 ]	34.6	31.6	28.9	22.2	NA	NA
Llava-Video [ 57 ]	42.6	36.6	32.9	26.0	NA	NA

Table 1: Performance comparison of different methods across VLMs on EgoEverything. "NA" means not available. The human annotators achieve an average accuracy of 83.5%.

Model	Videollama3-7b	Gemini 1.5 pro	LongVA
Accuracy (%)	22.9	35.9	21.8

Table 2: VLM model performance under blind setting.

(a) Dist. Gaze		(b) BBox area
Dist. bin (pixel)	Acc. (%)	Area (pixel 2 )	Acc. (%)
$0$ – $100$	54.2	100	44.3
$100$ – $200$	54.8	500	44.4
$200$ – $300$	51.8	2k	44.5
$300$ – $400$	47.2	5k	44.9
$400$ – $500$	42.6	10k	49.2

Table 3: Accuracy analysis under three factors. Dist. denotes

\|o-f\|

(pixel), area denotes bounding box area (pixel 2 ), and recall interval denotes

t_{q}-t_{0}

Egocentric Video Understanding

Egocentric video understanding studies how AI interprets first-person video to recognize actions, track interactions, reason about space and time, and assist users.

See all 203 papers

EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions
Oct 13, 2026Songruo Chen, Yuqian Fu, Yu Li +7Egocentric Video Understanding Egocentric Video QA
Fudan University · KAUST · Universiti Malaya +4
Ego-Grounding for Personalized Question-Answering in Egocentric Videos
Apr 2, 2026Junbin Xiao, Shenglang Zhang, Pengxiang Zhu +1Egocentric Video QA Question Answering
University of Science and Technology of China · National University of Singapore
Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding
May 14, 2026Arsha Nagrani, Jasper Uijilings, Shyamal Buch +6Egocentric Video QA Egocentric Video Understanding
Google DeepMind

Oct 13, 2026J/K move · Enter open · S save

EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions

Songruo Chen, Yuqian Fu, Yu Li, Mingkang Dong+6

Fudan University · KAUST · Universiti Malaya+4

Egocentric Video Understanding Egocentric Video QA Multimodal QA Audio-Visual Understanding

Open paper AskPDF arXiv

Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their tight coupling in real-world interaction. We propose the Ego-Action-World interaction loop as a unifying principle: the wearer perceives the world, forms goals, acts, changes the world, and uses feedback to guide subsequent behavior. Based on this view, we introduce EgoOmniWorld, a benchmark organized around four complementary interaction structures: Node captures ego, action, and world states; Edge captures directed relations among them; Chain captures linked dependencies; and Loop captures feedback-closed interactions. We develop an interaction-first, evidence-grounded data pipeline that extracts structured interactions from egocentric videos, grounds them in visual and auditory evidence, and generates questions according to their interaction structure. Required modalities and single- or multi-turn formats follow naturally from the supporting evidence and dependencies. EgoOmniWorld contains 24,166 QA groups, including 20,284 training and 3,882 test samples. Experiments on representative open- and closed-source multimodal large language models reveal substantial and uneven limitations across interaction structures. Models trained on EgoOmniWorld improve in-domain interaction understanding and generalize to existing egocentric benchmarks. EgoOmniWorld provides a unified foundation for evaluating and advancing egocentric video understanding. Data, code, and models will be released.

Abstract

Egocentric Video Understanding

Explore similar work

EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions

Ego-Grounding for Personalized Question-Answering in Egocentric Videos

Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding