Ego-Grounding for Personalized Question-Answering in Egocentric Videos

Junbin Xiao; Shenglang Zhang; Pengxiang Zhu; Angela Yao

cs.CVApr 2, 2026

Ego-Grounding for Personalized Question-Answering in Egocentric Videos

Authors: Junbin Xiao, Shenglang Zhang, Pengxiang Zhu, Angela Yao

Organizations: University of Science and Technology of China · National University of Singapore

Abstract

We present the first systematic analysis of multimodal large language models (MLLMs) in personalized question-answering requiring ego-grounding - the ability to understand the camera-wearer in egocentric videos. To this end, we introduce MyEgo, the first egocentric VideoQA dataset designed to evaluate MLLMs' ability to understand, remember, and reason about the camera wearer. MyEgo comprises 541 long videos and 5K personalized questions asking about "my things", "my activities", and "my past". Benchmarking reveals that competitive MLLMs across variants, including open-source vs. proprietary, thinking vs. non-thinking, small vs. large scales all struggle on MyEgo. Top closed- and open-source models (e.g., GPT-5 and Qwen3-VL) achieve only~46% and 36% accuracy, trailing human performance by near 40% and 50% respectively. Surprisingly, neither explicit reasoning nor model scaling yield consistent improvements. Models improve when relevant evidence is explicitly provided, but gains drop over time, indicating limitations in tracking and remembering "me" and "my past". These findings collectively highlight the crucial role of ego-grounding and long-range memory in enabling personalized QA in egocentric videos. We hope MyEgo and our analyses catalyze further progress in these areas for egocentric personalized assistance. Data and code are available at https://github.com/Ryougetsu3606/MyEgo

Figures & tables

Figure 1: Example of egocentric personalized QA concerning my demands. The narratives are provided for better understanding.

Figure 2: MyEgo examples. To answer my questions, the models must understand (a) which hand is mine, (b) memorize and track the green chess pieces I used (to be distinguished with the red ones of another person, and the blue ones which are not used).

Benchmark	#Q	#V	Ave. Len	Challenges	MP	MO	Task
QAEgo4D [ 4 ]	1,854	166	8.2 min	Episodic Memory	✗	✗	OE
EgoSchema [ 28 ]	5,063	5,063	3 min	Long Ego Video Understanding	✗	✗	MC
EgoThink [ 6 ]	700	595	-	General Ego Vision Understanding	✗	✗	OE
EgoMemoria [ 47 ]	7,026	629	0.5 to 60 min	General Ego Video Understanding	✗	✗	MC
EgolifeQA [ 46 ]	6,000	6	44.3 h	Multimodal Episodic Memory	✗	✗	MC
EgoTextVQA [ 57 ]	7,064	1,507	1.7 min	Ego Scene-Text Understanding	✗	✗	MC

Table 1: Benchmark comparison. MyEgo highlights ego-grounding and memorizing the camera wearers from ego video stream where multiple people and multiple similar objects present. MP: questions require distinguishing ego-person from other persons. MO: questions require distinguishing objected associated with ego-person from other similar-looking objects. OE/MC: Open-Ended/Multi-Choice.

Methods	Res.	Multiple-Choice					Open-Ended
Methods	Res.	MC-2	MC-5	Cur.	Pre.	Avg.	Act.	Obj.	Others	Cur.	Pre.	Avg. (Acc/Score)
Human	-	95.1	92.1	93.4	92.3	92.7	82.4	84.8	91.2	84.0	85.0	84.7
Closed-source Models
GPT-5 [ 29 ]	480P	66.4	53.7	56.9	55.8	56.1	50.0	44.6	43.1	51.1	44.0	46.1 / 2.5
Gemini-2.5 Pro [ 8 ]	720P	61.8	45.5	49.0	48.4	48.6	40.2	40.2	47.7	42.4	40.3	40.9 / 2.2
Open-source Models

Table 2: Evaluation results on MyEgo dataset. We present the performance on both multiple-choice (MC) and open-ended (OE) tasks. MCQA includes binary (MC-2) and five-option (MC-5) QA. For OEQA, we analyze the results across different categories and temporal locality. Cur.: question whose answer can be found in the current moment (within 2s). Pre.: question whose answer is located in the past video content. 480P: resize the height to be 480.

Models	Input	Acc@Cur.	Acc@Pre.
Qwen3-VL-8B-Think [ 2 ]	Uni. Q&A	38.4 41.3 $\uparrow$ 2.9	32.0 41.1 $\uparrow$ 9.1
Qwen3-VL-8B-Instruct [ 2 ]	Uni. Q&A	36.9 46.3 $\uparrow$ 9.4	35.5 41.4 $\uparrow$ 5.9
InternVL3-8B [ 58 ]	Uni. Q&A	35.6 41.3 $\uparrow$ 5.7	34.8 40.4 $\uparrow$ 5.6
LLaVA-Video [ 55 ]	Uni. Q&A	36.3 37.7 $\uparrow$ 1.4	33.7 41.6 $\uparrow$ 7.9
Qwen2.5-VL-7B [ 3 ]	Uni. Q&A	37.7 43.4 $\uparrow$ 5.7	33.2 42.4 $\uparrow$ 9.2
Gemini-2.5 Pro [ 8 ]	Uni. Q&A	42.4 49.3 $\uparrow$ 6.9	40.3 51.5 $\uparrow$ 11.2

Table 3: Effects of answer moment as input. Uni.: uniform sampling. Q&A: key frames at QA moments as input.

Figure 5: Accuracy distribution along the video stream. Accuracy drops along time with uniform sampling but remains relatively stable with GT moment input.

Figure 6: Effects with different number of frame inputs. The dashed lines in (b) are the results of uniformly sampling 32 frames.

Models	Input	OE	MC
InternVL3.5-8B [ 41 ]	Original Enhanced Remove	33.1 33.2 $\uparrow$ 0.1 31.7 $\downarrow$ 1.4	39.5 41.1 $\uparrow$ 1.6 38.5 $\downarrow$ 1.0
InternVL3-8B [ 58 ]	Original Enhanced Remove	35.0 34.7 $\downarrow$ 0.3 33.7 $\downarrow$ 1.3	41.8 40.4 $\downarrow$ 1.4 41.4 $\downarrow$ 0.4
LLaVA-Video [ 55 ]	Original Enhanced Remove	34.5 35.9 $\uparrow$ 1.4 35.3 $\uparrow$ 0.8	37.5 38.1 $\uparrow$ 0.6 37.5 $\uparrow$ 0.0

Table 4: Effects of different prompt modification. We test several models’ performance with a prompt where cues of "personalization" is removed ("Remove") and questions where first-person pronouns are replaced with "the camera wearer" ("Enhanced"), respectively. We select the thinking version of InternVL3.5-8B here.

Figure 7: Example prompts of (top) removing the personalized cues and (bottom) enhancing the clarification by reminding that ’I’ refers to the camera wearer.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Agreement between human and GPT evaluation on 100 instances.

Methods	Multiple-Choice					Open-Ended
Methods	MC-2	MC-5	Cur.	Pre.	Avg.	Action	Object	Others	Cur.	Pre.	Avg.
Closed-source Models
GPT-5 [ 29 ]	64.5	54.2	57.5	55.9	56.3	50.0	44.6	43.1	51.1	44.0	46.1
Gemini-2.5 Pro [ 8 ]	60.3	45.1	49.9	47.5	48.2	40.2	40.2	47.7	42.4	40.3	40.9
General Open-source Models
Parameters > 10B

Appendix

Table 6: Performances across all selected models (results based on 30% data). Dispider performs only MC-QA.

Figure 9: Result visualization on MyEgo. Models struggle to correctly answer even simple questions, indicating their severe deficiency in performing ego-grounding in videos to answer personalized questions from the camera wearers. Top: The camera wearer is cycling alongside another man whose bike and helmet differ in color from his. Middle: The camera wearer is playing a dart-ball game with others and achieves the highest score. Bottom: The camera wearer is repairing a bicycle with another man, repeatedly putting down and picking up his screwdriver.

Panel of Table 7 — Table 7: Prompts for Gemini-2.5 Pro to generate distracting answers in multi-choice QA.

Task	General Prompt
Open-ended	You are a first-person AI assistant integrated into a head-mounted camera. Your primary mission is to answer questions from the user (the camera wearer) about their own actions, objects, and environment as seen through your lens. You should answer directly to the user. The question is asked at the moment of the last frame in the video. Key Instructions: 1. First-Person Context: All references to ’I’, ’me’, or ’my’ are about the user. You must distinguish between the user’s actions/objects and those of other people visible in the video. 2. Grounding: Base your answers primarily on the visual evidence in the video. If the information is not present or cannot be reasonably inferred from the video, state that it is not shown. 3. For any question that requires a ’Yes’ or ’No’ response, you MUST follow it with a brief explanation for your reasoning. 4. All responses must be direct and clear. {VIDEO_CONTENT}. Question: {QUESTION}.
Multiple-choice	You are a first-person AI assistant integrated into a head-mounted camera. Your primary mission is to answer questions from the user (the camera wearer) about their own actions, objects, and environment as seen through your lens. You should answer directly to the user. The question is asked at the moment of the last frame in the video. Key Instructions: 1. All references to ’I’, ’me’, or ’my’ are about the user. You must distinguish between the user’s actions/objects and those of other people visible in the video. 2. Base your answers primarily on the visual evidence in the video. {VIDEO_CONTENT}. Question: {QUESTION}. Options: {2/5 OPTIONS}. There is only one correct option. Please only response with the letter of the correct option.
Removed personalized cues	You are a first-person AI assistant integrated into a head-mounted camera. Your primary mission is to answer questions from the user (the camera wearer) about their own actions, objects, and environment. You should answer directly to the user. The question is asked at the moment of the last frame in the video. Key Instructions: 1. Grounding: Base your answers primarily on the visual evidence in the video. If the information is not present or cannot be reasonably inferred from the video, state that it is not shown. 2. For any question that requires a ’Yes’ or ’No’ response, you MUST follow it with a brief explanation for your reasoning. 3. All responses must be direct and clear. {VIDEO_CONTENT}. Question: {QUESTION}.

Appendix

Table 8: Prompts for models to answer questions.

Panel of Table 9 — Table 9: Prompts for GPT-5 mini to evaluate open-ended answers.

Explore similar work

EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory
Aug 13, 2026Weitao Chen, Hu Jiaxin, Xie Tianyidan +15Egocentric Video QA Video-Language Model Evaluation
Nanjing University, China · Huawei Technologies Co., Ltd., China · Tianjin University, China
MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents
Mar 10, 2026Kangsan Kim, Yanlai Yang, Suji Kim +4Egocentric Video QA Embodied QA
KAIST · New York University · Samsung Electronics +1
EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning
Nov 23, 2025Yogesh Kulkarni, Pooyan FazliEgocentric Video Understanding Video Reasoning
Arizona State University

Aug 13, 2026J/K move · Enter open · S save

EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory

Weitao Chen, Hu Jiaxin, Xie Tianyidan, Yang Li+14

Nanjing University, China · Huawei Technologies Co., Ltd., China · Tianjin University, China

Egocentric Video QA Video-Language Model Evaluation Egocentric Video Understanding Temporal Video Understanding

Open paper AskPDF arXiv

Recent advances in Multimodal Large Language Models (MLLMs) have led to substantial progress in video understanding, accompanied by a growing number of long video benchmarks. However, existing benchmarks rely predominantly on web-sourced videos that lack inter-clip spatiotemporal continuity, making it difficult to assess whether models can maintain consistent memory across days or weeks of real-world experience. We introduce EgoMonth, the first month-level egocentric video understanding benchmark. EgoMonth comprises over 300 hours of first-person daily-life recordings from 20 participants spanning 20 to 120 days, paired with 1,443 human-crafted multiple-choice question-answer pairs. We design a cognitively grounded 14-task evaluation framework organized into three hierarchical cognitive levels: Schema Consolidation, Episodic Indexing, and Cascading Reasoning. Evaluation of state-of-the-art open-source and closed-source MLLMs reveals that even the best-performing model, Gemini 2.5 Pro, achieves only 71.8% macro-average accuracy, remaining 22.4 percentage points below the corrected human baseline of 94.2%. Several models perform near or below the 25% chance level on tasks such as Route Reasoning, Cross-view Spatial Reasoning, and Direction Judgement, while even the strongest closed-source model remains substantially below human performance. These results indicate that current MLLMs function as lossy summarizers rather than faithful memorizers, highlighting the need for architectures with genuine long-term spatiotemporal memory.

Abstract

Explore similar work

EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory

MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents

EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning