AI companions are envisioned as always-on assistants that support users in daily life. With this regard, we introduce BuddyVQA, a benchmark for companion-style question answering (QA) on egocentric streaming video. BuddyVQA contains 21.6K questions linked to 6K highlight moments across 1,012 long, egocentric videos. It features two key characteristics that are common in daily first-person QA assistance but are largely overlooked in existing VideoQA benchmarks: ego-deictic expressions and interactively chained questions (e.g., "Where is it?", "How to get there?"). These require models to infer a user's in-situation intent by resolving visual pronouns in the context of egocentric visual and QA contents, with both grounded in a long-form streaming setting. To tackle the challenges, we propose MyBuddy, a companion-style QA assistant that highlights a multimodal chain-of-thought reasoning mechanism to infer the final answer based on the historical QA and visual content. An additional question filter and multi-level memory are designed to facilitate efficient QA and visual information retrieval under streaming QA settings. Experiments show that MyBuddy significantly enhances the performance of foundation models on BuddyVQA. Moreover, these gains generalize to other streaming and common video QA benchmarks, demonstrating the applicability and effectiveness of our approach. Our code and dataset are available at https://github.com/QHUni/BuddyVQA
Figures & tables
Figure 1: BuddyVQA targets companion-style VQA assistance. It features two key challenges: ego-deictic expressions and chained questions, both under the egocentric streaming setting. MyBuddy achieves SOTA results across different question categories.
Figure 2: Examples of different QAs (Scrutinization, Recall, Recognition & Advisory) in BuddyVQA.
Figure 3: Video and question distributions in BuddyVQA.
Figure 4: Framework of MyBuddy. MyBuddy follows a memory-and-retrieval augmented QA pipeline. When a question arrives, it searches from the multi-level memory and historical QA buffer for related visual and textual contexts to infer the question meaning and drive the answer via multimodal chain-of-thought reasoning.
Model
Size
Sampling
Q. Categories
V. Duration
Overall
Recall
Recognition
Advisory
Scrutinization
0-60s
60-120s
120-180s
Human
-
-
92.4
92.1
91.9
92.0
92.0
92.1
92.3
92.1
General
VideoChat2 ( Li et al., 2024b )
8B
16
40.7
42.2
33.3
31.4
47.8
38.2
33.1
38.9
Video-LLaMA3 ( Zhang et al., 2025a )
7B
1 FPS
41.7
44.1
36.4
34.5
48.5
40.1
34.7
39.2
LongVA ( Zhang et al., 2024a )
7B
32
46.5
46.1
39.4
40.9
47.6
40.1
39.1
43.0
Table 2: Performance of MLLMs and Humans on BuddyVQA. MyBuddy outperforms both general and streaming models across all four assistance types, and shows more stable performance as video stream grows.
Figure 5: Result visualization on BuddyVQA. Streaming models generally perform better than general models over time, and MyBuddy is able to process interactively chained questions and ego-deictics regardless of the streaming length.
Model
Cha. (%)
Dei. (%)
All (%)
LongVA ( Zhang et al., 2024a )
41.5
38.7
43.0
VLLaMA3 ( Zhang et al., 2025a )
36.9
34.2
39.2
LLaMA-VID ( Li et al., 2024c )
41.8
39.4
43.1
Qwen3.5 ( QwenTeam, 2026 )
43.8
42.6
47.2
Gemini-3.1-pro ( DeepMind, 2026 )
46.1
42.6
48.3
GPT-5.5 ( OpenAI, 2026 )
47.6
43.9
50.5
Table 3: Performance comparison on chained and ego-deictic questions in BuddyVQA. MyBuddy consistently improves both capabilities, with particularly pronounced gains on ego-deictic questions.
Model
No Feedback
Correctness
GT Answer
Overall
Overall
0 – 90 s
>90 s
Overall
0 – 90 s
>90 s
Qwen3.5 ( QwenTeam, 2026 )
47.2
49.0
51.8
45.9
52.3
54.2
50.1
Gemini-3.1-pro ( DeepMind, 2026 )
48.3
50.4
53.0
47.4
54.0
55.9
51.7
GPT-5.5 ( OpenAI, 2026 )
50.5
53.1
55.0
50.8
56.9
58.2
55.3
MyBuddy (Qwen3.5)
60.7
63.2
63.8
62.5
67.1
66.2
68.2
MyBuddy (Gemini-3.1-pro)
63.2
66.0
66.4
65.5
69.8
68.8
70.9
Table 4: Dynamic streaming evaluation on BuddyVQA. We report accuracy (%) under three settings for better comparisons: 1) No Feedback: No user feedback of historical questions, 2) Correctness: Binary feedback (correct or not) of historical questions, and 3) GT Answer: Ground-truth answers of historical questions are provided by users. The best and second-best results are highlighted in bold and underline, respectively.
Model
RVS-Ego
RVS-Movie
Latency (↓)
GPU Usage (↓)
Acc.
Sco.
Acc.
Sco.
Flash-VStream ( Zhang et al., 2025c )
58.9
4.0
56.0
3.3
2.4s
20GB
ReKV(LLaVA-OV) ( Di et al., 2025 )
63.7
4.0
54.4
3.6
3.3s
38GB
LongVA ( Zhang et al., 2024a )
53.2
3.1
43.0
3.2
6.3s
37GB
MovieChat ( Song et al., 2025 )
52.2
3.4
39.1
2.3
8.8s
40GB
LLaMA-VID ( Li et al., 2024c )
53.4
3.9
48.6
3.3
11.9s
33GB
Table 5: Comparison on RVS-Ego and RVS-Movie ( Zhang et al., 2025c ) . Acc.: Accuracy. Sco.: Score. Latency & GPU Usage are tested on Tesla A100 GPU. Results are averaged on 10 videos with each 720 frames as inputs.
Model
Video-MME
EgoSchema
Video-LLaMA3 ( Zhang et al., 2025a )
47.9 / 49.7
57.3
VideoChat2 ( Li et al., 2024b )
39.5 / 43.8
56.7
MovieChat ( Song et al., 2025 )
38.2 / -
53.5
Qwen3.5 ( Bai et al., 2025b )
71.2 / 74.3
68.1
GPT-5.5 ( OpenAI, 2026 )
77.3 / 82.6
79.8
LongVA ( Zhang et al., 2024a )
52.6 / -
-
Table 6: Model comparison on Video-MME ( Fu et al., 2024 ) and EgoSchema ( Mangalam et al., 2024 ) . Video-MME: “without subs / with subs”. The best and second-best Accuracy results are bolded and underlined , respectively.
Ablation Model
Chain (%)
Deictic (%)
✗
✓
✗
✓
MyBuddy(Full)
66.4
62.8
66.1
63.5
w/o LM
65.7
61.3
63.3
58.9
w/o SM & LM
64.1
58.0
57.8
55.8
w/o AM & SM & LM
59.2
53.1
53.4
52.9
w/o Historical QA
62.4
44.5
61.3
54.3
Table 7: Ablation of MyBuddy on BuddyVQA w.r.t interactively chained questions and questions with deictic expressions. AM/SM/LM: Active/Semi-Active/Low-Active Memory. For the setting of w/o SM & LM & AM, we use the question frame as input.
Figure 6: Streaming VQA latency & GPU Usage w.r.t. number of frames, tested on A100 GPUs. MyBuddy is able to process long streams more efficiently.
Configuration
Acc.
Chained Acc.
Latency
GPU Usage
Extra Cost
Base (Raw F+His. QAs)
47.2
43.8
3.9s
51.1GB
–
+ Multi-Level Memory
51.4
48.2
2.5s
22.9GB
-1.4s / -28.2GB
+ Question Filter
53.6
54.5
2.3s
30.2GB
-1.6s / -20.9GB
+ Key Visual Info Retrieval
55.7
52.7
4.0s
62.3GB
+0.1s / +11.2GB
+ Hierarchical Rephrasing
56.9
53.4
4.3s
53.8GB
+0.4s / +2.7GB
MyBuddy(Full)
60.7
57.3
2.6s
29.1GB
-1.3s / -12.0GB
Table 8: Efficiency breakdown on A100 GPUs. Extract Cost denotes additional overhead (latency/memory) introduced by individual modules. Base (Qwen3.5-9B): processes video frames & concatenates all historical QAs for each question online.
Figure 7: More result visualizations on BuddyVQA in Scrutinization and Recognition categories. Baseline models perform poorly on deictic questions, underscoring the importance of jointly leveraging historical interaction contexts and embodied visual cues for ego-deictic reference resolution.
Figure 8: More result visualizations on BuddyVQA in Recall and Advisory categories. MyBuddy demonstrates substantially stronger performance on long video sequences compared to general MLLMs. Its consistent improvements on chained questions further highlight the importance of history-aware reasoning across question chains in BuddyVQA.
Failure
Num
Ratio (%)
Main Reasons
Que. Filter
124
6.2
Misclassifying chained questions
Vis. Retri.
420
21.0
Missing relevant frames or objects
Rephrase
504
25.2
Wrong referent grounding
Ans. Infer
556
27.8
Correct context, wrong inference
Others
396
19.8
Motion blur / Noisy input
Table 9: Step-wise failure analysis of MyBuddy on the subset of BuddyVQA. While early-stage errors exist, a significant portion of failures arises from intrinsic ambiguity and visual challenges, rather than pipeline design alone.
Evaluator
InternVL-3.5
ReKV
Acc.(%)
Sco.
Acc.(%)
Sco.
Human
62.3
3.1
55.3
2.9
GPT-5
61.8
3.1
53.7
2.8
Table 10: Comparison of human and LLM evaluators in terms of accuracy and score. The results show highly aligned scores between human and LLM judge.
Figure 9: Confusion matrices of evaluation results.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Activities in BuddyVQA.
Category
Q Num. & Ratio
Examples
Recall
5631(26.1%)
Who just passed by me? How much time passed after I started running? What was he doing several seconds ago? Did I lock the door before leaving?
Recognition
5069(23.5%)
What’s on that? What’s that red thing next to the greens? Who did I just hand those to?
Scrutinization
5523(25.6%)
Am I missing any spot while sweeping? Am I doing this fast enough? Did I take longer to complete this than before? Are these clothes of good quality?
Advisory
5351(24.8%)
What should I do next? How long will it take to get through this line? How do I know when the area is smooth enough?
Appendix
Table 11: Question categories and examples of BuddyVQA.
Method
D Acc.(%)
Latency
Direct Question Answering
62.4
423ms
Hierarchical Rephrasing (Ours)
86.7
552ms
Appendix
Table 12: Comparison of disambiguation accuracy (D Acc.) and latency w/ and w/o hierarchical rephrasing.
Deictic Type
Original Question
Hierarchically Rephrasing
Final Answer
Unclear referents
“I need more, where is it?”
“What does it refer to?” → “Does it refer to the cup?” → “Where is the cup?”
“On the table.”
Vague temporal / spatial cues
“Has it started yet?”
“Does it refer to the meeting?” → “Has the meeting started yet?”
“Yes, 2 minutes ago.”
Ambiguous actions
“Am I doing this right?”
“Does this refer to cutting the onion?” → “Am I cutting the onion correctly?”
“Yes, but hold the knife away from your hand.”
Appendix
Table 13: Examples of hierarchical input rephrasing of MyBuddy.
Method
Accuracy(%)
Corr. with Ques.
Corr.(%)
Score
Human
76.2
82.3
3.8
GPT-5 ( OpenAI, 2025a )
81.3
89.1
4.1
Appendix
Table 14: Accuracy of eye gaze points and hand pose detection, and the correlation of correctly detected features with questions in terms of correlated percentage (Corr.) and score.
You are an advanced assistant specialized in video analysis.
Your task is to analyze the provided video and identify its key highlights based on significant visual content, such as rapid action, noteworthy events, or shifts in visual dynamics. The input is a video file and the output should be a list of timestamps that correspond to the start and end of each highlight segment in the video. Please focus on identifying segments with high visual or contextual importance. Answer with ONLY timestamp pairs.
Ensure the timestamps are accurate and clearly marked in seconds, e.g., [00:30, 00:45], [02:21, 02:51].
Video: { video }
# Question generation:
Appendix
Table 16: Prompts for automatic question candidate generation of BuddyVQA.
Evaluation Prompts for BuddyVQA
You are an intelligent chatbot designed for evaluating the correctness of generative outputs for egocentric, streaming question-answer pairs in the BuddyVQA dataset. Your task is to compare the predicted answer with the correct answer set and determine whether they meaningfully match, taking into account the characteristics of egocentric video, ego-deictic expressions, and interactive question chains.
The following are the requirements of evaluation:
a. Focus on the meaningful match between the predicted answer and the correct answer sets. In BuddyVQA, many questions contain ego-deictic expressions such as “this”, “that”, “it”, “here”, or “there”. Ensure that the predicted answer resolves the same referent as the ground truth, even if expressed differently.
b. The correct answers are provided as a set . Your evaluation should treat the prediction as correct if it meaningfully matches any of the answers in the set. The match does not need to be exact, but must refer to the same grounded entity, event, or semantic content.
c. Consider the temporal continuity of chained questions. Some BuddyVQA questions depend on prior context ( e.g. follow-up questions like “Is it still there?” or “Did I do it correctly?”). When evaluating, ensure that the predicted answer maintains the same contextual interpretation and refers to the correct entity or past event, consistent with the ground-truth chain.
Appendix
Table 17: Prompts for GPT-5 ( OpenAI, 2025a ) to evaluate MLLMs on BuddyVQA.
Low-Active Memory Summarization Prompt
You are an intelligent multimodal assistant tasked with summarizing egocentric video segments to extract long-term, stable, and reusable contextual information. Your goal is to produce a concise textual summary that captures information in long videos, including key activities, recurring patterns, background environment, salient objects, user routines, and consistent spatial layout, etc.
# Instructions:
- Focus on information that remains stable or frequently appears across this time window. Identify recurring actions, repeated interactions, or common tasks performed by the user.
- Capture stable environmental structure such as room layout, tool locations, furniture arrangement, or habitual user workflows.
- Extract objects or entities that reliably appear or are frequently manipulated across the window.
Appendix
Table 18: Prompt for zero-shot summarization model S used to construct Low-Active Memory.
Question Filtering Prompt
You are an intelligent multimodal assistant designed to identify whether an incoming question in a streaming egocentric QA scenario is chained (dependent on prior QA turns) or unchained (independent). Your task is to analyze the current question, compare it with a buffer of historical QA pairs and the current frame, and determine: (1) whether the question relies on earlier context, and (2) if so, which prior QA pair is the most relevant for resolving this dependency.
# Instructions:
- Classify the incoming question as either “chained” or “unchained” . A chained question typically contains ego-deictic expressions (e.g., “this”, “that”, “it”, “there”) or implicit references to past events, actions, or answers (e.g., “Is it still there?”, “Did I do it correctly?”, “What about now?”) that require information from historical QAs to answer. An unchained question is self-contained and understandable without referencing earlier context.
- If the question is classified as unchained, return the label “unchained” and do not select any historical QA pair.
- If the question is classified as chained, inspect the historical QA buffer and identify the single most relevant QA pair that resolves the ambiguous referent or contextual dependency. Relevance may arise from: semantic similarity, shared objects or actions, temporal continuity, or matching referential expressions.
Appendix
Table 19: Prompt for the Question Filter R used to detect chained questions and retrieve relevant historical QA pairs.
Visual Information Retrieval Prompt
You are an intelligent multimodal assistant designed to retrieve visual information from egocentric video memories in order to support real-time question answering. Your task is to (1) generate concise visual captions, (2) filter long-term summaries for information relevant to the incoming question, and (3) detect embodied features such as eye-gaze points and hand bounding boxes that help ground deictic references. All three subtasks must be performed using the same lightweight MLLM capabilities.
# Instructions for Captioning (from Active and Semi-Active Memory):
- Generate a concise and accurate caption describing the visually salient content in the provided video frames. Focus on the objects, spatial relations, user interactions, and scene attributes that are relevant for answering a question.
- The caption should reflect only the visible content and be grounded in the egocentric viewpoint.
Appendix
Table 20: Prompt for the Visual Information Retrieval (VIR) module.
Zero-shot Question Answering Prompt
You are an intelligent multimodal assistant designed to answer egocentric, streaming questions using retrieved visual information, historical QA context, and embodied features. Your task is to first rephrase the incoming question into an explicit, fully disambiguated form and then generate a correct answer grounded in the provided multimodal context.
# Instructions for Hierarchical Input Rephrasing:
Rewrite the question in a clearer and explicit form that removes ambiguity. Resolve ego-deictic expressions such as “this”, “that”, “it”, “here”, or “there” by grounding them using the provided context. Incorporate historical QA, captions, and embodied features to determine the correct referent or event for rephrasing. The rephrased question must express the user’s intent explicitly and be fully interpretable without additional context. Provide only one rephrased question—concise, unambiguous, and grounded.
Here is an example:
Incoming Question: “Where is it now?”
Appendix
Table 21: Prompt for zero-shot question answering with hierarchical rephrasing.
We present the first systematic analysis of multimodal large language models (MLLMs) in personalized question-answering requiring ego-grounding - the ability to understand the camera-wearer in egocentric videos. To this end, we introduce MyEgo, the first egocentric VideoQA dataset designed to evaluate MLLMs' ability to understand, remember, and reason about the camera wearer. MyEgo comprises 541 long videos and 5K personalized questions asking about "my things", "my activities", and "my past". Benchmarking reveals that competitive MLLMs across variants, including open-source vs. proprietary, thinking vs. non-thinking, small vs. large scales all struggle on MyEgo. Top closed- and open-source models (e.g., GPT-5 and Qwen3-VL) achieve only~46% and 36% accuracy, trailing human performance by near 40% and 50% respectively. Surprisingly, neither explicit reasoning nor model scaling yield consistent improvements. Models improve when relevant evidence is explicitly provided, but gains drop over time, indicating limitations in tracking and remembering "me" and "my past". These findings collectively highlight the crucial role of ego-grounding and long-range memory in enabling personalized QA in egocentric videos. We hope MyEgo and our analyses catalyze further progress in these areas for egocentric personalized assistance. Data and code are available at https://github.com/Ryougetsu3606/MyEgo
Junbin Xiao, Shenglang Zhang, Pengxiang Zhu +1
University of Science and Technology of China · National University of Singapore
We introduce EgoSAT, the first comprehensive benchmark for egocentric video reasoning in streaming settings, designed to evaluate the capabilities of modern vision-language models (VLMs). The benchmark targets streaming interaction understanding, where video frames arrive sequentially and models must continuously interpret evolving visual context. EgoSAT unifies several previously distinct tasks within a single streaming framework. In this formulation, queries about completed events correspond to retrospective reasoning, queries about ongoing activities require online understanding, and queries about future actions involve prospective anticipation. This unified setting requires models to reason about the past, present, and future while operating under the constraint that only previously observed frames are available. EgoSAT contains 1,997 unique videos spanning 165 hours of egocentric footage and around 4,800 high-quality question-answer pairs, carefully designed to probe reasoning across varying temporal contexts. Using this benchmark, we evaluate a diverse set of both open-weight and closed-weight VLMs, providing a systematic assessment of their ability for streaming interaction understanding. By distinguishing answerability and conducting diagnostics on confidence of models, we find existing models not only struggle with prospective and retrospective modeling, but also exhibit severe mis-calibration: confidence often fails to track inherent answerability, leading to dangerous "confidently wrong" behaviors. Project page: https://leiyj23.github.io/EgoSAT/
Yijia Lei, Jinzhao Li, Yichi Zhang +3
College of AI, Tsinghua University · University of Wisconsin–Madison
Streaming video assistance requires models to answer asynchronous questions from an observed prefix under a fixed context budget. Existing approaches model response timing or compress history, but an online state formed before future questions are known can omit visual details before later questions reveal their relevance; the retained state alone cannot recover them. We introduce Watch-Think-Interact (WTI), a closed-loop framework for multi-question streaming video reasoning. WTI maintains compact natural-language memory entries tagged with source-video time ranges; these entries support direct reasoning when sufficient and otherwise anchor selective recall of finer visual evidence. For each question, WTI answers when current context and memory suffice, continues watching when required evidence has not appeared, or recalls a relevant past interval and decides again after incorporating the returned chunks, without replaying the full observed history. To train this behavior, we construct WTI-82K, comprising 82,335 timed questions across 4,812 causally aligned trajectories, and develop Stream-GDPO to optimize complete multi-question streaming rollouts using trajectory-level feedback for response timing, source-video recall, and memory updates. WTI achieves state-of-the-art aggregate performance among the compared open-source streaming baselines, reaching 83.3% on StreamingBench and 73.6% weighted overall accuracy on OVO-Bench.
Ziheng Huang, Yicheng Bao, Xueheng Li +8
East China Normal University · University of Science and Technology of China · Xiamen University