cs.CVSep 6, 2026
SaveCompanion-style QA Assistance in Ego-Vision
Organizations: National University of Singapore · University of Science and Technology of China
Abstract
AI companions are envisioned as always-on assistants that support users in daily life. With this regard, we introduce BuddyVQA, a benchmark for companion-style question answering (QA) on egocentric streaming video. BuddyVQA contains 21.6K questions linked to 6K highlight moments across 1,012 long, egocentric videos. It features two key characteristics that are common in daily first-person QA assistance but are largely overlooked in existing VideoQA benchmarks: ego-deictic expressions and interactively chained questions (e.g., "Where is it?", "How to get there?"). These require models to infer a user's in-situation intent by resolving visual pronouns in the context of egocentric visual and QA contents, with both grounded in a long-form streaming setting. To tackle the challenges, we propose MyBuddy, a companion-style QA assistant that highlights a multimodal chain-of-thought reasoning mechanism to infer the final answer based on the historical QA and visual content. An additional question filter and multi-level memory are designed to facilitate efficient QA and visual information retrieval under streaming QA settings. Experiments show that MyBuddy significantly enhances the performance of foundation models on BuddyVQA. Moreover, these gains generalize to other streaming and common video QA benchmarks, demonstrating the applicability and effectiveness of our approach. Our code and dataset are available at https://github.com/QHUni/BuddyVQA
Figures & tables
Figure 1: BuddyVQA targets companion-style VQA assistance. It features two key challenges: ego-deictic expressions and chained questions, both under the egocentric streaming setting. MyBuddy achieves SOTA results across different question categories.
Figure 2: Examples of different QAs (Scrutinization, Recall, Recognition & Advisory) in BuddyVQA.
| Benchmark | V Num. | Total V. | Q Num. | Dei(%) | Chain | EgoV | MA | QC | Task |
| Egocentric VQA Benchmarks | |||||||||
| AssistQ ( Wong et al., 2022 ) | 100 | 3.2h | 531 | 8 | ✗ | ✓ | ✗ | ✗ | MC |
| EgoSchema ( Mangalam et al., 2024 ) | 5k | 250h | 5k | 3 | ✗ | ✓ | ✗ | ✗ | MC |
| QAEgo4D ( Bärmann and Waibel, 2022 ) | 1.3k | 180h | 14k | 5 | ✗ | ✓ | ✗ | ✗ | OE |
| EgoMemoria ( Ye et al., 2025 ) | 629 | 149h | 7k | 8 | ✗ | ✓ | ✗ | ✓ | MC |
| EgolifeQA ( Yang et al., 2025b ) | 6 | 266h | 6k | 11 | ✗ | ✓ | ✗ | ✗ | MC |
Table 1: VideoQA datasets. Chain: chained question. Dei: question containing ego-deictic expressions. EgoV: egocentric videos. MA: multiple ground-truth answers for each question. QC: question category. OE/MC: Open-Ended/Multi-Choice.
Figure 3: Video and question distributions in BuddyVQA.
Figure 4: Framework of MyBuddy. MyBuddy follows a memory-and-retrieval augmented QA pipeline. When a question arrives, it searches from the multi-level memory and historical QA buffer for related visual and textual contexts to infer the question meaning and drive the answer via multimodal chain-of-thought reasoning.
| Model | Size | Sampling | Q. Categories | V. Duration | Overall | |||||
| Recall | Recognition | Advisory | Scrutinization | 0-60s | 60-120s | 120-180s | ||||
| Human | - | - | 92.4 | 92.1 | 91.9 | 92.0 | 92.0 | 92.1 | 92.3 | 92.1 |
| General | ||||||||||
| VideoChat2 ( Li et al., 2024b ) | 8B | 16 | 40.7 | 42.2 | 33.3 | 31.4 | 47.8 | 38.2 | 33.1 | 38.9 |
| Video-LLaMA3 ( Zhang et al., 2025a ) | 7B | 1 FPS | 41.7 | 44.1 | 36.4 | 34.5 | 48.5 | 40.1 | 34.7 | 39.2 |
| LongVA ( Zhang et al., 2024a ) | 7B | 32 | 46.5 | 46.1 | 39.4 | 40.9 | 47.6 | 40.1 | 39.1 | 43.0 |
Table 2: Performance of MLLMs and Humans on BuddyVQA. MyBuddy outperforms both general and streaming models across all four assistance types, and shows more stable performance as video stream grows.
Figure 5: Result visualization on BuddyVQA. Streaming models generally perform better than general models over time, and MyBuddy is able to process interactively chained questions and ego-deictics regardless of the streaming length.
| Model | Cha. (%) | Dei. (%) | All (%) |
| LongVA ( Zhang et al., 2024a ) | 41.5 | 38.7 | 43.0 |
| VLLaMA3 ( Zhang et al., 2025a ) | 36.9 | 34.2 | 39.2 |
| LLaMA-VID ( Li et al., 2024c ) | 41.8 | 39.4 | 43.1 |
| Qwen3.5 ( QwenTeam, 2026 ) | 43.8 | 42.6 | 47.2 |
| Gemini-3.1-pro ( DeepMind, 2026 ) | 46.1 | 42.6 | 48.3 |
| GPT-5.5 ( OpenAI, 2026 ) | 47.6 | 43.9 | 50.5 |
Table 3: Performance comparison on chained and ego-deictic questions in BuddyVQA. MyBuddy consistently improves both capabilities, with particularly pronounced gains on ego-deictic questions.
| Model | No Feedback | Correctness | GT Answer | ||||
| Overall | Overall | – s | s | Overall | – s | s | |
| Qwen3.5 ( QwenTeam, 2026 ) | 47.2 | 49.0 | 51.8 | 45.9 | 52.3 | 54.2 | 50.1 |
| Gemini-3.1-pro ( DeepMind, 2026 ) | 48.3 | 50.4 | 53.0 | 47.4 | 54.0 | 55.9 | 51.7 |
| GPT-5.5 ( OpenAI, 2026 ) | 50.5 | 53.1 | 55.0 | 50.8 | 56.9 | 58.2 | 55.3 |
| MyBuddy (Qwen3.5) | 60.7 | 63.2 | 63.8 | 62.5 | 67.1 | 66.2 | 68.2 |
| MyBuddy (Gemini-3.1-pro) | 63.2 | 66.0 | 66.4 | 65.5 | 69.8 | 68.8 | 70.9 |
Table 4: Dynamic streaming evaluation on BuddyVQA. We report accuracy (%) under three settings for better comparisons: 1) No Feedback: No user feedback of historical questions, 2) Correctness: Binary feedback (correct or not) of historical questions, and 3) GT Answer: Ground-truth answers of historical questions are provided by users. The best and second-best results are highlighted in bold and underline, respectively.
| Model | RVS-Ego | RVS-Movie | Latency | GPU Usage | ||
| Acc. | Sco. | Acc. | Sco. | |||
| Flash-VStream ( Zhang et al., 2025c ) | 58.9 | 4.0 | 56.0 | 3.3 | 2.4s | 20GB |
| ReKV(LLaVA-OV) ( Di et al., 2025 ) | 63.7 | 4.0 | 54.4 | 3.6 | 3.3s | 38GB |
| LongVA ( Zhang et al., 2024a ) | 53.2 | 3.1 | 43.0 | 3.2 | 6.3s | 37GB |
| MovieChat ( Song et al., 2025 ) | 52.2 | 3.4 | 39.1 | 2.3 | 8.8s | 40GB |
| LLaMA-VID ( Li et al., 2024c ) | 53.4 | 3.9 | 48.6 | 3.3 | 11.9s | 33GB |
Table 5: Comparison on RVS-Ego and RVS-Movie ( Zhang et al., 2025c ) . Acc.: Accuracy. Sco.: Score. Latency & GPU Usage are tested on Tesla A100 GPU. Results are averaged on 10 videos with each 720 frames as inputs.
| Model | Video-MME | EgoSchema |
| Video-LLaMA3 ( Zhang et al., 2025a ) | 47.9 / 49.7 | 57.3 |
| VideoChat2 ( Li et al., 2024b ) | 39.5 / 43.8 | 56.7 |
| MovieChat ( Song et al., 2025 ) | 38.2 / - | 53.5 |
| Qwen3.5 ( Bai et al., 2025b ) | 71.2 / 74.3 | 68.1 |
| GPT-5.5 ( OpenAI, 2026 ) | 77.3 / 82.6 | 79.8 |
| LongVA ( Zhang et al., 2024a ) | 52.6 / - | - |
Table 6: Model comparison on Video-MME ( Fu et al., 2024 ) and EgoSchema ( Mangalam et al., 2024 ) . Video-MME: “without subs / with subs”. The best and second-best Accuracy results are bolded and underlined , respectively.
| Ablation Model | Chain (%) | Deictic (%) | ||
| ✗ | ✓ | ✗ | ✓ | |
| MyBuddy(Full) | 66.4 | 62.8 | 66.1 | 63.5 |
| w/o LM | 65.7 | 61.3 | 63.3 | 58.9 |
| w/o SM & LM | 64.1 | 58.0 | 57.8 | 55.8 |
| w/o AM & SM & LM | 59.2 | 53.1 | 53.4 | 52.9 |
| w/o Historical QA | 62.4 | 44.5 | 61.3 | 54.3 |
Table 7: Ablation of MyBuddy on BuddyVQA w.r.t interactively chained questions and questions with deictic expressions. AM/SM/LM: Active/Semi-Active/Low-Active Memory. For the setting of w/o SM & LM & AM, we use the question frame as input.
Figure 6: Streaming VQA latency & GPU Usage w.r.t. number of frames, tested on A100 GPUs. MyBuddy is able to process long streams more efficiently.
| Configuration | Acc. | Chained Acc. | Latency | GPU Usage | Extra Cost |
| Base (Raw F+His. QAs) | 47.2 | 43.8 | 3.9s | 51.1GB | – |
| + Multi-Level Memory | 51.4 | 48.2 | 2.5s | 22.9GB | -1.4s / -28.2GB |
| + Question Filter | 53.6 | 54.5 | 2.3s | 30.2GB | -1.6s / -20.9GB |
| + Key Visual Info Retrieval | 55.7 | 52.7 | 4.0s | 62.3GB | +0.1s / +11.2GB |
| + Hierarchical Rephrasing | 56.9 | 53.4 | 4.3s | 53.8GB | +0.4s / +2.7GB |
| MyBuddy(Full) | 60.7 | 57.3 | 2.6s | 29.1GB | -1.3s / -12.0GB |
Table 8: Efficiency breakdown on A100 GPUs. Extract Cost denotes additional overhead (latency/memory) introduced by individual modules. Base (Qwen3.5-9B): processes video frames & concatenates all historical QAs for each question online.
Figure 7: More result visualizations on BuddyVQA in Scrutinization and Recognition categories. Baseline models perform poorly on deictic questions, underscoring the importance of jointly leveraging historical interaction contexts and embodied visual cues for ego-deictic reference resolution.
Figure 8: More result visualizations on BuddyVQA in Recall and Advisory categories. MyBuddy demonstrates substantially stronger performance on long video sequences compared to general MLLMs. Its consistent improvements on chained questions further highlight the importance of history-aware reasoning across question chains in BuddyVQA.
| Failure | Num | Ratio (%) | Main Reasons |
| Que. Filter | 124 | 6.2 | Misclassifying chained questions |
| Vis. Retri. | 420 | 21.0 | Missing relevant frames or objects |
| Rephrase | 504 | 25.2 | Wrong referent grounding |
| Ans. Infer | 556 | 27.8 | Correct context, wrong inference |
| Others | 396 | 19.8 | Motion blur / Noisy input |
Table 9: Step-wise failure analysis of MyBuddy on the subset of BuddyVQA. While early-stage errors exist, a significant portion of failures arises from intrinsic ambiguity and visual challenges, rather than pipeline design alone.
| Evaluator | InternVL-3.5 | ReKV | ||
| Acc.(%) | Sco. | Acc.(%) | Sco. | |
| Human | 62.3 | 3.1 | 55.3 | 2.9 |
| GPT-5 | 61.8 | 3.1 | 53.7 | 2.8 |
Table 10: Comparison of human and LLM evaluators in terms of accuracy and score. The results show highly aligned scores between human and LLM judge.
Figure 9: Confusion matrices of evaluation results.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Activities in BuddyVQA.
| Category | Q Num. & Ratio | Examples |
| Recall | 5631(26.1%) | Who just passed by me? How much time passed after I started running? What was he doing several seconds ago? Did I lock the door before leaving? |
| Recognition | 5069(23.5%) | What’s on that? What’s that red thing next to the greens? Who did I just hand those to? |
| Scrutinization | 5523(25.6%) | Am I missing any spot while sweeping? Am I doing this fast enough? Did I take longer to complete this than before? Are these clothes of good quality? |
| Advisory | 5351(24.8%) | What should I do next? How long will it take to get through this line? How do I know when the area is smooth enough? |
Table 11: Question categories and examples of BuddyVQA.
| Method | D Acc.(%) | Latency |
| Direct Question Answering | 62.4 | 423ms |
| Hierarchical Rephrasing (Ours) | 86.7 | 552ms |
Table 12: Comparison of disambiguation accuracy (D Acc.) and latency w/ and w/o hierarchical rephrasing.
| Deictic Type | Original Question | Hierarchically Rephrasing | Final Answer |
| Unclear referents | “I need more, where is it?” | “What does it refer to?” “Does it refer to the cup?” “Where is the cup?” | “On the table.” |
| Vague temporal / spatial cues | “Has it started yet?” | “Does it refer to the meeting?” “Has the meeting started yet?” | “Yes, 2 minutes ago.” |
| Ambiguous actions | “Am I doing this right?” | “Does this refer to cutting the onion?” “Am I cutting the onion correctly?” | “Yes, but hold the knife away from your hand.” |
Table 13: Examples of hierarchical input rephrasing of MyBuddy.
| Method | Accuracy(%) | Corr. with Ques. | |
| Corr.(%) | Score | ||
| Human | 76.2 | 82.3 | 3.8 |
| GPT-5 ( OpenAI, 2025a ) | 81.3 | 89.1 | 4.1 |
Table 14: Accuracy of eye gaze points and hand pose detection, and the correlation of correctly detected features with questions in terms of correlated percentage (Corr.) and score.
| Configuration | Overall Acc. |
| Base GPT-5.5 Backbone (Raw F+His. QAs) | 50.5 |
| Buddy w/o Memory + Q Filter + Visual Info + Rephrasing | 52.2 |
| Buddy w/o Memory + Q Filter + Visual Info | 56.1 |
| Buddy w/o Memory + Q Filter | 59.2 |
| Buddy w/o Memory | 62.4 |
| Full MyBuddy(GPT-5.5) | 65.1 |
Table 15: Controlled ablation study on Buddy.
| Automatic Question Candidate Generation Prompts |
| # Highlight detection: |
| You are an advanced assistant specialized in video analysis. |
| Your task is to analyze the provided video and identify its key highlights based on significant visual content, such as rapid action, noteworthy events, or shifts in visual dynamics. The input is a video file and the output should be a list of timestamps that correspond to the start and end of each highlight segment in the video. Please focus on identifying segments with high visual or contextual importance. Answer with ONLY timestamp pairs. |
| Ensure the timestamps are accurate and clearly marked in seconds, e.g., [00:30, 00:45], [02:21, 02:51]. |
| Video: { video } |
| # Question generation: |
Table 16: Prompts for automatic question candidate generation of BuddyVQA.
| Evaluation Prompts for BuddyVQA |
| You are an intelligent chatbot designed for evaluating the correctness of generative outputs for egocentric, streaming question-answer pairs in the BuddyVQA dataset. Your task is to compare the predicted answer with the correct answer set and determine whether they meaningfully match, taking into account the characteristics of egocentric video, ego-deictic expressions, and interactive question chains. |
| The following are the requirements of evaluation: |
| a. Focus on the meaningful match between the predicted answer and the correct answer sets. In BuddyVQA, many questions contain ego-deictic expressions such as “this”, “that”, “it”, “here”, or “there”. Ensure that the predicted answer resolves the same referent as the ground truth, even if expressed differently. |
| b. The correct answers are provided as a set . Your evaluation should treat the prediction as correct if it meaningfully matches any of the answers in the set. The match does not need to be exact, but must refer to the same grounded entity, event, or semantic content. |
| c. Consider the temporal continuity of chained questions. Some BuddyVQA questions depend on prior context ( e.g. follow-up questions like “Is it still there?” or “Did I do it correctly?”). When evaluating, ensure that the predicted answer maintains the same contextual interpretation and refers to the correct entity or past event, consistent with the ground-truth chain. |
Table 17: Prompts for GPT-5 ( OpenAI, 2025a ) to evaluate MLLMs on BuddyVQA.
| Low-Active Memory Summarization Prompt |
| You are an intelligent multimodal assistant tasked with summarizing egocentric video segments to extract long-term, stable, and reusable contextual information. Your goal is to produce a concise textual summary that captures information in long videos, including key activities, recurring patterns, background environment, salient objects, user routines, and consistent spatial layout, etc. |
| # Instructions: |
| - Focus on information that remains stable or frequently appears across this time window. Identify recurring actions, repeated interactions, or common tasks performed by the user. |
| - Capture stable environmental structure such as room layout, tool locations, furniture arrangement, or habitual user workflows. |
| - Extract objects or entities that reliably appear or are frequently manipulated across the window. |
Table 18: Prompt for zero-shot summarization model used to construct Low-Active Memory.
| Question Filtering Prompt |
| You are an intelligent multimodal assistant designed to identify whether an incoming question in a streaming egocentric QA scenario is chained (dependent on prior QA turns) or unchained (independent). Your task is to analyze the current question, compare it with a buffer of historical QA pairs and the current frame, and determine: (1) whether the question relies on earlier context, and (2) if so, which prior QA pair is the most relevant for resolving this dependency. |
| # Instructions: |
| - Classify the incoming question as either “chained” or “unchained” . A chained question typically contains ego-deictic expressions (e.g., “this”, “that”, “it”, “there”) or implicit references to past events, actions, or answers (e.g., “Is it still there?”, “Did I do it correctly?”, “What about now?”) that require information from historical QAs to answer. An unchained question is self-contained and understandable without referencing earlier context. |
| - If the question is classified as unchained, return the label “unchained” and do not select any historical QA pair. |
| - If the question is classified as chained, inspect the historical QA buffer and identify the single most relevant QA pair that resolves the ambiguous referent or contextual dependency. Relevance may arise from: semantic similarity, shared objects or actions, temporal continuity, or matching referential expressions. |
Table 19: Prompt for the Question Filter used to detect chained questions and retrieve relevant historical QA pairs.
| Visual Information Retrieval Prompt |
| You are an intelligent multimodal assistant designed to retrieve visual information from egocentric video memories in order to support real-time question answering. Your task is to (1) generate concise visual captions, (2) filter long-term summaries for information relevant to the incoming question, and (3) detect embodied features such as eye-gaze points and hand bounding boxes that help ground deictic references. All three subtasks must be performed using the same lightweight MLLM capabilities. |
| # Instructions for Captioning (from Active and Semi-Active Memory): |
| - Generate a concise and accurate caption describing the visually salient content in the provided video frames. Focus on the objects, spatial relations, user interactions, and scene attributes that are relevant for answering a question. |
| - The caption should reflect only the visible content and be grounded in the egocentric viewpoint. |
Table 20: Prompt for the Visual Information Retrieval (VIR) module.
| Zero-shot Question Answering Prompt |
| You are an intelligent multimodal assistant designed to answer egocentric, streaming questions using retrieved visual information, historical QA context, and embodied features. Your task is to first rephrase the incoming question into an explicit, fully disambiguated form and then generate a correct answer grounded in the provided multimodal context. |
| # Instructions for Hierarchical Input Rephrasing: |
| Rewrite the question in a clearer and explicit form that removes ambiguity. Resolve ego-deictic expressions such as “this”, “that”, “it”, “here”, or “there” by grounding them using the provided context. Incorporate historical QA, captions, and embodied features to determine the correct referent or event for rephrasing. The rephrased question must express the user’s intent explicitly and be fully interpretable without additional context. Provide only one rephrased question—concise, unambiguous, and grounded. |
| Here is an example: |
| Incoming Question: “Where is it now?” |
Table 21: Prompt for zero-shot question answering with hierarchical rephrasing.
More in this topic
Egocentric Video QA
Egocentric video question answering uses first-person camera footage to answer grounded questions about activities, objects, space, and time.
See all 46 papers