cs.CVOct 22, 2026

Retrieval-Augmented Online Textual Memory for Episodic Memory Video Question Answering

Authors: Raffaele Cali, Giuseppe Lando, Rosario Forte, Antonino Furnari

Organizations: Department of Mathematics and Computer Science, University of Catania, Italy

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Long egocentric videos pose a central challenge for multimodal language models: the visual stream must be compressed into a memory representation that can be searched efficiently while still preserving the evidence needed for question answering. Building on an online textual-memory framework in which first-person videos are converted into temporally grounded captions, this work systematically studies the role of retrieval in episodic-memory video question answering. We evaluate how different retrieval units, query formulations, context budgets, and reranking strategies affect multiple-choice answering on QAEgo4D-Closed, isolating retrieval as the main variable while keeping the underlying generated memory fixed. Our best configuration, Parent, matches fine-grained caption subchunks, then provides their full parent captions to the reasoner. It achieves 56.4 % accuracy, matching strong full-memory and retrieval baselines while exposing only a small fraction of the textual memory at question time. More importantly, our analysis shows that better temporal coverage does not necessarily translate into better answers: increasing the retrieval budget improves temporal Hit@K and Recall@K, but can reduce accuracy by introducing visually plausible distractors and uncertain caption evidence. These findings suggest that retrieval-augmented egocentric memory should not be optimized only for evidence recall, but also for the reliability, specificity, and answer-discriminative value of the retrieved context. The study motivates more structured episodic memories that move beyond raw caption retrieval toward evidence-aware representations for long-form egocentric reasoning.

Figures & tables

Fig. 1 : Overview of our retrieval-augmented textual-memory pipeline for egocentric video question answering. The egocentric video stream is processed online by a video descriptor model into captions, which are stored in an online textual memory. At question-answering time, the system uses the question and answer options to retrieve the most relevant subchunks, which are expanded to their parent captions and passed to the reasoner to predict the answer.
Fig. 2 : Overview of the proposed retrieval-augmented online textual-memory pipeline. The video stream is processed incrementally by a descriptor model into captions, which are then chunked, embedded, and stored in a vector database. At question-answering time, the query and candidate answers are used to retrieve relevant memory entries, which are provided to the reasoner to predict the final answer.
Fig. 3 : The proposed retrieval strategies.
SystemAcc.Hit@KRecall@K
Parent56.4%79.0%70.7%
Flat55.6%80.4%72.3%
Sub54.2%76.6%67.5%
Parent-Rerank53.8%79.0%70.7%
Sub-Rerank53.4%82.4%74.3%
Hybrid53.0%78.8%69.1%
TABLE I : Ablation of the proposed RAG configurations on QAEgo4D-Closed using a fixed retrieval budget of K=12K=12 chunks.
Fig. 4 : Retrieval budget curve for Parent . Increasing KK improves temporal coverage, but answer accuracy peaks at an intermediate retrieval budget.
Fig. 5 : Qualitative examples of retrieval-augmented generation behavior.
MethodAcc.MemoryTokens
Ground VQA [ 3 ]48.7%1.29 MB/min–
ReKV-LLaVAOneVision 0.5B [ 7 ]50.0%68.92 MB/min12,544
ReKV-LLaVAOneVision 7B [ 7 ]56.0%329.64 MB/min12,544
Lando et al. Gemini-based [ 8 ]56.0%3.57 KB/min10,880
Parent (Ours)56.4%66.07 KB/min3,600
TABLE II : Comparison with state-of-the-art methods on QAEgo4D-Closed. For methods based on MLLMs, we report the number of tokens used at query time.
More in this topic

Egocentric Video QA

Egocentric video question answering uses first-person camera footage to answer grounded questions about activities, objects, space, and time.

See all 46 papers

Explore similar work

CardsList
Sep 15, 2026

CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video

Wearable assistants require episodic memory over egocentric video, yet current vision-language models face bounded frame budgets, growing visual-token costs, and long-context retrieval failures. Under these practical constraints, we study whether textual captions can serve as reusable episodic memory. We define the Episodic Memory Video Caption QA task and introduce CapMem, a human-annotated benchmark with 75 videos totaling 33.7 hours, and 1,000 multiple-choice questions across 16 scenarios. On long videos (>20 min), full-coverage CaptionQA with 30s and 60s caption windows outperforms direct VideoQA for 10/12 and 8/12 models, respectively. On the same video subset, a matched-frame control across six Qwen models retains mean accuracy gains of 3.22 and 2.55 points, respectively. Our caption-guided retrieve-and-verify harness further improves accuracy by up to 5.3 points. These results support the effectiveness of caption memory for episodic reasoning over long egocentric video.

Dingli Liang, Yiqiao Xie, Yukai Huang +6

University College London · Imperial College London · Durham University +4

Memory-Augmented VLMsEgocentric Video QAVideo QAEgocentric Video Understanding
AskPDF
Sep 1, 2026

EM^2Mem: Event-Centric Multimodal Memory for Large Language Models

Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM^2Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM^2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by 4.67 times and total inference tokens by 63.66% (The code will be integrated into https://github.com/zjunlp/LightMem).

Yijun Chen, Yaqi Zheng, Yanya Li +11

Zhejiang University · South China University of Technology · Lenovo Group Limited

Multimodal RAGEfficient Multimodal InferenceMultimodal MemoryMemory-Augmented Video UnderstandingLong-Video QA
AskPDF
Aug 12, 2026

EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory

Long-horizon egocentric memory transforms continuous first-person video and audio into a searchable record of past experiences. We demonstrate two bottlenecks in existing systems: indices built from context-poor captions are unreliable for agentic search, while retrieval ignores a question's temporal intent. To address both bottlenecks, we introduce EgoCITE (Egocentric Context-augmented Indexing and Time-aware Evidence retrieval), a long-horizon agentic memory framework for egocentric QA. EgoCITE comprises three components. EgoScheme uses local multimodal context to turn fragmentary video captions and speech transcripts into self-contained atomic memory indices. EgoIndex organizes complementary action, activity, utterance, and conversation representations into searchable multi-view memory indices at multiple granularities. EgoRetrv combines semantic search with question-conditioned temporal relevance scoring and curation of retrieved evidence. We evaluate EgoCITE on EgoLifeQA, EgoMem, and EgoR1-Bench in terms of answer accuracy and target-event retrieval alignment. EgoCITE improves accuracy over agentic memory baselines by at least 4.4--14.2% while achieving 36×\times lower cost than long-context LLM agents.

Le Zhang, Hao Chen, Vlad Roznyatovskiy +2

University of Michigan, Ann Arbor

Temporal IRMultimodal QAAgentic RetrievalLong-Horizon LLM Agents
AskPDF