CAM: Question Answering on Entity-Centric Videos with Continuous Extraction and Adaptive Querying

Yizhou Tian; Zizhe Chen; Shiyuan Deng; Garry Yang; Zijie Dai; Luohao Pan; Hao Lin; Peiqi Yin

cs.CVSep 6, 2026

CAM: Question Answering on Entity-Centric Videos with Continuous Extraction and Adaptive Querying

Authors: Yizhou Tian, Zizhe Chen, Shiyuan Deng, Garry Yang, Zijie Dai, Luohao Pan, Hao Lin, Peiqi Yin,

Organizations: Department of Computer Science and Engineering, The Chinese University of Hong Kong · Knowin AI · Institute for Math and AI, Wuhan University

Abstract

Memory facilitates question answering over long videos by extracting and retrieving facts to fit within the limited context windows of multimodal LLMs (MLLMs). Existing solutions typically extract independent memory entries from fixed-length video clips and thus cannot capture high-level semantics that need to be summarized over extended time periods, such as character traits and relations. Moreover, they rely solely on similarity-based retrieval and may fail to retrieve the fine-grained details required for question answering. To tackle these problems, we propose CAM, featuring continuous extraction for high-level semantics and adaptive querying for fine-grained details. In particular, CAM stores the entities and relations extracted from video clips in a knowledge graph. To capture the high-level semantics of each entity or relation, CAM summarizes the local subgraph of the target entity or relation once the subgraph reaches a predefined size. To retrieve the fine-grained details required for question answering, CAM supports multiple search methods, including knowledge graph traversal, video re-watching, and audio listening. It utilizes a planner-executor-verifier pipeline to adaptively compose these search methods according to question intent. Evaluations on three benchmarks show that CAM outperforms SOTA baselines and improves their accuracy by up to 23 percentage points. Code is available at https://github.com/Jake-Tian/CAM.

Figures & tables

Figure 1: Comparing existing video memory systems and our CAM. The blue “M" denotes memory construction, while yellow “R" denotes question answering. (a) EgoRAG [ 5 ] generates a text caption for each video clip and searches these captions. (b) M3-Agent [ 8 ] extracts facts related to each entity. (c)WorldMM [ 7 ] builds multiple memory modules but completely decouples visual features from textual knowledge, leading to fragmented retrieval. (d) CAM (Ours) organizes the entities and relations into a temporally grounded, heterogeneous knowledge graph, continuously summarizes high-level semantics about each entity and relation, and searches the knowledge graph adaptively using multiple methods.

Figure 2: The memory construction workflow of CAM. This unified architecture effectively bridges the gap between low-level episodic observations and emergent high-level semantics, supporting robust and efficient question answering over extended temporal horizons.

Figure 3: The adaptive reasoning workflow of CAM. This iterative feedback loop enables the system to dynamically refine its search strategy based on intermediate findings, ensuring that complex multi-step queries are resolved with high precision and minimal retrieval overhead.

Base Models
Method	M3-Bench-robot						M3-Bench-web						HippoVlog
Method	ME	MH	CM	PU	GK	All	ME	MH	CM	PU	GK	All	HippoVlog
Qwen3-VL-flash	9.3	9.2	8.4	10.7	7.7	9.0	11.9	10.5	13.4	14.0	20.9	14.9	24.2
Gemini-3-Pro	15.5	19.8	14.1	27.1	7.2	15.3	18.0	17.9	23.8	23.1	28.7	23.2	38.0
GPT-5.2	18.3	16.2	18.7	29.5	15.7	18.0	21.3	21.9	30.9	27.1	39.6	28.7	38.8
Online Video Understanding Methods

Table 1: Question answering accuracy. All video memory methods use GPT as the MLLM except HippoMM due to efficiency considerations. Besides GPT, we also report the accuracy of our CAM when using Qwen as the MLLM to demonstrate generality.

Panel of Figure 4 — Figure 4: Token consumption comparison. (a) Memorization tokens per minute. (b) Reasoning tokens per question.

Ablation on memory construction
	M3-Bench-robot
Method	ME	MH	CM	PU	GK	All
Fixed-interval summarization	62.4	75.8	61.5	74.2	32.8	60.3
CAM w/o graph	43.4	56.8	47.9	57.9	27.3	44.8
CAM w/o high-level	54.6	66.9	51.5	79.9	29.1	55.7
Ablation on question answering

Table 2: Ablation study for the designs of CAM. Structured graph memory and adaptive querying tools both contribute critically to overall reasoning performance.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Association method	Total errors	Splits	Merge/miss	Added latency
CAM natural-language association	14	14	0	0 s
InsightFace	18	1	17	8.5 s

Appendix

Table 3: Character-association errors on the manually annotated subset. “Merge/miss” combines false merges and missed identities. Added latency is measured per 30-second clip on an RTX 4090.

Setting ( $\tau_{\mathrm{char}}/\tau_{\mathrm{rel}}$ )	M3-Bench-robot						M3-Bench-web
Setting ( $\tau_{\mathrm{char}}/\tau_{\mathrm{rel}}$ )	ME	MH	CM	PU	GK	All	ME	MH	CM	PU	GK	All
Fixed-interval	62.4	75.8	61.5	74.2	32.8	60.3	58.6	50.0	56.3	75.6	71.7	66.9
$30/10$	63.6	73.6	63.6	81.9	33.0	61.9	63.6	57.8	47.9	74.0	67.9	67.5
$50/30$ (default)	65.5	80.8	64.8	82.3	35.3	64.5	64.6	54.7	52.1	75.6	75.5	69.3
$100/60$	63.7	74.8	64.6	82.8	35.0	63.1	61.6	51.6	50.0	76.3	73.6	68.1

Appendix

Table 4: Accuracy (%) under different abstraction settings on M3-Bench-robot and M3-Bench-web. Threshold pairs denote

\tau_{\mathrm{char}}/\tau_{\mathrm{rel}}

; the fixed-interval variant does not use evidence thresholds.

Method	M3-Bench-robot			M3-Bench-web			HippoVlog
Method	Avg rounds	correct	wrong	Avg rounds	correct	wrong	Avg rounds	correct	wrong
M3-Agent	3.14	2.55	3.58	3.11	2.68	3.52	2.76	2.46	3.33
WorldMM	2.74	1.85	3.22	2.21	2.08	2.41	1.28	1.2	1.49
CAM	2.2	1.78	2.92	2.48	2.36	2.71	2.13	2.03	2.41

Appendix

Table 5: Average reasoning rounds required by different methods across benchmarks, separated by correct and incorrect predictions.

Benchmark	1st Most Common (%)	2nd Most Common (%)	3rd Most Common (%)
M3-Bench-robot	general_search (70.2%)	watch_video_clip (16.8%)	search_temporal_context (8.6%)
M3-Bench-web	general_search (78.6%)	search_temporal_context (13.9%)	watch_video_clip (3.7%)
HippoVlog	general_search (73.8%)	watch_video_clip (13.3%)	listen_to_audio (7.4%)

Appendix

Table 6: Distribution of the top three most frequently invoked tools by the agent during the reasoning process across different benchmarks.

Injected noise ratio	M3-Bench-robot						M3-Bench-web
Injected noise ratio	ME	MH	CM	PU	GK	All	ME	MH	CM	PU	GK	All
$0\%$ (clean)	65.5	80.8	64.8	82.3	35.3	64.5	64.6	54.7	52.1	75.6	75.5	69.3
$2\%$	62.4	77.6	62.6	83.3	32.3	62.5	64.4	55.1	49.3	74.7	75.5	68.5
$5\%$	59.4	79.9	62.4	82.4	33.8	62.5	63.9	54.3	47.7	72.8	77.5	67.9
$10\%$	51.5	73.3	61.4	81.9	30.4	60.8	61.9	50.7	44.2	68.9	71.6	65.2

Appendix

Table 7: Robustness to graph noise. We inject low-level edges sampled from other graphs before abstraction and question answering, and report accuracy (%) on M3-Bench-robot and M3-Bench-web.

Method	Latency per question	Accuracy (%)
M3-Agent	71.4 s	41.4
WorldMM	30.0 s	34.5
CAM (top- $k=30$ )	33.8 s	62.9
CAM-full	47.3 s	64.5

Appendix

Table 8: Average reasoning latency per question and accuracy on M3-Bench-robot.

Figure 5: Qualitative comparison of graph topologies. (a) CAM memory graph for a 2-minute video. (b) CAM memory graph for a 20-minute video, demonstrating scalable and organized relational density over extended temporal horizons. (c) A standard text-based GraphRAG topology for a written document, highlighting the structural differences between general text abstraction and CAM’s entity-centric video memory.

Video QA

Video question answering studies how AI systems use visual and audio evidence over time to answer questions about video.

See all 180 papers

Jun 4, 2026

MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering

Long-video question answering remains challenging for Vision-Language Models (VLMs), as answer-relevant evidence is often sparse, transient, and temporally dispersed across lengthy video contexts. Existing frame-centric approaches improve efficiency through uniform sampling, query-aware frame selection, visual-token compression, and adaptive resolution strategies. However, they still rely on isolated and fragmented frames as the fundamental evidence units, limiting VLMs' ability to effectively capture coherent event-level semantics. To address this limitation, we propose MemoryCard, a video-memory-based augmentation framework that organizes long videos into self-contained Memory Cards. Specifically, MemoryCard first performs a self-reading process over videos and aligned utterances to segment the video into semantically coherent units, each corresponding to a distinct topic or event. For each unit, it generates an event-level video gist and selects representative visual moments, which are then rendered into unified Memory Cards for retrieval and question answering. Experimental results demonstrate that MemoryCard consistently improves long-video QA performance under comparable visual-token budgets, achieving up to a 21.8% relative improvement in accuracy. All code is available at https://github.com/NEUIR/MemoryCard.

Qing Yang, Pengcheng Huang, Xinze Li +6

School of Computer Science and Engineering, Northeastern University, Shenyang, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China · Digital China Group, Beijing, China

Video Summarization Video QA Memory-Augmented Video Understanding Long-Video QA

Ask PDF

Mar 23, 2026

VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understanding

Long video understanding remains challenging for multimodal large language models (MLLMs) due to limited context windows, which necessitate identifying sparse query-relevant video segments. However, existing methods predominantly localize clues based solely on the query, overlooking the video's intrinsic structure and varying relevance across segments. To address this, we propose VideoDetective, a framework that integrates query-to-segment relevance and inter-segment affinity for effective clue hunting in long-video question answering. Specifically, we divide a video into various segments and represent them as a visual-temporal affinity graph built from visual similarity and temporal proximity. We then perform a Hypothesis-Verification-Refinement loop to estimate relevance scores of observed segments to the query and propagate them to unseen segments, yielding a global relevance distribution that guides the localization of the most critical segments for final answering with sparse observation. Experiments show our method consistently achieves substantial gains across a wide range of mainstream MLLMs on representative benchmarks, with accuracy improvements of up to 7.5% on VideoMME-long. Our code is available at https://videodetective.github.io/

Ruoliu Yang, Chu Wu, Caifeng Shan +2

Nanjing University · Institute of Automation, Chinese Academy of Sciences

Long-Video Understanding Long-Video QA Query-Aware Video Frame Selection Temporal Video Understanding

Ask PDF

Sep 1, 2026

EM^2Mem: Event-Centric Multimodal Memory for Large Language Models

Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM^2Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM^2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by 4.67 times and total inference tokens by 63.66% (The code will be integrated into https://github.com/zjunlp/LightMem).

Yijun Chen, Yaqi Zheng, Yanya Li +11

Zhejiang University · South China University of Technology · Lenovo Group Limited

Multimodal RAG Efficient Multimodal Inference Multimodal Memory Memory-Augmented Video Understanding Long-Video QA

Ask PDF

Abstract

Video QA

Explore similar work

MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering

VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understanding

EM^2Mem: Event-Centric Multimodal Memory for Large Language Models