VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding

Yucheng Liu; Yufei Yin; Mingxiao Feng; Jiajun Deng; Wengang Zhou; Houqiang Li

cs.CVOct 5, 2026

VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding

Authors: Yucheng Liu, Yufei Yin, Mingxiao Feng, Jiajun Deng, Wengang Zhou, Houqiang Li

Organizations: University of Science and Technology of China · Jilin University · Hangzhou Dianzi University · Institute of Artificial Intelligence, Hefei Comprehensive National Science Center

Abstract

Long-video understanding places substantial demands on memory, as answering questions often requires retrieving information distributed across extended temporal spans. Existing approaches broadly follow two paradigms: query-driven exploration, which is sensitive to localization errors, and query-independent memory construction, which may omit question-specific details. We introduce VideoTapestry, a training-free multi-agent framework that adapts a preconstructed hierarchical video memory through coarse-to-fine, query-driven refinement. The preconstructed memory organizes video content into three levels, capturing global narrative context, event-level temporal structure, and fine-grained relational evidence, respectively. To support coarse-to-fine localization and observation, we assign a specialized agent to each level, keeping retrieval and refinement within a scale-specific context. Guided by the query, these agents revisit relevant video regions and enrich layer-wise memories with targeted multimodal observations. Their refinements are assembled according to the original hierarchy into a composite query-adaptive memory, preserving global context in a compact form while retaining fine-grained evidence along query-relevant branches for final reasoning. Compared with direct GPT-5.5 inference, VideoTapestry achieves absolute accuracy gains of 17.2%, 14.9%, 9.8%, and 7.0% on LVBench, LongVideoBench (Long), Video-MME (Long), and EgoSchema, respectively, achieving the state-of-the-art results among all competitors.

Figures & tables

Figure 1: Comparison of long-video context construction paradigms. Left: Top-down exploration may miss answer-critical evidence due to localization errors. Middle: Bottom-up memory provides broad coverage but may omit query-specific details. Right: VideoTapestry combines a query-independent hierarchy with targeted revisiting to form a composite query-adaptive memory that preserves global context and enriches relevant evidence.

Figure 2: Overview of VideoTapestry. Scale-aligned agents operate over a preconstructed hierarchical video memory, while layer-wise tool-augmented refinement selectively incorporates newly acquired query-relevant observations at different semantic levels. The resulting refined layer memories are hierarchically assembled into a composite query-adaptive memory for final prediction.

Direct Proprietary VLMs
Method	Query-time Model(s)	LVBench	LongVideoBench		Video-MME(w/o)	EgoSchema
		Overall	Val	Long	Long	Val
Gemini-2.5-Pro	Gemini-2.5-Pro	72.0	71.0	68.6	75.9	72.8
Gemini-3.1-Pro	Gemini-3.1-Pro	78.2	78.6	77.0	80.3	76.4
OpenAI o3	OpenAI o3	57.1	66.7	60.6	64.7	63.2
Direct Open-Source VLMs

Table 1: Comparison with representative VLMs and agentic approaches on four long-video understanding benchmarks. Results are reported as accuracy (%). Green parenthesized values denote percentage-point gains over GPT-5.5 (Direct).

\dagger

denotes our same-backbone end-to-end evaluation with frozen GPT-5.5.

\ddagger

denotes results from our reproduction. w/o: no provided subtitles.

Direct VLMs
Method	ER	EU	Rea	KIR	Sum	TG	Avg.
GPT-4o	48.9	49.5	50.3	48.1	50.0	40.9	48.9
OpenAI o3	57.6	56.4	50.8	62.9	67.2	46.8	57.1
GPT-5.5 (Direct)	63.2	66.0	72.7	63.8	65.4	43.1	64.3
Structured and Agentic Methods
VideoTree	30.3	25.1	31.9	26.5	25.5	27.7	28.8

Table 2: Comparison with representative VLMs and agentic approaches across the six capability dimensions of LVBench (accuracy, %). The best results are bolded; green parenthesized values denote absolute percentage-point gains over GPT-5.5 (Direct).

\ddagger

denotes results from our reproduction.

Backbone	Direct	VideoTapestry	Improve.
GPT-5.5	78.0	87.0	$+9.0$
GPT-5	75.5	82.5	$+7.0$
Qwen3.7-Plus	75.0	83.0	$+8.0$
Qwen3.6-27B	65.5	75.0	$+9.5$

Table 4: Generalization across backbone models.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Two examples of coarse-to-fine query-adaptive memory refinement.

Explore similar work

Sprout: Building Dynamic Memory While Reasoning for Agentic Video Understanding
Sep 28, 2026Wei Chen, Xuanyu Zheng, Yancheng Long +6Video Understanding Video Agent
HKUST · Kling AI
VideoEvolve: Co-Evolving Memory and Retrieval for Long Video Understanding
Oct 7, 2026Yongchao Xu, Bowen Ye, Jiefeng Gan +5Video Understanding Multimodal Memory
Alibaba Group, Hangzhou, China · University of Science and Technology of China, Hefei, China · Shanghai Jiao Tong University, Shanghai, China +2
Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding
Aug 7, 2026Yeeun Choi, Youngbeom Yoo, Joon-Young Lee +2Video Understanding Episodic Memory
Yonsei University · Adobe Research

Sep 28, 2026, cs.CVJ/K move · Enter open · S save

Sprout: Building Dynamic Memory While Reasoning for Agentic Video Understanding

Wei Chen, Xuanyu Zheng, Yancheng Long, Haoyang Xu+5

HKUST · Kling AI

Video Understanding Video Agent Streaming Video Understanding Precision Agriculture

Open paper AskPDF arXiv

Long video understanding relies on video memory to overcome the context limits of multimodal large language models. Existing methods follow a build-then-reasoning pipeline: memory is built offline for the entire video, then reasoned over as a static source. In practice a long video is shared by several questions, and this pipeline is costly at both ends: with few questions, building memory for the whole video costs far more than answering them; with many questions, the memory is never updated, so what is learned while answering questions is lost to the next question. To alleviate these, we introduce Sprout, an agentic framework that builds memory while reasoning: a temporal tree that sprouts detailed nodes as questions are answered. The agent watches the video segment by segment at a low frame rate, stopping when the current question can be answered, remembers each segment as a coarse node of the tree, and revisits key intervals at a higher frame rate to refine the tree with the recovered details. Once a segment is recorded as text, its video input is removed from the context history, while the original video remains reachable through the video tools. The memory tree and prior question--answer records persist across questions, so the memory is online and dynamic: built from the first question onward and updated by every question thereafter. We find that replacing accumulated video inputs with textual memory substantially reduces context usage while maintaining accuracy, with slight improvements in some settings. Across benchmarks on three models, Sprout achieves competitive or improved accuracy relative to representative offline memory methods, with no upfront construction stage and lower context cost per question.

Abstract

Explore similar work

Sprout: Building Dynamic Memory While Reasoning for Agentic Video Understanding

VideoEvolve: Co-Evolving Memory and Retrieval for Long Video Understanding

Keep It Simple: Multi-Key Episodic Memory Retrieval for Ultra-Long Video Understanding