Question-guided Visual Compression with Memory Feedback for Long-Term Video Understanding

Sosuke Yamao; Natsuki Miyahara; Yuankai Qi; Shun Takeuchi

cs.CVMar 16, 2026

Question-guided Visual Compression with Memory Feedback for Long-Term Video Understanding

Authors: Sosuke Yamao, Natsuki Miyahara, Yuankai Qi, Shun Takeuchi

Organizations: Fujitsu Research · Macquarie University

Abstract

In the context of long-term video understanding with large multimodal models, many frameworks have been proposed. Although transformer-based visual compressors and memory-augmented approaches are often used to process long videos, they usually compress each frame independently and therefore fail to achieve strong performance on tasks that require understanding complete events, such as temporal ordering tasks in MLVU and VNBench. This motivates us to rethink the conventional one-way scheme from perception to memory, and instead establish a feedbackdriven process in which past visual contexts stored in the context memory can benefit ongoing perception. To this end, we propose Question-guided Visual Compression with Memory Feedback (QViC-MF), a framework for long-term video understanding. At its core is a Question-guided Multimodal Selective Attention (QMSA), which learns to preserve visual information related to the given question from both the current clip and the past related frames from the memory. The compressor and memory feedback work iteratively for each clip of the entire video. This simple yet effective design yields large performance gains on longterm video understanding tasks. Extensive experiments show that our method achieves significant improvement over current state-of-the-art methods by 6.1% on MLVU test, 8.3% on LVBench, 18.3% on VNBench Long, and 3.7% on VideoMME Long. The code will be released publicly.

Figures & tables

Figure 1: (a) Conventional frameworks [ 11 , 7 , 29 , 20 ] compress each frame independently without feedback from memory to perception, resulting in limited temporal reasoning and contextual inconsistency. (b) The proposed Question-guided Visual Compression with Memory Feedback (QViC-MF) framework performs multi-frame compression guided by both the question and recalled memory, enabling feedback-driven perception that preserves temporal event completeness.

Figure 2: (a) Overview of the proposed Question-guided Visual Compression with Memory Feedback (QViC-MF) framework (Section 3.1 ). The video is processed sequentially in clip-wise steps indexed by $n$ . The visual encoder and projector extract visual embeddings from the current clip $\mathcal{V}_{\mathrm{c},n}$ and recalled frames $\mathcal{V}_{\mathrm{r},n}$ from the context memory $\mathcal{B}_{n-1}$ , which are fed into a visual compressor equipped with Question-guided Multimodal Selective Attention (QMSA). The compressor transforms the context seed embeddings $\mathcal{E}_{\mathrm{c}_{0}}$ into context embeddings $\mathcal{E}_{\mathrm{c},n}$ by selectively compressing visual features conditioned on the question. The context memory $\mathcal{B}_{n}$ stores the context embeddings with their frame indices $\mathcal{S}_{n}$ and relevance scores $\mathcal{R}_{n}$ , incrementally appending new entries and pruning low-relevance ones to form the updated memory. From this memory, the top- $K_{\mathrm{r}}$ most relevant frame indices are retrieved to form the recalled frames $\mathcal{V}_{\mathrm{r},n+1}$ used for the next clip. Finally, the LLM decoder generates the text answer based on the updated memory. Trainable components are marked with flame icons. (b) Illustration of QMSA (Section 3.2 ). QMSA regulates multimodal attention through Mask, Block, and Guide operations, enabling frame-wise compression while preserving cross-frame context and focusing on question-related features. Here, Ctx2Vis and Ctx2Txt denote context-to-visual and context-to-text attention, respectively.

Figure 3: (a) Without our blocking context-to-text (Ctx2Txt), i.e., when using a naive variant of our visual compressor with standard causal masking in the self-attention module, textual information from the question may leak into the context embeddings, causing compression hallucination, where the embeddings no longer represent pure visual features. (b) Impact of guiding context-to-visual (Ctx2Vis). Without guiding Ctx2Vis, the model exhibits question-insensitive attention, failing to adapt its focus to the question. Incorporating guiding Ctx2Vis enables question-adaptive attention, allowing the model to attend selectively to regions relevant to the queried content.

		#Visual	MLVU		LVBench	VideoMME w/o sub.		VNBench 4-try eval.
Method	Size (LLM)	Tokens	test	dev	Overall	Overall	Long	Overall	Long
Duration			3–120 min.		30–120 min.	1–60 min.	30–60 min.	10–180 sec.	60–180 sec.
Proprietary Models
GPT-4o [ 1 ]	-	-	54.9	64.6	30.8	71.9	65.3	64.4	56.3
Gemini 1.5 Pro [ 13 ]	-	-	-	-	33.1	75.0	67.4	66.7	65.1
Open-source Models (w/o temporal compression)

Table 1: Comparison of the proposed QViC-MF framework with SoTA methods on long-term video understanding benchmarks. The “#Visual Tokens” column denotes the average number of visual tokens per frame input to the LLM. The best and second-best results are highlighted in bold and underlined, respectively.

Method	Avg	TR	AR	NQA	ER	PQA	SQA	AO	AC	TQA
GPT-4o	54.9	83.7	68.8	42.9	47.8	57.1	63.6	46.2	35.0	48.7
LLaMA-VID	17.2	20.9	23.1	21.7	11.3	16.0	16.7	18.6	15.0	11.6
MovieChat	18.0	18.7	10.3	23.3	15.1	16.0	30.6	17.1	15.0	16.3
MA-LMM	22.0	44.0	23.1	13.3	30.2	14.0	27.8	18.6	13.3	14.0
LongVA	41.1	81.3	41.0	46.7	39.6	46.0	44.4	17.1	23.3	30.2
Video-XL	45.5	78.0	28.2	50.0	41.5	46.0	41.6	48.6	31.7	44.2

Table 2: Detailed comparison of QViC-MF with SoTA methods on the MLVU test. The best and second-best results are highlighted in bold and underlined, respectively.

		MLVU	VNBench
Settings	Input	test	ALL	Long
Visual Compressor (Vanilla)	64 frm	48.8	61.6	39.3
+ Context Memory	2 fps	42.4	58.2	40.9
+ Memory Feedback	2 fps	52.2	63.7	55.1
+ Framewise Ctx2Vis ( $\mathbf{M}$ )	2 fps	53.0	61.1	50.7
+ Blocking Ctx2Txt ( $\mathbf{B}$ )	2 fps	57.9	63.2	56.7

Table 3: Ablation study of QViC-MF on MLVU and VNBench. Vanilla refers to a naive variant of our visual compressor that uses standard causal masking. “+” indicates that each component is incrementally added to the Vanilla baseline in the listed order. The best results are highlighted in bold.

Figure 4: Comparison of the proposed visual compressor equipped with QMSA against single-frame and average pooling baseline compressors. “Single-frame” denotes frame-by-frame compression without temporal integration. Each point shows the accuracy keep rate (in %) with the actual score in parentheses. The horizontal axis represents the compression rate, which is the ratio of visual tokens after compression to those before. To isolate the effect of the visual compressor, context memory and memory feedback are disabled, and all videos are uniformly sampled to 64 frames.

Figure 5: Case study example from MLVU, illustrating how QViC-MF utilizes context memory to answer long-term reasoning questions. The example corresponds to an Action Order (AO) task question, where the model predicts the correct sequence of events over a long-term video. Input frames corresponding to stored context memory entries are shown, and answer-critical frames are highlighted in red. The relevance score plot indicates the relevance score of each context memory entry over time.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Setting
Batch size	1
Gradient accumulation steps	4
Learning rate	1e-4
Learning scheduler	Cosine decay
Warm-up ratio	0.03
Weight decay	0

Appendix

Table 4: Training settings of our QViC-MF framework. During training, the model operates in a one-way compression setting without applying feedback from the context memory, and thus the number of recalled frames is fixed to

K_{\mathrm{r}}=0

Setting
Context tokens $C$	16
Context memory capacity $L$	256
Clip frames $K$	32
Recalled frames $K_{\mathrm{r}}$	32
Top heads $K_{\mathrm{h}}$ for computing $r_{n,i}$	5
Layer range $[L_{1},L_{2}]$ for computing $r_{n,i}$	[17, 20]

Appendix

Table 5: Inference settings of QViC-MF, including the hyper-parameters for relevance score computation. Parameters shared with training settings are repeated for clarity.

Figure 6: Variations of masking strategies used in the visual compressor. (a) A simple baseline that performs conventional frame-to-frame compression using single-frame causal masking. This corresponds to the “Single-frame” setting in Figure 4 of the main paper. (b) A naive extension of (a) to multi-frame compression, corresponding to the “Vanilla” setting in Table 3 of the main paper. (c) A variant of (b) that applies framewise context-to-visual masking, matching the masking matrix $\mathbf{M}$ in QMSA. (d) A variant of (c) that additionally blocks context-to-text attention, corresponding to applying both $\mathbf{M}$ and the blocking matrix $\mathbf{B}$ in QMSA. (e) The complete QMSA configuration, which further introduces guiding for context-to-visual attention. This corresponds to applying all three matrices: $\mathbf{M}$ , $\mathbf{B}$ , and $\mathbf{G}$ . As in standard attention masking, $-\infty$ blocks token-pair attention (ignored after softmax), $0$ passes the logits unchanged, and the guiding values $g_{i}$ act as bias terms added to the attention logits.

Figure 7: Visualization of attention patterns and relevance scores for a sample from the Needle QA (NQA) task in MLVU. The frames relevant to the question are highlighted with red boxes (around 140 seconds). The middle row shows the mean text-to-visual attention weights: for each condition (all heads vs. top- $K_{\mathrm{h}}\!=\!5$ heads, and all layers vs. layers $L_{1}\!=\!17$ to $L_{2}\!=\!20$ ), we plot the head-averaged attention weights for each layer, with different colors indicating different layers. The bottom row shows the corresponding relevance scores obtained by averaging these layer-wise mean attention weights across layers. These results show that appropriately selecting heads and layers leads to more discriminative relevance scores aligned with the question. For visualization, we assign each visual token to the frame it originates from based on its fixed sequence position and aggregate the text-to-visual attention over the tokens of each frame. This provides an approximate yet practical frame-level attribution even in deep layers.

Model	Size (LLM)	Input	M-Avg	TR	AR	NQA	ER	PQA	AO	AC
GPT-4o [ 1 ]	-	0.5 fps	64.6	-	-	-	-	-	-	-
MovieChat [ 16 ]	7B (Vicuna-v0)	2048 frm	25.8	29.5	25.0	24.2	24.7	25.8	28.6	22.8
LLaMA-VID [ 11 ]	7B (Vicuna-v1.5)	1 fps	33.2	50.8	34.5	30.1	32.7	32.5	23.9	27.8
MA-LMM [ 7 ]	7B (Vicuna-v1.1)	1000 frm	36.4	51.9	35.5	43.1	38.9	35.8	25.1	24.3
LLaVA-Mini [ 29 ]	7B (Vicuna-v1.5)	1 fps	42.8	76.0	50.0	44.5	37.5	49.0	24.3	18.4
DynFocus [ 6 ]	7B (Vicuna-v1.5)	32 frm	49.6	76.2	60.9	55.5	41.5	54.0	26.8	32.8

Appendix

Table 6: Detailed comparison of the proposed QViC-MF framework with SoTA methods on the MLVU-dev. The best and second best results are highlighted in bold and underlined, respectively.

Model	Size (LLM)	Input	Overall	KIR	EU	ER	TG	Rea	Sum
GPT-4o [ 1 ]	-	348 frm	30.8	34.5	27.4	33.0	25.0	27.5	24.1
Gemini 1.5 Pro [ 13 ]	-	3600 frm	33.1	39.3	30.9	32.1	31.8	27.0	32.8
MovieChat [ 16 ]	7B (Vicuna-v0)	2048 frm	22.5	25.9	23.1	21.3	22.3	24.0	17.2
LLaMA-VID [ 11 ]	7B (Vicuna-v1.5)	1 fps	23.9	23.4	21.7	25.4	26.4	26.5	17.2
DynFocus [ 6 ]	7B (Vicuna-v1.5)	200 frm	31.5	31.3	32.6	30.1	25.5	33.3	30.5
LLaVA-Video [ 30 ]	7B (Qwen2)	64 frm	41.8	41.6	39.3	42.3	33.2	47.8	32.8

Appendix

Table 7: Detailed comparison of the proposed QViC-MF framework with SoTA methods on the LVBench. The best and second best results are highlighted in bold and underlined, respectively.

Model	Size (LLM)	Input	Overall	Short	Medium	Long
GPT-4o [ 1 ]	-	384 frm	71.9	80.0	70.3	65.3
Gemini 1.5 Pro [ 13 ]	-	1 fps	70.3	78.8	68.8	61.1
DynFocus [ 6 ]	7B (Vicuna-v1.5)	16 frm	44.1	50.9	43.7	37.3
LongVA [ 28 ]	7B (Qwen2)	128 frm	52.6	61.1	50.4	46.2
Video-XL [ 15 ]	7B (Qwen2)	128 frm	55.5	64.0	53.2	49.2
Frame-Voyager [ 22 ]	7B (Qwen2)	8 frm	57.5	67.3	56.3	48.9

Appendix

Table 8: Detailed comparison of the proposed QViC-MF framework with SoTA methods on the VideoMME without subtitle setting. The best and second best results are highlighted in bold and underlined, respectively.

			Overall	Retrieval				Ordering				Counting
Method	Size (LLM)	Input		Edit	Insert-1	Insert-2	Avg	Edit	Insert-1	Insert-2	Avg	Edit-1	Edit-2	Insert	Avg
ALL (10-180 sec.)
GPT-4o [ 1 ]	-	1 fps	64.4	100.0	98.0	87.3	95.3	88.4	86.6	45.2	73.4	36.8	0.0	36.1	24.5
Gemini 1.5 Pro [ 13 ]	-	1 fps	66.7	100.0	96.0	76.0	90.7	90.7	95.3	32.7	72.9	60.7	7.3	42.0	36.7
LLaMA-VID [ 11 ]	7B (Vicuna-v1.5)	1 fps	10.8	28.0	28.0	19.3	25.1	0.7	0.0	0.0	0.2	4.0	2.7	14.7	7.1
Qwen2-VL [ 18 ]	7B (Qwen2)	1 fps	33.9	98.0	76.0	33.3	69.1	16.0	12.7	8.7	12.4	26.0	9.3	24.7	20.0

Appendix

Table 9: Detailed comparison of the proposed QViC-MF framework with SoTA methods on the VNBench. The best and second best results are highlighted in bold and underlined, respectively.

	MLVU	VNBench
$K,K_{\mathrm{r}}$	test	ALL	Long
64, 0	48.7	62.4	54.7
48, 16	58.3	63.8	58.9
32, 32	59.4	64.7	58.7
16, 48	58.0	63.9	57.1

Appendix

Table 10: Ablation study on the balance between the number of current clip frames

K

and recall frames

K_{\mathrm{r}}

. All other inference hyper-parameters follow the settings in Table 5 . The magenta-highlighted configuration corresponds to the default setting used in Table 5 . The best and second-best results are highlighted in bold and underlined, respectively.

	MLVU	VNBench
$K_{\mathrm{h}}$	test	ALL	Long
1	57.1	64.7	57.8
5	59.4	64.7	58.7
14	56.9	64.7	58.2
28 (all heads)	57.6	64.6	58.0

Appendix

Table 11: Ablation study on the number of heads

K_{\mathrm{h}}

used in relevance score computation. Setting

K_{\mathrm{h}}=28

corresponds to using all attention heads. The magenta-highlighted configuration denotes the default setting used in Table 5 . The best and second-best results are highlighted in bold and underlined, respectively.

	MLVU	VNBench
$[L_{1},L_{2}]$	test	ALL	Long
$[1,28]$ (all layers)	55.8	64.9	58.9
$[17,20]$	59.4	64.7	58.7

Appendix

Table 12: Ablation study on the layer range

[L_{1},L_{2}]

used in relevance score computation. Setting

[L_{1},L_{2}]=[1,28]

corresponds to using all layers. The magenta-highlighted configuration denotes the default setting used in Table 5 . The best results are highlighted in bold.

	MLVU	VNBench
$C$	test	ALL	Long
1	54.5	56.7	51.6
4	54.3	65.3	60.2
16	59.4	64.7	58.7

Appendix

Table 13: Ablation study on the number of context tokens per frame

C

. The magenta-highlighted configuration denotes the default setting used in Table 5 . The best results are highlighted in bold.

$L$	M-Avg	NQA	ER	AO	TQA
32	56.1	68.3	73.6	48.6	48.8
64	58.1	76.7	79.3	54.3	44.2
128	57.7	73.3	79.3	54.3	46.5
256	59.4	71.7	71.7	61.4	55.8

Appendix

Table 14: Ablation study on the context memory capacity

L

. We report results on the Needle QA (NQA), Ego Reasoning (ER), Action Order (AO), and Tutorial QA (TQA) tasks of MLVU-test, together with their mean average (M-Avg). The magenta-highlighted configuration denotes the default setting used in Table 5 . The best and second-best results are highlighted in bold and underlined, respectively.

Figure 8: QViC-MF failure cases on counting tasks from VNBench and MLVU. For each example, the input question, relevance score plot, and the frames corresponding to context memory entries are shown. (a) VNBench example: the model correctly identifies and stores all four bicycle-containing segments in the context memory, yet it fails to produce the correct count in its final answer. (b) MLVU example: the model accurately detects and stores the single occurrence of the “playing harp” action, but it likewise fails to output the correct count.

Explore similar work

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
Apr 15, 2026Zheyu Zhang, Ziqi Pang, Shixing Chen +3Long-Video Understanding Video Token Compression
University of Illinois Urbana-Champaign · Amazon Prime Video
Small Vision-Language Models are Smart Compressors for Long Video Understanding
Apr 9, 2026Junjie Fei, Jun Chen, Zechun Liu +13Long-Video Understanding Video Token Compression
Meta AI · King Abdullah University of Science and Technology (KAUST)
MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering
Jun 4, 2026Qing Yang, Pengcheng Huang, Xinze Li +6Video Summarization Video QA
School of Computer Science and Engineering, Northeastern University, Shenyang, China · Department of Computer Science and Technology, Tsinghua University, Beijing, China · Digital China Group, Beijing, China

Apr 15, 2026J/K move · Enter open · S save

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

Zheyu Zhang, Ziqi Pang, Shixing Chen, Xiang Hao+2

University of Illinois Urbana-Champaign · Amazon Prime Video

Long-Video Understanding Video Token Compression Long-Context VLMs Query-Aware Video Frame Selection

Open paper AskPDF arXiv

Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames. With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information. To address this, we explore extreme video token compression towards one token per frame at the final LLM layer. Our key insight is that heuristic-based compression, widely adopted by previous methods, is prone to information loss, and this necessitates supervising LLM layers into learnable and progressive modules for token-level compression (LP-Comp). Such compression enables our VLM to digest 2x-4x more frames with improved performance. To further increase the token efficiency, we investigate frame-level compression, which selects the frames most relevant to the queries via the internal attention scores of the LLM layers, named question-conditioned compression (QC-Comp). As a notable distinction from previous studies, we mitigate the position bias of LLM attention in long contexts, i.e., the over-concentration on the beginning and end of a sequence, by splitting long videos into short segments and employing local attention. Collectively, our combined token-level and frame-level leads to an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling. Our XComp is finetuned from VideoChat-Flash with a data-efficient supervised compression tuning stage that only requires 2.5% of the supervised fine-tuning data, yet boosts the accuracy from 42.9% to 46.2% on LVBench and enhances multiple other long video benchmarks.

Abstract

Explore similar work

One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

Small Vision-Language Models are Smart Compressors for Long Video Understanding

MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering