REEL: Same Frames, Different Answers and Different Costs

Xin Liang; Jiangpeng He

cs.CVOct 28, 2026

REEL: Same Frames, Different Answers and Different Costs

Authors: Xin Liang, Jiangpeng He

Organizations: Indiana University Bloomington

Abstract

Video understanding keeps moving toward longer recordings, and the longer a recording is, the more questions can be asked about it. But processing it again for every question does not scale, so many systems watch the video once, build a memory, and answer later questions from it. Yet what to remember is decided before anyone knows what will be asked, and whatever is left out cannot be recovered later. Moreover, whether the savings pay off is judged by token counts, which fall when the input gets shorter but not when the model reuses what it has already computed. We present REEL, an evaluation framework that compares three saved representations built from exactly the same observed frames, so that a change in answers can come only from what the system remembered. Text writes a short record of what happened, Visual keeps the encoder outputs, and Prefix State also keeps the language-model prefix. Each is compared with full reprocessing (Frames) by paired correctness changes, cumulative GPU time, and storage size. Across six vision-language models and two benchmarks, Text and Frames differ in correctness on 20% to 29% of questions, even when their mean accuracy is almost the same. By contrast, Visual and Prefix State stay much closer to Frames. With unlimited storage, Prefix State is the cheapest in total GPU time among memories that keep almost all of the answers Frames gets right, for every model on both benchmarks. The trade-off is storage: Prefix State keeps up to about 21 GiB per video, whereas Text needs only kilobytes. Reusing computation and discarding information both look like efficiency, but only the second removes evidence that later questions may need, and neither mean accuracy nor token count tells them apart. REEL makes that difference measurable for any system that answers many questions from one saved memory.

Figures & tables

Panel of Figure 1 — Figure 1: Representation changes answers, cost, and storage. (a) Correct/wrong changes across 12 model-dataset pairs, not mean accuracy loss. (b) Lowest-GPU choice before and after answer-retention checks. (c) Retained bytes versus GPU saving across three model families. (d) Cumulative GPU cost on HD-EPIC.

Figure 2: The missing video file. All four paths see the same timestamps. Frames repeats the full model; Visual and Prefix State save intermediate model state; Text writes a question-blind description. Here, the frame reveals a video file, but Text records only “user opens file,” so its later answer differs.

Panel of Figure 3 — Figure 3: Many answers change even when mean accuracy changes little. Each model contributes one red circle (Loss) and one blue square (Gain) from Eq. 1 ; their sum is Flip. Yellow diamonds show $\Delta\mathrm{Acc}$ (Text minus Frames) with 95% video-bootstrap intervals. Points are model-level summaries, not individual questions.

Figure 4: Text-minus-Frames accuracy by question type. Left: CG-Bench; right: HD-EPIC. Each shows the four lowest and four highest model-averaged types; positive values favor Text, and † marks types with fewer than 30 questions.

	CG, $H=6$		HD, $H=20$
Model	GPU only	+ Quality	GPU only	+ Quality
Qwen3-VL-8B	F $\to$ P 1	F $\to$ P 1	F $\to$ P 2 $\to$ T 7	F $\to$ P 2
InternVL3-8B	F $\to$ T 3	F $\to$ P 5	F $\to$ T 3	F $\to$ P 5
Gemma-3-12B-IT	F $\to$ P 1	F $\to$ P 2	F $\to$ P 2	F $\to$ P 2
Qwen3-VL-4B	F $\to$ P 2	F $\to$ V 4 $\to$ P 6	F $\to$ P 2 $\to$ T 4	F $\to$ P 2
InternVL3.5-8B	F $\to$ T 4	F $\to$ P 6	F $\to$ T 3	F $\to$ P 5

Table 1: Lowest-cost eligible representation as questions arrive. “+ Quality” applies both quality checks. F/T/V/P denote Frames, Text, Visual, and Prefix State; subscripts mark the first question after a change.

	CG			HD
Rep.	Build	Query	Total 6	Build	Query 1:20	Total 20	Total All	Saving All
Qwen3-VL-8B
Frames	0.0	13.2	79.5	0.0	57.2	1,143.1	2,084.5	0.0%
Text	60.4	3.1	79.3	74.0	3.6	146.5	193.4	90.7%
Visual	9.5	10.0	69.5	47.9	46.4	976.3	1,737.6	16.6%
Prefix	9.5	3.9	32.8	46.9	7.3	192.5	298.5	85.7%

Table 2: GPU time by model and question horizon. Build is one-time construction cost, Query is mean per-question answer cost, and Total is cumulative GPU-seconds per video. The HD columns additionally report the all-question total and its saving relative to Frames. Within each model–dataset block, green shading indicates the representation with the lowest cumulative GPU cost.

Panel of Figure 5 — Figure 5: Storage, GPU saving, and accuracy. Position gives bytes and GPU saving; color gives $\Delta\mathrm{Acc}$ against Frames. Prefix State includes Visual storage and KV pages. Qwen, InternVL, and Gemma denote Qwen3-VL-8B, InternVL3-8B, and Gemma-3-12B-IT, respectively.

		Acc. (%)			Select (s)		Total GPU-s
Data	$K$	MaxVol	AKS	$\Delta$	MaxVol/video	AKS/question	MaxVol	AKS
CG ( $H=6$ )	32	38.5	46.0	+7.6	63.7	65.4	65.1	199.7
CG ( $H=6$ )	64	41.9	48.1	+6.2	62.6	65.4	101.5	236.2
CG ( $H=6$ )	128	48.1	54.7	+6.7	61.9	65.4	214.7	349.8
HD ( $H=20$ )	32	32.4	30.9	-1.5	196.8	222.9	168.0	938.5
HD ( $H=20$ )	64	31.3	30.2	-1.1	196.2	222.9	281.7	1,049.3

Table 3: MaxVol and AKS at the same frame budgets with Qwen3-VL-8B. MaxVol is question-blind and AKS is question-aware; costs include selection and answering.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Data	Model	Balanced24	Added	Fixed
CG	Qwen3-VL-8B	19.7/12.5	18.6/12.0	20.9/12.7
	InternVL3-8B	23.5/14.0	22.6/14.0	24.1/13.8
	Gemma-3-12B-IT	20.4/10.4	21.0/9.7	20.4/11.0
	Qwen3-VL-4B	22.4/16.4	23.0/17.0	20.7/16.0
	InternVL3.5-8B	24.2/15.7	23.0/14.7	26.7/17.8
	MiniCPM-V-4.5-8B	22.7/11.7	20.8/11.1	21.7/12.3

Appendix

Table 4: Text designs versus Frames in the summary ablation. Each cell reports Flip / Loss (%) over valid pairs, computed relative to the corresponding Frames run in the same ablation setting. Fixed is evaluated only on CG MQ-50.

Model	Hugging Face repository
Qwen3-VL-8B	Qwen/Qwen3-VL-8B-Instruct
InternVL3-8B	OpenGVLab/InternVL3-8B
Gemma-3-12B-IT	google/gemma-3-12b-it
Qwen3-VL-4B	Qwen/Qwen3-VL-4B-Instruct
InternVL3.5-8B	OpenGVLab/InternVL3_5-8B
MiniCPM-V-4.5-8B	openbmb/MiniCPM-V-4_5

Appendix

Table 5: Hugging Face repositories for the evaluated models.

		32 frames						64 frames
Model	Path	$G_{6}$	Save	Acc.	Flip	Loss	Cross	$G_{6}$	Save	Acc.	Flip	Loss	Cross
Qwen3-VL-8B	Frames	40.7	+0.0	36.3	0.0	0.0	–	79.5	+0.0	39.0	0.0	0.0	–
	Text	46.8	-6.1	29.7	20.2	13.5	–	79.3	+0.2	34.0	19.7	12.5	6
	Visual	35.8	+4.9	36.3	0.0	0.0	3	69.5	+10.0	39.0	0.0	0.0	3
	Prefix State	22.8	+17.9	35.3	1.0	1.0	2	32.8	+46.7	40.0	1.4	0.3	1
InternVL3-8B	Frames	35.4	+0.0	36.3	0.0	0.0	–	110.1	+0.0	36.7	0.0	0.0	–

Appendix

Table 6: All four paths at 32 and 64 frames on CG MQ-50.

G_{6}

and Save are GPU-seconds per video through six questions, with Save relative to Frames; Cross is the sustained compute crossing, and – means no crossing by

m=6

. Accuracy, Flip, and Loss are percentages.

Model	Uniform64	Clue window64	Interaction
Qwen3-VL-8B	$-5.1\,[-9.8,-0.7]$	$-11.6\,[-16.5,-6.8]$	$-6.5\,[-13.2,+0.3]$
InternVL3-8B	$-4.5\,[-10.1,+1.1]$	$-2.8\,[-9.2,+3.5]$	$+1.7\,[-5.8,+9.1]$
Gemma-3-12B-IT	$-0.3\,[-6.4,+5.7]$	$-3.0\,[-9.7,+3.7]$	$-2.7\,[-10.1,+5.1]$

Appendix

Table 7: Text-minus-Frames gaps under uniform and clue-window sampling. Values are percentage points with 95% video-bootstrap intervals. A positive interaction means clue-window sampling narrows the gap.

			Accuracy (%)
Data	Model	Valid/All	MaxVol	AKS	$\Delta$	95% CI	All $\Delta$	GPU ratio AKS/MaxVol
CG	Qwen3-VL-8B	289/300	41.9	48.1	$+6.2$	$[+0.0,\,+12.5]$	$+6.3$	$2.33\times$
CG	InternVL3-8B	294/300	42.2	44.9	$+2.7$	$[-2.0,\,+7.1]$	$+2.7$	$2.09\times$
CG	Gemma-3-12B-IT	298/300	27.2	28.9	$+1.7$	$[-2.7,\,+6.0]$	$+1.7$	$2.72\times$
CG	Qwen3-VL-4B	299/300	42.5	46.8	$+4.3$	$[-1.3,\,+10.0]$	$+4.3$	$2.74\times$
CG	InternVL3.5-8B	300/300	38.7	45.7	$+7.0$	$[+1.3,\,+12.7]$	$+7.0$	$2.00\times$

Appendix

Table 8: MaxVol and AKS across backbones at 64 frames. Accuracy uses questions with valid answers from both selectors. The interval is a 95% video-bootstrap interval for AKS minus MaxVol. All

\Delta

instead counts invalid answers as wrong. The GPU-time ratios include selection and answering over all evaluated questions: six per CG-Bench video and all 1,871 questions across the 51-video HD-EPIC-51 cohort. For its GPU-time totals, Table 3 instead uses the same six-question CG-Bench horizon but only the first 20 HD-EPIC questions per video.

Data	Model	Words/frame	Pairs	$\Delta$ Acc	Flip (%)
CG	Qwen3-VL-8B	$19.28{\to}19.49$	296/300	$+1.4$ [ $-1.0$ , $+3.7$ ]	6.8 [4.0, 9.8]
	InternVL3-8B	$17.49{\to}17.66$	299/300	$+1.3$ [ $-1.7$ , $+4.3$ ]	8.7 [5.7, 12.0]
	Gemma-3-12B-IT	$16.34{\to}16.79$	297/300	$-1.7$ [ $-5.7$ , $+2.7$ ]	9.8 [6.4, 13.4]
HD	Qwen3-VL-8B	$19.32{\to}19.40$	1,853/1,871	$+0.0$ [ $-1.2$ , $+1.2$ ]	8.6 [7.4, 9.9]
	InternVL3-8B	$17.41{\to}17.74$	1,869/1,871	$-0.5$ [ $-2.0$ , $+1.0$ ]	9.5 [8.0, 11.1]
	Gemma-3-12B-IT	$17.09{\to}17.23$	1,854/1,871	$-1.5$ [ $-3.0$ , $+0.1$ ]	8.9 [7.5, 10.4]

Appendix

Table 9: Balanced24 versus Detail24 at the same requested length. Pairs are valid under both writers,

\Delta

Acc is Detail24 minus Balanced24, and Flip is the fraction with different correctness, each with a 95% video-bootstrap interval. Valid-output rates are 98.7–100% for both writers.

Figure 6: Cumulative GPU time for Qwen3-VL-8B, InternVL3-8B, and Gemma-3-12B-IT. Rows are datasets; bands are 95% video-bootstrap intervals.

Figure 7: Cumulative GPU time for Qwen3-VL-4B, InternVL3.5-8B, and MiniCPM-V-4.5-8B. Rows are datasets; bands are 95% video-bootstrap intervals.

Model	Rep.	Valid	Pair	$\Delta$ [95% CI]	Loss	Flip	$B$	$G_{10}$	Save	GPU from	All from
Qwen3-VL-8B	Frames	493/500	493/500	$+0.00$ [ $+0.00$ , $+0.00$ ]	0.0%	0.0%	0.0	479.4	0.0%	–	–
Qwen3-VL-8B	Text	494/500	487/500	$-9.24$ [ $-14.49$ , $-4.29$ ]	16.2%	23.2%	70.0	93.6	80.5%	2	–
Qwen3-VL-8B	Visual	492/500	492/500	$+0.00$ [ $+0.00$ , $+0.00$ ]	0.0%	0.0%	42.4	421.2	12.1%	5	5
Qwen3-VL-8B	Prefix State	492/500	492/500	$-0.20$ [ $-0.61$ , $+0.00$ ]	0.2%	0.2%	42.2	104.1	78.3%	2	3
InternVL3-8B	Frames	499/500	499/500	$+0.00$ [ $+0.00$ , $+0.00$ ]	0.0%	0.0%	0.0	173.3	0.0%	–	–
InternVL3-8B	Text	493/500	492/500	$-3.46$ [ $-7.54$ , $+0.61$ ]	13.8%	24.2%	40.9	59.7	65.5%	3	–

Appendix

Table 10: HER-MQ10 results for two backbones. Valid counts each path’s outputs; Pair counts questions valid for Frames and the listed path.

\Delta

Acc, Loss, and Flip use those pairs.

B

and

G_{10}

are GPU-seconds per video, Save is relative to Frames, and the final columns give the sustained crossings.

Dataset	Model	Text	Visual	KV only	Prefix State
CG	Qwen	9.7 KiB	810.0 MiB	3.74 GiB	4.53 GiB
CG	InternVL	8.7 KiB	112.9 MiB	973.0 MiB	1.06 GiB
CG	Gemma	7.7 KiB	121.9 MiB	1.40 GiB	1.52 GiB
HD	Qwen	8.5 KiB	3.78 GiB	17.19 GiB	20.97 GiB
HD	InternVL	7.8 KiB	112.9 MiB	972.1 MiB	1.06 GiB
HD	Gemma	7.2 KiB	121.9 MiB	1.40 GiB	1.52 GiB

Appendix

Table 11: Median retained payload per video for the same 64 frames. Text counts serialized bytes including each timestamp once, Visual counts unique encoder-output storage, KV only counts reached pages, and Prefix State adds Visual and KV storage.

Figure 8: The best answer-preserving path depends on the storage budget. At each cap, color gives the lowest-GPU path among Frames and the reusable paths that fit the per-video cap and pass both quality checks. Rows are the three profiled models on CG-Bench (CG) and HD-EPIC (HD). Figure labels Qwen, InternVL, and Gemma denote Qwen3-VL-8B, InternVL3-8B, and Gemma-3-12B-IT.

	No cache			Observed hits
Model	Text	Visual	Prefix State	Text	Visual	Prefix State
CG, endpoint $H=6$
Qwen3-VL-8B	2	$>6$	$>6$	2	$>6$	2
InternVL3-8B	2	$>6$	$>6$	2	$>6$	2
Gemma-3-12B-IT	2	$>6$	$>6$	2	$>6$	2
HD-EPIC, endpoint $H=20$

Appendix

Table 12: Sustained monetary crossing under the GLM tariff. Costs include construction and answering;

>H

means no crossing by the endpoint.

Frames	Uniform	AKS	$\Delta$
32	36.0	45.7	$+9.7$
64	41.0	48.3	$+7.3$
128	47.0	54.7	$+7.7$

Appendix

Table 13: Uniform reference for AKS on CG-Bench. All questions are counted, with invalid answers treated as wrong. The Uniform values come from the dedicated selector-sweep runs and are therefore separate from the four-path representation runs in Table 6 .

Video-Language Model Evaluation

Video-language model evaluation measures whether multimodal systems understand, ground, and reliably use visual and audio evidence over time.

See all 104 papers

Sep 20, 2026

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding

Long-video understanding must capture transient visual evidence under strict token budgets, yet existing methods compress frames, append memory tokens, or alter internal key-value (KV) caches. We introduce Prefix-Steered Recurrent Memory (PREM), a memory-token-free framework for frozen vision-language models (VLMs). PREM separates video ingestion from query answering: a recurrent writer distills visual streams into a compact 256 KiB multi-slot associative state, while a question-conditioned readout adds memory-derived key/value (K/V) steering modulations to existing non-visual prompt prefixes during prefill. This enables write-once, query-many inference without extra prompt tokens or decoding recurrence. Across six long-video benchmarks in offline and streaming end-of-stream settings, PREM consistently outperforms frozen baselines at every evaluated visual budget. Under a constrained budget of 16 frames, PREM improves macro-average accuracy by 3.06% on Qwen2.5-VL-3B, with gains of 11.0% on action antonym identification and 9.9% on localized needle retrieval. These gains require tuning 0.24% of backbone parameters at 0.03 GiB of peak GPU memory overhead.

Siru Zhong, Qiongyan Wang, Xiaohui Lv +5

The Hong Kong University of Science and Technology (Guangzhou) · Beta Infinity · Tsinghua University

Memory-Augmented VLMs Recurrent Neural Networks Long-Video Understanding Memory-Augmented Video Understanding Long-Context VLMs

Ask PDF

Oct 17, 2026

Write Once, Query Many: Evolving Compact Reusable Memory for Long Videos

Long videos often exceed the context capacity of multimodal large language models (MLLMs), while agentic search repeats costly video processing and grounding for each query. Reusable memory can amortize this cost, but must preserve information for questions unknown during construction. We propose Video-EMem, an evolvable video memory harness built on time-anchored perceptual observations. This representation combines compact free-form text with sparse images to preserve semantic content and complementary visual details across temporal scales, supporting temporal navigation without a fixed semantic schema. A Writer encodes each video once by consolidating overlapping multiscale observations, and a Reader retrieves and organizes evidence solely from the resulting memory. With the underlying models frozen, we evolve their workflows and operators using distinct feedback: query-independent source fidelity and coverage for the Writer, and QA failures in evidence localization and utilization for the Reader. Experiments support this separation for generalization beyond development queries, with our evolution strategy improving average accuracy by 7.2% over the unevolved harness. While reducing storage by about 99.86% on average relative to raw videos, Video-EMem improves Qwen3-VL-30B by an average of 2.6% across three long-video QA benchmarks over direct input of 2,048 frames. Using the same o3 answer model as the state-of-the-art agentic method, it improves average accuracy by 2.5% with lower per-query latency.

Kerui Chen, Jinglu Wang, Yun Wang +2

College of Artificial Intelligence, Zhejiang University. · Microsoft Research Asia

Memory-Augmented Video Understanding Memory-Augmented VLMs Long-Video Understanding Long-Video QA

Ask PDF

Sep 26, 2026

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

Vision-language models are increasingly used to understand long videos and continuous streams. However, dense visual tokens accumulate with video duration, making long-context inference prohibitively expensive. Existing training-free visual-token selection methods reduce this cost by retaining informative tokens, but may lose coherent event evidence and fail to distinguish detailed recent observations from long-range history. We propose KeyRec, a training-free framework for constructing bounded visual memory. During query-agnostic writing, KeyRec preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank. Candidate events are proposed according to their novelty relative to previously stored events and maintained through an online add--merge--evict update. When a question arrives, a text-only router adaptively allocates a fixed readout budget between recent and event memory, without reprocessing historical frames. KeyRec operates on model-facing visual embeddings and supports both modular encoder--projector VLMs and the encoder- and projector-free NEO-ov architecture. Across four streaming and long-video benchmarks and three VLM backbones, KeyRec achieves the best compressed performance in 13 of 15 settings using only 10% of the dense decoder-facing visual-token budget. It outperforms the strongest compressed baseline by 2.21--18.37 points on real-time questions, achieves the best compressed result in five of six long-video settings, and performs best in every NEO-ov 2B setting.

Zihan Chen, Xuejian Rong, Xiaojuan Wang +4

University of Virginia · Google

Long-Video Understanding Memory-Augmented Video Understanding Long-Context VLMs Streaming Video Understanding Visual Token Compression

Ask PDF

Abstract

Video-Language Model Evaluation

Explore similar work

PREM: Prefix-Steered Recurrent Memory for Long-Video Understanding

Write Once, Query Many: Evolving Compact Reusable Memory for Long Videos

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding