RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling

Can Zhang; Xiaotian Han; Junyuan Shang; Yuchen Ding; Zhenyu Zhang; Shuohuan Wang; Dianhai Yu; Ruirui Li

cs.CVSep 30, 2026

RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling

Authors: Can Zhang, Xiaotian Han, Junyuan Shang, Yuchen Ding, Zhenyu Zhang, Shuohuan Wang, Dianhai Yu, Ruirui Li

Organizations: Beijing University of Chemical Technology · Baidu, Inc.

Abstract

Existing video language models encode sampled RGB frames independently, so a long video must either exhaust the token budget or drop the changes between sampled frames. Codec-aware front-ends read the motion vectors and residuals that encoding produced, but in their deployed form each predictive frame is still tokenized on its own: the tokens are a function of the current primitives, not of a carried reference. We argue that a more natural function is of both---the current primitives and a carried reference. A clip and its time reversal share the same frames and differ only in the order of changes---an axis that symmetric pooling discards by construction, and that is non-empty in the frozen vision features VideoLMs use---and the codec recurrence already composes those changes in order against a reference state. We introduce RESUME, a stateful codec representation: an anchor I-frame initializes a compact latent state, each subsequent predictive frame is consumed as an update to that state, and a shared readout exposes VideoLM-compatible tokens from the accumulated state. Codec prediction is thereby kept at the representation level and handed to the language model as a trajectory, not as a set of independent token groups. At the same per-predictive-frame token budget as prior codec-aware methods, a predictive frame enters the language model as a readout of what the front-end already knows, not as an encoding of the current primitives alone. Across ten benchmarks, the gains concentrate on temporal reasoning: on all three temporal benchmarks RESUME improves over both the RGB-frame baseline LLaVA-Video-7B (by 2.8, 5.1, and 3.9 points on TempCompass, TOMATO, and MVBench) and the codec-based baseline CoPE-7B, while staying competitive on general and long-form QA. Frozen-transition tests further show anchor dependence, order sensitivity, and useful rollout behavior beyond the training horizon.

Figures & tables

Figure 1: Three ways to turn a group of pictures into visual tokens. (a) Dense RGB encoding treats every frame as a full image. (b) Deployed codec-aware tokenization reads motion vectors and residuals, but emits an independent token group per predictive frame; temporal dependence is left to language-model attention. (c) RESUME initializes a compact latent state from the anchor I-frame and updates that same state with each codec observation; the tokens of a predictive frame are a readout of the accumulated state. The two codec columns tokenize each predictive frame; only (c) maintains a reference-dependent state in the visual front-end.

Figure 2: RESUME as a codec-driven transition system. An anchor I-frame initializes a compact latent state $z_{0}$ and, in parallel, keeps its own token path into the language model. Each predictive frame contributes one fused observation $o_{t}$ from motion vectors and residuals, written into the carried state. A shared head reads $Y_{t}$ from $z_{t}$ at the same per-predictive-frame token budget used by prior codec-aware front-ends. The state is reset at the next I-frame. A dedicated projector adapts state readouts to the language embedding space; the original multimodal projector is left untouched for I-frame tokens.

(a) General QA PT NQA AQA VMME Proprietary GPT-5 [-1pt] ( OpenAI, 2025 ) – 86.3 – 83.3 Gemini 3 Pro [-1pt] ( Google, 2025 ) – 84.3 – 88.6 Gemini 2.5 Pro [-1pt] ( Gemini Team, 2025 ) – 85.3 – 87.8 Claude 4.5 [-1pt] ( Anthropic, 2025 ) – 79.2 – 74.2 Open-source VILA-40B [-1pt] ( Lin et al., 2024 ) 54.0 67.9 58.0 60.1 IXC-2.5-7B [-1pt] ( Zhang et al., 2024a ) 34.4 71.0 52.8 55.8 LLaVA-OV-7B [-1pt] ( Li et al., 2024a ) 57.1 79.4 56.6 58.2 Oryx-7B [-1pt] ( Liu et al., 2024c ) 68.6 81.9 – 58.3 LLaVA-Video-7B [-1pt] ( Zhang et al., 2024c ) 67.9 83.2 56.5 63.3 CoPE-7B [-1pt] ( Sarkar et al., 2026 ) 70.3 82.1 60.3 61.9 RESUME 71.2 81.9 60.9 61.9

(b) Temporal TC TOM MVB Proprietary GPT-5 80.4 53.0 74.1 Gemini 3 Pro 82.8 48.3 70.4 Gemini 2.5 Pro 81.9 48.6 70.6 Claude 4.5 72.8 39.6 62.1 Open-source IXC-2.5-7B 67.1 – 69.1 LLaVA-OV-7B 64.8 25.5 56.7 VideoLLaMA2 [-1pt] ( Cheng et al., 2024 ) – 18.5 54.6 InternVL2-8B [-1pt] ( Chen et al., 2024b ) 65.3 21.7 65.8 VideoChat2-7B [-1pt] ( Li et al., 2023b ) 45.5 – 51.1 LLaVA-Video-7B 66.6 24.9 58.6 CoPE-7B 68.9 28.3 61.9 RESUME 69.4 30.0 62.5

(c) Long-form VTT VMMU LVB Proprietary GPT-5 – – 68.8 Gemini 3 Pro – – 78.0 Gemini 2.5 Pro – – 78.4 Claude 4.5 – – 50.5 Open-source LongVA-7B [-1pt] ( Zhang et al., 2024b ) – 23.9 – LLaVA-OV-7B 44.0 33.9 38.1 InternVL2-8B – 37.4 – LLaVA-Video-7B 41.8 36.1 44.2 CoPE-7B 45.5 38.2 46.4 RESUME 45.7 38.2 43.2

Table 1: Question answering across general, temporal, and long-form benchmarks. RESUME uses at most

64

I-frames, and each of the four predictive updates per I-frame produces one

8

-token readout. Prior numbers are as reported by CoPE-VideoLM (cited in the table); a dash means that source does not report the benchmark. Video-MME is without subtitles. ActivityNet-QA is scored by Claude Opus 4.8 ( Anthropic, 2026 ) for RESUME and by a language model for the rows above.

Figure 3: The reversal axis separates static content, net change, and temporal order. (a) We compare frame-symmetric (P0), difference-symmetric (P1), and order-sensitive (P2) probes on a clip and its time reversal. (b) The probes are evaluated on order questions and on two frozen vision towers.

Figure 4: The frozen transition carries state across predictive updates. (a) Anchor sensitivity, (b) order sensitivity, and (c) rollout behavior are compared with state controls.

	Video-MME		LVBench
$N_{I}$	I-frames only	+ readouts	I-frames only	+ readouts
$8$	$53.3$	$\mathbf{54.1}$ +0.8	$36.7$	$\mathbf{37.3}$ +0.6
$16$	$57.3$	$\mathbf{58.6}$ +1.3	$39.1$	$\mathbf{40.0}$ +0.9

Table 2: Ablation of the codec-driven state readouts under a matched I-frame budget. Each pair uses exactly the same

N_{I}

I-frames; “+ readouts” augments them with the

256

state readouts. Accuracy (%) on Video-MME (without subtitles) and LVBench.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Codec primitives inside one group of pictures. The I-frame is an independently coded RGB image. Each predictive frame is stored as block-wise motion vectors $\tau$ (where existing content moves) and residuals $\delta$ (what motion compensation cannot explain). RESUME consumes $(\tau,\delta)$ as a single observation that updates a state initialized from the I-frame, rather than encoding each predictive frame as an independent token group. Motion vectors are the bitstream quiver overlaid on a faded reconstruction of the same frame.

Figure 6: Video length covered at one frame per second, against the visual-token budget. Markers distinguish dense frame encoding from three dense-readout settings (

4

8

, or

16

eight-token readouts per I-frame). The star is the published Gemini 2.5 Pro point.

Input	TTFT (s)	E2EL (s)
$32$ I-frames + $32$ readout groups	$0.461$	$1.795$
$16$ I-frames + $48$ readout groups	$0.570$	$1.895$
$8$ I-frames + $56$ readout groups	$0.631$	$1.954$
$64$ I-frames (same weights)	$0.619$	$1.938$
$64$ frames, LLaVA-Video-7B	$0.686$	$2.094$

Appendix

Table 3: Inference latency for a

64

-second clip at one frame per second, generating

64

text tokens. Time to first token (TTFT) is prefill; E2EL is the time to emit

64

tokens.

Training data	Total	NextQA	PerceptionTest	Video-MME
LLaVA-Video ( Zhang et al., 2024c )
LLaVA-Hound	$0.25$ M	$64.4$	$51.4$	$54.1$
+ LLaVA-Video-178K	$1.58$ M	$80.1$	$57.1$	$63.2$
+ 3 QA datasets	$1.64$ M	$80.1$	$69.0$	$61.9$
+ LLaVA-OV (images)	$2.74$ M	$83.2$	$67.9$	$63.4$
LLaVA-Video-178K (sampled)	$1.08$ M	$73.2$	$55.9$	$59.6$

Appendix

Table 4: Training-data scale and distribution. LLaVA-Video rows are as reported in ( Zhang et al., 2024c ) across incremental training stages; RESUME fine-tunes Stage 2 on LLaVA-Video-178K only. The three QA datasets are the training splits of PerceptionTest, NextQA, and ActivityNet-QA.

Explore similar work

Jun 1, 2026cs.CV

AdaCodec: A Predictive Visual Code for Video MLLMs

Video is temporally redundant: adjacent frames usually share most objects, background, and layout. Yet existing video multimodal large language models (video MLLMs) usually encode each sampled frame as an independent RGB image, causing visual tokens to repeat content already present in earlier frames. This suggests a more direct video interface: send a full reference frame only when the scene cannot be predicted well from prior context, and otherwise transmit a compact description of inter-frame changes. We call this interface a \emph{predictive visual code}, and instantiate it for video MLLMs as \textbf{AdaCodec}. AdaCodec spends full visual tokens on a reference frame only when its conditional predictive cost is high; otherwise, it encodes inter-frame changes, including motion and prediction residuals, as compact P-tokens. Across all eleven benchmarks, AdaCodec improves over the Qwen3-VL-8B per-frame RGB baseline at a matched visual-token budget. Even at

1/7

the budget, AdaCodec with 32k tokens surpasses the 224k baseline on all long-video benchmarks; on five general-video benchmarks, it raises the average score while substantially cutting time-to-first-token from 9.26s to 1.62s.

Haowen Hou, Zhen Huang, Zheming Liang +8

Shanghai Jiao Tong University · Shanghai Innovation Institute

Video Multimodal Large Language Models Long-Video Benchmarks Multimodal Large Language Models

Ask PDF

May 8, 2026cs.CV

TTF: Temporal Token Fusion for Efficient Video-Language Model

Video-language models (VLMs) face rapid inference costs as visual token counts scale with video length. For example, 32 frames at

448{\times}448

resolution already yield >8,000 visual tokens in Qwen3-VL, making LLM prefill the dominant throughput bottleneck. Existing methods often rely on global similarity or attention-guided compression, incurring offsets to their gains. We propose \textbf{Temporal Token Fusion (TTF)}, a training-free, plug-and-play pre-LLM token compression framework that exploits structured temporal redundancy in video. TTF automatically selects an anchor frame, then for each subsequent frame, performs a local window similarity search (e.g.,

3\times 3

), fusing tokens that exceed a threshold. The compressed sequence maintains positional consistency across both prefill and decoding through coordinate realignment, enabling seamless integration with existing VLM pipelines. On Qwen3-VL-8B with threshold t=0.70, TTF removes about 67% of visual tokens while retaining 99.5% of the baseline accuracy and introducing only

{\approx}0.16

,GFLOPs of matching overhead. Overall, TTF offers a practical, efficient solution for video understanding. The code is available at https://github.com/Cominder/ttf

Simin Huo, Ning LI

Shanghai Jiao Tong University

Video-Language Models Qwen3

Ask PDF

May 11, 2026cs.CV

EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMs

Long-form video understanding remains challenging for Video Large Language Models (VideoLLMs), as the dense frame sampling introduces massive visual tokens while sparse sampling risks missing critical temporal evidence and leading to LLM hallucination. Existing training-free token reduction methods either treat videos equally as static images or rely on segment-level merging heuristics, which weaken fine-grained spatiotemporal modeling and introduce additional overhead. In this paper, we propose EchoPrune, a lightweight and training-free token pruning method that improves temporal resolution under a fixed LLM-side visual token budget. Our core idea is to interpret redundant video tokens as temporal echoes: if a token is well reconstructed from the previous frame, it is merely a temporally redundant echo; otherwise, it may capture new events, motion, or query-relevant visual evidence. Based on this insight, EchoPrune scores visual tokens by (i) query-guided crossmodal relevance and (ii) temporal reconstruction error, measured by correspondence matching and echo matching across consecutive frames. The selected tokens preserve task-relevant cues and temporal novelty while suppressing predictable redundancy, allowing VideoLLMs to observe more frames without increasing the decoding budget. Extensive experiments on LLaVA-OV, Qwen2.5VL, and Qwen3VL across six video understanding benchmarks show that EchoPrune enables VideoLLMs to process up to 20x frames under the same token budget, yielding improved performance (+8.6%) and inference speedup (5.6x for prefilling) on Qwen2.5VL-7B.

Jiameng Li, Minye Wu, Jiezhang Cao +2

KU Leuven · Shanghai Jiaotong University · Weill Cornell Medicine

Redundancy Large Language Model Hallucination

Ask PDF