Organizations: Peking University · Shanghai Jiao Tong University · Harbin Institute of Technology · Southeast University · Institute of Automation, Chinese Academy of Sciences · Xi’an Jiaotong University · Zhejiang University · Nanyang Technological University · Xiamen University
Video question answering under tight frame budgets requires preserving localized motion and event order, yet current video multimodal large language models (MLLMs) reduce comprehension to static multi-image reasoning by discarding unselected frames and encoding keyframes independently: (1) Spatially, whole-frame pooling dilutes small moving targets into static backgrounds; (2) Temporally, dropping intermediate frames or averaging tokens erases inter-frame motion, confusing direction, speed, and order. We propose VIVID (Visual Inter-frame Variations and Individual Dynamics), which moves beyond static snapshots like human vision: keyframes capture core states while inter-frame Patch Diffs preserve neighboring changes within Kf keyframes. Autonomous Delta-Driven Keyframe Planning (ADKP) selects anchors from a 3 fps pool and feeds surrounding patch differences into Dual-Road Change Memory (DRCM), forming Dynamic Tokens at zero extra anchor tokens alongside a camera-stabilized Active Motion Region of Interest (ROI) and chronological Entity State Chains. Adding 0.22% trainable parameters to a frozen 27B backbone, VIVID-Small (50 examples, ∼1.5 GPU hours) beats all eleven baselines in overall accuracy across 8,083 questions and trails only VIVID-Large (3,600 examples), which leads on MotionBench, Video-MME, and TempCompass.
Figures & tables
Figure 2: Overview of the VIVID architecture (ViT: Vision Transformer; MLP: multilayer perceptron; FFN: feedforward network; ROI: region of interest; LoRA: Low Rank Adaptation): (a) across 3fps candidates ( Sec. 3 ), the frozen ViT computes Patch Diffs ΔEt so ADKP ( Eqs. 6 , 7 and 8 ) selects Kf=8 transition anchors across intervals B1…B6 and crops a 320×180 Active Motion ROI ; (b) surrounding Patch Diffs enter Dual Road Change Memory (DRCM) , where the Visual Delta Module θΔ enriches static anchors Et^k into Dynamic Tokens E~t^k ( Eqs. 3 and 4 ) alongside precomputed Entity State Chains ( Eq. 5 ); (c) the Transition Conditioned Input feeds the shared reader, with the first 48 LLM layers frozen and the final 16 layers adapted via LoRA.
Method
Venue
Motion Recognition
Moving Objects
Location Motion
Action Order
Repetition Count
Camera Motion
Overall
Static Multi Keyframe Baseline
Uniform Sampling
–
62.86
72.03
64.10
44.59
27.75
63.90
58.85
Similarity Based Token Compression and Frame Selection
DyCoke [ 40 ]
CVPR’25
63.67
70.29
64.65
42.28
29.50
60.52
58.48
PruneVid [ 16 ]
ACL’25
56.22
66.23
54.76
42.47
24.25
36.62
50.91
HoliTom [ 35 ]
NeurIPS’25
63.87
70.00
63.55
42.66
27.50
62.86
58.43
Table 1: Per category accuracy (%) across the six official motion categories of MotionBench on the shared Qwen3.8 27B backbone under the unified dynamic 3fps candidate pool ( M(T)∈[8,96]→Kf=8 reader frames; bold indicates best and underline second best).
Method
Venue
Video-MME
TempCompass
Short
Medium
Long
Overall
Action
Direction
Speed
Order
Attribute Change
Overall
Static Multi Keyframe Baseline
Uniform Sampling
–
65.78
55.00
51.00
57.26
96.40
58.42
54.58
76.49
88.39
74.82
Similarity Based Token Compression and Frame Selection
DyCoke [ 40 ]
CVPR’25
63.67
53.11
49.33
55.37
97.48
60.14
54.58
75.75
87.64
75.11
PruneVid [ 16 ]
ACL’25
48.89
52.22
47.78
49.63
93.88
30.93
40.84
52.24
34.08
50.44
Table 2: Accuracy comparison (%) on Video-MME and TempCompass on the shared Qwen3.8 27B backbone under the unified dynamic 3fps candidate pool ( M(T)∈[8,96]→Kf=8 reader frames; all 12 Video-MME task types are in Tab. S1 ). Bold indicates best and underline second best.
Variant
Vis. Delta
Ent. State
LoRA
ADKP
MB
VMME
TC
Overall
Static Baseline
–
–
–
–
58.85
57.26
74.82
61.02
Visual Delta Only
✓
–
–
–
59.87
56.96
75.84
61.60
Entity State Only
–
✓
–
–
60.17
57.78
74.60
61.81
Visual Delta + LoRA
✓
–
✓
–
62.14
60.74
76.87
64.16
Dual Road (w/o ADKP)
✓
✓
✓
–
62.68
60.85
77.38
64.56
Full VIVID
✓
✓
✓
✓
62.78
61.48
77.38
64.82
Table 3: Component ablation of VIVID across MotionBench (MB), Video-MME (VMME), and TempCompass (TC), in accuracy (%) on the shared Qwen3.8 27B backbone ( Kf=8 reader frames; Overall pools all 8,083 questions).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Task Category
Questions
Uniform Sampling
DyCoke
PruneVid
HoliTom
FastVID
ST-SimDiff
EarlyTom
MovieChat
VideoStreaming
VideoMind
ProtoKV
VIVID-Small
VIVID-Large
Information Synopsis
323
76.16
76.78
69.35
74.30
77.40
75.23
75.23
75.23
78.02
73.37
76.47
76.16
78.33
Object Reasoning
454
57.93
55.73
51.32
55.51
56.83
57.49
57.71
57.93
58.81
57.71
56.83
63.66
63.88
Object Recognition
354
58.76
54.24
46.61
55.93
55.08
53.11
59.04
51.98
56.50
57.91
56.21
62.43
61.58
Action Recognition
313
53.35
50.16
50.16
54.31
55.59
52.40
49.84
49.84
53.67
52.08
50.16
55.59
55.91
Action Reasoning
285
57.19
58.25
51.23
56.14
55.44
56.49
57.54
55.44
55.09
54.74
59.30
58.25
59.30
Counting Problem
268
33.96
32.46
25.00
31.72
29.85
28.73
31.34
30.22
36.94
27.99
35.82
41.79
42.54
Appendix
Table S1 : Per category accuracy (%) across all 12 task types of the complete 2,700 question Video-MME benchmark on the shared 27B backbone (OCR: optical character recognition; bold indicates best and underline second best).
Vis. Delta
Ent. State
LoRA
Pathway Gate
Frames
Configuration / Policy
TempCompass ( 1,366 Questions)
Action
Direction
Speed
Order
Attribute Change
Overall
(A) Single Pathway Variants under Matched Kf=8 Uniform Keyframes
–
–
–
–
8
Uniform Sampling
96.40
58.42
54.58
76.49
88.39
74.82
–
✓
–
–
8
Entity State Pathway Only
96.40
54.30
59.54
78.73
84.64
74.60
✓
–
–
–
8
Visual Delta Pathway Only
97.48
59.11
56.49
77.99
88.39
75.84
✓
–
✓
–
8
Visual Delta + Final Layer LoRA
97.12
59.45
62.98
75.75
89.51
76.87
Appendix
Table S2 : Matched eight frame ( Kf=8 ) component, pathway gating, and keyframe planning breakdown across the five official dimensions of held out TempCompass ( 1,366 questions) under the unified 3fps candidate pool ( bold indicates best and underline second best).
Recent video multimodal large language models (MLLMs) increasingly couple step-by-step reasoning with on-demand visual evidence retrieval, allowing models to revisit relevant video segments during inference. However, two structural gaps remain in existing thinking-with-video systems. (i) Sampling density is not a learnable decision: existing methods may let the model decide where to look, but the per-window frame rate is largely fixed. As a result, fine-grained evidence is often recovered through repeated retrieval calls, which increases inference context length and training difficulty. (ii) Retrieval and answer generation are usually optimized with a single trajectory-level advantage, so the "where to look" tokens and the "how to answer" tokens receive the same credit even when one is correct and the other is not. To address these gaps, we present DynFrame, a framework that emits the temporal window and the sampling density as native tokens within a single autoregressive pass. This learnable span-density retrieval enables acquiring multi-granularity evidence with a single retrieval step. Based on the above tokenized retrieval interface, we further introduce Segment-Decoupled GRPO (SD-GRPO), which splits each rollout at the retrieval boundary and assigns role-specific token-level advantages, separately crediting the sampling decision and the answer. Trained on the curated DM-CoT-74k and DM-RL-45k, DynFrame-4B is competitive with strong 7B-8B baselines across six benchmarks (NExT-GQA, Charades-STA, ActivityNet-MR, Video-MME, MLVU, LVBench), and DynFrame-8B sets new state-of-the-art on most metrics. Code is available at https://github.com/zhangguanghao523/DynFrame.
Multimodal Large Language Models (MLLMs) have made strong progress in video understanding, yet long videos remain difficult: the visual token budget grows with video length, so temporally sparse evidence is easily lost. Existing methods compress the input through uniform sampling or frame selection, but these strategies optimize different objectives, either broad temporal coverage or local question relevance, and neither preserves both global storyline context and fine-grained evidence. We propose VideoRouter (VR), which rethinks long-video understanding as coordinating complementary evidence views rather than selecting a single subset of frames. VideoRouter first organizes each video into a question-agnostic temporal hierarchy that partitions it into coarse-to-fine temporally coherent segments. Upper-level nodes capture broad storyline context and event progression, while lower-level nodes preserve fine-grained local details and evidence-bearing moments. This gives rise to two complementary views: a global view for coverage-oriented reasoning and a local view for detail-oriented evidence recovery. We further introduce a verification-guided router that judges which view is better supported by its own selected evidence and decides the final answer. Across six backbones, routing improves over both views in all settings, and the choice of view is shown to be dataset-dependent, confirming that no single evidence granularity is universally preferable. On VideoMME, our method outperforms state-of-the-art frame selection methods by 2.5 points, under the LLaVA-Video-7B backbone. We will release the code.
Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension to video has enabled tasks such as video question answering and video captioning, their pixel-level visual grounding abilities are less studied. In this work, we raise the pertinent question of whether motion is used in pixel-level visual grounding and whether video MLLMs can segment objects based on natural language expressions describing their motion patterns. We identify the shortcomings in the current benchmarks, where we show that a single frame can often suffice for capturing the referring expression without any temporal reasoning. To address this, we introduce novel motion-centric probing, particularly designed for the visual grounding task, to study video MLLMs' ability to identify true motion from a fake one and their ability to grasp the ordering of motion. Consequently, we introduce MoCentric-Bench, a motion-centric benchmark designed to evaluate video MLLMs based on their ability to capture the interaction between motion and language, rather than relying primarily on static cues. We further establish strong single-image baselines that either match or surpass prior methods. Finally, we explore a simple motion-centric adaptation that provides state-of-the-art performance on our MoCentric-Bench. Code is available at https://msiam.github.io/PixFoundationSeries/.