Organizations: Peking University · Shanghai Jiao Tong University · Harbin Institute of Technology · Southeast University · Institute of Automation, Chinese Academy of Sciences · Xi’an Jiaotong University · Zhejiang University · Nanyang Technological University · Xiamen University
Video question answering under tight frame budgets requires preserving localized motion and event order, yet current video multimodal large language models (MLLMs) reduce comprehension to static multi-image reasoning by discarding unselected frames and encoding keyframes independently: (1) Spatially, whole-frame pooling dilutes small moving targets into static backgrounds; (2) Temporally, dropping intermediate frames or averaging tokens erases inter-frame motion, confusing direction, speed, and order. We propose VIVID (Visual Inter-frame Variations and Individual Dynamics), which moves beyond static snapshots like human vision: keyframes capture core states while inter-frame Patch Diffs preserve neighboring changes within Kf keyframes. Autonomous Delta-Driven Keyframe Planning (ADKP) selects anchors from a 3 fps pool and feeds surrounding patch differences into Dual-Road Change Memory (DRCM), forming Dynamic Tokens at zero extra anchor tokens alongside a camera-stabilized Active Motion Region of Interest (ROI) and chronological Entity State Chains. Adding 0.22% trainable parameters to a frozen 27B backbone, VIVID-Small (50 examples, ∼1.5 GPU hours) beats all eleven baselines in overall accuracy across 8,083 questions and trails only VIVID-Large (3,600 examples), which leads on MotionBench, Video-MME, and TempCompass.
Figures & tables
Figure 2: Overview of the VIVID architecture (ViT: Vision Transformer; MLP: multilayer perceptron; FFN: feedforward network; ROI: region of interest; LoRA: Low Rank Adaptation): (a) across 3fps candidates ( Sec. 3 ), the frozen ViT computes Patch Diffs ΔEt so ADKP ( Eqs. 6 , 7 and 8 ) selects Kf=8 transition anchors across intervals B1…B6 and crops a 320×180 Active Motion ROI ; (b) surrounding Patch Diffs enter Dual Road Change Memory (DRCM) , where the Visual Delta Module θΔ enriches static anchors Et^k into Dynamic Tokens E~t^k ( Eqs. 3 and 4 ) alongside precomputed Entity State Chains ( Eq. 5 ); (c) the Transition Conditioned Input feeds the shared reader, with the first 48 LLM layers frozen and the final 16 layers adapted via LoRA.
Method
Venue
Motion Recognition
Moving Objects
Location Motion
Action Order
Repetition Count
Camera Motion
Overall
Static Multi Keyframe Baseline
Uniform Sampling
–
62.86
72.03
64.10
44.59
27.75
63.90
58.85
Similarity Based Token Compression and Frame Selection
DyCoke [ 40 ]
CVPR’25
63.67
70.29
64.65
42.28
29.50
60.52
58.48
PruneVid [ 16 ]
ACL’25
56.22
66.23
54.76
42.47
24.25
36.62
50.91
HoliTom [ 35 ]
NeurIPS’25
63.87
70.00
63.55
42.66
27.50
62.86
58.43
Table 1: Per category accuracy (%) across the six official motion categories of MotionBench on the shared Qwen3.8 27B backbone under the unified dynamic 3fps candidate pool ( M(T)∈[8,96]→Kf=8 reader frames; bold indicates best and underline second best).
Method
Venue
Video-MME
TempCompass
Short
Medium
Long
Overall
Action
Direction
Speed
Order
Attribute Change
Overall
Static Multi Keyframe Baseline
Uniform Sampling
–
65.78
55.00
51.00
57.26
96.40
58.42
54.58
76.49
88.39
74.82
Similarity Based Token Compression and Frame Selection
DyCoke [ 40 ]
CVPR’25
63.67
53.11
49.33
55.37
97.48
60.14
54.58
75.75
87.64
75.11
PruneVid [ 16 ]
ACL’25
48.89
52.22
47.78
49.63
93.88
30.93
40.84
52.24
34.08
50.44
Table 2: Accuracy comparison (%) on Video-MME and TempCompass on the shared Qwen3.8 27B backbone under the unified dynamic 3fps candidate pool ( M(T)∈[8,96]→Kf=8 reader frames; all 12 Video-MME task types are in Tab. S1 ). Bold indicates best and underline second best.
Variant
Vis. Delta
Ent. State
LoRA
ADKP
MB
VMME
TC
Overall
Static Baseline
–
–
–
–
58.85
57.26
74.82
61.02
Visual Delta Only
✓
–
–
–
59.87
56.96
75.84
61.60
Entity State Only
–
✓
–
–
60.17
57.78
74.60
61.81
Visual Delta + LoRA
✓
–
✓
–
62.14
60.74
76.87
64.16
Dual Road (w/o ADKP)
✓
✓
✓
–
62.68
60.85
77.38
64.56
Full VIVID
✓
✓
✓
✓
62.78
61.48
77.38
64.82
Table 3: Component ablation of VIVID across MotionBench (MB), Video-MME (VMME), and TempCompass (TC), in accuracy (%) on the shared Qwen3.8 27B backbone ( Kf=8 reader frames; Overall pools all 8,083 questions).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Task Category
Questions
Uniform Sampling
DyCoke
PruneVid
HoliTom
FastVID
ST-SimDiff
EarlyTom
MovieChat
VideoStreaming
VideoMind
ProtoKV
VIVID-Small
VIVID-Large
Information Synopsis
323
76.16
76.78
69.35
74.30
77.40
75.23
75.23
75.23
78.02
73.37
76.47
76.16
78.33
Object Reasoning
454
57.93
55.73
51.32
55.51
56.83
57.49
57.71
57.93
58.81
57.71
56.83
63.66
63.88
Object Recognition
354
58.76
54.24
46.61
55.93
55.08
53.11
59.04
51.98
56.50
57.91
56.21
62.43
61.58
Action Recognition
313
53.35
50.16
50.16
54.31
55.59
52.40
49.84
49.84
53.67
52.08
50.16
55.59
55.91
Action Reasoning
285
57.19
58.25
51.23
56.14
55.44
56.49
57.54
55.44
55.09
54.74
59.30
58.25
59.30
Counting Problem
268
33.96
32.46
25.00
31.72
29.85
28.73
31.34
30.22
36.94
27.99
35.82
41.79
42.54
Appendix
Table S1 : Per category accuracy (%) across all 12 task types of the complete 2,700 question Video-MME benchmark on the shared 27B backbone (OCR: optical character recognition; bold indicates best and underline second best).
Vis. Delta
Ent. State
LoRA
Pathway Gate
Frames
Configuration / Policy
TempCompass ( 1,366 Questions)
Action
Direction
Speed
Order
Attribute Change
Overall
(A) Single Pathway Variants under Matched Kf=8 Uniform Keyframes
–
–
–
–
8
Uniform Sampling
96.40
58.42
54.58
76.49
88.39
74.82
–
✓
–
–
8
Entity State Pathway Only
96.40
54.30
59.54
78.73
84.64
74.60
✓
–
–
–
8
Visual Delta Pathway Only
97.48
59.11
56.49
77.99
88.39
75.84
✓
–
✓
–
8
Visual Delta + Final Layer LoRA
97.12
59.45
62.98
75.75
89.51
76.87
Appendix
Table S2 : Matched eight frame ( Kf=8 ) component, pathway gating, and keyframe planning breakdown across the five official dimensions of held out TempCompass ( 1,366 questions) under the unified 3fps candidate pool ( bold indicates best and underline second best).