RIVER: A Real-Time Interaction Benchmark for Video LLMs
Authors: Yansong Shi, Qingsong Zhao, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, Limin Wang
Organizations: School of Information Science And Technology, University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · College of Computer Science and Artificial Intelligence, Fudan University · State Key Lab of Novel Software Technology, Nanjing University
The rapid advancement of multimodal large language models has demonstrated impressive capabilities, yet nearly all operate in an offline paradigm, hindering real-time interactivity. Addressing this gap, we introduce the Real-tIme Video intERaction Bench (RIVER Bench), designed for evaluating online video comprehension. RIVER Bench introduces a novel framework comprising Retrospective Memory, Live-Perception, and Proactive Anticipation tasks, closely mimicking interactive dialogues rather than responding to entire videos at once. We conducted detailed annotations using videos from diverse sources and varying lengths, and precisely defined the real-time interactive format. Evaluations across various model categories reveal that while offline models perform well in single question-answering tasks, they struggle with real-time processing. Addressing the limitations of existing models in online video interaction, especially their deficiencies in long-term memory and future perception, we proposed a general improvement method that enables models to interact with users more flexibly in real time. We believe this work will significantly advance the development of real-time interactive video understanding models and inspire future research in this emerging field. Datasets and code are publicly available at https://github.com/OpenGVLab/RIVER.
Figures & tables
Figure 1: Illustration of different online interaction tasks. The question (Query), reference events (Cue), and answers timings are represented by , and , respectively. Based on the frequency and timing of reference events, questions, and answers, we further categorize online interaction tasks into four distinct subclasses, as visually depicted in the figure. For the Retro-Memory, the clue is drawn from the past; for the Live-Perception, it comes from the present—both demand an immediate response. For the Pro-Response task, Video LLMs need to wait until the corresponding clue appears and then respond as quickly as possible.
Figure 2: The pie chart on the left illustrates the quantitative distribution of various tasks within the benchmark. The two bar charts in the middle depict the statistics of video duration and the proportion of question timestamps, respectively. On the right, a word cloud which is constructed from the annotated textual data within the dataset, visually emphasizing the most frequently occurring terms.
Benchmark
Videos
Questions
Memory & Perception
Anticipation
General
Short
Medium
Long
Very Long
Instant
Stream
VStream-QA ( Zhang et al., 2024a )
32
3,500
✓
✗
✗
✗
✗
✗
✗
StreamingBench ( Lin et al., 2024 )
900
4,500
✓
✗
✗
✗
✗
✗
✗
OVBench ( Huang et al., 2024 )
1,463
4,874
✓
✗
✗
✗
✗
✗
✗
OVO-Bench ( Li et al., 2025 )
644
3,100
✓
✗
✗
✗
✗
✓
✓
RIVER (ours)
1,067
4,278
✓
✓
✓
✓
✓
✓
✓
Table 1: Comparison of RIVER and other video online benchmarks.
Figure 3: Illustration of data processing process.
Figure 4: Pipeline to enable MLLMs to support online inference capabilities. The Long Short-Term Memory module continuously receives new visual features and selects the most important parts. After a query is posed at t0 , the model is queried at each time window; if it decides to answer, the final response is output.
Models
LLMs
#F
Retro-Memory
Live-Perception
Pro-Response
OE
MC
OE
MC
Instant
Streaming
Loc
MC
OE
Closed-Source Models
GPT-4o
-
50
39.09
59.56
40.08
61.05
∅
∅
1.63
Gemini-1.5-pro
-
50
24.24
36.35
34.53
52.19
∅
∅
1.51
Open-Source Models
Table 2: RIVER Bench evaluation results regarding the core online video understanding capability. "#F" indicates the total number of frames processed by the model during inference. "Loc" measures how precisely the model’s response falls within the correct time window in Pro-Response task. " ∅ " means that some models do not have the corresponding capabilities. All numbers in the table are presented in percentage (%). The gray background indicates methods that either natively support online inference or have been adapted to enable online inference capabilities.
Models
#F
Pro-Response
Instant
Streaming
Loc
MC
OE
Flash-VStream
1fps
-
-
1.31
VideoLLM-Online
2fps
23.88
6.67
4.41
VideoLLM-Online +RIVER
2fps
33.28
9.84
5.03
VideoLLM-Online +RIVER
4fps
35.16
10.53
5.47
Table 3: Online evaluation results regarding the Pro-Anticipation capability.
Models
LLMs
#F
Short
Medium
Long
Very Long
MC
OE
MC
OE
MC
OE
MC
OE
Closed-Source Models
GPT-4o
-
50
63.26
41.99
63.26
41.99
58.01
38.12
52.21
34.25
Gemini-1.5-pro
-
50
41.77
28.45
36.98
24.03
34.73
23.20
31.94
20.99
Open-Source Models
VideoChat2
Mistral-7B ( Jiang et al., 2023 )
16
32.04
19.34
32.32
20.44
31.49
19.34
32.04
18.51
Table 4: Evaluation results of Retro-Memory type questions.
Figure 9
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Illustration of data processing.
Parameter
Value
attn_implementation
flash_attention_2
learning_rate
0.0002
optim
adamw_torch
lr_scheduler_type
cosine
lora_modules
model.*(q_proj|k_proj|v_proj|o_proj|
gate_proj|up_proj|down_proj)|lm_head
Appendix
Table 6: Training parameters of online model.
Figure 7: Design of long-term memory.
Dataset
License / Restriction
Our Action for Release
LongVideoBench
CC-BY-NC-SA 4.0 (non-commercial only)
Index-only release – we provide JSON/CSV metadata and download links; videos are not re-hosted.
Vript-RR
Academic-use-only, no redistribution
Index-only release – we release scripts, questions, and timestamps only; users must download videos themselves under the original license terms.
LVBench
MIT (code) + original video copyrights
Index-only release – README explicitly states: “Video copyrights remain with original platforms; users must comply with their respective terms.”