StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding

Guowei Tang; Tianwen Qian; Huanran Zheng; Yifei Wang; Xiaoling Wang

cs.CVMar 23, 2026

StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding

Authors: Guowei Tang, Tianwen Qian, Huanran Zheng, Yifei Wang, Xiaoling Wang

Organizations: East China Normal University, Shanghai, China

Abstract

Real-time, continuous understanding of visual signals is essential for real-world interactive AI applications, and poses a fundamental system-level challenge. Existing research on streaming video understanding, however, typically focuses on isolated aspects such as question-answering accuracy under limited visual context or improvements in encoding efficiency, while largely overlooking practical deployability under realistic resource constraints. To bridge this gap, we introduce StreamingEval, a unified evaluation framework for assessing the streaming video understanding capabilities of Video-LLMs under realistic constraints. StreamingEval benchmarks both mainstream offline models and recent online video models under a standardized protocol, explicitly characterizing the trade-off between efficiency, storage and accuracy. Specifically, we adopt a fixed-capacity memory bank to normalize accessible historical visual context, and jointly evaluate visual encoding efficiency, text decoding latency, and task performance to quantify overall system deployability. Extensive experiments across multiple datasets reveal substantial gaps between current Video-LLMs and the requirements of realistic streaming applications, providing a systematic basis for future research in this direction. Codes will be released at https://github.com/wwgTang-111/StreamingEval1.

Figures & tables

Figure 1: Illustration of the conventional offline video understanding paradigm versus the streaming paradigm. Top: offline inference with full access to the video. Middle: pseudo-streaming inference, which truncates videos at query timestamps but still processes each clip in an offline manner. Bottom: realistic streaming inference with real-time incremental input and limited memory bank.

Figure 2: Overview of the StreamingEval framework. The framework standardizes streaming video understanding by modeling continuous input ingestion, incremental visual memory updates, and query-driven inference within a unified protocol.

Figure 3: Overall accuracy versus memory_bank budget for three representative models. Detailed results are provided in Table 6 .

Input Res.	Qwen3-VL	InternVL 3.5	FlashVstream
224 $\times$ 224	$51.99$	$48.93$	$33.15$
336 $\times$ 336	$52.95$	$51.70$	$34.67$
448 $\times$ 448	$53.34$	$52.75$	$35.73$

Table 2: Overall results under different input resolutions on OVO-Bench. We report the overall score for each model under 224

\times

224, 336

\times

336, and 448

\times

448 inputs. Detailed results are provided in Table 7 .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model / Type	Best Answer ( $w_{a}=0.4$ )	Interaction First ( $w_{t}=0.4$ )	Edge Resource-Saving ( $w_{r}=0.4$ )	Throughput First ( $w_{f}=0.4$ )
Qwen3-VL-7B (Offline)	Rank 1	Rank 2	Rank 2	Rank 2
Flash-VStream (Online)	Rank 2	Rank 1	Rank 1	Rank 1
MiniCPM-V4.5 (Offline)	Rank 3	Rank 3	Rank 3	Rank 5
StreamForest (Online)	Rank 9	Rank 10	Rank 9	Rank 9

Appendix

Table 4: Ranking changes of representative models under different deployment preferences.

Factor	Value
GPU model	NVIDIA GeForce RTX 4090-48G
Peak throughput	40.32 TFLOPS
GPU memory (VRAM)	48 GB
GPU memory bandwidth	1008.10 GB/s
PCIe lanes	16
PCIe bandwidth	31.50 GB/s

Appendix

Table 5: Hardware factors affecting performance

Figure 4: Comparison of two classic paradigms for video understanding.

Streaming Video Understanding

Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.

See all 132 papers

Harnessing Streaming Video in the Wild
Jun 7, 2026Dingyu Yao, Shuhuan Gu, Qingyi Si +8VLM Evaluation Streaming Video Understanding
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
A Simple Baseline for Streaming Video Understanding
Apr 2, 2026Yujiao Shen, Shulin Tian, Jingkang Yang +1Streaming Video Understanding Streaming Inference
S-Lab, Nanyang Technological University
VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant Models
Apr 8, 2026Pavan Kumar Anasosalu Vasu, Cem Koc, Fartash Faghri +6VLM Evaluation Video-Language Model Evaluation
Apple

Jun 7, 2026J/K move · Enter open · S save

Harnessing Streaming Video in the Wild

Dingyu Yao, Shuhuan Gu, Qingyi Si, Junhao Zhou+7

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

VLM Evaluation Streaming Video Understanding VLM Adaptation Streaming Inference

Open paper AskPDF arXiv

Vision-Language Models (VLMs) are increasingly required to process unbounded video streams in applications such as video-call assistants, live commentary, and embodied robots. An ideal streaming system should support proactive interaction, long-horizon memory, and real-time processing, while resting on a VLM backbone capable of handling diverse in-the-wild streaming tasks. However, existing VLMs excel at offline video understanding but fall short in streaming capabilities and lack dedicated infrastructure for streaming deployment. We address this gap on three fronts. (i) For backbone capability, we construct Streaming-Train-248K, a streaming dataset paired with a novel training objective for adapting VLMs to streaming interaction and understanding. (ii) For real-world deployment, we introduce Streaming Harness, a plug-and-play system that endows any VLM with three core abilities: proactive interaction (per-second response decisions), long-term memory (12-hour context retention), and real-time processing (sub-second latency). (iii) To drive continued community progress on streaming capabilities, we design Streaming-Eval, a benchmark that reflects models' capabilities across diverse in-the-wild scenarios. Extensive experiments demonstrate consistent gains from our approach across all core capabilities required for streaming video understanding. We will open-source our data, code, and benchmark to advance the community's shift from offline video understanding to deployable streaming intelligence.

Abstract

Streaming Video Understanding

Explore similar work

Harnessing Streaming Video in the Wild

A Simple Baseline for Streaming Video Understanding

VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant Models