cs.CVApr 8, 2026

VSAS-Bench: Real-Time Evaluation of Visual Streaming Assistant Models

Authors: Pavan Kumar Anasosalu Vasu, Cem Koc, Fartash Faghri, Chun-Liang Li, Bo Feng, Zhengfeng Lai, Meng Cao, Oncel Tuzel, 

Organizations: Apple

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core mechanism for real-time visual assistants. Existing VLM frameworks predominantly assess models in offline settings. In contrast, the performance of a streaming VLM depends on additional metrics beyond pure video understanding, including proactiveness, which reflects the timeliness of the model's responses, and consistency, which captures the robustness of its responses over time. To address this limitation, we propose VSAS-Bench, a new framework and benchmark for Visual Streaming Assistants. In contrast to prior benchmarks that primarily employ single-turn question answering on video inputs, VSAS-Bench features temporally dense annotations with over 18,000 annotations across diverse input domains and task types. We introduce standardized synchronous and asynchronous evaluation protocols, along with metrics that isolate and measure distinct capabilities of streaming VLMs. Using this framework, we conduct large-scale evaluations of recent video and streaming VLMs, analyzing the accuracy-latency trade-off under key design factors such as memory buffer length, memory access policy, and input resolution, yielding several practical insights. Finally, we show empirically that conventional VLMs can be adapted to streaming settings without additional training, and demonstrate that these adapted models outperform recent streaming VLMs. For example, Qwen3-VL-4B surpasses Dispider, the best streaming VLM on our benchmark, by 3% under the asynchronous protocol. The benchmark and code will be available at https://github.com/apple/ml-vsas-bench.

Figures & tables

Figure 1: VSAS-Bench accuracy uncovers advantage of small models as visual streaming assistants. Under the synchronous evaluation protocol, models run in lockstep with the camera and larger models benefit from unlimited processing time. However, in the more realistic asynchronous evaluation, smaller models outperform because their high inference speed allows them to respond more rapidly and effectively to the live stream.
BenchmarkLong HorizonDenseFree-formAsync. Eval.Time-aware
ReasoningAnnotationsResponseProtocolMetrics
Ego4D [ 8 ]✓✓✗✗✗
COIN [ 23 ]✗✓✗✗✗
VideoLLM-Online [ 4 ]✗✓✓✗✓
VStream-QA [ 35 ]✗✗✓✗✗
E.T. Bench [ 15 ]✓✗✓✗✗
Table 1: Comparison to prior video streaming benchmarks.
Figure 2: VSAS-Bench comprises densely annotated videos with a wide range of actions, varying durations, and reasoning horizons. (a) Distribution of video categories, showing the diversity of tasks covered. (b) Mean video duration (with standard deviation bars) for each video category. (c) Distribution of task types, highlighting the split between short- and long-horizon temporal reasoning.
Figure 3: Examples of VSAS-Bench task types with frame-level annotations. VSAS-Bench involves three task types, Present tasks, which focus on currently occurring events; Cumulative tasks, which require the model to reason over past events; and Future tasks, which focus on predicting upcoming events based on ongoing visual cues.
Figure 4: Traditional versus realistic evaluation protocols. (top) Synchronous Protocol : Frame generation is temporally aligned with the VLM’s processing rate, ensuring camera input and model inference occur in lockstep. (bottom) Asynchronous Protocol : Frame acquisition is decoupled from inference; the VLM retrieves frames from a buffered queue, emulating real-world streaming on edge devices.
ModelSync. Accuracy ↑\uparrowSync. Consistency ↑\uparrowAsync. Accuracy ↑\uparrowAsync. Consistency ↑\uparrow
PresentCumulativeFutureOverallPresentCumulativeFutureOverall
API Models
GPT-5 [ 20 ] (low)83.164.681.877.179.534.617.510.125.199.4
GPT-5 [ 20 ] (medium)81.565.384.477.079.336.321.912.327.899.6
GPT-5 [ 20 ] (high)80.964.385.176.480.337.526.215.930.399.7
Open-Source Video VLMs
Table 2: Streaming-adapted video VLMs surpass prior streaming VLMs on VSAS-Bench . Models are grouped by their streaming capability and being open- or closed-source. Hardware, model configurations, and protocol settings are detailed in Sec. 4.2 . Video models are adapted for asynchronous evaluation using the method in Sec. 4.1 with SW policy. We highlight the best numbers in each column per model category and for accuracies, we highlight numbers within maximum standard deviation of judge evaluations across tasks (0.6%). GPT-5 is accessed via a web API, and the resulting network latency hurts performance under the asynchronous protocol relative to locally deployed models such as Qwen3-VL and InternVL3; we therefore gray out these results in the table.
Memory PolicyTasksOverall
PresentCumulativeFuture
GPT-5
SW81.565.384.477.0
U81.172.382.178.5
SW+U83.772.685.780.6
Qwen3-VL-8B
Table 3: Effect of memory policies. SW+U performs best or within one standard deviation of the judge (0.6%). We use SW in other experiments as the only feasible policy for long videos. We report under synchronous protocol with maximum context size of 64 frames. We highlight the best numbers per model.
ModelAccel.TasksOverall
PresentCumulativeFuture
Qwen3-VL-2BA10052.122.89.435.8
H10053.827.19.538.0
Qwen3-VL-4BA10058.931.916.343.3
H10062.336.417.946.8
Qwen3-VL-8BA10055.332.516.841.7
Table 4: Effect of Hardware Accelerators. We evaluate Qwen3-VL model family using the asynchronous protocol on Nvidia A100 and H100. All runs use memory buffer size 64 and SW policy.
Memory Buffer SizeTasksOverall
PresentCumulativeFuture
142.015.831.042.0
866.124.530.047.0
1663.430.026.546.8
3260.135.323.046.1
6457.934.320.744.3
Table 5: Effect of memory buffer size. We evaluate Qwen3-VL-8B model using the asynchronous protocol. Image resolution is fixed across all buffer sizes, and all runs use the SW policy. We highlight the best numbers in each column.
Min PixelsTasksOverall
PresentCumulativeFuture
16,38445.029.317.835.5
50,17655.528.521.941.4
200,70457.934.320.744.3
262,14459.838.019.946.3
409,60058.536.618.645.0
Table 6: Effect of input resolution. We evaluate Qwen3-VL-8B using the asynchronous protocol. All runs use memory buffer size 64 and SW policy. We highlight the best numbers in each column.
Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

BenchmarkOVO-BenchRTV-BenchOurs
Number of video clips64452292
MTQA median. duration between annotations (sec.)4.061.81.0
Total Annotations3207460818410
Avg. unique events per video clip22.511.8
Total number of unique events320746082117
Single GPU eval. duration (mins.)127395345
Appendix
Table 7: Benchmark Comparison Single GPU eval duration is measured for Qwen2.5VL-7B-Instruct model.
ReweightingQwen3-VL-4BQwen3-VL-32B
PresentCumulativeFutureOverallPresentCumulativeFutureOverall
Uniform (reported in main paper Tab. 2)62.336.417.946.854.832.114.541.0
Inverse Category64.638.526.448.458.137.319.943.5
Inverse Task63.338.424.847.856.235.918.942.3
Inverse Category and Task62.343.417.345.854.741.113.741.1
Appendix
Table 8: Model accuracies for various reweighting schemes.
Figure 5: Overview of self-speculative decoding for streaming VLMs . The prior response serves as a draft to be verified for the next frame.
Self-Speculative DecodingCumulative Task (OCR)Mean Avg. Latency (s)
AccuracyConsistency
✗21.593.05.8
✓55.196.21.5
Appendix
Table 9: Self-Speculative Decoding for Streaming VLMs. We evaluate streaming-adapted Qwen3-VL-8B model using the asynchronous protocol and compare the performance of the model with and without self-speculative decoding optimization.
Figure 15
Figure 16
Figure 6: Judge prompt used to evaluate accuracy of model response.
Figure 7: Examples of VSAS-Bench task types with frame-level annotations and full prompts. VSAS-Bench involves three task types, Present tasks, which focus on currently occurring events; Cumulative tasks, which require the model to reason over past events; and Future tasks, which focus on predicting upcoming events based on ongoing visual cues.
 Qwen3VL-4BQwen3VL-8BQwen3VL-32BFlashVStream (7B)Dispider (7B)GPT-5 †
Present1.31.62.40.81.742.4
Cumulative2.12.24.11.41.263.7
Future2.12.73.50.33.365.8
Appendix
Table 10: Average end-to-end latency in seconds per task-type. Video and streaming models were evaluated on NVIDIA H100. †\dagger For GPT-5 its the measure of its API latency.
ModelAsync. Accuracy ↑\uparrowAsync. Consistency ↑\uparrow
PresentCumulativeFutureOverall
Qwen3-VL-2B [ 24 ]54.029.99.639.096.5
Qwen3-VL-4B [ 24 ]61.639.318.447.495.0
Qwen3-VL-8B [ 24 ]57.841.622.046.895.8
Qwen3-VL-32B [ 24 ]55.636.115.642.896.7
Appendix
Table 11: Async protocol performance for Qwen3-VL family of models evaluated with SW+U memory policy.
Camera Buffer SizeTasksOverall
PresentCumulativeFuture
Qwen3-VL-2B
153.828.49.638.5
854.230.49.839.3
1654.331.210.139.7
60053.630.510.539.1
Appendix
Table 12: Effect of camera buffer size. We evaluate Qwen3-VL 2B and 8B sizes using the asynchronous protocol. All runs use memory buffer size 64 and SW+U policy. We highlight the best numbers in each column.
More in this topic

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 986 papers

Explore similar work

CardsList
Jun 7, 2026

Harnessing Streaming Video in the Wild

Vision-Language Models (VLMs) are increasingly required to process unbounded video streams in applications such as video-call assistants, live commentary, and embodied robots. An ideal streaming system should support proactive interaction, long-horizon memory, and real-time processing, while resting on a VLM backbone capable of handling diverse in-the-wild streaming tasks. However, existing VLMs excel at offline video understanding but fall short in streaming capabilities and lack dedicated infrastructure for streaming deployment. We address this gap on three fronts. (i) For backbone capability, we construct Streaming-Train-248K, a streaming dataset paired with a novel training objective for adapting VLMs to streaming interaction and understanding. (ii) For real-world deployment, we introduce Streaming Harness, a plug-and-play system that endows any VLM with three core abilities: proactive interaction (per-second response decisions), long-term memory (12-hour context retention), and real-time processing (sub-second latency). (iii) To drive continued community progress on streaming capabilities, we design Streaming-Eval, a benchmark that reflects models' capabilities across diverse in-the-wild scenarios. Extensive experiments demonstrate consistent gains from our approach across all core capabilities required for streaming video understanding. We will open-source our data, code, and benchmark to advance the community's shift from offline video understanding to deployable streaming intelligence.

Dingyu Yao, Shuhuan Gu, Qingyi Si +8

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

VLM EvaluationStreaming Video UnderstandingVLM AdaptationStreaming Inference
AskPDF
Oct 8, 2026

FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?

Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events. We introduce FastBench to evaluate high-dynamic perception in real-world video streams. Its trajectory-grounded pipeline combines QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection. FastBench contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals. We also present ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. A dual-tier sliding window retains recent high-FPS observations while downsampling older ones into sparse history. Experiments reveal substantial limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%. Denser sampling improves Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, yet remains well below oracle-guided focusing, showing that current VLMs struggle to determine from the stream alone when finer temporal perception is needed. FastBench provides a testbed for high-dynamic streaming video understanding. Code and data: https://github.com/Ashone3/FastBench.

Yuxuan Hu, Weikang Shi, Yang Bo +10

CUHK MMLab

Adaptive Video Frame SamplingStreaming Video UnderstandingVideo-Language Model EvaluationAdaptive Sampling
AskPDF
Mar 23, 2026

StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding

Real-time, continuous understanding of visual signals is essential for real-world interactive AI applications, and poses a fundamental system-level challenge. Existing research on streaming video understanding, however, typically focuses on isolated aspects such as question-answering accuracy under limited visual context or improvements in encoding efficiency, while largely overlooking practical deployability under realistic resource constraints. To bridge this gap, we introduce StreamingEval, a unified evaluation framework for assessing the streaming video understanding capabilities of Video-LLMs under realistic constraints. StreamingEval benchmarks both mainstream offline models and recent online video models under a standardized protocol, explicitly characterizing the trade-off between efficiency, storage and accuracy. Specifically, we adopt a fixed-capacity memory bank to normalize accessible historical visual context, and jointly evaluate visual encoding efficiency, text decoding latency, and task performance to quantify overall system deployability. Extensive experiments across multiple datasets reveal substantial gaps between current Video-LLMs and the requirements of realistic streaming applications, providing a systematic basis for future research in this direction. Codes will be released at https://github.com/wwgTang-111/StreamingEval1.

Guowei Tang, Tianwen Qian, Huanran Zheng +2

East China Normal University, Shanghai, China

Streaming Video UnderstandingVideo-Language Model EvaluationEfficient VLM Inference
AskPDF