FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?

Yuxuan Hu; Weikang Shi; Yang Bo; Xudong Lu; Xintong Guo; Shuhan Li; Yuyang He; Huankang Guan

cs.CVOct 8, 2026

FastBench: Can Streaming VLMs Perceive High-Dynamic Real-World Streams?

Authors: Yuxuan Hu, Weikang Shi, Yang Bo, Xudong Lu, Xintong Guo, Shuhan Li, Yuyang He, Huankang Guan, +5 more

Organizations: CUHK MMLab

Abstract

Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events. We introduce FastBench to evaluate high-dynamic perception in real-world video streams. Its trajectory-grounded pipeline combines QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection. FastBench contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals. We also present ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. A dual-tier sliding window retains recent high-FPS observations while downsampling older ones into sparse history. Experiments reveal substantial limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%. Denser sampling improves Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, yet remains well below oracle-guided focusing, showing that current VLMs struggle to determine from the stream alone when finer temporal perception is needed. FastBench provides a testbed for high-dynamic streaming video understanding. Code and data: https://github.com/Ashone3/FastBench.

Figures & tables

Figure 1: A perception failure in a high-dynamic video event. A wooden separator (highlighted in purple using SAM 3) is removed, allowing the metal rods to touch and produce sparks. Both models provide generic electrical explanations that could be inferred from the sparks alone, but fail to identify the brief sequence of separator removal and subsequent contact that triggers the discharge. This illustrates the challenge of capturing critical visual details in fast-paced events.

Panel of Figure 2 — Figure 2: Overview of FastBench. Evidence Start and Evidence End in (a) mark the human-annotated interval containing the visual evidence required to answer each question.

Figure 3: FastBench data construction pipeline. A VLM generates candidate QA pairs from short, high-frame-rate video clips. Questions answerable at 2 FPS are discarded. The remaining pairs undergo automated vision-expert verification, where the VLM uses SAM3 segmentation and CoTracker3 trajectories to verify or refine answers; pairs that cannot be verified or corrected are discarded. Retained QA pairs are mapped back to their source videos and undergo human quality checks to form FastBench. The lower panel illustrates how trajectory evidence supports correcting a punch count from two to three.

Figure 4: Overview of ProactiveFrame. (a) The model switches between low- and high-FPS sampling through <Focus Start> and <Focus End> ; <Silent> preserves the current mode. (b) A 120-frame General Window includes a 90-frame Focus Window. When a new high-FPS chunk arrives, the oldest focused chunk is downsampled into sparse history, and the oldest historical chunk is evicted to maintain the budget. Each chunk spans one second; F denotes retained frames.

Model	Capability Breakdown (by Sample Ratio $\downarrow$ )						Temporal Scope			Overall
	Action & Phys. Inter.	Predictive & Causal	Motion & Spatiotemp.	Entity & Visual	Temporal & Dynamics	Streaming & Online	Forward	Instant	Backward
	(51.96%)	(14.38%)	(13.40%)	(9.48%)	(8.17%)	(2.61%)	(54.90%)	(23.86%)	(21.24%)
Closed-source Models
Doubao-Seed-2.0-Lite ( ByteDance Seed, 2026 )	25.0	51.4	16.6	35.2	30.4	42.5	33.2	24.9	25.2	29.5
GPT-5.6-Sol ( OpenAI, 2026 )	42.6	56.8	57.6	45.5	43.2	30.0	45.1	51.8	44.9	46.7
Gemini-3.5-Flash ( Google DeepMind, 2026b )	43.6	67.3	62.9	44.8	52.0	52.5	52.7	48.5	47.7	50.7

Table 1: Main evaluation results on FastBench . Capabilities are ordered from left to right by sample ratio in descending order. Scores are presented in percentage (%). Values in green or red parentheses indicate gains or drops relative to their respective base models.

Method	Capability Breakdown (by Sample Ratio $\downarrow$ )						Temporal Scope			Overall
	Action & Phys. Inter.	Predictive & Causal	Motion & Spatiotemp.	Entity & Visual	Temporal & Dynamics	Streaming & Online	Forward	Instant	Backward
	(51.96%)	(14.38%)	(13.40%)	(9.48%)	(8.17%)	(2.61%)	(54.90%)	(23.86%)	(21.24%)
Qwen3-VL (8B)
Full 30 FPS	41.0	40.9	57.6	37.2	42.4	7.5	42.0	46.3	37.5	42.1
Proactive Frame	36.1	42.7	47.3	33.8	34.4	40.0	43.0	32.3	32.9	38.3
Mixed FPS (Focus)	41.6	41.4	49.3	35.2	36.8	7.5	39.4	43.8	40.6	40.7

Table 2: Ablation study on temporal context formulation using Qwen3-VL (8B) and Gemini-3.5-Flash on FastBench. Mixed FPS (Focus) provides original 30 FPS video chunks during the annotated focus timestamps and 2 FPS video chunks otherwise.

FPS	Capability Breakdown (by Sample Ratio $\downarrow$ )						Temporal Scope			Overall
	Action & Phys. Inter.	Predictive & Causal	Motion & Spatiotemp.	Entity & Visual	Temporal & Dynamics	Streaming & Online	Forward	Instant	Backward
	(51.96%)	(14.38%)	(13.40%)	(9.48%)	(8.17%)	(2.61%)	(54.90%)	(23.86%)	(21.24%)
2 FPS	30.6	34.1	47.3	32.4	32.0	2.5	33.9	34.5	28.3	32.9
4 FPS	37.0 (+6.4)	40.5 (+6.4)	50.2 (+2.9)	35.2 (+2.8)	41.6 (+9.6)	2.5 (0.0)	37.4 (+3.5)	41.1 (+6.6)	38.8 (+10.5)	38.6 (+5.7)
8 FPS	40.3 (+9.7)	38.6 (+4.5)	54.1 (+6.8)	33.8 (+1.4)	43.2 (+11.2)	0.0 (-2.5)	38.1 (+4.2)	44.1 (+9.6)	42.5 (+14.2)	40.5 (+7.6)
16 FPS	43.0 (+12.4)	44.5 (+10.4)	59.0 (+11.7)	40.0 (+7.6)	40.0 (+8.0)	2.5 (0.0)	41.8 (+7.9)	45.8 (+11.3)	46.8 (+18.5)	43.8 (+10.9)

Table 3: Ablation study on input video frame rate (FPS) using Qwen3-VL (8B) on FastBench . Capabilities are ordered from left to right by sample ratio in descending order. Scores are presented in percentage (%). Values in green or red parentheses indicate performance gains or drops relative to the 2 FPS baseline.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: Additional temporal statistics of FastBench. (a) Duration distribution of the 300 video clips. (b) Query timestamp distribution of the 306 QA pairs, measured from the start of each clip. Both panels use 10-second bins; bar labels indicate counts and dashed lines mark the means.

Model	Sports (89)	Video Games (57)	Performing Arts (49)	Animals (39)	Lifestyle & Recreation (26)	Transpor- tation (25)	Science & Technology (11)	Food & Cooking (10)	Overall
Closed-source Models
Doubao-Seed-2.0-Lite	38.4	28.1	20.4	28.2	15.4	41.6	23.6	22.0	29.5
GPT-5.6-Sol	53.7	37.5	44.9	45.1	36.2	48.0	65.5	54.0	46.7
Gemini-3.5-Flash	56.4	43.2	49.8	52.3	46.2	49.6	65.5	38.0	50.7
Open-source Models
Aura (8B)	36.0	29.1	29.0	27.7	40.0	28.0	41.8	26.0	32.1

Appendix

Table A.1: Evaluation results across domains. Scores use a 0–100 scale. Parentheses in the header indicate the number of QA pairs in each domain. Best displayed scores in each column are bolded.

Model	Action & Physical Interaction			Predictive & Causal Reasoning			Motion & Spatiotemporal Tracking
	Fwd.	Inst.	Back.	Fwd.	Inst.	Back.	Fwd.	Inst.	Back.
Closed-source Models
Doubao-Seed-2.0-Lite	26.0	29.5	19.6	51.7	40.0	56.0	12.6	16.0	28.6
GPT-5.6-Sol	41.1	49.5	39.1	55.6	40.0	76.0	50.5	65.3	60.0
Gemini-3.5-Flash	43.8	48.0	39.6	69.4	26.7	76.0	60.0	60.0	77.1
Open-source Models

Appendix

Table A.2: Capability-wise evaluation results by temporal scope. Scores use a 0–100 scale. Fwd., Inst., and Back. denote Forward, Instant, and Backward, respectively. A dash indicates an empty subset. Best displayed scores in each column are bolded.

Panel of Figure 11 — Figure A.2: Qualitative examples from FastBench. Each row is one QA pair. Focus is the annotated timestamp that requires high-frame-rate perception; the query time is when the question is asked.

Explore similar work

Harnessing Streaming Video in the Wild
Jun 7, 2026Dingyu Yao, Shuhuan Gu, Qingyi Si +8VLM Evaluation Streaming Video Understanding
Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China
StreamFlow: Dynamic Memory Flows for Streaming Video Understanding
Aug 11, 2026Muxin Fu, Yifan Zhang, Wentao Zhang +5Memory-Augmented VLMs Streaming Video Understanding
Tongji University · Nanyang Technological University · University of Michigan +2
StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs
Aug 13, 2026Joya Chen, Zeyun Zhong, Mike Zheng ShouMemory-Augmented VLMs Streaming Video Understanding
National University of Singapore · Karlsruhe Institute of Technology

Jun 7, 2026, cs.CVJ/K move · Enter open · S save

Harnessing Streaming Video in the Wild

Dingyu Yao, Shuhuan Gu, Qingyi Si, Junhao Zhou+7

Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

VLM Evaluation Streaming Video Understanding VLM Adaptation

Open paper AskPDF arXiv

Vision-Language Models (VLMs) are increasingly required to process unbounded video streams in applications such as video-call assistants, live commentary, and embodied robots. An ideal streaming system should support proactive interaction, long-horizon memory, and real-time processing, while resting on a VLM backbone capable of handling diverse in-the-wild streaming tasks. However, existing VLMs excel at offline video understanding but fall short in streaming capabilities and lack dedicated infrastructure for streaming deployment. We address this gap on three fronts. (i) For backbone capability, we construct \textbf{Streaming-Train-248K}, a streaming dataset paired with a novel training objective for adapting VLMs to streaming interaction and understanding. (ii) For real-world deployment, we introduce \textbf{Streaming Harness}, a plug-and-play system that endows any VLM with three core abilities: proactive interaction (per-second response decisions), long-term memory (12-hour context retention), and real-time processing (sub-second latency). (iii) To drive continued community progress on streaming capabilities, we design \textbf{Streaming-Eval}, a benchmark that reflects models' capabilities across diverse in-the-wild scenarios. Extensive experiments demonstrate consistent gains from our approach across all core capabilities required for streaming video understanding. We will open-source our data, code, and benchmark to advance the community's shift from offline video understanding to deployable streaming intelligence.

Abstract

Explore similar work

Harnessing Streaming Video in the Wild

StreamFlow: Dynamic Memory Flows for Streaming Video Understanding

StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs