StreamVibe: Proactive Video Agents for Evolving User Intent

Chengzhi Liu; Yuxiang Lai; Yuzhe Yang; Yanqing Liu; Jiheng Hou; Yue Cao; Haonan Wang; Ruixi Fan

cs.CVOct 26, 2026

StreamVibe: Proactive Video Agents for Evolving User Intent

Authors: Chengzhi Liu, Yuxiang Lai, Yuzhe Yang, Yanqing Liu, Jiheng Hou, Yue Cao, Haonan Wang, Ruixi Fan,

Organizations: Google · University of California, Santa Barbara · Emory University · University of California, Santa Cruz ·

Abstract

Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, prevailing methods largely rely on explicit requests while miss the implicit user intent (e.g., "where should I click next?" while a user navigate a website), meanwhile fail to update the state simultaneously as user requests might update continuously. These limitations make them less suited to intent-aware interaction in practical. To address these challenges, we introduce StreamVibe-Bench, a new benchmark serving as testbed for user intent interaction, with 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents. Building on this benchmark, we develop StreamVibe-VL, a streaming proactive model that infers implicit user intents from continuous observations and dynamically tailors its interactions as these intents evolve. We further introduce StreamVibe-Harness, which maintains user intents as persistent states with a fleixble lifecycles via a hierarchical memory to track the latest user intent and environmental state. Our experiments reveal three key findings: (i) StreamVibe-VL preserves strong general video understanding while improving response timing and answer quality by 10.7% over other baselines; (ii) existing proactive models remain limited in understanding user intent and deciding when to respond, especially under implicit and dynamic intents; and (iii) StreamVibe-Harness further improves response consistency and adaptation over sustained interaction, resulting in a 12.3% gain in proactive interaction.

Figures & tables

Figure 1: Overview of StreamVibe. (a) StreamVibe studies sustained proactive interaction under explicit, implicit, and dynamically evolving user intents, where the agent continuously observes the environment and decides when and how to assist. (b) Existing paradigms rely on explicit intents, with limited tracking of interaction state and evolving intents. StreamVibe addresses these limitations through proactive streaming policy learning and persistent interaction state modeling that tracks task progress and updates active intents.

Figure 2: Construction and statistics of StreamVibe-Bench. (a) Data construction from task videos and user context to evidence annotation, human verification, and causal stream generation. (b) Sample allocation across three requirement types, with examples from physical and digital environments. (c) Pointwise and trajectory evaluation settings for local response decisions and continuous interaction behavior. TP: true positives; FP: false positives; FN: false negatives. (d) Joint distribution of video duration, sample count, and average number of evidence points per sample.

Figure 3: Overview of StreamVibe. (a) During inference, StreamVibe-Harness maintains persistent interaction state, tracks each intent through an independent lifecycle, and retrieves relevant history from hierarchical memory, providing StreamVibe-VL with a unified context for response decisions. (b) Adaptive streaming SFT balances wait/speak decisions and jointly supervises timing and content. (c) Localized trajectory RL samples causal rollouts around help points, combining point-level timing and content rewards with trajectory-level rewards on coverage, density, order, consistency, and redundancy.

Model	Explicit						Implicit						Dynamic
	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	FPR $\downarrow$	ANS $\uparrow$	C-F1 $\uparrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	FPR $\downarrow$	ANS $\uparrow$	C-F1 $\uparrow$	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	FPR $\downarrow$	ANS $\uparrow$	C-F1 $\uparrow$
Pointwise Evaluation (gold interaction history)
Proprietary Models
Gemini-3.1-Pro	62.34	68.52	65.28	41.40	45.22	29.52	70.92	85.58	77.56	43.57	67.46	52.32	62.87	84.59	72.13	58.81	51.15	36.89
GPT-Realtime-2.1	57.78	74.09	64.93	64.87	51.10	33.18	69.67	70.95	70.30	29.83	55.92	39.31	62.99	84.47	72.17	55.37	45.64	32.94
Gemini-3.8-Flash	60.34	77.40	67.81	50.69	51.37	34.84	81.55	80.45	81.00	27.09	82.63	66.93	62.96	84.52	72.16	60.25	60.56	43.70

Table 2: Main results on StreamVibe-Bench (

\pm 2

s tolerance). Pointwise evaluation scores each decision given the gold interaction history, whereas trajectory evaluation runs the model over the full stream with its self-generated history. Streaming models do not support pointwise evaluation and are thus reported only under trajectory evaluation. bold and underline mark the best and second-best results within each block.

Figure 4: Generalization to external benchmarks. (a) Macro score on EgoProactive; (b) submetric scores on IPIBench; (c) accuracy on general video benchmarks, with darker cells indicating higher scores within each row.

Figure 5: Diagnostic analysis of proactive agents . (a)-(b) Timing and Content F1 across intent categories and subtypes; (c) Recall versus near-miss FPR across models; (d) Content F1 degradation under accumulated self-generated history; (e) pointwise versus trajectory F1 per category and metric.

Figure 6: Effects of interaction length, policy learning and the harness. (1) Timing F1 versus video duration, and Recall in the early versus late half of the trajectory for each Dynamic Intent subtype. (2) Absolute metric changes of SFT over the base model and of RL over SFT. (3) Gains from adding the Harness to the SFT and RL policies across intent categories. Tolerance: $\tau=2$ s.

Figure 7: Case studies of implicit proactive interaction. Across physical and digital tasks, StreamVibe-VL better aligns response timing and content with the evolving visual evidence and task state, while streaming baselines more often produce premature, missing, or inconsistent assistance.

Appendix figures & tables47 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Explicit						Implicit						Dynamic
	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1
Proprietary Models
Gemini-3.1-Pro	70.92	85.58	77.56	43.57	67.46	52.32	62.34	68.52	65.28	41.40	45.22	29.52	62.87	84.59	72.13	58.81	51.15	36.89
GPT-Realtime-2.1	69.67	70.95	70.30	29.83	55.92	39.31	57.78	90.09	70.41	64.87	51.10	35.98	62.99	84.47	72.17	55.37	45.64	32.94
Gemini-3.8-Flash	81.55	80.45	81.00	27.09	82.63	66.93	60.34	77.40	67.81	50.69	51.37	34.83	62.96	84.52	72.16	60.25	60.56	43.70
Open-Source Offline Models

Appendix

Table 3: Pointwise evaluation with gold interaction history on StreamVibe-Bench without the Harness (

\pm 2

s tolerance).

Model	Explicit						Implicit						Dynamic
	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1
Proprietary Models
Gemini-3.1-Pro	67.47	81.43	73.80	51.13	68.92	50.86	53.46	58.76	55.98	52.96	46.73	26.16	58.03	78.07	66.57	68.27	52.94	35.24
GPT-Realtime-2.1	63.10	64.25	63.67	42.00	57.79	36.79	50.76	79.14	61.85	77.83	51.78	32.03	55.68	74.67	63.79	69.60	46.60	29.73
Gemini-3.8-Flash	79.04	77.97	78.50	31.60	83.41	65.48	53.48	68.61	60.11	61.10	52.45	31.53	59.01	79.21	67.63	67.94	61.25	41.42
Open-Source Offline Models

Appendix

Table 4: Pointwise evaluation with gold interaction history on StreamVibe-Bench without the Harness (

\pm 0

s tolerance).

Model	Explicit						Implicit						Dynamic
	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1
Proprietary Models
Gemini-3.1-Pro	66.28	82.45	73.49	33.17	40.95	30.09	68.23	70.21	69.21	29.31	29.20	20.21	73.03	80.05	76.38	28.13	36.70	28.03
GPT-Realtime-2.1	65.42	72.84	68.93	34.54	54.10	37.29	61.47	85.03	71.36	50.95	48.94	34.92	66.86	79.83	72.77	42.50	41.99	30.56
Gemini-3.8-Flash	89.93	75.22	81.92	5.79	71.88	58.88	79.24	79.49	79.36	13.77	46.90	37.22	85.98	82.75	84.33	7.27	56.27	47.45
Open-Source Offline Models

Appendix

Table 5: Pointwise evaluation with self-generated interaction history on StreamVibe-Bench without the Harness (

\pm 2

s tolerance).

Model	Explicit						Implicit						Dynamic
	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1
Proprietary Models
Gemini-3.1-Pro	54.69	68.03	60.63	59.37	41.25	25.01	45.91	47.24	46.57	56.49	27.20	12.67	50.21	55.03	52.51	64.44	33.65	17.67
GPT-Realtime-2.1	57.03	63.50	60.09	51.52	55.53	33.37	49.81	68.90	57.82	70.04	49.80	28.79	54.90	65.55	59.75	63.23	43.11	25.76
Gemini-3.8-Flash	72.18	60.37	65.75	32.78	73.90	48.59	35.61	35.72	35.66	65.59	49.80	17.76	43.63	42.00	42.80	66.43	58.46	25.02
Open-Source Offline Models

Appendix

Table 6: Pointwise evaluation with self-generated interaction history on StreamVibe-Bench without the Harness (

\pm 0

s tolerance).

Model	Explicit						Implicit						Dynamic
	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1
Proprietary Models
Gemini-3.1-Pro	32.74	79.21	46.33	4.60	85.60	39.66	17.73	58.78	27.24	12.80	52.12	14.20	14.23	57.82	22.84	8.92	64.95	14.83
GPT-Realtime-2.1	9.70	55.56	16.52	11.63	66.84	11.04	14.91	56.26	23.57	15.71	48.21	11.36	9.69	52.67	16.37	14.49	46.86	7.67
Gemini-3.8-Flash	47.69	79.10	59.50	2.37	87.49	52.06	21.03	71.82	32.53	12.94	59.96	19.50	16.00	66.17	25.77	8.42	72.48	18.68
Open-Source Offline Models

Appendix

Table 7: Trajectory evaluation with self-generated interaction history on StreamVibe-Bench without the Harness (

\pm 2

s tolerance).

Model	Explicit						Implicit						Dynamic
	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1
Proprietary Models
Gemini-3.1-Pro	19.21	46.49	27.19	8.43	85.54	23.26	9.01	29.88	13.85	16.13	55.51	7.69	7.74	31.46	12.42	11.71	69.94	8.69
GPT-Realtime-2.1	4.84	27.70	8.24	14.90	71.20	5.87	6.62	24.96	10.46	19.32	52.70	5.51	4.34	23.61	7.33	17.57	51.95	3.81
Gemini-3.8-Flash	27.51	45.63	34.33	6.29	88.34	30.33	10.60	36.20	16.40	17.04	64.64	10.60	8.52	35.23	13.72	11.70	76.23	10.46
Open-Source Offline Models

Appendix

Table 8: Trajectory evaluation with model-generated interaction history on StreamVibe-Bench without the Harness (

\pm 0

s tolerance).

Model	$\tau=2$ s						$\tau=0$ s
	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1
Proprietary Models
Gemini-3.1-Pro	73.78	78.71	76.17	34.17	47.57	36.23	65.06	69.40	67.16	47.97	49.82	33.46
GPT-Realtime-2.1	65.29	82.16	72.76	48.30	40.64	29.57	56.15	70.66	62.57	65.34	41.09	25.71
Gemini-3.8-Flash	75.14	79.58	77.30	35.71	47.59	36.79	68.79	72.85	70.76	45.70	49.06	34.71
Open-Source Offline Models

Appendix

Table 9: Pointwise evaluation with gold interaction history on general timing without the Harness.

Open-Source Offline Models
Model	OVO-Bench
	BT	RT	FAR	Overall
Qwen3.5-9B	53.94	63.94	55.24	57.71
JoyAI-VL-Interaction	47.09	67.99	38.87	51.32
Open-Source Streaming Models
MMDuet2	47.02	61.59	37.91	48.84

Appendix

Table 12: Results on OVO-Bench without the Harness.

Model	Explicit						Implicit						Dynamic
	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1	P	R	F1	FPR	ANS	C-F1
Ours + Harness
StreamVibe-VL (SFT) + Harness	18.49	28.51	22.43	3.80	61.79	13.86	18.20	29.39	22.48	6.33	38.93	8.75	10.01	43.20	16.25	12.25	38.98	6.33
StreamVibe-VL (RL) + Harness	24.57	16.20	19.53	1.49	72.83	14.22	21.53	29.48	24.89	2.02	40.86	10.17	14.80	30.65	19.96	5.68	44.62	8.91
Ours
StreamVibe-VL (SFT)	11.55	40.66	17.99	11.40	57.97	10.43	13.13	47.91	20.61	15.59	38.06	7.84	5.38	56.76	9.83	26.13	35.88	3.53

Appendix

Table 13: Trajectory evaluation of StreamVibe-VL with and without the Harness on StreamVibe-Bench (

\pm 2

s tolerance).

Model	General timing						IPIBench				EgoProactive
	P	R	F1	FPR	ANS	C-F1	Timing	Underst.	Repeated	Macro	Macro	G-mean	Interrupt	Silent
Ours + Harness
StreamVibe-VL (SFT) + Harness	48.39	68.25	56.63	12.57	61.44	34.79	44.13	15.67	38.68	32.83	46.71	43.48	63.75	29.66
StreamVibe-VL (RL) + Harness	53.23	67.22	59.41	6.84	62.88	37.36	45.34	16.88	40.05	35.67	48.79	44.0	3.87	29.80
Ours
StreamVibe-VL (SFT)	22.95	86.92	36.31	49.13	47.87	17.38	34.85	13.40	28.77	25.67	35.74	23.33	62.82	8.66

Appendix

Table 14: StreamVibe-Harness comparison on general timing (Trajectory,

\pm 2

s), IPIBench, and EgoProactive.

Figure 8: Category difficulty across StreamVibe-Bench. Each cell gives the mean score across all models available in that setting; the line spans the per-model range and the dot marks the median. Outlines identify the lowest mean within each intent type and metric. Pointwise uses gold interaction history ( $n{=}7$ models), whereas Trajectory uses model-generated interaction history and also includes native streaming baselines ( $n{=}12$ models). All results use the $\pm 2$ s tolerance.

Figure 9: Absolute subtype profiles of StreamVibe-VL (SFT) and StreamVibe-VL (RL) under Trajectory evaluation. Hollow and filled endpoints show SFT and RL, respectively; the connecting segment exposes both direction and magnitude. Background bands encode the three intent types. Scores are percentage points at

\pm 2

Figure 10: How RL changes the operating point by subtype. Cells report StreamVibe-VL (RL) minus StreamVibe-VL (SFT) in percentage points under Trajectory evaluation. Numbers retain the metric’s natural sign; color encodes benefit, so a negative $\mathrm{FPR}_{\rm near}$ change is blue. The visualization jointly exposes timing, coverage, false interventions, answer quality, and Content-F1.

Category	Subtype	Operational definition	Samples	Evidence	Hours
Explicit	I	One requested event or answer; verified Explicit Intents pool	200	200	7.5
	P	State transition or irreversible point; verified Explicit Intents pool	200	723	8.2
	R	Repeated occurrences of a requested event; verified Explicit Intents pool	200	794	6.4
Implicit	G	Guidance for the next task action	500	3,702	26.7
	L	Learning and explanation	500	3,287	18.4
	M	State monitoring for task progress and changes	500	2,701	18.9

Appendix

Table 15: StreamVibe-Bench target allocation and source coverage. “Evidence” counts proactive gold evidence points; reactive questions are excluded. Implicit and Dynamic statistics are measured on their balanced manifests. Explicit evidence-point and duration statistics are proportional estimates from the qualified I/P/R pool at 200 samples per subtype.

Criterion	Standard for approval
Intent fidelity	The instruction is coherent and compatible with the task. The assigned Explicit, Implicit, or Dynamic category matches the interaction; inferred intents are supported by the user’s context, not invented.
Evidence and causality	Every annotated intervention has identifiable visual or interaction evidence available by its response time. Future frames and later user turns are not needed to justify the response.
Coverage and non-redundancy	Distinct, necessary intervention opportunities are covered, without duplicate evidence points or repeated assistance after the relevant intent has already been satisfied.
Temporal validity	Each response window and preferred timestamp correspond to a moment when the evidence is sufficient and assistance is still useful; the preferred time lies within the window and neither anticipates evidence nor misses the actionable moment.
Reference-answer quality	The reference is correct, relevant, and useful for the active intent. It avoids unsupported details, contradictory advice, and unnecessary instructions, while remaining consistent with the visible state.
Dynamic-intent consistency	For Dynamic samples, timestamped revisions, compositions, and interruptions are coherent. References follow the latest valid intent and preserve unaffected constraints without acting on superseded ones; this criterion is not applicable to other categories.

Appendix

Table 16: Human review rubric. Each reviewer checks every applicable criterion. Acceptance requires four independent approvals of the same sample version, with no unresolved issue.

Audit subset	Candidates	Human–Human Fleiss’ $\kappa\,\uparrow$	Gemini-3.8-Flash–Human Cohen’s $\kappa\,\uparrow$ (mean of four)
Explicit	872	0.87	0.79 (0.76–0.82)
Implicit	2,946	0.84	0.75 (0.72–0.78)
Dynamic	2,187	0.82	0.73 (0.70–0.77)
All candidates	6,005	0.84	0.76 (0.70–0.82)

Appendix

Table 17: Inter-annotator agreement. Human–human agreement is measured by Fleiss’

\kappa

; Gemini–human agreement is the mean of four pairwise Cohen’s

\kappa

coefficients, with ranges in parentheses.

Training component	Recorded source families	Examples	Use
Task-aligned Explicit	Explicit Intents event families	35,000	Streaming SFT
Task-aligned Implicit	Guidance, learning, monitoring, and tool/GUI	35,000	Streaming SFT
General streaming core	JoyAI mixture: Live-WhisperX, Shot2Story, TimeLens, GUI-World, VideoGPT+, Charades-Ego, InternVid, ActivityNet, WebVid, and smaller sources	763,607	Streaming SFT
General streaming auxiliary	TimeChat-Online, VideoChat-Flash, TimeLens, EgoIT, and StreamO-Instruct	4,393	Streaming SFT
Full SFT manifest		838,000	1 FPS streams
Localized RL seed	Implicit / Explicit / general streaming, 5K each	15,000	Local rollout windows

Appendix

Table 18: Training-data provenance. The SFT mixture contains 70K task-aligned examples and 768K general streaming examples. Localized RL uses a separate 15K seed pool, yielding 85K task-oriented training examples in total when combined with task-aligned SFT. Dataset names indicate source families and do not imply a shared annotation protocol.

Evaluation	What is scored	Reported measures	Judge / scorer
Pointwise	Independent probes; supplied history	Precision, recall, F1, $\mathrm{FPR}_{\rm near}$ , ANS, Content-F1	Temporal matcher; Claude Sonnet 4.6 ( Anthropic, 2026 )
Trajectory	Complete 1 FPS rollout; generated history	Same six scores at $\pm 2$ , $\pm 1$ , $\pm 0$ s	Same matcher and judge
General Timing	Streaming transfer cohort	Same timing and content scores	Same protocol
EgoProactive / IPIBench	Session-level interaction	Macro, G-mean, Interrupt, Silent; Timing, Understanding, Repeated	Official protocols
Video suites	MVBench, Video-MME, OVO-Bench, OVBench	Accuracy or official aggregate	Official scorers

Appendix

Table 19: Metrics and judges used in the paper. External benchmarks retain their official evaluation protocols and are not rescored by the StreamVibe-Bench content judge.

Component	Setting
Memory model	Qwen3-VL-4B-Instruct; separate inference process
Memory input	At most 64 frames per call; 262,144 pixels per frame
Chunk and submission schedule	$T_{s}=32$ steps; submission lookahead $=8$ steps
Consolidation and retention	$M=5$ mid-term summaries; $L=8$ long-term blocks
Evidence selection	Up to 3 facts and 3 ordered transitions
Compact state evidence	First rendered evidence line, capped at 180 characters

Appendix

Table 20: Harness configuration. Temporal parameters correspond to the chunked-streaming setting; benchmark-specific native adapters are described separately below. Retrieval limits denote maximum capacities rather than fixed numbers of items injected into the policy context.

Figure 11: Explicit intent, single event (I), example A.

Figure 12: Explicit intent, single event (I), example B.

Figure 13: Explicit intent, state transition (P), example A.

Figure 14: Explicit intent, state transition (P), example B.

Figure 15: Explicit intent, repeated event (R), example A.

Figure 16: Explicit intent, repeated event (R), example B.

Figure 17: Implicit intent, guidance (G), example A.

Figure 18: Implicit intent, guidance (G), example B.

Figure 19: Implicit intent, learning (L), example A.

Figure 20: Implicit intent, learning (L), example B.

Figure 21: Implicit intent, monitoring (M), example A.

Figure 22: Implicit intent, monitoring (M), example B.

Figure 23: Implicit intent, tool / GUI (T), example A.

Figure 24: Implicit intent, tool / GUI (T), example B.

Figure 25: Dynamic intent, revision (RV), example A.

Figure 26: Dynamic intent, revision (RV), example B.

Figure 27: Dynamic intent, composition (CP), example A.

Figure 28: Dynamic intent, composition (CP), example B.

Figure 29: Dynamic intent, interruption (IT), example A.

Figure 30: Dynamic intent, interruption (IT), example B.

Figure 31: Additional case study 01. Drawing and saving shapes in GIMP. Model responses at two annotated evidence points, matched within $\pm 2$ s.

Figure 32: Additional case study 02. Editing and exporting a video in Filmora. Model responses at two annotated evidence points, matched within $\pm 2$ s.

Figure 33: Additional case study 03. Setting a French-learning reminder. Model responses at two annotated evidence points, matched within $\pm 2$ s.

Figure 34: Additional case study 04. Finding and selecting jeans in a shopping app. Model responses at two annotated evidence points, matched within $\pm 2$ s.

Figure 35: Additional case study 05. Making capsule coffee. Model responses at two annotated evidence points, matched within $\pm 2$ s.

Figure 36: Additional case study 06. Assembling a metal shelf. Model responses at two annotated evidence points, matched within $\pm 2$ s.

Figure 37: Additional case study 07. Browning butter without scorching. Model responses at two annotated evidence points, matched within $\pm 2$ s.

Figure 38: Additional case study 08. Making successive creases for a paper butterfly. Model responses at two annotated evidence points, matched within $\pm 2$ s.

Vision-Language Agents

Vision-language agents use visual and language models to perceive environments, reason about goals, and take actions through interactive interfaces.

See all 45 papers

LiveProBench: Can Streaming Video Models Really Interact Like Humans?
Sep 11, 2026Kaixuan Du, Xin Wan, Hang Zhang +4Streaming Video Understanding Video-Language Model Evaluation
School of Automation Science and Electrical Engineering, Beihang University · Qwen Business Unit of Alibaba
EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams
May 8, 2026Dongchuan Ran, Linyu Ou, Xueheng Li +5Egocentric Video Understanding AI Agent Benchmarks
Sensetime Reasearch · Beijing Institute of Technology · University of Science and Technology of China
From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios
Oct 15, 2026Boyu Cui, Junhao Luo, Zhihao Wang +5Video-Language Model Evaluation Video-Language Models
Fudan University · The Chinese University of Hong Kong · Shanghai Innovation Institute

Sep 11, 2026J/K move · Enter open · S save

LiveProBench: Can Streaming Video Models Really Interact Like Humans?

Kaixuan Du, Xin Wan, Hang Zhang, Meng Cao+3

School of Automation Science and Electrical Engineering, Beihang University · Qwen Business Unit of Alibaba

Streaming Video Understanding Video-Language Model Evaluation Temporal Video Understanding

Open paper AskPDF arXiv

Streaming video understanding requires models to process continuous multimodal input while maintaining temporal context. Existing evaluations are predominantly reactive: they query a model at a selected timestamp and therefore do not assess when it should respond. Proactive interaction instead requires monitoring a standing request, responding within an appropriate interval after the target event, and otherwise remaining silent. We introduce LiveProBench, which evaluates models at one-second stream intervals without an explicit response cue. Its six subtasks vary trigger ambiguity and timing tolerance. Event Sensitivity geometrically combines response and silence rates on the same recording; four window-based subtasks distinguish early, in-window, and missed responses; and Duplicate Counting penalizes omissions and repetitions. Premature responses outnumber missed responses for half of the evaluated models, revealing a substantial gap in the temporal decision-making required for human-like interaction.

Abstract

Vision-Language Agents

Explore similar work

LiveProBench: Can Streaming Video Models Really Interact Like Humans?

EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams

From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios