Organizations: Google · University of California, Santa Barbara · Emory University · University of California, Santa Cruz · Columbia University · Stanford University · University of Oxford
Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, prevailing methods largely rely on explicit requests while miss the implicit user intent (e.g., "where should I click next?" while a user navigate a website), meanwhile fail to update the state simultaneously as user requests might update continuously. These limitations make them less suited to intent-aware interaction in practical. To address these challenges, we introduce StreamVibe-Bench, a new benchmark serving as testbed for user intent interaction, with 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents. Building on this benchmark, we develop StreamVibe-VL, a streaming proactive model that infers implicit user intents from continuous observations and dynamically tailors its interactions as these intents evolve. We further introduce StreamVibe-Harness, which maintains user intents as persistent states with a fleixble lifecycles via a hierarchical memory to track the latest user intent and environmental state. Our experiments reveal three key findings: (i) StreamVibe-VL preserves strong general video understanding while improving response timing and answer quality by 10.7% over other baselines; (ii) existing proactive models remain limited in understanding user intent and deciding when to respond, especially under implicit and dynamic intents; and (iii) StreamVibe-Harness further improves response consistency and adaptation over sustained interaction, resulting in a 12.3% gain in proactive interaction.
Figures & tables
Figure 1: Overview of StreamVibe. (a) StreamVibe studies sustained proactive interaction under explicit, implicit, and dynamically evolving user intents, where the agent continuously observes the environment and decides when and how to assist. (b) Existing paradigms rely on explicit intents, with limited tracking of interaction state and evolving intents. StreamVibe addresses these limitations through proactive streaming policy learning and persistent interaction state modeling that tracks task progress and updates active intents.
Figure 2: Construction and statistics of StreamVibe-Bench. (a) Data construction from task videos and user context to evidence annotation, human verification, and causal stream generation. (b) Sample allocation across three requirement types, with examples from physical and digital environments. (c) Pointwise and trajectory evaluation settings for local response decisions and continuous interaction behavior. TP: true positives; FP: false positives; FN: false negatives. (d) Joint distribution of video duration, sample count, and average number of evidence points per sample.
Figure 3: Overview of StreamVibe. (a) During inference, StreamVibe-Harness maintains persistent interaction state, tracks each intent through an independent lifecycle, and retrieves relevant history from hierarchical memory, providing StreamVibe-VL with a unified context for response decisions. (b) Adaptive streaming SFT balances wait/speak decisions and jointly supervises timing and content. (c) Localized trajectory RL samples causal rollouts around help points, combining point-level timing and content rewards with trajectory-level rewards on coverage, density, order, consistency, and redundancy.
Model
Explicit
Implicit
Dynamic
P ↑
R ↑
F1 ↑
FPR ↓
ANS ↑
C-F1 ↑
P ↑
R ↑
F1 ↑
FPR ↓
ANS ↑
C-F1 ↑
P ↑
R ↑
F1 ↑
FPR ↓
ANS ↑
C-F1 ↑
Pointwise Evaluation (gold interaction history)
Proprietary Models
Gemini-3.1-Pro
62.34
68.52
65.28
41.40
45.22
29.52
70.92
85.58
77.56
43.57
67.46
52.32
62.87
84.59
72.13
58.81
51.15
36.89
GPT-Realtime-2.1
57.78
74.09
64.93
64.87
51.10
33.18
69.67
70.95
70.30
29.83
55.92
39.31
62.99
84.47
72.17
55.37
45.64
32.94
Gemini-3.8-Flash
60.34
77.40
67.81
50.69
51.37
34.84
81.55
80.45
81.00
27.09
82.63
66.93
62.96
84.52
72.16
60.25
60.56
43.70
Table 2: Main results on StreamVibe-Bench ( ±2 s tolerance). Pointwise evaluation scores each decision given the gold interaction history, whereas trajectory evaluation runs the model over the full stream with its self-generated history. Streaming models do not support pointwise evaluation and are thus reported only under trajectory evaluation. bold and underline mark the best and second-best results within each block.
Figure 4: Generalization to external benchmarks. (a) Macro score on EgoProactive; (b) submetric scores on IPIBench; (c) accuracy on general video benchmarks, with darker cells indicating higher scores within each row.
Figure 5: Diagnostic analysis of proactive agents . (a)-(b) Timing and Content F1 across intent categories and subtypes; (c) Recall versus near-miss FPR across models; (d) Content F1 degradation under accumulated self-generated history; (e) pointwise versus trajectory F1 per category and metric.
Figure 6: Effects of interaction length, policy learning and the harness. (1) Timing F1 versus video duration, and Recall in the early versus late half of the trajectory for each Dynamic Intent subtype. (2) Absolute metric changes of SFT over the base model and of RL over SFT. (3) Gains from adding the Harness to the SFT and RL policies across intent categories. Tolerance: τ=2 s.
Figure 7: Case studies of implicit proactive interaction. Across physical and digital tasks, StreamVibe-VL better aligns response timing and content with the evolving visual evidence and task state, while streaming baselines more often produce premature, missing, or inconsistent assistance.
Appendix figures & tables47 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Explicit
Implicit
Dynamic
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
Proprietary Models
Gemini-3.1-Pro
70.92
85.58
77.56
43.57
67.46
52.32
62.34
68.52
65.28
41.40
45.22
29.52
62.87
84.59
72.13
58.81
51.15
36.89
GPT-Realtime-2.1
69.67
70.95
70.30
29.83
55.92
39.31
57.78
90.09
70.41
64.87
51.10
35.98
62.99
84.47
72.17
55.37
45.64
32.94
Gemini-3.8-Flash
81.55
80.45
81.00
27.09
82.63
66.93
60.34
77.40
67.81
50.69
51.37
34.83
62.96
84.52
72.16
60.25
60.56
43.70
Open-Source Offline Models
Appendix
Table 3: Pointwise evaluation with gold interaction history on StreamVibe-Bench without the Harness ( ±2 s tolerance).
Model
Explicit
Implicit
Dynamic
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
Proprietary Models
Gemini-3.1-Pro
67.47
81.43
73.80
51.13
68.92
50.86
53.46
58.76
55.98
52.96
46.73
26.16
58.03
78.07
66.57
68.27
52.94
35.24
GPT-Realtime-2.1
63.10
64.25
63.67
42.00
57.79
36.79
50.76
79.14
61.85
77.83
51.78
32.03
55.68
74.67
63.79
69.60
46.60
29.73
Gemini-3.8-Flash
79.04
77.97
78.50
31.60
83.41
65.48
53.48
68.61
60.11
61.10
52.45
31.53
59.01
79.21
67.63
67.94
61.25
41.42
Open-Source Offline Models
Appendix
Table 4: Pointwise evaluation with gold interaction history on StreamVibe-Bench without the Harness ( ±0 s tolerance).
Model
Explicit
Implicit
Dynamic
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
Proprietary Models
Gemini-3.1-Pro
66.28
82.45
73.49
33.17
40.95
30.09
68.23
70.21
69.21
29.31
29.20
20.21
73.03
80.05
76.38
28.13
36.70
28.03
GPT-Realtime-2.1
65.42
72.84
68.93
34.54
54.10
37.29
61.47
85.03
71.36
50.95
48.94
34.92
66.86
79.83
72.77
42.50
41.99
30.56
Gemini-3.8-Flash
89.93
75.22
81.92
5.79
71.88
58.88
79.24
79.49
79.36
13.77
46.90
37.22
85.98
82.75
84.33
7.27
56.27
47.45
Open-Source Offline Models
Appendix
Table 5: Pointwise evaluation with self-generated interaction history on StreamVibe-Bench without the Harness ( ±2 s tolerance).
Model
Explicit
Implicit
Dynamic
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
Proprietary Models
Gemini-3.1-Pro
54.69
68.03
60.63
59.37
41.25
25.01
45.91
47.24
46.57
56.49
27.20
12.67
50.21
55.03
52.51
64.44
33.65
17.67
GPT-Realtime-2.1
57.03
63.50
60.09
51.52
55.53
33.37
49.81
68.90
57.82
70.04
49.80
28.79
54.90
65.55
59.75
63.23
43.11
25.76
Gemini-3.8-Flash
72.18
60.37
65.75
32.78
73.90
48.59
35.61
35.72
35.66
65.59
49.80
17.76
43.63
42.00
42.80
66.43
58.46
25.02
Open-Source Offline Models
Appendix
Table 6: Pointwise evaluation with self-generated interaction history on StreamVibe-Bench without the Harness ( ±0 s tolerance).
Model
Explicit
Implicit
Dynamic
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
Proprietary Models
Gemini-3.1-Pro
32.74
79.21
46.33
4.60
85.60
39.66
17.73
58.78
27.24
12.80
52.12
14.20
14.23
57.82
22.84
8.92
64.95
14.83
GPT-Realtime-2.1
9.70
55.56
16.52
11.63
66.84
11.04
14.91
56.26
23.57
15.71
48.21
11.36
9.69
52.67
16.37
14.49
46.86
7.67
Gemini-3.8-Flash
47.69
79.10
59.50
2.37
87.49
52.06
21.03
71.82
32.53
12.94
59.96
19.50
16.00
66.17
25.77
8.42
72.48
18.68
Open-Source Offline Models
Appendix
Table 7: Trajectory evaluation with self-generated interaction history on StreamVibe-Bench without the Harness ( ±2 s tolerance).
Model
Explicit
Implicit
Dynamic
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
Proprietary Models
Gemini-3.1-Pro
19.21
46.49
27.19
8.43
85.54
23.26
9.01
29.88
13.85
16.13
55.51
7.69
7.74
31.46
12.42
11.71
69.94
8.69
GPT-Realtime-2.1
4.84
27.70
8.24
14.90
71.20
5.87
6.62
24.96
10.46
19.32
52.70
5.51
4.34
23.61
7.33
17.57
51.95
3.81
Gemini-3.8-Flash
27.51
45.63
34.33
6.29
88.34
30.33
10.60
36.20
16.40
17.04
64.64
10.60
8.52
35.23
13.72
11.70
76.23
10.46
Open-Source Offline Models
Appendix
Table 8: Trajectory evaluation with model-generated interaction history on StreamVibe-Bench without the Harness ( ±0 s tolerance).
Model
τ=2 s
τ=0 s
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
Proprietary Models
Gemini-3.1-Pro
73.78
78.71
76.17
34.17
47.57
36.23
65.06
69.40
67.16
47.97
49.82
33.46
GPT-Realtime-2.1
65.29
82.16
72.76
48.30
40.64
29.57
56.15
70.66
62.57
65.34
41.09
25.71
Gemini-3.8-Flash
75.14
79.58
77.30
35.71
47.59
36.79
68.79
72.85
70.76
45.70
49.06
34.71
Open-Source Offline Models
Appendix
Table 9: Pointwise evaluation with gold interaction history on general timing without the Harness.
Model
OVO-Bench
BT
RT
FAR
Overall
Open-Source Offline Models
Qwen3.5-9B
53.94
63.94
55.24
57.71
JoyAI-VL-Interaction
47.09
67.99
38.87
51.32
Open-Source Streaming Models
MMDuet2
47.02
61.59
37.91
48.84
Appendix
Table 12: Results on OVO-Bench without the Harness.
Model
Explicit
Implicit
Dynamic
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
P
R
F1
FPR
ANS
C-F1
Ours + Harness
StreamVibe-VL (SFT) + Harness
18.49
28.51
22.43
3.80
61.79
13.86
18.20
29.39
22.48
6.33
38.93
8.75
10.01
43.20
16.25
12.25
38.98
6.33
StreamVibe-VL (RL) + Harness
24.57
16.20
19.53
1.49
72.83
14.22
21.53
29.48
24.89
2.02
40.86
10.17
14.80
30.65
19.96
5.68
44.62
8.91
Ours
StreamVibe-VL (SFT)
11.55
40.66
17.99
11.40
57.97
10.43
13.13
47.91
20.61
15.59
38.06
7.84
5.38
56.76
9.83
26.13
35.88
3.53
Appendix
Table 13: Trajectory evaluation of StreamVibe-VL with and without the Harness on StreamVibe-Bench ( ±2 s tolerance).
Model
General timing
IPIBench
EgoProactive
P
R
F1
FPR
ANS
C-F1
Timing
Underst.
Repeated
Macro
Macro
G-mean
Interrupt
Silent
Ours + Harness
StreamVibe-VL (SFT) + Harness
48.39
68.25
56.63
12.57
61.44
34.79
44.13
15.67
38.68
32.83
46.71
43.48
63.75
29.66
StreamVibe-VL (RL) + Harness
53.23
67.22
59.41
6.84
62.88
37.36
45.34
16.88
40.05
35.67
48.79
44.0
3.87
29.80
Ours
StreamVibe-VL (SFT)
22.95
86.92
36.31
49.13
47.87
17.38
34.85
13.40
28.77
25.67
35.74
23.33
62.82
8.66
Appendix
Table 14: StreamVibe-Harness comparison on general timing (Trajectory, ±2 s), IPIBench, and EgoProactive.
Figure 8: Category difficulty across StreamVibe-Bench. Each cell gives the mean score across all models available in that setting; the line spans the per-model range and the dot marks the median. Outlines identify the lowest mean within each intent type and metric. Pointwise uses gold interaction history ( n=7 models), whereas Trajectory uses model-generated interaction history and also includes native streaming baselines ( n=12 models). All results use the ±2 s tolerance.
Figure 9: Absolute subtype profiles of StreamVibe-VL (SFT) and StreamVibe-VL (RL) under Trajectory evaluation. Hollow and filled endpoints show SFT and RL, respectively; the connecting segment exposes both direction and magnitude. Background bands encode the three intent types. Scores are percentage points at ±2 s.
Figure 10: How RL changes the operating point by subtype. Cells report StreamVibe-VL (RL) minus StreamVibe-VL (SFT) in percentage points under Trajectory evaluation. Numbers retain the metric’s natural sign; color encodes benefit, so a negative FPRnear change is blue. The visualization jointly exposes timing, coverage, false interventions, answer quality, and Content-F1.
Category
Subtype
Operational definition
Samples
Evidence
Hours
Explicit
I
One requested event or answer; verified Explicit Intents pool
200
200
7.5
P
State transition or irreversible point; verified Explicit Intents pool
200
723
8.2
R
Repeated occurrences of a requested event; verified Explicit Intents pool
200
794
6.4
Implicit
G
Guidance for the next task action
500
3,702
26.7
L
Learning and explanation
500
3,287
18.4
M
State monitoring for task progress and changes
500
2,701
18.9
Appendix
Table 15: StreamVibe-Bench target allocation and source coverage. “Evidence” counts proactive gold evidence points; reactive questions are excluded. Implicit and Dynamic statistics are measured on their balanced manifests. Explicit evidence-point and duration statistics are proportional estimates from the qualified I/P/R pool at 200 samples per subtype.
Criterion
Standard for approval
Intent fidelity
The instruction is coherent and compatible with the task. The assigned Explicit, Implicit, or Dynamic category matches the interaction; inferred intents are supported by the user’s context, not invented.
Evidence and causality
Every annotated intervention has identifiable visual or interaction evidence available by its response time. Future frames and later user turns are not needed to justify the response.
Coverage and non-redundancy
Distinct, necessary intervention opportunities are covered, without duplicate evidence points or repeated assistance after the relevant intent has already been satisfied.
Temporal validity
Each response window and preferred timestamp correspond to a moment when the evidence is sufficient and assistance is still useful; the preferred time lies within the window and neither anticipates evidence nor misses the actionable moment.
Reference-answer quality
The reference is correct, relevant, and useful for the active intent. It avoids unsupported details, contradictory advice, and unnecessary instructions, while remaining consistent with the visible state.
Dynamic-intent consistency
For Dynamic samples, timestamped revisions, compositions, and interruptions are coherent. References follow the latest valid intent and preserve unaffected constraints without acting on superseded ones; this criterion is not applicable to other categories.
Appendix
Table 16: Human review rubric. Each reviewer checks every applicable criterion. Acceptance requires four independent approvals of the same sample version, with no unresolved issue.
Audit subset
Candidates
Human–Human Fleiss’ κ↑
Gemini-3.8-Flash–Human Cohen’s κ↑ (mean of four)
Explicit
872
0.87
0.79 (0.76–0.82)
Implicit
2,946
0.84
0.75 (0.72–0.78)
Dynamic
2,187
0.82
0.73 (0.70–0.77)
All candidates
6,005
0.84
0.76 (0.70–0.82)
Appendix
Table 17: Inter-annotator agreement. Human–human agreement is measured by Fleiss’ κ ; Gemini–human agreement is the mean of four pairwise Cohen’s κ coefficients, with ranges in parentheses.
TimeChat-Online, VideoChat-Flash, TimeLens, EgoIT, and StreamO-Instruct
4,393
Streaming SFT
Full SFT manifest
838,000
1 FPS streams
Localized RL seed
Implicit / Explicit / general streaming, 5K each
15,000
Local rollout windows
Appendix
Table 18: Training-data provenance. The SFT mixture contains 70K task-aligned examples and 768K general streaming examples. Localized RL uses a separate 15K seed pool, yielding 85K task-oriented training examples in total when combined with task-aligned SFT. Dataset names indicate source families and do not imply a shared annotation protocol.
Evaluation
What is scored
Reported measures
Judge / scorer
Pointwise
Independent probes; supplied history
Precision, recall, F1, FPRnear , ANS, Content-F1
Temporal matcher; Claude Sonnet 4.6 ( Anthropic, 2026 )
Table 19: Metrics and judges used in the paper. External benchmarks retain their official evaluation protocols and are not rescored by the StreamVibe-Bench content judge.
Component
Setting
Memory model
Qwen3-VL-4B-Instruct; separate inference process
Memory input
At most 64 frames per call; 262,144 pixels per frame
Chunk and submission schedule
Ts=32 steps; submission lookahead =8 steps
Consolidation and retention
M=5 mid-term summaries; L=8 long-term blocks
Evidence selection
Up to 3 facts and 3 ordered transitions
Compact state evidence
First rendered evidence line, capped at 180 characters
Appendix
Table 20: Harness configuration. Temporal parameters correspond to the chunked-streaming setting; benchmark-specific native adapters are described separately below. Retrieval limits denote maximum capacities rather than fixed numbers of items injected into the policy context.
Figure 11: Explicit intent, single event (I), example A.
Figure 12: Explicit intent, single event (I), example B.
Figure 13: Explicit intent, state transition (P), example A.
Figure 14: Explicit intent, state transition (P), example B.
Figure 15: Explicit intent, repeated event (R), example A.
Figure 16: Explicit intent, repeated event (R), example B.
Figure 17: Implicit intent, guidance (G), example A.
Figure 18: Implicit intent, guidance (G), example B.
Figure 19: Implicit intent, learning (L), example A.
Figure 20: Implicit intent, learning (L), example B.
Figure 21: Implicit intent, monitoring (M), example A.
Figure 22: Implicit intent, monitoring (M), example B.
Figure 23: Implicit intent, tool / GUI (T), example A.
Figure 24: Implicit intent, tool / GUI (T), example B.
Figure 25: Dynamic intent, revision (RV), example A.
Figure 26: Dynamic intent, revision (RV), example B.
Figure 27: Dynamic intent, composition (CP), example A.
Figure 28: Dynamic intent, composition (CP), example B.
Figure 29: Dynamic intent, interruption (IT), example A.
Figure 30: Dynamic intent, interruption (IT), example B.
Figure 31: Additional case study 01. Drawing and saving shapes in GIMP. Model responses at two annotated evidence points, matched within ±2 s.
Figure 32: Additional case study 02. Editing and exporting a video in Filmora. Model responses at two annotated evidence points, matched within ±2 s.
Figure 33: Additional case study 03. Setting a French-learning reminder. Model responses at two annotated evidence points, matched within ±2 s.
Figure 34: Additional case study 04. Finding and selecting jeans in a shopping app. Model responses at two annotated evidence points, matched within ±2 s.
Figure 35: Additional case study 05. Making capsule coffee. Model responses at two annotated evidence points, matched within ±2 s.
Figure 36: Additional case study 06. Assembling a metal shelf. Model responses at two annotated evidence points, matched within ±2 s.
Figure 37: Additional case study 07. Browning butter without scorching. Model responses at two annotated evidence points, matched within ±2 s.
Figure 38: Additional case study 08. Making successive creases for a paper butterfly. Model responses at two annotated evidence points, matched within ±2 s.
Streaming video understanding requires models to process continuous multimodal input while maintaining temporal context. Existing evaluations are predominantly reactive: they query a model at a selected timestamp and therefore do not assess when it should respond. Proactive interaction instead requires monitoring a standing request, responding within an appropriate interval after the target event, and otherwise remaining silent. We introduce LiveProBench, which evaluates models at one-second stream intervals without an explicit response cue. Its six subtasks vary trigger ambiguity and timing tolerance. Event Sensitivity geometrically combines response and silence rates on the same recording; four window-based subtasks distinguish early, in-window, and missed responses; and Duplicate Counting penalizes omissions and repetitions. Premature responses outnumber missed responses for half of the evaluated models, revealing a substantial gap in the temporal decision-making required for human-like interaction.
Kaixuan Du, Xin Wan, Hang Zhang +4
School of Automation Science and Electrical Engineering, Beihang University · Qwen Business Unit of Alibaba
Existing Multimodal Large Language Models (MLLMs) remain primarily reactive, failing to continuously perceive environments or proactively assist users. While emerging benchmarks address proactivity, they are largely confined to alert scenarios, neglect personalized context, and fail to evaluate the precise timing of human-machine interactions (HMI).In this paper, we introduce EgoPro-Bench, a novel benchmark for training and evaluating proactive interaction capabilities based on streaming egocentric videos; it comprises 2,400 videos in the evaluation set and over 12,000 videos in the training set.Unlike previous works, EgoPro-Bench leverages simulated user profiles to generate diverse user intentions and to construct high-fidelity HMI data across 12 distinct domains.Subsequently, we propose a specialized evaluation protocol and metrics, train proactive interaction models designed for efficient reasoning and low-latency interaction on streaming video data, and conduct comprehensive evaluations.Furthermore, we introduce an interaction principle termed "short thinking, better interaction", which allocates a limited token budget prior to intent recognition, thereby enhancing interaction performance.The experiments demonstrate that EgoPro-Bench substantially enhances the intention understanding capabilities of MLLMs and enables accurate identification of appropriate timings for HMI, thereby laying a solid foundation for next-generation user-centric proactive interactive agents.
Dongchuan Ran, Linyu Ou, Xueheng Li +5
Sensetime Reasearch · Beijing Institute of Technology · University of Science and Technology of China
Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed visual content. In real-world scenarios, assistants must additionally anticipate what assistance is needed and decide when to provide it. To address this gap, we introduce \textsc{P3IBench}, a benchmark for proactive video understanding in real-life streaming scenarios. It evaluates three complementary capabilities: daily perception, risk prediction and intervention. We benchmark 12 representative open-source and proprietary video-language models, including streaming and adapted non-streaming systems. Our evaluation shows that current models struggle to make well-calibrated response decisions. Finally, we construct additional proactive training data and train \textsc{P3I-Assistant-4B}, achieving improvements on several tasks, though gains remain uneven across capabilities. Together, these findings show that effective proactive assistance requires not only visual understanding, but also precise response timing and appropriate restraint.
Boyu Cui, Junhao Luo, Zhihao Wang +5
Fudan University · The Chinese University of Hong Kong · Shanghai Innovation Institute