Most AI assistants respond only when prompted, remaining silent even when events in the video stream offer meaningful opportunities for conversation. To move beyond purely user-driven interaction, we present AURA 2.0, a streaming video large language model that introduces three core upgrades to AURA. The key advance is Model-Initiated Conversation (MIC), which enables the model to identify topics worth discussing in the evolving visual stream and initiate conversations at appropriate moments without a user query. Stream-Aware Tool Calling (SATC) enables the model to invoke external tools using the current visual and conversational context and incorporate returned information into the ongoing interaction. Interactive Visual Grounding (IVG) supports visual grounding in both user inputs and model outputs. The model can identify objects marked by users on video frames and produce frame-aligned bounding boxes to link its responses to specific objects and regions. To develop these capabilities, we construct a dedicated streaming interaction dataset and adopt a three-stage training scheme to integrate the new behaviors with AURA's existing streaming interaction abilities. We further incorporate Long-Horizon Interaction Memory (LHIM) to maintain contextual continuity over extended video streams. Evaluations on public streaming benchmarks show that AURA 2.0 achieves leading performance among open-source streaming models, while targeted evaluations further validate its new interaction capabilities. We open-source the model weights and inference code.
Figures & tables
Figure 1: Comparison of AURA 2.0 with MOSS-VL-Realtime and JoyAI-VL-Interaction across four public streaming benchmarks. The Tool use score for MOSS-VL-Realtime on StreamArena is plotted as zero because the model does not support tool calling.
Model
Real-Time QA
Proactive QA
Multi-Response QA
MIC
SATC
IVG
AURA ( Lu et al., 2026a )
✓
✓
✓
✗
✗
✗
MMDuet2 ( Wang et al., 2025c )
✓
✓
✓
✗
✗
✗
MOSS-VL-Realtime ( Wang et al., 2026 )
✓
✓
✓
✗
✗
✗
JoyAI-VL-Interaction ( Yao et al., 2026b )
✓
✓
✓
✗
✓
✗
MiniCPM-o 4.5 ( Cui et al., 2026 )
✓
✓
✓
✗
✗
✗
AURA 2.0 (Ours)
✓
✓
✓
✓
✓
✓
Table 1: Comparison of representative streaming interaction models. A check mark (✓) indicates a publicly reported capability, while a cross (✗) indicates that supporting evidence was not identified, rather than a confirmed lack of capability.
Figure 2: Overview of the AURA 2.0 streaming interaction framework. The model incrementally processes video chunks together with user inputs and interaction history, producing either <|silent|> for silent observation or <|respond|> followed by a text response or tool call. Tool results are incorporated into the context to support subsequent responses. Dashed outlines indicate historical messages beyond the active visual window.
Figure 3: Data construction pipeline for AURA 2.0. The pipeline transforms collected videos into a verified streaming training mixture through three stages: (1) Video collection and preparation standardizes videos into clips and generates clip-level descriptions. (2) Task-specific data synthesis uses the clips and descriptions to construct General QA, MIC, SATC, and IVG trajectories. (3) Streaming assembly and quality assessment aligns the trajectories to video chunks and filters invalid or low-quality samples.
Figure 4: Training data composition and response-control statistics across the three training stages. (a) Sample distribution by stage (inner ring) and task (outer ring). (b) Stage-wise sample and token allocations and silent/response decision proportions. (c) Task-wise sample counts and shares aggregated across stages. Together, these distributions characterize the task coverage and supervision balance of the training mixture.
Model
Proactive Monitoring
Proactive Task Management
Interleaved Reactive–Proactive
Timing
Under.
Repeat.
Avg.
Cancel
Modify
Multi
Avg.
R2P
RuP
RaP
Avg.
Human Level
98.00
94.00
92.00
94.67
98.00
92.00
98.00
96.00
96.00
96.00
98.00
96.67
Proprietary Offline Models
Gemini 3 Pro
43.49
24.90
18.40
28.93
19.44
21.84
19.39
20.22
24.76
17.68
26.69
23.04
Gemini 2.5 Pro
45.36
25.26
13.21
27.94
24.54
20.81
16.78
20.71
21.17
17.54
28.24
22.32
GPT-5.4
54.64
23.52
8.96
29.04
38.43
24.50
7.57
23.50
27.04
15.06
29.28
23.79
Table 2: Main results on IPIBench. Results for previously evaluated models are reproduced from the original IPIBench paper, while results for AURA 2.0, JoyAI-VL-Interaction, MOSS-VL-Realtime, and Qwen3.5-9B (with thinking disabled and enabled) are obtained from our evaluations. All scores are reported as accuracies (%), and Avg. denotes the arithmetic mean of the three subtask scores within each category. Boldface indicates the best result among open-source streaming models in each metric column.
Model
RTP ↑
HR ↑
Tool ↑
Pro. ↑
Avg.
L1
L2
L3
L4
Avg.
L1
L2
L3
Human
91.8
80.7
84.2
82.1
79.5
75.8
95.2
91.5
94.6
91.3
83.4
63.4
78.6
68.9
56.4
45.1
Proprietary Offline Models
Gemini 3.5 Flash
51.3
51.4
45.8
57.8
51.4
45.3
70.8
–
–
–
–
Qwen3.5-Omni
41.8
35.8
31.5
44.8
34.1
25.4
49.4
–
–
–
–
Table 3: Main results on StreamArena. We report the performance of AURA 2.0 and representative baselines across four evaluation dimensions: RTP, HR, Tool, and Pro. Results for previously evaluated models, including the human reference, are reproduced from the original StreamArena paper, while results for AURA 2.0, JoyAI-VL-Interaction, and MOSS-VL-Realtime are obtained from our evaluations. Human HR results are reported with video rewatching in the upper row and without rewatching in the lower row. All scores are reported as accuracies (%), with higher values indicating better performance. Boldface indicates the best result among open-source streaming models in each metric column.
Model
WEB
EGO
TV
VAD
ω=
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Human
36.5
38.6
40.7
35.0
38.2
41.3
38.1
47.0
55.9
47.4
53.6
59.8
Proprietary Offline Models
GPT-4.1
44.6
51.7
58.9
53.6
58.8
64.0
45.0
56.8
68.5
40.8
46.2
51.6
GPT-4.1-mini
41.2
47.8
54.5
59.1
65.8
72.5
48.5
59.4
70.3
41.4
47.7
54.0
Gemini-1.5-pro
37.1
42.1
47.0
47.0
49.7
52.4
41.7
52.0
62.4
34.3
36.1
37.9
Table 4: Main results on ProactiveVideoQA. We report the performance of AURA 2.0 and representative baselines across four categories: WEB, EGO, TV, and VAD. The published baseline results, including the human reference, are reproduced from the original ProactiveVideoQA paper. Results for JoyAI-VL-Interaction and AURA 2.0 are obtained from our evaluations, while the MOSS-VL-Realtime results are taken from its official technical report. All scores are reported using Proactive Area Under Curve (PAUC), where higher values indicate better performance. Boldface indicates the best result among open-source streaming models in each metric column.
Model
Evaluation Score ( ↑ )
Overall ( ↑ )
Forward Task Analysis (%)
Instant
Backward
Forward
ER ( ↓ )
NR ( ↓ )
PC ( ↑ )
Proprietary Offline Models
Gemini 3 Pro
80.83
82.19
16.40
48.70
79.12
0.11
20.77
Doubao-Seed-1.6
71.28
62.94
44.26
56.01
29.76
13.38
56.85
Doubao-Seed-1.8
80.45
77.31
33.38
56.15
56.46
2.36
41.18
Open-source Offline Models
Table 5: Main results on PhoStream. We report the performance of AURA 2.0 and representative baselines across three task categories: Instant, Backward, and Forward. Published baseline results are reproduced from the original PhoStream paper, while results for AURA 2.0, JoyAI-VL-Interaction, MOSS-VL-Realtime, and Qwen3.5-9B are obtained from our evaluations. Evaluation scores are reported on a 0–100 scale, and Overall denotes the reported aggregate score. Boldface indicates the best result among open-source streaming models in each metric column, with lower values preferred for ER and NR and higher values preferred for the other metrics; arrows indicate the preferred direction.
Figure 5: MIC example in the Visual-Dialogue Context setting. Before playback, the user mentions wanting to get home early despite heavy traffic, and the assistant advises caution around big trucks. Without any further user query, AURA 2.0 initiates two messages: at t=66 s it warns about a large Amazon truck ahead, and at t=78 s it advises keeping a steady speed and a safe distance as traffic tightens. Both messages build on the user’s expressed concern and the assistant’s earlier advice. Each frame is shown one second before the corresponding message onset.
Setting
Evaluated over
Trigger rate (%)
Quality (0–5)
Msgs/min
Visual-Dialogue Context
Annotated windows
80.0 (20/25)
4.10
3.0
Visual-Only Context
Entire videos
96.0 (24/25)
3.62
3.5
Table 6: MIC results on internal MIC-Bench (25 videos). Msgs/min is the median across videos over the entire playback. The two settings differ in evaluation interval and scoring procedure, so their trigger rates and quality scores are not directly comparable.
Figure 6: SATC examples from the internal benchmark. Top: WebSearch identifies the “Zaochake” reservation as a breakfast experience at the water market. Bottom: WebSearch uses character names from the subtitles to identify Brotherhood of Blades II . Both examples correct answers generated without tools.
Figure 7: Real-world weather query at t=1 s. AURA 2.0 identifies Rome from the Colosseum and calls get_current_datetime and get_city_weather to retrieve the forecast. Without tools, the model reports that weather information is unavailable in the video; with tools, it provides the forecast for the visually identified city.
Model
mIoU
Accuracy (%)
Positive queries with a box
Positive hits
Correct abstentions
Qwen3.5-9B
0.635
76.0
26/37
25/37
13/13
AURA 2.0 (Ours)
0.906
94.0
37/37
36/37
11/13
Table 7: IVG results on a 50-image diagnostic set. mIoU is computed on present targets, with missing boxes assigned zero. Positive hits require IoU ≥0.5 ; correct abstentions require no box for an absent target.
Figure 8: IVG examples with less salient and peripheral targets. AURA 2.0 (top, red boxes) localizes both the handbag and the partially visible bus. Qwen3.5-9B (bottom, blue boxes) incorrectly declares the handbag absent and produces no response for the bus. Both models accurately localize the remote control despite the salient cat.
Concurrent sessions
TTFT
TPOT
TTFP
RTF
1
157
7.6
344
0.135
8
184
12.8
449
0.161
16
234
21.1
601
0.203
Table 8: Latency under concurrent sessions (median over pooled turns). TTFT, TPOT, and TTFP are in milliseconds; RTF is unitless. Lower is better for all metrics.
Figure 9: Cumulative effect of the inference optimizations on median TTFP with a single session. Each row adds one optimization to the preceding configuration, and the full configuration reduces median TTFP by 54.6%. The right column reports the change relative to the previous row.
Streaming video understanding studies how AI systems interpret continuous video in real time while retaining useful history and responding at the right moment.