Organizations: The Hong Kong Polytechnic University · Alibaba Token Hub, Alibaba Group · Shanghai Jiaotong University · Tsinghua University · Fudan University
Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams. However, existing evaluations provide limited insight into how reliably these models connect stream evidence and user needs to timely, sustained interventions. We introduce Omni-InteractScope, a benchmark of 1,479 human-annotated, cross-reviewed interactions with 1,790 temporal targets across eight task groups. It links input modalities, event triggers, intervention types, and user needs, covering event-timed tool invocation and query-free assistance alongside notifications, warnings, state tracking, and continuous responses. Our experimental results on the proposed benchmark reveal three critical limitations of current Omni-LLMs: (1) Timely tool invocation remains unreliable: all 12 evaluated models produce an on-time first response for at most 40% of tool-invocation targets, and joint timing-and-tool accuracy reaches at most 32.0%. (2) Query-free reminders remain poorly covered: all evaluated models achieve lower timely reference-target recall on query-free reminders than on explicit questions, measured on their respective target sets. Reminder recall never exceeds 23.0%. (3) End-to-end guidance coverage remains limited: nine models cover none of the 26 guidance sequences in full, and the remaining three cover only one to three sequences each. Targeted controls further show that tool-invocation scores alone do not establish event grounding and that reminder coverage depends on response history and frequency. These findings motivate developing and evaluating streaming assistants for timely action, initiative under general user needs, and coverage of entire response sequences.
Figures & tables
Figure 1: Overview of Omni-InteractScope. Top: Representative interactions connect user requests or general assistance needs with events in audiovisual streams and the content and timing of the required responses. The examples span event-timed tool invocation, a gesture reminder conditioned on an image and text request, and hearing assistance under a general instruction without an event-specific query. Bottom: Distributions of task types, scene categories, and stream durations across the benchmark’s 1,479 interactions. Clips reused across multiple interactions are counted once for each interaction in which they appear.
Resource
Evaluation
Intervention objectives
Benchmark
Modality
#Videos
Open- ended
Streaming
Proactive
Tool
Query- free
Successive
(A) Offline video understanding
Video-MME ( Fu et al., 2025 )
V, A, S
900
–
–
–
–
–
–
LongVideoBench ( Wu et al., 2024 )
V, S
3,763
–
–
–
–
–
–
(B) Streaming understanding and interaction
StreamingBench ( Lin et al., 2026 )
V, A
900
–
✓
✓ p
–
–
–
Table 1: From video understanding to streaming intervention. Coverage of scene inputs and interaction objectives at benchmark level.
Figure 2: Representative interactions across the eight task groups of Omni-InteractScope. Each panel links a user request (Q), timestamped stream observations, and reference responses (A) to illustrate what intervention is required and when. The colored diamonds mark reference response times. T6 includes an image and text request, while T7 and the T8-C guidance example show how a single request can require successive responses as the stream unfolds.
Model configuration
Joint content–timing accuracy (%) ↑
Model
Size
Input
T1 Tool
T2 Event
T3 Risk
T4 State
T5* Time
T6 Cue
T7 Cont.
Avg. 6 tasks
Targets
200
206
207
206
115
37
457
Proprietary Models (prompted)
Gemini-3.1-Pro
N/R
V, A
22.5
28.2
11.6
11.2
N/A
27.0
11.2
18.6
Gemini-3.5-Flash
N/R
V, A
29.5
31.6
14.0
10.2
N/A
24.3
11.4
20.2
Qwen3.5-Omni-Plus
N/R
V, A
32.0
24.3
4.8
3.9
N/A
10.8
7.9
13.9
Table 2: Main results on explicit-request tasks (T1–T7). Joint content–timing accuracy (%). Avg. equally weights T1–T4, T6, and T7.
Model configuration
Timely target recall (%) ↑
Coverage
Model
Size
Input
Overall
T8-A
T8-B
T8-C
Complete T8-C
Proprietary Models
Gemini-3.1-Pro
N/R
V, A
46.39
22.95
62.25
51.72
3/26
Gemini-3.5-Flash
N/R
V, A
41.11
22.13
57.62
39.08
2/26
Qwen3.5-Omni-Plus
N/R
V, A
31.11
9.02
49.67
29.89
1/26
Open-source MLLMs
Table 3: Main results on assistance tasks (T8). Timely reference-target recall (%) for query-free reminders (T8-A), audiovisual QA (T8-B), and guidance steps (T8-C), with 122/151/87 targets (360 overall). Overall pools hits across all targets. Complete T8-C counts fully covered guidance sequences out of 26. All configurations use full-clip causal collection, response-availability timestamps, and ordered matching (Section 4.1 ). Bold marks column maxima.
Figure 3: Scene access and response opportunities shape intervention scores. (a) T1 outcomes for Gemini-3.5-Flash with and without scene input (200 targets, two-second tolerance). Labels report target counts. (b) T8-A timely recall versus output rate for Gemini-3.5-Flash with no history, the five most recent responses, or history plus a ten-second cooldown. (c) Change in T8-A recall from general to event-specific instructions under scheduled and autonomous responding. Error bars show 95% paired cluster-bootstrap confidence intervals. Appendices D.1.2 , D.2.2 , and D.2.1 detail the protocols and results.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Behavior
Interactions
Targets
V
A
AV
Txt
Oth
T1
Tool invocation
200
200
189
10
1
0
0
T2
Event notification
207
207
150
36
18
3
0
T3
Risk/error interjection
208
208
206
0
2
0
0
T4
Cumulative-state tracking
206
206
171
3
6
3
23
T5
Temporal conditions
115
115
36
0
0
0
79
T6
Nonverbal/workflow cues
37
37
32
0
4
1
0
Appendix
Table 4: Task coverage and annotated trigger modalities. V, A, and AV denote visual, auditory, and joint audiovisual triggers. Txt and Oth denote textual and other triggers. The five columns partition the reference targets within each task.
Figure 4: An annotated cumulative-state interaction (T4). Three source frames show successive dish preparations. The request and reference response are reproduced from the annotation. Count labels identify the illustrated preparations, not model predictions. The evaluator derives [46,50] seconds from the 48-second point target. The frame strip provides selected evidence; the model observes the causal stream rather than this summary.
Rubric
Required content
Failure examples
General
Correct facts, names, quantities, and yes/no decisions.
Wrong facts, hallucinations, or refusals.
Event
The same event with its key subject, object, and action.
Generic narration, an unrelated event, or an incorrect participant or action.
Translation
The reference meaning, allowing synonyms and differences in fluency.
Omitted meaning or added content.
Assistance
The target information and any essential action. A guidance step must state the currently actionable instruction.
Contradictory directions, unsafe advice, invented critical details, a generic warning, or repetition of the user’s intention.
Appendix
Table 5: Content criteria. Equivalent wording is accepted. Each rubric evaluates correspondence to the reference target, independently of response timing.
Figure 5: Timing, content, and assignment play separate roles. (a) T1 assigns the first response, even when a later response would be eligible. The 6- and 10-second outputs are in separate groups because they are more than two seconds apart. (b) T8 can match a later eligible response after an incorrect attempt. Shaded intervals are inclusive valid windows. Diamonds mark correct content and crosses mark incorrect content; a correct response can still fail timing or remain unassigned. These are constructed scoring examples.
Model
Interface
Scene audio
Spoken-query attachment
Proprietary Models
Gemini-3.1-Pro
Prefix
Native prefix
Audio and text
Gemini-3.5-Flash
Prefix
Native prefix
Audio and text
Qwen3.5-Omni-Plus
Prefix
Native prefix
Audio and text
Open-source MLLMs
Qwen3-Omni
Prefix
Native prefix
Audio and text
Appendix
Table 6: Assistance input paths. “Text” in the last column means the annotated query text without consuming the separate query-audio attachment. Every configuration also receives the annotated text.
Model
On-time
Joint
Tool ∣ time
Image
Web
Audio
Gemini-3.1-Pro
38.00
22.50
59.21
19.33
24.64
41.67
Gemini-3.5-Flash
37.50
29.50
78.67
21.85
42.03
33.33
Qwen3.5-Omni-Plus
39.00
32.00
82.05
26.89
39.13
41.67
Qwen3-Omni
37.00
26.50
71.62
18.49
37.68
41.67
Qwen2.5-Omni
28.00
11.00
39.29
7.56
15.94
16.67
Qwen3-VL
34.00
19.50
57.35
12.61
30.43
25.00
Appendix
Table 7: T1 timing and tool accuracy (%). Joint and on-time scores use 200 targets; tool ∣ time uses each model’s timely responses. Per-tool denominators are 119 image, 69 web, and 12 audio targets. Incomplete targets remain in the denominator.
Predictor
Accuracy
Mean recall
Image
Web
Audio
Abstain
Always image search
59.50
33.33
100.00
0.00
0.00
0
Gemini-3.5-Flash
96.50
97.43
96.64
95.65
100.00
4
Qwen3-VL
51.50
65.13
21.85
98.55
75.00
24
Qwen2.5-Omni
63.00
69.28
41.18
100.00
66.67
0
Appendix
Table 8: Request and timing priors on 200 T1 targets, without scene input. (a) Request-only tool accuracy and class recalls (%); class sizes are 119/69/12. Mean recall weights classes equally, and abstentions count as errors. (b) Joint hits (%) when each tool policy is emitted at a fixed delay after query arrival. All tested delays are shown; their maximum was not independently selected.
Event
General
Model
Scheduled
Autonomous
Scheduled
Autonomous
Gemini-3.5-Flash
30 (24.59)
30 (24.59)
19 (15.57)
27 (22.13)
Qwen2.5-Omni
8 (6.56)
2 (1.64)
0 (0.00)
3 (2.46)
Qwen3-Omni
8 (6.56)
5 (4.10)
4 (3.28)
3 (2.46)
Appendix
Table 9: Event guidance and response scheduling on the same 122 T8-A episodes. (a) Hits (recall, %) in the four conditions. Scheduled (S) supplies an annotated response opportunity; Autonomous (A) uses the full causal grid. (b) Event-minus-general effects and their S-minus-A interaction in percentage points, with 95% paired intervals from 114 video-record clusters.
Condition
Hits (recall, %)
Outputs
Outputs/min
No history
22 (18.03)
7,833
58.39
History
22 (18.03)
3,283
24.47
History + cooldown
10 (8.20)
808
6.02
Paired recall changes (percentage points)
Contrast
Difference
95% CI
History − no history
+0.00
[−8.13,+8.00]
Appendix
Table 10: Gemini-3.5-Flash history and generation-time cooldown controls on 122 T8-A episodes. History contains the five most recent timestamped responses; cooldown enforces ten seconds between outputs. Paired intervals use 114 video-record clusters.
Figure 6: Step coverage and complete guidance on 20 three-step episodes. (a) Matched targets at each step position, with a denominator of 20 per cell. (b) Observed complete counts, permutation expectations, and central 95% reference intervals. Each permutation preserves the marginal counts in (a). The reference does not model episode difficulty or identify a memory mechanism.
Model
Target-local
Continuous
Difference [95% CI]
Gemini-3.5-Flash
26.80
19.17
−7.63[−13.96,−1.77]
Qwen3-VL
4.28
3.36
−0.92[−4.17,+2.10]
Appendix
Table 11: Decision-schedule effects on 119 paired interactions (142 targets). Accuracy equally weights T1–T4, T6, and T7. Differences are continuous minus target-local, in percentage points, with 95% paired intervals.
Tolerance (s)
Scene present
Scene removed
Difference [95% CI]
1
11.00 (22/200)
12.50 (25/200)
−1.50[−4.00,+1.00]
2 (primary)
29.50 (59/200)
33.00 (66/200)
−3.50[−7.00,−0.50]
4
53.00 (106/200)
66.00 (132/200)
−13.00[−18.50,−7.50]
Appendix
Table 12: Temporal-tolerance sensitivity of the Gemini-3.5-Flash scene control. Cells give joint accuracy (%) and hits out of 200. Differences are scene present minus removed, in percentage points, with 95% paired intervals. The output-grouping rule remains fixed at two seconds.
Model
Task
Targets
Success
Premature
Content fail
Late/missing
Gemini-3.1-Pro
T4
206
23
178
5
0
Gemini-3.1-Pro
T7
457
51
229
146
31
Gemini-3.5-Flash
T4
206
21
172
10
3
Gemini-3.5-Flash
T7
457
52
233
152
20
Appendix
Table 13: First-response outcomes on state tracking (T4) and continuous responses (T7), using the main-table trajectories. Counts sum to the target denominator in each row. Content failure denotes an on-time response with incorrect content.
Figure 7: Acting early and giving the wrong reminder. (a) Gemini-3.5-Flash calls a tool at query arrival, before the scored window, and repeats it at 29.00 s. The reference tool is image_search ; timing already determines failure. (b) The still-lit M button is outlined in the source frame. All five in-window Gemini-3.1-Pro responses recommend exit instead of waiting. Two are quoted in full; other unmatched outputs are not thereby classified as invalid assistance. Neither case establishes an internal cause.
Figure 8: C# diagnosis across three editing states. Source-frame crops accompany recorded replies at the indicated times. The request and replies are translated from Chinese; [...] marks omissions. The original target is a reminder to run tests at 60.50 s, against which the stored outcome is premature. The displayed diagnoses illustrate agreement with the screen, not additional scored targets.
Figure 9: Following a DaVinci Resolve import workflow. Original frames appear on the left. Panels (a,c) show source crops with matching asset labels; (b) is a redrawn English reference dialog. † † footnotemark: The diamond marks the annotated second-method point (189.87 s), with its ±2 -second window shaded. Circles mark the request and first replies.
Figure 10: Two covered steps with an incomplete final instruction. Each column pairs a source state with the full reference and a recorded Gemini-3.1-Pro response. The first two columns show stored matches; the third shows one of four incorrect final-target candidates. The shelf location and closing action distinguish the reference from the prediction. Frame timestamps identify sampled evidence, while output timestamps identify model emissions. An incorrect 21-second attempt does not prevent the later 22-second match under ordered assignment.
We introduce OmniInteract, a streaming benchmark for real-time omnimodal large language models evaluated through native online inference over audio-visual streams. Unlike offline video understanding or text-prompted streaming QA, OmniInteract preserves the original audio-visual stream and requires models to process it online, without access to future content. User queries and ambient sounds are embedded in the audio track, requiring models to detect multimodal triggers, decide when to respond, and answer while the stream unfolds. OmniInteract contains 250 videos with 1,430 temporally grounded response slots: 1,062 1Q1A slots across real-time, proactive, and nested scenarios, and 368 1QnA slots for continuous task monitoring and step guidance. Each slot includes a trigger, response window, and target answer. We evaluate response correctness, timing, invalid outputs, interruption handling, and context continuity using Interaction-Aware Quality-Timeliness F1, Interruption Diagnostic Suite, and Nested Chain Completion Score. Experiments show that current models remain weak in streaming interaction, with the best overall IA-QTF1 reaching only 0.368 and the best 1QnA IA-QTF1 only 0.052. Further study on mathematical reasoning in full-duplex settings shows that offline capability does not necessarily transfer to online interaction. Code and datasets will be made publicly accessible at https://github.com/Lucky-Lance/OmniInteract.
Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.
Xianyun Sun, Chaoyou Fu, Zhengye Zhang +6
Nanjing University · Nankai University · University of Waterloo
Omni-proactive streaming video understanding, i.e., autonomously deciding when to speak and what to say from continuous audio-visual streams, is an emerging capability of omni-modal large language models. Existing benchmarks fall short in three key aspects: they rely primarily on visual signals, adopt polling or fixed-timestamp protocols instead of true proactive evaluation, and cover only a limited range of tasks, preventing reliable assessment and differentiation of omni-proactive streaming models. We present OmniPro, the first benchmark to jointly evaluate omni-modal perception, proactive responding, and diverse video understanding tasks. It comprises 2,700 human-verified samples spanning 9 sub-tasks and 3 cognitive levels, covering 6 basic video understanding capabilities. Notably, 84% of samples require audio signals (speech or non-speech), and each sample is annotated with modality-isolation labels to enable fine-grained multimodal analysis. We further introduce a dual-mode evaluation protocol: Probe mode assesses content understanding by querying the model before and after each ground-truth trigger, while Online mode evaluates full proactive ability by requiring models to autonomously decide when to respond in streaming input. Evaluating 11 representative models reveals three key findings: (1) audio provides consistent gains but with highly variable utilization across models, (2) performance degrades significantly over time, indicating limited long-horizon robustness, and (3) non-speech audio perception remains the weakest dimension.
Ruixiang Zhao, Jie Yang, Zijie Xin +4
Renmin University of China · WeChat Vision, Tencent Inc.