cs.AIOct 26, 2026
SaveOmni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams
Organizations: The Hong Kong Polytechnic University · Alibaba Token Hub, Alibaba Group · Shanghai Jiaotong University · Tsinghua University ·
Abstract
Omnimodal large language models (Omni-LLMs) are increasingly expected to assist users while observing continuous audiovisual streams. However, existing evaluations provide limited insight into how reliably these models connect stream evidence and user needs to timely, sustained interventions. We introduce Omni-InteractScope, a benchmark of 1,479 human-annotated, cross-reviewed interactions with 1,790 temporal targets across eight task groups. It links input modalities, event triggers, intervention types, and user needs, covering event-timed tool invocation and query-free assistance alongside notifications, warnings, state tracking, and continuous responses. Our experimental results on the proposed benchmark reveal three critical limitations of current Omni-LLMs: (1) Timely tool invocation remains unreliable: all 12 evaluated models produce an on-time first response for at most 40% of tool-invocation targets, and joint timing-and-tool accuracy reaches at most 32.0%. (2) Query-free reminders remain poorly covered: all evaluated models achieve lower timely reference-target recall on query-free reminders than on explicit questions, measured on their respective target sets. Reminder recall never exceeds 23.0%. (3) End-to-end guidance coverage remains limited: nine models cover none of the 26 guidance sequences in full, and the remaining three cover only one to three sequences each. Targeted controls further show that tool-invocation scores alone do not establish event grounding and that reminder coverage depends on response history and frequency. These findings motivate developing and evaluating streaming assistants for timely action, initiative under general user needs, and coverage of entire response sequences.
Figures & tables
Figure 1: Overview of Omni-InteractScope. Top: Representative interactions connect user requests or general assistance needs with events in audiovisual streams and the content and timing of the required responses. The examples span event-timed tool invocation, a gesture reminder conditioned on an image and text request, and hearing assistance under a general instruction without an event-specific query. Bottom: Distributions of task types, scene categories, and stream durations across the benchmark’s 1,479 interactions. Clips reused across multiple interactions are counted once for each interaction in which they appear.
| Resource | Evaluation | Intervention objectives | ||||||
| Benchmark | Modality | #Videos | Open- ended | Streaming | Proactive | Tool | Query- free | Successive |
| (A) Offline video understanding | ||||||||
| Video-MME ( Fu et al., 2025 ) | V, A, S | 900 | – | – | – | – | – | – |
| LongVideoBench ( Wu et al., 2024 ) | V, S | 3,763 | – | – | – | – | – | – |
| (B) Streaming understanding and interaction | ||||||||
| StreamingBench ( Lin et al., 2026 ) | V, A | 900 | – | p | – | – | – | |
Table 1: From video understanding to streaming intervention. Coverage of scene inputs and interaction objectives at benchmark level.
Figure 2: Representative interactions across the eight task groups of Omni-InteractScope. Each panel links a user request (Q), timestamped stream observations, and reference responses (A) to illustrate what intervention is required and when. The colored diamonds mark reference response times. T6 includes an image and text request, while T7 and the T8-C guidance example show how a single request can require successive responses as the stream unfolds.
| Model configuration | Joint content–timing accuracy (%) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model | Size | Input | T1 Tool | T2 Event | T3 Risk | T4 State | T5* Time | T6 Cue | T7 Cont. | Avg. 6 tasks |
| Targets | 200 | 206 | 207 | 206 | 115 | 37 | 457 | |||
| Proprietary Models (prompted) | ||||||||||
| Gemini-3.1-Pro | N/R | V, A | 22.5 | 28.2 | 11.6 | 11.2 | N/A | 27.0 | 11.2 | 18.6 |
| Gemini-3.5-Flash | N/R | V, A | 29.5 | 31.6 | 14.0 | 10.2 | N/A | 24.3 | 11.4 | 20.2 |
| Qwen3.5-Omni-Plus | N/R | V, A | 32.0 | 24.3 | 4.8 | 3.9 | N/A | 10.8 | 7.9 | 13.9 |
Table 2: Main results on explicit-request tasks (T1–T7). Joint content–timing accuracy (%). Avg. equally weights T1–T4, T6, and T7.
| Model configuration | Timely target recall (%) | Coverage | |||||
|---|---|---|---|---|---|---|---|
| Model | Size | Input | Overall | T8-A | T8-B | T8-C | Complete T8-C |
| Proprietary Models | |||||||
| Gemini-3.1-Pro | N/R | V, A | 46.39 | 22.95 | 62.25 | 51.72 | 3/26 |
| Gemini-3.5-Flash | N/R | V, A | 41.11 | 22.13 | 57.62 | 39.08 | 2/26 |
| Qwen3.5-Omni-Plus | N/R | V, A | 31.11 | 9.02 | 49.67 | 29.89 | 1/26 |
| Open-source MLLMs | |||||||
Table 3: Main results on assistance tasks (T8). Timely reference-target recall (%) for query-free reminders (T8-A), audiovisual QA (T8-B), and guidance steps (T8-C), with 122/151/87 targets (360 overall). Overall pools hits across all targets. Complete T8-C counts fully covered guidance sequences out of 26. All configurations use full-clip causal collection, response-availability timestamps, and ordered matching (Section 4.1 ). Bold marks column maxima.
Figure 3: Scene access and response opportunities shape intervention scores. (a) T1 outcomes for Gemini-3.5-Flash with and without scene input (200 targets, two-second tolerance). Labels report target counts. (b) T8-A timely recall versus output rate for Gemini-3.5-Flash with no history, the five most recent responses, or history plus a ten-second cooldown. (c) Change in T8-A recall from general to event-specific instructions under scheduled and autonomous responding. Error bars show 95% paired cluster-bootstrap confidence intervals. Appendices D.1.2 , D.2.2 , and D.2.1 detail the protocols and results.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Behavior | Interactions | Targets | V | A | AV | Txt | Oth |
| T1 | Tool invocation | 200 | 200 | 189 | 10 | 1 | 0 | 0 |
| T2 | Event notification | 207 | 207 | 150 | 36 | 18 | 3 | 0 |
| T3 | Risk/error interjection | 208 | 208 | 206 | 0 | 2 | 0 | 0 |
| T4 | Cumulative-state tracking | 206 | 206 | 171 | 3 | 6 | 3 | 23 |
| T5 | Temporal conditions | 115 | 115 | 36 | 0 | 0 | 0 | 79 |
| T6 | Nonverbal/workflow cues | 37 | 37 | 32 | 0 | 4 | 1 | 0 |
Table 4: Task coverage and annotated trigger modalities. V, A, and AV denote visual, auditory, and joint audiovisual triggers. Txt and Oth denote textual and other triggers. The five columns partition the reference targets within each task.
Figure 4: An annotated cumulative-state interaction (T4). Three source frames show successive dish preparations. The request and reference response are reproduced from the annotation. Count labels identify the illustrated preparations, not model predictions. The evaluator derives seconds from the 48-second point target. The frame strip provides selected evidence; the model observes the causal stream rather than this summary.
| Rubric | Required content | Failure examples |
|---|---|---|
| General | Correct facts, names, quantities, and yes/no decisions. | Wrong facts, hallucinations, or refusals. |
| Event | The same event with its key subject, object, and action. | Generic narration, an unrelated event, or an incorrect participant or action. |
| Translation | The reference meaning, allowing synonyms and differences in fluency. | Omitted meaning or added content. |
| Assistance | The target information and any essential action. A guidance step must state the currently actionable instruction. | Contradictory directions, unsafe advice, invented critical details, a generic warning, or repetition of the user’s intention. |
Table 5: Content criteria. Equivalent wording is accepted. Each rubric evaluates correspondence to the reference target, independently of response timing.
Figure 5: Timing, content, and assignment play separate roles. (a) T1 assigns the first response, even when a later response would be eligible. The 6- and 10-second outputs are in separate groups because they are more than two seconds apart. (b) T8 can match a later eligible response after an incorrect attempt. Shaded intervals are inclusive valid windows. Diamonds mark correct content and crosses mark incorrect content; a correct response can still fail timing or remain unassigned. These are constructed scoring examples.
| Model | Interface | Scene audio | Spoken-query attachment |
| Proprietary Models | |||
| Gemini-3.1-Pro | Prefix | Native prefix | Audio and text |
| Gemini-3.5-Flash | Prefix | Native prefix | Audio and text |
| Qwen3.5-Omni-Plus | Prefix | Native prefix | Audio and text |
| Open-source MLLMs | |||
| Qwen3-Omni | Prefix | Native prefix | Audio and text |
Table 6: Assistance input paths. “Text” in the last column means the annotated query text without consuming the separate query-audio attachment. Every configuration also receives the annotated text.
| Model | On-time | Joint | Tool time | Image | Web | Audio |
|---|---|---|---|---|---|---|
| Gemini-3.1-Pro | 38.00 | 22.50 | 59.21 | 19.33 | 24.64 | 41.67 |
| Gemini-3.5-Flash | 37.50 | 29.50 | 78.67 | 21.85 | 42.03 | 33.33 |
| Qwen3.5-Omni-Plus | 39.00 | 32.00 | 82.05 | 26.89 | 39.13 | 41.67 |
| Qwen3-Omni | 37.00 | 26.50 | 71.62 | 18.49 | 37.68 | 41.67 |
| Qwen2.5-Omni | 28.00 | 11.00 | 39.29 | 7.56 | 15.94 | 16.67 |
| Qwen3-VL | 34.00 | 19.50 | 57.35 | 12.61 | 30.43 | 25.00 |
Table 7: T1 timing and tool accuracy (%). Joint and on-time scores use 200 targets; tool time uses each model’s timely responses. Per-tool denominators are 119 image, 69 web, and 12 audio targets. Incomplete targets remain in the denominator.
| Predictor | Accuracy | Mean recall | Image | Web | Audio | Abstain |
|---|---|---|---|---|---|---|
| Always image search | 59.50 | 33.33 | 100.00 | 0.00 | 0.00 | 0 |
| Gemini-3.5-Flash | 96.50 | 97.43 | 96.64 | 95.65 | 100.00 | 4 |
| Qwen3-VL | 51.50 | 65.13 | 21.85 | 98.55 | 75.00 | 24 |
| Qwen2.5-Omni | 63.00 | 69.28 | 41.18 | 100.00 | 66.67 | 0 |
Table 8: Request and timing priors on 200 T1 targets, without scene input. (a) Request-only tool accuracy and class recalls (%); class sizes are 119/69/12. Mean recall weights classes equally, and abstentions count as errors. (b) Joint hits (%) when each tool policy is emitted at a fixed delay after query arrival. All tested delays are shown; their maximum was not independently selected.
| Event | General | |||
|---|---|---|---|---|
| Model | Scheduled | Autonomous | Scheduled | Autonomous |
| Gemini-3.5-Flash | 30 (24.59) | 30 (24.59) | 19 (15.57) | 27 (22.13) |
| Qwen2.5-Omni | 8 (6.56) | 2 (1.64) | 0 (0.00) | 3 (2.46) |
| Qwen3-Omni | 8 (6.56) | 5 (4.10) | 4 (3.28) | 3 (2.46) |
Table 9: Event guidance and response scheduling on the same 122 T8-A episodes. (a) Hits (recall, %) in the four conditions. Scheduled (S) supplies an annotated response opportunity; Autonomous (A) uses the full causal grid. (b) Event-minus-general effects and their S-minus-A interaction in percentage points, with 95% paired intervals from 114 video-record clusters.
| Condition | Hits (recall, %) | Outputs | Outputs/min |
|---|---|---|---|
| No history | 22 (18.03) | 7,833 | 58.39 |
| History | 22 (18.03) | 3,283 | 24.47 |
| History + cooldown | 10 (8.20) | 808 | 6.02 |
| Paired recall changes (percentage points) | |||
| Contrast | Difference | 95% CI | |
| History no history | |||
Table 10: Gemini-3.5-Flash history and generation-time cooldown controls on 122 T8-A episodes. History contains the five most recent timestamped responses; cooldown enforces ten seconds between outputs. Paired intervals use 114 video-record clusters.
Figure 6: Step coverage and complete guidance on 20 three-step episodes. (a) Matched targets at each step position, with a denominator of 20 per cell. (b) Observed complete counts, permutation expectations, and central 95% reference intervals. Each permutation preserves the marginal counts in (a). The reference does not model episode difficulty or identify a memory mechanism.
| Model | Target-local | Continuous | Difference [95% CI] |
|---|---|---|---|
| Gemini-3.5-Flash | 26.80 | 19.17 | |
| Qwen3-VL | 4.28 | 3.36 |
Table 11: Decision-schedule effects on 119 paired interactions (142 targets). Accuracy equally weights T1–T4, T6, and T7. Differences are continuous minus target-local, in percentage points, with 95% paired intervals.
| Tolerance (s) | Scene present | Scene removed | Difference [95% CI] |
|---|---|---|---|
| 1 | 11.00 (22/200) | 12.50 (25/200) | |
| 2 (primary) | 29.50 (59/200) | 33.00 (66/200) | |
| 4 | 53.00 (106/200) | 66.00 (132/200) |
Table 12: Temporal-tolerance sensitivity of the Gemini-3.5-Flash scene control. Cells give joint accuracy (%) and hits out of 200. Differences are scene present minus removed, in percentage points, with 95% paired intervals. The output-grouping rule remains fixed at two seconds.
| Model | Task | Targets | Success | Premature | Content fail | Late/missing |
|---|---|---|---|---|---|---|
| Gemini-3.1-Pro | T4 | 206 | 23 | 178 | 5 | 0 |
| Gemini-3.1-Pro | T7 | 457 | 51 | 229 | 146 | 31 |
| Gemini-3.5-Flash | T4 | 206 | 21 | 172 | 10 | 3 |
| Gemini-3.5-Flash | T7 | 457 | 52 | 233 | 152 | 20 |
Table 13: First-response outcomes on state tracking (T4) and continuous responses (T7), using the main-table trajectories. Counts sum to the target denominator in each row. Content failure denotes an on-time response with incorrect content.
Figure 7: Acting early and giving the wrong reminder. (a) Gemini-3.5-Flash calls a tool at query arrival, before the scored window, and repeats it at 29.00 s. The reference tool is image_search ; timing already determines failure. (b) The still-lit M button is outlined in the source frame. All five in-window Gemini-3.1-Pro responses recommend exit instead of waiting. Two are quoted in full; other unmatched outputs are not thereby classified as invalid assistance. Neither case establishes an internal cause.
Figure 8: C# diagnosis across three editing states. Source-frame crops accompany recorded replies at the indicated times. The request and replies are translated from Chinese; [...] marks omissions. The original target is a reminder to run tests at 60.50 s, against which the stored outcome is premature. The displayed diagnoses illustrate agreement with the screen, not additional scored targets.
Figure 9: Following a DaVinci Resolve import workflow. Original frames appear on the left. Panels (a,c) show source crops with matching asset labels; (b) is a redrawn English reference dialog. † † footnotemark: The diamond marks the annotated second-method point (189.87 s), with its -second window shaded. Circles mark the request and first replies.
Figure 10: Two covered steps with an incomplete final instruction. Each column pairs a source state with the full reference and a recorded Gemini-3.1-Pro response. The first two columns show stored matches; the third shows one of four incorrect final-target candidates. The shelf location and closing action distinguish the reference from the prediction. Frame timestamps identify sampled evidence, while output timestamps identify model emissions. An incorrect 21-second attempt does not prevent the later 22-second match under ordered assignment.