Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute · Alibaba Token Hub, Alibaba Group · The Chinese University of Hong Kong · Tsinghua University · Hong Kong Polytechnic University · Nankai University · Johns Hopkins University
Natural audio-visual interaction is emerging as an important interface for AI assistants, allowing users to communicate through speech and vision rather than carefully composed text prompts. However, existing benchmarks of interactive capabilities still focus primarily on response quality, leaving a more fundamental question underexplored: can a model correctly infer the user's underlying demand from complex multimodal interaction? Real-world user demands are often underspecified in speech and must be inferred from multimodal cues and dialogue history. This inference is further complicated by ambiguous or disfluent expression and noisy acoustic environments. Conversely, request-like speech may not constitute a demand to the assistant, leading to false triggers. We establish Omni Demand Understanding (ODU) as a distinct multimodal contextual inference problem: given an interaction stream, a model must detect whether a user demand is present and infer intent from multimodal and conversational context. ODU evaluates this capability along five dimensions, covering both single-turn and multi-turn interactions. We construct ODU-Bench using a challenge-driven taxonomy, taxonomy-guided agentic video generation, and human-recorded interactions, followed by media-grounded annotation and human verification. We evaluate 14 native MLLMs. Even the strongest, Gemini 3.1 Pro, recovers only 44.7% of key information that must be inferred from visual, acoustic, or conversational context. Moreover, 11 of the 14 models exhibit false-trigger rates above 50% on non-demand scenarios. These results reveal a systematic capability gap in current MLLMs' ability to infer contextual user demands. We hope ODU can establish the evaluation of a previously underexplored yet essential capability in multimodal interaction: correctly understanding user demands before generating an appropriate response.
Figures & tables
Figure 1: Representative challenging scenarios in ODU. Demand-bearing cases (a–d) require integrating visual, acoustic, and conversational context while handling challenging expression forms and acoustic conditions, while no-demand cases (e–g) contain demand-like expressions that should not trigger responses.
Benchmark
Demand Presence
Intent Prediction Format
Spoken Utterance Input
Multi-Turn Dialogue Input
Visual Cues
Acoustic Cues
Multimodal interaction benchmarks
OmniMMI ( Wang et al., 2025 )
◐
✗
◐
✓
◐
✗
MultiVox ( Selvakumar et al., 2025 )
✗
✗
✓
✗
✓
✓
OmniInteract ( Lu et al., 2026 )
◐
✗
✓
✓
◐
✗
Full-Duplex-Bench-v2 ( Lin et al., 2026 )
◐
✗
✓
✓
✗
✗
Intent and goal understanding benchmarks
Table 1: Benchmark comparison. Visual and acoustic cues refer to intent inference.
Output measured
Metric
Weight
M1
demand present
binary macro-F1 over demand and no-demand scenes
0.15
M2
structured intent, required context
key-point hit-rate
0.60
M3
demand span
temporal IoU
0.10
M4
transcript
max(0,1−adaptive CER/WER)
0.10
M5
user profile
mean per-field accuracy
0.05
Table 2: Evaluation dimensions and their contribution to the overall score.
Figure 2: Taxonomy coverage in ODU-Bench. (a) and (b) show positive and negative sample distributions across their taxonomy axes; segment angles are proportional to sample counts within each axis. (c) shows task and scenario coverage, with word size reflecting frequency.
Figure 3: Overview of the ODU-Bench construction pipeline. Challenge-driven targets are expanded into coarse-to-fine scripts, screened for plausibility and challenge validity, and realized as synthesized or human-recorded interactions. Annotations are then reconstructed from complementary media evidence and verified through automatic checks and human review.
Audio–visual (%)
Audio-only (%)
Model
Size
Avg.
M1 Detect
M2 Keypt.
M3 Locate
M4 Trans.
M5 Profile
FTR Neg.
Avg.
M1 Detect
M2 Keypt.
M3 Locate
M4 Trans.
M5 Profile
FTR Neg.
Closed-source Models
Gemini 3.1 Pro
–
72.6
83.0
63.4
81.9
91.3
95.7
40.6
75.4
87.3
70.1
61.5
91.7
97.6
31.7
Gemini 3.7 Flash
–
69.6
83.9
60.2
78.7
85.9
88.3
27.5
72.9
89.6
66.1
62.7
88.4
94.1
16.0
Gemini 3.5 Flash Lite
–
57.4
63.9
49.9
68.1
71.5
79.3
55.6
62.2
74.5
56.4
56.7
74.3
82.4
38.5
Qwen3.5-Omni-Plus
–
69.6
63.9
65.8
75.7
83.3
93.5
73.8
74.7
76.2
71.5
66.1
89.3
95.8
55.3
Table 3: Evaluation results on the ODU task. Audio–visual (AV) and audio-only (AO) results are reported separately as percentages. M1–M5 measure detection, key-point coverage, localization, transcription, and user profile; Avg. uses weights 0.15/0.60/0.10/0.10/0.05 over M1–M5, and FTR is the no-demand false-trigger rate (lower is better). Size denotes the model parameter count; for MoE models, “A” indicates the number of active parameters. The text-only system receives ASR transcripts and sentence-level timestamps but no media.
Figure 7
Metric
Gemini 3.1 Pro
Seed 2.0 Lite
Qwen3.5-Omni-Plus
Qwen3-Omni-Think
Ming-Flash-Omni 2.0
Syn.
Rec.
Δ
Syn.
Rec.
Δ
Syn.
Rec.
Δ
Syn.
Rec.
Δ
Syn.
Rec.
Δ
Avg.
71.9
74.2
2.4
67.5
67.6
0.1
68.5
65.7
-2.8
61.3
53.6
-7.6
54.2
49.3
-4.8
M1 Detect
83.3
86
2.7
58.2
55.9
-2.2
60.7
53.7
-6.9
60.9
48.6
-12.3
49.2
43.5
-5.6
M2 Keypt.
62
65.1
3.2
63.9
63.7
-0.2
64.9
64.2
-0.7
56.4
49.5
-6.9
50.8
47.6
-3.2
M3 Locate
81.8
84.9
3.1
82.6
82.9
0.3
76.3
73.1
-3.3
45.9
49.8
3.9
31
37.5
6.5
M4 Trans.
92.3
88.3
-4.1
76.3
79.8
3.5
81.9
71.6
-10.3
90.7
71.3
-19.5
89.5
62.3
-27.3
Table 4: Model performance (%) on Chinese audio–visual scenes from synthetic (Syn.) and human-recorded (Rec.) interactions. Δ= Rec. − Syn. in percentage points.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Property
Main split
Total scenes
2 078
Audio–visual / audio-only
1 347 / 731
Chinese / English
1 170 / 908
Positive / no-demand negative
1 631 / 447
Generated / human-recorded interactions
1 801 / 277
Multi-turn interactions
969
Appendix
Table 5: Composition of the released dataset.
Figure 7: Annotation-review interface, shown in two consecutive views. (a) Generated media, reference annotation, sample acceptance, and transcript. The original field order is retained; selected options are interface defaults.
Figure 8: Annotation-review interface (continued). (b) Demand time spans, required context, all key points, and user profile fields from the same generated sample.
System
Channel
Reference points ( n )
Coverage (%)
Gap (pp) Request − context
95% confidence interval for gap (pp)
Gemini 3.1 Pro
Spoken request
1 984
82.9
–
–
All context
1 972
44.7
38.2
[ 35.4 , 41.1 ]
Visual
804
48.6
34.2
[ 30.3 , 38.3 ]
Audio
557
47.2
35.6
[ 31.2 , 40.2 ]
History
475
42.1
40.8
[ 35.9 , 45.7 ]
Qwen3.5-Omni-Plus
Spoken request
1 984
83.6
–
–
Appendix
Table 6: Point-weighted coverage by evidence channel (Figure 5 ). Gap subtracts each context row’s coverage from the same model’s Spoken request coverage; positive values mean lower context coverage. The 95% confidence interval quantifies the gap’s uncertainty under scene resampling. Both are in percentage points (pp). n counts reference points; dashes mark the request reference. Gaps use unrounded rates.
Setting
Quantity
Judge
Stability readout
Qwen3.6
DeepSeek
GPT-5.4
Panel A: target recoverability
A
Recoverability
0.998
0.996
0.996
3 956 key points
A
Orphan rate
0.002
0.004
0.004
ceiling loss
Panel B: stored-output re-judging
B
Gemini 3.1 Pro coverage
0.648
0.639
0.609
range 0.039
Appendix
Table 7: LLM judge stability. Panel A tests whether benchmark key points can be recovered from their source annotations. Panel B re-judges identical stored outputs; all three judges yield the same model ordering.
Figure 9: Positive challenge profiles: audio–visual. Category-minus-system-mean M2 on the common positive cohort. Labels give category codes and names; the final row is the unweighted mean across the displayed systems. The color scale is shared with Figure 10 .
Subset
Scenes
Given preferred
Tie
Not given preferred
All
200
103 ( 51.5 %)
80 ( 40.0 %)
17 ( 8.5 %)
Demand present
160
85 ( 53.1 %)
58 ( 36.3 %)
17 ( 10.6 %)
No-demand
40
18 ( 45.0 %)
22 ( 55.0 %)
0 ( 0.0 %)
Chinese
100
47 ( 47.0 %)
45 ( 45.0 %)
8 ( 8.0 %)
English
100
56 ( 56.0 %)
35 ( 35.0 %)
9 ( 9.0 %)
Appendix
Table 8: Human response preferences: counts (percentages) within each subset. “Given” and “Not given” abbreviate the “Given demand annotation” and “Not given demand annotation” conditions. Ties remain in the denominator.
Figure 11: Blinded response-comparison interface. The human reviewer is asked to choose a preferred reply between A and B.
Figure 12: Qualitative error analysis on English generated audio–visual scenes. (a)–(b) Demand-present cases; M2 reports covered/reference key points.
We introduce OmniInteract, a streaming benchmark for real-time omnimodal large language models evaluated through native online inference over audio-visual streams. Unlike offline video understanding or text-prompted streaming QA, OmniInteract preserves the original audio-visual stream and requires models to process it online, without access to future content. User queries and ambient sounds are embedded in the audio track, requiring models to detect multimodal triggers, decide when to respond, and answer while the stream unfolds. OmniInteract contains 250 videos with 1,430 temporally grounded response slots: 1,062 1Q1A slots across real-time, proactive, and nested scenarios, and 368 1QnA slots for continuous task monitoring and step guidance. Each slot includes a trigger, response window, and target answer. We evaluate response correctness, timing, invalid outputs, interruption handling, and context continuity using Interaction-Aware Quality-Timeliness F1, Interruption Diagnostic Suite, and Nested Chain Completion Score. Experiments show that current models remain weak in streaming interaction, with the best overall IA-QTF1 reaching only 0.368 and the best 1QnA IA-QTF1 only 0.052. Further study on mathematical reasoning in full-duplex settings shows that offline capability does not necessarily transfer to online interaction. Code and datasets will be made publicly accessible at https://github.com/Lucky-Lance/OmniInteract.
Smart-home assistants are expected to handle diverse, realistic requests that arise in daily life. In such interactions, users often rely on the surrounding multimodal context-pointing at objects or referring to what they see or hear, leaving their requests underspecified in language alone. Existing smart-home benchmarks, however, express user requests solely through language, leaving context-dependent real-world requests underexplored. To bridge this gap, we introduce OmniSmartHome, a multimodal smart-home benchmark where each spoken request is paired with the surrounding visual and spatial-audio context, providing complementary cues to disambiguate underspecified requests. OmniSmartHome comprises 1,360 synthetic and 272 real-world episodes. We evaluate 16 omnimodal large language models (Omni-LLMs) and reveal that, while they perform strongly when speech alone sufficiently conveys the user's intent, performance drops substantially when resolving it requires reasoning over multimodal contextual cues. As a simple agent baseline, we provide PROME (PROcedural Memory for multimodal Evidence gathering), which equips agents with specialized audio-visual perception tools and procedural memory for orchestrating their use. PROME generally improves performance across six Omni-LLMs. Demos and examples are available at https://omni-smart-home.github.io
Jihoo Jung, Suho Yoo, Jeongsoo Choi +5
Korea Advanced Institute of Science and Technology (KAIST) · Hankuk University of Foreign Studies
Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.
Xianyun Sun, Chaoyou Fu, Zhengye Zhang +6
Nanjing University · Nankai University · University of Waterloo