DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams

Yuhang Chen; Guangyue Liu; Shilong Liu

cs.AIOct 20, 2026

DIVE: Benchmarking Full-Duplex Interaction with Visual Engagement in Continuous Streams

Authors: Yuhang Chen, Guangyue Liu, Shilong Liu

Organizations: Huazhong University of Science and Technology · Princeton University

Abstract

Recent advances in multimodal large language models (MLLMs) have substantially improved spoken interaction and video understanding. Existing online and streaming benchmarks primarily assess causal multimodal perception through timestamped question answering, whereas real-world interaction requires models to continuously identify task-relevant multimodal cues and decide whether, when, and how to respond. This leaves open how well current models can sustain real-time interaction throughout an evolving multimodal stream. To address this gap, we introduce DIVE, a benchmark for evaluating full-duplex interaction with visual engagement under continuous and realistic multimodal streams. It organizes six tasks into three complementary interaction scenarios. (1) Proactive Intervention evaluates whether models can identify emerging visual or semantic cues and intervene at the appropriate moment. (2) Continuous Engagement evaluates sustained participation through continuous visual tracking and visually grounded query interaction. (3) Interaction Coordination evaluates how models coordinate listening and speaking under overlapping and multi-party interactions. DIVE contains 525 audio/video samples with fine-grained human annotations and precise temporal metadata. Its evaluation protocol separates response quality, participation decisions, and response timing. We evaluate representative real-time models using response quality, window response rates, and reaction latency. Experiments show that leading real-time models achieve sub-second average response latency, yet their average window response rates remain at or below 53.77%. Response quality varies substantially across models and tasks, with adapting ongoing responses to newly arriving information remaining particularly challenging. We hope DIVE facilitates progress toward more natural and capable real-time multimodal interactive systems.

Figures & tables

Offline Benchmarks
Benchmark	Modality	Duplex	Video Cue	Proactive Intervention		Continuous Engagement	Interaction Coordination		Continuous Engagement
Benchmark	Modality	Duplex	Video Cue	VTI	SPI	CQI	ORA	AVAT	CAT
MVBench Li et al. (2024)	V	✗	✗	✗	✗	✗	✗	✗	✗
Video-MME Fu et al. (2025)	V	✗	✗	✗	✗	✗	✗	✗	✗
MLVU Zhou et al. (2025)	V	✗	✗	✗	✗	✗	✗	✗	✗
LongVideoBench Wu et al. (2024)	V	✗	✗	✗	✗	✗	✗	✗	✗

Table 1: Comparison of DIVE with representative offline, streaming, and full-duplex interaction benchmarks. V and A denote video and audio, respectively. Duplex indicates concurrent perception and generation, while Video Cue indicates that task-relevant visual cues explicitly determine when or how the model should respond during an ongoing interaction. The six task dimensions correspond to our interaction taxonomy: VTI denotes Video Timely Intervention, SPI Speech Proactive Intervention, CQI Continuous Query Interaction, ORA Overlapping Response Adaptation, AVAT Audio-Visual Addressee Tracking, and CAT Continuous Action Tracking.

Figure 1: Examples of the tasks in Proactive Intervention. <Interrupted.> marks where the ongoing utterance is interrupted, whereas <Intervention.> marks where the model proactively intervenes.

Figure 2: Examples of the tasks in Continuous Engagement. <Visual.> indicates visually grounded queries, while <Independent.> indicates general queries independent of the visual content.

Figure 3: Example of each task in Interaction Coordination. <Silence.> indicates that the model remains silent.

Figure 4: Overview of the DIVE evaluation framework. Interaction Decision , which measures whether the model responds when required and remains silent when appropriate; Temporal Alignment , which categorizes required responses as early, on-time, late, or missed relative to the annotated response windows; and Response Quality , which assesses whether the first response assigned to each required interaction event satisfies its task-specific content requirements.

Native Audio-Visual Models
Model	VTI $\uparrow$	SPI $\uparrow$	CQI $\uparrow$	ORA $\uparrow$	AVAT $\uparrow$	CAT $\uparrow$	Avg. $\uparrow$
Aero-Realtime-4B	53.7	12.3	8.7	8.2	19.9	20.3	20.52
MiniCPM-o 4.5	14.6	24.8	13.1	7.6	9.9	19.7	14.95
DuplexOmni	21.1	10.6	28.6	9.1	11.8	23.4	17.43
Qwen3-Omni-30B-A3B-Instruct	19.8	20.3	21.2	10.1	18.5	10.0	16.65
GPT-Realtime-2.1	100.0	20.0	44.3	56.5	41.9	59.9	53.77

Table 3: Window response rate (%) on the six DIVE tasks. Higher is better. Avg. is the unweighted mean across the six tasks.

Native Audio-Visual Models
Model	VTI $\uparrow$	SPI $\uparrow$	CQI $\uparrow$	ORA $\uparrow$	AVAT $\uparrow$	CAT $\uparrow$	Avg. $\uparrow$
Aero-Realtime-4B	2.6	13.4	9.8	2.3	0.9	4.8	5.63
MiniCPM-o 4.5	13.2	18.7	11.6	8.8	2.3	10.3	10.82
DuplexOmni	11.2	16.5	9.5	7.3	14.2	8.7	11.23
Qwen3-Omni-30B-A3B-Instruct	16.6	21.5	17.2	12.3	21.2	14.1	17.15
GPT-Realtime-2.1	65.3	59.7	43.2	37.8	71.2	34.4	51.93

Table 4: Response quality on the six DIVE tasks. Scores are scaled by

\times 100

for readability (higher is better). Avg. is the unweighted mean across the six tasks.

Native Audio-Visual Models
Model	VTI $\downarrow$	SPI $\downarrow$	CQI $\downarrow$	ORA $\downarrow$	AVAT $\downarrow$	CAT $\downarrow$	Avg. $\downarrow$
Aero-Realtime-4B	2.475	2.583	1.912	3.206	3.502	1.724	2.567
MiniCPM-o 4.5	1.811	0.692	7.798	2.784	4.297	1.453	3.139
DuplexOmni	3.108	4.775	2.110	9.918	7.380	4.772	5.344
Qwen3-Omni-30B-A3B-Instruct	1.155	0.701	2.301	2.566	6.137	8.466	3.554
GPT-Realtime-2.1	0.552	0.413	0.395	1.393	1.671	0.561	0.831

Table 5: Response latency (seconds) on the six DIVE tasks. Latency is measured relative to the earliest valid response time, and lower is better.

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

May 17, 2026

Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction

Real-time duplex interaction is essential for multimodal AI systems operating in real-world scenarios, where models must continuously process streaming inputs and respond at appropriate moments. However, most existing multimodal large language models (MLLMs) are evaluated in offline settings, where the entire video input is processed before any response is generated. While recent work has started to explore real-time duplex MLLMs, there is still no comprehensive benchmark or automatic evaluation method for this setting. To address this gap, we propose Omni-DuplexEval, a benchmark for systematically evaluating real-time duplex interaction. The benchmark consists of two complementary scenarios: (1) Real-Time Description, which evaluates the ability to generate continuous, time-aligned responses that track evolving multimodal inputs, and (2) Proactive Reminder, which evaluates the ability to identify salient events and respond at appropriate moments. Omni-DuplexEval contains 660 videos with fine-grained, human-annotated labels and precise temporal metadata, spanning 9 tasks grounded in real-world scenarios, where all questions are formulated as open-ended queries. We further introduce an automatic evaluation framework based on LLM-as-a-Judge, which enables systematic assessment by jointly evaluating response-content alignment and response timing through timestamp-aware and sequential reasoning, achieving strong alignment with human judgments. Experiments on state-of-the-art duplex MLLMs reveal substantial limitations. The best-performing model achieves only 39.6% overall, while scoring only 20.0% on Proactive Reminder. Our analysis identifies two key challenges: models struggle to balance timely responses with coherent, holistic content generation, and they often fail to determine both when to respond and what to produce. We hope our work facilitates further progress in MLLMs.

Chaoqun He, Mingyang Xiang, Yingjing Xu +5

Tsinghua University · Shanghai Qi Zhi Institute · ModelBest Inc.

Multimodal Large Language Models Multimodal Agents Multimodal Model Evaluation Multimodal Understanding

Ask PDF

May 26, 2026

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants

We introduce OmniInteract, a streaming benchmark for real-time omnimodal large language models evaluated through native online inference over audio-visual streams. Unlike offline video understanding or text-prompted streaming QA, OmniInteract preserves the original audio-visual stream and requires models to process it online, without access to future content. User queries and ambient sounds are embedded in the audio track, requiring models to detect multimodal triggers, decide when to respond, and answer while the stream unfolds. OmniInteract contains 250 videos with 1,430 temporally grounded response slots: 1,062 1Q1A slots across real-time, proactive, and nested scenarios, and 368 1QnA slots for continuous task monitoring and step guidance. Each slot includes a trigger, response window, and target answer. We evaluate response correctness, timing, invalid outputs, interruption handling, and context continuity using Interaction-Aware Quality-Timeliness F1, Interruption Diagnostic Suite, and Nested Chain Completion Score. Experiments show that current models remain weak in streaming interaction, with the best overall IA-QTF1 reaching only 0.368 and the best 1QnA IA-QTF1 only 0.052. Further study on mathematical reasoning in full-duplex settings shows that offline capability does not necessarily transfer to online interaction. Code and datasets will be made publicly accessible at https://github.com/Lucky-Lance/OmniInteract.

Xudong Lu, Xueying Li, Annan Wang +8

CUHK MMLab · SJTU · NTU +3

Multimodal Agents Multimodal Understanding Multimodal Foundation Models Streaming Video Understanding Multimodal Model Evaluation Full-Duplex Spoken Dialogue Systems+1

Ask PDF

Apr 30, 2026

MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction

Recent progress in multimodal large language models (MLLMs) has brought AI capabilities from static offline data processing to real-time streaming interaction, yet they still remain far from human-level multimodal interaction. The key bottlenecks are no longer modality coverage or latency alone, but the interaction paradigm itself. First, perception and response are still separated into alternating phases, preventing models from incorporating new inputs for timely adjustment during generation. Second, most current models remain reactive, responding only to explicit user requests instead of acting proactively in the evolving multimodal environment. We present MiniCPM-o 4.5, our latest effort towards human-like multimodal interaction, which mitigates these gaps by real-time full-duplex omni-modal interaction. It can see, listen, and speak simultaneously in real-time, while also exhibiting proactive behaviors such as issuing reminders or comments based on its continuous understanding of the live scene. The key technique behind MiniCPM-o 4.5 is Omni-Flow, a unified streaming framework that aligns omni-modal inputs and outputs along a shared temporal axis. This formulation converts conventional turn-based interaction into a full-duplex, time-aligned process, enabling simultaneous perception and response and allowing proactive behavior to arise within the same framework. With a total of 9B parameters, MiniCPM-o 4.5 approaches Gemini 2.5 Flash in vision-language capabilities, delivering state-of-the-art open-source performance at its scale. It also surpasses Qwen3-Omni-30B-A3B in omni-modal understanding and delivers better speech generation, with significantly higher computation efficiency. Driven by its efficient architecture design and inference optimization, the model can perform real-time full-duplex omni-modal interaction on edge devices with less than 12GB RAM cost.

Junbo Cui, Bokai Xu, Chongyi Wang +33

MiniCPM-o Team, OpenBMB

Efficient Multimodal Inference Unified Multimodal Models Multimodal Large Language Models Multimodal Foundation Models Multimodal Agents Full-Duplex Spoken Dialogue Systems+1

Ask PDF

Abstract

Multimodal Model Evaluation

Explore similar work

Omni-DuplexEval: Evaluating Real-time Duplex Omni-modal Interaction

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants

MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction