OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

Haolin He; Yunfei Chu; Qi Chen; Wen Huang; Yuan Feng; Muzhi Zhu; Zheqi Dai; Haoning Xu

eess.ASSep 18, 2026

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

Authors: Haolin He, Yunfei Chu, Qi Chen, Wen Huang, Yuan Feng, Muzhi Zhu, Zheqi Dai, Haoning Xu, +10 more

Organizations: The Chinese University of Hong Kong · Alibaba Token Hub, Alibaba Group · Shanghai Jiao Tong University · Shanghai Innovation Institute · Zhejiang University

Abstract

We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, good replies often depend on multimodal context and can be phrased in many ways, making keyword matching unreliable for evaluation. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. Replies are judged by a large language model based on explicit scoring criteria. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.

Figures & tables

Figure 1: A glance at OmniVChat-Bench : each card shows one user frame from a synthesized dialogue. The benchmark spans 5 ability categories, 17 subcategories, and 22 scenario domains.

Figure 2: OmniVChat-Studio framework with single-turn (left) and multi-turn (right) subsystems. Each module uses the same color as the agent responsible for it.

Panel of Figure 4 — Figure 4: Synthesis of one single-turn dialogue, with each module’s decisions and outputs.

Figure 5: Overview of OmniVChat-Bench. Left: The five ability categories and their 17 subcategories. Bar length shows the instance count from the base ring. Angular span shows the number of subcategories in each band. Right: Turn and duration distributions for 250 multi-turn instances, with counts stacked by subcategory.

Figure 6: OmniVChat-RL over 1,000 iterations. Top: unweighted reward terms (Equation 4 ), reply efficiency, and mean length. Bottom: rubric scores (Equation 1 ) every twentieth step on the development set, OmniVChat-Bench, and OmniVChat-Bench-Human. Right includes a Chinese-gated synthetic control. The dashed line marks step 940. Scores across corpora are not directly comparable.

Closed-Source Models	AH	ER	MSA	MEA	DSLP	Single	Multi	Mean	Human	RE	Style
Gemini-3.5-Flash	0.600	0.778	0.649	0.728	0.623	0.664	0.682	0.667	0.509	10.59	0.755
Gemini-3.6-Flash	0.570	0.658	0.671	0.716	0.620	0.658	0.610	0.641	0.562	15.74	0.740
Gemini-3.7-Flash	0.604	0.623	0.663	0.708	0.620	0.658	0.607	0.640	0.575	16.96	0.631
Gemini-3.1-Pro	0.637	0.787	0.341	0.666	0.629	0.602	0.623	0.607	0.537	11.66	0.871
Qwen3.5-Omni-Plus	0.361	0.662	0.413	0.630	0.513	0.514	0.590	0.533	0.482	9.45	0.869
Doubao Seed 2.0 Lite	0.330	0.550	0.189	0.602	0.481	0.434	0.671	0.494	0.274	6.10	0.817

Table 1: Model comparison under one system prompt. Bold , straight , and wavy mark the top three ranks, excluding ablations. AH–DSLP, Single, and Multi give category, single-turn, and multi-turn means. Mean equally weights 17 subcategories. Human uses recorded dialogues; other columns use OmniVChat-Bench. Mean and Human aggregate

r(y)

, while Style averages

s(y)

from Equation 4 . RE measures rubric credit per thousand counted words, the evaluation counterpart of

e(y)

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Two-turn dialogue synthesis, showing shared setup and per-turn decisions and outputs.

Figure 8: OmniVChat-Bench-Human composition. The 360 recordings cover twelve single-turn subcategories with 30 recordings in each.

Judge	AH	ER	MSA	MEA	DSLP	Pooled	Mean	Gate
gpt-5.6-terra	0.587	0.628	0.655	0.729	0.634	0.660	0.651	99.9%
gpt-5.6-luna	0.585	0.620	0.653	0.694	0.626	0.646	0.638	99.8%
gpt-5.6-sol	0.622	0.673	0.675	0.747	0.636	0.678	0.673	99.9%
qwen3.6-flash	0.644	0.614	0.653	0.741	0.632	0.671	0.657	99.1%
qwen3.7-plus	0.595	0.593	0.638	0.702	0.619	0.643	0.631	99.9%
qwen3.7-max	0.605	0.614	0.663	0.712	0.619	0.653	0.641	99.9%

Appendix

Table 2: Agreement across six judges. Each judge receives the same replies; metrics use replies with valid judgments from all six judges. Pooled is earned over available non-gate points. Ability-category columns use this ratio within categories; Mean averages subcategory ratios. Gate is the Tier 0 language-pass rate. Category labels match those in Table 1 .

Policy And Objective		Rollout And Reward
Base Model	Qwen3-Omni-Instruct	Prompts Per Iteration	32
Adaptation	LoRA $r{=}64$ , $\alpha{=}256$ , Dropout 0	Samples Per Prompt $N$	4
Adapter Targets	Attention + MoE Experts	Completions Per Step	128
Adapter / Total Params	325M / 30B (3B Active)	Sampling	$T{=}1.0$ , Top- $k$ 20, Top- $p$ 1
Objective	GSPO-based (Eq. 3 )	Prompt / Reply Budget	11,072 / 1,024 Tokens
Clip Range	$(2,\,4)\times 10^{-4}$ (Lo, Hi)	Video Sampling	2 FPS, $\leq 16$ Frames

Appendix

Table 3: OmniVChat-RL training configuration . Reward weights follow Equation 4 . Each dialogue contains up to ten clips with timestamp-aligned audio. Video uses 2 fps sampling and at most 16 frames for each video clip.

Figure 9: Per-subcategory benchmark learning curves , grouped by ability category. Solid lines show OmniVChat-Bench, and dashed lines show OmniVChat-Bench-Human. The five multi-turn subcategories are marked Bench Only because human recordings cover only single-turn dialogues.

Figure 10: Reward ablations . Each row shows one run’s unweighted reward terms, absolute Reply Efficiency, and Subcategory Mean on the development set. The right-side label identifies the run. Dots mark checkpoints with benchmark evaluations, and the vertical line marks the checkpoint in Table 1 . Scales are shared within each column.

Explore similar work

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence
Aug 11, 2026Haoyu Zhang, Zhipeng Li, Xiaoying Tang +2Omni-Modal Omni
The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher
Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation
May 12, 2026Che Liu, Lichao Ma, Xiangyu Tony Zhang +4Omni-Modal Post-Training
StepFun-Audio Team
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
May 26, 2026Xudong Lu, Xueying Li, Annan Wang +8Omni-Modal Omni
CUHK MMLab · SJTU · NTU +3

Aug 11, 2026, cs.AIJ/K move · Enter open · S save

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

Haoyu Zhang, Zhipeng Li, Xiaoying Tang, Tianshu Yu+1

The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher

Omni-Modal Omni Dialogue

Open paper AskPDF arXiv

Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, but a spoken answer still leaves the agent visually absent. We introduce \textbf{Ex-Omni-2D}, a framework that answers a multimodal query with coordinated text, personalized speech, and reference-conditioned video. The dialogue model first writes a structured \textit{Visual Thought Plan} (VTP) for scene, emotion, and motion, then generates the response text and multi-codebook speech units. These speech units are decoded into audio and aligned with video frames, giving the speech and avatar modules a common timing signal while allowing them to learn from different data sources. The video module is trained as a full-sequence Teacher conditioned on reference appearance, VTP semantics, and frame-aligned speech units. We further explore to distill it into a few-step block-causal \emph{Streaming Student}; its Prefix Streaming mechanism carries the previous clean latent into the next chunk and is analyzed as a partial mitigation for late-chunk subject drift. At

400\times720

720\times400

, the four-step four-GPU Student provides incremental output with lower startup latency than the full-sequence Teacher.

Abstract

Explore similar work

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

Boosting Omni-Modal Language Models: Staged Post-Training with Visually Debiased Evaluation

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants