Organizations: The Chinese University of Hong Kong · Alibaba Token Hub, Alibaba Group · Shanghai Jiao Tong University · Shanghai Innovation Institute · Zhejiang University
We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly and simultaneously receive audio and video from a user and return text. The user's query is embedded in the audio and video, without a separate text question, external captioning, or speech recognition. Direct audio-visual input reduces external latency and computation while preserving perceptual cues. However, research on OmniVChat faces two constraints: data availability and evaluation. Recordings of people using their own devices are scarce. Furthermore, good replies often depend on multimodal context and can be phrased in many ways, making keyword matching unreliable for evaluation. Recent progress in agent systems and video generation makes generation for comprehension viable, which means using synthesized dialogues for training and evaluation. Therefore, we present OmniVChat-Studio, a multi-agent data engine for synthesizing single- and multi-turn audio-visual dialogues. We use synthesized dialogues to build OmniVChat-Bench, an evaluation benchmark that evaluates omni models' basic dialogue abilities across five ability categories. Replies are judged by a large language model based on explicit scoring criteria. We also present OmniVChat-RL, a reinforcement learning reward design that jointly targets reply correctness, efficiency, and style in OmniVChat. Training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues improves its performance on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. These gains validate the reward design and show transfer to real-world dialogues in training and evaluation.
Figures & tables
Figure 1: A glance at OmniVChat-Bench : each card shows one user frame from a synthesized dialogue. The benchmark spans 5 ability categories, 17 subcategories, and 22 scenario domains.
Figure 2: OmniVChat-Studio framework with single-turn (left) and multi-turn (right) subsystems. Each module uses the same color as the agent responsible for it.
Figure 4: Synthesis of one single-turn dialogue, with each module’s decisions and outputs.
Figure 5: Overview of OmniVChat-Bench. Left: The five ability categories and their 17 subcategories. Bar length shows the instance count from the base ring. Angular span shows the number of subcategories in each band. Right: Turn and duration distributions for 250 multi-turn instances, with counts stacked by subcategory.
Figure 6: OmniVChat-RL over 1,000 iterations. Top: unweighted reward terms (Equation 4 ), reply efficiency, and mean length. Bottom: rubric scores (Equation 1 ) every twentieth step on the development set, OmniVChat-Bench, and OmniVChat-Bench-Human. Right includes a Chinese-gated synthetic control. The dashed line marks step 940. Scores across corpora are not directly comparable.
Closed-Source Models
AH
ER
MSA
MEA
DSLP
Single
Multi
Mean
Human
RE
Style
Gemini-3.5-Flash
0.600
0.778
0.649
0.728
0.623
0.664
0.682
0.667
0.509
10.59
0.755
Gemini-3.6-Flash
0.570
0.658
0.671
0.716
0.620
0.658
0.610
0.641
0.562
15.74
0.740
Gemini-3.7-Flash
0.604
0.623
0.663
0.708
0.620
0.658
0.607
0.640
0.575
16.96
0.631
Gemini-3.1-Pro
0.637
0.787
0.341
0.666
0.629
0.602
0.623
0.607
0.537
11.66
0.871
Qwen3.5-Omni-Plus
0.361
0.662
0.413
0.630
0.513
0.514
0.590
0.533
0.482
9.45
0.869
Doubao Seed 2.0 Lite
0.330
0.550
0.189
0.602
0.481
0.434
0.671
0.494
0.274
6.10
0.817
Table 1: Model comparison under one system prompt. Bold , straight , and wavy mark the top three ranks, excluding ablations. AH–DSLP, Single, and Multi give category, single-turn, and multi-turn means. Mean equally weights 17 subcategories. Human uses recorded dialogues; other columns use OmniVChat-Bench. Mean and Human aggregate r(y) , while Style averages s(y) from Equation 4 . RE measures rubric credit per thousand counted words, the evaluation counterpart of e(y) .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Two-turn dialogue synthesis, showing shared setup and per-turn decisions and outputs.
Figure 8: OmniVChat-Bench-Human composition. The 360 recordings cover twelve single-turn subcategories with 30 recordings in each.
Judge
AH
ER
MSA
MEA
DSLP
Pooled
Mean
Gate
gpt-5.6-terra
0.587
0.628
0.655
0.729
0.634
0.660
0.651
99.9%
gpt-5.6-luna
0.585
0.620
0.653
0.694
0.626
0.646
0.638
99.8%
gpt-5.6-sol
0.622
0.673
0.675
0.747
0.636
0.678
0.673
99.9%
qwen3.6-flash
0.644
0.614
0.653
0.741
0.632
0.671
0.657
99.1%
qwen3.7-plus
0.595
0.593
0.638
0.702
0.619
0.643
0.631
99.9%
qwen3.7-max
0.605
0.614
0.663
0.712
0.619
0.653
0.641
99.9%
Appendix
Table 2: Agreement across six judges. Each judge receives the same replies; metrics use replies with valid judgments from all six judges. Pooled is earned over available non-gate points. Ability-category columns use this ratio within categories; Mean averages subcategory ratios. Gate is the Tier 0 language-pass rate. Category labels match those in Table 1 .
Policy And Objective
Rollout And Reward
Base Model
Qwen3-Omni-Instruct
Prompts Per Iteration
32
Adaptation
LoRA r=64 , α=256 , Dropout 0
Samples Per Prompt N
4
Adapter Targets
Attention + MoE Experts
Completions Per Step
128
Adapter / Total Params
325M / 30B (3B Active)
Sampling
T=1.0 , Top- k 20, Top- p 1
Objective
GSPO-based (Eq. 3 )
Prompt / Reply Budget
11,072 / 1,024 Tokens
Clip Range
(2,4)×10−4 (Lo, Hi)
Video Sampling
2 FPS, ≤16 Frames
Appendix
Table 3: OmniVChat-RL training configuration . Reward weights follow Equation 4 . Each dialogue contains up to ten clips with timestamp-aligned audio. Video uses 2 fps sampling and at most 16 frames for each video clip.
Figure 9: Per-subcategory benchmark learning curves , grouped by ability category. Solid lines show OmniVChat-Bench, and dashed lines show OmniVChat-Bench-Human. The five multi-turn subcategories are marked Bench Only because human recordings cover only single-turn dialogues.
Figure 10: Reward ablations . Each row shows one run’s unweighted reward terms, absolute Reply Efficiency, and Subcategory Mean on the development set. The right-side label identifies the run. Dots mark checkpoints with benchmark evaluations, and the vertical line marks the checkpoint in Table 1 . Scales are shared within each column.
Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, but a spoken answer still leaves the agent visually absent. We introduce \textbf{Ex-Omni-2D}, a framework that answers a multimodal query with coordinated text, personalized speech, and reference-conditioned video. The dialogue model first writes a structured \textit{Visual Thought Plan} (VTP) for scene, emotion, and motion, then generates the response text and multi-codebook speech units. These speech units are decoded into audio and aligned with video frames, giving the speech and avatar modules a common timing signal while allowing them to learn from different data sources. The video module is trained as a full-sequence Teacher conditioned on reference appearance, VTP semantics, and frame-aligned speech units. We further explore to distill it into a few-step block-causal \emph{Streaming Student}; its Prefix Streaming mechanism carries the previous clean latent into the next chunk and is analyzed as a partial mitigation for late-chunk subject drift. At 400×720/720×400, the four-step four-GPU Student provides incremental output with lower startup latency than the full-sequence Teacher.
Haoyu Zhang, Zhipeng Li, Xiaoying Tang +2
The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher
Omni-modal language models are intended to jointly understand audio, visual inputs, and language, but benchmark gains can be inflated when visual evidence alone is enough to answer a query. We study whether current omni-modal benchmarks separate visual shortcuts from genuine audio-visual-language evidence integration, and how post-training behaves under a visually debiased evaluation setting. We audit nine omni-modal benchmarks with visual-only probing, remove visually solvable queries, and retain full subsets when filtering is undefined or would make comparisons unstable. This yields OmniClean, a cleaned evaluation view with 8,551 retained queries from 16,968 audited queries. On OmniClean, we evaluate OmniBoost, a three-stage post-training recipe based on Qwen2.5-Omni-3B: mixed bi-modal SFT, mixed-modality RLVR, and SFT on self-distilled data. Balanced bi-modal SFT gives limited and uneven gains, RLVR provides the first broad improvement, and self-distillation reshapes the benchmark profile. After SFT on self-distilled data, the 3B model reaches performance comparable to, and in aggregate slightly above, Qwen3-Omni-30B-A3B-Instruct without using a stronger omni-modal teacher. These results show that omni-modal progress is easier to interpret when evaluation controls visual leakage, and that small omni-modal models can benefit from staged post-training with self-distilled omni-query supervision. Project page: https://cheliu-computation.github.io/omni/
We introduce OmniInteract, a streaming benchmark for real-time omnimodal large language models evaluated through native online inference over audio-visual streams. Unlike offline video understanding or text-prompted streaming QA, OmniInteract preserves the original audio-visual stream and requires models to process it online, without access to future content. User queries and ambient sounds are embedded in the audio track, requiring models to detect multimodal triggers, decide when to respond, and answer while the stream unfolds. OmniInteract contains 250 videos with 1,430 temporally grounded response slots: 1,062 1Q1A slots across real-time, proactive, and nested scenarios, and 368 1QnA slots for continuous task monitoring and step guidance. Each slot includes a trigger, response window, and target answer. We evaluate response correctness, timing, invalid outputs, interruption handling, and context continuity using Interaction-Aware Quality-Timeliness F1, Interruption Diagnostic Suite, and Nested Chain Completion Score. Experiments show that current models remain weak in streaming interaction, with the best overall IA-QTF1 reaching only 0.368 and the best 1QnA IA-QTF1 only 0.052. Further study on mathematical reasoning in full-duplex settings shows that offline capability does not necessarily transfer to online interaction. Code and datasets will be made publicly accessible at https://github.com/Lucky-Lance/OmniInteract.