The evolution of Omni-Modal Large Language Models~(Omni-LLMs) has revolutionized human--computer interaction, enabling unified audio-visual perception and speech response. However, existing Omni-LLMs struggle with complex real-world scenarios, often leading to superficial understanding and contextually mismatched emotional responses. This issue is further intensified by Omni-LLM's Thinker-Talker architectures, which are implicitly connected through hidden states, leading to the loss of emotional details. In this work, we present EmoOmni, a unified framework for accurate understanding and expression in multimodal emotional dialogue. At its core, we introduce the emotional Chain-of-Thought~(E-CoT), which enforces a reasoning from fine-grained multimodal perception to textual response. Moreover, we explicitly treat E-CoT as high-level emotional instructions that guide the talker, enabling accurate emotional expression. Complementing the model, we construct EmoOmniPipe to obtain the real-world annotated dialogue data and establish a benchmark, EmoOmniEval, to facilitate systematic assessment of multimodal emotional dialogue task. Experiments show that EmoOmni-7B achieves comparable performance with Qwen3Omni-30B-A3B-Thinking under the same talker.
Figures & tables
Figure 1: The overall framework of EmoOmni. The system mimics human affective cognition through a Perception-Reasoning-Expression causal chain.
Figure 2: Overview of the EmoOmniPipe. The data pipeline consists of three main stages: (1) processing of raw data, (2) multimodal annotation, and (3) construction of E-CoT and dialogue data.
Figure 3: Overview of EmoOmni. The left part illustrates the overall pipeline of our multimodal emotional dialogue system. Given audio–visual inputs, the Thinker module performs high-level reasoning and produces the emotional Chain-of-Thought, which consists of four components. The generated textual response is then fed into the talker module for expressive speech synthesis. The right part shows the architecture of the talker, which is built upon an autoregressive TTS model. The response strategy is further processed by a lightweight language model to generate controllable emotion instructions, enabling the talker to synthesize expressive speech.
Model/Metrics
Inference Mode
Param
MELD
ch-sims-v2
VS-RES ↑
VS-RC ↑
VT-EA ↑
VT-RES ↑
VT-RC ↑
IF ↑
VS-RES ↑
VS-RC ↑
VT-EA ↑
VT-RES ↑
VT-RC ↑
IF ↑
Gemini2.5Pro ( Comanici et al., 2025 )
-
-
1.42 ∗
1.60 ∗
1.97
1.98
1.86
1.29 ∗
1.87 ∗
1.92 ∗
2.00
2.00
1.99
1.60*
Intern-S1 ( Bai et al., 2025 )
thinking
9B
0.52 ∗
0.62 ∗
0.45
0.42
0.49
1.16 ∗
1.05 ∗
1.27 ∗
0.76
1.03
1.18
1.33 ∗
MiniCPM-o2.6 ( Team, 2025 )
no-thinking
8B
0.56
0.82
-
-
0.90
-
0.60
0.89
-
-
1.01
-
Qwen3Omni-Instruct ( Xu et al., 2025b )
no-thinking
30B
0.91
1.25
-
-
1.22
-
1.43
1.60
-
-
1.59
-
Qwen3Omni-Thinking ( Xu et al., 2025b )
thinking
30B
1.28 ∗
1.34 ∗
1.32
1.57
1.47
1.34 ∗
1.81 ∗
1.91 ∗
1.56
1.86
1.89
1.70 ∗
Table 1: Comparison of model performance on EmoOmniEval, which includes two sets: MELD and ch-sims-v2. ∗ denotes that the original model cannot synthesize speech and we therefore use EmoOmni-Talker to generate speech from its textual output for fair comparison. “Param” indicates the parameters of the models. The best and second-best results are shown in bold and underlined, respectively.
Model/Metrics
VS-RES ↑
VS-RC ↑
VT-EA ↑
VT-RES ↑
VT-RC ↑
EmoOmni
w/ E-CoT
1.67
1.75
1.97
1.99
1.81
w/o Emotion Analysis
1.53
1.61
-
1.58
1.62
w/o User Intent
1.63
1.70
1.96
1.93
1.73
w/o Response Strategy
-
-
1.97
-
1.57
w ASR
1.59
1.65
1.75
1.76
1.69
Table 2: Ablation study of each component within EmoOmni-Thinker conducted on ch-sims-v2 in EmoOmniEval. The best and second-best results are shown in bold and underlined, respectively.
Model/Metrics
VS-RES ↑
VS-RC ↑
IF ↑
WER ↓ (%)
UTMOS ↑
EmoOmni-Thinker
w/ EmoOmni-Talker
1.67
1.75
1.57
4.72
2.69
w/ VoiceSculptor ( Hu et al., 2026 )
1.64
1.71
1.48
6.62
2.67
w/ Llasa-3B ( Ye et al., 2025 )
1.61
1.71
1.30 ∗
4.45
2.73
Table 3: Ablation study on EmoOmni’s performance when paired with different talkers conducted on ch-sims-v2 in EmoOmniEval. ∗ denotes that the original model lacks instruction control capability, we directly evaluate IF score between the generated speech and the fixed instruction. The best results are shown in bold.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
ch2-sims-v2
Model/Metrics
WER ↓ (%)
N-MOS ↑
MiniCPM-o2.6
12.12
3.39 ± 0.29
Qwen3Omni-Instruct
3.35
4.08 ± 0.37
Qwen2.5Omni
1.17
3.96 ± 0.44
EmoOmni
4.72
4.17 ± 0.21
Appendix
Table 4: Speech Generation Performance on ch2-sims-v2 Dataset
Subjective
ch2-sims-v2
Model/Metrics
VS-RES-MOS ↑
VS-RC-MOS ↑
Gemini2.5Pro
1.78 ± 0.14
1.98 ± 0.11
Intern-S1
0.88 ± 0.22
1.01 ± 0.14
MiniCPM-o2.6
0.33 ± 0.07
0.46 ± 0.08
Qwen3Omni-Instruct
1.30 ± 0.16
1.60 ± 0.14
Qwen3Omni-Thinking
1.68 ± 0.12
1.95 ± 0.13
Appendix
Table 5: Subjective Human Evaluation Results on ch2-sims-v2 Dataset
Model/Metrics
VS-RES ↑
VS-RC ↑
IF ↑
WER ↓ (%)
UTMOS ↑
EmoOmni-Thinker
w/ EmoOmni-Talker
1.67
1.75
1.57
4.72
2.69
w/ Seed-tts-2.0
1.70
1.77
1.60
1.73
3.22
w/ Qwen3-TTS
1.76
1.79
1.81
2.40
3.03
Appendix
Table 6: Ablation study on EmoOmni’s performance when paired with different talkers conducted on ch-sims-v2 in EmoOmniEval.
Dataset
Language
Emotion Understanding Data
Emotion Dialogue Data
Scale
Source
Scale
Source
MEIJU25 (CN) ( Liu et al., 2025 )
CN
114.1k
TV
50.6k
Gemini
MEIJU25 (EN) ( Liu et al., 2025 )
EN
85.9k
TV
39.1k
Gemini
MER25 ( Lian et al., 2025b )
CN
132.2k
TV
114.7k
Gemini
MELD ( Poria et al., 2019 )
EN
13.7k
TV
3.1k
GroundTruth
MultiDialog ( Park et al., 2024 )
EN
187.9k
Acted
85.3k
GroundTruth
Appendix
Table 7: Overview of training data. “Gemini” indicates the answer is generated by Gemini2.5Pro.
Task1 Data/Hours
VT-EA ↑
VT-RES ↑
VT-RC ↑
0
1.88
1.89
1.67
100
1.97
1.99
1.81
500
1.97
1.96
1.71
1000
1.96
1.98
1.75
Appendix
Table 8: The impact of data volume on multi-task in task 1 learning performance evaluated on the ch2-sims-v2 test set.
Emotion recognition in conversation has been widely studied, but applying Large Language Models (LLMs) to continuous dimensional emotion evaluation in multimodal dialogue remains largely unexplored. We propose an LLM-based framework that performs discrete emotion recognition and Valence-Arousal-Dominance (VAD) dimensional evaluation on IEMOCAP, incorporating acoustic cues as natural language descriptions following the SpeechCueLLM approach. We evaluate six models spanning the LLaMA, GPT, and Qwen families under zero-shot prompting, few-shot prompting, and LoRA fine-tuning. LoRA fine-tuned LLaMA models substantially outperform prompt-engineered GPT models on both tasks despite GPT's larger scale, a gap we attribute to domain adaptation rather than model capacity. Our best model achieves a Valence CCC of 0.7822, a new state-of-the-art on IEMOCAP. Ablation studies confirm that textual audio descriptions meaningfully improve smaller models (+3.5 to 3.6 weighted F1) while contributing little for the largest model, suggesting audio cues are most valuable when linguistic capacity is limited. The performance asymmetry across VAD dimensions closely mirrors the annotator agreement hierarchy in IEMOCAP's own annotations.
Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, but a spoken answer still leaves the agent visually absent. We introduce Ex-Omni-2D, a framework that answers a multimodal query with coordinated text, personalized speech, and reference-conditioned video. The dialogue model first writes a structured Visual Thought Plan (VTP) for scene, emotion, and motion, then generates the response text and multi-codebook speech units. These speech units are decoded into audio and aligned with video frames, giving the speech and avatar modules a common timing signal while allowing them to learn from different data sources. The video module is trained as a full-sequence Teacher conditioned on reference appearance, VTP semantics, and frame-aligned speech units. We further explore to distill it into a few-step block-causal Streaming Student; its Prefix Streaming mechanism carries the previous clean latent into the next chunk and is analyzed as a partial mitigation for late-chunk subject drift. At 400×720/720×400, the four-step four-GPU Student provides incremental output with lower startup latency than the full-sequence Teacher.
Haoyu Zhang, Zhipeng Li, Xiaoying Tang +2
The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher
Multimodal Large Language Models (MLLMs) have shown remarkable progress in visual reasoning and understanding tasks but still struggle to capture the complexity and subjectivity of human emotions. Existing approaches based on supervised fine-tuning often suffer from limited generalization and poor interpretability, while reinforcement learning methods such as Group Relative Policy Optimization fail to align with the intrinsic characteristics of emotional cognition. To address these challenges, we propose Reflective Reinforcement Learning for Emotional Reasoning (EMO-R3), a framework designed to enhance the emotional reasoning ability of MLLMs. Specifically, we introduce Structured Emotional Thinking to guide the model to perform step-by-step emotional reasoning in a structured and interpretable manner, and design a Reflective Emotional Reward that enables the model to re-evaluate its reasoning based on visual-text consistency and emotional coherence. Extensive experiments demonstrate that EMO-R3 significantly improves both the interpretability and emotional intelligence of MLLMs, achieving superior performance across multiple visual emotional understanding benchmarks.
Yiyang Fang, Wenke Huang, Pei Fu +5
School of Computer Science, Wuhan University. · MiLM Plus, Xiaomi Inc.