EmoOmni: Bridging Emotional Understanding and Expression in Omni-Modal LLMs

Wenjie Tian; Zhixian Zhao; Jingbin Hu; Huakang Chen; Haohe Liu; Binshen Mu; Lei Xie

cs.SDFeb 25, 2026

EmoOmni: Bridging Emotional Understanding and Expression in Omni-Modal LLMs

Authors: Wenjie Tian, Zhixian Zhao, Jingbin Hu, Huakang Chen, Haohe Liu, Binshen Mu, Lei Xie

Organizations: Northwestern Polytechnical University, Xi’an, China · University of Surrey, Guildford, United Kingdom

Abstract

The evolution of Omni-Modal Large Language Models~(Omni-LLMs) has revolutionized human--computer interaction, enabling unified audio-visual perception and speech response. However, existing Omni-LLMs struggle with complex real-world scenarios, often leading to superficial understanding and contextually mismatched emotional responses. This issue is further intensified by Omni-LLM's Thinker-Talker architectures, which are implicitly connected through hidden states, leading to the loss of emotional details. In this work, we present EmoOmni, a unified framework for accurate understanding and expression in multimodal emotional dialogue. At its core, we introduce the emotional Chain-of-Thought~(E-CoT), which enforces a reasoning from fine-grained multimodal perception to textual response. Moreover, we explicitly treat E-CoT as high-level emotional instructions that guide the talker, enabling accurate emotional expression. Complementing the model, we construct EmoOmniPipe to obtain the real-world annotated dialogue data and establish a benchmark, EmoOmniEval, to facilitate systematic assessment of multimodal emotional dialogue task. Experiments show that EmoOmni-7B achieves comparable performance with Qwen3Omni-30B-A3B-Thinking under the same talker.

Figures & tables

Figure 1: The overall framework of EmoOmni. The system mimics human affective cognition through a Perception-Reasoning-Expression causal chain.

Figure 2: Overview of the EmoOmniPipe. The data pipeline consists of three main stages: (1) processing of raw data, (2) multimodal annotation, and (3) construction of E-CoT and dialogue data.

Figure 3: Overview of EmoOmni. The left part illustrates the overall pipeline of our multimodal emotional dialogue system. Given audio–visual inputs, the Thinker module performs high-level reasoning and produces the emotional Chain-of-Thought, which consists of four components. The generated textual response is then fed into the talker module for expressive speech synthesis. The right part shows the architecture of the talker, which is built upon an autoregressive TTS model. The response strategy is further processed by a lightweight language model to generate controllable emotion instructions, enabling the talker to synthesize expressive speech.

Model/Metrics	Inference Mode	Param	MELD						ch-sims-v2
Model/Metrics	Inference Mode	Param	VS-RES $\uparrow$	VS-RC $\uparrow$	VT-EA $\uparrow$	VT-RES $\uparrow$	VT-RC $\uparrow$	IF $\uparrow$	VS-RES $\uparrow$	VS-RC $\uparrow$	VT-EA $\uparrow$	VT-RES $\uparrow$	VT-RC $\uparrow$	IF $\uparrow$
Gemini2.5Pro ( Comanici et al., 2025 )	-	-	1.42 ∗	1.60 ∗	1.97	1.98	1.86	1.29 ∗	1.87 ∗	1.92 ∗	2.00	2.00	1.99	1.60*
Intern-S1 ( Bai et al., 2025 )	thinking	9B	0.52 ∗	0.62 ∗	0.45	0.42	0.49	1.16 ∗	1.05 ∗	1.27 ∗	0.76	1.03	1.18	1.33 ∗
MiniCPM-o2.6 ( Team, 2025 )	no-thinking	8B	0.56	0.82	-	-	0.90	-	0.60	0.89	-	-	1.01	-
Qwen3Omni-Instruct ( Xu et al., 2025b )	no-thinking	30B	0.91	1.25	-	-	1.22	-	1.43	1.60	-	-	1.59	-
Qwen3Omni-Thinking ( Xu et al., 2025b )	thinking	30B	1.28 ∗	1.34 ∗	1.32	1.57	1.47	1.34 ∗	1.81 ∗	1.91 ∗	1.56	1.86	1.89	1.70 ∗

Table 1: Comparison of model performance on EmoOmniEval, which includes two sets: MELD and ch-sims-v2. ∗ denotes that the original model cannot synthesize speech and we therefore use EmoOmni-Talker to generate speech from its textual output for fair comparison. “Param” indicates the parameters of the models. The best and second-best results are shown in bold and underlined, respectively.

Model/Metrics	VS-RES $\uparrow$	VS-RC $\uparrow$	VT-EA $\uparrow$	VT-RES $\uparrow$	VT-RC $\uparrow$
EmoOmni
w/ E-CoT	1.67	1.75	1.97	1.99	1.81
w/o Emotion Analysis	1.53	1.61	-	1.58	1.62
w/o User Intent	1.63	1.70	1.96	1.93	1.73
w/o Response Strategy	-	-	1.97	-	1.57
w ASR	1.59	1.65	1.75	1.76	1.69

Table 2: Ablation study of each component within EmoOmni-Thinker conducted on ch-sims-v2 in EmoOmniEval. The best and second-best results are shown in bold and underlined, respectively.

Model/Metrics	VS-RES $\uparrow$	VS-RC $\uparrow$	IF $\uparrow$	WER $\downarrow$ (%)	UTMOS $\uparrow$
EmoOmni-Thinker
w/ EmoOmni-Talker	1.67	1.75	1.57	4.72	2.69
w/ VoiceSculptor ( Hu et al., 2026 )	1.64	1.71	1.48	6.62	2.67
w/ Llasa-3B ( Ye et al., 2025 )	1.61	1.71	1.30 ∗	4.45	2.73

Table 3: Ablation study on EmoOmni’s performance when paired with different talkers conducted on ch-sims-v2 in EmoOmniEval. ∗ denotes that the original model lacks instruction control capability, we directly evaluate IF score between the generated speech and the fixed instruction. The best results are shown in bold.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

ch2-sims-v2
Model/Metrics	WER $\downarrow$ (%)	N-MOS $\uparrow$
MiniCPM-o2.6	12.12	3.39 $\pm$ 0.29
Qwen3Omni-Instruct	3.35	4.08 $\pm$ 0.37
Qwen2.5Omni	1.17	3.96 $\pm$ 0.44
EmoOmni	4.72	4.17 $\pm$ 0.21

Appendix

Table 4: Speech Generation Performance on ch2-sims-v2 Dataset

Subjective	ch2-sims-v2
Model/Metrics	VS-RES-MOS $\uparrow$	VS-RC-MOS $\uparrow$
Gemini2.5Pro	1.78 $\pm$ 0.14	1.98 $\pm$ 0.11
Intern-S1	0.88 $\pm$ 0.22	1.01 $\pm$ 0.14
MiniCPM-o2.6	0.33 $\pm$ 0.07	0.46 $\pm$ 0.08
Qwen3Omni-Instruct	1.30 $\pm$ 0.16	1.60 $\pm$ 0.14
Qwen3Omni-Thinking	1.68 $\pm$ 0.12	1.95 $\pm$ 0.13

Appendix

Table 5: Subjective Human Evaluation Results on ch2-sims-v2 Dataset

Model/Metrics	VS-RES $\uparrow$	VS-RC $\uparrow$	IF $\uparrow$	WER $\downarrow$ (%)	UTMOS $\uparrow$
EmoOmni-Thinker
w/ EmoOmni-Talker	1.67	1.75	1.57	4.72	2.69
w/ Seed-tts-2.0	1.70	1.77	1.60	1.73	3.22
w/ Qwen3-TTS	1.76	1.79	1.81	2.40	3.03

Appendix

Table 6: Ablation study on EmoOmni’s performance when paired with different talkers conducted on ch-sims-v2 in EmoOmniEval.

Dataset	Language	Emotion Understanding Data		Emotion Dialogue Data
Dataset	Language	Scale	Source	Scale	Source
MEIJU25 (CN) ( Liu et al., 2025 )	CN	114.1k	TV	50.6k	Gemini
MEIJU25 (EN) ( Liu et al., 2025 )	EN	85.9k	TV	39.1k	Gemini
MER25 ( Lian et al., 2025b )	CN	132.2k	TV	114.7k	Gemini
MELD ( Poria et al., 2019 )	EN	13.7k	TV	3.1k	GroundTruth
MultiDialog ( Park et al., 2024 )	EN	187.9k	Acted	85.3k	GroundTruth

Appendix

Table 7: Overview of training data. “Gemini” indicates the answer is generated by Gemini2.5Pro.

Task1 Data/Hours	VT-EA $\uparrow$	VT-RES $\uparrow$	VT-RC $\uparrow$
0	1.88	1.89	1.67
100	1.97	1.99	1.81
500	1.97	1.96	1.71
1000	1.96	1.98	1.75

Appendix

Table 8: The impact of data volume on multi-task in task 1 learning performance evaluated on the ch2-sims-v2 test set.

Explore similar work

Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue
Sep 30, 2026Yutong Hu, Jinho ChoiEmotion Recognition in Conversations Multimodal Emotion Recognition
Emory University Atlanta, GA
Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence
Aug 11, 2026Haoyu Zhang, Zhipeng Li, Xiaoying Tang +2Audio-Video Generation Streaming Video Generation
The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher
EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models
Feb 27, 2026Yiyang Fang, Wenke Huang, Pei Fu +5Multimodal Large Language Models Multimodal Reasoning
School of Computer Science, Wuhan University. · MiLM Plus, Xiaomi Inc.

Sep 30, 2026J/K move · Enter open · S save

Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue

Yutong Hu, Jinho Choi

Emory University Atlanta, GA

Emotion Recognition in Conversations Multimodal Emotion Recognition Emotion Recognition

Open paper AskPDF arXiv

Emotion recognition in conversation has been widely studied, but applying Large Language Models (LLMs) to continuous dimensional emotion evaluation in multimodal dialogue remains largely unexplored. We propose an LLM-based framework that performs discrete emotion recognition and Valence-Arousal-Dominance (VAD) dimensional evaluation on IEMOCAP, incorporating acoustic cues as natural language descriptions following the SpeechCueLLM approach. We evaluate six models spanning the LLaMA, GPT, and Qwen families under zero-shot prompting, few-shot prompting, and LoRA fine-tuning. LoRA fine-tuned LLaMA models substantially outperform prompt-engineered GPT models on both tasks despite GPT's larger scale, a gap we attribute to domain adaptation rather than model capacity. Our best model achieves a Valence CCC of 0.7822, a new state-of-the-art on IEMOCAP. Ablation studies confirm that textual audio descriptions meaningfully improve smaller models (+3.5 to 3.6 weighted F1) while contributing little for the largest model, suggesting audio cues are most valuable when linguistic capacity is limited. The performance asymmetry across VAD dimensions closely mirrors the annotator agreement hierarchy in IEMOCAP's own annotations.

Abstract

Explore similar work

Beyond Text: LLM-Based Dimensional Emotion Evaluation in Multimodal Dialogue

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models