HumDial-EIBench: A Human-Recorded Multi-Turn Emotional Intelligence Benchmark for Audio Language Models

Shuiyuan Wang; Zhixian Zhao; Hongfei Xue; Chengyou Wang; Shuai Wang; Hui Bu; Xin Xu; Lei Xie

eess.ASApr 13, 2026

HumDial-EIBench: A Human-Recorded Multi-Turn Emotional Intelligence Benchmark for Audio Language Models

Authors: Shuiyuan Wang, Zhixian Zhao, Hongfei Xue, Chengyou Wang, Shuai Wang, Hui Bu, Xin Xu, Lei Xie

Organizations: Nanjing University, China · AISHELL, China

Abstract

Evaluating the emotional intelligence (EI) of audio language models (ALMs) is critical. However, existing benchmarks mostly rely on synthesized speech, are limited to single-turn interactions, and depend heavily on open-ended scoring. This paper proposes HumDial-EIBench, a comprehensive benchmark for evaluating ALMs' EI. Using real-recorded human dialogues from the ICASSP 2026 HumDial Challenge, it reformulates emotional tracking and causal reasoning into multiple-choice questions with adversarial distractors, mitigating subjective scoring bias for cognitive tasks. It retains the generation of empathetic responses and introduces an acoustic-semantic conflict task to assess robustness against contradictory multimodal signals. Evaluations of eight ALMs reveal that most models struggle with multi-turn emotional tracking and implicit causal reasoning. Furthermore, all models exhibit decoupled textual and acoustic empathy, alongside a severe text-dominance bias during cross-modal conflicts.

Figures & tables

Benchmark	Input Source	Turns	Targeted Capabilities				Eval Method
			Emotion	Causal	Empathetic	A-S
			Trajectory	Reasoning	Response	Conflict
Dynamic-SUPERB [ 12 ]	Mixed	Single	$\times$	$\times$	$\times$	$\times$	Obj.
AIR-Bench [ 13 ]	Human-rec.	Single	$\times$	$\times$	$\times$	$\times$	LLM / Human
HPSU [ 14 ]	Human-rec.	Single	$\times$	$\times$	$\times$	$\times$	Obj.
VoiceBench [ 15 ]	Mixed	Single	$\times$	$\times$	$\times$	$\times$	Obj. / LLM

Table 1: Comparison of HumDial-EIBench with existing benchmarks. A-S Conflict denotes Acoustic-Semantic Conflict.

Figure 1: Data construction and task overview of HumDial-EIBench. Left: Three-stage pipeline— Stage 1 : Dialogue Script Design; Stage 2 : Authentic Human Enactment and Quality Control; Stage 3 : Multiple-Choice Reformulation and Distractor Construction. Right: Representative examples of the four tasks.

Task	Type	Turns	Dialogue (CN/EN)	Utterance	Metric
Task 1	MCQ	3–5	150/150	1200	ACC
Task 2	MCQ	3–5	134/149	1132	ACC
Task 3	Gen.	3–5	144/150	1170	LLM/Hum
Task 4	MCQ	1	100/100	200	ACC

Table 2: Data and evaluation overview of HumDial-EIBench. Gen.: Generation.

Open-Source
Model	Task 1 (Trajectory)			Task 2 (Reasoning)
Model	CN	EN	Avg	CN	EN	Avg
Freeze-Omni	16.00	6.00	11.00	23.88	24.16	24.02
GLM-4-Voice	42.67	18.00	30.34	58.96	52.35	55.66
Kimi-Audio	70.00	62.67	66.34	71.64	65.77	68.71
Step-Audio-2-mini	34.67	34.67	34.67	67.16	76.51	71.84

Table 3: Accuracy (%) on Tasks 1 and 2. Bold : best in column; underline : second best.

Open-Source
Model	D1 (Q / G)		D2 / D3 (Human)
Model	CN	EN	CN	EN
Freeze-Omni	3.69 / 2.69	3.25 / 2.52	3.29 / 3.42	3.37 / 3.45
GLM-4-Voice	3.62 / 3.38	2.77 / 2.42	3.27 / 3.31	3.37 / 3.41
Kimi-Audio	3.62 / 3.59	1.17 / 1.19	3.52 / 3.60	3.50 / 3.49
Qwen2.5-Omni	3.41 / 3.19	3.37 / 2.87	3.55 / 3.65	3.49 / 3.59

Table 4: Scores on Task 3. D1: text empathy, scored by LLM judges (Qwen3-Omni / Gemini2.5-Pro); D2/D3: acoustic empathy / naturalness. Bold : best in column; underline : second best; italics : anomalous low.

Open-Source
Model	CN		EN
Model	Conflict	Nonconf.	Conflict	Nonconf.
Freeze-Omni	18.00	50.00	22.00	60.00
GLM-4-Voice	30.00	82.00	20.00	54.00
Kimi-Audio	48.00	80.00	38.00	80.00
Step-Audio-2-mini	24.00	82.00	26.00	82.00

Table 5: Accuracy (%) on Task 4. “Conflict” denotes samples where acoustic emotion contradicts text sentiment; “Nonconf.” denotes consistent samples. Bold : best in column; underline : second best.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 206 papers

EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models
Aug 10, 2026Junyu Wang, Siyuan Zhang, Peiyuan Jiang +11Speech Language Models
Tianjin Key Laboratory of Cognitive Computing and Application Tianjin University Tianjin, China · Tianjin University Tianjin, China · Fuzhou University Fuzhou, China +2
AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence
May 20, 2026Kate M. Lubrano, Faisal Sayed, Ankita Rathod +4LLM Evaluation Emotion Recognition in Conversations
1Pareto, California, US · 2Thoughtful, California, US · University of Pennsylvania, Pennsylvania, US
SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
Jun 24, 2026Liang-Yuan Wu, Zih-Ching Chen, Tongshuang Wu +2Audio-Language Model Evaluation Voice Agent Evaluation
New York University · NVIDIA · Carnegie Mellon University +1

Aug 10, 2026J/K move · Enter open · S save

EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models

Junyu Wang, Siyuan Zhang, Peiyuan Jiang, Jian Zong+10

Tianjin Key Laboratory of Cognitive Computing and Application Tianjin University Tianjin, China · Tianjin University Tianjin, China · Fuzhou University Fuzhou, China+2

Speech Language Models

Open paper AskPDF arXiv

Despite significant advances in instruction-following and auditory comprehension, the evaluation of Emotional Intelligence (EI) in Spoken Language Models (SLMs) remains confined to rudimentary paralinguistic perception, lacking a systematic, theory-driven cognitive framework. We introduce EmoSBench, the first comprehensive EI evaluation benchmark for SLMs constructed upon the four-branch theoretical model, covering Perceiving, Understanding, Using, and Managing Emotion across ten sub-tasks. Preliminary assessments on EmoSBench reveal a substantial gap: even leading proprietary models like GPT-4o-Audio achieve only 52.6%, significantly trailing human baselines. To bridge this gap, we develop EmoS, a specialized evaluator model optimized via Supervised Fine-Tuning (SFT) and Group Relative Policy Optimization (GRPO). To facilitate its effective training, we curate EmoDialogue, a bilingual dataset providing necessary fine-grained supervision through response pairs with rigorously defined EI gradations. Concurrently, we introduce a reward mechanism integrating a Steep Exponential Accuracy Reward (SEAR) and a Rationale Fidelity Reward (RFR) to enforce precise ordinal scoring and valid reasoning. Experiments demonstrate that EmoS reaches 83.8% accuracy, approaching human-level performance. Furthermore, evaluations on authentic, unconstrained spoken interactions validate its robust real-world generalization, establishing a foundational framework for advancing emotionally intelligent dialogue systems.

Abstract

Audio-Language Model Evaluation

Explore similar work

EmoS: A Theory-Grounded Framework for Evaluating and Aligning Emotional Intelligence in Spoken Language Models

AttuneBench: A Conversation-Based Benchmark for LLM Emotional Intelligence

SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models