Empathetic spoken dialogue is a sophisticated cognitive process that requires not only recognizing emotions but also inferring a user's latent mental states to provide appropriate support. However, current SpeechLLMs often treat empathy as a direct input-to-response mapping, leading to "superficially warm" but emotionally hollow interactions. In addition, since empathy relies on a multi-stage process with strong inter-step dependency, errors at any intermediate step can cascade through subsequent steps and lead to inappropriate responses, while existing training paradigms lack mechanisms to precisely localize and improve such errors. In this work, we propose EchoChat, a unified framework that reformulates empathetic spoken dialogue as a structured cognitive reasoning process integrating perception, mental-state reasoning, and response generation. To support this paradigm, we first construct EchoDialogue-400K, an acoustically rich dataset for multi-stage empathetic supervision. During the SFT stage, we strengthen acoustic grounding through proposed Acoustic-Anchored Attention (AAA). During the RL stage, we further introduce a novel stage-aware optimization objective with Step-Decomposed Credit Assignment (SDCA) to localize reasoning errors and mitigate cascaded error propagation. In addition, we introduce EchoEval, an expert-annotated benchmark for multi-dimensional empathy evaluation. Extensive experiments demonstrate that EchoChat achieves state-of-the-art performance in perception, reasoning, and response alignment. Project page: https://github.com/dingdongwang/EchoChat
Figures & tables
Figure 1: Reframing empathetic dialogue as structured cognitive reasoning. Existing methods simplify empathy to direct response generation, leading to cascaded reasoning errors across intermediate stages. EchoChat instead models empathy as a structured process from perception to response, supported by EchoDialogue-400K and EchoEval, enabling cognitively grounded and emotionally aligned spoken interaction.
Dataset
Scale
Open Source
Multi-Turn
Reason.
Mental State
Prosody Labels
Intensity
Sarcasm
Affective Granularity
EChat-200K Geng et al. (2025)
200K
✗
✗
✗
✗
✗
✗
✗
Coarse emotion labels
EmotionalQA Wang et al. (2025b)
200K
✗
✗
✗
✗
✗
✗
✗
Coarse emotion labels
ParaS2S Yang et al. (2025)
100K
✗
✗
✗
✗
✗
✗
✓
Coarse emotion labels
OpenS2S Wang et al. (2025a)
100K
✓
✗
✓
✗
✗
✗
✗
Coarse emotion labels
EchoDialogue-400K
400K
✓
✓
✓
✓
✓
✓
✓
Fine-grained variants
Table 1: Comparison with existing speech empathetic dialogue datasets. EchoDialogue-400K supports structured empathetic reasoning with fine-grained affective annotations and rich acoustic variations.
Figure 2: Data construction pipeline for EchoDialogue-400K.
Figure 3: Overview of the EchoChat framework. (a) SFT uses Acoustic-Anchored Attention (AAA) to ground reasoning in acoustic signals. (b) RL performs structured reasoning optimization, where Step-Decomposed Credit Assignment (SDCA) and gating mitigate error propagation, and a response reward model aligns content and speech. Unlike vanilla GRPO, credit is stage-aware at the token level rather than uniform.
Figure 4: EchoEval benchmark overview. Left: diverse empathetic dialogue scenarios. Right: a three-level evaluation pipeline (perception, reasoning, response) that measures models’ empathetic ability.
Models
Perception ↑
Reasoning ↑
Response ↑
All ↑
Implicit
Neutral
Conflict
Intense
Shift
Avg.
Avg.
Content
Audio
Avg.
Avg.
General Speech Large Language Models
GLM-4-Voice
5.23
4.67
4.26
5.06
4.05
4.75
2.57
2.87
3.85
6.72
4.68
Step-Audio2 mini
3.57
3.15
3.12
2.13
1.41
2.80
1.80
2.15
3.22
5.37
3.32
Kimi-Audio
6.61
6.53
5.74
7.32
7.62
6.67
3.70
3.45
3.42
6.87
5.75
Mimo-Audio-7B-Instruct
6.98
5.70
5.62
8.73
8.44
6.96
6.61
4.12
3.80
7.92
7.16
Table 2: Performance comparison across models on EchoEval. Perception, reasoning, and response scores are reported on a 10-point scale; the response score consists of content quality (5 points) and audio quality (5 points). For speech-to-text models (MERaLiON and BLSP-Emo), the response score is computed using content quality only. The top two results are highlighted in bold and underline , respectively.
Dimension
ρ
Perception
0.88
Reasoning
0.77
Response
0.83
All (Avg.)
0.83
Table 3: Agreement between LLM-based and human judgments (Spearman’s ρ ).
Table 8Table 9
Figure 5: AAA doubles the attention mass that perception tokens place on acoustic input ρˉ , indicating it better grounds reasoning in audio rather than relying on textual priors.
Model
P(R∣P)↑
P(Resp∣R)↑
Err. ↓
Baseline
0.52
0.48
0.41
w/o gating
0.60
0.51
0.36
w/o SDCA
0.58
0.55
0.33
EchoChat
0.71
0.68
0.21
Table 8: Error propagation analysis. We report conditional correctness across stages and the overall error cascade ratio.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Overview of the topic taxonomy in EchoDialogue-400K.
Statistics
Number
Total samples (single-turn)
400K
Total samples (multi-turn)
4K sessions
Turns per multi-turn session
5
Total audio hours (query)
9.8K hours
Total audio hours (response)
3.3K hours
Average query length
24 words
Appendix
Table 9: Key statistics of EchoDialogue-400K.
Statistics
Number
Total user utterances
1,000
Scenario count
5
Implicit utterance
220 (22.00%)
Audio-dominant utterance
220 (22.00%)
High-arousal utterance
220 (22.00%)
Modality conflict utterance
220 (22.00%)
Appendix
Table 10: Key statistics of the EchoEval benchmark.
Model
Reasoning
Implicit Utterance
User Query: “I lied to myself again about starting the literature review, and now I’m ashamed because it’s the fourth time I’ve procrastinated this month. I hate this pattern; can you help me face it without beating myself up?” (Audio emotion: contempt)
Reasoning and Response: <think>
[Step 1: Perception Analysis]: Situation Framing: The speaker is describing repeated procrastination on a literature review and feeling ashamed and fed up with the pattern. Emotional Valence: Negative. Prosodic Cues: High pitch, normal speed and energy, falling intonation, and emphasis on words like “again,” “starting,” “literature,” “review,” and “fourth,” which conveys frustration and self-directed criticism. Primary Emotion and Intensity: Dominant emotion is self-directed disgust/anger mixed with shame; intensity is normal (noticeable but not overwhelming).
[Step 2: Mental State Modeling]: Mental State Attribution: She likely feels frustrated, disappointed in herself, and stuck—thinking this is a personal failure rather than a manageable setback. Underlying Drivers: A desire for competence and control, fear of falling behind, and self-expectations that make small delays feel like moral failures. Psychological Needs: Most urgent is to be understood and to regain a sense of agency without harsh self-judgment. Constraints: She likely does not want lecturing, moralizing, or being told to “just try harder.”
[Step 3: Support Strategy Selection]: Support Strategy Explanation: Primary strategy will be emotional validation combined with collaborative exploration and gentle guidance—first acknowledge the shame and frustration to reduce self-blame, then offer small, concrete steps to restore control. This addresses her need to be heard and to regain agency without shaming. Response Audio Style Selection: ⟨ calm ⟩
</think>
Appendix
Table 11: Case study of EchoChat on EchoEval’s all scenarios.
Figure 7: EchoDialogue-400K query data construction prompts.
Figure 8: EchoDialogue-400K reasoning and response data construction prompts.
Figure 9: EchoDialogue-400K multi-turn data construction prompts.
Figure 10: EchoEval inference prompts.
Figure 11: EchoEval evaluation prompts for perception and reasoning assessment.
Figure 12: EchoEval evaluation prompts for response assessment.
Emotion-aware dialogue generation adapts conversational responses to users’ feelings, context, and interaction goals across text, speech, and multimodal settings.
Empathetic spoken dialogue requires models to use both what is said and how it is said to decide how to respond. Explicit CoT can improve paralinguistic perception and make acoustic cues more explicit in replies, yet does not ensure their effective use in response planning. We call this mismatch the perception-reasoning gap. In addition, CoT may not fully capture acoustic cues in words, and generating it adds inference latency. To address these limitations, we introduce LoopSLM, which builds on looped Transformers for latent reasoning, reusing a decoder block to refine hidden states with acoustic grounding at every pass. Its two-stage training further narrows the perception-reasoning gap by separating learning to reason from learning to respond, enabling direct inference without CoT. On EchoMind, LoopSLM improves paralinguistic understanding, reasoning, and reply quality over Qwen2.5-Omni-7B. Against the CoT-SFT baseline, LoopSLM gains over 20 points in reasoning accuracy while generating 64.5% fewer tokens at half the latency. It also outperforms Qwen3-Omni-Thinking on most empathetic reply metrics with 34x lower latency. Despite training only on dialogue data, LoopSLM improves accuracy on general audio benchmarks.
Shengbo Cai, Yuxiang Wang, Jingran Xie +5
Tsinghua University · Tencent Hunyuan · The Chinese University of Hong Kong, Shenzhen
Empathetic spoken dialogue systems require not only semantically appropriate responses but also emotionally aligned prosodic expression. However, cascade pipelines often discard acoustic cues during speech-to-text conversion, while end-to-end speech models lack interpretable control over emotion and knowledge integration. To address these challenges, we propose PRISM, a multi-agent framework for empathetic spoken dialogue that decouples speech perception, response generation, and speech synthesis into coordinated components. PRISM introduces a prosody-to-language translation mechanism to stabilize large language model reasoning and enables on-demand invocation of external knowledge tools for empathetic dialogue generation. Experimental results demonstrate that PRISM achieves consistent improvements in empathy, prosodic appropriateness, and text response generation quality across objective and subjective metrics. Our code is available at: https://github.com/Bxzfrm/PRISM.
Wen Zhang, Xiaocui Yang, Zhuoyue Gao +3
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China
In human conversation, empathy involves not only recognizing a partner's emotion but also regulating it, listeners identify the emotional state, select a suitable regulation strategy, and implement it in their reply. Current large audio-language models (LALMs) largely map perceived emotion directly onto a response, skipping this regulation process. As a result, their responses tend to mirror the user's affect or offer generic sympathy. We propose ER-EDF, a psychology-grounded framework for empathetic dialogue that follows the three stages of Gross's Extended Process Model of emotion regulation, identifying the user's emotion from speech, selecting a regulation strategy matched to that emotion, and implementing the strategy through a LALM. The framework is model-agnostic and integrates seamlessly into existing LALMs. We further construct a spoken empathetic dialogue dataset and introduce empathy-aware evaluation metrics beyond lexical matching. Experiments across five LALMs and two datasets show that ER-EDF consistently improves empathetic response quality in both automatic and human evaluations. These results suggest that modeling how humans regulate others' emotions, rather than only how they perceive them, is key to empathetic spoken dialogue, paving the way for psychologically grounded empathetic AI.