Emotional Validation is a psychotherapy communication technique that involves recognizing, understanding, and explicitly acknowledging another person's feelings and actions, which strengthens alliance and reduces negative affect. To maximize the emotional support provided by validation, it is crucial to deliver it with appropriate timing and frequency. This study investigates validation timing detection from the speech perspective. Leveraging both paralinguistic and emotional information, we propose a paralinguistic- and emotion-aware model for validation timing detection without relying on textual context. Specifically, we first conduct continued self-supervised training and fine-tuning on different HuBERT backbones to obtain (i) a paralinguistics-aware Self-Supervised Learning (SSL) encoder and (ii) a multi-task speech emotion classification encoder. We then fuse these encoders and further fine-tune the combined model on the downstream validation timing detection task. Experimental evaluations on the TUT Emotional Storytelling Corpus (TESC) compare multiple models, fusion mechanisms, and training strategies, and demonstrate that the proposed approach achieves significant improvements over conventional speech baselines. Our results indicate that non-linguistic speech cues, when integrated with affect-related representations, carry sufficient signal to decide when validation should be expressed, offering a speech-first pathway toward more empathetic human-robot interaction.
Figures & tables
Figure 1: Architecture of the proposed method. The method consists of (1) a multitask speech emotion recognition model, (2) a paralinguistic-aware self-supervised learning model, and (3) feature fusion for the validation timing detection task.
Dataset
#Validation
#Non-Validation
TESC
489
881
-train
340
579
-val
74
157
-test
75
145
Table 1: Distribution of the dataset
V-Prec.
V-F1
NV-F1
M-F1
Language Model
BERT
39.96
41.62
66.76
54.19
ModernBERT
41.32
44.65
68.65
56.65
Llama 3.1 8B
- Zero-shot
33.91
49.04
40.48
44.76
- 3-shot
33.69
50.30
46.10
48.20
Table 2: Results of the validation timing detection task [%].
V-Prec.
V-F1
NV-F1
M-F1
Para. HuBERT
52.54
46.27
76.47
61.37
Emo. HuBERT
45.63
52.81
67.94
60.73
Proposed (Ours)
47.96
54.34
70.41
62.37
Table 3: Ablation study [%].
V-Prec.
V-F1
NV-F1
M-F1
Attention
43.81
51.11
66.15
58.63
Gated
44.00
50.29
67.17
58.73
MHA
46.67
50.91
70.55
60.73
Concat (Ours)
47.96
54.34
70.41
62.37
Table 4: Evaluation of different fusion strategies [%].
Para.
Emo.
V-Prec.
V-F1
NV-F1
M-F1
Freeze
Freeze
0.00
0.00
79.45
39.72
Fine T.
Freeze
44.90
50.87
68.16
59.52
Freeze
Fine T.
61.90
27.08
79.65
53.37
LoRA
Fine T.
48.53
46.15
74.07
60.11
Fine T.
LoRA
48.61
47.62
73.72
60.67
Fine T.
Fine T.
47.96
54.34
70.41
62.37
Table 5: Evaluation of different training strategies [%]. Para.: Paralinguistic Encoder; Emo.: Emotion Encoder
Emotional validation - explicitly acknowledging that a user's feelings make sense - has proven therapeutic value but has received little computational attention. Emotional validation in dialogue systems can be decomposed into (i) validating response identification, (ii) validation timing detection, and (iii) validating response generation. To support research on all three subtasks, we release M-EDESConv, a 120k English-Japanese multilingual corpus created through hybrid manual and automatic annotation, and M-TESC, a multilingual spoken-dialogue test set. For timing detection, we propose MEGUMI, a Multilingual Emotion-aware Gated Unit for Mutual Integration, that fuses frozen XLM-RoBERTa semantics with language-specific emotion encoders via cross-modal attention and gated fusion. MEGUMI shows superior performance on both the M-EDESConv and M-TESC datasets, both objectively and subjectively. Finally, our EmoValidBench benchmarks of GPT-4.1 Nano and Llama-3.1 8B indicate that current LLMs generate contextually similar and diverse validating responses, but emotional understanding remains a major area for improvement. Project page: https://github.com/zihaurpang/Multilingual-Emotional-Validation
Zi Haur Pang, Yahui Fu, Koji Inoue +1
Graduate School of Informatics, Kyoto University, Japan
In human conversation, empathy involves not only recognizing a partner's emotion but also regulating it, listeners identify the emotional state, select a suitable regulation strategy, and implement it in their reply. Current large audio-language models (LALMs) largely map perceived emotion directly onto a response, skipping this regulation process. As a result, their responses tend to mirror the user's affect or offer generic sympathy. We propose ER-EDF, a psychology-grounded framework for empathetic dialogue that follows the three stages of Gross's Extended Process Model of emotion regulation, identifying the user's emotion from speech, selecting a regulation strategy matched to that emotion, and implementing the strategy through a LALM. The framework is model-agnostic and integrates seamlessly into existing LALMs. We further construct a spoken empathetic dialogue dataset and introduce empathy-aware evaluation metrics beyond lexical matching. Experiments across five LALMs and two datasets show that ER-EDF consistently improves empathetic response quality in both automatic and human evaluations. These results suggest that modeling how humans regulate others' emotions, rather than only how they perceive them, is key to empathetic spoken dialogue, paving the way for psychologically grounded empathetic AI.
Large speech language models have demonstrated strong capabilities in unified cross-modal understanding and generation, yet paralinguistic cues, especially emotion, remain difficult to preserve. Existing systems typically rely on entangled acoustic representations, which allow the underlying language model to depend excessively on recovered lexical content instead of grounding its behavior in acoustic-prosodic evidence. We address this limitation with EMODE, an emotion-aware speech language model built around Dynamic Para-Semantic Experts (DPSE). DPSE decomposes continuous speech features into semantic and paralinguistic pathways, routes them dynamically, and fuses them before integration into the language model. To turn this structural decomposition into functional specialization, EMODE is trained with a three-stage curriculum consisting of semantic warm-up, paralinguistic activation, and joint refinement, guided by Orthogonal Expert Guidance (OEG), Semantic-to-Acoustic Alignment (SAA), and Gating Diversity Regularization (GDR). Experiments on SER test, empathetic response evaluation, and the newly constructed bilingual MEPA benchmark show that EMODE improves the balance between lexical fidelity and emotional sensitivity, strengthens affect-grounded response generation, and exposes the value of explicit para-semantic factorization for robust cross-corpus emotion understanding.