The rapid evolution of end-to-end spoken dialogue systems demands transcending mere textual semantics to incorporate paralinguistic nuances and the spontaneous nature of human conversation. However, current methods struggle with two critical gaps: the modality gap, involving prosody and emotion, and the colloquialness gap, distinguishing written scripts from natural speech. To address these challenges, we introduce SDiaReward, an end-to-end multi-turn reward model trained on SDiaReward-Dataset, a novel collection of episode-level preference pairs explicitly targeting these gaps. It operates directly on full multi-turn speech episodes and is optimized with pairwise preference supervision, enabling joint assessment of modality and colloquialness in a single evaluator. We further establish ESDR-Bench, a stratified benchmark for robust episode-level evaluation. Experiments demonstrate that SDiaReward achieves state-of-the-art pairwise preference accuracy, significantly outperforming general-purpose audio LLMs. Further analysis suggests that SDiaReward captures relative conversational expressiveness beyond superficial synthesis cues, improving generalization across domains and recording conditions. Code, data, and demos are available at https://github.com/MM-Speech/SDiaReward/.
Figures & tables
Figure 1: Challenges in spoken dialogue and our proposed framework. Text-based systems face modality (prosody/emotion) and colloquialness (style) gaps. Unlike rule-based methods, our end-to-end Reward Model learns these features from multi-turn dialogues via data-driven preference signals.
Figure 2: Overview of dataset construction. (a) Collection: We collect wild conversational audio (main) along with semi-wild/scripted data. (b1–b2) Processing & Pairing: We process audio into speaker-aware turns and group them into dialogues. We then construct two types of pairs: modality-aware pairs (center) via real vs. TTS audio , and colloquialness pairs (bottom right) via text-style vs. spoken-style generation and style change. (c1–c2) Post-processing: We filter episodes and attach hierarchical metadata (emotion, sentiment, act) for benchmark stratification. The detailed data processing pipeline can be found in Appendix B .
Category
Train
Val
Total
Modality
Wild
6,879
824
7,703
Semi-Wild
309
186
495
Scripted
2,192
466
2,658
Colloquialness
2,250
250
2,500
Total All
11,630
1,726
13,356
Table 1: Statistics of the dataset. We categorize data by modality types and colloquialness. The unit is the pairs of dialogue.
Figure 3: Architecture of our reward model.
Model
Modality Acc
Modality
Modality
Colloq.
Overall
Overall
Wild
Semi-wild
Scripted
Micro
Macro
Acc
Micro
Macro
Closed-source Judges
Gemini 2.5 Pro
76.12
65.60
69.77
72.63
70.50
98.80
76.42
84.65
Gemini 2.5 Flash
56.73
53.67
53.91
55.41
54.77
99.60
61.81
77.19
GPT-4o Audio
52.61
49.54
49.26
51.12
50.47
98.00
57.91
74.23
Open-source Audio LMs
Table 2: Main results on ESDR-Bench. We report pairwise preference accuracy (%) on the modality benchmark split into wild/semi-wild/scripted and on the colloquialness benchmark. Modality Micro is the weighted accuracy over all modality pairs; Modality Macro is the unweighted mean across the three modality subsets, serving as a stricter metric for generalization. Overall Micro is the weighted accuracy over all benchmark pairs, while Overall Macro averages modality macro and colloquialness accuracy.
Sample Subset
Count
Avg. Margin
Human Agree.
SE
Validation of Model Capability
High Confidence
20
1.65
88.3%
± 7.2%
Low Confidence
20
0.06
78.3%
± 9.2%
Random Sampling
20
0.77
76.7%
± 9.5%
Error Analysis (Hard Negatives)
Model Wrong †
15
-0.19
93.3%
± 6.5%
Table 3: Human Verification Results. We evaluate 75 stratified samples with averaged multi-annotator ratings. Human Agree. denotes agreement with the dataset ground truth; Avg. Margin is the model margin on each subset, and SE reports the standard error.
OOD Engine
Wav2Vec2-DF
SDiaReward-3B
SDiaReward-7B
Acc ↑
Srej
Acc ↑
Srej
Acc ↑
Srej
OpenAI TTS
89.9%
-
93.0%
-0.58
98.3%
-0.62
CosyVoice 2
38.6%
-
93.1%
-0.12
95.3%
-0.04
FireRedTTS-2
94.5%
-
72.7%
0.18
90.9%
0.29
Table 4: Modality Accuracy and Rejected Scores on OOD TTS Engines. Srej denotes the average scalar reward assigned to the rejected (synthetic) audio. Higher Srej indicates that the synthetic speech is more challenging to distinguish from human speech.
Figure 4: Ablation Analysis on SDiaReward Model (7B). (a) Score Alignment : The proposed center loss (Orange) effectively anchors the chosen reward distribution to μ≈0.32 , whereas the baseline (Blue) suffers from significant drift ( μ>5.0 ). (b) Margin Stability : The discriminative margin remains robust. (c) Density Modes : Split violin plots visualize reward density, showing high confidence in Wild data. (d) Statistical Ranges : Box plots reveal domain-dependent decision boundaries; notably, Scripted responses receive lower absolute scores despite being correct choices.
Setting
Modality
Colloq.
Overall
Pooling Strategy (3B Backbone)
Last Hidden
63.75
48.80
61.59
Attention
87.94
93.60
88.76
Mean
88.62
92.00
89.10
Pooling Strategy (7B Backbone)
Last Hidden
51.83
40.00
50.12
Table 5: Ablation Study. Performance comparison across pooling strategies and model scales. The Mean strategy with center loss achieves the best trade-off between stability and accuracy.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Primary Dimension
Secondary Dimension
YouTube (Wild)
Anger Disgust Fear Happiness Neutral Sadness Surprise
Cough Cry Filled Pauses Laughter Listener Feedback Sigh/Breath No Feature
MELD (Semi-wild)
Negative Neutral Positive
Anger Disgust Fear Joy Neutral Sadness Surprise
DailyTalk (Scripted)
Commissive Directive Inform Question Unknown
Anger Disgust Fear Happiness No Emotion Sadness Surprise
Appendix
Table 6: Hierarchical Classification across Datasets.
Figure 5: Overview of the ESDR-Bench
Figure 6: Overview of the SDiaReward-Dataset
Figure 7: Word Cloud of Colloquial Data
Figure 8: Ablation Analysis on SDiaReward Model (3B).