Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences. However, evaluation protocols for such data were designed for text and have not been validated for speech. We present the first ICC-based, controlled cross-modal study of human and synthetic preference annotations, comparing text and audio evaluations of identical semantic content across 100 prompts. We show that achieving good agreement within either modality (ICC(2,k) ≈ .80) requires ∼9 raters. At the same time, modalities show marked differences in how people report preferences: audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement. We demonstrate that synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations. Together, these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text.
Figures & tables
Figure 1: Cumulative distribution of the mean per-prompt rating gap on trials where raters declared a preference (A or B rather than Tie).
Predictor
b
95% CI
p
Intercept
74.80
[70.72,78.87]
<.001
Presented first
−2.61
[−4.65,−0.58]
.012
Modality (text)
−6.21
[−11.28,−1.14]
.017
Character length ( z )
3.35
[1.10,5.60]
.004
Trial number ( z )
0.03
[−1.02,1.07]
.960
Presented first × Modality
−0.51
[−3.42,2.40]
.732
Table 1: Fixed effects from the linear mixed-effects model predicting raw ratings.
Metric
Audio
Text
p
Avg. raters per stimulus
9.2
8.9
—
ICC(2,1)
.333 [.286, .365]
.295 [.250, .325]
.127
ICC(2, k )
.821 [.787, .841]
.788 [.748, .811]
.113
Krippendorff’s α (cont.)
.315 [.230, .388]
.228 [.157, .298]
.069
Krippendorff’s α (ord.)
.157 [.109, .206]
.217 [.167, .268]
.119
Krippendorff’s α (nom.)
.010 [ −.007 , .026]
.031 [.012, .049]
.437
Table 2: Inter-rater reliability metrics by modality.
Comparison
dˉ
p
AI-Human Absolute Error
Audio MAE
12.27
—
Text MAE
14.29
—
Text vs. Audio (LMM)
2.02
.035
PRISM vs. External Annotators
vs. AI-Text
−1.84
.015
Table 3: Model estimates for rating comparisons.
Predictor
B
p
AI Differentiation
0.014
< .001
Modality (Text)
− 0.010
.76
AI Diff × Modality
− 0.003
.15
Spearman correlations
Audio
r=.23 , p=.02
Text
r=.38 , p<.001
Table 4: Regression: AI Differentiation and Human Agreement
Figure 2: ICC(2, k ) as a function of the number of raters, derived from variance components of a crossed random-effects model. Shaded bands indicate conservative interpretation thresholds per Koo and Li (2016) . The observed average number of raters per stimulus was k≈9.2 (audio) and k≈8.9 (text).
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Interface for the Audio Task of preference selection. It includes conversation history, most recent query and two responses. Raters must rate each response from 1-100 on response quality (the content of the answer) and audio quality (the clarity and naturalness of speech). We also collect discrete preferences with the option to tie and optional free-response reasoning justification.
Figure 4: Interface for the Text Task of preference selection. It includes conversation history, most recent query and two responses. Raters must rate each response from 1-100 on response quality (the content of the answer). We also collect discrete preferences with the option to tie and optional free-response reasoning justification.
Text
Audio
p
Completed (%)
92.5
96.2
.678 a
Session duration, min
27.0 (13.3)
47.3 (12.4)
<.001 b
Trial duration, sec
78.8 (41.6)
136.4 (35.9)
<.001 b
Difficulty rating
2.45 (0.90)
2.50 (0.96)
.868 b
Failed attention (%)
9.4
13.2
.761 a
Appendix
Table 5: Session and trial characteristics by condition.
Figure 5: Cross-modality agreement between audio and text evaluations as a function of raw rating gap threshold. The blue line shows agreement percentage when both modalities produce a decisive winner (excluding ties), while the red line shows agreement when tie-versus-decisive outcomes are counted as disagreements. Green bars indicate the number of prompts where both modalities had a decisive winner at each threshold.
Crowdsourced pairwise evaluation has emerged as a scalable approach for assessing foundation models. However, applying it to Text to Speech(TTS) introduces high variance due to linguistic diversity and multidimensional nature of speech perception. We present a controlled multidimensional pairwise evaluation framework for multilingual TTS that combines linguistic control with perceptually grounded annotation. Using 5K+ native and code-mixed sentences across 10 Indic languages, we evaluate 7 state-of-the-art TTS systems and collect over 120K pairwise comparisons from over 1900 native raters. In addition to overall preference, raters provide judgments across 6 perceptual dimensions: intelligibility, expressiveness, voice quality, liveliness, noise, and hallucinations. Using Bradley-Terry modeling, we construct a multilingual leaderboard, interpret human preference using SHAP analysis and analyze leaderboard reliability alongside model strengths and trade-offs across perceptual dimensions.
Srija Anand, Ashwin Sankar, Ishvinder Sethi +10
Indian Institute of Technology Madras, India · AI4Bharat, India · Josh Talks, India
Popular ASR test sets adopt inconsistent conventions for numbers, disfluencies, entities, and casing, while standard normalizers erase the format distinctions users care about. Current benchmarks therefore cannot measure whether a model follows user preferences for output style. We introduce PreferenceASR, a test set evaluating ASR systems on their ability to follow natural-language preference instructions across four categories: normalization, entities, disfluencies, and case. Built from seven open-source corpora via a two-stage LLM-assisted pipeline with human verification, it is evaluated with a preference-aware normalizer that selectively skips steps matching the active instruction. Benchmarking four models shows rankings shift across preference types, exposing quality differences traditional evaluation obscures. We publicly release the dataset.
Nithin Rao Koluguri, Sasha Meister, Nikolay Karpov +4
Current audio captioning relies on supervised learning with paired audio-caption data, which is costly to curate and may not reflect human preferences in real-world scenarios. To address this, we propose a preference-aligned audio captioning framework based on Reinforcement Learning from Human Feedback (RLHF). To capture nuanced preferences, we train a Contrastive Language-Audio Pretraining (CLAP) based reward model using human-labeled pairwise preference data. This reward model is integrated into an RL framework to fine-tune any baseline captioning system without ground-truth annotations. Extensive human evaluations across multiple datasets show that our method produces captions preferred over baseline models, particularly when baselines fail to provide correct and natural captions. Furthermore, our framework achieves performance comparable to supervised approaches with ground-truth data, demonstrating effective alignment with human preferences and scalability in real-world use.
Kartik Hegde, Rehana Mahfuz, Yinyi Guo +1
Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA