Same Words, Different Judgments: How Preferences Vary Across Modalities

Aaron Broukhim; Nadir Weibel; Eshin Jolly

cs.SDFeb 26, 2026

Same Words, Different Judgments: How Preferences Vary Across Modalities

Authors: Aaron Broukhim, Nadir Weibel, Eshin Jolly

Abstract

Preference-based reinforcement learning (PbRL) is the dominant framework for aligning AI systems to human preferences. However, evaluation protocols for such data were designed for text and have not been validated for speech. We present the first ICC-based, controlled cross-modal study of human and synthetic preference annotations, comparing text and audio evaluations of identical semantic content across 100 prompts. We show that achieving $\textit{good}$ agreement within either modality (ICC(2, $k$ ) $\approx$ .80) requires $\sim$ 9 raters. At the same time, modalities show marked differences in how people report preferences: audio raters exhibit narrower decision thresholds, reduced length bias, and more user-oriented evaluation criteria, with near-chance cross-modality agreement. We demonstrate that synthetic ratings can be used to effectively predict inter-rater agreement, thus serving as an early signal for stimulus selection and proxy for human annotations. Together, these findings argue that evaluation protocols for audio preference data require modality-specific design rather than direct adaptation from text.

Figures & tables

Figure 1: Cumulative distribution of the mean per-prompt rating gap on trials where raters declared a preference (A or B rather than Tie).

Predictor	$b$	95% CI	$p$
Intercept	74.80	$[70.72,\ 78.87]$	$<.001$
Presented first	$-2.61$	$[-4.65,\ -0.58]$	$.012$
Modality (text)	$-6.21$	$[-11.28,\ -1.14]$	$.017$
Character length ( $z$ )	3.35	$[1.10,\ 5.60]$	$.004$
Trial number ( $z$ )	0.03	$[-1.02,\ 1.07]$	$.960$
Presented first $\times$ Modality	$-0.51$	$[-3.42,\ 2.40]$	$.732$

Table 1: Fixed effects from the linear mixed-effects model predicting raw ratings.

Metric	Audio	Text	$p$
Avg. raters per stimulus	9.2	8.9	—
ICC(2,1)	.333 [.286, .365]	.295 [.250, .325]	$.127$
ICC(2, $k$ )	.821 [.787, .841]	.788 [.748, .811]	$.113$
Krippendorff’s $\alpha$ (cont.)	.315 [.230, .388]	.228 [.157, .298]	$.069$
Krippendorff’s $\alpha$ (ord.)	.157 [.109, .206]	.217 [.167, .268]	$.119$
Krippendorff’s $\alpha$ (nom.)	.010 [ $-.007$ , .026]	.031 [.012, .049]	$.437$

Table 2: Inter-rater reliability metrics by modality.

AI-Human Absolute Error
Comparison	$\bar{d}$	$p$
Audio MAE	12.27	—
Text MAE	14.29	—
Text vs. Audio (LMM)	2.02	$.035$
PRISM vs. External Annotators
vs. AI-Text	$-1.84$	$.015$

Table 3: Model estimates for rating comparisons.

Spearman correlations
Predictor	$B$	$p$
AI Differentiation	0.014	$<$ .001
Modality (Text)	$-$ 0.010	.76
AI Diff $\times$ Modality	$-$ 0.003	.15
Audio	$r=.23$ , $p=.02$
Text	$r=.38$ , $p<.001$

Table 4: Regression: AI Differentiation and Human Agreement

Figure 2: ICC(2, k ) as a function of the number of raters, derived from variance components of a crossed random-effects model. Shaded bands indicate conservative interpretation thresholds per Koo and Li (2016) . The observed average number of raters per stimulus was

k\approx 9.2

(audio) and

k\approx 8.9

(text).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Interface for the Audio Task of preference selection. It includes conversation history, most recent query and two responses. Raters must rate each response from 1-100 on response quality (the content of the answer) and audio quality (the clarity and naturalness of speech). We also collect discrete preferences with the option to tie and optional free-response reasoning justification.

Figure 4: Interface for the Text Task of preference selection. It includes conversation history, most recent query and two responses. Raters must rate each response from 1-100 on response quality (the content of the answer). We also collect discrete preferences with the option to tie and optional free-response reasoning justification.

	Text	Audio	$p$
Completed (%)	92.5	96.2	$.678$ a
Session duration, min	27.0 (13.3)	47.3 (12.4)	$<.001$ b
Trial duration, sec	78.8 (41.6)	136.4 (35.9)	$<.001$ b
Difficulty rating	2.45 (0.90)	2.50 (0.96)	$.868$ b
Failed attention (%)	9.4	13.2	$.761$ a

Appendix

Table 5: Session and trial characteristics by condition.

Figure 5: Cross-modality agreement between audio and text evaluations as a function of raw rating gap threshold. The blue line shows agreement percentage when both modalities produce a decisive winner (excluding ties), while the red line shows agreement when tie-versus-decisive outcomes are counted as disagreements. Green bars indicate the number of prompts where both modalities had a decisive winner at each threshold.

	Text ( $n=48$ )	Audio ( $n=46$ )	Total ( $N=94$ )
Age (years)
$M$ ( $SD$ )	45.00 (13.77)	40.17 (11.70)	42.64 (12.96)
Median	44	40	41
Range	22–75	20–72	20–75
Gender
Woman	31 (64.6%)	25 (54.3%)	56 (59.6%)

Appendix

Table 6: Participant Demographics by Condition

Human Preference Evaluation

Human preference evaluation measures how people compare, rate, and explain AI outputs to assess and guide model behavior.

See all 56 papers

Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages
Apr 23, 2026Srija Anand, Ashwin Sankar, Ishvinder Sethi +10Pairwise Preference Learning Human Preference Evaluation
Indian Institute of Technology Madras, India · AI4Bharat, India · Josh Talks, India
Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs
Jun 28, 2026Nithin Rao Koluguri, Sasha Meister, Nikolay Karpov +4ASR Evaluation Automatic Speech Recognition
NVIDIA, USA
Aligning Audio Captions with Human Preferences
Sep 18, 2025Kartik Hegde, Rehana Mahfuz, Yinyi Guo +1Reward Modeling Audio-Language Models
Qualcomm Technologies, Inc., India · Qualcomm Technologies, Inc., USA

Apr 23, 2026J/K move · Enter open · S save

Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages

Srija Anand, Ashwin Sankar, Ishvinder Sethi, Aaditya Pareek+9

Indian Institute of Technology Madras, India · AI4Bharat, India · Josh Talks, India

Pairwise Preference Learning Human Preference Evaluation Pairwise Comparison TTS Synthesis Speech Generation Evaluation Pairwise Preference Evaluation+4

Open paper AskPDF arXiv

Crowdsourced pairwise evaluation has emerged as a scalable approach for assessing foundation models. However, applying it to Text to Speech(TTS) introduces high variance due to linguistic diversity and multidimensional nature of speech perception. We present a controlled multidimensional pairwise evaluation framework for multilingual TTS that combines linguistic control with perceptually grounded annotation. Using 5K+ native and code-mixed sentences across 10 Indic languages, we evaluate 7 state-of-the-art TTS systems and collect over 120K pairwise comparisons from over 1900 native raters. In addition to overall preference, raters provide judgments across 6 perceptual dimensions: intelligibility, expressiveness, voice quality, liveliness, noise, and hallucinations. Using Bradley-Terry modeling, we construct a multilingual leaderboard, interpret human preference using SHAP analysis and analyze leaderboard reliability alongside model strengths and trade-offs across perceptual dimensions.

Abstract

Human Preference Evaluation

Explore similar work

Preferences of a Voice-First Nation: Large-Scale Pairwise Evaluation and Preference Analysis for TTS in Indian Languages

Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs

Aligning Audio Captions with Human Preferences