Louder, Longer, Livelier: Acoustic Shortcuts and Underspecified Rationales in Speech LLM Judges

Mingyue Huo; Shivam Mehta; Bhavin Jawade; Yinghong Lan; Haoqi Li

eess.ASSep 29, 2026

Louder, Longer, Livelier: Acoustic Shortcuts and Underspecified Rationales in Speech LLM Judges

Authors: Mingyue Huo, Shivam Mehta, Bhavin Jawade, Yinghong Lan, Haoqi Li

Organizations: University of Illinois Urbana-Champaign · Netflix

Abstract

LLM-as-a-judge is widely used for evaluating text, but extending this paradigm to speech requires models to interpret acoustic as well as linguistic evidence. This introduces a modality-specific risk: a speech judge may treat a perceptually salient cue as evidence of quality even when that cue is irrelevant to the target criterion or receives more weight than human listeners give it. We call this behavior an acoustic shortcut. To study it, we audit six speech LLM judges using controlled manipulations of intensity, content richness, and emotional delivery. We evaluate both pointwise scoring and pairwise comparison, using human preference calibration to interpret the results. The judges consistently reward louder audio, prefer content-rich speech more strongly than human listeners do, and map emotional delivery into quality preferences. These effects are most visible in pairwise comparison, while pointwise scores often obscure them. More concerningly, the accompanying rationales rarely identify the acoustic cue that changes a judgment and instead repeatedly rely on a limited vocabulary, leaving them acoustically underspecified. Together, these findings show that reliable speech judges must both resist acoustic shortcuts and ground their rationales in the acoustic evidence behind their decisions. To support reproducibility and future audits, we also release SpeechJudgeAudit, the controlled stimuli and evaluation tools used in this study.

Figures & tables

Figure 1: Speech LLM judges may turn salient acoustic cues such as louder, longer, or livelier speech into quality preferences, while their rationales often use broad rubric terms rather than naming the cue that changed the judgment.

Judge	Backbone	Quality criterion	Pairwise	Pointwise	Scale	Rationale
SpeechJudge-BTRM ( Zhang et al., 2025 )	Qwen2.5-Omni	Naturalness	$\times$	✓	$(-\infty,\infty)$	$\times$
SpeechJudge-GRM ( Zhang et al., 2025 )	Qwen2.5-Omni	Naturalness	✓	$\times$	1–10	✓
UniSRM ( Wang et al., 2026b )	Qwen2.5-Omni	Overall quality	✓	✓	1–5	✓
SQ-LLM ( Wang et al., 2026a )	Qwen2.5-Omni	Overall quality	✓	✓	1–5	✓
Gemini-2.5-pro ( Gemini Team, Google, 2025 )	Gemini	Overall quality	✓	✓	1–10	✓
Gemini-3.1-pro ( Gemini Team, 2026 )	Gemini	Overall quality	✓	✓	1–10	✓

Table 1: Audited speech LLM judges. The Gemini models are general-purpose LALMs evaluated with zero-shot prompts; the other four judges are specialized models fine-tuned for speech-quality evaluation. SpeechJudge-BTRM is a scalar reward model rather than a generative judge, but we include it as a LALM-based speech evaluator. The rationale column indicates whether the judge returns an explanation with its judgment.

Figure 2: Mean pairwise preference for the louder clip after intensity adjustment. Points show individual judges; full per-judge results are in Appendix B.2 Loudness and Silence Results .

Comparison	What changes	Mean preference
Silence padding vs. concise	Duration only	0.41
Repetition vs. concise	Duplicated speech	0.46
Richer vs. concise	Content and varied speech material	0.71
Richer vs. duration-matched padding	Content richness at matched duration	0.70

Table 2: Content-richness controls averaged over the five pairwise judges. Values are preferences for the first condition, with ties counted as 0.5. Full per-judge results are in Appendix B.3 Content-Richness Results .

Figure 3: Human and mean judge preference for the emotional clip over the matched Neutral clip. White points show individual pairwise judges.

Figure 4: Rationale vocabulary for Neutral and emotional speech. Across acoustically different inputs, rationales repeatedly use similar quality and prosody terms rather than adapting to the specific emotional delivery.

Figure 5: Rationale audit summary. Rationales rarely name the manipulated acoustic cue, while generic rubric terms remain common. Repetition is a partial exception: judges can sometimes name a conspicuous acoustic change, but do not do so reliably.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

System	Generation structure	Representation
CosyVoice2	AR LLM + flow matching	Codec tokens
Qwen3-TTS	AR LLM + codec decoder	Codec tokens
F5-TTS	Non-AR DiT flow matching	Mel-spectrogram
StyleTTS2	Non-AR diffusion + GAN	Mel-spectrogram

Appendix

Table 3: TTS systems used for generated-speech stimuli. They are chosen to span autoregressive (AR) and non-autoregressive (non-AR) generation, as well as codec-token and mel-spectrogram output representations.

Judge	Condition	Pairwise N	Pairwise pref.	Pairwise sig.	Pointwise N	Pointwise $\Delta$	Pointwise sig.	Notes
BTRM	$-6$ dB vs. ref	–	–	–	100	-0.54	**	quiet penalty
BTRM	$-3$ dB vs. ref	–	–	–	99	-0.42	**	quiet penalty
BTRM	$+3$ dB vs. ref	–	–	–	99	+0.05	n.s.
BTRM	$+6$ dB vs. ref	–	–	–	99	-0.12	n.s.
GRM	$-6$ dB vs. ref	80	0.500	n.s.	–	–	–	pairwise-only judge
GRM	$-3$ dB vs. ref	80	0.475	n.s.	–	–	–

Appendix

Table 4: Combined loudness results. Pairwise preference is

P(\mathrm{manipulated}>\mathrm{reference})

with ties counted as 0.5; pointwise

\Delta

is the within-judge manipulated–reference score difference. All pairwise judges prefer

+6

dB clips, while pointwise scoring often hides this preference.

Judge	Condition	Pairwise N	Pairwise pref.	Pairwise sig.	Pointwise N	Pointwise $\Delta$	Pointwise sig.	Notes
BTRM	pad500 vs. base	–	–	–	80	-1.59		pad penalty
BTRM	pad1000 vs. base	–	–	–	80	-2.87		pad penalty grows
UniSRM	pad500 vs. base	80	0.250		80	-0.01	n.s.	pointwise 84% identical
UniSRM	pad1000 vs. base	80	0.069		80	+0.01	n.s.	pointwise 86% identical
SQ-LLM	pad500 vs. base	80	0.537	n.s.	80	+0.01	n.s.	96% identical scores
SQ-LLM	pad1000 vs. base	80	0.422	n.s.	80	-0.04	n.s.	trend anti-pad

Appendix

Table 5: Combined silence-padding results. Pointwise

\Delta

is the padded–base score difference; pairwise pad preference is

P(\mathrm{padded}>\mathrm{base})

with ties counted as 0.5. UniSRM illustrates the diagnostic pattern most clearly: pointwise scores are nearly unchanged, but pairwise comparison strongly rejects padding.

Setting	Richer preference	Takeaway
Baseline	0.701	reference
Position/length instruction	0.695	preference remains

Appendix

Table 6: Gemini-3.1 position/length prompt ablation on ASSET content-richness pairs. The explicit instruction has negligible effect: the richer-speech preference remains near 0.70.

Judge	Richer $>$ concise	Pad $>$ concise	Sig.	Repeat $>$ concise	Sig.	Richer $>$ pad	Sig.
UniSRM	0.846	0.082		0.447	**	0.954
GRM	0.570	0.527	n.s.	0.362		0.567
SQ-LLM	0.755	0.424		0.515	n.s.	0.765
Gemini-2.5	0.664	0.507	n.s.	0.515	n.s.	0.544	**
Gemini-3.1	0.701	0.502	n.s.	0.471		0.661

Appendix

Table 7: Full content-richness decomposition. Values are manipulated-clip preference rates with ties counted as 0.5. Richer denotes the original ASSET sentence, C = concise, Pad = silence-padded concise, and Rep = repeated concise; significance markers follow Appendix B.1 Statistical Reporting .

Judge	Setting	Text fidelity?	Richer pref.	What it shows
BTRM	Concise target transcript supplied	yes	0.705	target transcript changes the outcome
BTRM	Richer target transcript supplied	yes	0.262	target mismatch reverses the direction
BTRM	Concise and richer targets averaged	yes	0.483	target effects cancel; no target-free analogue
UniSRM	Target transcript plus text-fidelity dimension	yes	–	text fidelity masks content-richness preference
UniSRM	Target-free prompt variant	no	–	output became template-like, so excluded
Gemini-3.1	Anti-position/length instruction only	no	0.695	preference remains despite the instruction

Appendix

Table 8: Text-fidelity and prompt controls for the content-richness experiment. Target-text conditions can confound acoustic preference by rewarding or penalizing extra words through transcript matching.

Judge	Emotion	Pairwise N	Pairwise pref.	Pairwise sig.	Pointwise $\beta$	Pointwise sig.	Overall sig.	Notes
BTRM	Happy	–	–	–	-0.085	n.s.	**
BTRM	Sad	–	–	–	-0.205		**	lower than Neutral
BTRM	Angry	–	–	–	-0.129		**	lower than Neutral
GRM	Happy	400	0.478	n.s.	–	–	–	flat
GRM	Sad	400	0.392		–	–	–	Neutral wins
GRM	Angry	400	0.395		–	–	–	Neutral wins

Appendix

Table 9: Combined emotion results. Pairwise preference is

P(\mathrm{emotion}>\mathrm{Neutral})

with ties counted as 0.5; pointwise

\beta

is the z-scored coefficient relative to Neutral. Overall significance is from the likelihood-ratio test for the emotion fixed effects.

Cue / example	Pointwise evidence	Pairwise evidence	Takeaway
Content richness	no consistent richer-speech advantage; Gemini-2.5 leans concise	richer preference 0.570–0.846	pairwise comparison reveals a consistent preference
Content richness correlations	Gemini-3.1 $r=+0.14$ ; SQ-LLM $r=+0.096$	same items, pairwise preferences	per-item pointwise and pairwise verdicts are weakly related
Silence / UniSRM	$\Delta\approx 0$ , 84–86% identical scores	pad1000 preference 0.069	integer scale hides anti-pad preference
Loudness / SQ-LLM	loud6 $\Delta=0.000$ , degenerate	loud6 preference 0.591	pointwise resolution ceiling hides pro-loud preference
Loudness / Gemini	all pointwise deltas n.s.	quiet6 loses, loud6 wins	pointwise noise hides directional level preference
Emotional delivery	Kendall $W=0.33$ across pointwise ranks	Kendall $W=0.84$ across pairwise ranks	pairwise rankings are more consistent across judges

Appendix

Table 10: Pointwise and pairwise evidence across controlled acoustic manipulations. Pairwise comparison often reveals a directional preference that is weak, noisy, or absent in pointwise scores.

Setting	Position effect observed	Role in this audit
Content richness	GRM strongly favors the second clip in decisive comparisons (B wins 91.6%); Gemini-2.5 also favors B (71.8% of trials), while UniSRM and SQ-LLM lean toward A (64.8% and 56.4%).	We average the two presentation orders before computing the richer-speech preference, so the reported effect is not a single-order artifact.
Loudness	GRM shows order asymmetries of roughly 10–15 percentage points across loudness conditions even when its order-averaged loudness preference is near chance.	Order averaging prevents a positional preference from being interpreted as evidence for or against a loudness preference.
Silence padding	GRM and SQ-LLM show clear AB/BA differences, while their order-averaged padded-clip preferences remain near chance.	This supports using order-averaged item preferences and treating the strongest silence result as judge-specific rather than universal.
Emotional delivery	The average AB–BA gap is about 0.10 for SQ-LLM, 0.48 for GRM, 0.61 for Gemini-2.5, 0.68 for Gemini-3.1, and 0.80 for UniSRM.	The emotion preferences in Table 9 are interpretable because they are averaged across both orders.

Appendix

Table 11: Position-bias diagnostics for pairwise experiments. The numbers are used to justify order counterbalancing, not as acoustic-shortcut results.

Manipulation	Judge(s) audited	Explicit acoustic-term rate	Generic rubric terms
Content richness / length	Gemini-3.1, GRM, UniSRM	length 0–3%; richer/original 0%	rubric/prosody terms frequent
Loudness	UniSRM	loud/loudness/volume 0%; amplitude/dB/level 0.94%	prosody/naturalness/quality 100%
Silence padding	UniSRM	silence 0%; duration 0%; pad/leading/trailing 1.25%	prosody/naturalness/quality 100%
Emotion	rationale-producing judges	emotion-specific terms rarely identify the true emotion (3/16 pointwise cells)	generic prosody terms excluded from lexicon
Repetition	GRM	repetition keywords 22.1% for rep-vs-concise	partial exception; detection incomplete

Appendix

Table 12: Summary of explicit acoustic-term rates in rationales. Generic rubric terms refer to broad quality or prosody language rather than the manipulated acoustic change.

Judge	Mentions rep.	$P(\mathrm{rep\ loses}\mid\mathrm{mentioned})$	$P(\mathrm{rep\ loses}\mid\mathrm{not})$
Gemini-2.5	5.7%	11.8%	13.9%
Gemini-3.1	27.9%	16.9%	2.3%
GRM	21.8%	78.5%	59.7%
UniSRM	10.5%	80.6%	45.1%

Appendix

Table 13: Repetition detection versus penalty. Mentions rep. is the fraction of rep-vs-concise rationales mentioning repetition keywords; the conditional columns report how often the repeated clip loses when repetition is or is not named.

Condition	Items	N	Human mean [CI]	Human sig.	Judge mean [range]	Sig. judges	Residual	Human entropy	Human variance
+6 dB vs. ref	12	88	0.612 [0.548, 0.671]		0.578 [0.541, 0.616]	5/5	-0.035	1.009	0.071
Richer vs. concise	18	134	0.547 [0.464, 0.627]	n.s.	0.707 [0.570, 0.846]	5/5	+0.160	1.260	0.111
Sad vs. Neutral	15	115	0.217 [0.129, 0.320]		0.403 [0.328, 0.555]	5/5	+0.186	0.756	0.098
Happy vs. Neutral	15	112	0.407 [0.326, 0.488]	n.s.	0.531 [0.470, 0.598]	3/5	+0.124	1.305	0.155

Appendix

Table 14: Human preference calibration and agreement diagnostics. Human mean is the item-level manipulated-clip preference rate with bootstrap 95% confidence intervals. Human significance uses a two-sided sign-flip test against 0.5. Judge mean is averaged over the five pairwise judges, the bracketed range shows their spread, and Sig. judges counts how many individual judges are significant in either direction.

Rater	Valid N	Mean pref.	Tie rate	Majority agreement	Item-mean MAE
Lowest-agreement rater (rater_14)	29	0.379	0.207	0.345	0.478
Most extreme rater (rater_11)	30	0.450	0.100	0.433	0.465
All raters mean	29.9	0.441	0.343	0.481	0.311

Appendix

Table 15: Human listener quality-control diagnostics. Majority agreement is the fraction of a rater’s responses that match the item-level majority label; item-mean MAE is the mean absolute error between the rater’s numeric response and the item mean after coding manipulated/reference/tie as 1/0/0.5.

Analysis	Statistic	Result
Speaker gender	female vs. male speakers	no preference (n.s.; all $p\geq.21$ )
Speaker accent	North-American vs. British speakers	no preference (n.s.)
Emotion gender interaction	gender $\times$ emotion	no interaction (n.s.)
Low-level descriptors	descriptor–score correlations	no corrected correlation (n.s.)
Model self-preference	Qwen judges on Qwen3-TTS vs. other TTS	no self-preference (n.s.)
Gemini TTS preference	TTS-system effect	preference differs by system (***)

Appendix

Table 16: Natural-variance and supporting analyses. These diagnostics are within-pool evidence and should not be read as causal mechanisms or broad fairness guarantees.

Explore similar work

Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation
Jul 15, 2026Joonyong Park, David M. Chan, Yuki Saito +1Large Audio Language Models Large Language Model Judges
Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Berkeley AI Research (BAIR), University of California, Berkeley, CA, USA
SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferences
Sep 28, 2026Mingyue Huo, Shivam Mehta, Bhavin Jawade +2Large Audio Language Models Critic
University of Illinois Urbana-Champaign · Netflix
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Paralinguistic Cues Large Audio Language Models
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1

Jul 15, 2026, cs.SDJ/K move · Enter open · S save

Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation

Joonyong Park, David M. Chan, Yuki Saito, Hiroshi Saruwatari

Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Berkeley AI Research (BAIR), University of California, Berkeley, CA, USA

Large Audio Language Models Large Language Model Judges Audio Understanding Model Auditing

Open paper AskPDF arXiv

Large audio-language models (LALMs) are increasingly used as automatic judges for speech evaluation. However, high agreement with human ratings does not guarantee that their verdicts are grounded in the audio. A judge may instead rely on specialist labels or reference data supplied by the evaluation protocol itself, taking a shortcut in place of listening to the audio. In this paper, we audit such protocol-level ``shortcuts'' in LALM judges across three common deployment protocols: feature-blueprint judging, where the audio is replaced by a structured text description of acoustic features, reference-conditioned judging, and pairwise A/B comparison. Across six judges and four attributes, we find that several LALMs rely on protocol-level shortcuts. For example, in feature-blueprint judging, incorrect specialist labels reduce five judges' emotion accuracy to 0.10 or below, and in concatenated A/B comparisons, Qwen3-Omni-Thinking often picks the same slot regardless of order swaps. These results indicate that aggregate agreement can overstate the validity of LALM judges unless the model and the evaluation protocol are assessed jointly, and that each model-protocol pair should be evaluated with a matched shortcut probe.

Abstract

Explore similar work

Auditing Protocol-Level Shortcuts in Large Audio Language Model Judges for Speech Evaluation

SpeechCritic: Learning a Diagnostic Speech Judge from Limited Human Preferences

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge