Expert L2 English Speech Evaluation on a Gaming PC: Accuracy, Fluency, and Prosody with XLS-R, XEUS, and PhoneticXeus

Richard Rose

cs.CLOct 19, 2026

Expert L2 English Speech Evaluation on a Gaming PC: Accuracy, Fluency, and Prosody with XLS-R, XEUS, and PhoneticXeus

Authors: Richard Rose

Organizations: Hankuk University of Foreign Studies

Abstract

oollim is a locally executable system for evaluating L2 English pronunciation accuracy, fluency, and prosody on a consumer computer. The system combines pretrained XLS-R, XEUS, and PhoneticXeus representations with reference-text features and ridge regressions trained on SpeechOcean762, the Korean Learners' L2 Speech Corpus (English), J-AESOP, and their combinations. On 305 independently held-out recordings assessed by three expert raters, several ridge configurations substantially outperform archived Azure API scores obtained in January 2026 in both Spearman correlation and quadratic weighted kappa across all three constructs. These results support expert speech evaluation on consumer hardware using pretrained representations and construct-specific regressions, without neural-model fine-tuning or calibration on the private evaluation data.

Figures & tables

	Accuracy		Fluency		Prosody
Method	$\rho$	QWK	$\rho$	QWK	$\rho$	QWK
Human consensus	0.809	0.704	0.807	0.739	0.807	0.742
Azure	0.761	0.161	0.633	0.078	0.737	0.370
speechocean	0.844	0.546	0.842	0.671	0.847	0.570
korean	0.856	0.702	0.860	0.495	0.874	0.489
aesop	0.852	0.241	0.850	0.276	0.850	0.261

Table 1: Agreement with human consensus (

N=305

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Agreement comparison	$N$	$\rho$	95% CI	QWK	95% CI
Human raters	Human–Human pairwise agreement	305	0.759	[0.710, 0.799]	0.642	[0.597, 0.685]
Human raters	Human–Human consensus agreement (other two raters)	305	0.809	[0.769, 0.842]	0.704	[0.663, 0.743]
Azure	Model–Human consensus agreement (two raters)	305	0.761	[0.709, 0.804]	0.161	[0.129, 0.196]
Azure	Model–Human consensus agreement (three raters)	305	0.776	[0.723, 0.820]	0.158	[0.126, 0.193]
speechocean	Model–Human consensus agreement (two raters)	305	0.844	[0.807, 0.872]	0.546	[0.494, 0.592]
speechocean	Model–Human consensus agreement (three raters)	305	0.862	[0.825, 0.890]	0.555	[0.500, 0.605]

Appendix

Table 2: Supplementary accuracy comparisons on the matched cohort.

Method	Agreement comparison	$N$	$\rho$	95% CI	QWK	95% CI
Human raters	Human–Human pairwise agreement	305	0.747	[0.697, 0.791]	0.680	[0.631, 0.724]
Human raters	Human–Human consensus agreement (other two raters)	305	0.807	[0.768, 0.840]	0.739	[0.695, 0.777]
Azure	Model–Human consensus agreement (two raters)	305	0.633	[0.556, 0.698]	0.078	[0.056, 0.107]
Azure	Model–Human consensus agreement (three raters)	305	0.649	[0.571, 0.715]	0.077	[0.055, 0.105]
speechocean	Model–Human consensus agreement (two raters)	305	0.842	[0.801, 0.874]	0.671	[0.627, 0.707]
speechocean	Model–Human consensus agreement (three raters)	305	0.859	[0.819, 0.891]	0.704	[0.660, 0.740]

Appendix

Table 3: Supplementary fluency comparisons on the matched cohort.

Method	Agreement comparison	$N$	$\rho$	95% CI	QWK	95% CI
Human raters	Human–Human pairwise agreement	305	0.751	[0.702, 0.795]	0.681	[0.634, 0.722]
Human raters	Human–Human consensus agreement (other two raters)	305	0.807	[0.768, 0.842]	0.742	[0.701, 0.778]
Azure	Model–Human consensus agreement (two raters)	305	0.737	[0.672, 0.792]	0.370	[0.318, 0.421]
Azure	Model–Human consensus agreement (three raters)	305	0.750	[0.684, 0.806]	0.369	[0.318, 0.422]
speechocean	Model–Human consensus agreement (two raters)	305	0.847	[0.810, 0.878]	0.570	[0.525, 0.610]
speechocean	Model–Human consensus agreement (three raters)	305	0.865	[0.829, 0.895]	0.593	[0.547, 0.634]

Appendix

Table 4: Supplementary prosody comparisons on the matched cohort.

System	Reference	Metric	Difference	95% CI	Adjusted CI
azure	Human	$\rho$	-0.049	[-0.096, -0.005]	[-0.137, 0.027]
azure	Human	QWK	-0.543	[-0.580, -0.504]	[-0.606, -0.471]
speechocean	Human	$\rho$	+0.035	[0.004, 0.065]	[-0.021, 0.091]
speechocean	Human	QWK	-0.158	[-0.206, -0.110]	[-0.243, -0.073]
speechocean	Azure	$\rho$	+0.083	[0.041, 0.128]	[0.009, 0.167]
speechocean	Azure	QWK	+0.385	[0.341, 0.426]	[0.306, 0.455]

Appendix

Table 5

System	Reference	Metric	Difference	95% CI	Adjusted CI
azure	Human	$\rho$	-0.174	[-0.244, -0.110]	[-0.304, -0.067]
azure	Human	QWK	-0.661	[-0.697, -0.621]	[-0.723, -0.588]
speechocean	Human	$\rho$	+0.035	[0.000, 0.069]	[-0.029, 0.095]
speechocean	Human	QWK	-0.068	[-0.113, -0.024]	[-0.145, 0.010]
speechocean	Azure	$\rho$	+0.209	[0.145, 0.279]	[0.099, 0.338]
speechocean	Azure	QWK	+0.593	[0.554, 0.627]	[0.523, 0.653]

Appendix

Table 6

System	Reference	Metric	Difference	95% CI	Adjusted CI
azure	Human	$\rho$	-0.070	[-0.124, -0.023]	[-0.172, 0.010]
azure	Human	QWK	-0.372	[-0.419, -0.325]	[-0.455, -0.289]
speechocean	Human	$\rho$	+0.040	[0.009, 0.069]	[-0.017, 0.092]
speechocean	Human	QWK	-0.172	[-0.214, -0.128]	[-0.247, -0.094]
speechocean	Azure	$\rho$	+0.110	[0.063, 0.163]	[0.032, 0.211]
speechocean	Azure	QWK	+0.201	[0.145, 0.256]	[0.100, 0.298]

Appendix

Table 7

Construct	System	Mean	SD	Bias	MAE
Accuracy	azure	89.29	7.50	+23.63	23.63
Accuracy	speechocean	72.54	6.47	+6.87	8.22
Accuracy	korean	58.57	12.39	-7.10	7.66
Accuracy	aesop	38.49	13.31	-27.17	27.17
Accuracy	koreanocean	64.85	9.77	-0.81	4.38
Accuracy	aesopocean	55.27	11.66	-10.40	10.51

Appendix

Table 8

Construct	Family	SpeechOcean (500)	Korean (636)	J-AESOP (36)
Accuracy	speechocean	0.816	0.699	0.967
Accuracy	korean	0.735	0.759	0.969
Accuracy	aesop	0.659	0.673	0.987
Accuracy	koreanocean	0.808	0.755	0.964
Accuracy	aesopocean	0.807	0.715	0.970
Accuracy	koreanaesop	0.683	0.755	0.976

Appendix

Table 9

Family	Construct	Numerical best	Best $\rho$	Retained/gated	Promotion
speechocean	accuracy	X+P+I	0.8276	X+P+I	No
speechocean	fluency	X+P+I	0.8655	X+R	No
speechocean	prosody	X+I	0.8640	X+R	No
korean	accuracy	X+R+I	0.8322	X+P+I	No
korean	fluency	X+I	0.8544	X+R	No
korean	prosody	X	0.8560	X+R	No

Appendix

Table 10

Automated Spoken Language Assessment

See all 15 papers

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Audio-Language Model Evaluation Pairwise Comparison
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1
OpenEnded: An Open-Response Speech Corpus for Speaking Proficiency Assessment with Human Annotations and ALM Supervision
Sep 14, 2026Yu-Wen Chen, Eric Zhou, Evelyn Ding +3Speech Processing Automated Spoken Language Assessment
Department of Computer Science, Columbia University, USA
A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationales
Jun 8, 2026Aditya Kamlesh Parikh, Cristian Tejedor-Garcia, Catia Cucchiarini +1Audio-Language Model Evaluation Speech Processing
Centre for Language Studies, Radboud University, Nijmegen, The Netherlands

Jun 23, 2026J/K move · Enter open · S save

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

Jisu Jeon, Seungyeon Jwa, Joosung Lee, Jinhyeon Kim+5

Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea+1

Audio-Language Model Evaluation Pairwise Comparison Speech Generation Evaluation Pairwise Preference Evaluation Speech Quality Assessment Pairwise Ranking+1

Open paper AskPDF arXiv

Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches predominantly focus on holistic naturalness, leaving fine-grained paralinguistic distinctions underexplored. We introduce ParaPairAudioBench, a pairwise benchmark of 5,175 audio pairs across five paralinguistic dimensions: Style, Rate, Emphasis, Age, and Gender. Our experiments show that current LALM judges still lag behind human judgments by 32%p on average and exhibit severe calibration failures, particularly in Tie cases where the correct decision is to abstain. To further analyze lexical versus acoustic reliance, the benchmark includes both same-transcript and cross-transcript conditions. ParaPairAudioBench enables multi-dimensional, calibration-aware assessment of the reliability of LALM-as-a-Judge for paralinguistic speech evaluation.

Abstract

Automated Spoken Language Assessment

Explore similar work

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

OpenEnded: An Open-Response Speech Corpus for Speaking Proficiency Assessment with Human Annotations and ALM Supervision

A Finetuned SpeechLLM for Joint Multi-Granular L2 Assessment and Natural-Language Rationales