AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Style

Joonyong Park; Jerry Li

cs.SDMar 12, 2026

AnimeScore: A Preference-Based Dataset and Framework for Evaluating Anime-Like Speech Style

Authors: Joonyong Park, Jerry Li

Organizations: Spellbrush, USA

Abstract

Evaluating 'anime-like' voices currently relies on costly subjective judgments, yet no standardized objective metric exists. A key challenge is that anime-likeness, unlike naturalness, lacks a shared absolute scale, making conventional Mean Opinion Score (MOS) protocols unreliable. To address this gap, we propose AnimeScore, a preference-based framework for automatic anime-likeness evaluation via pairwise ranking. We collect 15,000 pairwise judgments from 187 evaluators with free-form descriptions, and acoustic analysis reveals that perceived anime-likeness is driven by controlled resonance shaping, prosodic continuity, and deliberate articulation rather than simple heuristics such as high pitch. We show that handcrafted acoustic features reach a 69.3% AUC ceiling, while SSL-based ranking models achieve up to 90.8% AUC, providing a practical metric that can also serve as a reward signal for preference-based optimization of generative speech models.

Figures & tables

Corpus	Train	Test	Total
Anim-400k [ 20 ]	1,065	250	1,315
ReazonSpeech [ 21 ]	828	120	948
Coco-Nut [ 22 ]	607	130	737
Total	2,500	500	3,000

Table 1: Statistics of speech samples for subjective evaluation

Category	Group	Count
Age	20s or younger	8
	30s	48
	40s	80
	50s or older	51
Gender	Male	142
Gender	Female	45

Table 2: Demographics of participants (

n=187

)

Figure 1: Corpus-wise distribution of empirical win rate for (a) train and (b) test data.

Figure 2: Visualizations of acoustic proxies for anime-likeness speech. (a) F1-F2 formant space distribution comparing high-win rate (top 25%) and low-win rate (bottom 25%) utterances. (b) Split violin plots of normalized spectral flux and voicing ratio distributions for the top and bottom 25% groups. (c) Overall syllable rate and articulation rate across win rate.

Category	Count
Emotional Explicitness	62
Timbre Difference	48
Prosodic Salience	38
Articulation Clarity	34
Temporal Control	5

Table 3: Distribution of Annotator-Identified Key Features of Anime-Like Speech (

n=187

)

Dimension	Proxy Metrics	PCR (%)
Emotional Explicitness	Intensity (Arousal)	52.2 ( $\uparrow$ )
Timbre Difference	$F1$ median (Hz)	59.6 ( $\downarrow$ )
	$F2$ median (Hz)	61.5 ( $\downarrow$ )
	$F3$ median (Hz)	60.1 ( $\downarrow$ )
Prosodic Salience	Mean F0 (Hz)	55.1 ( $\downarrow$ )
	Voicing ratio	59.5 ( $\uparrow$ )

Table 4: Pairwise Concordance Rate (PCR) between acoustic proxies and subjective anime-likeness.

\uparrow

\downarrow

: the preferred utterance tends to have a higher/lower value.

\dagger

Figure 3: Architecture of an anime score prediction model

Feature Set	#Feat.	Acc. (%)	AUC (%)
Emot. Explicit.	1	52.1 $\pm$ 0.6	52.9 $\pm$ 0.9
Timbre Diff.	3	60.0 $\pm$ 1.3	65.7 $\pm$ 1.4
Prosodic Sal.	3	62.0 $\pm$ 1.2	66.0 $\pm$ 1.5
Artic. Clarity	4	59.8 $\pm$ 1.3	66.8 $\pm$ 1.4
All Combined	11	63.4 $\pm$ 1.2	69.3 $\pm$ 1.5

Table 5: Multivariate pairwise prediction via logistic regression on acoustic feature differences (5-fold CV,

N{=}15{,}000

pairs).

Backbone	NLL	Acc.(%)	AUC(%)
wav2vec2 [ 5 ]	0.5139	0.7430	0.8247
WavLM [ 6 ]	0.4284	0.8105	0.8944
HuBERT [ 27 ]	0.3852	0.8243	0.9082
data2vec [ 28 ]	0.4686	0.7709	0.8580

Table 6: Ablation study on SSL backbones for anime-likeness preference prediction (pairwise test set,

N=2500

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions
Aug 10, 2026Oluwanifemi Bamgbose, Jash Shah, Simon Rosen +6Audio-Language Model Evaluation Human Preference Evaluation
ServiceNow
Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs
Jun 28, 2026Nithin Rao Koluguri, Sasha Meister, Nikolay Karpov +4ASR Evaluation Automatic Speech Recognition
NVIDIA, USA
ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge
Jun 23, 2026Jisu Jeon, Seungyeon Jwa, Joosung Lee +6Audio-Language Model Evaluation Pairwise Comparison
Hongik University, South Korea · Seoul National University, South Korea · NAVER Cloud, South Korea +1

Aug 10, 2026J/K move · Enter open · S save

Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

Oluwanifemi Bamgbose, Jash Shah, Simon Rosen, Lindsay Devon Brin+5

ServiceNow

Audio-Language Model Evaluation Human Preference Evaluation TTS Synthesis Speech Generation Evaluation TTS Evaluation Speech Quality Assessment+1

Open paper AskPDF arXiv

Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Model (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors are more sensitive to acoustic signal-level properties at the prosodic and paralinguistic levels, while Audio-LLM judges show selective, prompt-dependent detection that favors word-level errors and does not generalize across all dimensions. Neither class reliably captures a breadth of linguistically structured, human-perceptible speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.

Abstract

Automated Evaluation

Explore similar work

Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge