With recent advances in Text-to-Speech (TTS) systems, synthetic audiobook narration has seen increased interest, reaching unprecedented levels of naturalness. However, larger gaps remain in synthetic narration systems' ability to impersonate fictional characters, and convey complex emotions or prosody. A promising direction to enhance character identification is the assignment of plausible voices to each fictional characters in a book. This step typically requires complex inference of attributes in book-length contexts, such as a character's age, gender, origin or physical health, which in turns requires dedicated benchmark datasets to evaluate extraction systems' performances. We present S-VoCAL (Speaking Voice Character Attributes in Literature), the first dataset and evaluation framework dedicated to evaluate the inference of voice-related fictional character attributes. S-VoCAL entails 8 attributes grounded in sociophonetic studies, and 952 character-book pairs derived from Project Gutenberg. Its evaluation framework addresses the particularities of each attribute, and includes a novel similarity metric based on recent Large Language Models embeddings. We demonstrate the applicability of S-VoCAL by applying a simple Retrieval-Augmented Generation (RAG) pipeline to the task of inferring character attributes. Our results suggest that the RAG pipeline reliably infers attributes such as Age or Gender, but struggles on others such as Origin or Physical Health. The dataset and evaluation code are available at https://github.com/AbigailBerthe/S-VoCAL .
Figures & tables
Attribute
Voice relevance
Wikidata properties
Type
Coverage
Is New
Name
label
Text
100%
✗
Aliases
also known as, given name, native label, nickname, pseudonym
List
73.5%
✗
Age ∗
pitch range, formant frequencies, speech rate
relative age
Categorical (4)
4.1% ∣ 37.7%
✗
Gender
pitch range, resonance
sex or gender
Categorical (2)
96.5%
✗
Origin
accent, phonetic features, prosody
country of citizenship, place of birth, country of origin
List
58.8%
✓
Residence
accent, speech patterns
residence, work location, country, place of residence
List
26.5%
✓
Table 1: Attributes overview in S-VoCAL, along with their mapped Wikidata properties. Age ∗ is represented as a relative category inferred from voice features, with coverage reported as (initial | final, after manual annotation). The Is New column shows originality compared to Iosif and Mishra (2014) .
Figure 1: As attributes from S-VoCAL comes from various Gutenberg books, they are unbalanced, especially for Type that mostly contains humans and Age that mostly contains adults.
Age category
F1-score
Instances
Child
0.78
16
Teenager
0.44
10
Adult
0.90
242
Senior
0.71
25
Table 2: Per-class F1-scores of human annotations for the Age attribute. Adult , the most represented class, shows a high level of agreement, while Teenager shows only moderate agreement.
Attribute
Prediction
Gold reference
Human score
Interpretation Scale
Origin
America
France
0.0
Prediction is entirely incorrect
Occupation
Writer
Author, nurse
0.5
Partially correct, lacks some occupations
Physical health
Speech disorder
Stuttering
0.66
Not precise but correct
Type
Kitty
Cat
1.0
Correct or synonym
Table 3: Examples of the interpretation scale used by annotators to evaluate each systems’ predictions on open-class attributes.
Attribute
α
ρ -Qwen3
ρ -BERT
n
Origin
0.95
0.85
0.44
66
Residence
0.93
0.85
0.31 ( p=0.013 )
65
P. Health
0.74
0.52
-0.10 ( p=0.572 )
37
Occupation
0.75
0.66
0.32
121
Type (n-h.)
0.92
0.86
0.63
57
Table 4: Inter-annotator agreement ( α ) and Spearman correlation ( ρ ) between human scores and automatic metrics. All correlations are significant at p<0.001 , except where specified Abbreviations: P. Health = Physical Health ; Type (n-h.) = Type (non-human) .
Figure 2: Overview of the RAG pipeline. The first step extracts attribute-specific relevant excerpts for a character. These passages are then prompted to an LLM along with an attribute prompt.
Qwen3-8b
Phi4-14b
Baseline
Weighted F1
Gender
0.993
0.997
0.561
Age
0.783
0.800
0.660
Type (h./n-h.)
0.969
0.971
0.910
Soft F1
Age
0.927
0.941
0.908
Micro F1
Spoken- Languages
0.712
0.749
0.576
Table 5: Results for close and semi-closed class attributes, with their dedicated metric. Type (h./n-h.) corresponds to the binary classification of Type Human vs Non-Human.
mHAS ↑
Attribute
Qwen3-8b
Phi4-14b
Occupation
0.522
0.486
Residence
0.498
0.502
Origin
0.420
0.422
Type (n-h.)
0.785
0.781
Physical health
0.141
0.165
Table 6: Mean Human-Aligned Score (mHAS), for open attributes. For most attributes, values of 0.5 indicates a partially correct prediction, with lacking information. Type (n-h.) = Type (non-human).
High-fidelity character voice synthesis is a cornerstone of immersive multimedia applications, particularly for interacting with anime avatars and digital humans. However, existing systems struggle to maintain consistent persona traits across diverse emotional contexts. To bridge this gap, we present ATRIE, a unified framework utilizing a Persona-Prosody Dual-Track (P2-DT) architecture. Our system disentangles generation into a static Timbre Track (via Scalar Quantization) and a dynamic Prosody Track (via Hierarchical Flow-Matching), distilled from a 14B LLM teacher. This design enables robust identity preservation (Zero-Shot Speaker Verification EER: 0.04) and rich emotional expression. Evaluated on our extended AnimeTTS-Bench (50 characters), ATRIE achieves state-of-the-art performance in both generation and cross-modal retrieval (mAP: 0.75), establishing a new paradigm for persona-driven multimedia content creation.
Aoduo Li, Haoran Lv, Hongjian Xu +5
Guangdong University of Technology Guangzhou, China · South China University of Technology Guangzhou, China · Shenzhen Polytechnic University Shenzhen, China +2
Instruction-based text-to-speech (ITTS) systems enable natural-language control of expressive speech generation, but often offer limited transparency and fine-grained control over individual text units. Character-level controllable TTS systems provide explicit acoustic control, yet typically rely on user-specified acoustic attributes. To bridge this gap, we propose InstCharVoice, a unified framework that grounds natural-language instructions in character-level acoustic control. We first construct grounded instruction annotations on the WordVoice-5A-zh corpus using Qwen3-Omni. With this supervision, we train an autoregressive model to identify instruction-relevant characters and predict their acoustic attributes before generating the corresponding speech tokens. Keyword prediction and grounding-aware loss weighting help the model focus on instruction-relevant characters and attributes. Experiments show improved instruction following and keyword-level acoustic control over representative ITTS systems, with competitive speech naturalness and explicit character-level controllability. Audio samples are available at https://xxh333.github.io/instcharvoice-demo/.
Sihang Nie, Xueru Li, Xiaofen Xing +4
South China University of Technology · Huya Inc. · The Hongkong Polytechnic University
We introduce Conversational Voice Aesthetic Model, a speech large language model for describing the voice aesthetics of real or synthetic speech responses in natural conversational contexts. Given a context and a response speech, CVAM describes salient moments that characterize the voice and predicts nine categorical attributes spanning gender, pitch, pacing, emotion, and delivery. The key challenge lies in perceptual fields such as emotion and delivery, which are inherently subjective and lack definitive ground truth. Therefore, we collect ~10 human annotations for each of 3k real and synthetic responses derived from the CANDOR corpus. CVAM is supervised finetuned on synthesized aesthetic descriptions and labels, then optimized with Group Relative Policy Optimization on human judgments. Experiments show that CVAM better agrees with human listeners than Gemini 3.1 Pro and open-source speech LLMs, and outperforms single-human-vs.-rest agreement. Together, we demonstrate the importance of grounding voice aesthetics in human perception and propose a principled framework for human alignment.
Xilin Jiang, Shun Zhang, Tejas Jayashankar +2
Columbia University New York, NY, USA · Meta Superintelligence Labs Menlo Park, CA, USA