cs.CLMar 8, 2026
SaveAccent Vector: Controllable Accent Manipulation for Multilingual TTS Without Accented Data
Organizations: Signal Analysis and Interpretation Lab, University of Southern California, USA · Thomas Lord Department of Computer Science, University of Southern California, USA · Department of Linguistics, University of Southern California
Abstract
Accent is an integral part of society, reflecting multiculturalism and shaping how individuals express identity. The majority of English speakers are non-native (L2) speakers, yet current Text-To-Speech (TTS) systems primarily model American-accented English due limited accented data. We propose Accent Vector, a controllable representation that enables accent manipulation in multilingual TTS without requiring accented training data. Accent Vector is derived by fine-tuning a TTS system on native speech of a different language (i.e. non-English) and computing task vectors capturing accent characteristics (i.e. in English). By scaling and interpolating the vector, we achieve fine-grained control over accent strength and generate mixed-accent speech. In addition, it generalizes beyond English, enabling accent control across multiple languages. Objective and human evaluations confirm the effectiveness of Accent Vector for fine-grained and compositional accent control.
Figures & tables
| \multirow 2* Approach | Requiring | Explicit | Zero-shot Speaker | Cross-Accent | Multilingual |
|---|---|---|---|---|---|
| Accented Dataset | Accent Control | Adaptation | Mixing | Support | |
| Xinyuan et al. [ 20 ] | Required | ✗ | ✓ | ✗ | ✗ |
| Onda et al. [ 15 ] | Not Required | ✗ | ✓ | ✗ | ✗ |
| MacST [ 16 ] | Not Required | ✗ | ✓ | ✗ | ✗ |
| Lertpetchpun et at. [ 21 ] | Not Required | Phoneme-level | ✗ | ✗ | ✗ |
| Accent Vector (Ours) | Not Required | Weight Scaling | ✓ | ✓ | ✓ |
Table 1: Comparison between accented TTS models.
Figure 1: Accent Vector Framework. The top panel illustrates the fine-tuning procedure, and the bottom panel shows the inference process for generating accented speech. During inference, a language ID token (e.g., [en]) is concatenated with the transcript and provided as input to the model.
Figure 2: Accent Vector Arithmetic. The figure illustrates the computation of the Accent Vector, its scaling for accent strength control, and the interpolation of multiple Accent Vectors for mixed-accent synthesis.
| Language | Training Dataset | Subset | Duration (Hours) | UTMOS (Train) | UTMOS (Eval) |
|---|---|---|---|---|---|
| English | VCTK | English | 8.9 | 3.92 | 3.86 |
| Spanish | CommonVoice-sp | Penisular | 52.21 | 2.79 | 2.61 |
| Hindi | IndicVoices-R | Uttar Pradesh and Madhya Pradesh | 27.2 | 2.57 | 2.57 |
| German | CommonVoice-de | German-Non-NRW and -NRW Area | 375.12 | 2.93 | 2.86 |
| French | CommonVoice-fr | France | 337.56 | 2.63 | 2.50 |
| Mandarin | KeSpeech | Beijing, Northeastern | 125.53 | 2.57 | 2.68 |
Table 2: Languages and corresponding datasets. UTMOS is reported after filtering low-quality samples.
| \multirow 2* Target Accent | Target Accent Prob. (%) | Target Accent SIM. (%) | \multirow 2*WER / CER | \multirow 2*SSIM | \multirow 2*UTMOS | |||
|---|---|---|---|---|---|---|---|---|
| Pretrained | Finetuned | Pretrained | Finetuned | |||||
| British (England Accent) | 23.3 | 56.7 \color gray (+143.83%) | 0.47 | 0.73 | 5.46 / 4.31 | 0.90 | 3.61 | |
| Spanish | 15.5 | 39.7 \color gray (+156.37%) | 0.41 | 0.61 | 17.3 / 10.3 | 0.88 | 2.86 | |
| Hindi | 2.2 | 24.2 \color gray (+1021.29%) | 0.40 | 0.64 | 10.4 / 6.5 | 0.88 | 3.10 | |
| French | 12.2 | 23.2 \color gray (+90.10%) | 0.38 | 0.56 | 18.9 / 11.4 | 0.87 | 2.76 | |
| German | 14.3 | 27.4 \color gray (+91.14%) | 0.50 | 0.52 | 11.0 / 6.8 | 0.86 | 3.04 | |
Table 3: Objective evaluation on accent shifting from American English to each target accent. We report the accent prediction probabilities (Prob) and accent cosine similarities measured by VoxProfile, intelligibility (WER/CER), speaker similarity (SSIM), and naturalness (UTMOS).
| \multirow 2* Target Language | English Accent Prob. (%) | English Language Prob. (%) | \multirow 2*WER / CER | \multirow 2*SSIM | \multirow 2*UTMOS | ||
|---|---|---|---|---|---|---|---|
| Pretrained | Finetuned | Pretrained | Finetuned | ||||
| Spanish | 1.20 | 44.69 | 0.04 | 38.1 | 30.1 / 14.6 | 0.89 | 3.49 |
| German | 8.57 | 41.57 | 1.13 | 35.1 | 34.2 / 12.6 | 0.89 | 3.55 |
| Mandarin | 0.00 | 3.03 | 0.03 | 10.8 | – / 31.6 | 0.82 | 3.51 |
Table 4: Objective evaluation on English-accented speech generation for non-English languages. We report English accent probability measured by VoxProfile, English language prediction probability by the LID model, intelligibility (WER/CER), speaker similarity (SSIM), and naturalness (UTMOS).
Figure 3: Control of accent strength using the task vector coefficient. We evaluate the model fine-tuned on British and Hindi by measuring accent probability and word error rate (WER) across different task vector coefficients. The first and second rows correspond to the results of British- and Hindi-accented English, respectively. A clear trade-off between accent strength and intelligibility is observed; increasing the coefficient produces a stronger accent but results in higher WER.
Figure 4: Effect of the task vector coefficient when mixing multiple Accent Vectors. We combine Spanish and English Accent Vectors using coefficients and , respectively. Accent probabilities for Spanish and English accents are evaluated across different values of . The vertical dashed line indicates the accent probability of the pretrained model of each accent.
| \multirow 2* Target Accent | Prob on Target Accent (%) | SIM on Target Accent | \multirow 2*WER / CER | \multirow 2*SSIM | \multirow 2*MOS | ||
|---|---|---|---|---|---|---|---|
| Pretrained | Finetuned | Pretrained | Finetuned | ||||
| Spanish + British | 14.58/9.29 | 5.61/20.45 | 0.412/0.295 | 0.307/0.442 | 8.58/5.06 | 0.84 | 3.22 |
| Mandarin + British | 27.38/4.31 | 13.30/17.77 | 0.557/0.156 | 0.445/0.355 | 13.0/9.57 | 0.82 | 3.00 |
| Hindi + British | 2.16/3.39 | 7.11/9.64 | 0.397/0.057 | 0.447/0.157 | 8.24/5.60 | 0.86 | 3.32 |
| Spanish + Hindi | 14.58/1.05 | 24.29/10.92 | 0.412/0.271 | 0.561/0.508 | 11.3/7.14 | 0.88 | 3.03 |
| Mandarin + Hindi | 27.38/0.87 | 28.84/8.70 | 0.557/0.316 | 0.617/0.535 | 19.1/13.4 | 0.85 | 2.79 |
Table 5: Results of interpolating the Accent Vectors. VoxProfile measures accent characteristics, cosine similarity evaluates speaker similarity, and UTMOS. The order of accent probabilities follows the order of the target accent columns. For example, the probabilities for the first row are for Spanish/British. The LoRA weights for both languages are merged with a coefficient of 0.5 each.
| Accent | Accuracy (%) | Strength | Naturalness |
|---|---|---|---|
| US | 80.00 | 3.63 | 3.06 |
| British (England) | 78.46 | 3.55 | 3.91 |
| German | 53.85 | 3.23 | 3.03 |
| French | 66.15 | 3.53 | 3.00 |
| Spanish | 53.85 | 3.00 | 3.28 |
| Hindi | 78.46 | 3.65 | 3.15 |
Table 6: Subjective Evaluation Results: Accent Identification Accuracy, Perceived Accent Strength (Correct Responses Only), and Naturalness.
Figure 5: Confusion matrix on the target accents and perceived accents by human listeners.