Adaptive and High-Fidelity Matching Algorithm for Retrieval-Based Accent Conversion
Organizations: School of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, China
Abstract
Accent conversion (AC) aims to transform accent-related acoustic characteristics of the source speech to a target accent while preserving linguistic content and speaker identity. Unlike conventional generative AC models, the retrieval-based ones, which employ k-nearest neighbors (kNN) to directly retrieve acoustic features from target-accent speech and so do not need source-accent speech during training, are more effective. However, existing retrieval-based AC frameworks face three issues: limited intelligibility due to incomplete phoneme coverage from sequential matching pool construction, limited adaptability to source speech variations, and low spectral continuity from coarse-grained style modulation. To address these limitations, we propose Anisotropy-Aware Adaptive k-Nearest Neighbor Accent Conversion (AA-kNN-AC), an enhanced retrieval-based AC model. First, we design a phoneme coverage-based matching pool construction strategy, employing greedy selection with diversity-aware scoring to achieve balanced and near-complete phonetic representation under limited target accent data. Second, we propose an anisotropy-aware adaptive kNN retrieval mechanism that dynamically determines the neighborhood size according to the singular value decomposition (SVD)-based anisotropy analysis, enhancing adaptability to source speech variations. Third, we present a progressive Adaptive Instance Normalization (AdaIN)-based fine-grained decoder by integrating temporal up-sampling within AdaIN blocks, enhancing spectral continuity of the converted speech. Experimental results on the L2-ARCTIC and VCTK datasets demonstrate that the proposed method outperforms existing retrieval-based and generative baselines in both objective and subjective evaluations, achieving high intelligibility, speaker similarity, and naturalness. Demo page is available at https://aa-knn-ac.github.io/demo/.
Figures & tables
| Datasets | Role | Speaker (Gender) | Accent / L1 | Duration (h) |
| LibriSpeech [ 22 ] | Training | 251 speakers | American | 100.0 |
| CMU-ARCTIC [ 13 ] | Target accent | BDL (M) | American | 1.0 |
| CMU-ARCTIC [ 13 ] | Target accent | SLT (F) | American | 1.0 |
| L2-ARCTIC [ 34 ] | Source accent | ASI (M) | Hindi (L1) | 1.0 |
| L2-ARCTIC [ 34 ] | Source accent | ABA (M) | Arabic (L1) | 1.1 |
| VCTK | Source accent | P248 (F) | Indian (Regional) * | 0.4 |
| Categories | Models | Training data | Testing data |
| Reproducible | NN-VC [ 2 ] | LibriSpeech (100h) | L2-ARCTIC (ASI, ABA), VCTK (P248) |
| NN-AC [ 18 ] | LibriSpeech (100h) | ||
| TokAN [ 4 ] | LibriTTS (360h) + L2-ARCTIC | ||
| Seed-VC [ 15 ] | Emilia-101k5 (101000h) | ||
| Demos | MLVAE-ADV [ 20 ] | L2-ARCTIC + CMU-ARCTIC | Various official demo samples |
| TTS-AC [ 6 ] | LibriTTS + LJSpeech |
| Objective Metrics | Subjective Metrics | ||||||||
| Dataset | Model | WER(%) | CER(%) | SECS | PPG | RTF | NMOS | SMOS | ACT |
| L2 (ASI) [ 34 ] | NN-VC [ 2 ] | 38.84 | 26.57 | – | 0.93 | 0.05 | – | ||
| NN-AC [ 18 ] | 13.12 | 6.59 | 0.64 | 0.40 | 0.08 | ||||
| TokAN [ 4 ] | 13.24 | 7.53 | 0.82 | 0.38 | 0.61 | ||||
| Seed-VC [ 15 ] | 9.58 | 4.72 | – | 0.69 | 0.20 | – | |||
| Ours | 8.35 | 4.04 | 0.80 | 0.36 | 0.13 | ||||
| Objective Metrics | Subjective Metrics | |||||||
| Dataset | Model | WER(%) | CER(%) | SECS | PPG | NMOS | SMOS | ACT |
| Demo samples provided by corresponding baselines | MLVAE-ADV [ 20 ] | 14.30 | 11.10 | 0.74 | 0.64 | |||
| Ours | 18.20 | 11.00 | 0.79 | 0.50 | ||||
| TTS-AC [ 6 ] | 15.28 | 10.88 | 0.76 | 0.57 | ||||
| Ours | 5.00 | 4.41 | 0.81 | 0.46 | ||||
| VITS-AC [ 21 ] | 11.84 | 6.67 | 0.70 | 0.38 | ||||
| Configuration | Weights | WER(%) | CER(%) | SECS | PPG |
| Proposed setting | (0.20, 0.30, 0.05, 0.40, 0.05) | 8.35 | 4.04 | 0.80 | 0.36 |
| Equal Weights | (0.20, 0.20, 0.20, 0.20, 0.20) | 8.69 | 4.26 | 0.79 | 0.37 |
| w/o Unique Phonemes | (0.00, 0.40, 0.15, 0.30, 0.15) | 9.11 | 4.38 | 0.79 | 0.37 |
| w/o Diversity | (0.30, 0.00, 0.10, 0.50, 0.10) | 9.05 | 4.41 | 0.78 | 0.38 |
| w/o Phoneme Categories | (0.25, 0.35, 0.00, 0.30, 0.10) | 8.96 | 4.32 | 0.79 | 0.37 |
| w/o Rare Phonemes | (0.35, 0.45, 0.10, 0.00, 0.10) | 9.15 | 4.47 | 0.79 | 0.37 |
| Configuration | WER(%) | CER(%) | SECS | PPG |
| Proposed Model | 8.35 | 4.04 | 0.80 | 0.36 |
| w/o anisotropy score correction | 8.92 | 4.35 | 0.79 | 0.37 |
| Unit scale( ) | 8.92 | 4.32 | 0.79 | 0.37 |
| Linear scale( ) | 8.77 | 4.24 | 0.79 | 0.37 |
| Configuration | WER | CER | SECS | PPG |
| Proposed Model | 8.35 | 4.04 | 0.80 | 0.36 |
| w/o progressive AdaIN | 9.06 | 4.42 | 0.77 | 0.39 |
| Direction | Source | Target Pool | WER(%) | CER(%) | SECS | PPG |
| Other-to- American | ASI(M) | BDL(M) | 8.35 | 4.04 | 0.80 | 0.36 |
| SLT(F) | 9.21 | 4.58 | 0.78 | 0.33 | ||
| ABA(M) | BDL(M) | 15.72 | 7.31 | 0.82 | 0.51 | |
| SLT(F) | 16.56 | 8.01 | 0.80 | 0.46 | ||
| P248(F) | BDL(M) | 5.86 | 2.37 | 0.78 | 0.30 | |
| SLT(F) | 5.69 | 2.34 | 0.77 | 0.31 |
| Range | WER | CER | SECS | PPG |
| Ours | 8.35 | 4.04 | 0.80 | 0.36 |
| 9.32 | 4.47 | 0.79 | 0.36 | |
| 8.90 | 4.31 | 0.79 | 0.36 |
Voice Conversion
Voice conversion changes a speaker's vocal identity or delivery while preserving the spoken content.
See all 38 papers