cs.SDNov 5, 2026

Adaptive and High-Fidelity Matching Algorithm for Retrieval-Based Accent Conversion

Authors: Xinran Wang, Ning Chen, Yixin Luo, Hongqing Zhu, Li Tian

Organizations: School of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, China

Abstract

Accent conversion (AC) aims to transform accent-related acoustic characteristics of the source speech to a target accent while preserving linguistic content and speaker identity. Unlike conventional generative AC models, the retrieval-based ones, which employ k-nearest neighbors (kNN) to directly retrieve acoustic features from target-accent speech and so do not need source-accent speech during training, are more effective. However, existing retrieval-based AC frameworks face three issues: limited intelligibility due to incomplete phoneme coverage from sequential matching pool construction, limited adaptability to source speech variations, and low spectral continuity from coarse-grained style modulation. To address these limitations, we propose Anisotropy-Aware Adaptive k-Nearest Neighbor Accent Conversion (AA-kNN-AC), an enhanced retrieval-based AC model. First, we design a phoneme coverage-based matching pool construction strategy, employing greedy selection with diversity-aware scoring to achieve balanced and near-complete phonetic representation under limited target accent data. Second, we propose an anisotropy-aware adaptive kNN retrieval mechanism that dynamically determines the neighborhood size according to the singular value decomposition (SVD)-based anisotropy analysis, enhancing adaptability to source speech variations. Third, we present a progressive Adaptive Instance Normalization (AdaIN)-based fine-grained decoder by integrating temporal up-sampling within AdaIN blocks, enhancing spectral continuity of the converted speech. Experimental results on the L2-ARCTIC and VCTK datasets demonstrate that the proposed method outperforms existing retrieval-based and generative baselines in both objective and subjective evaluations, achieving high intelligibility, speaker similarity, and naturalness. Demo page is available at https://aa-knn-ac.github.io/demo/.

Figures & tables

Explore similar work

CardsList
  1. Accent Conversion: A Problem-Driven Survey of Sociolinguistic and Technical Constraints

    Apr 30, 2026Yurii Halychanskyi, Jianfeng Steven Guo, Volodymyr KindratenkoSpeech ProcessingVoice Conversion

  2. PHONOS: PHOnetic Neutralization for Online Streaming Applications

    Mar 27, 2026Waris Quamer, Mu-Ruei Tseng, Ghady Nasrallah +1Speech ProcessingSpeaker Anonymization

  3. TokAN: Accent Normalization Using Self-Supervised Speech Tokens

    Jul 4, 2026Qibing Bai, Shuai Wang, Yuhan Du +3