CosyAccent: Duration-Controllable Accent Normalization Using Source-Synthesis Training Data

Qibing Bai; Shuhao Shi; Shuai Wang; Yukai Ju; Yannan Wang; Haizhou Li

eess.ASFeb 22, 2026

CosyAccent: Duration-Controllable Accent Normalization Using Source-Synthesis Training Data

Authors: Qibing Bai, Shuhao Shi, Shuai Wang, Yukai Ju, Yannan Wang, Haizhou Li

Organizations: SDS, The Chinese University of Hong Kong, Shenzhen, China · SRIBD, The Chinese University of Hong Kong, Shenzhen, China · Tencent Ethereal Audio Lab, Tencent, Shenzhen, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China ·

Abstract

Accent normalization (AN) systems often struggle with unnatural outputs and undesired content distortion, stemming from both suboptimal training data and rigid duration modeling. In this paper, we propose a "source-synthesis" methodology for training data construction. By generating source L2 speech and using authentic native speech as the training target, our approach avoids learning from TTS artifacts and, crucially, requires no real L2 data in training. Alongside this data strategy, we introduce CosyAccent, a non-autoregressive model that resolves the trade-off between prosodic naturalness and duration control. CosyAccent implicitly models rhythm for flexibility yet offers explicit control over total output duration. Experiments show that, despite being trained without any real L2 speech, CosyAccent achieves significantly improved content preservation and superior naturalness compared to strong baselines trained on real-world data.

Figures & tables

Figure 1: Construction pipeline of the paired training data.

Figure 2: CosyAccent architecture. It implicitly models rhythm for prosodic flexibility, while allowing the total duration to be either specified or predicted.

Figure 3: Speech decoder’s alignment mechanism. Positional indices for the source content features are scaled to match the target’s length, creating a coarse alignment within RoPE-based cross attention.

System	Source-length	Subjective			Objective
System	Source-length	NAT ( $\uparrow$ )	ACT ( $\downarrow$ )	SIM ( $\uparrow$ )	WER (% $\downarrow$ )	UTMOS ( $\uparrow$ )	SECS ( $\uparrow$ )	$\Delta\text{PPG}$ ( $\downarrow$ )
Source	$\checkmark$	65.78 $\pm$ 2.18	50.45 $\pm$ 2.22	-	15.86	2.81	-	0.51
FramAN [ 2 ]	$\checkmark$	58.13 $\pm$ 2.19	44.08 $\pm$ 2.19	$-$ 0.075	21.54	2.56	0.8065	0.49
TokAN-1 [ 1 ]	$\times$	63.63 $\pm$ 1.97	29.44 $\pm$ 1.87	0.060	16.21	2.86	0.8563	0.30
TokAN-2 [ 1 ]	$\checkmark$	57.25 $\pm$ 2.19	31.98 $\pm$ 2.00	$-$ 0.027	16.71	2.76	0.8613	0.30
CosyAccent-1	$\times$	64.62 $\pm$ 1.92	31.04 $\pm$ 1.91	0.033	12.96	3.04	0.8213	0.38

Table 1: Evaluation results of the accent normalization systems

System	Zh	Hi	Vi	Ar	Es	Ko
Source	20.82	11.75	31.40	15.09	15.79	13.78
FramAN	26.76	19.98	34.36	22.96	22.28	18.91
TokAN-1	17.18	18.53	24.51	17.25	16.68	13.81
TokAN-2	17.80	18.95	25.20	17.36	17.52	14.43
CosyAccent-1	16.95	7.55	25.53	13.97	13.22	10.15
CosyAccent-2	17.77	7.52	24.41	13.97	14.40	11.03

Table 2: Accent-wise WERs with native-only ASR

System	WER ( $\%\downarrow$ )	SECS ( $\uparrow$ )	$\Delta\text{PPG}$ ( $\downarrow$ )
Source	15.86	-	0.51
CosyAccent-2	13.26	0.8291	0.37
w/o CTC loss	15.61	0.8324	0.41
w/o speaker embedding	13.51	0.6524	0.37
w/o position scaling	Model Collapse

Table 3: Ablation results of CosyAccent-2

Speech Processing

Speech processing develops computational methods for recognizing, synthesizing, analyzing, and transforming spoken audio.

See all 412 papers

Controllable Accent Normalization via Discrete Diffusion
Mar 15, 2026Qibing Bai, Yuhan Du, Tom Ko +3Audio Diffusion Models Speech Processing
SDS, The Chinese University of Hong Kong, Shenzhen, China · Tencent Ethereal Audio Lab, Tencent, Shenzhen, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China +3
TokAN: Accent Normalization Using Self-Supervised Speech Tokens
Jul 4, 2026Qibing Bai, Shuai Wang, Yuhan Du +3
School of Data Science (SDS), The Chinese University of Hong Kong, Shenzhen (CUHKSZ), China · Tencent Ethereal Audio Lab, Tencent, Shenzhen, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China +4
Joycent: Multi-Accent TTS via Disentangled Accent Modeling and Layer-Specific Conditioning
Jun 15, 2026Xintong Wang, Junchuan Zhao, Ye WangDisentangled Representation Learning TTS Synthesis
School of Computing, National University of Singapore, Singapore

Mar 15, 2026J/K move · Enter open · S save

Controllable Accent Normalization via Discrete Diffusion

Qibing Bai, Yuhan Du, Tom Ko, Shuai Wang+2

SDS, The Chinese University of Hong Kong, Shenzhen, China · Tencent Ethereal Audio Lab, Tencent, Shenzhen, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China+3

Audio Diffusion Models Speech Processing Diffusion Models

Open paper AskPDF arXiv

Existing accent normalization methods do not typically offer control over accent strength, yet many applications-such as language learning and dubbing-require tunable accent retention. We propose DLM-AN, a controllable accent normalization system built on masked discrete diffusion over self-supervised speech tokens. A Common Token Predictor identifies source tokens that likely encode native pronunciation; these tokens are selectively reused to initialize the reverse diffusion process. This provides a simple yet effective mechanism for controlling accent strength: reusing more tokens preserves more of the original accent. DLM-AN further incorporates a flow-matching Duration Ratio Predictor that automatically adjusts the total duration to better match the native rhythm. Experiments on multi-accent English data show that DLM-AN achieves the lowest word error rate among all compared systems while delivering competitive accent reduction and smooth, interpretable accent strength control. The implementation is available at https://github.com/P1ping/DLM-AN

Abstract

Speech Processing

Explore similar work

Controllable Accent Normalization via Discrete Diffusion

TokAN: Accent Normalization Using Self-Supervised Speech Tokens

Joycent: Multi-Accent TTS via Disentangled Accent Modeling and Layer-Specific Conditioning