Dialect-Robust Speech Language Models with Synthetic Pseudo-Dialect Augmentation

Shunsuke Mitsumori; Tomoya Mizumoto; Yusuke Fujita

cs.CLOct 7, 2026

Dialect-Robust Speech Language Models with Synthetic Pseudo-Dialect Augmentation

Authors: Shunsuke Mitsumori, Tomoya Mizumoto, Yusuke Fujita

Organizations: SB Intuitions · Waseda University, Tokyo, Japan

Abstract

Speech Language Model (SLM) performance often degrades on dialects due to data scarcity. Conventional text-to-speech (TTS) augmentation struggles to cover diverse dialects as it requires a certain amount of real dialect speech. We propose synthesizing pseudo-dialect speech by converting LLM-generated dialect text via a standard-language TTS model, requiring zero real dialect speech. Additionally, we introduce intermediate standard-text prediction during training, acting as semantic normalization for downstream tasks. We evaluate dialect understanding via dialect-to-English speech translation across Japanese, German, and Chinese dialects. Compared to synthetic standard speech baselines, pseudo-dialect augmentation improves scores for Japanese (from 25.38 to 26.24) and German (from 31.57 to 32.47). Furthermore, the intermediate standard-text prediction effectively bridges the semantic gap, boosting performance to 28.26 for Japanese and from 11.67 to 16.37 for Chinese. These results suggest that our approach scales to various languages without requiring speech resources specific to each dialect.

Figures & tables

Fig. 1: Proposed pseudo-dialect speech augmentation. From a standard pair $\{x_{s},t_{s}\}$ and translation $y_{s}$ , an LLM and standard TTS model generate pseudo-dialect text $\tilde{t}_{d}$ and speech $\tilde{x}_{d}$ . This yields $\{\tilde{x}_{d},y_{s}\}$ and $\{\tilde{x}_{d},t_{s},y_{s}\}$ for single-task and multi-task SLM training, respectively.

Output Targets	Transcription ( $\tilde{t}_{d}$ or $t_{s}$ )	Standard text ( $t_{s}$ )	Translation text ( $y$ )	Output Format
ASR+English	✓		✓	<ASR> $\tilde{t}_{d}$ or $t_{s}$ </ASR><en> $y$ </en>
Standard+English		✓	✓	<standard> $t_{s}$ </standard><en> $y$ </en>
ASR+Standard+English	✓	✓	✓	<ASR> $\tilde{t}_{d}$ or $t_{s}$ </ASR><standard> $t_{s}$ </standard><en> $y$ </en>

TABLE I: Output formats for each multi-task learning configuration. Variables follow the notation introduced in Section 2, where the transcription target (

\tilde{t}_{d}

t_{s}

) depends on the input speech.

Synthetic data added	Output Target	Japanese	German	Chinese
No Augmentation (baseline)	English	24.21	22.37	11.61
Synthetic standard speech	English	25.38	31.57	11.72
Pseudo-dialect speech	English	26.24	32.47	11.67
Pseudo-dialect speech	ASR+English	27.67	29.81	16.30
Pseudo-dialect speech	Standard+English	28.26	30.73	16.37
Pseudo-dialect speech	ASR+Standard+English	24.95	29.92	13.24

TABLE II: Comparison of dialect understanding capabilities evaluated via dialect-to-English speech translation performance (BLEU) across multiple languages, data augmentation settings, and task output formats. For the output targets, “English” denotes direct translation, while “ASR” and “Standard” represent intermediate speech transcription and standard-text prediction, respectively.

Synthetic data added	Avg.	Akita	Hiroshima	Iyo	Fukuoka	Awa	Hokkaido	Fukui	Kyoto	Miyazaki	Enshu
No Augmentation (baseline)	24.21	19.10	25.97	27.49	26.72	25.34	30.19	27.69	24.35	21.73	23.83
Synthetic standard speech	25.38	19.90	26.67	27.58	28.46	26.29	30.17	29.57	25.97	23.23	24.70
Pseudo-dialect speech	26.24	20.12	27.39	29.16	30.39	27.23	30.78	30.13	27.14	24.98	25.22

TABLE III: Comparison of English translation performance (BLEU) across different types of data augmentation, showing the average score and individual results for 20 Japanese dialects.

Output Targets	Avg.	Akita	Hiroshima	Iyo	Fukuoka	Awa	Hokkaido	Fukui	Kyoto	Miyazaki	Enshu
English (Single-task)	26.24	20.12	27.39	29.16	30.39	27.23	30.78	30.13	27.14	24.98	25.22
ASR+English	27.67	23.26	30.24	31.27	33.50	28.58	30.96	31.51	27.94	26.96	26.29
Standard+English	28.26	22.20	32.53	31.68	32.25	29.72	31.21	30.27	29.88	26.93	26.39
ASR+Standard+English	24.95	19.23	25.12	25.93	29.18	27.13	27.45	29.42	26.46	22.22	25.36

TABLE IV: Impact of intermediate prediction on English translation (BLEU), showing the average score and individual results for 20 Japanese dialects. All models were trained on the dataset augmented with pseudo-dialect speech.

Category	Input Audio (Dialect)	English Ground Truth	Baseline Output	Proposed Output
Improved 1	ほんで食材も	And we also ran out	And I also bought	And we also ran out of
(Kyoto)	たりひんかったんどす．	of ingredients.	some ingredients.	ingredients.
Improved 2	きっと素敵な休日に	I’m sure it will be a	I hope you have a	I’m sure it will be a
(Iwaki)	なっぺよ．	wonderful holiday.	wonderful holiday.	wonderful holiday.
Failure 1	がめるものがめるもの，	The things I pick up	There are a lot of	There are a lot of
(Kanazawa)*	古いものが多いがやて．	tend to be old.	old things here.	old things here.

TABLE V: Qualitative analysis of translation outputs from the English translation task. Asterisk (*) and dagger (†) indicate failure cases due to unseen vocabulary and acoustic gaps.

Output Target	Output Content
Ground Truth	ASR: 必要ねぇのにスマートフォンばもってる理由は、緊急連絡用に、親がわさけだはんで。 Standard: 必要がないのにスマートフォンをもっている理由は、緊急連絡用にと親が私に与えたから。 English: The reason I have a smartphone even though I don’t need one is because my parents gave it to me for emergency contact purposes.
English (Single-task)	English: The reason I have a smartphone even though I don’t need one is because my parents insisted, saying it’s for emergencies .
ASR+English	ASR: 必要ねぇのにスマートフォンば持ってる理由は、緊急連絡用に、親が話さけだはんで。 English: The reason for carrying a smartphone even though it’s not necessary is because it’s for emergency contact, and parents can’t talk .
Standard+English	Standard: 必要ないのにスマートフォンを持っている理由は緊急連絡用に親が買ってくれたから。 English: The reason I have a smartphone even though I don’t need one is because my parents bought it for me as an emergency contact.

TABLE VI: Comparison of English translation outputs across multi-task configurations. All models were trained on the identical dataset augmented with pseudo-dialect speech, differing only in their target output formats.

Dataset	SLM	Whisper	Cascade
# Total params	8,683.7M	1,543.5M	9,615.7M
# Trainable params	18.4M	1,543.5M	1,585.4M
No training	-	14.36	19.87
No Augmentation	24.21	17.78	29.78
+ Standard	25.38	18.44	29.65
+ Pseudo-dialect	26.24	18.98	30.44

TABLE VII: Effectiveness of pseudo-dialect augmentation across architectures in English translation (BLEU). “No training” means no fine-tuning. “+ Standard” and “+ Pseudo-dialect” add synthetic standard and pseudo-dialect speech. SLM scores match averages in Table III .

Explore similar work

DialectS2S: End-to-End Speech Dialogue Modeling for Low-Resource Chinese Dialects
Aug 8, 2026Yi Shu, Tianyu Peng, Yingzhuo Deng +5Dialects Self-Supervised Speech Models
School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research +1
Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models
Jun 24, 2026Tomoya Mizumoto, Yusuke Fujita, Hao Shi +3Dialects Speech Language Models
SB Intuitions, Tokyo, Japan
PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects
May 31, 2026Sicheng Yang, Shulan Ruan, Shiwei Wu +4Multilingual Automatic Speech Recognition Dialects
Shenzhen International Graduate School, Tsinghua University Shenzhen, China · Department of Electronic Engineering, Tsinghua University Beijing, China · JD AI Research Beijing, China

Aug 8, 2026, cs.CLJ/K move · Enter open · S save

DialectS2S: End-to-End Speech Dialogue Modeling for Low-Resource Chinese Dialects

Yi Shu, Tianyu Peng, Yingzhuo Deng, Wen Yang+4

School of Artificial Intelligence, University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Wuhan AI Research+1

Dialects Self-Supervised Speech Models Semantic Representations

Open paper AskPDF arXiv

Current end-to-end speech dialogue models are primarily optimized for mainstream languages and remain limited in low-resource dialect scenarios due to the scarcity of dialect speech data. Moreover, during dialect adaptation, the semantic representation space of speech dialogue models continuously evolves, while conventional speech supervision remains unchanged, leading to semantic inconsistency between hidden representations and speech targets and degrading speech stability and naturalness. To address these issues, we propose DialectS2S, an end-to-end speech dialogue model for Chinese dialects. We first develop a scalable dialect speech dialogue synthesis pipeline for efficient data construction. We further introduce a two-stage post-training strategy with self-aligned speech supervision, which aligns the semantic content of speech supervision with the evolved semantic representations of the model to improve dialect speech generation quality. Experimental results show that DialectS2S consistently outperforms existing baselines across multiple Chinese dialects in speech dialogue, achieving substantial improvements in dialect consistency, response quality, and speech intelligibility. Our work provides an efficient and scalable solution for end-to-end speech dialogue modeling in low-resource dialect scenarios. To facilitate future research and practical applications, we fully open-source the DialectS2S framework, including model checkpoints, training datasets, and fine-tuning code.

Abstract

Explore similar work

DialectS2S: End-to-End Speech Dialogue Modeling for Low-Resource Chinese Dialects

Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models

PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects