Efficient Training for Cross-lingual Speech Language Models
Authors: Yan Zhou, Qingkai Fang, Yun Hong, Yang Feng
Organizations: Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
Currently, large language models (LLMs) predominantly focus on the text modality. To enable more natural human-AI interaction, speech LLMs are emerging, but building effective end-to-end speech LLMs remains challenging due to limited data and the difficulty in expanding to more languages. In this paper, we introduce Cross-lingual Speech Language Model (CSLM), an efficient training method for cross-lingual speech LLMs based on discrete speech tokens. We propose a novel alignment strategy that achieves cross-modal and cross-lingual alignment through continual pre-training. By conducting instruction fine-tuning following a speech-text interleaved chain-of-modality generation process, we enhance modal alignment at a finer granularity, thereby improving generation quality and reducing latency. CSLM aligns different modalities and languages simultaneously without the need for massive speech data, thus exhibiting good language scalability. Evaluations on cross-modal tasks, mono-lingual conversational tasks, and cross-lingual conversational tasks demonstrate CSLM's strong cross-modal alignment capabilities and general task abilities. (Code is available at: https://github.com/ictnlp/CSLM)
Figures & tables
Figure 1: Alignment strategy of CSLM.
Figure 2: Model architecture and inference process of CSLM.
Figure 3: The construction process of the SFT dataset.
Task
Dataset
Model
GT
Whisper
CosyVoice
SpeechGPT
AnyGPT
GLM-4-Voice
Moshi
CSLM-base
CSLM-SFT
ASR
LibriSpeech
2.5
–
18.9
8.5
2.8
5.7
6.7
9.8
–
TTS
LibriSpeech
–
3.4
24.6
27.1
–
4.7
3.2
3.8
3.0
TTS
LibriTTS
–
2.9
29.1
27.9
5.6
–
3.2
4.5
2.9
TTS
VCTK
–
3.7
5.7
8.5
–
–
2.8
2.7
3.5
Table 1: Results of English ASR and TTS tasks. The test datasets include the test-clean set of LibriSpeech ( Panayotov et al., 2015 ) , the test-clean set of LibriTTS ( Zen et al., 2019 ) , and VCTK ( Yamagishi et al., 2019 ) . The Whisper model refers to whisper-large-v3, and the CosyVoice model refers to CosyVoice-300M-SFT. The last column “GT” is an abbreviation for “ground truth”, representing the error rates of the original speech-text pairs from the dataset calculated using whisper-large-v3.
Task
Dataset
Model
GT
Whisper
CosyVoice
GLM-4-Voice
CSLM-base
CSLM-SFT
ASR
AISHELL-1
9.3
–
2.5
8.6
9.0
–
ASR
AISHELL-2
5.4
–
–
7.6
8.6
–
ASR
AISHELL-3
14.8
–
–
9.2
9.6
–
TTS
AISHELL-1
–
3.3
–
3.8
3.7
1.9
TTS
AISHELL-2
–
5.0
–
5.3
5.2
2.8
Table 2: Results of Chinese ASR and TTS tasks. The test datasets include AISHELL-1 ( Bu et al., 2017 ) , AISHELL-2 ( Du et al., 2018 ) and AISHELL-3 ( Shi et al., 2021 ) . The last column “GT” represents the error rates of the original speech-text pairs from the dataset calculated using the paraformer large model.
Dataset
Hours
Speech Tokens
Text Tokens
Cross-modal Data
MLS English
44.7K
3.7B
0.6B
GigaSpeech
10.0K
0.7B
0.2B
WenetSpeech
10.0K
1.1B
0.5B
WenetSpeech4TTS
12.8K
0.6B
0.2B
Cross-lingual Data
Table 3: Statistics of CSLM’s continuing pre-training data.
Model
Speech Data (k hours)
SpeechGPT*
11+
AnyGPT
57
Moshi
7,000+
GLM-4-Voice**
10,000+
CSLM
77
Table 4: Comparison of amounts of speech data in speech-text pairs used by different models. *The data volume of SpeechGPT is calculated based on the datasets listed in its paper ( Zhang et al., 2023 ) . **The data volume of GLM-4-Voice is calculated from the number of speech tokens and its frequency reported in its paper ( Zeng et al., 2024 ) .
Model
GPT Score ↑
UTMOS ↑
ASR-ER ↓
Off-Target ↓
T
S
T
S
En ( InstructS2S-Eval )
SpeechGPT
2.19
2.98
3.9
45.0
–
–
AnyGPT
–
2.31
3.3
–
–
–
GLM-4-Voice
4.10
4.02
4.0
12.8
8.5
9.0
CSLM
3.50
3.27
4.4
9.0
0.0
0.0
Table 5: Evaluation results of SFT models on English and Chinese speech-to-speech conversational benchmarks. The “T” and “S” under GPT-Score denote the evaluation of the generated text and the transcription of the generated speech, respectively. The “T” and “S” under Off-Target denote the off-target ratio assessed for the generated text and speech, respectively. ASR-ER denotes ASR-WER for En or ASR-CER for Zh.
Model
GPT Score ↑
ASR-ER ↓
Off-Target ↓
T
S
T
S
En → Zh ( InstructS2S-Eval )
GLM-4-Voice
4.29
4.23
5.5
8.5
8.5
CSLM
3.31
2.95
17.5
1.0
0.5
Zh → En ( BELLE-eval-S2S )
GLM-4-Voice
1.20
1.16
42.6
97.2
96.8
Table 6: Results of SFT models on En-Zh and Zh-En speech-to-speech conversational tasks.
Model
C-MOS ↑
A-MOS ↑
En → En
En → Zh
Zh → Zh
Zh → En
Overall
SpeechGPT
1.83
-
-
-
3.22
GLM-4-Voice
4.17
3.50
4.42
3.08
4.11
CSLM
3.25
3.42
4.00
4.33
4.00
Table 7: Human evaluation results for C-MOS and A-MOS. En → X and Zh → X directions are evaluated on InstructS2S-Eval and BELLE-eval-S2S, respectively.
Model
Parallel Speech-Text Pairs
Random Speech-Text Pairs
SpeechGPT
1.2%
1.6%
GLM-4-Voice
39.4%
16.9%
CSLM
72.5%
56.7%
Table 8: Comparison of speech-text representation similarity on LibriSpeech test-clean set.
Model
GPT Score ↑
ASR-ER ↓
Off-Target ↓
T
S
T
S
En → Zh ( InstructS2S-Eval )
CSLM
3.31
2.95
17.5
1.0
0.5
– w/o MT
3.00
2.65
25.4
4.0
0.5
Zh → En ( BELLE-eval-S2S )
CSLM
3.53
3.20
7.4
1.6
0.8
Table 9: Performances of models with and without MT data on speech-to-speech conversational tasks.
Model
GPT Score ↑
ASR-ER ↓
Latency(s) ↓
Speedup ↑
T
S
En → En ( InstructS2S-Eval )
CSLM
3.50
3.27
9.0
466.46
× 2.87
– chunk=4
2.82
2.42
15.2
456.88
× 2.93
– full CoM
3.21
2.92
8.5
1338.68
× 1
– w/o TQ
2.01
1.20
8.9
–
–
Table 10: Impact of chain-of-modality forms on speech-to-speech conversations.
While the shift from cascaded dialogue systems to end-to-end (E2E) speech Large Language Models (LLMs) improves latency and paralinguistic modeling, E2E models often exhibit a significant performance degradation compared to their text-based counterparts. The standard Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) training methods fail to close this gap. To address this, we propose X-OPD, a novel Cross-Modal On-Policy Distillation framework designed to systematically align the capabilities of Speech LLMs to their text-based counterparts. X-OPD enables the Speech LLM to explore its own distribution via on-policy rollouts, where a text-based teacher model evaluates these trajectories and provides token-level feedback, effectively distilling teacher's capabilities into student's multi-modal representations. Extensive experiments across multiple benchmarks demonstrate that X-OPD significantly narrows the gap in complex tasks while preserving the model's inherent capabilities.
Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA.
Shreyas Gopal, Donghang Wu, Ashutosh Anshul +5
College of Computing and Data Science, Nanyang Technological University, Singapore · AI Singapore, National University of Singapore · Institute for Infocomm Research (I2R), A∗STAR, Singapore
Speech Language Models (SLMs) that understand spoken language questions support only a few high-resource languages, limiting access to millions of people worldwide. This gap stems from the scarcity of multilingual speech instruction-tuning datasets. We present MULTISPEECHQA, a large-scale, synthetically generated and human-verified dataset comprising 9200 hours of 10.8 million spoken question-answer pairs in 23 typologically diverse languages. Using MULTISPEECHQA, we also introduce MULTISPEECH-BENCH, a multi-task benchmark for evaluating SLM performance on 23 languages. We compare the performance of a cascading system to open-weight and closed SLMs on MULTISPEECH-BENCH and find that the cascading system outperforms open-weight SLMs but not all closed SLMs. We use MULTISPEECHQA to finetune Qwen 2.5-Omni, which improves its performance on our benchmark. Our findings show that high-quality synthetic datasets offer a cheap solution to improving the multilingual capabilities of SLMs.
Tolúlopé Ògúnrèmí, Dan Jurafsky, Chris Manning +2
Department of Computer Science, Stanford University, United States · Cohere Labs, Canada · Cohere, Canada