Efficient Training for Cross-lingual Speech Language Models
Authors: Yan Zhou, Qingkai Fang, Yun Hong, Yang Feng
Organizations: Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
Currently, large language models (LLMs) predominantly focus on the text modality. To enable more natural human-AI interaction, speech LLMs are emerging, but building effective end-to-end speech LLMs remains challenging due to limited data and the difficulty in expanding to more languages. In this paper, we introduce Cross-lingual Speech Language Model (CSLM), an efficient training method for cross-lingual speech LLMs based on discrete speech tokens. We propose a novel alignment strategy that achieves cross-modal and cross-lingual alignment through continual pre-training. By conducting instruction fine-tuning following a speech-text interleaved chain-of-modality generation process, we enhance modal alignment at a finer granularity, thereby improving generation quality and reducing latency. CSLM aligns different modalities and languages simultaneously without the need for massive speech data, thus exhibiting good language scalability. Evaluations on cross-modal tasks, mono-lingual conversational tasks, and cross-lingual conversational tasks demonstrate CSLM's strong cross-modal alignment capabilities and general task abilities. (Code is available at: https://github.com/ictnlp/CSLM)
Figures & tables
Figure 1: Alignment strategy of CSLM.
Figure 2: Model architecture and inference process of CSLM.
Figure 3: The construction process of the SFT dataset.
Task
Dataset
Model
GT
Whisper
CosyVoice
SpeechGPT
AnyGPT
GLM-4-Voice
Moshi
CSLM-base
CSLM-SFT
ASR
LibriSpeech
2.5
–
18.9
8.5
2.8
5.7
6.7
9.8
–
TTS
LibriSpeech
–
3.4
24.6
27.1
–
4.7
3.2
3.8
3.0
TTS
LibriTTS
–
2.9
29.1
27.9
5.6
–
3.2
4.5
2.9
TTS
VCTK
–
3.7
5.7
8.5
–
–
2.8
2.7
3.5
Table 1: Results of English ASR and TTS tasks. The test datasets include the test-clean set of LibriSpeech ( Panayotov et al., 2015 ) , the test-clean set of LibriTTS ( Zen et al., 2019 ) , and VCTK ( Yamagishi et al., 2019 ) . The Whisper model refers to whisper-large-v3, and the CosyVoice model refers to CosyVoice-300M-SFT. The last column “GT” is an abbreviation for “ground truth”, representing the error rates of the original speech-text pairs from the dataset calculated using whisper-large-v3.
Task
Dataset
Model
GT
Whisper
CosyVoice
GLM-4-Voice
CSLM-base
CSLM-SFT
ASR
AISHELL-1
9.3
–
2.5
8.6
9.0
–
ASR
AISHELL-2
5.4
–
–
7.6
8.6
–
ASR
AISHELL-3
14.8
–
–
9.2
9.6
–
TTS
AISHELL-1
–
3.3
–
3.8
3.7
1.9
TTS
AISHELL-2
–
5.0
–
5.3
5.2
2.8
Table 2: Results of Chinese ASR and TTS tasks. The test datasets include AISHELL-1 ( Bu et al., 2017 ) , AISHELL-2 ( Du et al., 2018 ) and AISHELL-3 ( Shi et al., 2021 ) . The last column “GT” represents the error rates of the original speech-text pairs from the dataset calculated using the paraformer large model.
Dataset
Hours
Speech Tokens
Text Tokens
Cross-modal Data
MLS English
44.7K
3.7B
0.6B
GigaSpeech
10.0K
0.7B
0.2B
WenetSpeech
10.0K
1.1B
0.5B
WenetSpeech4TTS
12.8K
0.6B
0.2B
Cross-lingual Data
Table 3: Statistics of CSLM’s continuing pre-training data.
Model
Speech Data (k hours)
SpeechGPT*
11+
AnyGPT
57
Moshi
7,000+
GLM-4-Voice**
10,000+
CSLM
77
Table 4: Comparison of amounts of speech data in speech-text pairs used by different models. *The data volume of SpeechGPT is calculated based on the datasets listed in its paper ( Zhang et al., 2023 ) . **The data volume of GLM-4-Voice is calculated from the number of speech tokens and its frequency reported in its paper ( Zeng et al., 2024 ) .
Model
GPT Score ↑
UTMOS ↑
ASR-ER ↓
Off-Target ↓
T
S
T
S
En ( InstructS2S-Eval )
SpeechGPT
2.19
2.98
3.9
45.0
–
–
AnyGPT
–
2.31
3.3
–
–
–
GLM-4-Voice
4.10
4.02
4.0
12.8
8.5
9.0
CSLM
3.50
3.27
4.4
9.0
0.0
0.0
Table 5: Evaluation results of SFT models on English and Chinese speech-to-speech conversational benchmarks. The “T” and “S” under GPT-Score denote the evaluation of the generated text and the transcription of the generated speech, respectively. The “T” and “S” under Off-Target denote the off-target ratio assessed for the generated text and speech, respectively. ASR-ER denotes ASR-WER for En or ASR-CER for Zh.
Model
GPT Score ↑
ASR-ER ↓
Off-Target ↓
T
S
T
S
En → Zh ( InstructS2S-Eval )
GLM-4-Voice
4.29
4.23
5.5
8.5
8.5
CSLM
3.31
2.95
17.5
1.0
0.5
Zh → En ( BELLE-eval-S2S )
GLM-4-Voice
1.20
1.16
42.6
97.2
96.8
Table 6: Results of SFT models on En-Zh and Zh-En speech-to-speech conversational tasks.
Model
C-MOS ↑
A-MOS ↑
En → En
En → Zh
Zh → Zh
Zh → En
Overall
SpeechGPT
1.83
-
-
-
3.22
GLM-4-Voice
4.17
3.50
4.42
3.08
4.11
CSLM
3.25
3.42
4.00
4.33
4.00
Table 7: Human evaluation results for C-MOS and A-MOS. En → X and Zh → X directions are evaluated on InstructS2S-Eval and BELLE-eval-S2S, respectively.
Model
Parallel Speech-Text Pairs
Random Speech-Text Pairs
SpeechGPT
1.2%
1.6%
GLM-4-Voice
39.4%
16.9%
CSLM
72.5%
56.7%
Table 8: Comparison of speech-text representation similarity on LibriSpeech test-clean set.
Model
GPT Score ↑
ASR-ER ↓
Off-Target ↓
T
S
T
S
En → Zh ( InstructS2S-Eval )
CSLM
3.31
2.95
17.5
1.0
0.5
– w/o MT
3.00
2.65
25.4
4.0
0.5
Zh → En ( BELLE-eval-S2S )
CSLM
3.53
3.20
7.4
1.6
0.8
Table 9: Performances of models with and without MT data on speech-to-speech conversational tasks.
Model
GPT Score ↑
ASR-ER ↓
Latency(s) ↓
Speedup ↑
T
S
En → En ( InstructS2S-Eval )
CSLM
3.50
3.27
9.0
466.46
× 2.87
– chunk=4
2.82
2.42
15.2
456.88
× 2.93
– full CoM
3.21
2.92
8.5
1338.68
× 1
– w/o TQ
2.01
1.20
8.9
–
–
Table 10: Impact of chain-of-modality forms on speech-to-speech conversations.
College of Computing and Data Science, Nanyang Technological University, Singapore · AI Singapore, National University of Singapore · Institute for Infocomm Research (I2R), A∗STAR, Singapore