Speech Language Models (SLMs) that understand spoken language questions support only a few high-resource languages, limiting access to millions of people worldwide. This gap stems from the scarcity of multilingual speech instruction-tuning datasets. We present MULTISPEECHQA, a large-scale, synthetically generated and human-verified dataset comprising 9200 hours of 10.8 million spoken question-answer pairs in 23 typologically diverse languages. Using MULTISPEECHQA, we also introduce MULTISPEECH-BENCH, a multi-task benchmark for evaluating SLM performance on 23 languages. We compare the performance of a cascading system to open-weight and closed SLMs on MULTISPEECH-BENCH and find that the cascading system outperforms open-weight SLMs but not all closed SLMs. We use MULTISPEECHQA to finetune Qwen 2.5-Omni, which improves its performance on our benchmark. Our findings show that high-quality synthetic datasets offer a cheap solution to improving the multilingual capabilities of SLMs.
Figures & tables
Figure 1: Summary of the dataset creation process. We translate the questions and answers of the Voice Assistant 400K dataset into each of the Aya Expanse languages (in Table 2 ) with SeamlessM4T, synthesise the questions with XTTS for the languages covered by the model and SeamlessM4T for the languages not covered by XTTS, leaving us with roughly 10.8 million text questions, speech questions and text answers.
Dataset
Number of pairs per language
Trivia (Multi-choice, 17K) [ 27 ]
16,528
Trivia (Single-choice, 20K) [ 27 ]
16,529
QA Assistant V1 (7K) [ 28 ]
5,769
QA Assistant V2 (20K) [ 29 ]
16,008
Alpaca GPT-4 (EN, 55K) [ 30 ]
31,293
Identity [ 25 ]
4,306
Table 1: Splits in our dataset drawn from the 470k entried in the VA 400K dataset translated into 22 languages with number of instruction-completion pairs per language.
Language
Language Family
ISO 639-1
TTS Model
Naturalness
Content Understood
Arabic
Afro-Asiatic (Semitic)
ara
XTTS
2.8
3.7
Chinese (Simplified)
Sino-Tibetan (Sinitic)
zho
XTTS
2.8
4.8
Czech
Indo-European (Slavic, West)
ces
XTTS
–
–
Dutch
Indo-European (Germanic, West)
nld
XTTS
3.1
4.4
English
Indo-European (Germanic, West)
eng
–
–
–
French
Indo-European (Romance)
fra
XTTS
3.6
4.3
Table 2: Languages in MultiSpeechQA with their language families, ISO 639-1 codes, TTS model used, and human evaluation scores for naturalness and content understanding.
Figure 2: MultiSpeech-Bench covers three tasks: (1) Spoken Question Answering (SQA), where models are prompted with speech only (no text prompt); (2) Automatic Speech Translation (AST) from speech to English, using CoVoST-2 (X to En) for languages that overlap with the 23 languages in our dataset; and (3) Automatic Speech Recognition (ASR) on CommonVoice for each of the 23 supported languages.
Figure 3: Win rates on MultiSpeech-Bench averaged across languages for the open-weight and commercial SLMs against the baseline cascading system of Whisper combined with Aya Expanse 8B using the Command-A LLM-as-a-Judge. Bars show % wins for each model and % ties (grey lines).
Figure 4: Win rates on MultiSpeech-Bench averaged across languages for the open-weight and commercial SLMs against the baseline cascading system of Whisper combined with Aya Expanse 8B using the GPT-4o LLM-as-a-Judge. Bars show % wins for each model and % ties (grey lines).
Lang.
ASR (WER %) ↓
AST (BLEU / chrF) ↑
Base
Q2-A
Phi4
Q2.5O
Q2.5O FT
Base
Q2-A
Phi4
Q2.5O
Q2.5O FT
ar
14.0
118.1
146.1
45.7
31.5
34.2/54.7
7.3/30.9
0.1/11.9
30.6/53.8
29.7/52.3
cs
26.0
117.4
115.2
100.3
101.6
–
–
–
–
–
de
9.4
33.3
7.0
7.1
7.5
–
–
–
–
–
el
21.8
118.3
114.1
108.0
108.3
–
–
–
–
–
en
3.2
34.6
13.9
13.8
16.6
–
–
–
–
–
Table 3: Speech Recognition (ASR) and Speech Translation (AST) performance across models and languages. The baseline results are for the cascaded Whisper and Aya-Expanse 8B model. We compare the baseline to Qwen2-Audio (Q2-Audio), Phi-4-Multimodal (Phi-4 MM), and Qwen2.5-Omni (Q2.5O). For AST, we select the languages in the dataset that have both X → En and En → X directions. We report CER for Chinese (zh) and Japanese (ja), and report chrF in addition to BLEU for AST performance.
Figure 5: Win rates comparison: Qwen2.5-Omni vs. Qwen2.5-Omni finetuned. Finetuning improves performance across most languages, meaning that our MultiSpeechQA enables better SQA capability. Hebrew, and Czech seem to lag behind, with most of the results being a tie.
Figure 6: Win rates on MultiSpeech-Bench averaged across languages for the Qwen2.5-Omni and the Qwen2.5-Omni model finetuned on MultiSpeechQA . Bars show % wins for each model and % ties (gray).
Hyperparameter
Stage 1
Stage 2
Learning rate
1e-5
1e-5
Warmup steps
800
400
LoRA Rank
64
64
No. of epochs
10
3
Batch size
128
256
Number of samples per language
20 hours
90 000
Table 4: Hyperparameters used to train Stage 1 and Stage 2 of MultiSpeech models.
Figure 7: Win rates on MultiSpeech-Bench averaged across languages for the from-scratch SALMONN models. ALL+AST versus TEN+AST is shown at the top and versus ALL at the bottom. Bars show % wins for each model and % ties (gray).