Speech Language Models (SLMs) that understand spoken language questions support only a few high-resource languages, limiting access to millions of people worldwide. This gap stems from the scarcity of multilingual speech instruction-tuning datasets. We present MULTISPEECHQA, a large-scale, synthetically generated and human-verified dataset comprising 9200 hours of 10.8 million spoken question-answer pairs in 23 typologically diverse languages. Using MULTISPEECHQA, we also introduce MULTISPEECH-BENCH, a multi-task benchmark for evaluating SLM performance on 23 languages. We compare the performance of a cascading system to open-weight and closed SLMs on MULTISPEECH-BENCH and find that the cascading system outperforms open-weight SLMs but not all closed SLMs. We use MULTISPEECHQA to finetune Qwen 2.5-Omni, which improves its performance on our benchmark. Our findings show that high-quality synthetic datasets offer a cheap solution to improving the multilingual capabilities of SLMs.
Figures & tables
Figure 1: Summary of the dataset creation process. We translate the questions and answers of the Voice Assistant 400K dataset into each of the Aya Expanse languages (in Table 2 ) with SeamlessM4T, synthesise the questions with XTTS for the languages covered by the model and SeamlessM4T for the languages not covered by XTTS, leaving us with roughly 10.8 million text questions, speech questions and text answers.
Dataset
Number of pairs per language
Trivia (Multi-choice, 17K) [ 27 ]
16,528
Trivia (Single-choice, 20K) [ 27 ]
16,529
QA Assistant V1 (7K) [ 28 ]
5,769
QA Assistant V2 (20K) [ 29 ]
16,008
Alpaca GPT-4 (EN, 55K) [ 30 ]
31,293
Identity [ 25 ]
4,306
Table 1: Splits in our dataset drawn from the 470k entried in the VA 400K dataset translated into 22 languages with number of instruction-completion pairs per language.
Language
Language Family
ISO 639-1
TTS Model
Naturalness
Content Understood
Arabic
Afro-Asiatic (Semitic)
ara
XTTS
2.8
3.7
Chinese (Simplified)
Sino-Tibetan (Sinitic)
zho
XTTS
2.8
4.8
Czech
Indo-European (Slavic, West)
ces
XTTS
–
–
Dutch
Indo-European (Germanic, West)
nld
XTTS
3.1
4.4
English
Indo-European (Germanic, West)
eng
–
–
–
French
Indo-European (Romance)
fra
XTTS
3.6
4.3
Table 2: Languages in MultiSpeechQA with their language families, ISO 639-1 codes, TTS model used, and human evaluation scores for naturalness and content understanding.
Figure 2: MultiSpeech-Bench covers three tasks: (1) Spoken Question Answering (SQA), where models are prompted with speech only (no text prompt); (2) Automatic Speech Translation (AST) from speech to English, using CoVoST-2 (X to En) for languages that overlap with the 23 languages in our dataset; and (3) Automatic Speech Recognition (ASR) on CommonVoice for each of the 23 supported languages.
Figure 3: Win rates on MultiSpeech-Bench averaged across languages for the open-weight and commercial SLMs against the baseline cascading system of Whisper combined with Aya Expanse 8B using the Command-A LLM-as-a-Judge. Bars show % wins for each model and % ties (grey lines).
Figure 4: Win rates on MultiSpeech-Bench averaged across languages for the open-weight and commercial SLMs against the baseline cascading system of Whisper combined with Aya Expanse 8B using the GPT-4o LLM-as-a-Judge. Bars show % wins for each model and % ties (grey lines).
Lang.
ASR (WER %) ↓
AST (BLEU / chrF) ↑
Base
Q2-A
Phi4
Q2.5O
Q2.5O FT
Base
Q2-A
Phi4
Q2.5O
Q2.5O FT
ar
14.0
118.1
146.1
45.7
31.5
34.2/54.7
7.3/30.9
0.1/11.9
30.6/53.8
29.7/52.3
cs
26.0
117.4
115.2
100.3
101.6
–
–
–
–
–
de
9.4
33.3
7.0
7.1
7.5
–
–
–
–
–
el
21.8
118.3
114.1
108.0
108.3
–
–
–
–
–
en
3.2
34.6
13.9
13.8
16.6
–
–
–
–
–
Table 3: Speech Recognition (ASR) and Speech Translation (AST) performance across models and languages. The baseline results are for the cascaded Whisper and Aya-Expanse 8B model. We compare the baseline to Qwen2-Audio (Q2-Audio), Phi-4-Multimodal (Phi-4 MM), and Qwen2.5-Omni (Q2.5O). For AST, we select the languages in the dataset that have both X → En and En → X directions. We report CER for Chinese (zh) and Japanese (ja), and report chrF in addition to BLEU for AST performance.
Figure 5: Win rates comparison: Qwen2.5-Omni vs. Qwen2.5-Omni finetuned. Finetuning improves performance across most languages, meaning that our MultiSpeechQA enables better SQA capability. Hebrew, and Czech seem to lag behind, with most of the results being a tie.
Figure 6: Win rates on MultiSpeech-Bench averaged across languages for the Qwen2.5-Omni and the Qwen2.5-Omni model finetuned on MultiSpeechQA . Bars show % wins for each model and % ties (gray).
Hyperparameter
Stage 1
Stage 2
Learning rate
1e-5
1e-5
Warmup steps
800
400
LoRA Rank
64
64
No. of epochs
10
3
Batch size
128
256
Number of samples per language
20 hours
90 000
Table 4: Hyperparameters used to train Stage 1 and Stage 2 of MultiSpeech models.
Figure 7: Win rates on MultiSpeech-Bench averaged across languages for the from-scratch SALMONN models. ALL+AST versus TEN+AST is shown at the top and versus ALL at the bottom. Bars show % wins for each model and % ties (gray).
Spoken language models (SLMs) unify speech perception and reasoning, but adapting them to sensitive domains is underexplored, especially when the original training data is inaccessible and the use case demands multilingual, spoken-query interaction. We adapt an open-source SLM to the Singaporean Home Team context across five speech tasks in Singapore's four official languages, combining LoRA fine-tuning, a surrogate text-QA dataset that guards against catastrophic forgetting, and a multi-task objective that adapts the CoBa reweighting scheme to speech. We also build HTD-multilingual-QA, a 504,853 sample multilingual QA dataset in text and spoken form. The resulting HT-Moonstone (5B) matches or outperforms SLMs up to 7x its size on most tasks, attains the best accent and gender recognition among all models evaluated, and loses under 2% of its original speech QA ability.
Ng Jia Sheng Jason
Language AI R&D, xData Home Team Science & Technology Agency (HTX), Singapore
Spoken Language Models (SLMs) generate textual responses directly from speech, offering an alternative to cascaded systems. Despite recent advances, existing SLMs still exhibit weaker instruction-following behavior and limited generalization across diverse tasks compared to text-based language models. Our analysis shows that speech and text representations in current SLMs remain weakly aligned despite strong downstream performance, indicating that structural differences between continuous, temporally varying speech and discrete text remain insufficiently addressed. To address this, we propose a simple framework that decouples length mismatch from semantic alignment and encourages closer correspondence between speech and text representations. Experiments across multiple benchmarks demonstrate competitive performance against strong baselines, underscoring the importance of explicitly addressing structural differences between speech and text in SLM training. Our code is publicly available at https://github.com/jaykim9870/Do_SLMs_Hear_Speech_as_They_Read_Text.
This paper describes our system for Task2 of the second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge. We adapt Qwen3-Omni-30B-A3B-Instruct with a segment-evidence-aware data and post-training pipeline. A language model converts timestamped ASR into coherent event spans, which are expanded by a boundary margin and cropped from the original recording. We then synthesize complementary semantic MCQs with Qwen3.6-27B and acoustic MCQs with Gemini3.1 Flash-Lite, followed by structural, grounding, answer-consistency, and target-model trainability checks, yielding 359,825 verified MCQs across 21 language and accent variants. A text-only probe partitions the data into weak, text-answerable items used for supervised fine-tuning and strong, audio-dependent items used for reinforcement learning with Group Sequence Policy Optimization (GSPO), stabilized by debiased advantages, sequence-level importance correction, and dynamic filtering. Our system obtains 90.92% accuracy on the final official evaluation set.