We present Walrus, a unified 8B speech-and-audio foundation model handling speech recognition (including code-switching and multi-talker speech), speech-to-text and speech-to-speech translation, text-to-speech, emotion recognition, audio captioning, and general audio understanding. Our main contribution is not a new architecture but a data-centric training recipe; we use a standard decoder-only backbone initialized from Gemma 4, so that gains can be traced to the data. For speech generation, Walrus predicts low-bitrate discrete speech tokens, and a lightweight token-to-waveform decoder converts them into high-fidelity audio. The recipe is a three-stage data curriculum that moves from broad multilingual data to multi-talker, code-switching, and domain-specific data. At each stage, task-aware and dataset-aware weighting balances tasks with very different token budgets and protects core skills from forgetting. Trained on over 600M samples covering more than 20 tasks and 100 languages, Walrus leads on most multilingual, multi-talker, and code-switching ASR, speech translation, and audio understanding benchmarks, outperforming both open speech language models of comparable size and dedicated task-specific models. On most benchmarks, it even matches or surpasses the 4× larger Qwen3-Omni's performance. Results suggest that data engineering is a primary driver of quality in multi-task speech and audio modeling.
Figures & tables
Figure 1: Overview of Walrus, our most capable SpeechLLM yet. Input speech is encoded by the USM encoder into continuous representations, which are passed to the decoder-only Gemma 4 backbone together with a natural-language task prompt. For recognition, translation, and understanding tasks, the backbone generates text. For TTS and S2ST, it generates discrete speech tokens at 25 Hz (325 bits per second), which a VQ decoder and a Vocos-based vocoder convert into a waveform. The vocoder is also conditioned on the source speech to transfer speaker identity and prosody.
Figure 2: Walrus against the best previous system (among AF3, Gemma4, Qwen2.5-Omni-7B, Phi4-MM) across recognition, translation, and understanding benchmarks. Scores are normalized so that higher is better (ASR and CS-ASR shown as word/character accuracy, translation as BLEU or ASR-BLEU, understanding as accuracy, captioning as CLAP-Sim × 100). Both Walrus variants advance the frontier on most axes, and the Research variant extends it further, most notably on code-switching, internal entertainment ASR, and audio understanding.
Table 1: Task families supported by Walrus, with language coverage. ASR and its variants support more than 100 languages. CS-ASR supports English-Japanese, and English or Japanese mixed with Chinese, French, German, Hindi, Italian, or Spanish (both as major and minor script). MT-ASR supports 2-, 3-, and 4-speaker mixtures in Chinese, English, French, German, Italian, Japanese, and Spanish. AST and S2ST support translation from 10 languages into English or Japanese, and from English or Japanese into the same 10 languages. Closed captioning supports Chinese, English, French, German, Italian, Japanese, and Spanish.
Figure 3: Training data composition of Walrus. Recognition dominates the mix (45%), followed by translation (27%) and understanding (21%), while generation is deliberately kept small (7%). The detailed breakdown shows that standard ASR and instruction-following data anchor the recognition and understanding families. This distribution reflects our data-centric design: core recognition is heavily represented to protect it from forgetting, while token-heavy generative tasks are downweighted.
Stage
Steps
Learning rate
Warmup
Decay
1: Foundation
87,500
2×10−6
1%
5%
2: Expansion
135,000
2×10−6
1%
5%
3: Specialization
150,000
1×10−6
1%
5%
Table 2: Training schedule per stage. Each stage is initialized from the previous checkpoint. Warmup and decay are given as a fraction of total steps.
Task
Stage 1
Stage 2
Stage 3
Recognition
ASR
5.0
5.0
4.0
Context / Keyword ASR
3.0
3.0
3.0
Timestamp ASR
–
1.0
1.0
CS-ASR
–
1.0
1.0
MT-ASR
–
1.0
1.0
Table 3: Task weights across the three stages. Weights scale each task’s contribution to the loss. A dash means the task is not yet present. Recognition is upweighted throughout to protect the core, while generation and understanding are raised in Stage 3.
Task
Dataset
Metric
Prior SOTA
Walrus-CR
Walrus-R
Recognition
ASR
LS Clean/Other Panayotov et al. (2015)
WER ↓
1.57 / 3.13 (AF3)
1.88/4.99
1.65/3.85
ASR
MLS Pratap et al. (2020)
WER ↓
8.20 (Phi4-MM)
9.14
6.89
ASR
FLEURS Conneau et al. (2023)
WER ↓
4.00 (Phi4-MM)
7.54
5.33
ASR
VoxPopuli Wang et al. (2021)
WER ↓
11.78 (Gemma4)
8.38
7.84
ASR
Internal Entertainment (avg)
WER ↓
24.35 (Gemma4)
26.71
17.45
Table 4: Performance comparison across all evaluated benchmarks. For each dataset, we report the best prior system among comparable 4-10B models, including Gemma4 , Phi4-MM , AF3 , Q2.5O (Prior SOTA), the Commercial-Ready Walrus ( Walrus-CR ), and the Research Walrus ( Walrus-R ). Metrics: WER/CER ( ↓ ) for ASR tasks, cpWER/cpCER ( ↓ ) for MT-ASR, BLEU/ASR-BLEU ( ↑ ) for AST/S2ST, accuracy ( ↑ ) for IF and SER, CIDEr/CLAP-Sim ( ↑ ) for captioning, R@1/5/10 ( ↑ ) for retrieval. Averages over languages; per-language breakdowns are in Appendix § C . Phi4-MM results on MLS and FLEURS are reported as averages across supported languages, i.e., with Ar/Hi/Ko/Nl/Pl excluded. Best results per dataset in bold .
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Slot
Filled with
{lang1} , {lang2}
Source and target language
{context}
Preceding text, as discourse context
{keywords}
List of candidate keywords
{labels}
Closed set of allowed labels, for example emotions
{speaker}
Target speaker for target-speaker queries
{start} , {end}
Time span for time-conditioned queries
Appendix
Table 5: Slot variables used in the prompt templates.
Pipeline
What we build
Tools
Quality control
Transcript cleaning
Casing and punctuation
Restored transcripts of real ASR corpora
Qwen3.5-9B with few-shot prompts
CER ≤ 0.05 against the original; length check
Text normalization
Consistent references across languages
Rules for European, CJK, and Hindi/Arabic scripts
Manual inspection of samples
Recognition
Context ASR
Target utterance with 1-5 preceding utterances
MLS chapters, internal domains
–
Keyword ASR
Keyword lists of 0-100 items with distractors
Named-entity recognition
Entities in at least two utterances
Appendix
Table 6: Overview of the data pipelines. “Tools” lists the models and recipes used, “Quality control” the filters applied to the output. A dash means that we apply no filter beyond the construction itself.
Model
Clean
Other
WLv3
2.31
4.29
Gemma4
3.22
7.74
Phi4-MM
1.68
3.83
AF3
1.57
3.13
Q25O
1.83
3.44
Q3O
1.22
2.48
Appendix
Table 7: ASR Results: LibriSpeech (WER ↓ ). WLv3 indicates Whisper-Large-v3, AF3 Audio-Flamingo3, Q250/Q30 denote Qwen2.5Omni and Qwen3Omni, respectively. Best results in bold , second-best underlined.
Model
DE
EN
ES
FR
IT
NL
PL
PT
AVG
WLv3
5.54
5.30
4.01
4.60
9.09
10.36
4.89
7.20
6.37
Gemma4
8.71
9.15
5.43
7.86
14.63
12.06
10.44
9.10
9.67
Phi4-MM
7.59
8.04
10.49
5.15
11.63
N/A
N/A
6.29
8.20 ∗
AF3
27.45
5.02
12.57
20.07
19.96
28.46
53.50
17.26
23.04
Q25O
6.54
6.14
8.21
7.92
20.27
17.56
19.21
11.96
12.23
Q3O
3.77
3.77
3.41
3.00
9.72
8.94
12.06
5.22
6.24
Appendix
Table 8: ASR Results: MLS (WER ↓ ). *This average score reflects only the supported languages.
Model
AR
DE
EN
ES
FR
HI
IT
JA
KO
PT
ZH
AVG
WLv3
16.40
5.96
6.24
4.26
6.96
18.61
4.07
6.55
4.34
5.55
9.03
7.99
Gemma4
16.20
6.10
6.50
3.50
8.00
8.60
3.20
7.80
5.30
4.60
13.60
7.58
Phi4-MM
N/A
3.96
3.38
3.02
4.35
N/A
1.98
4.50
N/A
3.98
6.83
4.00 ∗
AF3
N/A
22.53
5.06
25.83
39.82
74.45
19.74
N/A
54.65
49.65
N/A
36.47
Q25O
18.75
7.67
6.46
5.70
5.08
14.54
7.63
8.12
7.03
6.50
7.29
8.62
Q3O
14.84
5.02
7.36
4.27
5.10
7.59
3.94
4.85
3.31
5.25
6.78
6.21
Appendix
Table 9: ASR Results: FLEURS (WER/CER ↓ ). *This average score reflects only the supported languages.
Model
DE
EN
ES
FR
IT
AVG
WLv3
18.26
9.89
10.19
9.98
27.21
15.11
Gemma4
13.76
8.67
8.79
10.89
16.80
11.78
Phi4-MM
12.13
5.91
10.18
13.18
22.67
12.81
AF3
108.73
5.55
95.36
91.02
142.65
86.66
Q25O
15.84
6.64
13.24
11.39
23.67
14.16
Q3O
11.60
7.66
8.35
8.06
15.73
10.28
Appendix
Table 10: ASR Results: VoxPopuli (WER ↓ ). 5 major EU languages are considered.
Model
DE
ES
FR
IT
JA
PT
RU
ZH
AVG
WLv3
6.78
4.19
12.28
5.10
10.46
4.52
4.21
16.37
7.99
Gemma4
9.33
5.55
14.58
7.10
16.46
8.16
9.16
47.37
14.71
Phi4-MM
10.82
3.57
6.05
3.53
36.91
10.26
N/A
48.29
17.49 ∗
AF3
28.22
14.52
22.27
18.23
139.96
18.72
95.60
143.12
60.08
Q25O
6.67
7.51
8.44
8.62
19.85
12.88
7.95
10.69
10.33
Q3O
4.71
4.26
6.91
4.37
8.87
6.83
3.51
4.70
5.52
Appendix
Table 11: ASR Results: CoVoST-2 (WER/CER ↓ ). *This average score reflects only the supported languages.
Model
DE
EN-UK
EN-US
ES
FR
IT
JA
WLv3
22.33
8.31
11.10
11.67
22.53
13.56
24.77
Gemma4
39.20
16.86
17.99
17.72
34.96
18.35
25.37
Phi4-MM
69.58
17.44
20.25
26.91
66.01
22.40
202.35
AF3
63.90
16.05
17.74
47.20
88.49
41.51
169.11
Q25O
34.54
18.55
22.72
30.55
37.68
28.55
43.18
Q3O
27.11
11.85
11.14
15.50
21.95
13.00
25.88
Appendix
Table 12: A SR Results: Internal entertainment data (WER/CER ↓ ) . Per-language averages over different test sets.
Model
EN-HI
EN-JA
EN-DE
EN-FR
EN-ZH
AVG
Read Test Set
WLv3
42.75
35.41
6.10
12.21
17.58
22.81
Gemma4
47.50
48.07
19.18
18.17
28.56
32.30
Phi4-MM
N/A
69.29
29.15
32.78
81.71
53.23 ∗
AF3
97.95
85.09
64.50
73.62
86.69
81.57
Q25O
56.96
63.68
41.66
37.21
58.42
51.58
Appendix
Table 13: CS-ASR Results: CS-FLEURS Read (top) and XTTS (bottom) test sets, CER ( ↓ ). *This average score reflects only the supported languages.
LibriMix (Clean 2Mix)
Japanese Daily Talk
Model
cpWER
tcpWER
cpCER
tcpCER
VibeVoice-ASR [ 46 ]
59.34
59.36
71.99
72.17
Gemma4
N/A
N/A
N/A
N/A
Phi4-MM
N/A
N/A
N/A
N/A
AF3
N/A
N/A
N/A
N/A
Q25O
N/A
N/A
N/A
N/A
Appendix
Table 14: MT-ASR Results: LibriMix Clean 2Mix (left) and Japanese Daily Talk (right). Metrics: cpWER/cpCER and tcpWER/tcpCER ( ↓ ). tcpWER/tcpCER were calculated with equidistant_interval option and five-second collar tolerance. Japanese Daily Talk is evaluated on CEJC (Corpus of Everyday Japanese Conversation) on 30-second segments.
Model
DE
ES
FR
IT
JA
PT
RU
ZH
AVG
WLv3
36.14
37.06
33.87
35.96
23.53
53.05
41.61
11.50
34.09
Gemma4
42.01
37.15
41.04
43.02
25.50
51.12
46.44
17.90
38.02
Phi4-MM
39.81
43.60
42.24
41.42
30.54
55.28
N/A
22.39
39.33 ∗
AF3
32.80
24.19
23.34
26.68
13.08
36.46
33.49
13.24
25.41
Q25O
42.42
40.51
37.46
34.95
14.57
56.65
50.44
22.58
37.44
Q3O
46.06
43.01
41.95
42.61
30.54
59.65
54.02
28.09
43.24
Appendix
Table 15: AST Results: CoVoST-2, X → En (BLEU ↑ ). *This average score reflects only the supported languages.
Model
AR
DE
ES
FR
HI
IT
JA
KO
PT
ZH
AVG
X → En
WLv3
20.63
33.27
22.27
31.67
19.40
23.57
16.51
19.05
36.68
15.80
23.89
Gemma4
29.56
33.55
23.11
31.22
27.46
23.74
22.47
23.44
36.79
21.63
27.29
Phi4-MM
N/A
37.71
25.33
35.10
N/A
26.06
21.62
N/A
40.80
22.37
29.86 ∗
AF3
14.05
33.27
23.43
28.83
8.62
23.08
11.90
14.32
35.39
16.43
20.93
Q25O
18.79
35.30
24.81
31.68
16.09
23.94
18.36
19.01
41.48
19.43
24.89
Appendix
Table 16: AST Results: FLEURS, X → En (top) and En → X (bottom), BLEU ( ↑ ). *This average score reflects only the supported languages.
Model
DE
ES
FR
IT
JA
PT
RU
ZH
AVG
Gemma4
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Phi4-MM
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
AF3
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Q25O
40.39
38.32
35.54
33.67
15.61
55.10
48.15
21.39
36.02
Q3O
44.10
41.32
38.30
40.21
30.04
58.12
52.19
25.90
41.27
Walrus-CR
49.21
47.12
40.73
41.89
32.14
57.33
52.58
25.16
43.27
Appendix
Table 17: S2ST Results: CVSS-C, X → En (ASR-BLEU ↑ ).
Model
AR
DE
ES
FR
HI
IT
JA
KO
PT
ZH
AVG
X → En
Gemma4
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Phi4-MM
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
AF3
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Q25O
20.63
36.40
23.71
31.54
15.78
25.04
17.77
20.39
38.91
21.33
25.15
Q3O
32.80
43.10
29.30
41.00
31.52
30.34
25.61
29.30
45.41
28.30
33.60
Appendix
Table 18: S2ST Results: FLEURS, X → En (top) and En → X (bottom), ASR-BLEU ( ↑ ).
X → En
En → X
Model
ASR-BLEU
UTMOS
SSim
ESim
APCP
ASR-BLEU
UTMOS
SSim
ESim
APCP
Kraken [ 16 ]
32.30
3.13
0.75
0.77
2.48
N/A
N/A
N/A
N/A
N/A
Seamless [ 3 ]
32.10
3.06
0.66
0.76
2.05
27.33
2.82
0.58
0.56
2.25
Q2.5O
25.10
3.48
0.49
0.53
2.40
N/A
N/A
N/A
N/A
N/A
Q3O
33.60
3.59
0.61
0.75
2.19
25.67
3.27
0.49
0.44
2.12
Walrus-CR
28.59
3.17
0.75
0.76
2.54
17.19
3.01
0.67
0.69
2.41
Appendix
Table 19: Speech-to-speech translation on FLEURS, X → En (left) and En → X (right). Metrics are ASR-BLEU, UTMOS (predicted naturalness), speaker similarity (SSim), emotion similarity (ESim), and AutoPCP (APCP, predicted prosody similarity). Higher is better for all metrics, and scores are averages over languages. N/A means that the system does not support the direction. Qwen3-Omni (Q3O, shaded) is much larger and is not ranked. Among the other systems, the best score is in bold and the second best is underlined.
Task
Primary prompt
Recognition
ASR
Transcribe the {lang1} speech:
Context-ASR
Transcribe the {lang1} speech from the context {context}:
Keyword-biased ASR
Transcribe the {lang1} speech, biasing toward these keywords: {keywords}:
Code-switched ASR
Transcribe the code-switched speech in {lang1} and {lang2}:
Dialogue-contextual ASR
Given the previous conversation between {agent} and {user}: {context} Transcribe the {lang1} speech:
Appendix
Table 20: Primary (most frequently sampled) prompt for each task. During training, each prompt is drawn uniformly from a pool of 10-20 paraphrases with the listed prompt oversampled; slot variables ( {lang1} , {lang2} , {context} , {keywords} , {labels} , {speaker} , {start} , {end} , {agent} , {user} ) are filled at data-construction time.
A universal audio representation should capture fine-grained speech cues and high-level semantics for environmental sounds and music in a single encoder. Existing encoders often excel in one domain but degrade in others. We propose UniWhisper, an efficient continual multi-task training framework that casts heterogeneous audio tasks into a unified instruction and answer format. This enables standard next-token training without task-specific heads and losses. We train it on 38k hours of public audio and assess the encoder using shallow MLP probes and k-nearest neighbors (kNN) on 20 tasks spanning speech, environmental sound, and music. UniWhisper reaches normalized weighted averages of 0.81 with MLP probes and 0.61 with kNN, compared to 0.64 and 0.46 for Whisper, while retaining strong speech performance.
Yuxuan Chen, Peize He, Haoyuan Yu +1
Jilin University · University of Electronic Science and Technology of China · Hunan University +1
Speech Language Models (SLMs) that understand spoken language questions support only a few high-resource languages, limiting access to millions of people worldwide. This gap stems from the scarcity of multilingual speech instruction-tuning datasets. We present MULTISPEECHQA, a large-scale, synthetically generated and human-verified dataset comprising 9200 hours of 10.8 million spoken question-answer pairs in 23 typologically diverse languages. Using MULTISPEECHQA, we also introduce MULTISPEECH-BENCH, a multi-task benchmark for evaluating SLM performance on 23 languages. We compare the performance of a cascading system to open-weight and closed SLMs on MULTISPEECH-BENCH and find that the cascading system outperforms open-weight SLMs but not all closed SLMs. We use MULTISPEECHQA to finetune Qwen 2.5-Omni, which improves its performance on our benchmark. Our findings show that high-quality synthetic datasets offer a cheap solution to improving the multilingual capabilities of SLMs.
Tolúlopé Ògúnrèmí, Dan Jurafsky, Chris Manning +2
Department of Computer Science, Stanford University, United States · Cohere Labs, Canada · Cohere, Canada
Recent advances in speech technology predominantly favor high-resource languages, leaving speakers of Sub-Saharan African languages underserved. To bridge this divide, we introduce WAXAL, an open, large-scale speech dataset spanning 27 African languages representing over 100 million speakers. WAXAL comprises over 14,100 hours of speech collected in partnership with African academic and community organizations to build local research capacity: an Automatic Speech Recognition (ASR) corpus with 2,340 hours of transcribed, image-prompted natural speech across 19 languages (plus ~11,760 untranscribed hours) and a studio-quality Text-to-Speech (TTS) corpus of ~247 hours across 15 language configurations. We empirically benchmark three architecturally distinct ASR models -- Gemma 3n 2B, Whisper-Large-v3, and MMS-1B-all -- across the 19 ASR languages on strictly speaker-disjoint splits. Fine-tuning on WAXAL yields substantial gains, reducing macro-average WER by up to 58% for Whisper-Large-v3, 56% for Gemma 3n 2B, and 17% for MMS-1B-all (and Character Error Rate by up to 70%). WAXAL-fine-tuned models generalize zero-shot to three out-of-distribution benchmarks: FLEURS, Mozilla Common Voice, and Sunbird SALT. Finally, an LLM-as-judge semantic evaluation confirms that fine-tuning preserves speaker intent beyond surface-level transcription metrics. WAXAL is released at https://huggingface.co/datasets/google/WaxalNLP.
Abdoulaye Diack, Perry Nelson, MohamedElfatih MohamedKhair +44