Walrus Technical Report
Organizations: Speech & Language AI Lab, Sony Group Corporation, Tokyo, Japan
Abstract
We present Walrus, a unified 8B speech-and-audio foundation model handling speech recognition (including code-switching and multi-talker speech), speech-to-text and speech-to-speech translation, text-to-speech, emotion recognition, audio captioning, and general audio understanding. Our main contribution is not a new architecture but a data-centric training recipe; we use a standard decoder-only backbone initialized from Gemma 4, so that gains can be traced to the data. For speech generation, Walrus predicts low-bitrate discrete speech tokens, and a lightweight token-to-waveform decoder converts them into high-fidelity audio. The recipe is a three-stage data curriculum that moves from broad multilingual data to multi-talker, code-switching, and domain-specific data. At each stage, task-aware and dataset-aware weighting balances tasks with very different token budgets and protects core skills from forgetting. Trained on over 600M samples covering more than 20 tasks and 100 languages, Walrus leads on most multilingual, multi-talker, and code-switching ASR, speech translation, and audio understanding benchmarks, outperforming both open speech language models of comparable size and dedicated task-specific models. On most benchmarks, it even matches or surpasses the larger Qwen3-Omni's performance. Results suggest that data engineering is a primary driver of quality in multi-task speech and audio modeling.
Figures & tables
| Task family | Supported tasks |
|---|---|
| Recognition | ASR, Context-ASR, Keyword-ASR, Timestamp-ASR, |
| CS-ASR, MT-ASR, Target-Speaker ASR, Diarization | |
| Translation | MT, AST (direct, cascaded), S2ST (direct, cascaded) |
| Generation | TTS |
| Understanding | Audio Captioning, Closed Captioning, Audio QA, Timestamp-QA, |
| Audio Reasoning, Instruction Following, SER |
| Stage | Steps | Learning rate | Warmup | Decay |
|---|---|---|---|---|
| 1: Foundation | 87,500 | 1% | 5% | |
| 2: Expansion | 135,000 | 1% | 5% | |
| 3: Specialization | 150,000 | 1% | 5% |
| Task | Stage 1 | Stage 2 | Stage 3 |
|---|---|---|---|
| Recognition | |||
| ASR | 5.0 | 5.0 | 4.0 |
| Context / Keyword ASR | 3.0 | 3.0 | 3.0 |
| Timestamp ASR | – | 1.0 | 1.0 |
| CS-ASR | – | 1.0 | 1.0 |
| MT-ASR | – | 1.0 | 1.0 |
| Task | Dataset | Metric | Prior SOTA | Walrus-CR | Walrus-R |
|---|---|---|---|---|---|
| Recognition | |||||
| ASR | LS Clean/Other Panayotov et al. (2015) | WER | 1.57 / 3.13 (AF3) | 1.88/4.99 | 1.65/3.85 |
| ASR | MLS Pratap et al. (2020) | WER | 8.20 (Phi4-MM) | 9.14 | 6.89 |
| ASR | FLEURS Conneau et al. (2023) | WER | 4.00 (Phi4-MM) | 7.54 | 5.33 |
| ASR | VoxPopuli Wang et al. (2021) | WER | 11.78 (Gemma4) | 8.38 | 7.84 |
| ASR | Internal Entertainment (avg) | WER | 24.35 (Gemma4) | 26.71 | 17.45 |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Slot | Filled with |
|---|---|
| {lang1} , {lang2} | Source and target language |
| {context} | Preceding text, as discourse context |
| {keywords} | List of candidate keywords |
| {labels} | Closed set of allowed labels, for example emotions |
| {speaker} | Target speaker for target-speaker queries |
| {start} , {end} | Time span for time-conditioned queries |
| Pipeline | What we build | Tools | Quality control |
| Transcript cleaning | |||
| Casing and punctuation | Restored transcripts of real ASR corpora | Qwen3.5-9B with few-shot prompts | CER 0.05 against the original; length check |
| Text normalization | Consistent references across languages | Rules for European, CJK, and Hindi/Arabic scripts | Manual inspection of samples |
| Recognition | |||
| Context ASR | Target utterance with 1-5 preceding utterances | MLS chapters, internal domains | – |
| Keyword ASR | Keyword lists of 0-100 items with distractors | Named-entity recognition | Entities in at least two utterances |
| Model | Clean | Other |
|---|---|---|
| WLv3 | 2.31 | 4.29 |
| Gemma4 | 3.22 | 7.74 |
| Phi4-MM | 1.68 | 3.83 |
| AF3 | 1.57 | 3.13 |
| Q25O | 1.83 | 3.44 |
| Q3O | 1.22 | 2.48 |
| Model | DE | EN | ES | FR | IT | NL | PL | PT | AVG |
|---|---|---|---|---|---|---|---|---|---|
| WLv3 | 5.54 | 5.30 | 4.01 | 4.60 | 9.09 | 10.36 | 4.89 | 7.20 | 6.37 |
| Gemma4 | 8.71 | 9.15 | 5.43 | 7.86 | 14.63 | 12.06 | 10.44 | 9.10 | 9.67 |
| Phi4-MM | 7.59 | 8.04 | 10.49 | 5.15 | 11.63 | N/A | N/A | 6.29 | 8.20 ∗ |
| AF3 | 27.45 | 5.02 | 12.57 | 20.07 | 19.96 | 28.46 | 53.50 | 17.26 | 23.04 |
| Q25O | 6.54 | 6.14 | 8.21 | 7.92 | 20.27 | 17.56 | 19.21 | 11.96 | 12.23 |
| Q3O | 3.77 | 3.77 | 3.41 | 3.00 | 9.72 | 8.94 | 12.06 | 5.22 | 6.24 |
| Model | AR | DE | EN | ES | FR | HI | IT | JA | KO | PT | ZH | AVG |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| WLv3 | 16.40 | 5.96 | 6.24 | 4.26 | 6.96 | 18.61 | 4.07 | 6.55 | 4.34 | 5.55 | 9.03 | 7.99 |
| Gemma4 | 16.20 | 6.10 | 6.50 | 3.50 | 8.00 | 8.60 | 3.20 | 7.80 | 5.30 | 4.60 | 13.60 | 7.58 |
| Phi4-MM | N/A | 3.96 | 3.38 | 3.02 | 4.35 | N/A | 1.98 | 4.50 | N/A | 3.98 | 6.83 | 4.00 ∗ |
| AF3 | N/A | 22.53 | 5.06 | 25.83 | 39.82 | 74.45 | 19.74 | N/A | 54.65 | 49.65 | N/A | 36.47 |
| Q25O | 18.75 | 7.67 | 6.46 | 5.70 | 5.08 | 14.54 | 7.63 | 8.12 | 7.03 | 6.50 | 7.29 | 8.62 |
| Q3O | 14.84 | 5.02 | 7.36 | 4.27 | 5.10 | 7.59 | 3.94 | 4.85 | 3.31 | 5.25 | 6.78 | 6.21 |
| Model | DE | EN | ES | FR | IT | AVG |
|---|---|---|---|---|---|---|
| WLv3 | 18.26 | 9.89 | 10.19 | 9.98 | 27.21 | 15.11 |
| Gemma4 | 13.76 | 8.67 | 8.79 | 10.89 | 16.80 | 11.78 |
| Phi4-MM | 12.13 | 5.91 | 10.18 | 13.18 | 22.67 | 12.81 |
| AF3 | 108.73 | 5.55 | 95.36 | 91.02 | 142.65 | 86.66 |
| Q25O | 15.84 | 6.64 | 13.24 | 11.39 | 23.67 | 14.16 |
| Q3O | 11.60 | 7.66 | 8.35 | 8.06 | 15.73 | 10.28 |
| Model | DE | ES | FR | IT | JA | PT | RU | ZH | AVG |
|---|---|---|---|---|---|---|---|---|---|
| WLv3 | 6.78 | 4.19 | 12.28 | 5.10 | 10.46 | 4.52 | 4.21 | 16.37 | 7.99 |
| Gemma4 | 9.33 | 5.55 | 14.58 | 7.10 | 16.46 | 8.16 | 9.16 | 47.37 | 14.71 |
| Phi4-MM | 10.82 | 3.57 | 6.05 | 3.53 | 36.91 | 10.26 | N/A | 48.29 | 17.49 ∗ |
| AF3 | 28.22 | 14.52 | 22.27 | 18.23 | 139.96 | 18.72 | 95.60 | 143.12 | 60.08 |
| Q25O | 6.67 | 7.51 | 8.44 | 8.62 | 19.85 | 12.88 | 7.95 | 10.69 | 10.33 |
| Q3O | 4.71 | 4.26 | 6.91 | 4.37 | 8.87 | 6.83 | 3.51 | 4.70 | 5.52 |
| Model | DE | EN-UK | EN-US | ES | FR | IT | JA |
|---|---|---|---|---|---|---|---|
| WLv3 | 22.33 | 8.31 | 11.10 | 11.67 | 22.53 | 13.56 | 24.77 |
| Gemma4 | 39.20 | 16.86 | 17.99 | 17.72 | 34.96 | 18.35 | 25.37 |
| Phi4-MM | 69.58 | 17.44 | 20.25 | 26.91 | 66.01 | 22.40 | 202.35 |
| AF3 | 63.90 | 16.05 | 17.74 | 47.20 | 88.49 | 41.51 | 169.11 |
| Q25O | 34.54 | 18.55 | 22.72 | 30.55 | 37.68 | 28.55 | 43.18 |
| Q3O | 27.11 | 11.85 | 11.14 | 15.50 | 21.95 | 13.00 | 25.88 |
| Model | EN-HI | EN-JA | EN-DE | EN-FR | EN-ZH | AVG |
|---|---|---|---|---|---|---|
| Read Test Set | ||||||
| WLv3 | 42.75 | 35.41 | 6.10 | 12.21 | 17.58 | 22.81 |
| Gemma4 | 47.50 | 48.07 | 19.18 | 18.17 | 28.56 | 32.30 |
| Phi4-MM | N/A | 69.29 | 29.15 | 32.78 | 81.71 | 53.23 ∗ |
| AF3 | 97.95 | 85.09 | 64.50 | 73.62 | 86.69 | 81.57 |
| Q25O | 56.96 | 63.68 | 41.66 | 37.21 | 58.42 | 51.58 |
| LibriMix (Clean 2Mix) | Japanese Daily Talk | |||
| Model | cpWER | tcpWER | cpCER | tcpCER |
| VibeVoice-ASR [ 46 ] | 59.34 | 59.36 | 71.99 | 72.17 |
| Gemma4 | N/A | N/A | N/A | N/A |
| Phi4-MM | N/A | N/A | N/A | N/A |
| AF3 | N/A | N/A | N/A | N/A |
| Q25O | N/A | N/A | N/A | N/A |
| Model | DE | ES | FR | IT | JA | PT | RU | ZH | AVG |
|---|---|---|---|---|---|---|---|---|---|
| WLv3 | 36.14 | 37.06 | 33.87 | 35.96 | 23.53 | 53.05 | 41.61 | 11.50 | 34.09 |
| Gemma4 | 42.01 | 37.15 | 41.04 | 43.02 | 25.50 | 51.12 | 46.44 | 17.90 | 38.02 |
| Phi4-MM | 39.81 | 43.60 | 42.24 | 41.42 | 30.54 | 55.28 | N/A | 22.39 | 39.33 ∗ |
| AF3 | 32.80 | 24.19 | 23.34 | 26.68 | 13.08 | 36.46 | 33.49 | 13.24 | 25.41 |
| Q25O | 42.42 | 40.51 | 37.46 | 34.95 | 14.57 | 56.65 | 50.44 | 22.58 | 37.44 |
| Q3O | 46.06 | 43.01 | 41.95 | 42.61 | 30.54 | 59.65 | 54.02 | 28.09 | 43.24 |
| Model | AR | DE | ES | FR | HI | IT | JA | KO | PT | ZH | AVG |
|---|---|---|---|---|---|---|---|---|---|---|---|
| X En | |||||||||||
| WLv3 | 20.63 | 33.27 | 22.27 | 31.67 | 19.40 | 23.57 | 16.51 | 19.05 | 36.68 | 15.80 | 23.89 |
| Gemma4 | 29.56 | 33.55 | 23.11 | 31.22 | 27.46 | 23.74 | 22.47 | 23.44 | 36.79 | 21.63 | 27.29 |
| Phi4-MM | N/A | 37.71 | 25.33 | 35.10 | N/A | 26.06 | 21.62 | N/A | 40.80 | 22.37 | 29.86 ∗ |
| AF3 | 14.05 | 33.27 | 23.43 | 28.83 | 8.62 | 23.08 | 11.90 | 14.32 | 35.39 | 16.43 | 20.93 |
| Q25O | 18.79 | 35.30 | 24.81 | 31.68 | 16.09 | 23.94 | 18.36 | 19.01 | 41.48 | 19.43 | 24.89 |
| Model | DE | ES | FR | IT | JA | PT | RU | ZH | AVG |
|---|---|---|---|---|---|---|---|---|---|
| Gemma4 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
| Phi4-MM | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
| AF3 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
| Q25O | 40.39 | 38.32 | 35.54 | 33.67 | 15.61 | 55.10 | 48.15 | 21.39 | 36.02 |
| Q3O | 44.10 | 41.32 | 38.30 | 40.21 | 30.04 | 58.12 | 52.19 | 25.90 | 41.27 |
| Walrus-CR | 49.21 | 47.12 | 40.73 | 41.89 | 32.14 | 57.33 | 52.58 | 25.16 | 43.27 |
| Model | AR | DE | ES | FR | HI | IT | JA | KO | PT | ZH | AVG |
|---|---|---|---|---|---|---|---|---|---|---|---|
| X En | |||||||||||
| Gemma4 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
| Phi4-MM | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
| AF3 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A |
| Q25O | 20.63 | 36.40 | 23.71 | 31.54 | 15.78 | 25.04 | 17.77 | 20.39 | 38.91 | 21.33 | 25.15 |
| Q3O | 32.80 | 43.10 | 29.30 | 41.00 | 31.52 | 30.34 | 25.61 | 29.30 | 45.41 | 28.30 | 33.60 |
| X En | En X | |||||||||
| Model | ASR-BLEU | UTMOS | SSim | ESim | APCP | ASR-BLEU | UTMOS | SSim | ESim | APCP |
| Kraken [ 16 ] | 32.30 | 3.13 | 0.75 | 0.77 | 2.48 | N/A | N/A | N/A | N/A | N/A |
| Seamless [ 3 ] | 32.10 | 3.06 | 0.66 | 0.76 | 2.05 | 27.33 | 2.82 | 0.58 | 0.56 | 2.25 |
| Q2.5O | 25.10 | 3.48 | 0.49 | 0.53 | 2.40 | N/A | N/A | N/A | N/A | N/A |
| Q3O | 33.60 | 3.59 | 0.61 | 0.75 | 2.19 | 25.67 | 3.27 | 0.49 | 0.44 | 2.12 |
| Walrus-CR | 28.59 | 3.17 | 0.75 | 0.76 | 2.54 | 17.19 | 3.01 | 0.67 | 0.69 | 2.41 |
| Task | Primary prompt |
| Recognition | |
| ASR | Transcribe the {lang1} speech: |
| Context-ASR | Transcribe the {lang1} speech from the context {context}: |
| Keyword-biased ASR | Transcribe the {lang1} speech, biasing toward these keywords: {keywords}: |
| Code-switched ASR | Transcribe the code-switched speech in {lang1} and {lang2}: |
| Dialogue-contextual ASR | Given the previous conversation between {agent} and {user}: {context} Transcribe the {lang1} speech: |
Speech Foundation Models
Speech foundation models are large pretrained systems that learn reusable audio representations for speech understanding, recognition, and generation.
See all 71 papers