eess.ASNov 10, 2026

Walrus Technical Report

Authors: Alkis Koudounas, Osamu Take, Hayato Futami, Quentin Jodelet, Emiru Tsunoo

Organizations: Speech & Language AI Lab, Sony Group Corporation, Tokyo, Japan

Abstract

We present Walrus, a unified 8B speech-and-audio foundation model handling speech recognition (including code-switching and multi-talker speech), speech-to-text and speech-to-speech translation, text-to-speech, emotion recognition, audio captioning, and general audio understanding. Our main contribution is not a new architecture but a data-centric training recipe; we use a standard decoder-only backbone initialized from Gemma 4, so that gains can be traced to the data. For speech generation, Walrus predicts low-bitrate discrete speech tokens, and a lightweight token-to-waveform decoder converts them into high-fidelity audio. The recipe is a three-stage data curriculum that moves from broad multilingual data to multi-talker, code-switching, and domain-specific data. At each stage, task-aware and dataset-aware weighting balances tasks with very different token budgets and protects core skills from forgetting. Trained on over 600M samples covering more than 20 tasks and 100 languages, Walrus leads on most multilingual, multi-talker, and code-switching ASR, speech translation, and audio understanding benchmarks, outperforming both open speech language models of comparable size and dedicated task-specific models. On most benchmarks, it even matches or surpasses the 4×4\times larger Qwen3-Omni's performance. Results suggest that data engineering is a primary driver of quality in multi-task speech and audio modeling.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. UniWhisper: Efficient Continual Multi-task Training for Robust Universal Audio Representation

    Feb 25, 2026Yuxuan Chen, Peize He, Haoyuan Yu +1Audio Representation LearningAudio Understanding

  2. Turning Speech Language Models into Multilingual Listeners

    Sep 27, 2026Tolúlopé Ògúnrèmí, Dan Jurafsky, Chris Manning +2Speech Language ModelsSpoken Language Understanding

  3. WAXAL: A Large-Scale Multilingual African Language Speech Corpus

    Feb 2, 2026Abdoulaye Diack, Perry Nelson, MohamedElfatih MohamedKhair +44African Language ProcessingMultilingual ASR