Speech Language Models

Follow this topic Ask about this topic

Speech language models connect spoken audio to language-model capabilities, supporting tasks such as transcription, spoken question answering, translation, diarization, and speech generation. Common approaches pair pretrained audio encoders or speech tokenizers with language models, using projectors, alignment objectives, interleaved speech-text training, or distillation to bridge the modalities. Post-training and decoding methods extend these systems to streaming and full-duplex conversation, where turn-taking, latency, and expressive speech matter alongside answer quality. Models are becoming more capable across tasks, but robustness to noise, overlap, low-resource languages, and acoustic cues remains uneven, and end-to-end systems do not always outperform cascades of specialized components.

Latest papers

312 papers55 papers in the last four weeks, up 8% on the four weeks before. 0.3% of all new papers.

ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modeling
Feb 17, 2026Nicol Visser, Simon Malan, Danel Slabbert +1Audio Tokenization Speech Language Models
Electrical and Electronic Engineering, Stellenbosch University, South Africa
Text-only adaptation in LLM-based ASR through text denoising
Jan 28, 2026Andrés Carofilis, Sergio Burdisso, Esaú Villatoro-Tello +8Domain Adaptation for ASR Domain Adaptation
Idiap Research Institute · EPFL, Lausanne · Uniphore +2
VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models
Jan 27, 2026Yuxiang Wang, Hongyu Liu, Dekun Chen +2Privacy-Preserving Language Models Language Model Safety Evaluation
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · Amphion Technology Co., Ltd. +1
LOGIC: Efficient and Robust Contextual Biasing for Speech LLMs via Logit-Space Integration
Jan 21, 2026Peidong Wang, Jian Xue, Jinyu LiLanguage Model Decoding Speech Processing
Microsoft, USA
PRiSM: Benchmarking Phone Realization in Speech Models
Jan 20, 2026Shikhar Bharadwaj, Chin-Jou Li, Yoonjae Kim +13Automatic Speech Recognition Speech Language Models
1CMU · 2GIST · 3UT Austin +4
Zero-Shot Lombard Speech Synthesis with Controllable Style Embeddings
Jan 19, 2026Seymanur Akti, Alexander WaibelTTS Synthesis Zero-Shot TTS
KIT Campus Transfer GmbH (KCT) · Karlsruhe Institute of Technology (KIT) · Carnegie Mellon University (CMU)
TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding
Jan 11, 2026Mingyue Huo, Yiwen Shao, Yuheng ZhangMulti-Speaker ASR Audio Understanding
University of Illinois Urbana-Champaign · Johns Hopkins University
FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation
Jan 8, 2026Junseok Lee, Chang-Jae ChunMultimodal Large Language Models Speech Processing
OKESTRO · Sejong University
AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning
Dec 31, 2025Yiwen Shao, Wei Liu, Jiahong Li +4Speech Language Models Spoken Language Understanding
Tencent AI Lab
VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents
Oct 13, 2025Jiliang Hu, Wenfu Wang, Zuchao Li +6Audio-Language Model Evaluation Voice Agent Evaluation
School of Artificial Intelligence, Wuhan University, Wuhan, China · Tencent AI Lab, Beijing, China
Enhancing Speech Large Language Models through Reinforced Behavior Alignment
Aug 25, 2025Yansong Liu, Yuan LiuSpeech Language Models Language Model Robustness
Xi’an Jiaotong-Liverpool University · The Chinese University of Hong Kong
Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture
Apr 16, 2025Biao Fu, Donglei Yu, Minpeng Liao +5Speech Translation Simultaneous Speech Translation
School of Informatics, Xiamen University · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism · Alibaba Group Tongyi Lab

Feb 17, 2026J/K move · Enter open · S save

ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modeling

Nicol Visser, Simon Malan, Danel Slabbert, Herman Kamper

Electrical and Electronic Engineering, Stellenbosch University, South Africa

Audio Tokenization Speech Language Models Discrete Speech Representation Learning Subword Tokenization

Open paper AskPDF arXiv

Pure speech language models aim to learn language directly from raw audio without textual resources. A key challenge is that discrete tokens from self-supervised speech encoders result in excessively long sequences, motivating recent work on syllable-like units. However, methods like Sylber and SyllableLM rely on intricate multi-stage training pipelines. We propose ZeroSyl, a simple training-free method to extract syllable boundaries and embeddings directly from a frozen WavLM model. Using L2 norms of features in WavLM's intermediate layers, ZeroSyl achieves competitive syllable segmentation performance. The resulting segments are mean-pooled, discretized using K-means, and used to train a language model. ZeroSyl outperforms prior syllabic tokenizers across lexical, syntactic, and narrative benchmarks. Scaling experiments show that while finer-grained units are beneficial for lexical tasks, our discovered syllabic units exhibit better scaling behavior for syntactic modeling.