Audio Tokenization

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 75% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 33

All topics

Q-SPT: Learnable Query-Based Compression for Low-Frame-Rate Speech Tokenization
Oct 1, 2026Jeeyoung Yun, Seohwan Yun, Sungwoong KimAudio Token Compression Neural Audio Codecs
Korea University, Seoul, Republic of Korea
Multi-Rate Bandwidth Extension by Token Completion in Neural Audio Codecs
Sep 29, 2026Benoît Ginies, Olivier Fercoq, Gaël RichardAudio Representation Learning Neural Audio Codecs
LTCI, Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France
Interpreting and Evaluating Dynamic-Rate Speech Codec Boundaries
Sep 29, 2026Han Wang, Jiaqi Li, Yingda Shen +2Neural Audio Codecs Audio Tokenization
The Chinese University of Hong Kong, Shenzhen · Zhejiang University · Amphion Technology Co., Ltd.
UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation
Sep 23, 2026Runwu Shi, Kai Li, Yujin Wang +8Text-to-Audio Generation Audio Representation Learning
Institute of Science Tokyo · Tsinghua University · Wuhan University +2
Discrete vs. Continuous: A Comprehensive Study of Unified Audio Understanding in LALMs
Sep 19, 2026Jing Peng, Zichao Nie, Zhisheng Zhang +2Audio-Language Model Evaluation Audio Representation Learning
Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
TokenMapper: A Step Toward Interoperable Speech Token Translation
Sep 14, 2026Tal Kozakov, Tal Rosenwein, Eliya NachmaniNeural Audio Codecs Audio Tokenization
Department of Electrical and Computer Engineering, Ben-Gurion University of the Negev
StreamAlign: Streaming Text-Aligned Speech Tokenization
Sep 9, 2026Kang-wook Kim, Jinyoung Park, Jinsoo Kim +3Audio Tokenization Speech Language Models
Seoul National University · University of California, Berkeley · KRAFTON +1
BiMTokenizer: Preserving Semantic-Acoustic Balance in Low-Bitrate Speech Tokenization via Bidirectional State-Space Modeling
Sep 1, 2026Xin Zhang, Lin Li, Chuanbo Liu +2Neural Audio Codecs Vector Quantization
Wuhan University of Technology, Wuhan, China · NEC Laboratories Asia Pacific, Singapore · NEC Corporation, Japan +1
Language-Statistical Analysis of Neural Audio Codec Tokens Across Architectures, Corpora, and Noise Conditions
Aug 31, 2026Joonyong Park, Shinnosuke Takamichi, David M. Chan +3Neural Audio Codecs Residual VQ
Grad. School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Faculty of Science and Technology, Keio University, Tokyo, Japan · Berkeley Artificial Intelligence Research Lab (BAIR), University of California, Berkeley, CA, USA
Phoenix TTS: High-Fidelity Synthesis and Voice Conversion via Flow-Matching-Driven Speech Tokenization
Aug 12, 2026Peijie Chen, Zhuanling Zha, Zhipeng Nie +8Speech Processing Zero-Shot TTS
Didichuxing Co. Ltd
Pitch Contour Tokenization using VQ-VAE and Its Application on Korean Traditional Music Analysis
Aug 11, 2026Seonguk Ju, Seola Cho, Sooin Chung +2Audio Representation Learning Vector-Quantized Variational Autoencoder
Music & Arts Learning (MALer) Lab, Sogang University, South Korea · Graduate School of Culture Technology, KAIST, South Korea
ReLMCodec: Designing Predictable Speech Tokens from Pre-Quantization Phoneme Structure
Aug 8, 2026Zixiang Wan, Xusheng Yang, Zheng Wang +1Neural Audio Codecs Audio Tokenization
1Peking University, Shenzhen, China · 2Tencent, Shenzhen, China
KVAE: Family of Tokenizers for Multimodal Generative Models
Aug 6, 2026Andrey Shutkin, Denis Parkhomenko, Ivan Kirillov +11Variational Autoencoders Audio Token Compression
Kandinsky Lab
Content is What Remains: Invariant Speech Tokenization from Parallel Utterances
Jul 21, 2026Laurin Wagner, Bernhard Thallinger, Miroslav Stankovic +1Audio Tokenization Self-Supervised Speech Representation Learning
nyra labs · nyra labs, Austria
Do Speech Tokens Leak Voiceprints? Speaker Inversion Attacks Against End-to-End Speech Language Models
Jul 18, 2026Ye Lu, Yihan Yan, Zhaoyang Zhang +4Adversarial Attacks Privacy Leakage in Language Models
School of Cyberspace Science, Harbin Institute of Technology, Harbin 150001, China
Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization
Jul 5, 2026Ryota Komatsu, Kota Kawakita, Takuma Okamoto +1Audio Tokenization Self-Supervised Speech Representation Learning
Institute of Science Tokyo, Meguro, Tokyo 152-8550, Japan · National Institute of Information and Communications Technology, Kyoto 619-0289, Japan
Neural Audio Codec with Adjustable Token Temporal Resolution Using Sampling-Frequency-Independent Convolutional Layers
Jul 2, 2026Tomohiko Nakamura, Wataru Nakata, Kanami Imamura +1Neural Audio Codecs Audio Tokenization
The National Institute of Advanced Industrial Science and Technology (AIST), 2-4-7 Aomi, Koto-ku, Tokyo, 135-0064 Japan · The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8654 Japan
wav2tok 2.0: Scalable Audio Tokenization Maintaining Explicit Pairwise Token Alignment for Efficient Audio Retrieval
Jun 25, 2026Adhiraj Banerjee, Vipul AroraAudio Representation Learning Speech Processing
Department of Electrical Engineering, Indian Institute of Technology Kanpur, India · Department of Electrical Engineering, KU Leuven, Belgium
End-to-End Training for Discrete Token LLM based TTS System
Jun 8, 2026Changfeng Gao, Yong Ren, Jun Yuan +3TTS Synthesis Speech Processing
Tencent Yuanbao, China
TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech
Jun 8, 2026Yejin Lee, Junwon Moon, Hyoeun Kim +3Efficient Transformer Inference Audio Token Compression
Sungkyunkwan University · University of Seoul
Probing Token Spaces under Generator Shift in AI-Generated Music Detection
Jun 7, 2026Joonyong Park, Jungwoo Kim, Junyoung Koh +1Audio Representation Learning Audio Deepfake Detection
1The University of Tokyo, Tokyo, Japan · 3Yonsei University, Seoul, Republic of Korea. · 2MAAP Lab
Leveraging Soft Distributions of SSL-Derived Discrete Speech Tokens for Downstream Inference
Jun 5, 2026Kentaro Onda, Satoru Fukayama, Daisuke Saito +1Automatic Speech Recognition Audio Tokenization
The University of Tokyo, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Japan
F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
Jun 4, 2026Dinghao Zhou, Xingchen Song, Di Wu +3Audio Representation Learning Autoencoders
2WeNet Open Source Community · ∗Equal contribution · 1Nanjing University, China
CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding
Jun 3, 2026Eugene Kwek, Feng Liu, Rui Zhang +1Audio Token Compression Audio Representation Learning
Pennsylvania State University · Drexel University
EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
Jun 1, 2026Hui Li, Yangfan Gao, Junlin Shang +4Audio Representation Learning Neural Audio Codecs
Fudan University
UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception
May 29, 2026Yuhan Song, Linhao Zhang, Aiwei Liu +6Audio Representation Learning Audio Understanding
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Basic Model Technology Center, WeChat AI, Tencent Inc.
HoliTok:A Coutinuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding
May 28, 2026Bohan Li, Shi Lian, Hankun Wang +6Audio Representation Learning Speech Foundation Models
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China · hi lab, Xiaohongshu Inc, China
LoSATok: Low-dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation
May 27, 2026Zhisheng Zhang, Xiang Li, Yixuan Zhou +3Audio Token Compression Audio Representation Learning
1Shenzhen International Graduate School, Tsinghua University, China · *Work conducted while interning at ModelBest. · 2ModelBest Inc., China
Modeling Music as a Time-Frequency Image: A 2D Tokenizer for Music Generation
May 15, 2026Yuqing Cheng, Xingyu Ma, Guochen Yu +1Neural Audio Codecs Autoregressive Language Modeling
Department of Music AI and Information Technology, Central Conservatory of Music, Beijing, China · Zhipu AI, Beijing, China
Exploring Token-Space Manipulation in Latent Audio Tokenizers
May 11, 2026Francesco Paissan, Luca Della Libera, Mirco Ravanelli +1Audio Representation Learning Neural Audio Codecs
1Mila – Québec AI Institute · 2Université Laval · 3Concordia University
PairAlign: A Framework for Sequence Tokenization via Self-Alignment with Applications to Audio Tokenization
May 7, 2026Adhiraj Banerjee, Vipul AroraAudio Tokenization Self-Supervised Speech Representation Learning
Department of Electrical Engineering Indian Institute of Technology, Kanpur
LLM-Codec: Neural Audio Codec Meets Language Model Objectives
Apr 20, 2026Ho-Lam Chung, Yiming Chen, Hung-yi LeeNeural Audio Codecs Multi-Token Prediction
Graduate Institute of Communication Engineering, National Taiwan University · ASUS Intelligent Cloud Services · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
Speech Codec Probing from Semantic and Phonetic Perspectives
Mar 11, 2026Xuan Shi, Chang Zeng, Tiantian Feng +3Audio Representation Learning Neural Audio Codecs
University of Southern California, USA · Dolby Laboratories, USA

Oct 1, 2026, cs.SDJ/K move · Enter open · S save

Q-SPT: Learnable Query-Based Compression for Low-Frame-Rate Speech Tokenization

Jeeyoung Yun, Seohwan Yun, Sungwoong Kim

Korea University, Seoul, Republic of Korea

Audio Token Compression Neural Audio Codecs Audio Tokenization Speech Language Models

Open paper AskPDF arXiv

Neural speech codecs increasingly serve as tokenizers for speech language models (SLMs). Lowering the frame rate reduces the computational and memory costs of SLMs, but makes it difficult to preserve both linguistic information and acoustic detail. Existing approaches rely on rule-based compression: average pooling can discard linguistic information, whereas similarity-based merging uses a fixed threshold on adjacent-frame similarity and applies the resulting boundaries to the acoustic stream. We propose Q-SPT, a low-frame-rate dual-stream speech tokenizer with separate, context-aware, learnable query-based compressors specialized for semantic and acoustic representations. In particular, queries at a fixed rate independently attend to the semantic and acoustic streams as separate key-value sources, enabling stream-specific, context-aware aggregation through two separately learned compressors. In addition, an autoregressive text loss explicitly supervises the semantic compressor to preserve linguistic information. Experimental results show that Q-SPT achieves the best reconstruction among the evaluated codecs at the same frame rate. In downstream SLMs, it yields the best speech recognition accuracy and text-to-speech perceptual quality with competitive intelligibility.

Momentum

Related topics

Latest papers 33