Audio Representation Learning

Follow this topic Ask about this topic

Momentum

20 papers in the last four weeks, up 233% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 141

All topics

Frequency-Aware Self-Supervised Music Representation Learning
Jun 24, 2026Yicheng Gu, Junan Zhang, Jerry Li +2Audio Representation Learning Frequency-Domain Feature Learning
Spellbrush, Akihabara, Tokyo 101-0021, Japan · Acoustic Lab, Department of Information and Communications Engineering (DICE), Aalto University, Espoo 02150, Finland · School of Data Science, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China
Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?
Jun 24, 2026Tomoya Mizumoto, Yusuke FujitaAudio Representation Learning Self-Supervised Speech Representation Learning
SB Intuitions
PHAST-Net: Attention-Guided, Physics-Informed Network for Unified Estimation of Ideal Time-Frequency Representations
Jun 22, 2026James M. Cozens, Simon J. GodsillAudio Representation Learning Wavelet Methods
Engineering Department, University of Cambridge, CB2 1PZ Cambridge, U.K.
STAR-VAE: Structured Topology-Aware Regularization for Audio Reconstruction and Generation
Jun 22, 2026Huadai Liu, Wen Wang, Kaicheng Luo +3Variational Autoencoders Text-to-Audio Generation
Hong Kong University of Science and Technology · Tongyi Fun Team, Alibaba Group.
Toward Open-Set Speaker Attribute Prediction with Keyword-Appended LLM Embeddings
Jun 20, 2026Byoungjun So, Jaejun Lee, Kyogu LeeAudio Representation Learning Speech Processing
Department of Intelligence and Information, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University · Artificial Intelligence Institute, Seoul National University, Republic of Korea
AugCodec: A Low-Bitrate Disentangled Neural Speech Codec via Data Augmentation
Jun 20, 2026Dongmei Wang, Xiaohang Sun, Yang Liu +10Disentangled Representation Learning Audio Representation Learning
Amazon, USA
ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion
Jun 20, 2026Jeongsoo Choi, Ji-Hoon Kim, Shujie Hu +1Audio Representation Learning Neural Audio Codecs
KAIST, South Korea · Chung-Ang University, South Korea · The Chinese University of Hong Kong, China
Gradient-Based Learning of Parametric Engine Sound Representations for Real-Time Resynthesis and Tuning on Embedded Systems
Jun 19, 2026Robin Doerfler, Matthieu Kuntz, Clemens ZimmerAudio Representation Learning Audio Generation
Impulse Audio Lab GmbH Munich, Germany
CoughPhase-CLR: Designing an acoustics-informed foundation model for coughing sound classification
Jun 19, 2026Marius Moldovan, Anton Batliner, Thomas M. Berghaus +2Contrastive Learning Audio Representation Learning
CHI – the Chair of Health Informatics at the TUM University Hospital, Munich, Germany · MCML – the Munich Center for Machine Learning and MDSI – the Munich Data Science Institute, Munich, Germany · University Hospital Augsburg at the University of Augsburg, Augsburg, Germany +2
LISE : Listenable Interpretable Speaker Embeddings
Jun 19, 2026Xiaoliang Wu, Chongxin Gan, Ke Liu +2Audio Representation Learning Speaker Verification
University of Southampton, United Kingdom · The Hong Kong Polytechnic University, Hong Kong SAR, China · University of Edinburgh, United Kingdom
Segment-Level Mandarin Chinese Speech-Based Cognitive Impairment Detection via an Autoencoder with Contrastive Learning
Jun 18, 2026Yongqi Shao, Hong Huo, Flavio Bertini +2Audio Representation Learning Representation Learning
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University; Key Laboratory of System Control and Information Processing, Ministry of Education of China; Shanghai Key Laboratory of Perception and Control in Industrial Network Systems, Shanghai, 200240, China · Department of Mathematical, Physical and Computer Sciences, University of Parma, Parco Area delle Sciences 53/A, Parma, 43124, Italy · Department of Computer Science and Engineering, University of Bologna, Mura Anteo Zamboni 7, Bologna, 40126, Italy
Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models
Jun 14, 2026Hyebin Cho, Jaehyuk Jang, Changick Kim +1Few-Shot Audio Classification Audio Representation Learning
Korea Advanced Institute of Science and Technology, South Korea
Beyond task performance: Decoding bioacoustic embeddings with speech features
Jun 12, 2026Ines Nolasco, Jules Cauzinille, Marius Miron +8Audio Representation Learning Representation Probing
Earth Species Project
Spectro-Temporal Interference Confounds Phase Encoding in Spatial Audio Foundation Models
Jun 12, 2026Yuxuan Chen, Haoyuan Yu, Peize HeAudio Representation Learning Direction-of-Arrival Estimation
The Chinese University of Hong Kong, Shenzhen · Jilin University · Hunan University +1
Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources
Jun 12, 2026Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida +6Audio-Language Model Evaluation Audio Representation Learning
1POSTECH · 2Sony AI · 4Sungkyunkwan University +2
From Tokens to Faces: Investigating Discrete Speech Representations for 3D Facial Animation
Jun 11, 2026Pedro Correa, Olivier Perrotin, Samir Sadok +2Audio Representation Learning Neural Audio Codecs
Univ. Estadual de Campinas (UNICAMP), Brazil · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, France · Inria at Univ. Grenoble Alpes, CNRS, LJK, France
Decoding Insect Song: A Multitask Semisupervised Orthoptera Bioacoustic Classifier
Jun 11, 2026Olga Isupova, Danil Kuzin, Ella Browning +2Audio Representation Learning Audio Understanding
Leverhulme Centre for Nature Recovery, School of Geography and the Environment, University of Oxford, Oxford, UK
From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation
Jun 11, 2026Fengrui Liu, Ruiyang Huang, Qijian Zheng +2Synthetic Data Pretraining Audio Representation Learning
East China Normal University · Southeast University · Fudan University +1
Dolph2Vec: Self-Supervised Representations of Dolphin Vocalizations
Jun 10, 2026Chiara Semenzin, Faadil Mustun, Roberto Dessi +5Audio Representation Learning Audio Understanding
École Normale Supérieure, Paris, France · Not Diamond, San Francisco, USA · Institut du Cerveau, Paris, France +1
Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation
Jun 10, 2026Zhen Ye, Xu Tan, Yiming Li +10Spoken Language Understanding Audio Representation Learning
Hong Kong University of Science and Technology, Hong Kong SAR · Tencent, China · Independent Researcher +4
SARA: A Dual-Stream VAE for High-Fidelity Speech Generation via Integrating Semantic and Acoustic Representations
Jun 10, 2026Peijie Chen, Wenhao Guan, Weijie Wu +7Variational Autoencoders Audio Representation Learning
School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China · DiDi Global Inc., Beijing, China
Speaker Group Encoding in Self-supervised Speech Recognition Models
Jun 9, 2026Felix Herron, Solange Rossato Alexandre Allauzen, Benoit Favre +1Audio Representation Learning Demographic Bias in ASR
MILES Team, LAMSADE, Université Paris Dauphine-PSL, France (1) · GETALP Team, LIG, Université Grenoble Alpes, France (2) · NLP team, LIS, Aix-Marseille University, France (3)
ViP-VL: Vietnamese Self-supervised Speech Pretraining Model with Vector-Quantization Learning
Jun 9, 2026Khanh Le, Kiet Anh Hoang, Bao Nguyen +5Audio Representation Learning Speech Processing
VinUniversity, Vietnam · UNEY, Switzerland
Dual-Branch Gated Fusion for Open-Set Audio Deepfake Source Tracing
Jun 8, 2026Awais Khan, Kutub Uddin, Khalid MalikAudio Representation Learning Audio Deepfake Detection
College of Innovation and Technology, University of Michigan, Flint, MI, USA · ProbeTruth Inc., MI, USA
Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
Jun 8, 2026Ming-Hao Hsu, Yuxuan Hu, Shujie Liu +3Audio Representation Learning Audio Understanding
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research, Redmond, WA, USA. · Microsoft Research Asia, Hong Kong, China.
Multi-View Speech Representation Learning for Parkinson's Disease Detection Using Context-guided Cross-modal Attention
Jun 8, 2026George Theodosiou, Loukas Ilias, Dimitris AskounisAudio Representation Learning Speech Processing
Decision Support Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece
Probing Token Spaces under Generator Shift in AI-Generated Music Detection
Jun 7, 2026Joonyong Park, Jungwoo Kim, Junyoung Koh +1Audio Representation Learning Audio Deepfake Detection
1The University of Tokyo, Tokyo, Japan · 3Yonsei University, Seoul, Republic of Korea. · 2MAAP Lab
HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec
Jun 4, 2026Arjun Gangwar, S UmeshDisentangled Representation Learning Audio Representation Learning
Indian Institute of Technology, Madras, India
USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding
Jun 4, 2026Heng-Jui Chang, Alexander H. Liu, Saurabhchand Bhati +4Audio Representation Learning Audio Understanding
MIT CSAIL, USA · Amazon, USA
F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
Jun 4, 2026Dinghao Zhou, Xingchen Song, Di Wu +3Audio Representation Learning Autoencoders
2WeNet Open Source Community · ∗Equal contribution · 1Nanjing University, China
Do speech foundation models perceive speaker similarity as humans do?
Jun 4, 2026Minoru Kishi, Hayato Yagi, Shinnosuke Takamichi +1Audio Representation Learning Speech Foundation Models
Keio University, Japan · The University of Tokyo, Japan
InfoShield: Privacy-Preserving Speech Representations for Mental Health Screening via Information-Theoretic Optimization
Jun 4, 2026Xueyang Wu, Siyuan Liu, Kezhuo Yang +1Audio Representation Learning Depression Detection
Shenzhen NeurStar Inc., China · University of York, United Kingdom · Shanghai Jiao Tong University, China
Probing Spatial Structure in Pretrained Audio Representations
Jun 4, 2026Chuyang Chen, Sivan Ding, Adrian S. Roman +1Audio Representation Learning Representation Probing
Music and Audio Research Laboratory, New York University, USA
CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding
Jun 3, 2026Eugene Kwek, Feng Liu, Rui Zhang +1Audio Token Compression Audio Representation Learning
Pennsylvania State University · Drexel University
EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
Jun 1, 2026Hui Li, Yangfan Gao, Junlin Shang +4Audio Representation Learning Neural Audio Codecs
Fudan University
Echo: A Joint-Embedding Predictive Architecture for Speaker Diarization and Speech Recognition in a Shared Latent Space
Jun 1, 2026Louis MouchonAudio Representation Learning Joint-Embedding Predictive Architecture
Independent Research
UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception
May 29, 2026Yuhan Song, Linhao Zhang, Aiwei Liu +6Audio Representation Learning Audio Understanding
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Basic Model Technology Center, WeChat AI, Tencent Inc.
Extracting accent features in spoken Brazilian Portuguese without sociolinguistic labels
May 28, 2026Pedro H. L. Leite, Pedro Benevenuto Valadares, Luiz W. P. BiscainhoAudio Representation Learning Speech Processing
PEE/COPPE, UFRJ, Rio de Janeiro-RJ · Faculdade de Engenharia Elétrica e Computação (FEEC), UNICAMP, Campinas-SP · DEL/Poli & PEE/COPPE, UFRJ, Rio de Janeiro-RJ
HoliTok:A Coutinuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding
May 28, 2026Bohan Li, Shi Lian, Hankun Wang +6Audio Representation Learning Speech Foundation Models
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, China · hi lab, Xiaohongshu Inc, China
MELD: Mel-Spectrogram-Based Speech Language Modeling with Discrete Latent Variables
May 28, 2026Sung-Lin Yeh, Wei Zhou, Gil Keren +6Audio Representation Learning Autoregressive Language Modeling
University of Edinburgh · *Work done at Meta · 2Google DeepMind +1
LoSATok: Low-dimensional Semantic-Acoustic Tokenizer for Cross-Domain Audio Understanding and Generation
May 27, 2026Zhisheng Zhang, Xiang Li, Yixuan Zhou +3Audio Token Compression Audio Representation Learning
1Shenzhen International Graduate School, Tsinghua University, China · *Work conducted while interning at ModelBest. · 2ModelBest Inc., China
Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text
May 27, 2026Jiahao Mei, Heinrich Dinkel, Yadong Niu +7Text-to-Audio Generation Audio Representation Learning
1X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · 2MiLM Plus, Xiaomi Inc., Beijing, China
MERIT: Learning Disentangled Music Representations for Audio Similarity
May 26, 2026Abhinaba Roy, Junyi Liang, Dorien HerremansDisentangled Representation Learning Audio Representation Learning
Singapore University of Technology and Design
StreamSplit: Continuous Audio Representation Learning via Uncertainty-Guided Adaptive Splitting
May 26, 2026Minh K. Quan, Pubudu N. PathiranaContrastive Learning Audio Representation Learning
School of Engineering, Deakin University Waurn Ponds, Australia
Rethinking Continual Learning for Speech and Audio: A Representation-Centric Taxonomy and Open Problems
May 24, 2026Yang Xiao, Siyi Wang, Eun-Jung Holden +1Audio Representation Learning Speech Foundation Models
University of Melbourne, Melbourne, Australia.
SAME: A Semantically-Aligned Music Autoencoder
May 18, 2026Julian D. Parker, Zach Evans, CJ Carr +4Audio Representation Learning Neural Audio Codecs
Stability AI
SpeakerLLM: A Speaker-Specialized Audio-LLM for Speaker Understanding and Verification Reasoning
May 14, 2026KiHyun Nam, Jungwoo Heo, Siu Bae +2Audio Representation Learning Speaker Verification
Korea Advanced Institute of Science and Technology (KAIST) · University of Seoul
AudioMosaic: Contrastive Masked Audio Representation Learning
May 14, 2026Hanxun Huang, Qizhou Wang, Xingjun Ma +3Audio Representation Learning Self-Supervised Pre-Training
School of Computing and Information Systems, The University of Melbourne, Australia · Institute of Trustworthy Embodied AI, Fudan University, China · Baskin School of Engineering, University of California, Santa Cruz, USA
Exploring Token-Space Manipulation in Latent Audio Tokenizers
May 11, 2026Francesco Paissan, Luca Della Libera, Mirco Ravanelli +1Audio Representation Learning Neural Audio Codecs
1Mila – Québec AI Institute · 2Université Laval · 3Concordia University
AffectCodec: Emotion-Preserving Neural Speech Codec for Expressive Speech Modeling
May 11, 2026Jiacheng Shi, Hongfei Du, Xinyuan Song +3Audio Representation Learning Neural Audio Codecs
College of William & Mary · Emory University · George Mason University
Multi-layer attentive probing improves transfer of audio representations for bioacoustics
May 11, 2026Marius Miron, David Robinson, Masato Hagiwara +15Audio Representation Learning Bioacoustics
Earth Species Project
PoDAR: Power-Disentangled Audio Representation for Generative Modeling
May 11, 2026Alejandro Luebs, Mithilesh Vaidya, Ishaan Kumar +5Audio Representation Learning Latent Diffusion Models
Descript
Voice Biomarkers for Depression and Anxiety
May 11, 2026Oleksii Abramenko, Noah D. Stein, Colin VazAudio Representation Learning Depression Detection
machine learning team at Kintsugi Mindful Wellness, Inc.
BeeVe: Unsupervised Acoustic State Discovery in Honey Bee Buzzing
May 8, 2026Hamze Hammami, Nidhal AbdulazizAudio Representation Learning Unsupervised Clustering
TARNet: A Temporal-Aware Multi-Scale Architecture for Closed-Set Speaker Identification
May 8, 2026Yassin Terraf, Youssef IraqiAudio Representation Learning Temporal Convolutional Networks
College of Computing, University Mohammed VI Polytechnic, Benguerir, Morocco · CID Development, Rabat, Morocco
WavCube: Unifying Speech Representation for Understanding and Generation via Semantic-Acoustic Joint Modeling
May 7, 2026Guanrou Yang, Tian Tan, Qian Chen +12Audio Representation Learning TTS Synthesis
Shanghai Jiao Tong University · Independent Researcher · Shanghai Innovation Institute +4
Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB)
May 6, 2026Cyril Allauzen, Tom Bagby, Georg Heigold +2Audio-Language Model Evaluation LLM Evaluation
Google USA & Germany
PHALAR: Phasors for Learned Musical Audio Representations
May 5, 2026Davide Marincione, Michele Mancusi, Giorgio Strano +4Audio Representation Learning Representation Learning
Department of Computer Science, Sapienza University of Rome, Italy · Moises Systems, Inc. · Paradigma, Inc.
Shao: Scaling Acoustic Token Language Models Toward High-Fidelity Music Generation
May 3, 2026Jiafeng Liu, Yuanliang Dong, Hongjia Liu +8Text-to-Music Generation Audio Representation Learning
1Central Conservatory of Music · 2Tsinghua University
LASE: Language-Adversarial Speaker Encoding for Indic Cross-Script Identity Preservation
May 1, 2026Venkata Pushpak Teja MentaAdversarial Training Adversarial Representation Learning
Praxel Ventures

Jun 24, 2026, cs.SDJ/K move · Enter open · S save

Frequency-Aware Self-Supervised Music Representation Learning

Yicheng Gu, Junan Zhang, Jerry Li, Zhizheng Wu+1

Spellbrush, Akihabara, Tokyo 101-0021, Japan · Acoustic Lab, Department of Information and Communications Engineering (DICE), Aalto University, Espoo 02150, Finland · School of Data Science, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China

Audio Representation Learning Frequency-Domain Feature Learning Music Information Retrieval

Open paper AskPDF arXiv

Self-supervised learning (SSL) has emerged as an essential paradigm for music information retrieval (MIR). While current SSL models achieve state-of-the-art performance across various MIR tasks, they typically treat audio as 1D sequences, either operating on time-domain waveforms or on flattened time-frequency-domain spectrograms. This discards the rich spatial and structural information in time-frequency representations and overlooks a fundamental intuition in music production. In particular, music is naturally represented as time-frequency grids in MIDI-based workflows, a structure that tightly corresponds to 2D spectrograms and inherently makes many MIR tasks trivial. Motivated by this intuition, we propose PupuM2D, a musical Masked Modeling Duo (M2D) model that is trained directly on 2D music spectrograms. Instead of applying masked language modeling (MLM) to 1D sequences, PupuM2D learns robust representations by predicting the latent embeddings of masked 2D music spectrogram patches from unmasked contexts. To optimally adapt such a framework to music signals, we also apply domain-specific modifications to model architecture, training scheme, and inference paradigm, with comprehensive ablation studies showing their effectiveness. Evaluations on the MARBLEv2 benchmark show that PupuM2D outperforms the sequence-based SSL models across multiple MIR tasks in linear probing. Additionally, case studies of the attention maps also confirm that PupuM2D captures musically meaningful patterns within the 2D time-frequency domain. Codes and checkpoints are available at: https://www.yichenggu.com/PupuM2D/.

Momentum

Related topics

Latest papers 141