Audio Representation Learning

Follow this topic Ask about this topic

Momentum

20 papers in the last four weeks, up 233% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 141

All topics

Frequency-Aware Self-Supervised Music Representation Learning
Jun 24, 2026Yicheng Gu, Junan Zhang, Jerry Li +2Audio Representation Learning Frequency-Domain Feature Learning
Spellbrush, Akihabara, Tokyo 101-0021, Japan · Acoustic Lab, Department of Information and Communications Engineering (DICE), Aalto University, Espoo 02150, Finland · School of Data Science, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China
Does Translation-Enhanced Speech Encoder Pre-training Affect Speech LLMs?
Jun 24, 2026Tomoya Mizumoto, Yusuke FujitaAudio Representation Learning Self-Supervised Speech Representation Learning
SB Intuitions
PHAST-Net: Attention-Guided, Physics-Informed Network for Unified Estimation of Ideal Time-Frequency Representations
Jun 22, 2026James M. Cozens, Simon J. GodsillAudio Representation Learning Wavelet Methods
Engineering Department, University of Cambridge, CB2 1PZ Cambridge, U.K.
STAR-VAE: Structured Topology-Aware Regularization for Audio Reconstruction and Generation
Jun 22, 2026Huadai Liu, Wen Wang, Kaicheng Luo +3Variational Autoencoders Text-to-Audio Generation
Hong Kong University of Science and Technology · Tongyi Fun Team, Alibaba Group.
Toward Open-Set Speaker Attribute Prediction with Keyword-Appended LLM Embeddings
Jun 20, 2026Byoungjun So, Jaejun Lee, Kyogu LeeAudio Representation Learning Speech Processing
Department of Intelligence and Information, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University · Artificial Intelligence Institute, Seoul National University, Republic of Korea
AugCodec: A Low-Bitrate Disentangled Neural Speech Codec via Data Augmentation
Jun 20, 2026Dongmei Wang, Xiaohang Sun, Yang Liu +10Disentangled Representation Learning Audio Representation Learning
Amazon, USA
ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion
Jun 20, 2026Jeongsoo Choi, Ji-Hoon Kim, Shujie Hu +1Audio Representation Learning Neural Audio Codecs
KAIST, South Korea · Chung-Ang University, South Korea · The Chinese University of Hong Kong, China
Gradient-Based Learning of Parametric Engine Sound Representations for Real-Time Resynthesis and Tuning on Embedded Systems
Jun 19, 2026Robin Doerfler, Matthieu Kuntz, Clemens ZimmerAudio Representation Learning Audio Generation
Impulse Audio Lab GmbH Munich, Germany
CoughPhase-CLR: Designing an acoustics-informed foundation model for coughing sound classification
Jun 19, 2026Marius Moldovan, Anton Batliner, Thomas M. Berghaus +2Contrastive Learning Audio Representation Learning
CHI – the Chair of Health Informatics at the TUM University Hospital, Munich, Germany · MCML – the Munich Center for Machine Learning and MDSI – the Munich Data Science Institute, Munich, Germany · University Hospital Augsburg at the University of Augsburg, Augsburg, Germany +2
LISE : Listenable Interpretable Speaker Embeddings
Jun 19, 2026Xiaoliang Wu, Chongxin Gan, Ke Liu +2Audio Representation Learning Speaker Verification
University of Southampton, United Kingdom · The Hong Kong Polytechnic University, Hong Kong SAR, China · University of Edinburgh, United Kingdom
Segment-Level Mandarin Chinese Speech-Based Cognitive Impairment Detection via an Autoencoder with Contrastive Learning
Jun 18, 2026Yongqi Shao, Hong Huo, Flavio Bertini +2Audio Representation Learning Representation Learning
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University; Key Laboratory of System Control and Information Processing, Ministry of Education of China; Shanghai Key Laboratory of Perception and Control in Industrial Network Systems, Shanghai, 200240, China · Department of Mathematical, Physical and Computer Sciences, University of Parma, Parco Area delle Sciences 53/A, Parma, 43124, Italy · Department of Computer Science and Engineering, University of Bologna, Mura Anteo Zamboni 7, Bologna, 40126, Italy
Acoustic Prompting via Stage-wise Modulation for Few-Shot Learning in Audio Language Models
Jun 14, 2026Hyebin Cho, Jaehyuk Jang, Changick Kim +1Few-Shot Audio Classification Audio Representation Learning
Korea Advanced Institute of Science and Technology, South Korea
Beyond task performance: Decoding bioacoustic embeddings with speech features
Jun 12, 2026Ines Nolasco, Jules Cauzinille, Marius Miron +8Audio Representation Learning Representation Probing
Earth Species Project
Spectro-Temporal Interference Confounds Phase Encoding in Spatial Audio Foundation Models
Jun 12, 2026Yuxuan Chen, Haoyuan Yu, Peize HeAudio Representation Learning Direction-of-Arrival Estimation
The Chinese University of Hong Kong, Shenzhen · Jilin University · Hunan University +1
Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources
Jun 12, 2026Oh Hyun-Bin, Kazuki Shimada, Yuhta Takida +6Audio-Language Model Evaluation Audio Representation Learning
1POSTECH · 2Sony AI · 4Sungkyunkwan University +2
From Tokens to Faces: Investigating Discrete Speech Representations for 3D Facial Animation
Jun 11, 2026Pedro Correa, Olivier Perrotin, Samir Sadok +2Audio Representation Learning Neural Audio Codecs
Univ. Estadual de Campinas (UNICAMP), Brazil · Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, France · Inria at Univ. Grenoble Alpes, CNRS, LJK, France
Decoding Insect Song: A Multitask Semisupervised Orthoptera Bioacoustic Classifier
Jun 11, 2026Olga Isupova, Danil Kuzin, Ella Browning +2Audio Representation Learning Audio Understanding
Leverhulme Centre for Nature Recovery, School of Geography and the Environment, University of Oxford, Oxford, UK
From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation
Jun 11, 2026Fengrui Liu, Ruiyang Huang, Qijian Zheng +2Synthetic Data Pretraining Audio Representation Learning
East China Normal University · Southeast University · Fudan University +1
Dolph2Vec: Self-Supervised Representations of Dolphin Vocalizations
Jun 10, 2026Chiara Semenzin, Faadil Mustun, Roberto Dessi +5Audio Representation Learning Audio Understanding
École Normale Supérieure, Paris, France · Not Diamond, San Francisco, USA · Institut du Cerveau, Paris, France +1
Which Speech Representation Better Matches Text-Native Reasoning? A Study of Speech-Text Alignment on Frame Rate and Representation
Jun 10, 2026Zhen Ye, Xu Tan, Yiming Li +10Spoken Language Understanding Audio Representation Learning
Hong Kong University of Science and Technology, Hong Kong SAR · Tencent, China · Independent Researcher +4
SARA: A Dual-Stream VAE for High-Fidelity Speech Generation via Integrating Semantic and Acoustic Representations
Jun 10, 2026Peijie Chen, Wenhao Guan, Weijie Wu +7Variational Autoencoders Audio Representation Learning
School of Informatics, Xiamen University, China · School of Electronic Science and Engineering, Xiamen University, China · DiDi Global Inc., Beijing, China
Speaker Group Encoding in Self-supervised Speech Recognition Models
Jun 9, 2026Felix Herron, Solange Rossato Alexandre Allauzen, Benoit Favre +1Audio Representation Learning Demographic Bias in ASR
MILES Team, LAMSADE, Université Paris Dauphine-PSL, France (1) · GETALP Team, LIG, Université Grenoble Alpes, France (2) · NLP team, LIS, Aix-Marseille University, France (3)
ViP-VL: Vietnamese Self-supervised Speech Pretraining Model with Vector-Quantization Learning
Jun 9, 2026Khanh Le, Kiet Anh Hoang, Bao Nguyen +5Audio Representation Learning Speech Processing
VinUniversity, Vietnam · UNEY, Switzerland
Dual-Branch Gated Fusion for Open-Set Audio Deepfake Source Tracing
Jun 8, 2026Awais Khan, Kutub Uddin, Khalid MalikAudio Representation Learning Audio Deepfake Detection
College of Innovation and Technology, University of Michigan, Flint, MI, USA · ProbeTruth Inc., MI, USA
Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs
Jun 8, 2026Ming-Hao Hsu, Yuxuan Hu, Shujie Liu +3Audio Representation Learning Audio Understanding
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research, Redmond, WA, USA. · Microsoft Research Asia, Hong Kong, China.
Multi-View Speech Representation Learning for Parkinson's Disease Detection Using Context-guided Cross-modal Attention
Jun 8, 2026George Theodosiou, Loukas Ilias, Dimitris AskounisAudio Representation Learning Speech Processing
Decision Support Systems Laboratory, School of Electrical and Computer Engineering, National Technical University of Athens, 15780 Athens, Greece
Probing Token Spaces under Generator Shift in AI-Generated Music Detection
Jun 7, 2026Joonyong Park, Jungwoo Kim, Junyoung Koh +1Audio Representation Learning Audio Deepfake Detection
1The University of Tokyo, Tokyo, Japan · 3Yonsei University, Seoul, Republic of Korea. · 2MAAP Lab
HybridCodec: Fast Dual-Stream, Semantically Enhanced Neural Audio Codec
Jun 4, 2026Arjun Gangwar, S UmeshDisentangled Representation Learning Audio Representation Learning
Indian Institute of Technology, Madras, India
USAD 2.0: Scaling Representation Distillation for Universal Audio Understanding
Jun 4, 2026Heng-Jui Chang, Alexander H. Liu, Saurabhchand Bhati +4Audio Representation Learning Audio Understanding
MIT CSAIL, USA · Amazon, USA
F3-Tokenizer: Taming Audio Autoencoder Latents for Understanding and Generation
Jun 4, 2026Dinghao Zhou, Xingchen Song, Di Wu +3Audio Representation Learning Autoencoders
2WeNet Open Source Community · ∗Equal contribution · 1Nanjing University, China

Jun 24, 2026, cs.SDJ/K move · Enter open · S save

Frequency-Aware Self-Supervised Music Representation Learning

Yicheng Gu, Junan Zhang, Jerry Li, Zhizheng Wu+1

Spellbrush, Akihabara, Tokyo 101-0021, Japan · Acoustic Lab, Department of Information and Communications Engineering (DICE), Aalto University, Espoo 02150, Finland · School of Data Science, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China

Audio Representation Learning Frequency-Domain Feature Learning Music Information Retrieval

Open paper AskPDF arXiv

Self-supervised learning (SSL) has emerged as an essential paradigm for music information retrieval (MIR). While current SSL models achieve state-of-the-art performance across various MIR tasks, they typically treat audio as 1D sequences, either operating on time-domain waveforms or on flattened time-frequency-domain spectrograms. This discards the rich spatial and structural information in time-frequency representations and overlooks a fundamental intuition in music production. In particular, music is naturally represented as time-frequency grids in MIDI-based workflows, a structure that tightly corresponds to 2D spectrograms and inherently makes many MIR tasks trivial. Motivated by this intuition, we propose PupuM2D, a musical Masked Modeling Duo (M2D) model that is trained directly on 2D music spectrograms. Instead of applying masked language modeling (MLM) to 1D sequences, PupuM2D learns robust representations by predicting the latent embeddings of masked 2D music spectrogram patches from unmasked contexts. To optimally adapt such a framework to music signals, we also apply domain-specific modifications to model architecture, training scheme, and inference paradigm, with comprehensive ablation studies showing their effectiveness. Evaluations on the MARBLEv2 benchmark show that PupuM2D outperforms the sequence-based SSL models across multiple MIR tasks in linear probing. Additionally, case studies of the attention maps also confirm that PupuM2D captures musically meaningful patterns within the 2D time-frequency domain. Codes and checkpoints are available at: https://www.yichenggu.com/PupuM2D/.

Momentum

Related topics

Latest papers 141