Audio Representation Learning

Ask about this topic

Overview

Audio representation learning studies how to encode waveforms or spectrograms into features that preserve information useful for tasks such as speech recognition, sound classification, music retrieval, and generation. Common approaches include self-supervised masked prediction in wav2vec 2.0, HuBERT, and BEST-RQ; contrastive audio-language learning such as CLAP; and distillation or autoencoder-based continuous and discrete audio tokenizers. Researchers assess transfer with linear probes and fine-tuning on benchmarks including SUPERB, ESC-50, MARBLE, and X-ARES, while also probing which acoustic, semantic, spatial, or speaker attributes the features retain. A central tension is how to make representations broadly transferable and compact while preserving fine acoustic detail, disentangling nuisance factors, and supporting both understanding and high-fidelity generation.

Latest papers

CardsList

211 papers30 papers in the last four weeks, up 25% on the four weeks before. 0.2% of all new papers.

Jan 25, 2025

Audio-Language Models for Audio-Centric Tasks: A Systematic Survey

Audio-Language Models (ALMs), trained on paired audio-text data, are designed to process, understand, and reason about audio-centric multimodal content. Unlike traditional supervised approaches that use predefined labels, ALMs leverage natural language supervision to better handle complex real-world audio scenes with multiple overlapping events. While demonstrating impressive zero-shot and task generalization capabilities, there is still a notable lack of systematic surveys that comprehensively organize and analyze developments. In this paper, we present the first systematic review of ALMs with three main contributions: (1) comprehensive coverage of ALM works across speech, music, and sound from a general audio perspective; (2) a unified taxonomy of ALM foundations, including model architectures and training objectives; (3) establishment of a research landscape capturing mutual promotion and constraints among different research aspects, aiding in summarizing evaluations, limitations, concerns and promising directions. Our review contributes to helping researchers understand the development of existing technologies and future trends, while also providing valuable references for implementation in practical applications.

Yi Su, Jisheng Bai, Qisheng Xu +2

College of Computer Science and Technology, National University of Defense Technology, Changsha, China · School of Communications and Information Engineering, Xi’an University of Posts and Telecommunications, Xi’an, China

Audio Representation LearningAudio UnderstandingAudio-Language Models
AskPDF