Audio Representation Learning

Audio representation learning studies how to encode waveforms or spectrograms into features that preserve information useful for tasks such as speech recognition, sound classification, music retrieval, and generation. Common approaches include self-supervised masked prediction in wav2vec 2.0, HuBERT, and BEST-RQ; contrastive audio-language learning such as CLAP; and distillation or autoencoder-based continuous and discrete audio tokenizers. Researchers assess transfer with linear probes and fine-tuning on benchmarks including SUPERB, ESC-50, MARBLE, and X-ARES, while also probing which acoustic, semantic, spatial, or speaker attributes the features retain. A central tension is how to make representations broadly transferable and compact while preserving fine acoustic detail, disentangling nuisance factors, and supporting both understanding and high-fidelity generation.