AudioFM: Towards General Purpose Audio Representation Learning

Yamin Li; Ruidong Zhang; Anran Wang; Eduardo Fonseca; Dan Ellis; Zechen Li; Simon A. Lee; Tao Pang

cs.SDOct 25, 2026

AudioFM: Towards General Purpose Audio Representation Learning

Authors: Yamin Li, Ruidong Zhang, Anran Wang, Eduardo Fonseca, Dan Ellis, Zechen Li, Simon A. Lee, Tao Pang,

Organizations: Google Research · Vanderbilt University · Google DeepMind · University of Washington

Abstract

Learning audio representations that effectively bridge holistic global soundscapes with fine-grained acoustic moments is a long-standing goal in self-supervised learning. Recording-level contrastive objectives encourage agreement between pooled audio representations, but do not explicitly supervise local time-frequency representations. We introduce AudioFM, a self-supervised framework that combines recording-level contrast with patch-wise masked latent discrimination. Using visible acoustic context, a student predicts representations of masked patches and identifies their corresponding full-view teacher targets among candidates from other recordings. We further incorporate weaker supervision from eligible temporal neighbors to encourage local temporal consistency. Extensive evaluations show that AudioFM achieves state-of-the-art performance across diverse audio domains in linear probing, fine-tuning, and zero-shot transfer, demonstrating superior event detection, robust scene-level understanding, and strong domain generalizability.

Figures & tables

Study	General audio	Speech recog.	Bioacoustics	Medical	Speech emotion	SED
AudioMAE ( Huang et al., 2022a )	✓	✓	✗	✗	✗	✗
BEATs ( Chen et al., 2023 )	✓	✓	✗	✗	✓	✗
ATST-Frame ( Li et al., 2024 )	✓	✓	✗	✗	✗	✓
M2D ( Niizumi et al., 2024 )	✓	✓	✗	✓	✓	✗
EAT ( Chen et al., 2024 )	✓	✓	✗	✗	✗	✗
SSLAM ( Alex et al., 2025 )	✓	✓	✗	✗	✗	✗

Table 1: Downstream evaluation coverage. ✓ / ✗ : dedicated evaluation reported/not reported in the original study; SED: frame-level sound event detection.

Figure 1: Overview of AudioFM . (a) A shared student encoder learns from complementary time- and frequency-masked views through recording-level contrast (a.1) and masked-patch prediction (a.2). An EMA teacher encodes the corresponding unmasked views to provide latent targets. (b) Detailed view of masked-patch discrimination in (a.2).

Figure 2: Mask-aware target selection. Temporal masking can exclude neighbors; frequency masking retains all in-bounds temporal neighbors.

Model	Backbone	Data (PT)	Params		Audio				Speech
Model	Backbone	Data (PT)	PT	FT	AS-20K	AS-2M	FSD50K	ESC-50	SPC-2	SPC-1
BEATs iter3 ( Chen et al., 2023 )	ViT-B/16	AS	182M	91M	38.0 ‡	48.0 / 46.7	57.6	95.6	98.3	97.7
AudioMAE ( Huang et al., 2022a )	ViT-B/16	AS	137M	86M	35.5 ‡	47.3 / 46.5	57.3	94.1	98.3	96.9
A-JEPA ( Fei et al., 2023 )	ViT-B/16	AS	354M	86M	38.4	48.6 / -	-	96.3	98.5	97.7
EAT ( Chen et al., 2024 )	ViT-B/16	AS	93M	88M	40.5 ‡	48.6 / 48.8	67.5	95.9	98.3	-
SSLAM ( Alex et al., 2025 )	ViT-B/16	AS	93M	88M	39.3 ‡	50.2 / 50.0	66.6	96.2	98.1	98.8

Table 2: Comparison with representative audio self-supervised learning methods. AS-20K, AS-2M, and FSD50K report mAP; the other tasks report accuracy (%). AS-2M baseline entries show published / re-evaluated mAP. Re-evaluation uses released fine-tuned checkpoints on the same evaluation subset as our model, without additional training. PT/FT: pre-training/fine-tuning; AS: AudioSet. AS ∗ : our smaller available AudioSet subset (

\sim

1.68M clips). ‡ : Re-fine-tuned on our AudioSet-20K training split and evaluated on our retained evaluation subset.

			Strict Bare Probing (Raw Features)						Flexible Probing (+LayerNorm)
Domain	Dataset	Metric	A-MAE	EAT	SSLAM	BAT	AMos	AudioFM	A-MAE	EAT	SSLAM	BAT	AMos	AudioFM
General	AudioSet-20K	mAP	13.9	1.1	0.9	18.1	28.7	31.2	11.9	18.7	18.2	28.5	33.4	35.0
	FSD50K	mAP	32.7	31.4	30.3	53.9	53.7	55.3	33.8	39.3	37.0	53.3	55.4	58.6
	ESC-50	Acc	77.5	66.6	87.0	92.7	93.2	94.5	77.0	83.4	91.5	92.4	93.8	94.7
Bioacoustics	BEANS-Watkins	Acc	81.9	61.9	34.5	57.1	88.9	89.1	70.7	85.6	82.3	82.2	89.3	85.9
	BEANS-Dogs	Acc	86.1	60.9	37.4	59.5	88.7	85.9	74.6	85.4	78.7	82.5	85.1	83.5

Table 3: Linear Probing Performance . We compare frozen representations evaluated under Strict Bare Probing (raw features without normalization) versus Flexible Probing (with downstream LayerNorm). Results are means over three random seeds, and all metrics are percentages (%). Best score is in bold , and the runner-up is underlined .

Figure 3: Frozen-encoder SED on AS-Strong.

Figure 4: Ablations and scaling. (a) Local-loss weight

\lambda_{\mathrm{local}}

with the default temporal neighborhood (

K=1

). (b) Temporal-neighborhood radius. Panels (a,b) report frozen-encoder probing gains over training without local loss; (c,d) AS-20K fine-tuning performance as encoder size and pretraining data increase.

Figure 5: Cross-domain transfer and pretrained audio representations. (a-c) Direct transfer without target-domain training: (a) clip-level sound event tagging on RealDESED, (b) cross-corpus and (c) cross-speaker emotion recognition. Error bars indicate standard deviations over three seeds. (d-e) t-SNE visualizations of PSG-Audio embeddings extracted from pretrained encoders without fine-tuning: (d) normal breathing versus apnea using AudioFM , and (e) four apnea subtypes using AudioFM and AudioMosaic.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	Pre-training	Fine-tuning
Configuration	AS-2M PT	AS-2M	AS-20K	FSD50K	ESC	SPC-2	SPC-1
Optimizer	AdamW
Optimizer momentum	$\beta_{1}=0.9$ , $\beta_{2}=0.999$
Weight decay	0.05	0.01
LR layer decay	1.0	0.75	0.75	0.75	0.75	0.75	0.75
Peak learning rate	6e-4	1.5e-4	5e-5	5e-5	5e-5	5e-4	5e-4

Appendix

Table 4: Pre-training (PT) and fine-tuning (FT) hyperparameters for AudioFM .

Figure 6: Sensitivity to structured masking ratios. AudioSet-20K linear-probing mAP when varying (a)

r_{t}

with

r_{f}=0.40

fixed, and (b)

r_{f}

with

r_{t}=0.60

fixed. Both panels use the same vertical scale. Hollow markers identify the shared default configuration

(r_{t},r_{f})=(0.60,0.40)

Domain	Dataset	Prediction target
Utterance-level classification
General audio	AudioSet-20K	Sound event categories
	ESC-50	Environmental sound class
	FSD50K	Sound event categories
Bioacoustics	BEANS-Watkins	Marine mammal species
	BEANS-Dogs	Individual dog identity

Appendix

Table 5: Linear probing datasets and prediction targets. Utterance-level tasks span four acoustic domains; AudioSet Strong additionally supports temporal event detection.

RealDESED Target Class	AudioSet Ontology ID	AudioSet Label Description
bell_ringing	/m/0395lw	Bell
cutlery_dishes	/m/04brg2, /m/023pjk	Dishes, pots, and pans; Cutlery, silverware
door_open_close	/m/02dgv	Door
footsteps	/m/07pbtc8	Walk, footsteps
keyboard_typing	/m/01m2v, /m/0316dw	Computer keyboard; Typing
keychain	/m/03v3yw	Keys jangling

Appendix

Table 6: Ontological Mapping Between RealDESED Classes and AudioSet Categories for cross-dataset sound event tagging.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

AudioMosaic: Contrastive Masked Audio Representation Learning
May 14, 2026Hanxun Huang, Qizhou Wang, Xingjun Ma +3Audio Representation Learning Self-Supervised Pre-Training
School of Computing and Information Systems, The University of Melbourne, Australia · Institute of Trustworthy Embodied AI, Fudan University, China · Baskin School of Engineering, University of California, Santa Cruz, USA
Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
Aug 20, 2026Umberto Cappellazzo, Xubo Liu, Stavros Petridis +1Self-Supervised Speech Representation Learning Autoregressive Modeling
Imperial College London · University of Surrey
BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations
Jun 29, 2026Ludovic K. Tuncay, Etienne Labbé, Thomas PellegriniAudio Representation Learning Self-Supervised Pre-Training
IRIT-SAMoVA · IRIT, Université de Toulouse, CNRS, Toulouse INP, Toulouse, France

May 14, 2026J/K move · Enter open · S save

AudioMosaic: Contrastive Masked Audio Representation Learning

Hanxun Huang, Qizhou Wang, Xingjun Ma, Cihang Xie+2

School of Computing and Information Systems, The University of Melbourne, Australia · Institute of Trustworthy Embodied AI, Fudan University, China · Baskin School of Engineering, University of California, Santa Cruz, USA

Audio Representation Learning Self-Supervised Pre-Training Audio Understanding

Open paper AskPDF arXiv

Audio self-supervised learning (SSL) aims to learn general-purpose representations from large-scale unlabeled audio data. While recent advances have been driven mainly by generative reconstruction objectives, contrastive approaches remain less explored, partly due to the difficulty of designing effective audio augmentations and the large batch sizes required for contrastive pre-training. We introduce AudioMosaic, a contrastive learning-based audio encoder for general audio understanding. During pre-training, AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches, which reduces memory usage and enables efficient large-batch training. Compared with generative approaches, the AudioMosaic encoder learns more discriminative utterance-level representations that demonstrate strong transferability across datasets, domains, and acoustic conditions. Extensive experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks under both linear probing and fine-tuning. We further show that integrating the pretrained AudioMosaic encoder into audio-language models improves performance on audio-language tasks. The code is publicly available in our GitHub repository.

Abstract

Audio Representation Learning

Explore similar work

AudioMosaic: Contrastive Masked Audio Representation Learning

Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners

BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations