Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning

Daisuke Niizumi; Daiki Takeuchi; Masahiro Yasuda; Binh Thien Nguyen; Noboru Harada; Nobutaka Ono

eess.ASMar 25, 2026

Rethinking Masking Strategies for Masked Prediction-based Audio Self-supervised Learning

Authors: Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen, Noboru Harada, Nobutaka Ono

Organizations: Tokyo Metropolitan University, Tokyo, Japan · NTT Inc., Atsugi, Japan

Abstract

Since the introduction of Masked Autoencoders, various improvements to masking techniques have been explored. In this paper, we rethink masking strategies for audio representation learning using masked prediction-based self-supervised learning (SSL) on general audio spectrograms. While recent informed masking techniques have attracted attention, we observe that they incur substantial computational overhead. Motivated by this observation, we propose dispersion-weighted masking (DWM), a lightweight masking strategy that leverages the spectral sparsity inherent in the frequency structure of audio content. Our experiments show that inverse block masking, commonly used in recent SSL frameworks, improves audio event understanding performance while introducing a trade-off in generalization. The proposed DWM alleviates these limitations and computational complexity, leading to consistent performance improvements. This work provides practical guidance on masking strategy design for masked prediction-based audio representation learning.

Figures & tables

Fig. 1: Masking examples on an audio spectrogram (hint ratio $r_{h}=0.0$ ). Object-centric masking (SGIM) almost entirely covers audio events (objects), whereas the proposed DWM applies dispersion-weighted random masking based on patch-wise spectral variability, yielding masking patterns that loosely align with object-centric behavior.

	Env. sound tasks		Speech tasks				Music tasks
Masking strategy	ESC-50	US8K	SPCV2	VC1	VF	CRM-D	GTZAN	NSynth	Surge	Avg.
(MSM-MAE variants)
Random	89.2 $\pm$ 0.4	87.2 $\pm$ 0.2	96.1 $\pm$ 0.1	73.4 $\pm$ 0.4	97.9 $\pm$ 0.1	71.3 $\pm$ 0.2	80.3 $\pm$ 1.0	74.6 $\pm$ 0.4	43.2 $\pm$ 0.2	79.2
IBM [ 6 ]	90.1 $\pm$ 0.3	88.2 $\pm$ 0.4	95.4 $\pm$ 0.1	65.1 $\pm$ 0.5	97.7 $\pm$ 0.1	70.9 $\pm$ 0.6	80.5 $\pm$ 2.3	74.2 $\pm$ 0.6	42.8 $\pm$ 0.1	78.3
vs. random [pp]	+0.9	+1.0	-0.7	-8.3	-0.2	-0.4	+0.2	-0.4	-0.4	-0.9
DWM (proposed)	89.9 $\pm$ 0.7	87.4 $\pm$ 0.1	95.9 $\pm$ 0.1	72.5 $\pm$ 0.7	97.8 $\pm$ 0.1	70.9 $\pm$ 0.3	81.8 $\pm$ 1.1	75.5 $\pm$ 0.1	43.3 $\pm$ 0.3	79.5

TABLE I: Linear evaluation results (%) with 95% CI comparing different masking strategies.

(MAE variants)
Masking	AS2M	AS20K	ESC-50	SPCV2	VC1
strategy	mAP	mAP	acc(%)	acc(%)	acc(%)
Random	47.4 $\pm$ 0.1	37.9 $\pm$ 0.0	95.4 $\pm$ 0.1	98.4 $\pm$ 0.0	96.6 $\pm$ 0.2
IBM [ 6 ]	47.4 $\pm$ 0.1	38.6 $\pm$ 0.1	95.4 $\pm$ 0.3	98.3 $\pm$ 0.0	95.4 $\pm$ 0.5
diff.[pp]	0.0	+0.7	0.0	-0.1	-1.2
DWM	47.5 $\pm$ 0.1	38.3 $\pm$ 0.1	95.5 $\pm$ 0.2	98.4 $\pm$ 0.1	96.4 $\pm$ 0.2

TABLE II: Fine-tuning results with 95% CI comparing masking strategies.

	Env. sound tasks		Speech tasks				Music tasks
Scheduling	ESC-50	US8K	SPCV2	VC1	VF	CRM-D	GTZAN	NSynth	Surge	Avg.
Baseline MSM-MAE	89.2 $\pm$ 0.4	87.2 $\pm$ 0.2	96.1 $\pm$ 0.1	73.4 $\pm$ 0.4	97.9 $\pm$ 0.1	71.3 $\pm$ 0.2	80.3 $\pm$ 1.0	74.6 $\pm$ 0.4	43.2 $\pm$ 0.2	79.2
Hint ratio $r_{h}$ =0.0	89.9 $\pm$ 1.1	87.3 $\pm$ 0.7	95.5 $\pm$ 0.1	69.1 $\pm$ 0.1	97.7 $\pm$ 0.0	70.4 $\pm$ 0.6	83.4 $\pm$ 4.5	75.4 $\pm$ 0.2	42.5 $\pm$ 0.9	79.0
Hint ratio $r_{h}$ =0.5	87.4 $\pm$ 0.9	86.3 $\pm$ 0.6	96.0 $\pm$ 0.2	71.8 $\pm$ 0.5	97.6 $\pm$ 0.3	70.5 $\pm$ 0.9	82.1 $\pm$ 3.1	74.3 $\pm$ 0.4	42.5 $\pm$ 0.5	78.7
Schedule $\gamma$ =1	88.1 $\pm$ 0.2	87.3 $\pm$ 0.3	96.2 $\pm$ 0.2	70.0 $\pm$ 0.5	97.7 $\pm$ 0.2	71.4 $\pm$ 0.9	82.0 $\pm$ 6.4	76.0 $\pm$ 0.2	42.6 $\pm$ 0.7	79.0
Schedule $\gamma$ =2	89.9 $\pm$ 0.7	87.4 $\pm$ 0.1	95.9 $\pm$ 0.1	72.5 $\pm$ 0.7	97.8 $\pm$ 0.1	70.9 $\pm$ 0.3	81.8 $\pm$ 1.1	75.5 $\pm$ 0.1	43.3 $\pm$ 0.3	79.5

TABLE III: Ablation of hint ratio scheduling in DWM: Linear evaluation results (%) with 95% CI.

Fig. 2: Hint ratio $r_{h}$ schedules over training epochs for ablation settings.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

AudioMosaic: Contrastive Masked Audio Representation Learning
May 14, 2026Hanxun Huang, Qizhou Wang, Xingjun Ma +3Audio Representation Learning Self-Supervised Pre-Training
School of Computing and Information Systems, The University of Melbourne, Australia · Institute of Trustworthy Embodied AI, Fudan University, China · Baskin School of Engineering, University of California, Santa Cruz, USA
Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
Aug 20, 2026Umberto Cappellazzo, Xubo Liu, Stavros Petridis +1Self-Supervised Speech Representation Learning Autoregressive Modeling
Imperial College London · University of Surrey
Structured-Noise Masked Modeling for Video, Audio and Beyond
Mar 20, 2025Aritra Bhowmik, Carlos Hinojosa, Fida Mohammad Thoker +2Audio Representation Learning Video Representation Learning
University of Amsterdam · King Abdullah University of Science and Technology

May 14, 2026J/K move · Enter open · S save

AudioMosaic: Contrastive Masked Audio Representation Learning

Hanxun Huang, Qizhou Wang, Xingjun Ma, Cihang Xie+2

School of Computing and Information Systems, The University of Melbourne, Australia · Institute of Trustworthy Embodied AI, Fudan University, China · Baskin School of Engineering, University of California, Santa Cruz, USA

Audio Representation Learning Self-Supervised Pre-Training Audio Understanding

Open paper AskPDF arXiv

Audio self-supervised learning (SSL) aims to learn general-purpose representations from large-scale unlabeled audio data. While recent advances have been driven mainly by generative reconstruction objectives, contrastive approaches remain less explored, partly due to the difficulty of designing effective audio augmentations and the large batch sizes required for contrastive pre-training. We introduce AudioMosaic, a contrastive learning-based audio encoder for general audio understanding. During pre-training, AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches, which reduces memory usage and enables efficient large-batch training. Compared with generative approaches, the AudioMosaic encoder learns more discriminative utterance-level representations that demonstrate strong transferability across datasets, domains, and acoustic conditions. Extensive experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks under both linear probing and fine-tuning. We further show that integrating the pretrained AudioMosaic encoder into audio-language models improves performance on audio-language tasks. The code is publicly available in our GitHub repository.

Abstract

Audio Representation Learning

Explore similar work

AudioMosaic: Contrastive Masked Audio Representation Learning

Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners

Structured-Noise Masked Modeling for Video, Audio and Beyond