Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
Figures & tables
Figure 1: Left: NAPE’s overview. The input spectrogram is split into patches and embedded into a sequence of embeddings. At each step, the model predicts the embedding of the next patch (red border) using only the embeddings of the preceding patches ; patches at future positions are hidden by the causal attention mask. Middle: The NAPE pipeline: patch embeddings z are processed by the causal encoder h and predictor g to produce predictions z^ , which are compared against the targets z under stop-gradient using a similarity function D (i.e., negative cosine similarity). Right: Encoder architecture ( h ): multiple stacked Transformer layers with pre-norm design, causal self-attention, LayerScale, and query-key normalization.
Figure 2: Patch scanning orders. NAPE linearizes the 2D spectrogram patch grid into a 1D causal sequence in one of four ways: raster (left-to-right, bottom-to-top), time-major (bottom-to-top within each time column, then advance in time), zigzag (raster with alternating row directions), and diagonal (sweep by frequency-plus-time index). The numbers indicate the position of each patch in the resulting sequence.
Pred
stop
causal
Audio Tasks
Speech Tasks
shift
grad
mask
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
✘
✓
✓
Diverge
✓
✘
✓
Diverge
✓
✓
✘
41.8
24.8
68.9
96.1
97.3
57.0
✓
✓
✓
49.6
39.1
94.2
97.9
98.8
64.9
Table 1: Ablation on main NAPE’s design elements : prediction shift , stop-gradient , and causal objective .
Patch Emb.
Audio Tasks
Speech Tasks
Layer
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Convstem
46.7
34.3
89.1
97.4
98.3
63.6
Speechstem
47.6
33.1
88.4
98.1
98.9
63.0
Conv2d
49.6
39.1
94.2
97.9
98.8
64.9
Table 2: Ablation on the patch embedding layer.
Predictor
#Par.
Audio Tasks
Speech Tasks
Style
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
None
-
48.7
37.8
93.3
98.1
98.8
64.2
2-MLP
1.2M
49.4
38.5
93.6
98.0
98.8
64.2
Transformer
14.2M
49.2
38.4
93.0
98.2
98.7
65.0
SimSiam
1.8M
49.6
39.1
94.2
97.9
98.8
64.9
Table 3: Ablation on the predictor-style variants.
Figure 3: NAPE’s performance across four scan orders on six benchmarks.
Predicted
Audio Tasks
Speech Tasks
Target
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
1st enc. layer
Diverge
Raw Mel
49.7
38.0
94.8
97.7
98.6
64.2
Patch embed
49.6
39.1
94.2
97.9
98.8
64.9
Table 4: Ablation on NAPE’s target to predict.
Similarity
Audio Tasks
Speech Tasks
Function
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
L1
Diverge
L2
Diverge
Cross-entropy
48.8
37.4
93.6
98.0
98.7
64.5
Cosine
49.6
39.1
94.2
97.9
98.8
64.9
Table 5: Ablation on the similarity function.
Figure 4: Scaling comparison between Audio-MAE and NAPE, raster.
Figure 5: NAPE’s results at different scales under raster and diagonal scan orders.
Model
#Par.
Pre-train
Audio Tasks
Speech Tasks
Data
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Out-of-domain Supervised Pre-training
PSLA ( Gong et al., 2021b )
14M
IN
44.4
31.9
-
-
96.3
-
AST ( Gong et al., 2021a )
86M
IN
45.9
34.7
88.7
95.5
98.1
56.0
HTS-AT ( Chen et al., 2022a )
31M
IN
47.1
-
-
-
98.0
-
Audio-CLIP ( Guzhov et al., 2022 )
93M
TI+AS
25.9
-
96.7
-
-
-
Table 6: Comparison with audio methods on audio and speech downstream tasks. IN, AS, and LS denote the ImageNet, AudioSet, and LibriSpeech datasets, respectively. TI denotes the 400M text-image pairs for CLIP pre-training. We gray-out the models and results with additional supervised training on external datasets. Best results are in bold , second-best are underlined .
Figure 6: Layer-wise linear probing analysis.
Model
Layer
AS-2M
AS-20K
ESC-50
Small
2nd
23.2
18.9
79.8
Base
6th
25.0
19.7
81.7
Large
11th
27.1
20.4
83.5
Table 7: Linear probing results.
Figure 7: Top: Prediction Quality Analysis. We report the cosine similarity between the predicted and true patch embeddings, averaged over 500 audio clips ( left ) and for two individual clips ( middle , right ). Bottom: Attention/Embedding Analyses. Left: selected query patch; middle: attention map showing the patches NAPE attends to when predicting the next patch; right: embedding-similarity map between the predicted next-patch embedding z^t+1 and every actual patch embedding in the spectrogram.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameters
Pre-training
Fine-tuning
AS-2M
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
Optimizer
AdamW
Opt. Momentum ( β1 , β2 )
(0.9,0.95)
(0.9,0.999)
Weight Decay
0.05
Learning Rate Scheduler
Cosine Decay
Layer-Wise LR Decay
1.0
0.7/0.9
0.8/0.9
0.7/0.9
0.7/0.8
0.7/0.8
0.7/0.9
Appendix
Table 8: Hyperparameter list . When an hyperparameter h varies between the base ( hb ) and large ( hl ) model, we include both values like ( hb / hl ). ∗ Following ( Chen et al., 2022b ) , we balance each class to 50 % of the size of the unknown class for each training epoch.
Attention
Pooling
Task
Type
Mode
AS-2M
AS-20K
KS2
Bidirec
CLS Tok
49.7
38.9
98.7
Bidirec
Last Tok
49.6
38.7
98.8
Bidirec
Avg Pool
49.6
39.1
98.8
Causal
Last Tok
49.4
38.9
98.8
Appendix
Table 9: Ablation on the pooling method and attention type at fine-tuning.
Table 17
Norm
Audio Tasks
Speech Tasks
Type
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
LN
49.6
39.1
94.2
97.9
98.8
64.9
RMS
49.5
38.8
94.4
97.8
98.8
65.2
Appendix
Table 12: Ablation on normalization layer style.
Figure 8: Pre-training budget ablation . AS-2M (left) and AS-20K (right) mAP trend as a function of the number of epochs the model is trained on.
Mask
Audio Tasks
Speech Tasks
Ratio
AS-2M
AS-20K
ESC-50
KS1
KS2
ER
0
47.6
36.2
92.9
97.4
98.3
64.6
20
47.3
36.2
92.2
97.6
98.5
63.9
50
47.2
35.2
91.7
97.5
98.3
63.9
Appendix
Table 13: Ablation on random masking applied to the input embeddings.
Figure 9: Comparison between JEPA, LeWorldModel, and NAPE architectures.
Figure 10: Pre-training loss curves for multiple NEPA models across scales and configurations.
Figure 11: Additional qualitative results for NAPE-L with raster order.
Figure 12: Additional qualitative results for NAPE-L with diagonal order.
Self-supervised learning enables audio representations that transfer across domains and tasks. We present BEST-RQ-2, an evolution of BEST-RQ that retains frozen randomprojection-based discrete targets while introducing a two-step contextualize-then-predict pretraining scheme. A ViT context encoder processes only the unmasked spectrogram regions, and a lightweight predictor infers targets for the masked regions; the predictor is discarded after pretraining. Replacing the original Conformer encoder with a ViT shifts performance across domains, slightly reducing speech performance while improving music and environmental sounds, with comparable average scores. The main improvement comes from decomposing masked prediction into separate contextualization and prediction stages. On the X-ARES and XARES-LLM benchmarks, BEST-RQ-2 consistently outperforms one-stage baselines in overall transfer while keeping inference compute unchanged. Code and model checkpoints are publicly available.
Ludovic K. Tuncay, Etienne Labbé, Thomas Pellegrini
IRIT-SAMoVA · IRIT, Université de Toulouse, CNRS, Toulouse INP, Toulouse, France
Audio self-supervised learning (SSL) aims to learn general-purpose representations from large-scale unlabeled audio data. While recent advances have been driven mainly by generative reconstruction objectives, contrastive approaches remain less explored, partly due to the difficulty of designing effective audio augmentations and the large batch sizes required for contrastive pre-training. We introduce AudioMosaic, a contrastive learning-based audio encoder for general audio understanding. During pre-training, AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches, which reduces memory usage and enables efficient large-batch training. Compared with generative approaches, the AudioMosaic encoder learns more discriminative utterance-level representations that demonstrate strong transferability across datasets, domains, and acoustic conditions. Extensive experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks under both linear probing and fine-tuning. We further show that integrating the pretrained AudioMosaic encoder into audio-language models improves performance on audio-language tasks. The code is publicly available in our GitHub repository.
Hanxun Huang, Qizhou Wang, Xingjun Ma +3
School of Computing and Information Systems, The University of Melbourne, Australia · Institute of Trustworthy Embodied AI, Fudan University, China · Baskin School of Engineering, University of California, Santa Cruz, USA
While self-supervised learning (SSL) has revolutionized audio representation, the excessive parameterization and quadratic computational cost of standard Transformers limit their deployment on resource-constrained devices. To address this bottleneck, we propose HEAR (Human-inspired Efficient Audio Representation), a novel decoupled architecture. Inspired by the human cognitive ability to isolate local acoustic features from global context, HEAR splits the processing pipeline into two dedicated modules: an Acoustic Model for local feature extraction and a Task Model for global semantic integration. Coupled with an Acoustic Tokenizer trained via knowledge distillation, our approach enables robust Masked Audio Modeling (MAM). Extensive experiments demonstrate that HEAR requires only 15M parameters and 9.47 GFLOPs for inference, operating at a fraction of the computational cost of conventional foundation models (which typically require 85M-94M parameters). Despite this high efficiency, HEAR achieves highly competitive performance across diverse audio classification benchmarks. The code and pre-trained models are available at https://github.com/HarunoriKawano/HEAR
Harunori Kawano, Takeshi Sasaki
Faculty of Engineering and Information Technology, University of Technology Sydney · Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology