Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners

Umberto Cappellazzo; Xubo Liu; Stavros Petridis; Maja Pantic

eess.ASAug 20, 2026

Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners

Authors: Umberto Cappellazzo, Xubo Liu, Stavros Petridis, Maja Pantic

Organizations: Imperial College London · University of Surrey

Abstract

Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.

Figures & tables

Figure 1: Left: NAPE’s overview. The input spectrogram is split into patches and embedded into a sequence of embeddings. At each step, the model predicts the embedding of the next patch (red border) using only the embeddings of the preceding patches ; patches at future positions are hidden by the causal attention mask. Middle: The NAPE pipeline: patch embeddings $z$ are processed by the causal encoder $h$ and predictor $g$ to produce predictions $\hat{z}$ , which are compared against the targets $z$ under stop-gradient using a similarity function $\mathcal{D}$ (i.e., negative cosine similarity). Right: Encoder architecture ( $h$ ): multiple stacked Transformer layers with pre-norm design, causal self-attention, LayerScale, and query-key normalization.

Figure 2: Patch scanning orders. NAPE linearizes the 2D spectrogram patch grid into a 1D causal sequence in one of four ways: raster (left-to-right, bottom-to-top), time-major (bottom-to-top within each time column, then advance in time), zigzag (raster with alternating row directions), and diagonal (sweep by frequency-plus-time index). The numbers indicate the position of each patch in the resulting sequence.

Pred	stop	causal	Audio Tasks			Speech Tasks
shift	grad	mask	AS-2M	AS-20K	ESC-50	KS1	KS2	ER
✘	✓	✓	Diverge
✓	✘	✓	Diverge
✓	✓	✘	41.8	24.8	68.9	96.1	97.3	57.0
✓	✓	✓	49.6	39.1	94.2	97.9	98.8	64.9

Table 1: Ablation on main NAPE’s design elements : prediction shift , stop-gradient , and causal objective .

Patch Emb.	Audio Tasks			Speech Tasks
Layer	AS-2M	AS-20K	ESC-50	KS1	KS2	ER
Convstem	46.7	34.3	89.1	97.4	98.3	63.6
Speechstem	47.6	33.1	88.4	98.1	98.9	63.0
Conv2d	49.6	39.1	94.2	97.9	98.8	64.9

Table 2: Ablation on the patch embedding layer.

Predictor	#Par.	Audio Tasks			Speech Tasks
Style	#Par.	AS-2M	AS-20K	ESC-50	KS1	KS2	ER
None	-	48.7	37.8	93.3	98.1	98.8	64.2
2-MLP	1.2M	49.4	38.5	93.6	98.0	98.8	64.2
Transformer	14.2M	49.2	38.4	93.0	98.2	98.7	65.0
SimSiam	1.8M	49.6	39.1	94.2	97.9	98.8	64.9

Table 3: Ablation on the predictor-style variants.

Figure 3: NAPE’s performance across four scan orders on six benchmarks.

Predicted	Audio Tasks			Speech Tasks
Target	AS-2M	AS-20K	ESC-50	KS1	KS2	ER
1st enc. layer	Diverge
Raw Mel	49.7	38.0	94.8	97.7	98.6	64.2
Patch embed	49.6	39.1	94.2	97.9	98.8	64.9

Table 4: Ablation on NAPE’s target to predict.

Similarity	Audio Tasks			Speech Tasks
Function	AS-2M	AS-20K	ESC-50	KS1	KS2	ER
L1	Diverge
L2	Diverge
Cross-entropy	48.8	37.4	93.6	98.0	98.7	64.5
Cosine	49.6	39.1	94.2	97.9	98.8	64.9

Table 5: Ablation on the similarity function.

Figure 4: Scaling comparison between Audio-MAE and NAPE, raster.

Figure 5: NAPE’s results at different scales under raster and diagonal scan orders.

Model	#Par.	Pre-train	Audio Tasks			Speech Tasks
Model	#Par.	Data	AS-2M	AS-20K	ESC-50	KS1	KS2	ER
Out-of-domain Supervised Pre-training
PSLA ( Gong et al., 2021b )	14M	IN	44.4	31.9	-	-	96.3	-
AST ( Gong et al., 2021a )	86M	IN	45.9	34.7	88.7	95.5	98.1	56.0
HTS-AT ( Chen et al., 2022a )	31M	IN	47.1	-	-	-	98.0	-
Audio-CLIP ( Guzhov et al., 2022 )	93M	TI+AS	25.9	-	96.7	-	-	-

Table 6: Comparison with audio methods on audio and speech downstream tasks. IN, AS, and LS denote the ImageNet, AudioSet, and LibriSpeech datasets, respectively. TI denotes the 400M text-image pairs for CLIP pre-training. We gray-out the models and results with additional supervised training on external datasets. Best results are in bold , second-best are underlined .

Figure 6: Layer-wise linear probing analysis.

Model	Layer	AS-2M	AS-20K	ESC-50
Small	2nd	23.2	18.9	79.8
Base	6th	25.0	19.7	81.7
Large	11th	27.1	20.4	83.5

Table 7: Linear probing results.

Panel of Figure 7 — Figure 7: Top: Prediction Quality Analysis. We report the cosine similarity between the predicted and true patch embeddings, averaged over $500$ audio clips ( left ) and for two individual clips ( middle , right ). Bottom: Attention/Embedding Analyses. Left: selected query patch; middle: attention map showing the patches NAPE attends to when predicting the next patch; right: embedding-similarity map between the predicted next-patch embedding $\hat{z}_{t+1}$ and every actual patch embedding in the spectrogram.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters	Pre-training	Fine-tuning
Hyperparameters	AS-2M	AS-2M	AS-20K	ESC-50	KS1	KS2	ER
Optimizer	AdamW
Opt. Momentum ( $\beta_{1}$ , $\beta_{2}$ )	(0.9,0.95)	(0.9,0.999)
Weight Decay	0.05
Learning Rate Scheduler	Cosine Decay
Layer-Wise LR Decay	1.0	0.7/0.9	0.8/0.9	0.7/0.9	0.7/0.8	0.7/0.8	0.7/0.9

Appendix

Table 8: Hyperparameter list . When an hyperparameter

h

varies between the base (

h_{b}

) and large (

h_{l}

) model, we include both values like (

h_{b}

h_{l}

). ∗ Following ( Chen et al., 2022b ) , we balance each class to

50

% of the size of the unknown class for each training epoch.

Attention	Pooling	Task
Type	Mode	AS-2M	AS-20K	KS2
Bidirec	CLS Tok	49.7	38.9	98.7
Bidirec	Last Tok	49.6	38.7	98.8
Bidirec	Avg Pool	49.6	39.1	98.8
Causal	Last Tok	49.4	38.9	98.8

Appendix

Table 9: Ablation on the pooling method and attention type at fine-tuning.

Table 17

Norm	Audio Tasks			Speech Tasks
Type	AS-2M	AS-20K	ESC-50	KS1	KS2	ER
LN	49.6	39.1	94.2	97.9	98.8	64.9
RMS	49.5	38.8	94.4	97.8	98.8	65.2

Appendix

Table 12: Ablation on normalization layer style.

Figure 8: Pre-training budget ablation . AS-2M (left) and AS-20K (right) mAP trend as a function of the number of epochs the model is trained on.

Mask	Audio Tasks			Speech Tasks
Ratio	AS-2M	AS-20K	ESC-50	KS1	KS2	ER
0	47.6	36.2	92.9	97.4	98.3	64.6
20	47.3	36.2	92.2	97.6	98.5	63.9
50	47.2	35.2	91.7	97.5	98.3	63.9

Appendix

Table 13: Ablation on random masking applied to the input embeddings.

Figure 9: Comparison between JEPA, LeWorldModel, and NAPE architectures.

Panel of Figure 10 — Figure 10: Pre-training loss curves for multiple NEPA models across scales and configurations.

Panel of Figure 11 — Figure 11: Additional qualitative results for NAPE-L with raster order.

Panel of Figure 12 — Figure 12: Additional qualitative results for NAPE-L with diagonal order.

Explore similar work

BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations
Jun 29, 2026Ludovic K. Tuncay, Etienne Labbé, Thomas PellegriniAudio Representation Learning Self-Supervised Pre-Training
IRIT-SAMoVA · IRIT, Université de Toulouse, CNRS, Toulouse INP, Toulouse, France
AudioMosaic: Contrastive Masked Audio Representation Learning
May 14, 2026Hanxun Huang, Qizhou Wang, Xingjun Ma +3Audio Representation Learning Self-Supervised Pre-Training
School of Computing and Information Systems, The University of Melbourne, Australia · Institute of Trustworthy Embodied AI, Fudan University, China · Baskin School of Engineering, University of California, Santa Cruz, USA
A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning
Mar 27, 2026Harunori Kawano, Takeshi SasakiAudio Representation Learning Self-Supervised Pre-Training
Faculty of Engineering and Information Technology, University of Technology Sydney · Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology

Jun 29, 2026J/K move · Enter open · S save

BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations

Ludovic K. Tuncay, Etienne Labbé, Thomas Pellegrini

IRIT-SAMoVA · IRIT, Université de Toulouse, CNRS, Toulouse INP, Toulouse, France

Audio Representation Learning Self-Supervised Pre-Training Self-Supervised Speech Representation Learning

Open paper AskPDF arXiv

Self-supervised learning enables audio representations that transfer across domains and tasks. We present BEST-RQ-2, an evolution of BEST-RQ that retains frozen randomprojection-based discrete targets while introducing a two-step contextualize-then-predict pretraining scheme. A ViT context encoder processes only the unmasked spectrogram regions, and a lightweight predictor infers targets for the masked regions; the predictor is discarded after pretraining. Replacing the original Conformer encoder with a ViT shifts performance across domains, slightly reducing speech performance while improving music and environmental sounds, with comparable average scores. The main improvement comes from decomposing masked prediction into separate contextualization and prediction stages. On the X-ARES and XARES-LLM benchmarks, BEST-RQ-2 consistently outperforms one-stage baselines in overall transfer while keeping inference compute unchanged. Code and model checkpoints are publicly available.

Abstract

Explore similar work

BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations

AudioMosaic: Contrastive Masked Audio Representation Learning

A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning