BAT: Better Audio Transformer Guided by Convex Gated Probing

Houtan Ghaffari; Lukas Rauch; Christoph Scholz; Paul Devos

cs.SDFeb 18, 2026

BAT: Better Audio Transformer Guided by Convex Gated Probing

Authors: Houtan Ghaffari, Lukas Rauch, Christoph Scholz, Paul Devos

Organizations: Ghent University · University of Kassel · Now at Earth Species Project · Fraunhofer IEE

Abstract

Probing is widely adopted in computer vision to faithfully evaluate self-supervised learning (SSL) embeddings, as finetuning may misrepresent their inherent quality. In contrast, audio SSL models still rely on finetuning because simple probing fails to unlock their full potential and alters their rankings when competing on AudioSet. Hence, a robust and efficient probing mechanism is required to guide the trajectory of audio SSL towards reliable and reproducible methods. We introduce Convex Gated Probing (CGP), a prototype-based method that significantly closes the gap between finetuning and probing in audio. CGP efficiently utilizes all frozen layers via a gating mechanism and exposes the location of latent task-relevant information. Guided by CGP as a reliable post-hoc evaluation probe, we rework the entire SSL pipeline of current best performing audio models that use legacy implementations of prior SSL methods. By refining data preprocessing, model architecture, and pretraining recipe, we introduce Better Audio Transformer (BAT), and establish new SOTA on audio benchmarks.

Figures & tables

Figure 1: Convex Gated Probing (CGP) method. We illustrate the probing process of a spectrogram embedding for a ViT backbone. CGP applies a learnable soft-gating vector (softmax) to compute a weighted sum of embeddings from all layers (

L

). The gating aggregates the hierarchy into a single representation, which is then compared against

K

prototypes. The cosine similarities of the patch embeddings are min-max pooled and concatenated with the ones from the cls -token, resulting in

3K

features for a linear classifier.

Method	AS-20k		AS-2M
	mAP	F1	mAP	F1
Reported finetuning
EAT ( Chen et al., 2024 )	40.2	-	48.6	-
SSLAM ( Alex et al., 2025 )	40.9	-	50.2	-
Our finetuning
EAT	40.28	30.73	47.61	36.63

Table 1: Benchmark of EAT and SSLAM on AudioSet . Comparison between reported performance and our reproduction across finetuning and various probing settings (CGP, Protobin, LP).

Figure 2: CGP Ablation on AS-20k. Increasing the number of prototypes constantly improves results, but yields diminishing returns. Although the best computation-performance trade-off is dataset-dependent, we will use 10k prototypes for all experiments to showcase the efficacy and reliability of CGP.

Data Pipeline	Global Statistics	mAP	F1
EAT (reproduced)	✓	34.86	24.28
Ours	✗	35.03	24.75

Table 2: Audio frontend ablation. We compare different spectrogram representations and normalization strategies. The baseline denotes our reproduction of the filterbank inputs used in EAT and AudioMAE, while our BAT configuration is highlighted.

Figure 3: Impact of audio frontend. A recording containing the labels [Whimper, Gasp, Speech, Outside, urban or manmade]. (a) Our incorporated audio frontend: Mel-spectrogram with decibel compression and local min-max normalization, exhibiting clear spectral structure and high contrast. (b) Audio-MAE, EAT, and SSLAM: simple log, filtering, Mel-spectrogram, and global standardization. Note the artifacts and blurring, particularly at lower frequencies.

SSL Target	Gated Attention	mAP	F1
MLP	✗	35.03	24.75
EOB	✗	34.60	25.00
MLP	✓	35.09	26.12
EOB	✓	35.42	27.13

Table 3: Impact of target selection and attention gates. EOB denotes End-of-Block. The baseline from the previous section and our proposed BAT configuration are highlighted.

Figure 4: Layer-wise latent information. We display the layer-wise latent information quality across three models on AS-20k: (a) BAT with the lightweight CNN (best performer from Table 3), (b) EAT (baseline), and (c) our final BAT (ViT decoder). The top row displays the linear probing performance of each block. The bottom row visualizes the learned gating weights from CGP. Notably, the standard EAT (b) and the CNN-based BAT (a) exhibit a middle-heavy distribution where semantic information peaks early. In contrast, the heavier ViT decoder in the final BAT (c) shifts the semantic peak toward the later layers, improving linear separability at the output.

Figure 5: Impact of gating on attention maps. Gating distributes attention better and focuses more on the token itself, rather than sinking into one token, primarily the cls -token.

Decoder	Depth	Heads	MLP Ratio	mAP	F1
CNN	-	-	-	35.42	27.13
ViT	6	6	2	37.43	28.91
ViT	6	12	4	37.52	29.11

Table 4: Impact of decoder. We replace the lightweight CNN decoder with a ViT. The baseline from the previous section and our proposed BAT configuration are highlighted.

		AS-2M		AS-20k		ESC-50		SC-v2		SED		HSN
Method	Model	mAP	F1	mAP	F1	mAP	Acc	mAP	Acc	frame-mAP	onset-F1	mAP	F1
Reported	SSLAM	50.2	-	40.9	-	-	96.2	-	98.1	-	-	-	-
Finetune	EAT	48.6	-	40.2	-	-	95.9	-	98.3	-	-	-	-
	BEATs	48.0	-	38.3	-	-	95.6	-	98.3	-	-	-	-
Finetune	BAT	48.85	36.62	41.59	38.20	98.81 $\pm$ 0.9	95.95 $\pm$ 1.2	99.80	99.09	99.71	98.20	47.05	34.07
	SSLAM	47.82	36.33	40.26	38.00	98.21 $\pm$ 1.0	94.85 $\pm$ 1.7	98.30	96.87	99.59	97.13	44.20	31.26

Table 5: Downstream task probing performance comparison across audio and speech benchmarks. We evaluate models using convex gated probing (CGP), linear probing (LP), linear convex gated probing (LCGP), protobin (PB), visual query tokens (VQT), and Head2Toe (H2T). BAT outperforms other baselines and CGP significantly outperforms all probing methods. The DCASE2016 Task 2 SED reports frame-wise micro-averaged mAP and event-onset detection micro-averaged F1 score. All other tasks report the macro-averaged mAP, accuracy and F1 scores. The best model for each probing method is highlighted .

Metrics	BAT	SSLAM	EAT	BEATs
CER	7.27	10.06	9.26	8.12
WER	22.18	29.05	27.14	24.67

Table 6: Automatic speech recognition. Probing on the test set.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameters

Pretraining

Finetuning

Probing

AS-2M

AS-20K

ESC-50

SC-v2

HSN

SED

AS-2M

AS-20K

ESC-50

SC-v2

HSN

SED

ASR

Optimizer

AdamW

Weight Decay

0.05

Optimizer Momentum (

\beta_{1}

\beta_{2}

)

(0.9, 0.95)

(0.9, 0.999)

Learning Rate Scheduler

Cosine

Peak Learning Rate

5e-4

5e-5

1e-3

Appendix

Table 7: Hyperparameter configurations. We show our hyperparameters for pretraining, finetuning, and probing. All probing methods used the same hyperparameters, with no special advantage for CGP. We tuned the number of tokens for VQT and the best Lasso for Head2Toe. Sound Event Detection (SED) refers to DCASE2016 Task 2. Automatic Speech Recognition (ASR) uses LibriSpeech. The loss function acronyms are: Binary Cross-Entropy (BCE), Asymmetric Binary Cross-Entropy (A-BCE) ( Ridnik et al., 2021 ) , and Connectionist Temporal Classification (CTC).

Dataset	Train	Validation	Test	#Classes	Segments [seconds]
AudioSet unbalanced (AS-2M)	$1,912,024$	-	$18,886$	$527$	10
AudioSet balanced (AS-20k)	$20,550$	-	$18,886$	$527$	10
Environmental Sound Classification (ESC-50)	$1,600$	-	$400$	$50$	5
Speech Commands v2 (SC-v2)	$84,848$	$9,982$	$4,890$	$12$	1
High Sierra Nevada (HSN)	$5,460$	-	$12,000$	$21$	5 (random crop)
DCASE 2016 Task 2 (SED)	$44$	$14$	$14$	$11$	120

Appendix

Table 8: Overview of datasets.

		AS-2M		AS-20K
Method	Model	mAP	F1	mAP	F1
Finetune	BAT-S	45.77	30.86	37.55	34.38
CGP	BAT-S	41.63	34.43	33.68	31.70
PB	BAT-S	40.19	32.26	32.63	29.27
VQT	BAT-S	40.01	33.28	30.76	29.90
H2T	BAT-S	32.74	23.78	25.03	26.37

Appendix

Table 9: ViT-Small downstream adaptation on AudioSet.

Method	Prototypes / Queries	Classes	Params (M)	Input & Shapes	Peak VRAM (MB)	Latency (ms)	MAC (G)
Linear	N/A	527	0.41	last layer cls=(1, 768)	28.72	$0.09\pm 0.02$	0.0004
Linear-CGP	N/A	527	0.41	cls=(1, 12, 768)	28.72	$0.14\pm 0.02$	0.0004
Head2Toe	N/A	527	4.88	cls=(1, 12, 768)	63.84	$0.10\pm 0.02$	0.0050
VQT	10	527	49.07	spectrogram=(1, 1024, 128)	637.96	$10.30\pm 0.25$	53.2681
Protobin	10000	527	12.95	last layer patch=(1, 768, 64, 8)	166.27	$0.91\pm 0.02$	3.9374
CGP	10000	527	23.49	[patch=(1, 12, 768, 64, 8), cls=(1, 12, 768)]	184.20	$0.72\pm 0.03$	3.9556

Appendix

Table 10: Computational cost and latency of the probing methods.

Figure 6: CGP layer-wise gating weights across datasets. The task-relevant information in BAT is pushed towards the later blocks even more than BEATs, which is a contrastive method. The difference in the distribution of the gating weights in this figure and those in Figure 4 of the main paper is due to the final benchmarking experiments for AudioSet leveraging the asymmetric loss. We plot these CGP models here. The ablation in the methodological exposition used the conventional cross-entropy loss to avoid additional factors affecting the findings due to the asymmetric loss’s sensitivity to its hyperparameters.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

Multi-layer attentive probing improves transfer of audio representations for bioacoustics
May 11, 2026Marius Miron, David Robinson, Masato Hagiwara +15Audio Representation Learning Bioacoustics
Earth Species Project
ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence
Oct 16, 2026Ming-Hao Hsu, Shujie Liu, Yuxuan Hu +4Audio-Language Models Speech Language Models
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China. · Microsoft Research Asia, Hong Kong, China. · Microsoft Research, Redmond, WA, USA.
Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
Aug 20, 2026Umberto Cappellazzo, Xubo Liu, Stavros Petridis +1Self-Supervised Speech Representation Learning Autoregressive Modeling
Imperial College London · University of Surrey

May 11, 2026J/K move · Enter open · S save

Multi-layer attentive probing improves transfer of audio representations for bioacoustics

Marius Miron, David Robinson, Masato Hagiwara, Titouan Parcollet+14

Earth Species Project

Audio Representation Learning Bioacoustics Hidden-State Probing

Open paper AskPDF arXiv

Probing heads map the representations learned from audio by a machine learning model to downstream task labels and are a key component in evaluating representation learning. Most bioacoustic benchmarks use a fixed, low-capacity probe, such as a linear layer on the final encoder layer. While this standardization enables model comparisons, it may bias results by overlooking the interaction between encoder features and probe design. In this work, we systematically study different probing strategies across two bioacoustic benchmarks, BEANs and BirdSet. We evaluate last- and multi-layer probing, across linear and attention probes. We show that larger probe heads that leverage time information have superior performance. Our results suggest that current benchmarks may misrepresent encoder quality when relying on a last-layer probing setup. Multi-layer probing improves downstream task performance across all tested models, while attention probing has superior performance to linear probing for transformer models.

Abstract

Audio Representation Learning

Explore similar work

Multi-layer attentive probing improves transfer of audio representations for bioacoustics

ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence

Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners