From Masking to Merging: Rethinking SpecAugment for Efficient Audio Spectrogram Transformer

Minhee Park; Hyowon Ahn; Chanwoo Kim

cs.SDSep 6, 2026

From Masking to Merging: Rethinking SpecAugment for Efficient Audio Spectrogram Transformer

Authors: Minhee Park, Hyowon Ahn, Chanwoo Kim

Organizations: Korea University, Republic of Korea

Abstract

This paper proposes SpecAugment-Patch Merging, a simple yet effective method to accelerate Audio Spectrogram Transformer (AST) training. We first apply SpecAugment to mask input spectrograms at the patch level, and after positional embeddings are added, the method selects r pairs of masked patches and merges them, reducing the number of tokens processed by the Transformer. Increasing the number of merged pairs r from 0 to 100 keeps mAP on AudioSet nearly unchanged (34.07 to 34.08) while throughput increases from 43.3 to 49.3 samples/sec, which is a relatively 13.9% improvement. Similar patterns appear on ESC-50 and Speech Commands V2, where throughput steadily improves with only minor accuracy changes, demonstrating that this merging approach provides faster training with minimal performance loss.

Figures & tables

Figure 1: Overall architecture of the proposed SpecAugment-Patch Merging AST.

Method	mAP	S/s
SpecAugment [ 5 ]	34.11 $\pm$ 0.35	43.3
SpecAugment-Patch	34.07 $\pm$ 0.18	43.3

Table 1: Results (mAP) and training throughput S/s (samples/sec) comparison between SpecAugment and SpecAugment-Patch on Balanced AudioSet, both combined with Mixup, at

r=0

	Drop	Merge
	Drop	Max	Sum	Mean
mAP	34.03 $\pm$ 0.11	34.10 $\pm$ 0.31	34.07	34.00
S/s	48.5 S/s	48.63 S/s	48.6 S/s	48.9 S/s

Table 2: Comparison of token-reduction strategies on Balanced AudioSet (merged pairs

r=90

). Dimension-wise max, mean, and sum merging are compared with random drop.

Panel of Figure 2 — Figure 2: Comparison of spectrogram masking. (a) SpecAugment masks continuous frequency/time frames, so when the spectrogram is later split into 16×16 patches (stride = 10), a patch can contain both masked and unmasked regions. (b) SpecAugment-Patch masks along the patch grid, ensuring each patch is either fully masked or fully retained. The bottom row shows the 16×16 patches from the red-boxed area above.

AST (1212 Patches)				Merge rate(%)	PaSST-U (1190 Patches)
mAP	S/s	Mem (GB)	r(pairs)		r(pairs)	mAP	S/s	Mem (GB)
34.07 $\pm$ 0.18	43.3	24.64 ( 0.00 GB)	0	0%	0	29.43 $\pm$ 0.14	42.7	24.13 ( 0.00 GB)
34.12 $\pm$ 0.21	45.3	22.33 ( -2.31 GB)	30	5%	29	29.47 $\pm$ 0.15	43.3	23.11 ( -1.02 GB)
34.04 $\pm$ 0.38	46.2	22.79 ( -1.85 GB)	60	10%	59	29.62 $\pm$ 0.06	44.5	22.31 ( -1.82 GB)
34.10 $\pm$ 0.31	48.6	21.85 ( -2.79 GB)	90	15%	88	29.27 $\pm$ 0.21	46.2	21.29 ( -2.84 GB)
34.08 $\pm$ 0.24	49.3	21.50 ( -3.14 GB)	100	16.5%	98	29.58 $\pm$ 0.27	46.7	21.00 ( -3.13 GB)

Table 3: Results on Balanced AudioSet. We report mean ± standard deviation of mAP, training throughput S/s (samples/sec), and peak GPU memory (GB). Due to different numbers of patch tokens in AST and PaSST-U, we match the merging rate (%) and report the corresponding number of merged pairs r. Memory reductions relative to the 0% merging baseline are shown in parentheses.

Dataset	r	Acc	S/s
ESC-50	0	89.20 $\pm$ 0.43	127.3
	10	88.96 $\pm$ 0.45	129.3
	20	89.00 $\pm$ 0.62	130.8
	30	88.96 $\pm$ 0.23	134.6
	40	88.80 $\pm$ 0.17	138.9
	50	88.67 $\pm$ 0.21	142.9

Table 4: Results on ESC-50 and Speech Commands V2.

Token Merging

Token merging combines redundant transformer tokens to reduce sequence length, attention costs, and inference time while preserving task-relevant information.

See all 23 papers

LAST: Looped Audio Spectrogram Transformer
Oct 1, 2026Haider Al-Tahan, Sean O'Brien, Anastasia Razdaibiedina +1Audio Understanding Recurrent Transformers
Georgia Institute of Technology · University of California San Diego · Google DeepMind
STFT Frontend Mismatch in Pretrained Audio Spectrogram Transformers: Analysis and Adaptation
Oct 25, 2026Julia Wilkins, Wei-Ching Lin, Abinaya Kumar +2Audio Classification Transformer
New York University, USA · Bosch Research, USA
U-PAST: A Phase-Aware Audio Spectrogram Transformer-U-Net for Single-Channel Speech Enhancement
Aug 31, 2026Cao Duong Ly, Jörn AnemüllerHybrid CNN-Transformer Architectures Speech Enhancement
Department of Medical Physics and Acoustics Carl von Ossietzky University Oldenburg Oldenburg, Germany 26129

Oct 1, 2026J/K move · Enter open · S save

LAST: Looped Audio Spectrogram Transformer

Haider Al-Tahan, Sean O'Brien, Anastasia Razdaibiedina, N. Apurva Ratan Murty

Georgia Institute of Technology · University of California San Diego · Google DeepMind

Audio Understanding Recurrent Transformers Audio Classification

Open paper AskPDF arXiv

Increasing depth of transformer models improves recognition, but it comes at a substantial cost. Each additional layer requires more parameters, which makes the process computationally inefficient. We ask whether additional processing can focus on integrating features already computed. Looped Audio Spectrogram Transformer (LAST) first processes all tokens, then reuses the same blocks to refine only the class token over fixed audio features, thereby making later passes inexpensive. On AudioSet, ten-pass LAST achieves 0.345 mean average precision, exceeding a twelve-layer sequential transformer by 2.1% relative with 49.4% fewer parameters, 42% fewer multiply-accumulate operations, and 9.8% higher measured throughput. Across separately trained models, increasing the pass count from two to ten improves accuracy while adding only 1.2% computation. Further evaluations show improved robustness to temporal masking and various other auditory augmentations, with better generalization on classification tasks with music, environmental, and event sounds.

Abstract

Token Merging

Explore similar work

LAST: Looped Audio Spectrogram Transformer

STFT Frontend Mismatch in Pretrained Audio Spectrogram Transformers: Analysis and Adaptation

U-PAST: A Phase-Aware Audio Spectrogram Transformer-U-Net for Single-Channel Speech Enhancement