Spectrogram features for audio and speech analysis

Ian McLoughlin; Lam Pham; Yan Song; Xiaoxiao Miao; Huy Phan; Pengfei Cai; Qing Gu; Jiang Nan

eess.ASMar 16, 2026

Spectrogram features for audio and speech analysis

Authors: Ian McLoughlin, Lam Pham, Yan Song, Xiaoxiao Miao, Huy Phan, Pengfei Cai, Qing Gu, Jiang Nan,

Organizations: Singapore Institute of Technology · Austrian Institute of Technology · The University of Science and Technology of China · Meta Inc., Reality Labs

Abstract

Spectrogram-based representations have grown to dominate the feature space for deep learning audio analysis systems, and are often adopted for speech analysis also. Initially, the primary motivator for spectrogram-based representations was their ability to present sound as a two dimensional signal in the time-frequency plane, which not only provides an interpretable physical basis for analysing sound, but also unlocks the use of a wide range of machine learning techniques such as convolutional neural networks, that had been developed for image processing. A spectrogram is a matrix characterised by the resolution and span of its two dimensions, as well as by the representation and scaling of each element. Many possibilities for these three characteristics have been explored by researchers across numerous application areas, with different settings showing affinity for various tasks. This paper reviews the use of spectrogram-based representations and surveys the state-of-the-art to question how front-end feature representation choice allies with back-end classifier architecture for different tasks.

Figures & tables

Figure 1: Illustration of spectrogram creation from input audio data, as a stack of frequency vectors.

Description	Dimensions	Element scale	Frequency span
Linear spectrogram (LS)	time, frequency (T, F)	scalar $(0,1)$	$(0,$ Nyquist $)$
Log-scaled spectrogram (LSS)	T, F	log $(-100$ dB $,0)$	$(0,$ Nyquist $)$
A/ $\mu$ -law scaling	T, F	log $(0,255)$	$(0,$ Nyquist $)$
Mel-spectrogram (MS)	T, Mel-F	linear	$(0,$ Mel(Nyquist) $)$
Log-Mel-spectrogram (LMS)	T, Mel-F	log $(-100$ dB $,0)$	$(0,$ Mel(Nyquist) $)$
Gammatonegram (GTG)	trapeziodal/squared-T, F	log	$($ ERB(0), ERB(Nyquist) $)$

Table 1: Taxonomy of spectrograms.

Figure 2: Illustrations of two dimensional time-frequency spectrograms based on (a) stabilised auditory image, (b) Constant-Q transform, (c) Mel-scaled spectrogram, (d) stacked MFCC, (e) Linear magnitude spectrogram.

Figure 3: High level system diagram showing spectrogram features (a) being extracted from an input waveform as a stack of scaled transforms from windowed speech regions then (b) features gathered from patches, pooled regions or a downsampled spectrogram image for (c) input to a deep learning classification pipeline.

Task	Details	Fixed pooling	VNFs
SED	50 class RWCP, 20dB SNR	94.8% accuracy	96.3% accuracy
SED	50 class RWCP, 0dB SNR	75.1% accuracy	84.0% accuracy
LID	NIST LRE07 DNN x-vector 3s	10.17 $C_{avg}$	8.80 $C_{avg}$
LID	NIST LRE07 CLSTM 3s	7.15 $C_{avg}$	6.70 $C_{avg}$
DID	Arabic dialect challenge	3.20 $C_{avg}$	2.62 $C_{avg}$

Table 2: Performance of variance normalised features (VNF) compared to standard pooling for three tasks, aiming for higher accuracy and lower C avg .

Input	Feature extraction	Stack & Classify	Output
continuous,	raw waveform/spectrum,		one-hot class per instance,
full recording,	named features,		posterior probabilities,
utterance,	trained feature extractor or		vote over multiple instances,
segment or	pre-trained feature extractor		average/threshold over time or
frame			localisation in time

Table 3: Taxonomy of audio analysis. The input length (first column) can vary from a short frame to a continuous signal, one or more features extracted from this, and then am output class, or timing, obtained.

Year	Ref.	Task	Spectrogram type	Resolution & settings	Pooling
2014	( McLoughlin et al., 2015 )	50 class RWCP	LS	$[30{\times}24]$ , 16kHz	vote
2017	( Adavanne and Virtanen, 2017 )	6 class TUT events Heittola and Mesaros (2017) 1	LMS+LS	$[240{\times}256]$ , 44.1kHz	max
2019	( Lin and Wang, 2019 )	10 class from Gemmeke et al. (2017) + Fonseca et al. (2017)	LMS	$[64{\times}500]$ , 16kHz	median
2020	( McLoughlin et al., 2020 )	50 class RWCP	LS,GTG,CQT	$[52{\times}40]$ , 16kHz	mean
2022	( Ebbers and Haeb-Umbach, 2022 )	10 class DESED Turpault et al. (2019) 2	LMS	$[128{\times}960]$ , 16kHz	mean
2023	( Kim et al., 2023 )	10 class DESED Turpault et al. (2019) 2	MS	$[128{\times}1001]$ , 16kHz	mean

Table 4: Prominent sound event detection methods that utilise spectrogram features, including linear (LS), log-mel (LMS), mel scale (MS), constant-Q (CQT) and gammatonegram (GTG). Tasks include the Real World Computing Partnership (RWCP) sounds, the TUT sound events database, and Domestic Environment Sound Event Detection Dataset (DESED).

Method	Spectrogram	Pixels	Sample Rate	Scale
Chakrabarty et al. Chakrabarty and Elhilali (2016)	LMS	128 $\times$ T	8kHz	log
Zeng et al. Zeng et al. (2023)	LMS	128 $\times$ T	16kHz	inverted log
Li et al. Li et al. (2024)	LNS	128 $\times$ T	16kHz	log
Liu et al. Liu et al. (2022)	LMS+Tgram	128 $\times$ T	16kHz	log
Yin et al. Yin et al. (2025)	LMS	128 $\times$ T	16kHz	normalised

Table 5: Spectrograms used in ASD, primarily log-Mel (LMS, their resolution, sample rate and scaling.).

Technique	Input Type	Task(s)	Taxa
SincNet Ravanelli and Bengio (2018)	Raw waveform	Species classification	Birds
SampleCNN Lee et al. (2017)	Raw waveform	Music auto-tagging	Music
RawNet Jung et al. (2019)	Raw waveform	Speaker verification	Humans
CNN/ResNet Stowell (2022) ; Bravo Sánchez et al. (2021)	LMS	Species classification, SED	Birds, Frogs, Whales
PCEN-enhanced CNN Wang et al. (2017) ; Zeghidour et al. (2021) ; Allen et al. (2021)	PCEN-Mel	Low-SNR event detection	Birds, Whales
CNN on STFT Hexeberg et al. (2023)	LS	Call segmentation	Bats

Table 6: Deep learning front-ends in bioacoustic analysis that make use of raw waveforms (top three), linear and log-Mel spectrograms (LS, LMS) and stabilized auditory image (SAI) (middle four) and hybrid approaches (bottom two).

Method	Spectrogram	Resolution	Task
Ma et al. Jin et al. (2018)	PLP + bottleneck	48 $\times$ 21	23 languages, EER 4.38% 1
Kaiyr et al. Kaiyr et al. (2021)	LS, CNN-RNN	116 $\times$ 200 5-10s segments	7 languages, acc. 94.3% 2
Liu et al. Liu et al. (2022)	MFCC+delta+delta-delta	39+39+39, 25ms window, 10ms hop	14 languages, EER 3.82% 3
Miao et al. Miao et al. (2021)	MFCC+D-MONA	23 $\times$ 5 frames	14 languages, EER 1.15% 3
Tjanda et al. Tjandra et al. (2022)	LMS	80 $\times$ 4, 25ms window, 10ms hop	26 languages, acc. 90.3% 4
1 Evaluated on 10s utterances using NIST LRE2009.

Table 7: Prominent LID research showing various kinds of spectrogram, including linear (LS) and log-Mel spectrograms (LMS). EER is equal error rate.

Method	Spectrogram	Resolution
Reynolds et al. Reynolds et al. (2000)	MFCC+context	13 $\times$ T
Dehak et al. Dehak et al. (2011)	i-vector from 60d MFCC	200 WCCN
Variani et al. Variani et al. (2014)	trained from 40d FB+context	256 d-vector
Snyder et al. Snyder et al. (2018)	60d MFCC+delta+delta-delta	150 x-vector
Desplanques et al. Desplanques et al. (2020)	LMS	80 $\times$ 80
Liu et al. Liu et al. (2024)	LMS	128 $\times$ 304

Table 8: Prominent SV research showing a progression of spectrogram use, where LMS refers to log-Mel spectrogram and FB are filterbanks.

Method	Feature Type	Representation
Schuller et al. Schuller et al. (2011)	MFCC + prosody + energy	1582-d openSMILE
Satt et al. Satt et al. (2017)	LMS	$\sim$ 40–64 Mel bands, 25ms window, 10ms hop
Mirsamadi et al. Mirsamadi et al. (2017)	FB with frame attention	40dim FB, 25ms window, 10ms hop
Trigeorgis et al. Trigeorgis et al. (2016)	LMS + channel attention	40dim FB with 40ms frame, 5ms hop
Pepino et al. Pepino et al. (2021)	wav2vec 2.0 SSL embedding	768dim contextual frames
Chen et al. Chen et al. (2022)	WavLM SSL embedding	1024dim contextual frames

Table 9: Representative SER systems illustrating the evolution from hand-crafted features to high-resolution spectrograms and SSL-based embeddings. FB refers to filterbank, LMS is log-mel spectrogram.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

STFT Frontend Mismatch in Pretrained Audio Spectrogram Transformers: Analysis and Adaptation
Oct 25, 2026Julia Wilkins, Wei-Ching Lin, Abinaya Kumar +2Audio Classification Transformer
New York University, USA · Bosch Research, USA
Exploring the Design Space of Representation Learning for Audio Transformations
Aug 28, 2026Sungho Lee, Marco Martínez-Ramírez, Junghyun Koo +3Audio Representation Learning Equivariant Representation Learning
Department of Intelligence and Information, Seoul National University, Seoul, South Korea · Sony AI, Tokyo, Japan · Sony Group Corporation, Tokyo, Japan
LAST: Looped Audio Spectrogram Transformer
Oct 1, 2026Haider Al-Tahan, Sean O'Brien, Anastasia Razdaibiedina +1Audio Understanding Recurrent Transformers
Georgia Institute of Technology · University of California San Diego · Google DeepMind

Oct 25, 2026J/K move · Enter open · S save

STFT Frontend Mismatch in Pretrained Audio Spectrogram Transformers: Analysis and Adaptation

Julia Wilkins, Wei-Ching Lin, Abinaya Kumar, Shabnam Ghaffarzadegan+1

New York University, USA · Bosch Research, USA

Audio Classification Transformer Frequency-Domain Feature Learning

Open paper AskPDF arXiv

Pretrained audio spectrogram transformers are widely used as backbones of machine listening systems for sound event classification. However, they are typically trained using spectrograms with a fixed Short-Time Fourier Transform (STFT) configuration, implicitly binding the model to the time-frequency resolution seen during pretraining. In practice, adapting these systems to new domains or computational constraints often requires different STFT frontend parameters. In this work, we conduct a controlled study of the sensitivity of the foundational Audio Spectrogram Transformer to variations in STFT window and hop size, showing that audio classification performance degrades substantially as frontend parameters deviate from pretraining settings. As a mitigation strategy, we introduce a lightweight conditioning mechanism using FiLM that enables a pretrained model to adapt to variable input configurations using limited additional training data. Our method generalizes well to unseen STFT configurations, improving model robustness without sacrificing performance in the original setting.

Abstract

Audio Representation Learning

Explore similar work

STFT Frontend Mismatch in Pretrained Audio Spectrogram Transformers: Analysis and Adaptation

Exploring the Design Space of Representation Learning for Audio Transformations

LAST: Looped Audio Spectrogram Transformer