eess.ASFeb 17, 2026

What Do Neurons Listen To? A Neuron-level Dissection of a General-purpose Audio Model

Authors: Takao Kawamura, Daisuke Niizumi, Nobutaka Ono

Organizations: Graduate School of Systems Design, Tokyo Metropolitan University, Tokyo, Japan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

In this paper, we analyze the internal representations of a general-purpose audio self-supervised learning (SSL) model from a neuron-level perspective. Despite their strong empirical performance as feature extractors, the internal mechanisms underlying the robust generalization of SSL audio models remain unclear. Drawing on the framework of mechanistic interpretability, we identify and examine class-specific neurons by analyzing conditional activation patterns across diverse tasks. Our analysis reveals that SSL models foster the emergence of class-specific neurons that provide extensive coverage across novel task classes. These neurons exhibit shared responses across different semantic categories and acoustic similarities, such as speech attributes and musical pitch. We also confirm that these neurons have a functional impact on classification performance. To our knowledge, this is the first systematic neuron-level analysis of a general-purpose audio SSL model, providing new insights into its internal representation.

Figures & tables

 Mean class-specific neuronsClass coverage ratio
CREMA-D172.0100%
ESC-5037.998%
GISE-5142.3100%
GTZAN70.2100%
NSynth98.5100%
Surge164.9100%
TABLE I: Statistics of Class-specific Neurons (SSL). SSL achieves a class coverage ratio close to 100%, indicating that class-specific neurons span nearly all classes.
 Mean class-specific neuronsClass coverage ratio
CREMA-D98.7100%
ESC5024.794%
GISE-5123.480%
GTZAN48.175%
NSynth49.7100%
Surge81.7100%
TABLE II: Statistics of Class-specific Neurons (SL). SL covers substantially fewer classes (e.g., 49% in VC1), indicating that their neurons exhibit limited class-level coverage.
Fig. 1: Shared neurons between VC1 and CREMA-D under gender-based class definitions. SSL demonstrates clear cross-task sharing of neurons aligned with gender, whereas SL exhibits negligible sharing.
Fig. 2: Common neuron ratios for each octave class in NSynth and Surge. Despite differences in synthesis methods and dataset characteristics, octave-specific neurons are consistently observed across both tasks.
Fig. 3: Common neuron ratios for semantically overlapping event classes across ESC-50 and GISE-51. Compared with Fig. 1 and Fig. 2 , the ratios are generally lower.
Fig. 4: Common neuron ratios across genre classes in GTZAN. “Classical” and “jazz” share a relatively large number of neurons, exhibiting a distinct sharing pattern compared with other genres.
Fig. 5: Common neuron ratios across language classes in VoxForge. SSL results exhibit relatively high neuron sharing within Germanic (“de,” “en”) and Romance (“es,” “fr,” “it”) language families.
Fig. 6: Common neuron ratios across emotion classes in CREMA-D. Relatively higher neuron sharing is observed between “ANG” and “HAP,” as well as among “DIS,” “FEA,” “NEU,” and “SAD.”
Fig. 7: Ablation impact on GTZAN genre classification. Left: Original model. Middle/Right: Deviations after ablating class-specific vs. random neurons. The results indicate that class-specific neurons have a greater functional impact on classification than randomly selected ones.
Fig. 8: Ablation impact on CREMA-D genre classification. Left: Original model. Middle/Right: Deviations after ablating class-specific vs. random neurons. The results indicate that class-specific neurons have a clearer targeted impact on classification compared to the random baseline.
More in this topic

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 211 papers

Explore similar work

CardsList
Feb 24, 2026

AR&D: A Framework for Retrieving and Describing Concepts for Interpreting AudioLLMs

Despite strong performance in audio perception tasks, large audio-language models (AudioLLMs) remain opaque to interpretation. A major factor behind this lack of interpretability is that individual neurons in these models frequently activate in response to several unrelated concepts. We introduce the first mechanistic interpretability framework for AudioLLMs, leveraging sparse autoencoders (SAEs) to disentangle polysemantic activations into monosemantic features. Our pipeline identifies representative audio clips, assigns meaningful names via automated captioning, and validates concepts through human evaluation and steering. Experiments show that AudioLLMs encode structured and interpretable features, enhancing transparency and control. This work provides a foundation for trustworthy deployment in high-stakes domains and enables future extensions to larger models, multilingual audio, and more fine-grained paralinguistic features. Project URL: https://townim-faisal.github.io/AutoInterpret-AudioLLM/

Townim Faisal Chowdhury, Ta Duc Huy, Siqi Pan +2

Australian Institute for Machine Learning, University of Adelaide · Dolby Laboratories · School of Computer and Mathematical Sciences, University of Adelaide, Australia

Multimodal Model InterpretabilityMechanistic InterpretabilityRepresentation Disentanglement
AskPDF
Jul 7, 2026

InsideSSL: Understanding Self-Supervised Speech Representations using a Model-Centric Perspective

Self-supervised learning (SSL) models, such as Wav2Vec2, HuBERT, and WavLM, have become foundational across a wide range of speech and audio tasks. Despite their success, understanding their internal layer-wise dynamics remains an ongoing challenge. To address this, we propose a two-part model-centric framework called InsideSSL. First, we establish a task-agnostic analysis from three intrinsic per-layer perspectives: compression (entropy), geometry (curvature), and robustness to perturbations. We show that varying training objectives induce distinct regimes of acoustic compression and manifold unfolding. Second, we introduce the cross-layer Generative Compatibility Matrix (GCM) to evaluate functional transferability, exposing stable phonetic cores, identity volatility, and deep-layer semantic pruning. In addition to these evaluations, linear probing connects the model-centric perspective to downstream tasks, demonstrating how layer topology dictates phoneme, pitch, and speaker encoding.

Samir Sadok, Xavier Alameda-Pineda

Inria, Universit´e Grenoble Alpes CNRS, LJK, France

Representation GeometrySelf-Supervised Speech Representation Learning
AskPDF
Oct 20, 2026

Uncovering Audio-Text Alignment in Large Audio-Language Models through Cross-Modal Concept Neurons

While the language model backbone is pretrained on text alone, large audio-language models (LALMs) can, to a great extent, recognize acoustic attributes, such as emotion, speaker gender, and spoken language. However, it is unclear how the audio representation is integrated, if at all, with the other text representation before the model produces text. We hypothesize that there is a shared subspace where the audio and text representations are aligned. To test this hypothesis, we identify cross-modal neurons that are responsible for representing both audio and text. We find alignment between audio and text neurons to various degrees across three LALMs and four acoustic attribute types. Causal interventions, steering and deactivating the neurons, confirm that the cross-modal neurons are not just necessary for prediction but can also be manipulated.

Tzu-Quan Lin, Chih-Kai Yang, Yi-Cheng Lin +2

Centre for Speech Technology Research, University of Edinburgh, United Kingdom · Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan

Audio-Text AlignmentAudio-Language ModelsCross-Modal Representation LearningMechanistic Interpretability
AskPDF