Adaptive Discovery of Interpretable Audio Attributes with Multimodal LLMs for Low-Resource Classification

Kosuke Yoshimura; Hisashi Kashima

cs.SDMar 7, 2026

Adaptive Discovery of Interpretable Audio Attributes with Multimodal LLMs for Low-Resource Classification

Authors: Kosuke Yoshimura, Hisashi Kashima

Organizations: Kyoto University, Kyoto, Japan

Abstract

In predictive modeling for low-resource audio classification, extracting high-accuracy and interpretable attributes is critical. Particularly in high-reliability applications, interpretable audio attributes are indispensable. While human-driven attribute discovery is effective, its low throughput becomes a bottleneck. We propose a method for adaptively discovering interpretable audio attributes using Multimodal Large Language Models (MLLMs). By replacing humans in the AdaFlock framework with MLLMs, our method achieves significantly faster attribute discovery. Our method dynamically identifies salient acoustic characteristics via prompting and constructs an attribute-based ensemble classifier. Experimental results across various audio tasks demonstrate that our method outperforms direct MLLM prediction in the majority of evaluated cases. The entire training completes within 11 minutes, proving it a practical, adaptive solution that surpasses conventional human-reliant approaches.

Figures & tables

Figure 1: An iterative framework for MLLM-based attribute discovery and weak classifier training. Sampling weights are adaptively updated based on prediction results to discover attributes and refine model performance.

	Baselines		Proposed
Dataset	LR	MLLM Pred.	Ours	$\Delta$
CREMA-D	70.00	69.00 $\pm$ 2.32	72.45 $\pm$ 6.10	+3.45
RAVDESS	54.00	66.60 $\pm$ 1.82	68.55 $\pm$ 2.57	+1.95
Coswara	67.50	48.10 $\pm$ 2.00	55.70 $\pm$ 2.20	+7.60
ESC-50	94.00	88.35 $\pm$ 0.88	87.15 $\pm$ 1.49	-1.20

Table 1: Comparison of the proposed method against two baselines (MLLM prediction and LR). Values represent mean accuracy

\pm

standard deviation (%), except for the LR baseline which is reported as a single deterministic score.

\Delta

indicates the absolute improvement of Ours over the MLLM Prediction baseline. Bold indicates the best performance for each dataset.

Dataset	Attributes
CREMA-D	Is the speaker’s tone generally positive or upbeat?; Does the speaker sound relaxed rather than intense or secretive?; Does the speaker’s voice sound cheerful?;
RAVDESS	Does the speaker’s tone suggest more of a positive-neutral emotion, as opposed to a negative-neutral emotion?; Does the speaker sound excited?; Does the speaker’s voice exhibit signs of distress or agitation?;
Coswara	Is the cough followed by a distinct audible breath intake?; Does the cough appear to be prolonged or linger after it starts?; Does the frequency of coughing increase over the duration of the sample?;
ESC-50	Does the environment sound windy or breezy?; Is there a presence of moving water in the background?; Does the sound contain high-pitched elements like splashes or drips?;

Table 2: Top 3 attributes discovered by gemini-3-f-p for each dataset, listed in descending order of importance.

Dataset	gpt-audio-m	gpt-audio	gemini-3-f-p	$\Delta_{max}$
	Attribute Definition Model ( $\mathcal{M}_{def}$ )
CREMA-D	73.55 $\pm$ 4.05	75.30 $\pm$ 3.94	72.45 $\pm$ 6.10	2.88
RAVDESS	68.65 $\pm$ 3.81	66.90 $\pm$ 1.63	68.55 $\pm$ 2.57	1.75
Coswara	54.20 $\pm$ 1.84	53.35 $\pm$ 3.13	55.70 $\pm$ 2.20	2.35
ESC-50	87.25 $\pm$ 1.99	88.10 $\pm$ 2.07	87.15 $\pm$ 1.49	0.95

Table 3: Impact of Different Attribute Definition Models (

\mathcal{M}_{def}

) on Accuracy. Values represent mean accuracy

\pm

standard deviation (%).

\Delta_{max}

denotes the difference between the highest and lowest mean accuracy among

\mathcal{M}_{def}

models. Bold values indicate the best performance for each dataset.

	Attribute Definition Model ( $\mathcal{M}_{def}$ )
Dataset	gpt-audio	gpt-audio-m	Gemini-3-f-p
CREMA-D	7.89 $\pm$ 1.00	7.72 $\pm$ 1.11	8.57 $\pm$ 0.81
RAVDESS	9.87 $\pm$ 1.56	9.74 $\pm$ 1.81	8.84 $\pm$ 1.35
Coswara	10.53 $\pm$ 1.54	10.30 $\pm$ 1.78	8.43 $\pm$ 0.84
ESC-50	9.63 $\pm$ 1.02	9.50 $\pm$ 0.68	9.11 $\pm$ 0.92

Table 4: Training time [min] (mean

\pm

standard deviation) per dataset across different

\mathcal{M}_{def}

\mathcal{M}_{lab}

is fixed to gemini-3-f-p .

Audio Classification

Audio classification assigns recordings to categories such as speech, music, animal calls, acoustic scenes, and health-related events.

See all 147 papers

AR&D: A Framework for Retrieving and Describing Concepts for Interpreting AudioLLMs
Feb 24, 2026Townim Faisal Chowdhury, Ta Duc Huy, Siqi Pan +2Multimodal Model Interpretability Mechanistic Interpretability
Australian Institute for Machine Learning, University of Adelaide · Dolby Laboratories · School of Computer and Mathematical Sciences, University of Adelaide, Australia
AuditoryHuM: Auditory Scene Label Generation and Clustering using Human-MLLM Collaboration
Feb 23, 2026Henry Zhong, Jörg M. Buchholz, Julian Maclaren +2Clustering LLM-Assisted Annotation
Australian Hearing Hub, Macquarie University, Sydney, Australia · Google Research Australia, Sydney, Australia
Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering
Mar 6, 2026Neta Glazer, Lenny Aharon, Ethan FetayaAudio-Language Models Audio-Language Model Evaluation
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA

Feb 24, 2026J/K move · Enter open · S save

AR&D: A Framework for Retrieving and Describing Concepts for Interpreting AudioLLMs

Townim Faisal Chowdhury, Ta Duc Huy, Siqi Pan, Jeremy Stoddard+1

Australian Institute for Machine Learning, University of Adelaide · Dolby Laboratories · School of Computer and Mathematical Sciences, University of Adelaide, Australia

Multimodal Model Interpretability Mechanistic Interpretability Representation Disentanglement

Open paper AskPDF arXiv

Despite strong performance in audio perception tasks, large audio-language models (AudioLLMs) remain opaque to interpretation. A major factor behind this lack of interpretability is that individual neurons in these models frequently activate in response to several unrelated concepts. We introduce the first mechanistic interpretability framework for AudioLLMs, leveraging sparse autoencoders (SAEs) to disentangle polysemantic activations into monosemantic features. Our pipeline identifies representative audio clips, assigns meaningful names via automated captioning, and validates concepts through human evaluation and steering. Experiments show that AudioLLMs encode structured and interpretable features, enhancing transparency and control. This work provides a foundation for trustworthy deployment in high-stakes domains and enables future extensions to larger models, multilingual audio, and more fine-grained paralinguistic features. Project URL: https://townim-faisal.github.io/AutoInterpret-AudioLLM/

Abstract

Audio Classification

Explore similar work

AR&D: A Framework for Retrieving and Describing Concepts for Interpreting AudioLLMs

AuditoryHuM: Auditory Scene Label Generation and Clustering using Human-MLLM Collaboration

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering