In predictive modeling for low-resource audio classification, extracting high-accuracy and interpretable attributes is critical. Particularly in high-reliability applications, interpretable audio attributes are indispensable. While human-driven attribute discovery is effective, its low throughput becomes a bottleneck. We propose a method for adaptively discovering interpretable audio attributes using Multimodal Large Language Models (MLLMs). By replacing humans in the AdaFlock framework with MLLMs, our method achieves significantly faster attribute discovery. Our method dynamically identifies salient acoustic characteristics via prompting and constructs an attribute-based ensemble classifier. Experimental results across various audio tasks demonstrate that our method outperforms direct MLLM prediction in the majority of evaluated cases. The entire training completes within 11 minutes, proving it a practical, adaptive solution that surpasses conventional human-reliant approaches.
Figures & tables
Figure 1: An iterative framework for MLLM-based attribute discovery and weak classifier training. Sampling weights are adaptively updated based on prediction results to discover attributes and refine model performance.
Baselines
Proposed
Dataset
LR
MLLM Pred.
Ours
Δ
CREMA-D
70.00
69.00 ± 2.32
72.45 ± 6.10
+3.45
RAVDESS
54.00
66.60 ± 1.82
68.55 ± 2.57
+1.95
Coswara
67.50
48.10 ± 2.00
55.70 ± 2.20
+7.60
ESC-50
94.00
88.35 ± 0.88
87.15 ± 1.49
-1.20
Table 1: Comparison of the proposed method against two baselines (MLLM prediction and LR). Values represent mean accuracy ± standard deviation (%), except for the LR baseline which is reported as a single deterministic score. Δ indicates the absolute improvement of Ours over the MLLM Prediction baseline. Bold indicates the best performance for each dataset.
Dataset
Attributes
CREMA-D
Is the speaker’s tone generally positive or upbeat?; Does the speaker sound relaxed rather than intense or secretive?; Does the speaker’s voice sound cheerful?;
RAVDESS
Does the speaker’s tone suggest more of a positive-neutral emotion, as opposed to a negative-neutral emotion?; Does the speaker sound excited?; Does the speaker’s voice exhibit signs of distress or agitation?;
Coswara
Is the cough followed by a distinct audible breath intake?; Does the cough appear to be prolonged or linger after it starts?; Does the frequency of coughing increase over the duration of the sample?;
ESC-50
Does the environment sound windy or breezy?; Is there a presence of moving water in the background?; Does the sound contain high-pitched elements like splashes or drips?;
Table 2: Top 3 attributes discovered by gemini-3-f-p for each dataset, listed in descending order of importance.
Attribute Definition Model ( Mdef )
Dataset
gpt-audio-m
gpt-audio
gemini-3-f-p
Δmax
CREMA-D
73.55 ± 4.05
75.30 ± 3.94
72.45 ± 6.10
2.88
RAVDESS
68.65 ± 3.81
66.90 ± 1.63
68.55 ± 2.57
1.75
Coswara
54.20 ± 1.84
53.35 ± 3.13
55.70 ± 2.20
2.35
ESC-50
87.25 ± 1.99
88.10 ± 2.07
87.15 ± 1.49
0.95
Table 3: Impact of Different Attribute Definition Models ( Mdef ) on Accuracy. Values represent mean accuracy ± standard deviation (%). Δmax denotes the difference between the highest and lowest mean accuracy among Mdef models. Bold values indicate the best performance for each dataset.
Attribute Definition Model ( Mdef )
Dataset
gpt-audio
gpt-audio-m
Gemini-3-f-p
CREMA-D
7.89 ± 1.00
7.72 ± 1.11
8.57 ± 0.81
RAVDESS
9.87 ± 1.56
9.74 ± 1.81
8.84 ± 1.35
Coswara
10.53 ± 1.54
10.30 ± 1.78
8.43 ± 0.84
ESC-50
9.63 ± 1.02
9.50 ± 0.68
9.11 ± 0.92
Table 4: Training time [min] (mean ± standard deviation) per dataset across different Mdef . Mlab is fixed to gemini-3-f-p .
Despite strong performance in audio perception tasks, large audio-language models (AudioLLMs) remain opaque to interpretation. A major factor behind this lack of interpretability is that individual neurons in these models frequently activate in response to several unrelated concepts. We introduce the first mechanistic interpretability framework for AudioLLMs, leveraging sparse autoencoders (SAEs) to disentangle polysemantic activations into monosemantic features. Our pipeline identifies representative audio clips, assigns meaningful names via automated captioning, and validates concepts through human evaluation and steering. Experiments show that AudioLLMs encode structured and interpretable features, enhancing transparency and control. This work provides a foundation for trustworthy deployment in high-stakes domains and enables future extensions to larger models, multilingual audio, and more fine-grained paralinguistic features. Project URL: https://townim-faisal.github.io/AutoInterpret-AudioLLM/
Townim Faisal Chowdhury, Ta Duc Huy, Siqi Pan +2
Australian Institute for Machine Learning, University of Adelaide · Dolby Laboratories · School of Computer and Mathematical Sciences, University of Adelaide, Australia
Manual annotation and clustering of audio datasets is labour intensive. We introduce AuditoryHuM, a training-free framework for the unsupervised discovery and clustering of auditory scene labels using human-Multimodal Large Language Model (MLLM) collaboration. Leveraging MLLMs, our framework generates contextually relevant labels for audio data. To ensure label quality and mitigate hallucinations, zero-shot learning (Human-CLAP) quantifies the alignment between generated text labels and raw audio. A targeted human-in-the-loop intervention, refines only the lowest aligned pairs. The discovered labels form an interpretable alignment vector to group audio into cohesive clusters. The framework was evaluated across three auditory scene datasets (ADVANCE, AHEAD-DS, and TAU 2019), achieving a 96.1% reduction in human labour for ADVANCE during testing. Downstream models trained on our clusters exhibit enhanced classification accuracy vs the baseline (0.79 to 0.84) in clean acoustic environments, though performance scales down in dense, complex soundscapes. The project page: https://github.com/Australian-Future-Hearing-Initiative
Henry Zhong, Jörg M. Buchholz, Julian Maclaren +2
Australian Hearing Hub, Macquarie University, Sydney, Australia · Google Research Australia, Sydney, Australia
Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a ``listening'' signal. We show that this signal increases when audio evidence affects the model's output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio--silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model's audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates.
Neta Glazer, Lenny Aharon, Ethan Fetaya
Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA