AuditoryHuM: Auditory Scene Label Generation and Clustering using Human-MLLM Collaboration

Henry Zhong; Jörg M. Buchholz; Julian Maclaren; Simon Carlile; Richard F. Lyon

cs.SDFeb 23, 2026

AuditoryHuM: Auditory Scene Label Generation and Clustering using Human-MLLM Collaboration

Authors: Henry Zhong, Jörg M. Buchholz, Julian Maclaren, Simon Carlile, Richard F. Lyon

Organizations: Australian Hearing Hub, Macquarie University, Sydney, Australia · Google Research Australia, Sydney, Australia

Abstract

Manual annotation and clustering of audio datasets is labour intensive. We introduce AuditoryHuM, a training-free framework for the unsupervised discovery and clustering of auditory scene labels using human-Multimodal Large Language Model (MLLM) collaboration. Leveraging MLLMs, our framework generates contextually relevant labels for audio data. To ensure label quality and mitigate hallucinations, zero-shot learning (Human-CLAP) quantifies the alignment between generated text labels and raw audio. A targeted human-in-the-loop intervention, refines only the lowest aligned pairs. The discovered labels form an interpretable alignment vector to group audio into cohesive clusters. The framework was evaluated across three auditory scene datasets (ADVANCE, AHEAD-DS, and TAU 2019), achieving a 96.1% reduction in human labour for ADVANCE during testing. Downstream models trained on our clusters exhibit enhanced classification accuracy vs the baseline (0.79 to 0.84) in clean acoustic environments, though performance scales down in dense, complex soundscapes. The project page: https://github.com/Australian-Future-Hearing-Initiative

Figures & tables

Figure 1 : The dataflow diagram of the AuditoryHuM processes, showing the steps from MLLM label discovery, human-in-the-loop refinement, cleanup, alignment, and clustering.

$\mu_{c}$	Gemma 3N E2B	Qwen 2 Audio 7B	Qwen 2.5 Omni 3B
ADVANCE	0.51	0.56	0.61
AHEAD-DS	0.42	0.51	0.55
TAU 2019	0.48	0.48	0.49

Table 1 : The

\mu_{c}

values for each dataset and MLLM pair, rounded to two decimal places.

\mu_{c}

ranges from 1 to

-1

, a value of 1 represents perfect alignment and

-1

is perfect misalignment. A higher value is better.

$\mu_{1\%}$	Gemma 3N E2B	Qwen 2 Audio 7B	Qwen 2.5 Omni 3B
ADVANCE	-0.06	-0.14	0.13
AHEAD-DS	-0.19	-0.26	0.07
TAU 2019	0.07	0.00	0.10
	Human Labelled	Human Labelled	Human Labelled
ADVANCE	0.30	0.31	0.30
AHEAD-DS	0.48	0.48	0.22

Table 2 : The

\mu_{1\%}

values for each dataset and MLLM pair before and after relabelling by a human, rounded to two decimal places.

\mu_{1\%}

represents the mean of the

1st

percentile (bottom

1\%

) of CLAP scores .

\mu_{1\%}

ranges from 1 to

-1

, a value of 1 represents perfect alignment and

-1

is perfect misalignment. A higher value is better.

$\mu_{c}$	AuditoryHuM	Zero-Shot	Captioning
ADVANCE	0.61	0.25	0.42
AHEAD-DS	0.55	0.40	0.46
TAU 2019	0.49	0.33	0.19

Table 3 :

\mu_{c}

represents mean alignment of labels produced by AuditoryHum vs zero-shot using provided labels vs captioning. A higher value is better.

$\mu_{1\%}$	Qwen 2.5 Omni 3B + Human-CLAP	Qwen 2.5 Omni 3B + LAION-CLAP
ADVANCE	0.13	0.07
AHEAD-DS	0.07	0.07
TAU 2019	0.10	0.13
	Human Labelled + Human-CLAP	Human Labelled + LAION-CLAP
ADVANCE	0.30	0.10
AHEAD-DS	0.22	0.04

Table 4 : The

\mu_{1\%}

values comparing the results from Human-CLAP and LAION-CLAP rounded to 2 decimal places. A higher value indicates better alignment between label and audio content.

	Instances	Mean CLAP score Minimal Cleanup	Mean CLAP score Default Cleanup
ADVANCE Long Labels	578	0.60	0.57
ADVANCE Non-English Lab.	0
AHEAD-DS Long Labels	781	0.60	0.58
AHEAD-DS Non-Eng. Lab.	4	0.14	-0.07
TAU 2019 Long Labels	2726	0.50	0.49
TAU 2019 Non-English Lab.	2	0.13	0.18

Table 5 : The mean CLAP score values comparing the results with default or minimal label cleanup.

Figure 2 : Silhouette scores comparing AuditoryHuM vs zero-shot CLAP audio embeddings vs caption sentence transformer embeddings. A higher value is better.

	Provided Labels mAP	Provided Labels Accuracy	AuditoryHuM Labels mAP	AuditoryHuM Labels Accuracy
ADVANCE	0.84	0.79	0.84	0.84
AHEAD-DS	0.97	0.98	0.83	0.89
TAU 2019	0.94	0.95	0.76	0.75

Table 6 : The mean average precision (mAP) and accuracy of the dataset provided and AuditoryHuM generated labels. Higher values are better.

Panel of Figure 3 — Figure 3 : Comparison of CLAP scores between human and MLLM labels.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt	Notes
Describe the auditory scene using word pairs. Separate each pair with a comma.	Initial prompt.
Provide a short sentence to describe this set of audio samples. The frequency distribution of individual labels for this set of audio samples is provided: $<distribution>…</distribution>$ .	Prompt to generate a descriptive composite label for the entire cluster.
Describe this auditory scene.	Captioning prompt.

Appendix

Table 7 : The prompts for generating labels.

Step	Model or Algorithm	Params \| $\approx$ MACs	Complexity	Notes
1	Label generation (Qwen 2.5 Omni 3B)	4.7035B \| 971.995G	$O(H\cdot W\log W)+O(N\cdot P+N^{2}\cdot D)+O(P+N\cdot D)$	Steps include STFT, encoder, and decoder. $H$ is the number of hops. $W$ is the window length. $N$ is the total token length. $P$ number of params. $D$ is the hidden layer dimension.
2	CLAP score (Human-CLAP)	153.507M \| 6.287G	$O(H\cdot W\log W)+O(T_{a}^{2}\cdot D+T_{a}\cdot D^{2})+L\cdot O(1)+L\cdot O(T_{t}^{2}\cdot D+T_{t}\cdot D^{2})+L\cdot O(E)$	Steps include STFT, audio encoder, label clean up on $L$ labels, text encoder on $L$ labels, and Cosine similarity on $L$ labels. $W$ is the window length. $T_{a}$ is the audio token length. $T_{t}$ is the text token length. $D$ is the hidden layer dimension. $E$ is the CLAP embedding length.
3	Select best aligned label		$O(L)$	Find the largest value in an unsorted sequence of length $L$ .
				Steps $1$ to $3$ are repeated $A$ times. Where $A$ is the number of audio samples.
4	Create alignment vector (Human-CLAP)	153.507M \| 6.287G	$O(H\cdot W\log W)+O(T_{a}^{2}\cdot D+T_{a}\cdot D^{2})+L\cdot O(1)+L\cdot O(T_{t}^{2}\cdot D+T_{t}\cdot D^{2})+L\cdot O(E)$	Steps include STFT, audio encoder, label clean up on $L$ labels, text encoder on $L$ labels, and Cosine similarity on $L$ labels. $W$ is the window length. $T_{a}$ is the audio token length. $T_{t}$ is the text token length. $D$ is the hidden layer dimension. $E$ is the CLAP embedding length.
				Step $4$ is repeated up to $A^{2}$ times to get alignment vectors.

Appendix

Table 8 : Breakdown of the computational cost and complexity of each step in AuditoryHuM, using multiply and accumulate computations (MACs) and Big-O.

Panel of Figure 4 — Figure 4 : t-SNE visualisations using $k^{*}$ value for each dataset. Each unique colour represents a cluster of thematically related labels.

Dataset	Cluster Size / Total	Unique Labels	Composite Label for Cluster
ADVANCE	1582 / 5075	186	The audio samples consist mainly of ocean related sounds such as waves, thunder, and wind, with a few other elements like car, train, and aircraft sounds mixed in, along with various background noises and other natural sounds.
ADVANCE	1367 / 5075	375	The audio samples consist of a variety of sounds including thuds, frying noises, car sounds, choir voices, vibrations, horse trotting, train movement, drills, sewing machines, and other mechanical sounds, among others.
ADVANCE	983 / 5075	117	These audio samples contain various animal sounds, human activities, and natural elements, including bird vocalizations, insects, and weather sounds.
AHEADDS	4713 / 9968	198	This audio set contains a variety of sounds including clatter, child speaking, restaurant background noise, white noise, engine sounds, female conversations, dishes clinking, door slams, car passing, male laughter, crowd talk, and many other common everyday sounds, totaling 4713 samples.
AHEADDS	1668 / 9968	100	The audio set contains a diverse range of music genres and styles, including country, blues, heavy metal, opera, jazz, and more, with a focus on vocal performances and specific instruments like guitar and piano.
AHEADDS	1582 / 9968	93	The audio samples predominantly feature traffic noise, car passing, and various road-related sounds, with some additional elements like engine sounds, wind, and weather effects.

Appendix

Table 9 : Details of the 3 largest clusters for each dataset.

Component / Hyperparameter	Value / Strategy
Architecture	OpenYAMNet (Pretrained on AudioSet)
Optimiser	Adam
Loss Function	Focal Loss
Label Smoothing	0.1
Data Scheduling	Shuffled each epoch
Learning Rate Scheduler	Halved after 3 consecutive epochs without val improvement

Appendix

Table 10 : Downstream training configuration and hyperparameters.

Explore similar work

Audio-Zero: Label-Free Self-Evolution for Fine-Grained Audio Reasoning
Jul 22, 2026Siqian Tong, Xuan Li, Chaozhuo Li +5Audio Understanding Audio Understanding
Institute of Acoustics, Chinese Academy of Sciences · University of Chinese Academy of Sciences · 3Beijing Academy of Artificial Intelligence +3
From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models
Jun 24, 2026Pengfei Zhang, Hoang H Nguyen, Kazi Shaharair Sharif +6Large Audio Language Models
University of California Irvine · University of Illinois Chicago · Kennesaw State University
Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text
May 27, 2026Jiahao Mei, Heinrich Dinkel, Yadong Niu +7Modern Generative Audio Models Acoustic Latent Space
1X-LANCE Lab, Shanghai Jiao Tong University, Shanghai, China · 2MiLM Plus, Xiaomi Inc., Beijing, China

Jul 22, 2026, cs.SDJ/K move · Enter open · S save

Audio-Zero: Label-Free Self-Evolution for Fine-Grained Audio Reasoning

Siqian Tong, Xuan Li, Chaozhuo Li, Baolong Bi+4

Institute of Acoustics, Chinese Academy of Sciences · University of Chinese Academy of Sciences · 3Beijing Academy of Artificial Intelligence+3

Audio Understanding Audio Understanding Self-Play Self-Evolution

Open paper AskPDF arXiv

Large Audio Language models (LALMs) have made rapid progress on acoustic understanding, yet they still struggle with fine-grained audio reasoning (e.g., recognizing event order, repetitions and duration). Existing post-training methods heavily rely on expensive external labels or provide only coarse semantic signals. To bridge this gap, we introduce Audio-Zero, the first label-free self-evolution framework in the field of LALMs that improves fine-grained auditory perception and reasoning. Audio-Zero constructs an auditory self-play game from unlabeled audio contrast pairs: most players hear a reference audio, while one odd listener hears a subtle variant. The model first generates clues describing what it hears and then identifies the odd listener by reasoning over inconsistencies among clues. Since the odd listener is known by construction, the game provides verifiable rewards without any annotated answers. Experiments with Qwen2-Audio-7B-Instruct and Qwen2.5-Omni-7B on TREA, MMAU Test-mini and MMAR show that Audio-Zero improves fine-grained audio reasoning while preserving broad audio understanding. Evolutionary and diagnostic analyses further reveal that increasingly fine-grained auditory descriptions emerge naturally from game pressure.

Abstract

Explore similar work

Audio-Zero: Label-Free Self-Evolution for Fine-Grained Audio Reasoning

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text