AudioLens: Multi-Perspective Speech Clustering with Reasoning Audio-Language Models

Wenjun Huang; Qiaosong Chu; Tiger Shao; Pengfei Zhang; Yutong Song; Hanning Chen; Yezi Liu; Weiyi Wu

cs.SDAug 25, 2026

AudioLens: Multi-Perspective Speech Clustering with Reasoning Audio-Language Models

Authors: Wenjun Huang, Qiaosong Chu, Tiger Shao, Pengfei Zhang, Yutong Song, Hanning Chen, Yezi Liu, Weiyi Wu,

Organizations: University of California, Irvine · Independent Researcher · Dartmouth College · Purdue University Northwest

Abstract

Audio clustering is a fundamental task for organizing rapidly growing speech collections, supporting applications such as conversational analysis and speech-driven discovery. However, existing methods rely on fixed acoustic similarity metrics or ASR-based text pipelines, limiting their ability to reorganize the same audio collection under different user-specified perspectives, especially when clustering depends on both linguistic and paralinguistic cues. We introduce audio multi-perspective clustering, where a model directly partitions speech recordings according to a natural-language perspective while inferring both the number of clusters and their assignments. To study this setting, we construct AudioLens-Bench, a benchmark spanning multiple application domains and evaluating both in-perspective and cross-perspective generalization. We further propose AudioLens-R1, an end-to-end large audio-language model trained with reasoning distillation and preference optimization. Experiments show that AudioLens-R1 consistently outperforms all baselines, improving overall ARI by 12.99 points and V-measure by 11.62 points. These results demonstrate the promise of native audio-language models for flexible, perspective-conditioned structure discovery over speech collections.

Figures & tables

Figure 1: Three paradigms for audio clustering. (a) Traditional pipelines transcribe audio, embed transcripts, and apply a clustering algorithm. (b) Transcript-based LLM clustering follows user-specified perspectives, but remains limited when the perspective depends on paralinguistic cues. (c) AudioLens-R1 takes raw audio and natural-language perspectives as input, enabling the same speech collection to be clustered into different valid partitions under different perspectives.

Figure 2: Overview of AudioLens-Bench construction.

Dataset	Train		$\mathrm{\textbf{L}}_{0}$		$\mathrm{\textbf{L}}_{1}$		$\mathrm{\textbf{L}}_{2}$
Dataset	#Audio	#Data	#Audio	#Data	#Audio	#Data	#Audio	#Data
Banking77	641	4110	584	1863	519	1800	599	1849
MultiWOZ	466	1571	479	1617	440	1574	499	1578
ECHR	519	3795	478	1735	453	1665	478	1670
S&P 500	472	3907	404	1784	373	1700	396	1689

Table 1: Statistics of AudioLens-Bench. #Audio denotes the number of unique audio recordings, and #Data denotes the number of clustering instances.

Figure 3: Overview of preference optimization pipeline.

ASR (Whisper Radford et al. (2023) ) + Text Embedding + Clustering
Method	Clustering	ECHR			S&P 500			Banking77			MultiWOZ			Average			Overall
Method	Clustering	$\mathrm{\textbf{L}}_{0}$	$\mathrm{\textbf{L}}_{1}$	$\mathrm{\textbf{L}}_{2}$	$\mathrm{\textbf{L}}_{0}$	$\mathrm{\textbf{L}}_{1}$	$\mathrm{\textbf{L}}_{2}$	$\mathrm{\textbf{L}}_{0}$	$\mathrm{\textbf{L}}_{1}$	$\mathrm{\textbf{L}}_{2}$	$\mathrm{\textbf{L}}_{0}$	$\mathrm{\textbf{L}}_{1}$	$\mathrm{\textbf{L}}_{2}$	$\mathrm{\textbf{L}}_{0}$	$\mathrm{\textbf{L}}_{1}$	$\mathrm{\textbf{L}}_{2}$
	K-Means	4.98	4.24	8.82	9.90	5.71	5.42	8.29	6.83	6.83	4.73	4.77	8.81	6.97	5.39	7.47	6.61
InBedder Peng et al. (2024)	GMM	1.58	2.56	9.06	2.89	2.15	3.73	5.20	8.11	9.57	5.43	3.92	7.16	3.77	4.18	7.38	5.11
	K-Means	12.18	15.40	26.51	16.19	18.79	15.56	23.57	24.68	18.18	10.55	11.41	11.09	15.62	17.57	17.84	17.01
Instructor Su et al. (2023)	GMM	8.44	9.68	17.30	9.37	9.46	6.98	14.79	13.13	14.65	7.30	7.66	6.48	9.97	9.98	11.35	10.44

Table 2: ARI comparison across four corpora. AudioLens-R1 achieves the best overall ARI and demonstrates strong performance across most clustering settings. For embedding-based baselines, we report results with both K-Means and GMM clustering, while LLM/LALM-based methods directly produce clustering assignments. The best result is highlighted in bold blue , and the second-best result is underlined .

ASR (Whisper Radford et al. (2023) ) + Text Embedding + Clustering
Method	Clustering	ECHR			S&P 500			Banking77			MultiWOZ			Average			Overall
Method	Clustering	$\mathrm{\textbf{L}}_{0}$	$\mathrm{\textbf{L}}_{1}$	$\mathrm{\textbf{L}}_{2}$	$\mathrm{\textbf{L}}_{0}$	$\mathrm{\textbf{L}}_{1}$	$\mathrm{\textbf{L}}_{2}$	$\mathrm{\textbf{L}}_{0}$	$\mathrm{\textbf{L}}_{1}$	$\mathrm{\textbf{L}}_{2}$	$\mathrm{\textbf{L}}_{0}$	$\mathrm{\textbf{L}}_{1}$	$\mathrm{\textbf{L}}_{2}$	$\mathrm{\textbf{L}}_{0}$	$\mathrm{\textbf{L}}_{1}$	$\mathrm{\textbf{L}}_{2}$
	K-Means	50.71	51.51	57.84	53.69	50.88	48.79	52.13	52.87	50.09	30.24	29.38	29.55	46.69	46.16	46.57	46.47
InBedder Peng et al. (2024)	GMM	49.19	50.79	58.20	50.64	49.54	48.81	50.71	54.13	53.02	31.49	29.43	28.97	45.51	45.97	47.25	46.24
Instructor Su et al. (2023)	K-Means	55.50	57.76	67.36	57.81	59.13	54.67	60.18	61.91	57.21	34.75	34.41	31.18	52.06	53.30	52.61	52.66
Instructor Su et al. (2023)	GMM	53.50	55.07	62.85	54.41	53.58	50.40	56.26	57.21	55.96	32.98	32.15	28.51	49.29	49.50	49.43	49.41

Table 3: V-measure comparison across four corpora. The best result in each column is highlighted in bold blue , and the second-best result is underlined .

Panel of Figure 4 — Figure 4: Perspective-level performance averaged across the corpora. Each axis represents one clustering perspective, and each curve represents one model. Because polar coordinates cannot faithfully represent a negative radius, the ARI of $-1.50$ obtained by Whisper + GPT-4o for emotion clustering is displayed at zero. All other points show their exact values.

Metric	Corpus / Split	Answer-only SFT	RD SFT	Answer-only + DPO	RD + DPO
ARI	ECHR	33.17	33.85	35.38	42.02
	S&P 500	33.30	38.23	37.58	42.73
	Banking77	39.50	39.48	41.26	49.85
	MultiWOZ	33.34	32.34	34.07	44.50
	Overall	34.83	35.97	37.07	44.77
	$\Delta$	–	$+1.14$	$+2.24$	$\mathbf{+9.94}$

Table 4: Ablation study of answer-only supervised fine-tuning (SFT), RD, and DPO. RD SFT denotes supervised fine-tuning with RD, while Answer-only DPO and RD+DPO denote DPO initialized from the corresponding checkpoints.

\Delta

denotes the absolute improvement over the answer-only SFT baseline.

Panel of Figure 5 — Figure 5: (a) Evolution of mean output length, decomposed into total output tokens, <think> tokens, and <answer> tokens. (b) Evolution of the scaled preference margin and policy-reference KL divergence. The margin is computed as $m_{\mathrm{scaled}}=L_{\mathrm{ref}}\left(\overline{\log\pi_{\theta}(y^{+}\mid x)}-\overline{\log\pi_{\theta}(y^{-}\mid x)}\right)$ .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Transcript Generation	Validation	TTS Path
Banking77	Classify each short intent query under the refined taxonomy, then expand each classifiable query into a natural spoken transcript (monologue or short dialogue).	Re-classify the expanded transcript under the same taxonomy and retain it only if all transcript-level labels exactly match the original query-level assignments.	Synthesize validated transcripts with sampled emotion and speaker identity; use one speaker for monologues and multiple speakers for dialogues; augment the waveform with background-noise conditions.
ECHR	Assign taxonomy labels to long legal case texts, compress each accepted case into an evidence-preserving summary, and rewrite the summary into a spoken script.	Use dual-model agreement for source-text labeling, then re-classify the compressed text and retain it only if all previously accepted dimensions remain recoverable after compression.	Synthesize accepted spoken scripts with one to four speakers, sampled emotion, and background-noise conditions, with approximate balancing of paralinguistic factors within semantic strata.
S&P 500	Assign taxonomy labels to 10-K disclosure segments, compress each accepted document into an evidence-preserving summary, and rewrite the summary into a spoken script.	Use dual-model agreement for source-text labeling, then re-classify the compressed text and retain it only if all accepted dimensions are preserved after compression.	Synthesize accepted spoken scripts with sampled emotion, speaker count, and background-noise condition, again using a balanced assignment scheme within semantic strata.
MultiWOZ	Directly reuse sampled human-written dialogues as transcripts after assigning taxonomy labels.	Retain only dialogues for which two independent classifiers produce identical label assignments across all taxonomy dimensions.	Synthesize the validated dialogue turn by turn with distinct speakers for different roles, while injecting sampled emotion and background-noise variation.

Appendix

Table 5: Corpus-specific transcript generation, validation, and TTS paths. Banking77 uses expansion from short intent queries; ECHR and S&P 500 use compression followed by spoken rewriting; MultiWOZ directly reuses native dialogues.

Corpus	Held-in perspectives	Held-out perspectives
Banking77	• Cluster requests by the customer’s primary intended outcome • Cluster requests by the main banking problem or request category • Cluster requests by the dominant urgency, severity, or risk level • Cluster clips by the dominant emotional tone in speech • Cluster clips by the exact number of distinct speakers • Cluster clips by the real-world background-noise scene	• Cluster requests by the banking capability or product domain most directly involved • Cluster requests by the primary route through which the issue would be resolved • Cluster clips by the dominant speaker-gender pattern
ECHR	• Cluster cases by the dominant State act or omission producing the alleged violation • Cluster cases by the procedural or situational context of the dispute • Cluster cases by the structure of the dispute and party configuration • Cluster clips by the dominant emotional tone in speech • Cluster clips by the exact number of distinct speakers • Cluster clips by the real-world background-noise scene	• Cluster cases by the principal protected interest implicated • Cluster cases by the main kind of harm or outcome alleged • Cluster cases by the most salient applicant status affecting power imbalance or risk • Cluster clips by the dominant speaker-gender pattern
S&P 500	• Cluster firms by the primary way key properties are controlled • Cluster firms by the geographic breadth and concentration of properties • Cluster firms by the dominant type of legal matter discussed • Cluster firms by the dominant signal about legal-matter status and potential financial impact • Cluster clips by the dominant emotional tone in speech • Cluster clips by the exact number of distinct speakers • Cluster clips by the real-world background-noise scene	• Cluster disclosures by the dominant type of physical properties central to operations • Cluster disclosures by the dominant approach to shareholder return • Cluster disclosures by the dominant approach to repurchases or equity structure • Cluster clips by the dominant speaker-gender pattern
MultiWOZ	• Cluster dialogues by the presence and source of explicit repair turns • Cluster dialogues by how users provide, accumulate, revise, or confirm constraints • Cluster dialogues by the balance between information-seeking and transaction-execution content • Cluster dialogues by whether the task succeeds directly, partially succeeds, fails, or is recovered through fallback • Cluster clips by the dominant emotional tone in speech • Cluster clips by the exact number of distinct speakers • Cluster clips by the real-world background-noise scene	• Cluster dialogues by how much earlier information must be remembered and reused later • Cluster dialogues by who drives the interaction and how strongly negotiation shapes the exchange • Cluster dialogues by whether the interaction stays within a narrow goal or couples multiple domains or subgoals • Cluster clips by the dominant speaker-gender pattern

Appendix

Table 6: Corpus-specific clustering perspectives.

Source	Episodes	Unique audios	Clips / episode	Gold $K$	Duration (s)
All SFT train	1,876	2,008	5.64 (7, 10)	3.37 (6, 9)	373.5 (524.6, 539.9)
ECHR	539	501	5.51 (7, 7)	3.37 (6, 7)	452.8 (531.4, 539.9)
S&P 500	571	455	5.69 (7, 9)	3.45 (6, 9)	411.4 (525.2, 539.2)
Banking77	523	618	5.88 (8, 10)	3.66 (7, 9)	306.3 (431.1, 538.1)
MultiWOZ	243	434	5.33 (7, 7)	2.56 (4, 5)	253.0 (372.3, 445.3)

Appendix

Table 7: Episode-level statistics of the SFT training set. Entries for the final three columns are reported as mean (95th percentile, maximum).

Panel of Figure 6 — Figure 6: Source-specific episode and cluster profiles in the SFT training set. All axes report percentages. The first axis represents the percentage of multi-cluster episodes, i.e., $100\%-\Pr(K=1)$ . The three cluster-size axes form a compositional distribution and therefore sum to $100\%$ for each source.

Metric	Training	ECHR					S&P 500					Banking77					MultiWOZ					Overall
Metric	Training	BG	Emo.	Spk.	Gen.	Reason	BG	Emo.	Spk.	Gen.	Reason	BG	Emo.	Spk.	Gen.	Reason	BG	Emo.	Spk.	Gen.	Reason	BG	Emo.	Spk.	Gen.	Reason
ARI	RD SFT	0.2947	0.2809	0.2682	0.2806	0.3872	0.3754	0.3096	0.3212	0.3539	0.3968	0.3380	0.2616	0.3025	0.3803	0.4749	0.3127	0.3664	0.0000	0.3823	0.4067	0.3302	0.3046	0.2230	0.3493	0.4164
ARI	RD+DPO	0.3163	0.2882	0.2727	0.2993	0.3821	0.3965	0.3169	0.3293	0.3554	0.4014	0.3818	0.2437	0.3448	0.3786	0.4507	0.3407	0.3557	0.0000	0.3724	0.4010	0.3588	0.3012	0.2367	0.3514	0.4088
V-measure	RD SFT	0.6135	0.6772	0.6212	0.5756	0.8025	0.6460	0.6879	0.6294	0.5866	0.7717	0.5475	0.6769	0.4725	0.5801	0.8592	0.5115	0.7023	0.0000	0.5134	0.6240	0.5796	0.6861	0.4308	0.5639	0.7643
V-measure	RD+DPO	0.6221	0.6827	0.6223	0.5864	0.8000	0.6595	0.6920	0.6329	0.5837	0.7689	0.5834	0.6660	0.4921	0.5779	0.8485	0.5384	0.6985	0.0000	0.5141	0.6238	0.6008	0.6848	0.4368	0.5655	0.7603

Appendix

Table 8: Performance of using Qwen2.5-omni as the base model. BG, Emo., Spk., Gen., and Reason denote background noise, emotion, speaker count, gender, and the average over linguistic reasoning perspectives.

Panel of Figure 7 — Figure 7: Case study

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding
Sep 19, 2026Nishit Anand, Jiaqi Su, Ke Chen +5Audio Understanding Audio-Language Models
Adobe Research, USA · University of Maryland, College Park, USA · OpenAI, USA
Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs
Apr 14, 2026Linhao Zhang, Yuhan Song, Aiwei Liu +6Audio Understanding Audio-Language Models
Basic Model Technology Center, WeChat AI, Tencent Inc. · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University
ALARM: Audio-Language Alignment for Reasoning Models
Mar 10, 2026Petr Grinberg, Hassan ShahmohammadiAudio-Language Models Audio Reasoning
EPFL, Switzerland · Sony Europe Ltd., Germany

Sep 19, 2026J/K move · Enter open · S save

ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding

Nishit Anand, Jiaqi Su, Ke Chen, Yunyun Wang+4

Adobe Research, USA · University of Maryland, College Park, USA · OpenAI, USA

Audio Understanding Audio-Language Models Audio Reasoning Audio-Language Model Evaluation

Open paper AskPDF arXiv

Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.

Abstract

Audio-Language Models

Explore similar work

ParA-LLM: A Unified Approach to Paralinguistic and Acoustic Speech Understanding

Beyond Transcription: Unified Audio Schema for Perception-Aware AudioLLMs

ALARM: Audio-Language Alignment for Reasoning Models