Audio clustering is a fundamental task for organizing rapidly growing speech collections, supporting applications such as conversational analysis and speech-driven discovery. However, existing methods rely on fixed acoustic similarity metrics or ASR-based text pipelines, limiting their ability to reorganize the same audio collection under different user-specified perspectives, especially when clustering depends on both linguistic and paralinguistic cues. We introduce audio multi-perspective clustering, where a model directly partitions speech recordings according to a natural-language perspective while inferring both the number of clusters and their assignments. To study this setting, we construct AudioLens-Bench, a benchmark spanning multiple application domains and evaluating both in-perspective and cross-perspective generalization. We further propose AudioLens-R1, an end-to-end large audio-language model trained with reasoning distillation and preference optimization. Experiments show that AudioLens-R1 consistently outperforms all baselines, improving overall ARI by 12.99 points and V-measure by 11.62 points. These results demonstrate the promise of native audio-language models for flexible, perspective-conditioned structure discovery over speech collections.
Figures & tables
Figure 1: Three paradigms for audio clustering. (a) Traditional pipelines transcribe audio, embed transcripts, and apply a clustering algorithm. (b) Transcript-based LLM clustering follows user-specified perspectives, but remains limited when the perspective depends on paralinguistic cues. (c) AudioLens-R1 takes raw audio and natural-language perspectives as input, enabling the same speech collection to be clustered into different valid partitions under different perspectives.
Figure 2: Overview of AudioLens-Bench construction.
Dataset
Train
L0
L1
L2
#Audio
#Data
#Audio
#Data
#Audio
#Data
#Audio
#Data
Banking77
641
4110
584
1863
519
1800
599
1849
MultiWOZ
466
1571
479
1617
440
1574
499
1578
ECHR
519
3795
478
1735
453
1665
478
1670
S&P 500
472
3907
404
1784
373
1700
396
1689
Table 1: Statistics of AudioLens-Bench. #Audio denotes the number of unique audio recordings, and #Data denotes the number of clustering instances.
Figure 3: Overview of preference optimization pipeline.
Method
Clustering
ECHR
S&P 500
Banking77
MultiWOZ
Average
Overall
L0
L1
L2
L0
L1
L2
L0
L1
L2
L0
L1
L2
L0
L1
L2
ASR (Whisper Radford et al. (2023) ) + Text Embedding + Clustering
K-Means
4.98
4.24
8.82
9.90
5.71
5.42
8.29
6.83
6.83
4.73
4.77
8.81
6.97
5.39
7.47
6.61
InBedder Peng et al. (2024)
GMM
1.58
2.56
9.06
2.89
2.15
3.73
5.20
8.11
9.57
5.43
3.92
7.16
3.77
4.18
7.38
5.11
K-Means
12.18
15.40
26.51
16.19
18.79
15.56
23.57
24.68
18.18
10.55
11.41
11.09
15.62
17.57
17.84
17.01
Instructor Su et al. (2023)
GMM
8.44
9.68
17.30
9.37
9.46
6.98
14.79
13.13
14.65
7.30
7.66
6.48
9.97
9.98
11.35
10.44
Table 2: ARI comparison across four corpora. AudioLens-R1 achieves the best overall ARI and demonstrates strong performance across most clustering settings. For embedding-based baselines, we report results with both K-Means and GMM clustering, while LLM/LALM-based methods directly produce clustering assignments. The best result is highlighted in bold blue , and the second-best result is underlined .
Method
Clustering
ECHR
S&P 500
Banking77
MultiWOZ
Average
Overall
L0
L1
L2
L0
L1
L2
L0
L1
L2
L0
L1
L2
L0
L1
L2
ASR (Whisper Radford et al. (2023) ) + Text Embedding + Clustering
K-Means
50.71
51.51
57.84
53.69
50.88
48.79
52.13
52.87
50.09
30.24
29.38
29.55
46.69
46.16
46.57
46.47
InBedder Peng et al. (2024)
GMM
49.19
50.79
58.20
50.64
49.54
48.81
50.71
54.13
53.02
31.49
29.43
28.97
45.51
45.97
47.25
46.24
Instructor Su et al. (2023)
K-Means
55.50
57.76
67.36
57.81
59.13
54.67
60.18
61.91
57.21
34.75
34.41
31.18
52.06
53.30
52.61
52.66
GMM
53.50
55.07
62.85
54.41
53.58
50.40
56.26
57.21
55.96
32.98
32.15
28.51
49.29
49.50
49.43
49.41
Table 3: V-measure comparison across four corpora. The best result in each column is highlighted in bold blue , and the second-best result is underlined .
Figure 4: Perspective-level performance averaged across the corpora. Each axis represents one clustering perspective, and each curve represents one model. Because polar coordinates cannot faithfully represent a negative radius, the ARI of −1.50 obtained by Whisper + GPT-4o for emotion clustering is displayed at zero. All other points show their exact values.
Metric
Corpus / Split
Answer-only SFT
RD SFT
Answer-only + DPO
RD + DPO
ARI
ECHR
33.17
33.85
35.38
42.02
S&P 500
33.30
38.23
37.58
42.73
Banking77
39.50
39.48
41.26
49.85
MultiWOZ
33.34
32.34
34.07
44.50
Overall
34.83
35.97
37.07
44.77
Δ
–
+1.14
+2.24
+9.94
Table 4: Ablation study of answer-only supervised fine-tuning (SFT), RD, and DPO. RD SFT denotes supervised fine-tuning with RD, while Answer-only DPO and RD+DPO denote DPO initialized from the corresponding checkpoints. Δ denotes the absolute improvement over the answer-only SFT baseline.
Figure 5: (a) Evolution of mean output length, decomposed into total output tokens, <think> tokens, and <answer> tokens. (b) Evolution of the scaled preference margin and policy-reference KL divergence. The margin is computed as mscaled=Lref(logπθ(y+∣x)−logπθ(y−∣x)) .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Corpus
Transcript Generation
Validation
TTS Path
Banking77
Classify each short intent query under the refined taxonomy, then expand each classifiable query into a natural spoken transcript (monologue or short dialogue).
Re-classify the expanded transcript under the same taxonomy and retain it only if all transcript-level labels exactly match the original query-level assignments.
Synthesize validated transcripts with sampled emotion and speaker identity; use one speaker for monologues and multiple speakers for dialogues; augment the waveform with background-noise conditions.
ECHR
Assign taxonomy labels to long legal case texts, compress each accepted case into an evidence-preserving summary, and rewrite the summary into a spoken script.
Use dual-model agreement for source-text labeling, then re-classify the compressed text and retain it only if all previously accepted dimensions remain recoverable after compression.
Synthesize accepted spoken scripts with one to four speakers, sampled emotion, and background-noise conditions, with approximate balancing of paralinguistic factors within semantic strata.
S&P 500
Assign taxonomy labels to 10-K disclosure segments, compress each accepted document into an evidence-preserving summary, and rewrite the summary into a spoken script.
Use dual-model agreement for source-text labeling, then re-classify the compressed text and retain it only if all accepted dimensions are preserved after compression.
Synthesize accepted spoken scripts with sampled emotion, speaker count, and background-noise condition, again using a balanced assignment scheme within semantic strata.
MultiWOZ
Directly reuse sampled human-written dialogues as transcripts after assigning taxonomy labels.
Retain only dialogues for which two independent classifiers produce identical label assignments across all taxonomy dimensions.
Synthesize the validated dialogue turn by turn with distinct speakers for different roles, while injecting sampled emotion and background-noise variation.
Appendix
Table 5: Corpus-specific transcript generation, validation, and TTS paths. Banking77 uses expansion from short intent queries; ECHR and S&P 500 use compression followed by spoken rewriting; MultiWOZ directly reuses native dialogues.
Corpus
Held-in perspectives
Held-out perspectives
Banking77
• Cluster requests by the customer’s primary intended outcome • Cluster requests by the main banking problem or request category • Cluster requests by the dominant urgency, severity, or risk level • Cluster clips by the dominant emotional tone in speech • Cluster clips by the exact number of distinct speakers • Cluster clips by the real-world background-noise scene
• Cluster requests by the banking capability or product domain most directly involved • Cluster requests by the primary route through which the issue would be resolved • Cluster clips by the dominant speaker-gender pattern
ECHR
• Cluster cases by the dominant State act or omission producing the alleged violation • Cluster cases by the procedural or situational context of the dispute • Cluster cases by the structure of the dispute and party configuration • Cluster clips by the dominant emotional tone in speech • Cluster clips by the exact number of distinct speakers • Cluster clips by the real-world background-noise scene
• Cluster cases by the principal protected interest implicated • Cluster cases by the main kind of harm or outcome alleged • Cluster cases by the most salient applicant status affecting power imbalance or risk • Cluster clips by the dominant speaker-gender pattern
S&P 500
• Cluster firms by the primary way key properties are controlled • Cluster firms by the geographic breadth and concentration of properties • Cluster firms by the dominant type of legal matter discussed • Cluster firms by the dominant signal about legal-matter status and potential financial impact • Cluster clips by the dominant emotional tone in speech • Cluster clips by the exact number of distinct speakers • Cluster clips by the real-world background-noise scene
• Cluster disclosures by the dominant type of physical properties central to operations • Cluster disclosures by the dominant approach to shareholder return • Cluster disclosures by the dominant approach to repurchases or equity structure • Cluster clips by the dominant speaker-gender pattern
MultiWOZ
• Cluster dialogues by the presence and source of explicit repair turns • Cluster dialogues by how users provide, accumulate, revise, or confirm constraints • Cluster dialogues by the balance between information-seeking and transaction-execution content • Cluster dialogues by whether the task succeeds directly, partially succeeds, fails, or is recovered through fallback • Cluster clips by the dominant emotional tone in speech • Cluster clips by the exact number of distinct speakers • Cluster clips by the real-world background-noise scene
• Cluster dialogues by how much earlier information must be remembered and reused later • Cluster dialogues by who drives the interaction and how strongly negotiation shapes the exchange • Cluster dialogues by whether the interaction stays within a narrow goal or couples multiple domains or subgoals • Cluster clips by the dominant speaker-gender pattern
Appendix
Table 6: Corpus-specific clustering perspectives.
Source
Episodes
Unique audios
Clips / episode
Gold K
Duration (s)
All SFT train
1,876
2,008
5.64 (7, 10)
3.37 (6, 9)
373.5 (524.6, 539.9)
ECHR
539
501
5.51 (7, 7)
3.37 (6, 7)
452.8 (531.4, 539.9)
S&P 500
571
455
5.69 (7, 9)
3.45 (6, 9)
411.4 (525.2, 539.2)
Banking77
523
618
5.88 (8, 10)
3.66 (7, 9)
306.3 (431.1, 538.1)
MultiWOZ
243
434
5.33 (7, 7)
2.56 (4, 5)
253.0 (372.3, 445.3)
Appendix
Table 7: Episode-level statistics of the SFT training set. Entries for the final three columns are reported as mean (95th percentile, maximum).
Figure 6: Source-specific episode and cluster profiles in the SFT training set. All axes report percentages. The first axis represents the percentage of multi-cluster episodes, i.e., 100%−Pr(K=1) . The three cluster-size axes form a compositional distribution and therefore sum to 100% for each source.
Metric
Training
ECHR
S&P 500
Banking77
MultiWOZ
Overall
BG
Emo.
Spk.
Gen.
Reason
BG
Emo.
Spk.
Gen.
Reason
BG
Emo.
Spk.
Gen.
Reason
BG
Emo.
Spk.
Gen.
Reason
BG
Emo.
Spk.
Gen.
Reason
ARI
RD SFT
0.2947
0.2809
0.2682
0.2806
0.3872
0.3754
0.3096
0.3212
0.3539
0.3968
0.3380
0.2616
0.3025
0.3803
0.4749
0.3127
0.3664
0.0000
0.3823
0.4067
0.3302
0.3046
0.2230
0.3493
0.4164
RD+DPO
0.3163
0.2882
0.2727
0.2993
0.3821
0.3965
0.3169
0.3293
0.3554
0.4014
0.3818
0.2437
0.3448
0.3786
0.4507
0.3407
0.3557
0.0000
0.3724
0.4010
0.3588
0.3012
0.2367
0.3514
0.4088
V-measure
RD SFT
0.6135
0.6772
0.6212
0.5756
0.8025
0.6460
0.6879
0.6294
0.5866
0.7717
0.5475
0.6769
0.4725
0.5801
0.8592
0.5115
0.7023
0.0000
0.5134
0.6240
0.5796
0.6861
0.4308
0.5639
0.7643
RD+DPO
0.6221
0.6827
0.6223
0.5864
0.8000
0.6595
0.6920
0.6329
0.5837
0.7689
0.5834
0.6660
0.4921
0.5779
0.8485
0.5384
0.6985
0.0000
0.5141
0.6238
0.6008
0.6848
0.4368
0.5655
0.7603
Appendix
Table 8: Performance of using Qwen2.5-omni as the base model. BG, Emo., Spk., Gen., and Reason denote background noise, emotion, speaker count, gender, and the average over linguistic reasoning perspectives.
Recent advances in Audio LLMs have achieved human-level speech recognition, yet existing systems struggle to capture paralinguistic aspects such as speaker traits, expressive variations, and environmental acoustic conditions. To address this, we design a framework of 22 paralinguistic characteristics and create a dataset of over 1.2M Audio-QA pairs. We develop ParA-LLM, trained with a two-stage curriculum: first on single-attribute questions to build foundational knowledge, then on multi-attribute questions for joint reasoning over speaker and acoustic characteristics. We also release ParA-Bench, a benchmark of 6,000 multiple-choice questions across speaker-speech, acoustic, and mixed categories, where frontier models like GPT-4o-Audio achieve only 36% accuracy. ParA-LLM surpasses state-of-the-art Audio LLMs like GPT-4o-Audio by 7.5% on ParA-Bench, with additional gains of 1.13% on MMAU-Pro Speech and 7.49% on MMAR Speech.
Nishit Anand, Jiaqi Su, Ke Chen +5
Adobe Research, USA · University of Maryland, College Park, USA · OpenAI, USA
Recent Audio Large Language Models (AudioLLMs) exhibit a striking performance inversion: while excelling at complex reasoning tasks, they consistently underperform on fine-grained acoustic perception. We attribute this gap to a fundamental limitation of ASR-centric training, which provides precise linguistic targets but implicitly teaches models to suppress paralinguistic cues and acoustic events as noise. To address this, we propose Unified Audio Schema (UAS), a holistic and structured supervision framework that organizes audio information into three explicit components -- Transcription, Paralinguistics, and Non-linguistic Events -- within a unified JSON format. This design achieves comprehensive acoustic coverage without sacrificing the tight audio-text alignment that enables reasoning. We validate the effectiveness of this supervision strategy by applying it to both discrete and continuous AudioLLM architectures. Extensive experiments on MMSU, MMAR, and MMAU demonstrate that UAS-Audio yields consistent improvements, boosting fine-grained perception by 10.9% on MMSU over the same-size state-of-the-art models while preserving robust reasoning capabilities. Our code and model are publicly available at https://github.com/Tencent/Unified_Audio_Schema.
Linhao Zhang, Yuhan Song, Aiwei Liu +6
Basic Model Technology Center, WeChat AI, Tencent Inc. · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University
Large audio language models (ALMs) extend LLMs with auditory understanding. A common approach freezes the LLM and trains only an adapter on self-generated targets. However, this fails for reasoning LLMs (RLMs) whose built-in chain-of-thought traces expose the textual surrogate input, yielding unnatural responses. We propose self-rephrasing, converting self-generated responses into audio-understanding variants compatible with RLMs while preserving distributional alignment. We further fuse and compress multiple audio encoders for stronger representations. For training, we construct a 6M-instance multi-task corpus (2.5M unique prompts) spanning 19K hours of speech, music, and sound. Our 4B-parameter ALM outperforms similarly sized models and surpasses most larger ALMs on related audio-reasoning benchmarks, while preserving textual capabilities with a low training cost. Notably, we achieve the best open-source result on the MMAU-speech and MMSU benchmarks and rank third among all the models.