Organizations: Basic Model Technology Center, WeChat AI, Tencent Inc. · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University
Recent Audio Large Language Models (AudioLLMs) exhibit a striking performance inversion: while excelling at complex reasoning tasks, they consistently underperform on fine-grained acoustic perception. We attribute this gap to a fundamental limitation of ASR-centric training, which provides precise linguistic targets but implicitly teaches models to suppress paralinguistic cues and acoustic events as noise. To address this, we propose Unified Audio Schema (UAS), a holistic and structured supervision framework that organizes audio information into three explicit components -- Transcription, Paralinguistics, and Non-linguistic Events -- within a unified JSON format. This design achieves comprehensive acoustic coverage without sacrificing the tight audio-text alignment that enables reasoning. We validate the effectiveness of this supervision strategy by applying it to both discrete and continuous AudioLLM architectures. Extensive experiments on MMSU, MMAR, and MMAU demonstrate that UAS-Audio yields consistent improvements, boosting fine-grained perception by 10.9% on MMSU over the same-size state-of-the-art models while preserving robust reasoning capabilities. Our code and model are publicly available at https://github.com/Tencent/Unified_Audio_Schema.
Figures & tables
Figure 1: Overview of the Unified Audio Schema (UAS) and evaluation results. (a) UAS structures audio information into three components: Transcription, Paralinguistics, and Non-linguistic Events. (b) Reasoning vs. Perception accuracy on MMSU. UAS-Audio significantly enhances perception while maintaining robust reasoning.
Figure 2: Overview of the UAS Data Generation Pipeline. (Stage 1) Acoustic captions are generated to capture paralinguistic and environmental context. (Stage 2) A structured synthesizer merges these captions with ground-truth transcripts. (Stage 3) An automated validation stage filters hallucinations and enforces logical consistency, yielding a final UAS that is ‘Structured & Complete’
Figure 3: Overview of UAS-Audio architectures . Audio input is processed via either discrete tokenization or continuous encoding (left). The LLM generates text-only or bi-modal outputs (right).
Model
MMSU
MMAR
MMAU
Avg.
Perception
Reasoning
Overall
Speech
Sound
Music
Overall
Speech
Sound
Music
Overall
Discrete Input Architecture
GLM-4-Voice
11.04
16.16
13.30
34.35
29.70
19.90
29.60
35.44
27.63
27.84
30.30
24.4
UAS-Audio-D
31.32
48.55
39.66
44.56
40.00
36.89
43.30
46.25
59.16
62.57
56.00
44.2
Continuous Input Architecture
Kimi-Audio
44.8
75.7
59.8
58.5
49.7
33.0
48.0
62.2
75.7
66.8
68.2
58.7
Table 1: Unified evaluation results on MMSU, MMAR, and MMAU benchmarks. The models are categorized by their audio input representation (Discrete vs. Continuous). Bold denotes the best result; underline denotes the second best. UAS-Audio-D denotes the discrete-input variant of UAS-Audio.
Figure 4: Impact of individual components . We observe a consistent performance drop when removing the UAS module or the UAS-QA strategy from the full model.
Model
Seed-Zh
Seed-EN
Avg
Baichuan-Audio-Instruct
2.9
4.7
3.8
Qwen2.5-Omni
1.4
2.3
1.9
Step-Audio2-mini
2.1
3.2
2.7
UAS-Audio
1.4
1.7
1.6
Table 2: Evaluation results on TTS benchmarks. Avg reports the average score of Seed-Zh and Seed-EN. Bold denotes the best result.
Prompt Strategy
Output
LS-Clean
AISHELL
Targeted ASR
Raw Text
2.2
2.3
Holistic UAS
JSON
2.3
2.4
Performance Deviation
≈ 0.1
≈ 0.1
Table 3: Robustness of ASR Capability under Multi-Task Generation. Comparing standard ASR generation versus extracting transcription from the unified JSON output (Holistic). The model maintains high recognition accuracy even when handling complex structured predictions.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: The web-based human evaluation interface. Annotators listen to the audio and verify if the predicted JSON values (black boxes) match the acoustic reality by selecting "Correct", "Incorrect", or "Unsure".
UAS Domain
Field
Accuracy (%)
95% CI
Paralinguistics
Age
98.50
[96.77,99.31]
Gender
95.25
[92.70,96.94]
Emotion
89.00
[85.55,91.70]
Accent
96.00
[93.60,97.52]
Prosody
96.50
[94.21,97.90]
Timbre
95.50
[93.00,97.13]
Appendix
Table 4: Human evaluation accuracy across UAS fields ( N=400 ). 95% Confidence Intervals (CI) are calculated using the Wilson score interval method.
Dataset
Duration (#hours)
LibriSpeech ( Panayotov et al., 2015 )
960
Multilingual LibriSpeech ( Pratap et al., 2020 )
27,322
GigaSpeech ( Chen et al., 2021 )
10,000
Yodas ( Li et al., 2023 )
29,155
Hi-Fi TTS ( Bakhturina et al., 2021 )
292
VCTK ( Yamagishi et al., 2019 )
44
Appendix
Table 5: Summary of datasets used for training UAS-Audio
Model Settings
Perception
Reasoning
Average
UAS-Audio (Full)
55.7
77.4
66.2
− w/o GRPO (Stage 4)
54.8
76.0
65.2
Best Baseline (Kimi-Audio)
44.8
75.7
62.2
Appendix
Table 6: Ablation study on the impact of GRPO training (Stage 4) on the MMSU benchmark. The results show that while GRPO provides further optimization, the majority of the performance gain stems from the UAS supervision.
Target Format
Perception
Reasoning
Average
Unstructured Caption
48.4
75.5
61.5
Structured UAS
54.8
76.0
65.2
Appendix
Table 7: Impact of supervision format on MMSU. The "Unstructured Caption" setting serves as a proxy for caption-based approaches (e.g., MiDashengLM). Both settings use the exact same synthetic data source . All models are trained without GRPO.
Hyperparameter
Stage 1
Stage 2
Stage 3
Stage 4
Peak Learning Rate
5e-4
2e-4
1e-4
5e-6
Warmup Iterations
500
1,000
1,000
200
LR Schedule
Cosine with Linear Warmup
Optimizer
AdamW ( β1=0.9 , β2=0.95 )
Weight Decay
0.1
Gradient Clipping
1.0
Appendix
Table 8: Hyperparameter configurations across training stages.