Organizations: HKUST, Hong Kong SAR, China · Independent Researcher · Sogang University, Seoul, Korea · KAIST, Daejeon, Korea · NYU Shanghai, China · JIUTIAN Research, China Mobile, Beijing, China · The State Key Laboratory of Multimedia Information Processing, Peking University, Beijing, China · China Mobile (Hong Kong) Innovation Research Institute, Hong Kong SAR, China · Central Conservatory of Music, Beijing, China
Recent advances in large audio-language models (LALMs) have significantly improved performance in tasks such as music captioning, genre classification, and sound event detection. However, limited attention has been paid to improving their adaptability across diverse musical traditions, particularly folk music rooted in distinct cultural contexts. Folk-music traditions are typically resource-scarce, unevenly represented across regions, and poorly documented. Even when such samples appear in large-scale pre-training, LALMs often fail to capture their structural and stylistic characteristics, partly due to the absence of dedicated evaluation protocols and training solutions. To address these limitations, we introduce UniVerse, a reproducible solution for low-resource music understanding. Specifically, we propose UniVerseBench, a benchmark of 5,042 Q&A pairs across more than 38 cultural and linguistic entities, constructed via an expert-guided yet highly automated pipeline. In parallel, we construct a fully automated, model-generated multi-turn dialogue training dataset UniVerseSet. By training LALMs on UniVerseSet, we systematically adapt and investigate representative multimodal imbalance learning strategies across both dense and Mixture-of-Experts (MoE) architectures. Experimental results indicate that fully automated data curation combined with imbalance-aware training yields non-trivial improvements, but models still struggle to capture fine-grained acoustic features, indicating a gap between surface-level alignment and deep musical comprehension.
Figures & tables
Figure 1: The pipeline of UniVerseBench construction and LALMs training schema on UniVerSet .
Figure 2: Composition of UniVerseBench . (a) Language distribution ; Chinese and World slices are further broken down by language family (Chinese-region labels from metadata language/ISO codes; World languages mapped to macro families). (b) Question-type distribution over the final benchmark: feature extraction, causal reasoning, music pattern recognition, comparison analysis, error detection, other (includes music completion, etc.). (c) Audio Source and Quality distribution from the quality audit sample: indoor professional, indoor non-professional, and low-quality recording.
Figure 3: Acoustic structure of the benchmark audio excerpts in Qwen2.5-Omni embedding space. Points in panel (a) are colored by language family and each cluster’s convex hull is outlined by dashed polygons; Sino-Tibetan branches are labeled ST--Sinitic / ST--Tibeto-Burman (parallel to IE--* )). Panel (b) shows cosine distances between language family centroids.
Figure 4: Overview of the three post-training strategies. Lang. Loss SFT reweights the supervised loss by language frequency to boost low-resource traditions. Text / Audio DPO applies complementary preference stages post-SFT, optimizing text responses under fixed audio and audio inputs under fixed responses. Latent Reasoning + REPA provides architecture-specific alignment: latent reasoning with REPA on dense Qwen2.5-Omni-7B (Qwen2.5-Omni), and encoder-side REPA with a frozen decoder on MoE Qwen3-Omni-30B-A3B-Instruct (Qwen3-Omni).
Model
Overall
LoC*
ko
L-oth.
FE
CR
PR
CA
ED
Kimi-audio
46.5
37.8
57.8
46.2
44.2
47.7
47.3
62.6
50.0
MidashengLM
40.9
36.4
46.5
40.9
39.5
41.6
39.9
54.4
31.9
Music Flamingo
54.4
47.3
61.6
55.1
54.1
57.1
53.6
51.8
55.3
Gemini 3 Flash
70.8
67.6
77.6
69.5
68.7
73.3
71.3
84.9
55.6
Qwen3.5-Omni-plus
74.4
71.0
75.5
75.7
73.6
80.1
72.6
71.4
77.1
Qwen2.5-Omni-7B-Instruct
33.9
29.2
45.2
32.0
32.5
38.3
35.7
30.0
37.1
Table 1: Overall accuracy (%) on the UniVerseBench across aggregated, language, and skill categories. Language strata include LoC (Languages of China) , ko (Korean), and L-oth. (other languages). Skill strata encompass FE (feature extraction), CR (causal reasoning), PR (pattern recognition), CA (comparison analysis), and ED (error detection). “Post-trained” denotes the best performance of models fine-tuned on our thinking-format post-training data.
Method
Qwen2.5
Qwen3
Method
Qwen2.5
Qwen3
SFT (w\ think)
40.6
53.4
Audio DPO / Enc.-side REPA
45.7
52.9
Lang. Loss
41.8
51.9
LR. + REPA (Phase a)
42.9
—
Lang. Loss + Text DPO
42.6
50.9
LR. + REPA (Phase b) / Route-b Δ CE
48.8
52.9
Table 2: Ablation study of imbalanced learning and alignment strategies on the UniVerseBench . Values report strict parseable accuracy (%) formatted as Qwen2.5-Omni ∣ Qwen3-Omni . Underlined entries denote the best performance for each backbone.
Overall
FE
CR
PR
CA
Strategy
r
p
ρ
p
r
ρ
r
ρ
r
ρ
r
ρ
Q2.5 SFT (think)
0.255
0.219
0.028
0.895
0.279
0.105
0.346
0.096
− 0.014
0.204
− 0.272
− 0.531
Q2.5 Lang. Loss
0.270
0.193
0.095
0.651
0.350
0.140
0.099
− 0.077
0.062
0.031
− 0.130
− 0.480
Q2.5 Text DPO
0.339
0.097
0.064
0.760
0.327
− 0.108
0.150
0.017
0.105
0.164
0.252
− 0.133
Q2.5 Audio DPO
0.366
0.072
0.130
0.536
0.355
0.051
0.222
0.103
0.209
0.312
− 0.205
− 0.186
Q2.5 LR Phase-a
0.312
0.129
− 0.037
0.862
0.353
− 0.121
0.207
0.113
0.049
0.211
− 0.158
− 0.442
Table 3: Correlation between post-training utterance count and per-language adversarial accuracy ( N=25 ; n≥10 ). LR. represents Latent Reasoning. REPA++ represents REPA+ Δ CE+gates. Comparison Analysis merges plain + neutral variants. Category columns report Pearson’s r and Spearman’s ρ within each skill.
Figure 5: Per-language tuning gain versus training volume ( log10(n+1) ). Point size scales with sample size; quadrant axes indicate median splits on training volume and accuracy gain.
Region
Qwen2.5-Omni
Qwen3-Omni
Region
Qwen2.5-Omni
Qwen3-Omni
Chongqing
31.2 | 75.0
62.5 | 75.0
Ningxia
8.7 | 43.5
56.5 | 43.5
Hebei
31.0 | 56.9
37.9 | 48.3
Guizhou
31.1 | 42.6
39.3 | 42.6
Shanghai
34.5 | 56.4
40.0 | 47.3
Taiwan
29.1 | 41.8
41.8 | 48.1
Liaoning
18.5 | 55.6
55.6 | 59.3
Shandong
26.1 | 41.3
52.2 | 52.2
Jiangsu
34.4 | 54.1
59.0 | 49.2
Gansu
28.3 | 41.3
30.4 | 43.5
Henan
39.1 | 52.2
52.2 | 52.2
Jiangxi
26.2 | 41.0
47.5 | 50.8
Table 4: Best post-training performance vs. original performance of Qwen2.5-Omni and Qwen3-Omni on the Chinese regional subset (sorted by mean trained accuracy). Guangdong* includes Hong Kong and Macau. Regions printed in violet indicate those with official regional and indigenous languages. Format: original ∣ trained.
Region
Qwen2.5-Omni
Qwen3-Omni
Mode
Qwen2.5-Omni
Qwen3-Omni
Gyeonggi
41.5 | 51.8
52.3 | 52.3
Yukjabaegi
50.2 | 56.5
55.9 | 66.1
Gangwon
58.7 | 61.9
50.8 | 60.3
Menari
46.7 | 54.4
50.9 | 56.2
Gyeongsang
39.7 | 53.8
47.4 | 60.3
Susimga
37.1 | 61.0
40.0 | 65.7
Jeolla
50.0 | 56.2
56.0 | 65.6
Jin-gyeong
42.2 | 54.1
55.0 | 55.0
Jeju
44.4 | 59.3
54.3 | 59.3
Gyeong
28.6 | 71.4
57.1 | 64.3
Hwanghae
45.2 | 65.8
50.0 | 69.2
Ban-gyeong
50.0 | 55.3
46.1 | 57.9
Table 5: Korean accuracy stratified by region (left) and melodic mode ( tori , right). Format: original ∣ trained.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Language distribution of the synthesized corpus plotted on a logarithmic scale. The label oth aggregates all remaining low-frequency languages with fewer than 100 dialogues.
Language [#]
Qwen2.5-omni
Qwen3-omni
Language [#]
Qwen2.5-omni
Qwen3-omni
ru (Russian) [88]
30.67 | 48.00
50.67 | 54.67
ro (Romanian) [76]
12.16 | 58.11
48.65 | 67.57
hu (Hungarian) [41]
25.00 | 42.65
39.71 | 47.06
id (Indonesian) [35]
37.50 | 55.00
42.50 | 56.25
sv (Swedish) [20]
23.75 | 51.25
41.25 | 46.25
bg (Bulgarian) [0]
31.43 | 45.71
52.86 | 50.00
da (Danish) [0]
25.33 | 50.67
36.00 | 56.00
es (Spanish) [36476]
25.97 | 54.55
37.66 | 51.95
en (English) [29936]
20.90 | 52.24
58.21 | 65.67
pt (Portuguese) [13911]
33.33 | 47.62
41.67 | 63.10
hi (Hindi) [1864]
35.62 | 54.79
60.27 | 61.64
tr (Turkish) [1624]
26.32 | 43.42
52.63 | 50.00
Appendix
Table 6: Per-language accuracy (original ∣ trained) for all benchmark languages. Rows are grouped by the Qwen2.5-Omni training-count vs. gain quadrant (median splits on log10(n+1) and Δ=trained−original ): green = low-resource/high-gain, blue = high-resource/high-gain, grey = low-resource/low-gain, red = high-resource/low-gain; within each block, sorted by n (high → low).
Figure 7: Two systematic failure modes of LALMs on Raag Abhogi and A Li Ge Cai: misidentification & prompt-induced bias.
Observation
Count
Proportion (%)
Correct option contains Undulating
178/191
93.19
CantoCore contains only Undulating
95/178
53.37
CantoCore is multi-coded and includes Undulating
83/178
46.63
CantoCore does not include Undulating
0/178
0.00
Appendix
Table 7: Answer-label concentration and CantoCore alignment.
Full response
Options only
Configuration
Valid
Acc (%)
Acc (%)
Original
176
44.38 / 44.89
48.31
Default SFT
163
41.01 / 44.79
42.13
Lang. Loss
166
44.38 / 47.59
42.70
Text DPO
168
53.93 / 57.14
42.13
Audio DPO
161
42.70 / 47.20
39.89
Appendix
Table 8: Qwen2.5-Omni and Qwen3-Omni results on 178 Undulating-positive questions. "Valid" denotes valid response count ( N=178 ). Accuracy is reported as Overall/Valid-only (%) for Full response, and Overall (%) for Options only. A dash denotes no evaluated setting.
Independent Researcher · The Hong Kong University of Science and Technology, Hong Kong SAR, China · The Chinese University of Hong Kong, Hong Kong SAR, China
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · National Taiwan University, Taiwan · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE), Taiwan