CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction

Yinghao Ma; Haiwen Xia; Hewei Gao; Weixiong Chen; Yuxin Ye; Yuchen Yang; Sungkyun Chang; Mingshuo Ding

cs.SDFeb 28, 2026

CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction

Authors: Yinghao Ma, Haiwen Xia, Hewei Gao, Weixiong Chen, Yuxin Ye, Yuchen Yang, Sungkyun Chang, Mingshuo Ding, +4 more

Organizations: Queen Mary University of London, London, UK · Peking University, Beijing, China · Technical University of Munich, Munich, Germany · Technical University of Denmark, Lyngby, Denmark · Beijing University of Post and Telecommunications, Beijing, China · Soochow University, Suzhou, China · University of Manchester, Manchester, UK · Hong Kong University of Science and Technology, Hong Kong, China

Abstract

While music generation models have evolved to handle complex multimodal inputs mixing text, lyrics, and reference audio, evaluation mechanisms have lagged behind. In this paper, we bridge this critical gap by establishing a comprehensive ecosystem for music reward modeling under Compositional Multimodal Instruction (CMI), where the generated music may be conditioned on text descriptions, lyrics, and audio prompts. We first introduce CMI-Pref-Pseudo, a large-scale preference dataset comprising 110k pseudo-labeled samples, and CMI-Pref, a high-quality, human-annotated corpus tailored for fine-grained alignment tasks. To unify the evaluation landscape, we propose CMI-RewardBench, a unified benchmark that evaluates music reward models on heterogeneous samples across musicality, text-music alignment, and compositional instruction alignment. Leveraging these resources, we develop CMI reward models (CMI-RMs), a parameter-efficient reward model family capable of processing heterogeneous inputs. We evaluate their correlation with human judgment scores on musicality and alignment on CMI-Pref along with previous datasets. Further experiments demonstrate that CMI-RM not only correlates strongly with human judgments, but also enables effective inference-time scaling via top-k filtering. Code is available at GitHub (https://github.com/Haiwen-Xia/CMI-RewardBench). Model weights: CMI-RM (https://huggingface.co/HaiwenXia/CMI-RM). Datasets: CMI-Pref-Pseudo (https://huggingface.co/datasets/HaiwenXia/cmi-pref-pseudo) and CMI-Pref (https://huggingface.co/datasets/HaiwenXia/cmi-pref)

Figures & tables

Figure 1: Reward models should act as proxies for human evaluation under compositional multimodal instructions (CMI). Human judges evaluate a generated waveform with respect to the provided prompt—text descriptions, lyrics, and/or reference audio—considering both musicality and instruction alignment. However, current reward models are typically fragmented: they either score musicality from audio alone or capture alignment for a single modality pair, and thus cannot reliably proxy human judgments across flexible CMI inputs.

Data set	PAM	MusicEval	SongEval	AIME	MusicPref	Music Arena	CMI-Pref-Pseudo	CMI-Pref
Text Condition	✔	unavailable	✗	✔	✔	✔	✔	✔
Lyrics Condition	✗	✗	unavailable	✗	✗	✔	✔	✔
Audio Condition	✗	✗	✗	✗	✗	✗	✔	✔
#samples	500	2748	2,399	15,600	2,520	2,800	110k	4,027
#Testing samples	500	413	-	-	-	1,340	-	500
#Duration of audio(hours)	0.83	16.62	140.54	16.67	unavailable	88.30	808.85	133.80

Table 1: Statistics of CMI-Pref and CMI-Pref-Pseudo compared with previous datasets. Samples here refer to pairs for preference datasets and individual audio clips for MOS datasets.

Figure 2: Model architecture of compositional music instruction reward model (CMI-RM).

Musicality	PAM (Music Subset)			MusicEval (Test Split)			Music Arena	CMI-Pref
Method&Model	LCC	SRCC	K-Tau	LCC	SRCC	K-Tau	ACC	ACC
PAM score	0.5873	0.6099	0.4367	0.6466	0.6724	0.4874	63.13%	65.40%
audiobox-CE	0.5283	0.5204	0.3665	0.6393	0.6599	0.4830	64.25%	71.80%
audiobox-CU	0.4645	0.4704	0.3279	0.6272	0.6764	0.4950	67.76%	71.40%
audiobox-PC	0.2505	0.2230	0.1552	0.1225	0.0768	0.0514	58.73%	59.00%
audiobox-PQ	0.4636	0.4513	0.3166	0.6016	0.6335	0.4620	67.54%	73.80%

Table 2: Musicality results on CMI-RewardBench. For each metric, the best performance is marked in boldface and second with underline.

	PAM (Music Subset)			CMI-Pref (Subsets)
Text-Music	✔	✔	✔	✔	✔	✔	✔	CMI-Pref w/o Audio	CMI-Pref w/ Audio
Lyrics-Music	✗	✗	✗	✗	✔	✗	✔
Audio-Music	✗	✗	✗	✗	✗	✔	✔
Metrics	LCC	SRCC	K-Tau	ACC	ACC	ACC	ACC	ACC	ACC
CLAP score (default)	0.4692	0.4517	0.3171	60.80%	64.00%	-	-	62.40%	-
CLAP score (music)	0.3192	0.2881	0.1978	67.20%	73.60 %	-	-	70.40%	-

Table 3: Benchmark Results on Compositional Multimodal Instruction Alignment.

Training variant	PAM	MusicEval	Music Arena	CMI-Pref
Metric $\uparrow$	Mean SRCC	SRCC	Acc.	Mean Acc.
Distill only	0.3925	0.5117	64.25%	71.10%
Distill+CMI-Pref	0.6116	0.7315	71.41%	75.90%
Distill+MusicEval	0.4338	0.3460	64.78%	69.00%
Distill+Both	0.5464	0.8266	73.43%	76.05%
Scratch+Both	0.2630	0.4986	71.34%	72.15%

Table 4: Aggregated ablation results with a fixed CMI-RM architecture. Mean SRCC on PAM averages the musicality and text–music alignment regression tasks. Music Arena accuracy uses musicality score

s_{\text{MUS}}

for pairwise prediction. Mean Acc. on CMI-Pref averages musicality and alignment preference accuracy.

Panel of Figure 3 — Figure 3: Pairwise preference matrices for test-time scaling with RM reranking. Each cell reports the percentage of trials in which annotators preferred system $A$ (row) over system $B$ (column).

Model	Alignment	AudioBox		SongEval
Model	MuQ-MuLan $\uparrow$	CE $\uparrow$	CU $\uparrow$	Mus $\uparrow$
MusicGen	0.298	6.046	6.989	2.143
MusicGen (N=3)	0.323	6.405	7.255	2.213
MusicGen (N=10)	0.339	6.647	7.416	2.273
Stable Audio	0.293	5.567	7.170	2.055
Stable Audio (N=3)	0.301	5.732	7.245	2.078

Table 5: Test-time scaling results of MusicGen-small and Stable-audio-open-small on objective metrics on text-music alignment (MuQ-Mulan) and musicality (AudioBox, and SongEval).

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Distribution of audio duration. The x-axis denotes generated audio duration (seconds), and the y-axis denotes sample count ( $N=6458$ unique audios). To improve readability, bar heights are clipped at 500 and overflow counts are annotated in red; a tiny fraction of samples with duration $>$ 300 seconds is omitted from the plot.

Figure 5: Distribution of total listening time per audio. “Total listening time” denotes accumulated playback time spent by annotators on one audio. Starting from 8,054 records (4,027 votes $\times$ 2 audios), we remove 8 outliers with listening time $>$ 500 seconds and report the remaining $N=8046$ samples.

Figure 6: Correlation between audio duration (x-axis) and total listening time (y-axis), computed on $N=8046$ filtered samples. The dashed diagonal denotes $y=x$ (full listening).

Metric	Instruction Following	Music Quality
Agreement Count	445	466
Disagreement Count	199	178
Agreement Rate	0.691	0.724
Krippendorff’s $\alpha$	0.382	0.447

Appendix

Table 6: Inter-annotator agreement on overlapping votes, computed over 644 vote pairs from 492 comparisons with multiple annotations.

Metric	Alignment Preference	Musicality Preference
Agreement Rate	0.750	0.752
Krippendorff’s $\alpha$	0.500	0.504
Wilson 95% CI	(70.9%, 78.5%)	(71.1%, 78.7%)

Appendix

Table 7: Reliability of CMI-Pref-test, computed by comparing three annotators’ re-annotations of all 500 test samples with the original test labels.

Figure 7: Distribution of annotators’ confidence scores for instruction following and music quality.

Metric	Value
#Votes with same preference	3309
#Votes with different preference	718
Agreement rate	0.822
Avg. confidence (same Pref.)	IF: 3.674, MQ: 3.734
Avg. confidence (diff. Pref.)	IF: 3.171, MQ: 3.311

Appendix

Table 8: Agreement between instruction-following and music-quality preferences, and the corresponding average confidence.

Configuration	Win A (%)	Win B (%)	Tie (%)
Musicality
Original $(A,B)$	51.96	40.96	7.08
Reversed $(B,A)$	59.27	33.48	7.25
Agreed (Filtered)	57.80	36.97	5.22
Alignment
Original $(A,B)$	33.65	43.71	22.64

Appendix

Table 9: Distribution of pseudo-labels across varying query configurations. “Original” and “Reversed” denote the presentation order of the audio pair. “Agreed” represents the subset of labels where the model’s preference remained consistent across both permutations. The discrepancy between Original and Reversed highlights positional bias, which is rectified in the Agreed set.

Dimension	Positive Descriptors	Negative Descriptors
Melody	Catchy, Memorable, Distinct	Repetitive, Generic, Wandering
Structure	Coherent, Progression, Build-up	Disjointed, Abrupt, Random Loops
Rhythm	Groovy, Steady, Driving	Off-beat, Unstable, Chaotic
Audio Quality	Clean mix, Balanced, High-fidelity	Harsh, Muddy, Distorted/Clipping
Vocals	Natural, Clear articulation	Robotic, Slurred, Artifacts

Appendix

Table 10: Taxonomy of musical attributes used in the annotation process.

Training pipeline	Musicality Acc.	Alignment Acc.
Pseudo pretrain (before FT)	71.0%	71.8%
Pseudo pretrain $\rightarrow$ FT (no smoothing)	75.2%	71.0%
Direct FT (from scratch)	75.0%	69.3%
Pseudo pretrain $\rightarrow$ FT (with smoothing)	77.8%	74.0%

Appendix

Table 11: Real-test reward-head accuracy (%) under different training pipelines.

Figure 9: Cross-entropy loss of checkpoints trained on CMI-Pseudo. Pseudo indicates metrics on the CMI-Pseudo validation set, and Real denotes results on the test set of CMI-Pref.

Figure 10: Accuracy of checkpoints trained on CMI-Pseudo. Pseudo indicates metrics on the CMI-Pseudo validation set, and Real denotes results on the test set of CMI-Pref.

Pseudo Size	Pref-Test Acc. $\uparrow$	Pref-Test CE $\downarrow$	Mus-Acc $\uparrow$	Align-Acc $\uparrow$
4k	0.582	0.674	0.726	0.664
8k	0.640	0.656	0.750	0.692
16k	0.605	0.655	0.754	0.700
32k	0.646	0.627	0.782	0.700
64k	0.711	0.593	0.778	0.740
110k	0.720	0.559	0.761	0.735

Appendix

Table 12: Pseudo data size ablation under the fixed epoch-progress protocol (same-epoch comparison).

Pseudo Size	Pref-Test Acc. $\uparrow$	Pref-Test CE $\downarrow$	Mus-Acc $\uparrow$	Align-Acc $\uparrow$
4k	0.588	0.724	0.742	0.696
8k	0.623	0.685	0.768	0.694
16k	0.646	0.636	0.760	0.682
32k	0.566	0.725	0.752	0.710
64k	0.711	0.593	0.778	0.740
110k	0.712	0.570	0.762	0.714

Appendix

Table 13: Pseudo data size ablation under the fixed-step protocol (same-step comparison).

Pseudo-label setup	PAM Mean SRCC	MusicEval SRCC	Music Arena Acc	CMI-Pref Mean Acc
Smooth (Qwen only)	0.529	0.742	0.701	0.756
Mix (70k Qwen + 40k Gemini)	0.550	0.761	0.699	0.768
Smooth + Mix	0.450	0.765	0.707	0.775

Appendix

Table 14: Aggregated results under different pseudo-label source designs.

Variant	Trainable Params	PAM Mean SRCC	MusicEval SRCC	MusicArena Acc	CMI-Pref Mean Acc
Small	8.3M	0.540	0.816	0.680	0.744
Base	38M	0.578	0.811	0.729	0.746
Large	102M	0.589	0.848	0.719	0.765

Appendix

Table 15: Model size scaling of the trainable reward head.

Metric	Arena T	Arena T+L	CMI T	CMI T+L	CMI T+L+A	CMI T+A
$\Delta$ Acc.	+2.7%	-1.4%	-4.0%	0.0%	-7.2%	-6.8%

Appendix

Table 16: Accuracy change from replacing the default text encoder with Flan-T5-Large. T/L/A denote text, lyrics, and audio conditions.

Datasets	PAM	MusicEval	Music Arena	CMI-Pref
Metrics $\uparrow$	Mean SRCC	SRCC	ACC	Mean ACC
None	0.5069	0.4659	73.73%	75.3%
Tanh	0.6024	0.4702	73.8%	74.8%
Linear	0.4595	0.4691	73.58%	74.40%
Ordinal	0.6079	0.4665	72.31%	75.80%

Appendix

Table 17: Aggregated results on mapping ablations

		Duration (s), Acc (%)
Metric	Variant	[10,30)	[30,60)	[60,120)	[120,500]
Music Arena	n	119	88	531	600
	first 10	72.3	53.4	64.2	59.5
	mean 10	75.6	48.9	75.7	67.0
	first 120	75.6	53.4	81.1	67.3
CMI-Pref Musicality	n	196	77	73	154

Appendix

Table 18: Accuracy (%) by audio duration across inference variants. Highest per bin is bold .

Pair	[10,30)	[30,60)	[60,120)	[120,500]	Overall
	n = 1311	n = 824	n = 346	n = 1639	n = 4120
RMSE
f10 vs f120	0.3022	0.3960	0.5465	0.7734	0.5687
m10 vs f120	0.2289	0.2067	0.4292	0.5014	0.3751
f10 vs m10	0.2836	0.3583	0.3485	0.4478	0.3758
LCC

Appendix

Table 19: Effect of inference duration on predicted Musicality (RMSE & LCC between three configurations). Statistics for first 120 is shown

Pair	[10,30)	[30,60)	[60,120)	[120,500]	Overall
	n = 1311	n = 824	n = 346	n = 1639	n = 4120
RMSE
f10 vs f120	0.2726	0.3104	0.4164	0.6110	0.4539
m10 vs f120	0.2084	0.1777	0.3082	0.3410	0.2727
f10 vs m10	0.2568	0.2922	0.2928	0.4022	0.3311
LCC

Appendix

Table 20: Effect of inference duration on predicted Text-Music Alignment (RMSE & LCC between three configurations). Statistics for first 120 is shown

Method	Accuracy (%)	AUC	MSE
Single Metric Threshold (Threshold = 0)
Musicality Diff	73.4	0.8001	0.1884
Alignment Diff	69.7	0.7714	0.2031
Mus + Ali Diff (Equal)	72.6	0.7962	0.1851
Regression Models (5-fold CV)
Logistic Regression	73.1	0.7991	0.1845

Appendix

Table 21: Regression analysis of overall human preference on Music Arena using predicted Musicality (Mus) and Alignment (Ali) scores. The results indicate that general user preference is predominantly driven by musicality.

Prompt Modality	w/ Condition Acc (%)	w/o Condition Acc (%)	$\Delta$
Text + Audio + Lyrics	82.40	68.80	+13.60
Text + Audio w/o Lyrics	79.20	68.80	+10.40
Text + Lyrics w/o Audio	72.00	76.00	-4.00
Text only	68.80	67.20	+1.60
Overall ( $N=500$ )	75.60	70.20	+5.40

Appendix

Table 22: Ablation study on prompt conditions for Musicality prediction on CMI-Pref. We compare models trained and evaluated with and without compositional conditions across different prompt modalities (

n=125

for each subset).

Backbone / Comparison	Ordering	$p$ -value
Stable Audio	GT $>$ Top-10	0.003
MusicGen	GT $>$ Top-1	0.0001
Stable Audio	GT $>$ Top-1	0.0001
MusicGen	Top-10 $>$ Top-3	0.016
MusicGen	Top-3 $>$ Top-1	0.101
Stable Audio	Top-10 $>$ Top-3	0.890

Appendix

Table 23: Significance checks for human listening results in test-time scaling.

Model	Total	Time Period (Month)					Data Type
Model	(%)	Jul-Aug	Sep	Oct	Nov	Dec	Instru	Vocal
PAM score	63.13	69.13	75.41	58.56	53.14	55.27	56.84	68.35
audiobox-CE	64.25	67.79	73.77	63.36	60.14	55.27	64.25	64.25
audiobox-CU	67.76	71.36	77.0	66.78	60.14	60.73	68.04	67.53
audiobox-PC	58.73	63.75	70.49	55.82	54.54	48.00	52.88	63.57
audiobox-PQ	67.54	68.01	74.86	67.47	65.04	63.27	68.04	67.12

Appendix

Table 24: Benchmark Musicality ACC Results on Music Arena (Time & Data Type Analysis).

Musicality	Total	Confidence Level			Data Type
Musicality	(500)	Conf $<3$ (66)	Conf $=3$ (128)	Conf $>3$ (306)	Instru (250)	Vocal (250)
PAM score	65.40%	63.64%	64.84%	66.01%	67.60%	63.20%
audiobox-CE	71.80%	63.64%	72.66%	73.20%	70.80%	72.80%
audiobox-CU	71.40%	66.67%	71.88%	72.22%	68.80%	74.00%
audiobox-PC	59.00%	46.97%	54.69%	63.40%	60.00%	58.00%
audiobox-PQ	73.80%	74.24%	78.13%	71.90%	73.20%	74.40%

Appendix

Table 25: Benchmark Musicality ACC Results on CMI-Pref (Ablation Analysis).

Text-Music Alignment	Total	Confidence Level			Data Type
Text-Music Alignment	(250)	Conf $<3$ (47)	Conf $=3$ (58)	Conf $>3$ (145)	Instru (125)	Vocal (125)
audiobox-CE	60.00%	45.95%	63.64%	61.90%	56.80%	63.20%
audiobox-CU	59.60%	54.05%	63.64%	59.18%	53.60%	65.60%
audiobox-PC	56.00%	35.13%	59.09%	59.86%	55.20%	56.80%
audiobox-PQ	59.60%	54.05%	65.15%	58.50%	55.20%	64.00%
CLAP score	62.40%	51.35%	59.09%	66.67%	60.80%	64.00%

Appendix

Table 26: Benchmark Text-Music Alignment ACC Results on CMI-Pref(Ablation Analysis).

	Instrumental		Song
Model	w/ audio	w/o audio	w/ audio	w/o audio
suno-v5	212	221	213	288
mureka-o2	0	0	15	163
suno-v4.5-plus	214	221	216	288
mureka-v7.5	7	117	8	211
minimax-music-2	0	0	27	429

Appendix

Table 28: Generation counts per model by modality

	Inst. (w/ audio)		Inst. (w/o audio)		Song (w/ audio)		Song (w/o audio)
Model	Alignment	Musicality	Alignment	Musicality	Alignment	Musicality	Alignment	Musicality
suno-v5	1	1	3	3	3	3	2	3
mureka-o2	-	-	-	-	2	1	4	1
suno-v4.5-plus	2	2	1	2	4	4	3	5
mureka-v7.5	6	4	2	1	7	6	5	2
minimax-music-2.0	-	-	-	-	1	2	1	4

Appendix

Table 29: Per-column ranks (1=best). Inst. refers to instrumental music generation, Song referes to song generation, w/audio indicates if an audio prompt is used. Unavailable modalities are left out.

Explore similar work

TuneJury: An Open Metric for Improving Music Generation Preference Alignment
Jun 15, 2026Yonghyun Kim, Junwon Lee, Haiwen Xia +5Text-to-Music Generation Music Generation Evaluation
Georgia Tech · KAIST · Peking University +3
MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques
Aug 12, 2026Jiabao Zhuang, Changhao Jiang, Hanchen Wang +11Music Generation Evaluation Reward Modeling
Fudan NLP Group, Fudan University
Rubric-Based Optimization for Text-to-Music Generation
Oct 2, 2026Ping Wang, Guang Yang, Shao-Rong Su +3Text-to-Music Generation Rubric-Based RL
Paul G. Allen School of Computer Science & Engineering, University of Washington · Department of Electrical and Computer Engineering, University of Washington · Allen Institute for AI

Jun 15, 2026J/K move · Enter open · S save

TuneJury: An Open Metric for Improving Music Generation Preference Alignment

Yonghyun Kim, Junwon Lee, Haiwen Xia, Yinghao Ma+4

Georgia Tech · KAIST · Peking University+3

Text-to-Music Generation Music Generation Evaluation Reward Modeling Pairwise Comparison Pairwise Preference Evaluation Pairwise Ranking+2

Open paper AskPDF arXiv

We introduce TuneJury, an open, instance-level pairwise reward model for text-to-music that predicts a music preference score from a text prompt and an audio clip. The released checkpoint is trained on publicly available human-preference labels covering arena-style (A vs. B) votes, metric-alignment preference pairs, crowdsourced pairwise comparisons, and expert aesthetic ratings. The predicted score margin between two clips is well calibrated on our held-out test split, supporting data filtering via a simple score threshold. TuneJury generalizes to both held-out test pairs and out-of-distribution benchmarks, remaining competitive with prior baselines on the latter. For generators released after training, we introduce anchor calibration, a post-hoc, per-system Bradley-Terry calibration that recovers agreement at substantially better data efficiency than from-scratch retraining. The same frozen reward drives consistent reward-axis gains across three downstream applications: inference-time best-of-N selection, DITTO-style latent optimization, and expert-iteration post-training. TuneJury is available at https://github.com/yonghyunk1m/TuneJury.

Abstract

Explore similar work

TuneJury: An Open Metric for Improving Music Generation Preference Alignment

MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques

Rubric-Based Optimization for Text-to-Music Generation