MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation

Di Zhu; Zixuan Li

cs.AIMar 24, 2026

MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation

Authors: Di Zhu, Zixuan Li

Organizations: Stevens Institute of Technology · Columbia University

Abstract

Distributional metrics such as Fréchet Audio Distance cannot score individual music clips and correlate poorly with human judgments, while the only per-sample learned metric achieving high human correlation is closed-source. We introduce MUQ-EVAL, an open-source per-sample quality metric for AIgenerated music built by training lightweight prediction heads on frozen MuQ-310M features using MusicEval, a dataset of generated clips from 31 text-to-music systems with expert quality ratings. Our simplest model, frozen features with attention pooling and a two-layer MLP, achieves system-level SRCC = 0.957 and utterance-level SRCC = 0.838 with human mean opinion scores. A systematic ablation over training objectives and adaptation strategies shows that no addition meaningfully improves the frozen baseline, indicating that frozen MuQ representations already capture quality-relevant information. Encoder choice is the dominant design factor, outweighing all architectural and training decisions. LoRA-adapted models trained on as few as 150 clips already achieve usable correlation, enabling personalized quality evaluators from individual listener annotations. A controlled degradation analysis reveals selective sensitivity to signal-level artifacts but insensitivity to musical-structural distortions. Our metric, MUQ-EVAL, is fully open-source, outperforms existing open per-sample metrics, and runs in real time on a single consumer GPU. Code, model weights, and evaluation scripts are available at https://github.com/dgtql/MuQ-Eval.

Figures & tables

Fig. 1 : Training dynamics: (a) validation SRCC vs. epoch and (b) training loss vs. epoch for all configurations (fold 0). All MuQ variants converge to similar validation SRCC.

Method	SRCC ${}_{\text{sys}}$	PCC ${}_{\text{sys}}$	$\tau_{\text{sys}}$
FAD (VGGish) †	0.295 $\pm$ .031 ‡	—	0.14
Audiobox Aes. ∗	0.200	—	—
A1 (Frozen+MSE)	0.957 $\pm$ .014	0.960 $\pm$ .013	0.839 $\pm$ .019
	[.898, .986]	[.916, .980]	[.716, .910]
A2 (Frozen+Ord.)	0.953 $\pm$ .015	0.963 $\pm$ .007	0.839 $\pm$ .034
	[.873, .986]	[.912, .982]	[.696, .915]

TABLE I : System-level correlation with human MOS (31 TTM models, MusicEval 5-fold CV). Best in bold ; 95% BCa bootstrap CIs in brackets.

Fig. 2 : System-level scatter: predicted MuQ-Eval scores vs. human MOS for 31 TTM models (A1, fold 0). Dashed: linear fit; dotted:

y=x

reference.

Method	MI PCC	MI SRCC	TA PCC	TA SRCC
A1	.828 $\pm$ .017	.838 $\pm$ .016	.594 $\pm$ .041	.591 $\pm$ .034
A2	.824 $\pm$ .020	.833 $\pm$ .019	.604 $\pm$ .024	.601 $\pm$ .023
A3a	.827 $\pm$ .020	.835 $\pm$ .019	.614 $\pm$ .029	.614 $\pm$ .029
A3b	.829 $\pm$ .019	.840 $\pm$ .019	.588 $\pm$ .036	.585 $\pm$ .034
A3c	.829 $\pm$ .019	.840 $\pm$ .019	.587 $\pm$ .038	.584 $\pm$ .035
A4	.808 $\pm$ .019	.825 $\pm$ .015	.613 $\pm$ .034	.619 $\pm$ .028

TABLE II : Utterance-level correlation with human MOS (MI dimension,

\sim

385 clips/fold, 5-fold CV). 95% BCa CIs in brackets.

Fig. 3 : Utterance-level scatter: per-clip predicted vs. human MOS (MI dimension, A1, fold 0,

\sim

385 clips). Higher density along the diagonal indicates strong per-sample agreement.

Step	Component	$\Delta$ SRCC ${}_{\text{sys}}$	Cohen’s $q$	$\geq 0.02$ ?
A2 $-$ A1	Ordinal CE	$-0.004$	0.050	No
A3a $-$ A2	LoRA tuning	$+0.007$	0.085	No
A3b $-$ A3a	Contrastive	$-0.004$	0.054	No
A3c $-$ A3b	Unc. weight	$-0.000$	0.003	No

TABLE III : Progressive ablation deltas (system-level SRCC). Pre-registered threshold:

\Delta\geq 0.02

. None met.

Fig. 4 : Progressive ablation: system-level (solid bars) and utterance-level (hatched bars) SRCC(MI) per experiment. Trend lines connect values across experiments. All MuQ variants cluster tightly at both granularities, confirming that progressive complexity does not yield cumulative improvement.

Comparison	$z$	$p$	Cohen’s $q$	Sig.?
A3c vs. A1	$-0.485$	0.284	0.074	No
A3c vs. A2	$1.466$	0.002	0.109	Yes
A3c vs. A3a	$-3.714$	0.009	0.084	Yes
A3c vs. A3b	$-0.766$	0.496	0.005	No
A3c vs. A4	$2.732$	0.200	0.192	No

TABLE IV : Steiger test for dependent correlations (system-level SRCC). Bonferroni

\alpha_{\text{adj}}=0.01

Degradation	Severe	Moderate	Mild
MP3 compression	0.97	0.84	0.52
Gaussian noise	0.99	0.89	0.39
Pitch shift	0.51	0.43	0.43
Tempo stretch	0.58	0.49	0.46

TABLE V : Degradation concordance (A3b, fold 0,

N=114

clips). Values are the fraction of pairs where the original scores higher than the degraded version. Chance level is 0.50.

Fig. 5 : Degradation concordance heatmap (A3b, fold 0). The metric reliably detects signal-level artifacts (MP3, noise) at severe levels but is insensitive to musical-structural distortions (pitch, tempo) at all severities.

Method	Encoder	Params	VRAM	Inf. (ms)
A1	MuQ-310M	$\sim$ 1M	$\sim$ 3 GB	$\sim$ 35
A3a–c	MuQ-310M	$\sim$ 2M	$\sim$ 4 GB	$\sim$ 35
A4	MERT-95M	95M	$\sim$ 12 GB	$\sim$ 20

TABLE VI : Computational cost comparison.

$N_{\text{train}}$	A1 (Frozen+MSE)		A3a (LoRA+Ord.)
$N_{\text{train}}$	SRCC	PCC	SRCC	PCC
100	0.635	0.623	0.757	0.742
150	0.685	0.679	0.761	0.751
250	0.731	0.720	0.788	0.778
500	0.781	0.763	0.808	0.799
750	0.814	0.804	0.823	0.815

TABLE VII : Data efficiency: utterance-level MI correlation as a function of training set size (fold 0). A3a (LoRA) achieves higher correlation at every sample size and matches A1@500 performance with only 250 samples.

Fig. 6 : Data efficiency: utterance-level MI SRCC and PCC as a function of training set size. A3a (LoRA) consistently outperforms A1 (frozen) and requires

{\sim}50\%

fewer samples to reach comparable performance (annotated crossover: A3a@250

>

A1@500).

Music Generation Evaluation

Music Generation Evaluation measures the quality, musical structure, prompt alignment, controllability, and robustness of generated music.

See all 39 papers

What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings
Oct 9, 2026Mateusz Modrzejewski, Tomasz RadzikowskiAudio-Language Model Evaluation LLM-as-a-Judge
Institute of Computer Science, Warsaw University of Technology
Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals
Sep 29, 2026Xiaosha Li, Chun Liu, Ziyu WangMusic Generation Evaluation Automated Evaluation
Georgia Institute of Technology · ByteDance Inc. · New York University · MBZUAI
APEX: Large-scale Multi-task Aesthetic-Informed Popularity Prediction for AI-Generated Music
May 5, 2026Jaavid Aktar Husain, Dorien HerremansMulti-Task Learning Music Information Retrieval
AMAAI Lab, Singapore University of Technology and Design

Oct 9, 2026J/K move · Enter open · S save

What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings

Mateusz Modrzejewski, Tomasz Radzikowski

Institute of Computer Science, Warsaw University of Technology

Audio-Language Model Evaluation LLM-as-a-Judge Music Generation Evaluation

Open paper AskPDF arXiv

Open audio-language models are used as judges of generated music: given a clip and its text prompt, the model rates the clip, and the judge is validated by how well its ratings correlate with expert ratings. The correlation does not show what the judge responds to: the music, the text prompt, or the way the question is asked. We propose four probes that test each of these, three of them read against a control computed from the human ratings: removing the audio, replacing the clip with another model's clip for the same prompt, swapping the order of two clips in pairwise judging, and rewording the instruction. We apply them to five open judges on all 2,748 clips of MusicEval, and the last two to the SongEval songs. Without audio, no judge exceeds what the prompt alone predicts from the human ratings, and for prompt alignment every judge falls below it. With audio, every judge follows the clip it hears when rating prompt alignment, two do when rating musical quality, and the strongest judge's quality score tracks the experts' alignment rating more than their quality rating. The pairwise verdicts of the four smaller judges follow the position of a clip in the pair (its slot) rather than its quality on both rating sets. For musical quality, rewording the instruction moves the correlation of most judges as much as changing the judge. MusicEval's prompt assignment bounds any prompt-only judge at the model level. We recommend reporting the four probes with every music judge.

Abstract

Music Generation Evaluation

Explore similar work

What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings

Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals

APEX: Large-scale Multi-task Aesthetic-Informed Popularity Prediction for AI-Generated Music