Distributional metrics such as Fréchet Audio Distance cannot score individual music clips and correlate poorly with human judgments, while the only per-sample learned metric achieving high human correlation is closed-source. We introduce MUQ-EVAL, an open-source per-sample quality metric for AIgenerated music built by training lightweight prediction heads on frozen MuQ-310M features using MusicEval, a dataset of generated clips from 31 text-to-music systems with expert quality ratings. Our simplest model, frozen features with attention pooling and a two-layer MLP, achieves system-level SRCC = 0.957 and utterance-level SRCC = 0.838 with human mean opinion scores. A systematic ablation over training objectives and adaptation strategies shows that no addition meaningfully improves the frozen baseline, indicating that frozen MuQ representations already capture quality-relevant information. Encoder choice is the dominant design factor, outweighing all architectural and training decisions. LoRA-adapted models trained on as few as 150 clips already achieve usable correlation, enabling personalized quality evaluators from individual listener annotations. A controlled degradation analysis reveals selective sensitivity to signal-level artifacts but insensitivity to musical-structural distortions. Our metric, MUQ-EVAL, is fully open-source, outperforms existing open per-sample metrics, and runs in real time on a single consumer GPU. Code, model weights, and evaluation scripts are available at https://github.com/dgtql/MuQ-Eval.
Figures & tables
Fig. 1 : Training dynamics: (a) validation SRCC vs. epoch and (b) training loss vs. epoch for all configurations (fold 0). All MuQ variants converge to similar validation SRCC.
Method
SRCC sys
PCC sys
τsys
FAD (VGGish) †
0.295 ± .031 ‡
—
0.14
Audiobox Aes. ∗
0.200
—
—
A1 (Frozen+MSE)
0.957 ± .014
0.960 ± .013
0.839 ± .019
[.898, .986]
[.916, .980]
[.716, .910]
A2 (Frozen+Ord.)
0.953 ± .015
0.963 ± .007
0.839 ± .034
[.873, .986]
[.912, .982]
[.696, .915]
TABLE I : System-level correlation with human MOS (31 TTM models, MusicEval 5-fold CV). Best in bold ; 95% BCa bootstrap CIs in brackets.
Fig. 2 : System-level scatter: predicted MuQ-Eval scores vs. human MOS for 31 TTM models (A1, fold 0). Dashed: linear fit; dotted: y=x reference.
Method
MI PCC
MI SRCC
TA PCC
TA SRCC
A1
.828 ± .017
.838 ± .016
.594 ± .041
.591 ± .034
A2
.824 ± .020
.833 ± .019
.604 ± .024
.601 ± .023
A3a
.827 ± .020
.835 ± .019
.614 ± .029
.614 ± .029
A3b
.829 ± .019
.840 ± .019
.588 ± .036
.585 ± .034
A3c
.829 ± .019
.840 ± .019
.587 ± .038
.584 ± .035
A4
.808 ± .019
.825 ± .015
.613 ± .034
.619 ± .028
TABLE II : Utterance-level correlation with human MOS (MI dimension, ∼ 385 clips/fold, 5-fold CV). 95% BCa CIs in brackets.
Fig. 3 : Utterance-level scatter: per-clip predicted vs. human MOS (MI dimension, A1, fold 0, ∼ 385 clips). Higher density along the diagonal indicates strong per-sample agreement.
Fig. 4 : Progressive ablation: system-level (solid bars) and utterance-level (hatched bars) SRCC(MI) per experiment. Trend lines connect values across experiments. All MuQ variants cluster tightly at both granularities, confirming that progressive complexity does not yield cumulative improvement.
Comparison
z
p
Cohen’s q
Sig.?
A3c vs. A1
−0.485
0.284
0.074
No
A3c vs. A2
1.466
0.002
0.109
Yes
A3c vs. A3a
−3.714
0.009
0.084
Yes
A3c vs. A3b
−0.766
0.496
0.005
No
A3c vs. A4
2.732
0.200
0.192
No
TABLE IV : Steiger test for dependent correlations (system-level SRCC). Bonferroni αadj=0.01 .
Degradation
Severe
Moderate
Mild
MP3 compression
0.97
0.84
0.52
Gaussian noise
0.99
0.89
0.39
Pitch shift
0.51
0.43
0.43
Tempo stretch
0.58
0.49
0.46
TABLE V : Degradation concordance (A3b, fold 0, N=114 clips). Values are the fraction of pairs where the original scores higher than the degraded version. Chance level is 0.50.
Fig. 5 : Degradation concordance heatmap (A3b, fold 0). The metric reliably detects signal-level artifacts (MP3, noise) at severe levels but is insensitive to musical-structural distortions (pitch, tempo) at all severities.
Method
Encoder
Params
VRAM
Inf. (ms)
A1
MuQ-310M
∼ 1M
∼ 3 GB
∼ 35
A3a–c
MuQ-310M
∼ 2M
∼ 4 GB
∼ 35
A4
MERT-95M
95M
∼ 12 GB
∼ 20
TABLE VI : Computational cost comparison.
Ntrain
A1 (Frozen+MSE)
A3a (LoRA+Ord.)
SRCC
PCC
SRCC
PCC
100
0.635
0.623
0.757
0.742
150
0.685
0.679
0.761
0.751
250
0.731
0.720
0.788
0.778
500
0.781
0.763
0.808
0.799
750
0.814
0.804
0.823
0.815
TABLE VII : Data efficiency: utterance-level MI correlation as a function of training set size (fold 0). A3a (LoRA) achieves higher correlation at every sample size and matches A1@500 performance with only 250 samples.
Fig. 6 : Data efficiency: utterance-level MI SRCC and PCC as a function of training set size. A3a (LoRA) consistently outperforms A1 (frozen) and requires ∼50% fewer samples to reach comparable performance (annotated crossover: A3a@250 > A1@500).
Open audio-language models are used as judges of generated music: given a clip and its text prompt, the model rates the clip, and the judge is validated by how well its ratings correlate with expert ratings. The correlation does not show what the judge responds to: the music, the text prompt, or the way the question is asked. We propose four probes that test each of these, three of them read against a control computed from the human ratings: removing the audio, replacing the clip with another model's clip for the same prompt, swapping the order of two clips in pairwise judging, and rewording the instruction. We apply them to five open judges on all 2,748 clips of MusicEval, and the last two to the SongEval songs. Without audio, no judge exceeds what the prompt alone predicts from the human ratings, and for prompt alignment every judge falls below it. With audio, every judge follows the clip it hears when rating prompt alignment, two do when rating musical quality, and the strongest judge's quality score tracks the experts' alignment rating more than their quality rating. The pairwise verdicts of the four smaller judges follow the position of a clip in the pair (its slot) rather than its quality on both rating sets. For musical quality, rewording the instruction moves the correlation of most judges as much as changing the judge. MusicEval's prompt assignment bounds any prompt-only judge at the model level. We recommend reporting the four probes with every music judge.
Mateusz Modrzejewski, Tomasz Radzikowski
Institute of Computer Science, Warsaw University of Technology
Current music generative models can produce high-quality music, but does this ability imply that they ``understand'' the musical qualities of their outputs, and is that understanding aligned with human evaluation? Previous attempts to use the likelihood of a generative model to evaluate music, an approach commonly used in text, have proven unsuccessful, leading researchers to rely on standalone supervised music evaluation models. In this paper, we answer this question affirmatively: we show that a model's intrinsic signals---derived from its hidden representations and predictions---are strongly correlated with human ratings. In particular, we study MusicGen and consider three types of features: (1) prediction loss, (2) prediction entropy, and (3) concepts extracted from the model using a sparse autoencoder (SAE). Using these features, we train a lightweight prediction model to estimate subjective ratings. We evaluate these features both individually and in combination. We hypothesize that these signals parallel the listening process: the temporal and frequency-domain structure of loss and entropy reflects listeners' expectation and surprise, while gradient directions in SAE latent space predict perceived quality. Experiments on five human-evaluation benchmarks spanning continuous ratings and pairwise preferences confirm this hypothesis, with SAE latents carrying most of the predictive signal.
Xiaosha Li, Chun Liu, Ziyu Wang
Georgia Institute of Technology · ByteDance Inc. · New York University · MBZUAI
Music popularity prediction has attracted growing research interest, with relevance to artists, platforms, and recommendation systems. However, the explosive rise of AI-generated music platforms has created an entirely new and largely unexplored landscape, where a surge of songs is produced and consumed daily without the traditional markers of artist reputation or label backing. Key, yet unexplored in this pursuit is aesthetic quality. We propose APEX, the first large-scale multi-task learning framework for AI-generated music, trained on over 211k songs (10k hours of audio) from Suno and Udio, that jointly predicts engagement-based popularity signals - streams and likes scores - alongside five perceptual aesthetic quality dimensions from frozen audio embeddings extracted from MERT, a self-supervised music understanding model. Aesthetic quality and popularity capture complementary aspects of music that together prove valuable: in an out-of-distribution evaluation on the Music Arena dataset, comprising pairwise human preference battles across eleven generative music systems unseen during training, including aesthetic features consistently improves preference prediction, demonstrating strong generalisation of the learned representations across generative architectures.
Jaavid Aktar Husain, Dorien Herremans
AMAAI Lab, Singapore University of Technology and Design