Open audio-language models are used as judges of generated music: given a clip and its text prompt, the model rates the clip, and the judge is validated by how well its ratings correlate with expert ratings. The correlation does not show what the judge responds to: the music, the text prompt, or the way the question is asked. We propose four probes that test each of these, three of them read against a control computed from the human ratings: removing the audio, replacing the clip with another model's clip for the same prompt, swapping the order of two clips in pairwise judging, and rewording the instruction. We apply them to five open judges on all 2,748 clips of MusicEval, and the last two to the SongEval songs. Without audio, no judge exceeds what the prompt alone predicts from the human ratings, and for prompt alignment every judge falls below it. With audio, every judge follows the clip it hears when rating prompt alignment, two do when rating musical quality, and the strongest judge's quality score tracks the experts' alignment rating more than their quality rating. The pairwise verdicts of the four smaller judges follow the position of a clip in the pair (its slot) rather than its quality on both rating sets. For musical quality, rewording the instruction moves the correlation of most judges as much as changing the judge. MusicEval's prompt assignment bounds any prompt-only judge at the model level. We recommend reporting the four probes with every music judge.
Figures & tables
Figure 1: Two of the four probes for the five judges, with 95% bootstrap intervals. Left: in the same-prompt swap, the standardized coefficient of the judge’s score on the expert rating of the clip played (filled) and of the clip it replaced (open), for the quality and the alignment target; a judge that listens follows the clip played. Right: correlation with expert quality ratings under three wordings of the instruction on the same 300 clips.
Overall quality
Align.
Model
Judge
Audio
No audio
Δw
Audio
level
Qwen2-Audio
0.13
0.03
0.06
0.17
0.42
Qwen2.5-Omni
0.11
0.01
0.05
0.19
0.32
Audio Flamingo 3
0.24
0.02
0.12
0.23
0.73
Music Flamingo
0.21
0.05
0.07
0.23
0.55
Qwen3-Omni (FP8)
0.31
0.04
0.23
0.34
0.36
Table 1: Spearman correlation with expert ratings. The Audio and No audio columns are computed over all 2,748 clips; No audio omits the audio and keeps the instruction used with audio. Δw is audio minus no audio within a model, and every interval is above zero. Align. is the prompt-alignment target with audio, within a model. Model level is over 31 models after adjusting for each model’s prompts; the expert’s 0.98 is unadjusted. The expert rows give one expert against the mean of the others, pooled or within a model. Intervals are in Table 6 .
Swap, qual.
Swap, al.
Order swap
Judge
played
repl.
played
same
2nd
agree
Qwen2-Audio
0.10
0.06
0.12
0.95
0.86
0.46
Qwen2.5-Omni
0.02
0.08
0.12
0.69
0.42
0.55
Audio Flamingo 3
0.17
0.07
0.19
0.78
0.40
0.50
Music Flamingo
0.15
0.11
0.23
0.99
0.99
0.66
Qwen3-Omni (FP8)
0.24
0.07
0.31
0.25
0.21
0.67
Table 2: The same-prompt swap columns give standardized regression coefficients of the judge’s score on the expert rating of the clip played and of the clip it replaced (2,327 swaps). The order-swap columns give the same-slot rate (the share of 600 pairs in which the judge picks the same slot in both orders, so that its preferred clip changes with the order), the share in which that slot is the second, and the share in which the order-averaged preference agrees with the experts (chance: 0.50, 0.25, and 0.50). Intervals are in Tables 13 and 12 .
Table 3: Judges with their Hugging Face revision (first ten characters), parameters, precision, and software (transf.: transformers). The score is the expected value over the digits 1 to 5 (the letters A and B in pairwise judging) under greedy decoding. Qwen3-Omni is a community FP8 quantization of Qwen/Qwen3-Omni-30B-A3B-Instruct on two RTX 4090 GPUs; the other judges run on one.
Comparison
Target
Correlations
Difference
pHolm
MusicEval, all clips
Qwen2-Audio – Qwen2.5-Omni
quality
0.13 – 0.11
0.02 [-0.02, 0.05]
0.362
Qwen2-Audio – Audio Flamingo 3
quality
0.13 – 0.24
-0.11 [-0.15, -0.08]
< 0.001
Qwen2-Audio – Music Flamingo
quality
0.13 – 0.21
-0.09 [-0.12, -0.05]
< 0.001
Qwen2-Audio – Qwen3-Omni (FP8)
quality
0.13 – 0.31
-0.18 [-0.22, -0.13]
< 0.001
Qwen2.5-Omni – Audio Flamingo 3
quality
0.11 – 0.24
-0.13 [-0.16, -0.10]
< 0.001
Appendix
Table 4: Paired tests, with Holm-corrected p and 95% paired bootstrap intervals. (a) The judge and wording comparisons in the text: differences of Spearman correlations on the same clips, adjusted within each block and target. (b) No-audio correlation (instruction unchanged) against the prompt-only floor (0.08 for quality, 0.19 for alignment) on the 2,501 clips where the floor is defined, adjusted over the ten tests. (c) Same-prompt swap: coefficient of the clip played minus that of the clip replaced (2,327 swaps), adjusted over the five judges of each target.
FP8
Unquantized (bf16)
Thinking
Cell
value
value
difference
value
difference
pooled, audio, overall
0.31
0.31
0.00 [0.00, 0.01]
0.38
0.07 [0.04, 0.10]
pooled, no audio (instruction unchanged), overall
0.04
0.04
0.01 [0.00, 0.02]
–
–
within model, audio, overall
0.30
0.30
0.00 [0.00, 0.01]
0.26
-0.04 [-0.07, -0.01]
within-model audio gain ( Δw ), overall
0.23
0.22
-0.01 [-0.02, 0.00]
–
–
swap, clip played, overall
0.24
0.25
0.01 [0.00, 0.01]
0.32
0.08 [0.04, 0.12]
Appendix
Table 5: Qwen3-Omni-30B-A3B unquantized (bf16) and Thinking against the FP8 judge of the main tables, on identical inputs: the cells of Tables 1 and 2 and the SongEval correlations, with the difference from FP8. Unquantized and FP8 clip scores correlate at 0.91 to 0.99. The Thinking readout follows its reasoning (up to 4,096 tokens); 128 of its 23,140 MusicEval judgments, mostly without audio, give no answer and are excluded.
No audio (Table 1 )
No audio, explicit wording
Silence
Judge
Audio
pooled
Δw
pooled
Δw
pooled
Δw
Qwen2-Audio
0.13 [0.09, 0.16]
0.03
0.06 [0.04, 0.08]
0.03
0.10 [0.07, 0.13]
0.04
0.07 [0.05, 0.10]
Qwen2.5-Omni
0.11 [0.08, 0.15]
0.01
0.05 [0.03, 0.08]
0.00
0.05 [0.03, 0.07]
0.00
0.03 [0.01, 0.05]
Audio Flamingo 3
0.24 [0.21, 0.27]
0.02
0.12 [0.09, 0.15]
0.02
0.10 [0.07, 0.14]
0.01
0.12 [0.09, 0.16]
Music Flamingo
0.21 [0.18, 0.25]
0.05
0.07 [0.04, 0.10]
0.04
0.07 [0.04, 0.10]
0.02
0.09 [0.07, 0.11]
Qwen3-Omni (FP8)
0.31 [0.27, 0.34]
0.04
0.23 [0.18, 0.28]
0.02
0.23 [0.18, 0.29]
0.00
0.28 [0.22, 0.34]
Appendix
Table 6: Table 1 with intervals (over models for the model level; paired for Δw , audio minus control within a model). The controls omit the audio under the same instruction, omit it under the explicit no-audio wording (Appendix C ), or play 30 s of silence.
Correlation with experts, with audio
Agreement between formats
Judge
Target
1–5
1–7
1–9
yes/no
graded
yes/no vs. graded
Qwen2-Audio
quality
0.09
0.11
0.08
0.07
0.76–0.86
0.58–0.62
Qwen2.5-Omni
quality
0.04
0.07
0.04
0.07
0.98
0.78
Audio Flamingo 3
quality
0.17
0.19
0.18
0.32
0.96–0.98
0.67–0.72
Music Flamingo
quality
0.15
0.18
0.21
0.26
0.96–0.98
0.81–0.82
Qwen3-Omni (FP8)
quality
0.24
0.25
0.24
0.23
0.98–0.99
0.95–0.96
Appendix
Table 7: Answer formats on 300 clips, with audio: scales read as expected values over the digits, yes/no as the probability of Yes over Yes and No. Last two columns: range of correlations between a judge’s scores under two formats, among the graded scales and between yes/no and each scale.
Turn 1: verbal probability
Turn 2: rating from 1 to 5
Turn 2, replaced clip
Judge
top 3
vs. P(yes)
own prob. vs. single
vs. single
own prob.
clip played
Qwen2-Audio
0.54
-0.09 [-0.21, 0.03]
0.37 [0.28, 0.46]
-0.01 [-0.12, 0.10]
0.27 [0.15, 0.38]
-0.03 [-0.16, 0.11]
Qwen2.5-Omni
0.98
-0.20 [-0.31, -0.09]
0.30 [0.23, 0.39]
0.03 [-0.07, 0.13]
0.19 [0.04, 0.35]
0.14 [0.01, 0.27]
Audio Flamingo 3
0.94
-0.16 [-0.29, -0.05]
0.04 [-0.04, 0.13]
0.10 [0.00, 0.20]
–
–
Music Flamingo
0.82
-0.14 [-0.22, -0.05]
0.23 [0.16, 0.31]
0.03 [-0.06, 0.13]
–
–
Qwen3-Omni (FP8)
0.77
0.01 [-0.04, 0.06]
-0.01 [-0.03, 0.00]
-0.02 [-0.05, 0.01]
0.58 [0.48, 0.68]
0.33 [0.22, 0.43]
Appendix
Table 8: Probability, then rating, in one conversation (alignment, 300 clips). Turn 1 asks for the probability that the clip matches the prompt, written as a number between 0 and 1; turn 2 repeats the rating question, alone or with a clip attached. Top 3: share of the three most frequent probabilities. Vs. P(yes): expert correlation of the verbal probability minus that of the yes/no token probability. Own prob. vs. single: the turn-2 rating’s correlation with the turn-1 probability minus that with the single-turn rating. Vs. single: turn-2 minus single-turn expert correlation. Replaced clip: coefficients of the turn-2 rating on the turn-1 probability and on the expert rating of another model’s clip attached in turn 2.
Agrees with own pointwise
Agrees with the experts
Judge
Input
one order
averaged
pointwise
pairwise
difference
pHolm
Qwen2-Audio
concatenated
0.50
0.48
0.58
0.46
0.11 [0.06, 0.17]
0.002
Qwen2.5-Omni
concatenated
0.56
0.61
0.60
0.55
0.05 [0.00, 0.10]
0.148
Audio Flamingo 3
concatenated
0.51
0.54
0.66
0.50
0.16 [0.11, 0.22]
0.000
Music Flamingo
concatenated
0.50
0.68
0.63
0.66
-0.03 [-0.08, 0.02]
0.193
Qwen3-Omni (FP8)
concatenated
0.77
0.81
0.61
0.67
-0.05 [-0.09, -0.02]
0.017
Appendix
Table 9: Pointwise against pairwise verdicts on the 600 order-swap pairs, with pointwise ratings of the same 12 s crops (0.91 to 0.96 correlation with full-clip ratings). Agreement with the experts uses pairs with a nonzero expert gap; the difference is pointwise minus pairwise, Holm-adjusted per input type.
Judge
Target
Wording 1
Wording 2
Wording 3
Qwen2-Audio
quality
0.09 [-0.02, 0.20]
-0.01 [-0.12, 0.11]
-0.04 [-0.16, 0.08]
alignment
0.10 [-0.02, 0.20]
0.04 [-0.08, 0.15]
0.06 [-0.06, 0.16]
Qwen2.5-Omni
quality
0.04 [-0.08, 0.16]
0.06 [-0.07, 0.18]
0.03 [-0.08, 0.15]
alignment
0.16 [0.05, 0.27]
0.19 [0.08, 0.30]
0.00 [-0.12, 0.12]
Audio Flamingo 3
quality
0.17 [0.06, 0.28]
0.01 [-0.11, 0.12]
0.15 [0.05, 0.26]
alignment
0.16 [0.05, 0.26]
0.14 [0.02, 0.25]
0.16 [0.05, 0.27]
Appendix
Table 10: Correlation with expert ratings under three wordings, with audio, on the same 300 clips.
Audio gain
Swap: played
Same-slot
Wording range
Pairwise vs. own
Judge
(within)
/ replaced
rate
(clips)
pointwise
Qwen2-Audio
0.06 [0.04, 0.08]
0.10 / 0.06
0.95 [0.93, 0.97]
0.13 (300)
0.48 [0.44, 0.52]
Qwen2.5-Omni
0.05 [0.03, 0.08]
0.02 / 0.08
0.69 [0.65, 0.72]
0.03 (300)
0.61 [0.57, 0.65]
Audio Flamingo 3
0.12 [0.09, 0.15]
0.17 / 0.07
0.78 [0.75, 0.81]
0.16 (300)
0.54 [0.50, 0.58]
Music Flamingo
0.07 [0.04, 0.10]
0.15 / 0.11
0.99 [0.98, 1.00]
0.24 (all)
0.68 [0.65, 0.72]
Qwen3-Omni (FP8)
0.23 [0.18, 0.28]
0.24 / 0.07
0.24 [0.21, 0.27]
0.18 (all)
0.81 [0.78, 0.84]
Appendix
Table 11: Metric card for musical quality: the five diagnostics we recommend reporting. Audio gain: within-model correlation with minus without audio. Swap: coefficients of the clip played and replaced. Same-slot rate: concatenated order swap. Wording range: maximum minus minimum over three wordings. Last column: pairwise against own pointwise verdicts on the same 12 s crops.
Order swap, concatenated clips
Separate
No audio
Judge
same slot
gap < 0.4
0.4–0.8
≥ 0.8
same slot
+ name
Qwen2-Audio
0.95 [0.93, 0.97]
0.93
0.93
0.96
0.97 [0.95, 0.98]
-0.05 [-0.09, -0.02]
Qwen2.5-Omni
0.69 [0.65, 0.72]
0.66
0.71
0.69
0.77 [0.74, 0.81]
0.03 [-0.01, 0.06]
Audio Flamingo 3
0.78 [0.75, 0.81]
0.77
0.79
0.78
–
0.05 [0.02, 0.09]
Music Flamingo
0.99 [0.98, 1.00]
0.99
1.00
0.98
–
0.08 [0.04, 0.11]
Qwen3-Omni (FP8)
0.25 [0.21, 0.28]
0.30
0.31
0.19
0.19 [0.16, 0.22]
0.10 [0.06, 0.13]
Appendix
Table 12: Order swap on 600 pairs: same-slot rate overall, by expert quality gap (bins of 164, 107, and 329 pairs), and with separate inputs (the Flamingo processors take one audio input). Last column: no-audio quality correlation with the generating model’s name in the instruction.
Same-prompt swap, quality
Cross-prompt drop
Decomposition
Judge
played
replaced
quality
align.
on quality
on align.
Qwen2-Audio
0.10 [0.05, 0.14]
0.06 [0.02, 0.11]
0.04 [0.02, 0.06]
0.06 [0.05, 0.08]
0.10 [0.04, 0.15]
0.11 [0.05, 0.16]
Qwen2.5-Omni
0.02 [-0.02, 0.07]
0.08 [0.04, 0.13]
0.11 [0.09, 0.14]
0.46 [0.42, 0.51]
-0.07 [-0.13, -0.01]
0.17 [0.11, 0.23]
Audio Flamingo 3
0.17 [0.13, 0.22]
0.07 [0.03, 0.11]
0.09 [0.06, 0.11]
0.31 [0.27, 0.36]
0.11 [0.05, 0.16]
0.06 [0.01, 0.12]
Music Flamingo
0.15 [0.11, 0.19]
0.11 [0.06, 0.15]
0.11 [0.09, 0.14]
0.41 [0.37, 0.45]
0.12 [0.07, 0.17]
0.06 [0.01, 0.11]
Qwen3-Omni (FP8)
0.24 [0.20, 0.28]
0.07 [0.03, 0.11]
1.05 [0.98, 1.13]
1.57 [1.47, 1.67]
0.07 [0.02, 0.12]
0.37 [0.32, 0.42]
Appendix
Table 13: Same-prompt swap for quality (2,327 swaps), mean score drop under another prompt (points on the 1 to 5 scale, 999 clips), and within-model regression of the quality score on both expert ratings, which correlate at 0.74.
Pooled, all clips
Within model
Judge
Target
Wording 1
Wording 2
Wording 3
1
2
3
Kendall τ
Music Flamingo
quality
0.21 [0.18, 0.25]
0.25 [0.21, 0.28]
0.00 [-0.03, 0.04]
0.18
0.18
0.08
0.47–0.67
Music Flamingo
alignment
0.20 [0.17, 0.24]
0.18 [0.15, 0.22]
0.11 [0.07, 0.15]
0.23
0.19
0.14
0.58–0.64
Qwen3-Omni (FP8)
quality
0.31 [0.27, 0.34]
0.21 [0.17, 0.25]
0.13 [0.09, 0.17]
0.30
0.28
0.23
0.70–0.77
Qwen3-Omni (FP8)
alignment
0.35 [0.32, 0.38]
0.34 [0.30, 0.37]
0.35 [0.31, 0.38]
0.34
0.35
0.35
0.82–0.84
Appendix
Table 14: All three wordings on all 2,748 clips: pooled correlation with the experts, mean within-model correlation over the 25 shared-prompt models, and the range of Kendall τ between a judge’s scores under two wordings.
Quality
Alignment
Judge
wording 1
wording 2
wording 3
wording 1
wording 2
wording 3
Qwen2-Audio
0.11 [0.02, 0.19]
-0.01 [-0.08, 0.07]
-0.06 [-0.14, 0.01]
0.11 [0.01, 0.21]
0.00 [-0.09, 0.10]
0.08 [0.00, 0.16]
Qwen2.5-Omni
0.09 [0.00, 0.19]
0.08 [0.01, 0.14]
0.04 [-0.02, 0.11]
0.11 [0.00, 0.22]
0.17 [0.09, 0.26]
-0.02 [-0.11, 0.05]
Audio Flamingo 3
0.23 [0.14, 0.32]
0.05 [-0.03, 0.12]
0.20 [0.13, 0.29]
0.18 [0.08, 0.29]
0.15 [0.08, 0.23]
0.21 [0.13, 0.30]
Music Flamingo
0.15 [0.07, 0.22]
0.20 [0.14, 0.25]
-0.01 [-0.10, 0.07]
0.21 [0.10, 0.32]
0.19 [0.12, 0.26]
0.15 [0.06, 0.23]
Qwen3-Omni (FP8)
0.27 [0.11, 0.42]
0.16 [0.01, 0.32]
0.07 [-0.09, 0.23]
0.29 [0.13, 0.43]
0.31 [0.16, 0.45]
0.32 [0.17, 0.48]
Appendix
Table 15: Audio gain per wording on 300 clips: pooled correlation with audio minus without audio under the same wording (paired).
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
Distributional metrics such as Fréchet Audio Distance cannot score individual music clips and correlate poorly with human judgments, while the only per-sample learned metric achieving high human correlation is closed-source. We introduce MUQ-EVAL, an open-source per-sample quality metric for AIgenerated music built by training lightweight prediction heads on frozen MuQ-310M features using MusicEval, a dataset of generated clips from 31 text-to-music systems with expert quality ratings. Our simplest model, frozen features with attention pooling and a two-layer MLP, achieves system-level SRCC = 0.957 and utterance-level SRCC = 0.838 with human mean opinion scores. A systematic ablation over training objectives and adaptation strategies shows that no addition meaningfully improves the frozen baseline, indicating that frozen MuQ representations already capture quality-relevant information. Encoder choice is the dominant design factor, outweighing all architectural and training decisions. LoRA-adapted models trained on as few as 150 clips already achieve usable correlation, enabling personalized quality evaluators from individual listener annotations. A controlled degradation analysis reveals selective sensitivity to signal-level artifacts but insensitivity to musical-structural distortions. Our metric, MUQ-EVAL, is fully open-source, outperforms existing open per-sample metrics, and runs in real time on a single consumer GPU. Code, model weights, and evaluation scripts are available at https://github.com/dgtql/MuQ-Eval.
Di Zhu, Zixuan Li
Stevens Institute of Technology · Columbia University
We introduce TuneJury, an open, instance-level pairwise reward model for text-to-music that predicts a music preference score from a text prompt and an audio clip. The released checkpoint is trained on publicly available human-preference labels covering arena-style (A vs. B) votes, metric-alignment preference pairs, crowdsourced pairwise comparisons, and expert aesthetic ratings. The predicted score margin between two clips is well calibrated on our held-out test split, supporting data filtering via a simple score threshold. TuneJury generalizes to both held-out test pairs and out-of-distribution benchmarks, remaining competitive with prior baselines on the latter. For generators released after training, we introduce anchor calibration, a post-hoc, per-system Bradley-Terry calibration that recovers agreement at substantially better data efficiency than from-scratch retraining. The same frozen reward drives consistent reward-axis gains across three downstream applications: inference-time best-of-N selection, DITTO-style latent optimization, and expert-iteration post-training. TuneJury is available at https://github.com/yonghyunk1m/TuneJury.
Text-to-music alignment is commonly evaluated by the global similarity between text and audio embeddings, which collapses all described attributes into a single score and obscures which ones are missing. We instead introduce MQAScore, which verifies each atomic musical concept along five clip-level dimensions (genre, instrument, mood/theme, vocal, and usage) by the probability that a large audio language model (LALM) answers affirmatively to a binary question. To assess evaluator reliability, we construct verification labels from presence/absence annotations and LLM-guided concept substitutions, and propose the Sensitivity-Invariance Ratio, which measures how selectively an evaluator responds to meaning-changing substitutions. Experiments show that concept-level scoring improves both question-answering and similarity-based evaluators, and MQAScore can be a more reliable and interpretable fine-grained complement to global similarity for evaluating text-to-music alignment.
Wei-Jaw Lee, Chun-Ping Wang, Xuanjun Chen +1
GICE, National Taiwan University · AI-CoRE, National Taiwan University · Taiwan AILabs