What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings

Mateusz Modrzejewski; Tomasz Radzikowski

cs.SDOct 9, 2026

What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings

Authors: Mateusz Modrzejewski, Tomasz Radzikowski

Organizations: Institute of Computer Science, Warsaw University of Technology

Abstract

Open audio-language models are used as judges of generated music: given a clip and its text prompt, the model rates the clip, and the judge is validated by how well its ratings correlate with expert ratings. The correlation does not show what the judge responds to: the music, the text prompt, or the way the question is asked. We propose four probes that test each of these, three of them read against a control computed from the human ratings: removing the audio, replacing the clip with another model's clip for the same prompt, swapping the order of two clips in pairwise judging, and rewording the instruction. We apply them to five open judges on all 2,748 clips of MusicEval, and the last two to the SongEval songs. Without audio, no judge exceeds what the prompt alone predicts from the human ratings, and for prompt alignment every judge falls below it. With audio, every judge follows the clip it hears when rating prompt alignment, two do when rating musical quality, and the strongest judge's quality score tracks the experts' alignment rating more than their quality rating. The pairwise verdicts of the four smaller judges follow the position of a clip in the pair (its slot) rather than its quality on both rating sets. For musical quality, rewording the instruction moves the correlation of most judges as much as changing the judge. MusicEval's prompt assignment bounds any prompt-only judge at the model level. We recommend reporting the four probes with every music judge.

Figures & tables

Figure 1: Two of the four probes for the five judges, with 95% bootstrap intervals. Left: in the same-prompt swap, the standardized coefficient of the judge’s score on the expert rating of the clip played (filled) and of the clip it replaced (open), for the quality and the alignment target; a judge that listens follows the clip played. Right: correlation with expert quality ratings under three wordings of the instruction on the same 300 clips.

	Overall quality			Align.	Model
Judge	Audio	No audio	$\Delta_{w}$	Audio	level
Qwen2-Audio	0.13	0.03	0.06	0.17	0.42
Qwen2.5-Omni	0.11	0.01	0.05	0.19	0.32
Audio Flamingo 3	0.24	0.02	0.12	0.23	0.73
Music Flamingo	0.21	0.05	0.07	0.23	0.55
Qwen3-Omni (FP8)	0.31	0.04	0.23	0.34	0.36

Table 1: Spearman correlation with expert ratings. The Audio and No audio columns are computed over all 2,748 clips; No audio omits the audio and keeps the instruction used with audio.

\Delta_{w}

is audio minus no audio within a model, and every interval is above zero. Align. is the prompt-alignment target with audio, within a model. Model level is over 31 models after adjusting for each model’s prompts; the expert’s 0.98 is unadjusted. The expert rows give one expert against the mean of the others, pooled or within a model. Intervals are in Table 6 .

	Swap, qual.		Swap, al.	Order swap
Judge	played	repl.	played	same	2nd	agree
Qwen2-Audio	0.10	0.06	0.12	0.95	0.86	0.46
Qwen2.5-Omni	0.02	0.08	0.12	0.69	0.42	0.55
Audio Flamingo 3	0.17	0.07	0.19	0.78	0.40	0.50
Music Flamingo	0.15	0.11	0.23	0.99	0.99	0.66
Qwen3-Omni (FP8)	0.24	0.07	0.31	0.25	0.21	0.67

Table 2: The same-prompt swap columns give standardized regression coefficients of the judge’s score on the expert rating of the clip played and of the clip it replaced (2,327 swaps). The order-swap columns give the same-slot rate (the share of 600 pairs in which the judge picks the same slot in both orders, so that its preferred clip changes with the order), the share in which that slot is the second, and the share in which the order-averaged preference agrees with the experts (chance: 0.50, 0.25, and 0.50). Intervals are in Tables 13 and 12 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Judge	Hugging Face identifier	Revision	Params	Precision	Software
Qwen2-Audio	Qwen/Qwen2-Audio-7B-Instruct	0a095220c3	8.4B	bf16	transf. 5.17.0
Qwen2.5-Omni	Qwen/Qwen2.5-Omni-7B	ae9e169054	10.7B	bf16	transf. 5.17.0
Audio Flamingo 3	nvidia/audio-flamingo-3-hf	7d4bae64ee	8.3B	bf16	transf. 5.17.0
Music Flamingo	nvidia/music-flamingo-2601-hf	6b5be086d5	8.3B	bf16	transf. 5.17.0
Qwen3-Omni	sammysun0711/Qwen3-Omni-30B- A3B-Instruct-FP8-Dynamic	622f7242c4	35.3B (MoE)	FP8 (dynamic)	vLLM 0.30

Appendix

Table 3: Judges with their Hugging Face revision (first ten characters), parameters, precision, and software (transf.: transformers). The score is the expected value over the digits 1 to 5 (the letters A and B in pairwise judging) under greedy decoding. Qwen3-Omni is a community FP8 quantization of Qwen/Qwen3-Omni-30B-A3B-Instruct on two RTX 4090 GPUs; the other judges run on one.

MusicEval, all clips
Comparison	Target	Correlations	Difference	$p_{\mathrm{Holm}}$
Qwen2-Audio – Qwen2.5-Omni	quality	0.13 – 0.11	0.02 [-0.02, 0.05]	0.362
Qwen2-Audio – Audio Flamingo 3	quality	0.13 – 0.24	-0.11 [-0.15, -0.08]	$<$ 0.001
Qwen2-Audio – Music Flamingo	quality	0.13 – 0.21	-0.09 [-0.12, -0.05]	$<$ 0.001
Qwen2-Audio – Qwen3-Omni (FP8)	quality	0.13 – 0.31	-0.18 [-0.22, -0.13]	$<$ 0.001
Qwen2.5-Omni – Audio Flamingo 3	quality	0.11 – 0.24	-0.13 [-0.16, -0.10]	$<$ 0.001

Appendix

Table 4: Paired tests, with Holm-corrected

p

and 95% paired bootstrap intervals. (a) The judge and wording comparisons in the text: differences of Spearman correlations on the same clips, adjusted within each block and target. (b) No-audio correlation (instruction unchanged) against the prompt-only floor (0.08 for quality, 0.19 for alignment) on the 2,501 clips where the floor is defined, adjusted over the ten tests. (c) Same-prompt swap: coefficient of the clip played minus that of the clip replaced (2,327 swaps), adjusted over the five judges of each target.

	FP8	Unquantized (bf16)		Thinking
Cell	value	value	difference	value	difference
pooled, audio, overall	0.31	0.31	0.00 [0.00, 0.01]	0.38	0.07 [0.04, 0.10]
pooled, no audio (instruction unchanged), overall	0.04	0.04	0.01 [0.00, 0.02]	–	–
within model, audio, overall	0.30	0.30	0.00 [0.00, 0.01]	0.26	-0.04 [-0.07, -0.01]
within-model audio gain ( $\Delta_{w}$ ), overall	0.23	0.22	-0.01 [-0.02, 0.00]	–	–
swap, clip played, overall	0.24	0.25	0.01 [0.00, 0.01]	0.32	0.08 [0.04, 0.12]

Appendix

Table 5: Qwen3-Omni-30B-A3B unquantized (bf16) and Thinking against the FP8 judge of the main tables, on identical inputs: the cells of Tables 1 and 2 and the SongEval correlations, with the difference from FP8. Unquantized and FP8 clip scores correlate at 0.91 to 0.99. The Thinking readout follows its reasoning (up to 4,096 tokens); 128 of its 23,140 MusicEval judgments, mostly without audio, give no answer and are excluded.

		No audio (Table 1 )		No audio, explicit wording		Silence
Judge	Audio	pooled	$\Delta_{w}$	pooled	$\Delta_{w}$	pooled	$\Delta_{w}$
Qwen2-Audio	0.13 [0.09, 0.16]	0.03	0.06 [0.04, 0.08]	0.03	0.10 [0.07, 0.13]	0.04	0.07 [0.05, 0.10]
Qwen2.5-Omni	0.11 [0.08, 0.15]	0.01	0.05 [0.03, 0.08]	0.00	0.05 [0.03, 0.07]	0.00	0.03 [0.01, 0.05]
Audio Flamingo 3	0.24 [0.21, 0.27]	0.02	0.12 [0.09, 0.15]	0.02	0.10 [0.07, 0.14]	0.01	0.12 [0.09, 0.16]
Music Flamingo	0.21 [0.18, 0.25]	0.05	0.07 [0.04, 0.10]	0.04	0.07 [0.04, 0.10]	0.02	0.09 [0.07, 0.11]
Qwen3-Omni (FP8)	0.31 [0.27, 0.34]	0.04	0.23 [0.18, 0.28]	0.02	0.23 [0.18, 0.29]	0.00	0.28 [0.22, 0.34]

Appendix

Table 6: Table 1 with intervals (over models for the model level; paired for

\Delta_{w}

, audio minus control within a model). The controls omit the audio under the same instruction, omit it under the explicit no-audio wording (Appendix C ), or play 30 s of silence.

		Correlation with experts, with audio				Agreement between formats
Judge	Target	1–5	1–7	1–9	yes/no	graded	yes/no vs. graded
Qwen2-Audio	quality	0.09	0.11	0.08	0.07	0.76–0.86	0.58–0.62
Qwen2.5-Omni	quality	0.04	0.07	0.04	0.07	0.98	0.78
Audio Flamingo 3	quality	0.17	0.19	0.18	0.32	0.96–0.98	0.67–0.72
Music Flamingo	quality	0.15	0.18	0.21	0.26	0.96–0.98	0.81–0.82
Qwen3-Omni (FP8)	quality	0.24	0.25	0.24	0.23	0.98–0.99	0.95–0.96

Appendix

Table 7: Answer formats on 300 clips, with audio: scales read as expected values over the digits, yes/no as the probability of Yes over Yes and No. Last two columns: range of correlations between a judge’s scores under two formats, among the graded scales and between yes/no and each scale.

	Turn 1: verbal probability		Turn 2: rating from 1 to 5		Turn 2, replaced clip
Judge	top 3	vs. P(yes)	own prob. vs. single	vs. single	own prob.	clip played
Qwen2-Audio	0.54	-0.09 [-0.21, 0.03]	0.37 [0.28, 0.46]	-0.01 [-0.12, 0.10]	0.27 [0.15, 0.38]	-0.03 [-0.16, 0.11]
Qwen2.5-Omni	0.98	-0.20 [-0.31, -0.09]	0.30 [0.23, 0.39]	0.03 [-0.07, 0.13]	0.19 [0.04, 0.35]	0.14 [0.01, 0.27]
Audio Flamingo 3	0.94	-0.16 [-0.29, -0.05]	0.04 [-0.04, 0.13]	0.10 [0.00, 0.20]	–	–
Music Flamingo	0.82	-0.14 [-0.22, -0.05]	0.23 [0.16, 0.31]	0.03 [-0.06, 0.13]	–	–
Qwen3-Omni (FP8)	0.77	0.01 [-0.04, 0.06]	-0.01 [-0.03, 0.00]	-0.02 [-0.05, 0.01]	0.58 [0.48, 0.68]	0.33 [0.22, 0.43]

Appendix

Table 8: Probability, then rating, in one conversation (alignment, 300 clips). Turn 1 asks for the probability that the clip matches the prompt, written as a number between 0 and 1; turn 2 repeats the rating question, alone or with a clip attached. Top 3: share of the three most frequent probabilities. Vs. P(yes): expert correlation of the verbal probability minus that of the yes/no token probability. Own prob. vs. single: the turn-2 rating’s correlation with the turn-1 probability minus that with the single-turn rating. Vs. single: turn-2 minus single-turn expert correlation. Replaced clip: coefficients of the turn-2 rating on the turn-1 probability and on the expert rating of another model’s clip attached in turn 2.

		Agrees with own pointwise		Agrees with the experts
Judge	Input	one order	averaged	pointwise	pairwise	difference	$p_{\mathrm{Holm}}$
Qwen2-Audio	concatenated	0.50	0.48	0.58	0.46	0.11 [0.06, 0.17]	0.002
Qwen2.5-Omni	concatenated	0.56	0.61	0.60	0.55	0.05 [0.00, 0.10]	0.148
Audio Flamingo 3	concatenated	0.51	0.54	0.66	0.50	0.16 [0.11, 0.22]	0.000
Music Flamingo	concatenated	0.50	0.68	0.63	0.66	-0.03 [-0.08, 0.02]	0.193
Qwen3-Omni (FP8)	concatenated	0.77	0.81	0.61	0.67	-0.05 [-0.09, -0.02]	0.017

Appendix

Table 9: Pointwise against pairwise verdicts on the 600 order-swap pairs, with pointwise ratings of the same 12 s crops (0.91 to 0.96 correlation with full-clip ratings). Agreement with the experts uses pairs with a nonzero expert gap; the difference is pointwise minus pairwise, Holm-adjusted per input type.

Judge	Target	Wording 1	Wording 2	Wording 3
Qwen2-Audio	quality	0.09 [-0.02, 0.20]	-0.01 [-0.12, 0.11]	-0.04 [-0.16, 0.08]
	alignment	0.10 [-0.02, 0.20]	0.04 [-0.08, 0.15]	0.06 [-0.06, 0.16]
Qwen2.5-Omni	quality	0.04 [-0.08, 0.16]	0.06 [-0.07, 0.18]	0.03 [-0.08, 0.15]
	alignment	0.16 [0.05, 0.27]	0.19 [0.08, 0.30]	0.00 [-0.12, 0.12]
Audio Flamingo 3	quality	0.17 [0.06, 0.28]	0.01 [-0.11, 0.12]	0.15 [0.05, 0.26]
	alignment	0.16 [0.05, 0.26]	0.14 [0.02, 0.25]	0.16 [0.05, 0.27]

Appendix

Table 10: Correlation with expert ratings under three wordings, with audio, on the same 300 clips.

	Audio gain	Swap: played	Same-slot	Wording range	Pairwise vs. own
Judge	(within)	/ replaced	rate	(clips)	pointwise
Qwen2-Audio	0.06 [0.04, 0.08]	0.10 / 0.06	0.95 [0.93, 0.97]	0.13 (300)	0.48 [0.44, 0.52]
Qwen2.5-Omni	0.05 [0.03, 0.08]	0.02 / 0.08	0.69 [0.65, 0.72]	0.03 (300)	0.61 [0.57, 0.65]
Audio Flamingo 3	0.12 [0.09, 0.15]	0.17 / 0.07	0.78 [0.75, 0.81]	0.16 (300)	0.54 [0.50, 0.58]
Music Flamingo	0.07 [0.04, 0.10]	0.15 / 0.11	0.99 [0.98, 1.00]	0.24 (all)	0.68 [0.65, 0.72]
Qwen3-Omni (FP8)	0.23 [0.18, 0.28]	0.24 / 0.07	0.24 [0.21, 0.27]	0.18 (all)	0.81 [0.78, 0.84]

Appendix

Table 11: Metric card for musical quality: the five diagnostics we recommend reporting. Audio gain: within-model correlation with minus without audio. Swap: coefficients of the clip played and replaced. Same-slot rate: concatenated order swap. Wording range: maximum minus minimum over three wordings. Last column: pairwise against own pointwise verdicts on the same 12 s crops.

	Order swap, concatenated clips				Separate	No audio
Judge	same slot	gap $<$ 0.4	0.4–0.8	$\geq$ 0.8	same slot	+ name
Qwen2-Audio	0.95 [0.93, 0.97]	0.93	0.93	0.96	0.97 [0.95, 0.98]	-0.05 [-0.09, -0.02]
Qwen2.5-Omni	0.69 [0.65, 0.72]	0.66	0.71	0.69	0.77 [0.74, 0.81]	0.03 [-0.01, 0.06]
Audio Flamingo 3	0.78 [0.75, 0.81]	0.77	0.79	0.78	–	0.05 [0.02, 0.09]
Music Flamingo	0.99 [0.98, 1.00]	0.99	1.00	0.98	–	0.08 [0.04, 0.11]
Qwen3-Omni (FP8)	0.25 [0.21, 0.28]	0.30	0.31	0.19	0.19 [0.16, 0.22]	0.10 [0.06, 0.13]

Appendix

Table 12: Order swap on 600 pairs: same-slot rate overall, by expert quality gap (bins of 164, 107, and 329 pairs), and with separate inputs (the Flamingo processors take one audio input). Last column: no-audio quality correlation with the generating model’s name in the instruction.

	Same-prompt swap, quality		Cross-prompt drop		Decomposition
Judge	played	replaced	quality	align.	on quality	on align.
Qwen2-Audio	0.10 [0.05, 0.14]	0.06 [0.02, 0.11]	0.04 [0.02, 0.06]	0.06 [0.05, 0.08]	0.10 [0.04, 0.15]	0.11 [0.05, 0.16]
Qwen2.5-Omni	0.02 [-0.02, 0.07]	0.08 [0.04, 0.13]	0.11 [0.09, 0.14]	0.46 [0.42, 0.51]	-0.07 [-0.13, -0.01]	0.17 [0.11, 0.23]
Audio Flamingo 3	0.17 [0.13, 0.22]	0.07 [0.03, 0.11]	0.09 [0.06, 0.11]	0.31 [0.27, 0.36]	0.11 [0.05, 0.16]	0.06 [0.01, 0.12]
Music Flamingo	0.15 [0.11, 0.19]	0.11 [0.06, 0.15]	0.11 [0.09, 0.14]	0.41 [0.37, 0.45]	0.12 [0.07, 0.17]	0.06 [0.01, 0.11]
Qwen3-Omni (FP8)	0.24 [0.20, 0.28]	0.07 [0.03, 0.11]	1.05 [0.98, 1.13]	1.57 [1.47, 1.67]	0.07 [0.02, 0.12]	0.37 [0.32, 0.42]

Appendix

Table 13: Same-prompt swap for quality (2,327 swaps), mean score drop under another prompt (points on the 1 to 5 scale, 999 clips), and within-model regression of the quality score on both expert ratings, which correlate at 0.74.

Judge	Target	Wording 1	Wording 2	Wording 3	1	2	3	Kendall $\tau$
		Pooled, all clips			Within model
Music Flamingo	quality	0.21 [0.18, 0.25]	0.25 [0.21, 0.28]	0.00 [-0.03, 0.04]	0.18	0.18	0.08	0.47–0.67
Music Flamingo	alignment	0.20 [0.17, 0.24]	0.18 [0.15, 0.22]	0.11 [0.07, 0.15]	0.23	0.19	0.14	0.58–0.64
Qwen3-Omni (FP8)	quality	0.31 [0.27, 0.34]	0.21 [0.17, 0.25]	0.13 [0.09, 0.17]	0.30	0.28	0.23	0.70–0.77
Qwen3-Omni (FP8)	alignment	0.35 [0.32, 0.38]	0.34 [0.30, 0.37]	0.35 [0.31, 0.38]	0.34	0.35	0.35	0.82–0.84

Appendix

Table 14: All three wordings on all 2,748 clips: pooled correlation with the experts, mean within-model correlation over the 25 shared-prompt models, and the range of Kendall

\tau

between a judge’s scores under two wordings.

	Quality			Alignment
Judge	wording 1	wording 2	wording 3	wording 1	wording 2	wording 3
Qwen2-Audio	0.11 [0.02, 0.19]	-0.01 [-0.08, 0.07]	-0.06 [-0.14, 0.01]	0.11 [0.01, 0.21]	0.00 [-0.09, 0.10]	0.08 [0.00, 0.16]
Qwen2.5-Omni	0.09 [0.00, 0.19]	0.08 [0.01, 0.14]	0.04 [-0.02, 0.11]	0.11 [0.00, 0.22]	0.17 [0.09, 0.26]	-0.02 [-0.11, 0.05]
Audio Flamingo 3	0.23 [0.14, 0.32]	0.05 [-0.03, 0.12]	0.20 [0.13, 0.29]	0.18 [0.08, 0.29]	0.15 [0.08, 0.23]	0.21 [0.13, 0.30]
Music Flamingo	0.15 [0.07, 0.22]	0.20 [0.14, 0.25]	-0.01 [-0.10, 0.07]	0.21 [0.10, 0.32]	0.19 [0.12, 0.26]	0.15 [0.06, 0.23]
Qwen3-Omni (FP8)	0.27 [0.11, 0.42]	0.16 [0.01, 0.32]	0.07 [-0.09, 0.23]	0.29 [0.13, 0.43]	0.31 [0.16, 0.45]	0.32 [0.17, 0.48]

Appendix

Table 15: Audio gain per wording on 300 clips: pooled correlation with audio minus without audio under the same wording (paired).

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 206 papers

MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation
Mar 24, 2026Di Zhu, Zixuan LiMusic Generation Evaluation
Stevens Institute of Technology · Columbia University
TuneJury: An Open Metric for Improving Music Generation Preference Alignment
Jun 15, 2026Yonghyun Kim, Junwon Lee, Haiwen Xia +5Text-to-Music Generation Music Generation Evaluation
Georgia Tech · KAIST · Peking University +3
MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model
Oct 22, 2026Wei-Jaw Lee, Chun-Ping Wang, Xuanjun Chen +1Audio-Text Alignment Automated Evaluation
GICE, National Taiwan University · AI-CoRE, National Taiwan University · Taiwan AILabs

Mar 24, 2026J/K move · Enter open · S save

MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation

Di Zhu, Zixuan Li

Stevens Institute of Technology · Columbia University

Music Generation Evaluation

Open paper AskPDF arXiv

Distributional metrics such as Fréchet Audio Distance cannot score individual music clips and correlate poorly with human judgments, while the only per-sample learned metric achieving high human correlation is closed-source. We introduce MUQ-EVAL, an open-source per-sample quality metric for AIgenerated music built by training lightweight prediction heads on frozen MuQ-310M features using MusicEval, a dataset of generated clips from 31 text-to-music systems with expert quality ratings. Our simplest model, frozen features with attention pooling and a two-layer MLP, achieves system-level SRCC = 0.957 and utterance-level SRCC = 0.838 with human mean opinion scores. A systematic ablation over training objectives and adaptation strategies shows that no addition meaningfully improves the frozen baseline, indicating that frozen MuQ representations already capture quality-relevant information. Encoder choice is the dominant design factor, outweighing all architectural and training decisions. LoRA-adapted models trained on as few as 150 clips already achieve usable correlation, enabling personalized quality evaluators from individual listener annotations. A controlled degradation analysis reveals selective sensitivity to signal-level artifacts but insensitivity to musical-structural distortions. Our metric, MUQ-EVAL, is fully open-source, outperforms existing open per-sample metrics, and runs in real time on a single consumer GPU. Code, model weights, and evaluation scripts are available at https://github.com/dgtql/MuQ-Eval.

Abstract

Audio-Language Model Evaluation

Explore similar work

MuQ-Eval: An Open-Source Per-Sample Quality Metric for AI Music Generation Evaluation

TuneJury: An Open Metric for Improving Music Generation Preference Alignment

MQAScore: Fine-Grained Text-Music Alignment Evaluation via Large Audio Language Model