Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input. We introduce VocalCoachBench, a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing. VocalCoachBench contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,056 expert submissions and 12,051 atomic coaching claims. It comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions. To accommodate the open-ended nature of expert feedback, VocalCoachBench sep- arates deterministic structured targets from claim-based assessment of free-form diagnosis and corrective guidance. Human annotation analysis shows that expert agreement varies strongly with label granularity, motivating hierarchical structured metrics and claim-based evaluation of open-ended feedback. Experiments with 12 recent audio-language models reveal a consistent gap: while models can compare performances and identify broad issue domains in free-form feedback, Top-3 fine-grained issue-label identification remains below label-prior baselines and strict diagnosis alignment stays below 7%. To our knowledge, VocalCoachBench pro- vides the first public testbed for evaluating audio-grounded expert feedback for singing, moving audio-language evaluation beyond description toward analytic feedback.
Figures & tables
Figure 1: Overview of VocalCoachBench. Expert vocal trainers annotate two complementary subsets: a same-song subset for controlled comparison, and a diverse-song subset for diverse coaching scenarios and segment-grounded feedback. Both subsets include overall coaching reviews and ranked Top-3 issue labels. Expert feedback is post-processed into atomic coaching claims, which support structured and open-ended evaluation.
Work
Main target
Structured Eval
Expert FB
Diag/Corr Eval
Segment-level FB
Singing assessment
SingEval [ 13 ]
Quality score/rank
✓
×
×
×
Music performance assessment and feedback
PercePiano [ 28 ]
Perceptual evaluation
✓
×
×
×
NeuroPiano [ 45 ]
Rating + feedback
✓
✓
×
×
CROCUS [ 24 ]
Formative critiques
×
✓
×
△
Table 1: Comparison with representative resources for singing assessment, music performance feedback, and expert skill feedback.
Dataset field
Same
Diverse
Total
Audio
207
308
515
Submissions
450
602
1,052
Blocks
1,397
3,046
4,443
Claims
3,972
8,079
12,051
Diagnosis
2,090
3,997
6,087
Correction
1,642
2,885
4,527
Table 2: VocalCoachBench dataset summary.
Figure 2: Annotation statistics and agreement in VocalCoachBench. (a) Top-3 issue distributions across subsets, with percentage-point shifts shown on the right. (b) Feedback richness in words and atomic claims per annotator-audio submission. (c) Expert agreement on structured targets, including parent-level Top-3 agreement, fine-label agreement, and same-song triplet rankings.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Parent
Label
Guidance
Technical Production
Breath
The overall use and control of breath, including phrase sustain, breath support, release of support during descending passages, and breath length or distribution.
Vocalization
Overall vocal quality and production method, including phonation, placement, blocked or constricted sound, muffled or pulled-back placement, high-note stability, and resonance.
Technique
The use and proficiency of special vocal skills, such as vibrato, vocal fry, bending, and ornamentation.
Musical Accuracy
Pitch
All cases related to pitch accuracy and stability, including leaps, final-note stability, pitch sagging in descending passages, and the accuracy of ornaments or modulations.
Rhythm
Beat and timing, including rushing or dragging, pushing or pulling against the beat, and rhythmic expression through accent or dynamics.
Delivery
Diction
Lyric intelligibility and pronunciation habits, including vowel maintenance, nasal tone, hard or blocked articulation, and genre-stylistic pronunciation.
Smule Research Data License; research/education use only, no redistribution or commercial use
Diverse-song
PopBuTFy [ 21 ]
85
CC BY-NC-SA 4.0 with dataset terms of access
Diverse-song
N20EMv2 [ 47 ]
65
CC BY-SA 4.0
Diverse-song
LM-SSD [ 31 ]
7
CC BY 4.0
Diverse-song
NUS-48E [ 8 ]
41
Research-use-only corpus terms
Diverse-song
MedleyDB [ 2 ]
32
CC BY-NC-SA 4.0
Appendix
Table 11: Audio source datasets and license terms.
Error type
Rate
Atomicity violation
0.8%
Fragmentation error
0.0%
Omission
0.0%
Hallucinated addition
0.0%
Contextual role-assignment error
0.4%
Correction target-link error
0.0%
Appendix
Table 12: Single-author audit results for claim extraction. Correction target-link errors are computed over audited correction claims; other error rates are computed over audited extracted claims.
Comparison
Valid cases
Agreement
Cohen’s κ
LLM judge vs. Annotator 1
99
84.8
.798
LLM judge vs. Annotator 2
94
81.9
.759
LLM judge vs. authors, pooled
193
83.4
.779
Annotator 1 vs. Annotator 2, exact
93
80.6
.737
Annotator 1 vs. Annotator 2, binary
93
93.5
.870
Appendix
Table 13: Diagnosis judge validation summary. Exact agreement uses the four-way diagnosis labels. Binary agreement collapses Strict Hit and Coarse Hit into Hit , and Miss and Contradict into Non-Hit .
LLM judge \ author
Strict
Coarse
Miss
Contra.
Strict
48
2
0
0
Coarse
13
33
2
0
Miss
0
0
47
1
Contra.
1
5
8
33
Appendix
Table 14: Pooled four-way confusion matrix for diagnosis judge validation. Rows are LLM judge labels and columns are author labels, after excluding BAD_CASE and blank labels.
Annotator 2 \ Annotator 1
Strict
Coarse
Miss
Contra.
Strict
26
2
0
0
Coarse
7
12
2
2
Miss
0
2
24
3
Contra.
0
0
0
13
Appendix
Table 15: Author–author four-way confusion matrix for diagnosis validation
LLM judge / Expert
Valid
Weak
Invalid
Total
Valid
21
3
0
24
Weak
5
19
1
25
Invalid
4
2
17
23
Total
30
24
18
72
Appendix
Table 16: Correction judge validation confusion matrix for the three main correction labels. Rows are LLM judge labels and columns are expert labels.
Protocol
Prompt / task
Max tokens
Main
Open-ended coaching feedback
1,200
Main
Objective Top-3 issue prediction
420
Main
Segment-level issue classification
256
Simple
Overall coaching review
650
Simple
Short-form label prompts
40–80
Appendix
Table 17: Generation budgets used for model inference.
Figure 3: Annotation interfaces. Top: same-song interface for holistic coaching feedback. Bottom: diverse-song interface for segment-level issue annotation.
Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.
Effective vocal coaching connects performance issues with corrective advice, but recordings paired with expert diagnoses and advice are scarce. Online singing communities provide a potential source of supervision, but observations and advice are scattered across unstructured comments. We introduce PitchIn, audio-language models for natural-language vocal coaching, and the PeerCoach Dataset: 14,445 instruction examples covering Focused QA, Overall Feedback, and Recognition QA. We compare target combinations on Qwen2.5-Omni and train PitchIn on Focused QA and Overall Feedback using Qwen2.5-Omni and MOSS-Music. On VocalCoachBench, the proportion of requested diagnosis-correction pairs matching an expert issue with valid advice rises from 21.3% to 61.0% for Qwen2.5-Omni and from 31.4% to 51.2% for MOSS-Music. Direct training on source comments yields only 19.8% for Qwen2.5-Omni. For Qwen, Focused QA alone reaches 65.6%, but adding Overall Feedback improves alignment with expert reviews and human preference. Both models improve expert-review alignment over pretrained baselines and are significantly preferred for usefulness (92.0% and 67.3%, respectively). Demo examples are available at https://hayeonbang.github.io/PitchIn.
Hayeon Bang, Hounsu Kim, Joonyong Park +2
KAIST, Republic of Korea · The University of Tokyo, Japan · ETRI, Republic of Korea
Mixing requires not only applying processing but also assessing the resulting mix. Audio-language models can provide conversa- tional mixing advice, but existing evaluations do not separately test whether they can judge specific mixing issues. Since professional supervision for such judgments is scarce, we construct a curriculum of 11 vocal-mix issues at multiple severity levels and use it to train MixJudge, a language-queryable audio critic. MixJudge shows its ability to retrieve mixing issues, distinguish opposite issues, and order the severity of an issue. Moreover, its judgments transfer to menu selection: given a set of processing candidates, MixJudge often favors versions acoustically closer to a held-out professional version, demonstrating how explicit mixing judgment can support comparison and downstream mixing decisions.
Kuan-Hsun Ho, Yuan-Chiao Cheng, Yen-Tung Yeh +1
Independent Researcher · Music Informatics Group, Georgia Institute of Technology · Music and AI Lab, National Taiwan University +1
Open audio-language models are used as judges of generated music: given a clip and its text prompt, the model rates the clip, and the judge is validated by how well its ratings correlate with expert ratings. The correlation does not show what the judge responds to: the music, the text prompt, or the way the question is asked. We propose four probes that test each of these, three of them read against a control computed from the human ratings: removing the audio, replacing the clip with another model's clip for the same prompt, swapping the order of two clips in pairwise judging, and rewording the instruction. We apply them to five open judges on all 2,748 clips of MusicEval, and the last two to the SongEval songs. Without audio, no judge exceeds what the prompt alone predicts from the human ratings, and for prompt alignment every judge falls below it. With audio, every judge follows the clip it hears when rating prompt alignment, two do when rating musical quality, and the strongest judge's quality score tracks the experts' alignment rating more than their quality rating. The pairwise verdicts of the four smaller judges follow the position of a clip in the pair (its slot) rather than its quality on both rating sets. For musical quality, rewording the instruction moves the correlation of most judges as much as changing the judge. MusicEval's prompt assignment bounds any prompt-only judge at the model level. We recommend reporting the four probes with every music judge.
Mateusz Modrzejewski, Tomasz Radzikowski
Institute of Computer Science, Warsaw University of Technology