VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing

Hayeon Bang; Hounsu Kim; Wonil Kim; Juhan Nam

cs.SDAug 6, 2026

VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing

Authors: Hayeon Bang, Hounsu Kim, Wonil Kim, Juhan Nam

Organizations: KAIST

Abstract

Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input. We introduce VocalCoachBench, a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing. VocalCoachBench contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,056 expert submissions and 12,051 atomic coaching claims. It comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions. To accommodate the open-ended nature of expert feedback, VocalCoachBench sep- arates deterministic structured targets from claim-based assessment of free-form diagnosis and corrective guidance. Human annotation analysis shows that expert agreement varies strongly with label granularity, motivating hierarchical structured metrics and claim-based evaluation of open-ended feedback. Experiments with 12 recent audio-language models reveal a consistent gap: while models can compare performances and identify broad issue domains in free-form feedback, Top-3 fine-grained issue-label identification remains below label-prior baselines and strict diagnosis alignment stays below 7%. To our knowledge, VocalCoachBench pro- vides the first public testbed for evaluating audio-grounded expert feedback for singing, moving audio-language evaluation beyond description toward analytic feedback.

Figures & tables

Figure 1: Overview of VocalCoachBench. Expert vocal trainers annotate two complementary subsets: a same-song subset for controlled comparison, and a diverse-song subset for diverse coaching scenarios and segment-grounded feedback. Both subsets include overall coaching reviews and ranked Top-3 issue labels. Expert feedback is post-processed into atomic coaching claims, which support structured and open-ended evaluation.

Singing assessment
Work	Main target	Structured Eval	Expert FB	Diag/Corr Eval	Segment-level FB
SingEval [ 13 ]	Quality score/rank	$\checkmark$	$\times$	$\times$	$\times$
Music performance assessment and feedback
PercePiano [ 28 ]	Perceptual evaluation	$\checkmark$	$\times$	$\times$	$\times$
NeuroPiano [ 45 ]	Rating + feedback	$\checkmark$	$\checkmark$	$\times$	$\times$
CROCUS [ 24 ]	Formative critiques	$\times$	$\checkmark$	$\times$	$\triangle$

Table 1: Comparison with representative resources for singing assessment, music performance feedback, and expert skill feedback.

Dataset field	Same	Diverse	Total
Audio	207	308	515
Submissions	450	602	1,052
Blocks	1,397	3,046	4,443
Claims	3,972	8,079	12,051
Diagnosis	2,090	3,997	6,087
Correction	1,642	2,885	4,527

Table 2: VocalCoachBench dataset summary.

Figure 2: Annotation statistics and agreement in VocalCoachBench. (a) Top-3 issue distributions across subsets, with percentage-point shifts shown on the right. (b) Feedback richness in words and atomic claims per annotator-audio submission. (c) Expert agreement on structured targets, including parent-level Top-3 agreement, fine-label agreement, and same-song triplet rankings.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Parent	Label	Guidance
Technical Production	Breath	The overall use and control of breath, including phrase sustain, breath support, release of support during descending passages, and breath length or distribution.
	Vocalization	Overall vocal quality and production method, including phonation, placement, blocked or constricted sound, muffled or pulled-back placement, high-note stability, and resonance.
	Technique	The use and proficiency of special vocal skills, such as vibrato, vocal fry, bending, and ornamentation.
Musical Accuracy	Pitch	All cases related to pitch accuracy and stability, including leaps, final-note stability, pitch sagging in descending passages, and the accuracy of ornaments or modulations.
Musical Accuracy	Rhythm	Beat and timing, including rushing or dragging, pushing or pulling against the beat, and rhythmic expression through accent or dynamics.
Delivery	Diction	Lyric intelligibility and pronunciation habits, including vowel maintenance, nasal tone, hard or blocked articulation, and genre-stylistic pronunciation.

Appendix

Table 10: Fine-grained vocal-issue taxonomy guidance.

Subset	Source dataset	Tracks	License / terms
Same-song	DAMP-S-AG [ 32 ]	207	Smule Research Data License; research/education use only, no redistribution or commercial use
Diverse-song	PopBuTFy [ 21 ]	85	CC BY-NC-SA 4.0 with dataset terms of access
Diverse-song	N20EMv2 [ 47 ]	65	CC BY-SA 4.0
Diverse-song	LM-SSD [ 31 ]	7	CC BY 4.0
Diverse-song	NUS-48E [ 8 ]	41	Research-use-only corpus terms
Diverse-song	MedleyDB [ 2 ]	32	CC BY-NC-SA 4.0

Appendix

Table 11: Audio source datasets and license terms.

Error type	Rate
Atomicity violation	0.8%
Fragmentation error	0.0%
Omission	0.0%
Hallucinated addition	0.0%
Contextual role-assignment error	0.4%
Correction target-link error	0.0%

Appendix

Table 12: Single-author audit results for claim extraction. Correction target-link errors are computed over audited correction claims; other error rates are computed over audited extracted claims.

Comparison	Valid cases	Agreement	Cohen’s $\kappa$
LLM judge vs. Annotator 1	99	84.8	.798
LLM judge vs. Annotator 2	94	81.9	.759
LLM judge vs. authors, pooled	193	83.4	.779
Annotator 1 vs. Annotator 2, exact	93	80.6	.737
Annotator 1 vs. Annotator 2, binary	93	93.5	.870

Appendix

Table 13: Diagnosis judge validation summary. Exact agreement uses the four-way diagnosis labels. Binary agreement collapses Strict Hit and Coarse Hit into Hit , and Miss and Contradict into Non-Hit .

LLM judge $\backslash$ author	Strict	Coarse	Miss	Contra.
Strict	48	2	0	0
Coarse	13	33	2	0
Miss	0	0	47	1
Contra.	1	5	8	33

Appendix

Table 14: Pooled four-way confusion matrix for diagnosis judge validation. Rows are LLM judge labels and columns are author labels, after excluding BAD_CASE and blank labels.

Annotator 2 $\backslash$ Annotator 1	Strict	Coarse	Miss	Contra.
Strict	26	2	0	0
Coarse	7	12	2	2
Miss	0	2	24	3
Contra.	0	0	0	13

Appendix

Table 15: Author–author four-way confusion matrix for diagnosis validation

LLM judge / Expert	Valid	Weak	Invalid	Total
Valid	21	3	0	24
Weak	5	19	1	25
Invalid	4	2	17	23
Total	30	24	18	72

Appendix

Table 16: Correction judge validation confusion matrix for the three main correction labels. Rows are LLM judge labels and columns are expert labels.

Protocol	Prompt / task	Max tokens
Main	Open-ended coaching feedback	1,200
Main	Objective Top-3 issue prediction	420
Main	Segment-level issue classification	256
Simple	Overall coaching review	650
Simple	Short-form label prompts	40–80

Appendix

Table 17: Generation budgets used for model inference.

Panel of Figure 3 — Figure 3: Annotation interfaces. Top: same-song interface for holistic coaching feedback. Bottom: diverse-song interface for segment-level issue annotation.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 205 papers

PitchIn: Natural-Language Vocal Coaching from Unstructured Community Feedback
Oct 12, 2026Hayeon Bang, Hounsu Kim, Joonyong Park +2Audio-Language Models Audio Understanding
KAIST, Republic of Korea · The University of Tokyo, Japan · ETRI, Republic of Korea
MixJudge: Learning a Language-Queryable Critic for Vocal Mixing
Oct 22, 2026Kuan-Hsun Ho, Yuan-Chiao Cheng, Yen-Tung Yeh +1Audio-Language Models Audio Editing
Independent Researcher · Music Informatics Group, Georgia Institute of Technology · Music and AI Lab, National Taiwan University +1
What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings
Oct 9, 2026Mateusz Modrzejewski, Tomasz RadzikowskiAudio-Language Model Evaluation LLM-as-a-Judge
Institute of Computer Science, Warsaw University of Technology

Oct 12, 2026J/K move · Enter open · S save

PitchIn: Natural-Language Vocal Coaching from Unstructured Community Feedback

Hayeon Bang, Hounsu Kim, Joonyong Park, Jung-Jae Yu+1

KAIST, Republic of Korea · The University of Tokyo, Japan · ETRI, Republic of Korea

Audio-Language Models Audio Understanding Instruction Tuning

Open paper AskPDF arXiv

Effective vocal coaching connects performance issues with corrective advice, but recordings paired with expert diagnoses and advice are scarce. Online singing communities provide a potential source of supervision, but observations and advice are scattered across unstructured comments. We introduce PitchIn, audio-language models for natural-language vocal coaching, and the PeerCoach Dataset: 14,445 instruction examples covering Focused QA, Overall Feedback, and Recognition QA. We compare target combinations on Qwen2.5-Omni and train PitchIn on Focused QA and Overall Feedback using Qwen2.5-Omni and MOSS-Music. On VocalCoachBench, the proportion of requested diagnosis-correction pairs matching an expert issue with valid advice rises from 21.3% to 61.0% for Qwen2.5-Omni and from 31.4% to 51.2% for MOSS-Music. Direct training on source comments yields only 19.8% for Qwen2.5-Omni. For Qwen, Focused QA alone reaches 65.6%, but adding Overall Feedback improves alignment with expert reviews and human preference. Both models improve expert-review alignment over pretrained baselines and are significantly preferred for usefulness (92.0% and 67.3%, respectively). Demo examples are available at https://hayeonbang.github.io/PitchIn.

Abstract

Audio-Language Model Evaluation

Explore similar work

PitchIn: Natural-Language Vocal Coaching from Unstructured Community Feedback

MixJudge: Learning a Language-Queryable Critic for Vocal Mixing

What Do Audio-Language Judges of Generated Music Listen For? Four Probes with Controls from Expert Ratings