A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models

Shreeram Suresh Chandra; Ashwyn Sharma; Liang Lu; Xiutian Zhao; Leda Sari; Björn Schuller; Berrak Sisman

cs.CLOct 15, 2026

A Benchmark for Controllable Speaking-Style Captioning in Audio-Language Models

Authors: Shreeram Suresh Chandra, Ashwyn Sharma, Liang Lu, Xiutian Zhao, Leda Sari, Björn Schuller, Berrak Sisman

Organizations: Center for Language and Speech Processing, Johns Hopkins University, USA · Otter AI · The University of Texas at Dallas, USA · GLAM – Group on Language, Audio, & Music, Imperial College London, UK

Abstract

Speaking-style captioning (SSC) aims to generate natural language descriptions of how speech is delivered, capturing paralinguistic attributes such as vocal timbre, prosody, and expressivity. Many downstream applications, including conversational AI agents, controllable speech generation, and large-scale audio annotation, require dimension-specific style captions that describe targeted aspects of speech (e.g., speaker traits, emotion, or delivery style), rather than a single undifferentiated description. However, existing work lacks a unified formulation for controllable speaking-style captioning. We address this gap by formulating SSC as an instruction-following audio-language task, where explicit instructions specify the speaking-style dimensions to be described. We introduce StyleInstructCaps, the first standardized benchmark for controllable speaking-style captioning in audio-language models. StyleInstructCaps provides a dataset and evaluation framework that measures metadata groundedness, hallucination, instruction-following ability, generalization to unseen instructions and audio datasets, and speaker-style consistency. Across nine state-of-the-art audio-language models, we find that strong instruction-following ability does not necessarily imply grounded speaking-style understanding, and that SSC-specific instruction-diverse training substantially improves groundedness, controllability, and speaker consistency.

Figures & tables

Figure 1: Speaking-style captioning as an instruction-following audio-language task

Figure 2: Overview of StyleInstructCaps , which evaluates models on metadata groundedness, hallucination, instruction-following, generalization to unseen prompts and speech data, and speaker-style consistency.

Model name	LLM metadata groundedness		LLM hallucination severity (%)			Instruction following
	Meta score $\uparrow$	Hall. $\downarrow$	Low	Mid	High	IFA score $\uparrow$	IFA rate(%) $\uparrow$
Oracle Caption (PSC)	9.48	0.03	99.5	0.5	0	-	-
Oracle caption ( SIC-DB )	9.86	0.22	99.8	0.2	0	8.48	71.3
Audio-Flamingo-3	3.88	6.43	15.4	20.7	63.8	8.21	62.4
Voxtral-Mini-3B	4.99	4.29	48.6	35.0	16.4	8.15	68.0
Voxtral-Small-24B	4.38	1.99	73.5	16.8	9.68	7.51	67.8

Table 1: Metadata groundedness, hallucination severity, and instruction-following performance evaluated on StyleInstructCaps-MetaSet . Blue shading indicates relative performance for scalar metrics. Metadata groundedness and hallucination are averaged over a panel of three LLM judges (Qwen3-32B, Gemma-4-31B-it, GPT-OSS-20B) with three runs per judge (9 runs per model); instruction-following is scored with a single Qwen3-32B run.

Model	ID-balanced						UD-VCTK
Model	ARI $\uparrow$	NMI $\uparrow$	Purity $\uparrow$	InterSep $\uparrow$	EER $\downarrow$	AUC $\uparrow$	ARI $\uparrow$	NMI $\uparrow$	Purity $\uparrow$	InterSep $\uparrow$	EER $\downarrow$	AUC $\uparrow$
Audio-Flamingo-3	0.009	0.046	0.100	0.046	0.476	0.532	0.011	0.048	0.121	0.057	0.463	0.556
Voxtral-Mini-3B	0.033	0.122	0.152	0.133	0.399	0.643	0.003	0.028	0.103	0.063	0.486	0.519
Voxtral-Small-24B	0.067	0.182	0.186	0.157	0.363	0.674	0.011	0.052	0.119	0.076	0.480	0.532
Qwen2-Audio-7B-Instruct	-0.002	0.010	0.072	0.026	0.498	0.499	-0.002	0.007	0.083	0.025	0.496	0.498
MERaLiON-AudioLLM	0.081	0.276	0.213	0.233	0.428	0.604	0.039	0.146	0.167	0.169	0.476	0.534

Table 2: Speaker-style consistency evaluation using clustering and verification metrics on the ID-balanced and UD-VCTK sets. Darker shading indicates better performance per column. Higher is better for ARI, NMI, Purity, InterSep, and ROC-AUC; lower is better for EER.

Model	Meta score $\uparrow$	Hall. $\downarrow$
OpenAI-GPT-Audio	4.98	7.02
Audio-Flamingo-3	3.85	5.97
Voxtral-Mini-3B	4.57	7.12
Voxtral-Small-24B	3.45	6.45
Qwen2-Audio-7B-Instruct	4.62	6.97
MERaLiON-AudioLLM	3.95	5.72

Table 3: Evaluation on the unseen speech dataset. Metadata groundedness (1–10,

\uparrow

) and hallucination count (0–10,

\downarrow

) are averaged over the three-judge panel (Qwen3-32B, Gemma-4-31B-it, GPT-OSS-20B) with two runs per judge (6 runs per model) on the GigaSpeech-derived out-of-domain evaluation set.

Model	Narrative	Key-value	Tags/list	Min. imper.	Persona
Audio-Flamingo-3	7.26	3.34	4.12	6.39	7.47
Voxtral-Mini-3B	6.23	6.30	7.31	4.85	7.41
Voxtral-Small-24B	5.43	7.68	7.30	3.46	5.82
Qwen2-Audio-7B-Instruct	7.27	4.00	5.63	5.84	6.96
MERaLiON-AudioLLM	4.50	6.91	5.70	5.15	5.79
MOSS-Audio	8.00	8.57	8.34	8.39	8.33

Table 4: Instruction-following on unseen prompts, broken down by the output format each instruction requires. Each cell aggregates 32 clips over the instructions in that family, giving 128 to 416 judgements per system and 1,152 per system in total.

Figure 3: t-SNE visualization of speaker-style caption embeddings on UD-VCTK (6 speaker subset)

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Name	Size (hrs)
ParlerTTS Lyth and King (2024)	45k
PromptSpeech Guo et al. (2023)	37 (real)
Expresso Nguyen et al. (2023)	47
EARS Richter et al. (2024)	60
TextrolSpeech Ji et al. (2024)	300
MEAD-TTS Guan et al. (2024)	36

Appendix

Table 5: Summary of existing datasets relevant to SSC.

Benchmark	What Task It Benchmarks	Evaluation Metrics
CP-Bench Wang et al. (2025b)	Contextual Paralinguistic Reasoning: Evaluates the model’s ability to reason over both verbal content and non-verbal cues like emotion and prosody to answer questions	Answer accuracy, human preference ratings
SD-Eval Ao et al. (2024)	Spoken Dialogue Understanding: assesses understanding of dialogue that relies on paralinguistic cues (e.g., sarcasm, tone) for correct interpretations	Task accuracy, BLEU (for generation), MOS-style human ratings.
ParaS2SBench Yang et al. (2026)	Paralinguistic-Aware Speech-to-Speech: Evaluates if generated speech responses maintain appropriate paralinguistic alignment (style/emotion) with the input	Content similarity, style similarity, MOS (Mean Opinion Score)
ParaLBench Zhang et al. (2024)	General Computational Paralinguistics: A comprehensive test suite across multiple existing paralinguistic datasets to test generalizability	Accuracy, UAR (Unweighted Average Recall), F1 score, correlation metrics
BERSt Tuttösí et al. (2026)	Expressive Emotion Recognition: Specifically benchmarks emotion recognition in challenging conditions like shouted or highly expressive speech	SER (Speech Emotion Recognition) accuracy, UAR, WER (Word Error Rate - as a stress test)
EmoNet-Voice-Bench Schuhmann et al. (2026)	Fine-grained Emotion Understanding: Tests understanding of nuanced emotions beyond basic categories (e.g., distinguishing "cold anger" from "hot anger")	Classification accuracy, macro-F1, confusion matrix analysis

Appendix

Table 6: Benchmarks for Speech Paralinguistics Understanding

Panel of Figure 5 — Figure 5: Prompt template for generating six style-specific captions from an input metadata object.

Instruction type	Style instruction	Generated caption
Speaker idiosyncratic style	Describe the speaker’s inherent vocal identity by combining their pitch, accent, gender, and timbre-related characteristics.	A high-pitched, crisp, and nasal voice with an American accent, delivered by a female speaker, instantly conveys a sense of authority and confidence.
Situational contextual style	Set the scene by combining the speaking rate, noise level, and contextual intrinsic tags to describe a person speaking with intention in a specific setting.	Speaking with measured speed and authority, a female voice with an American accent cuts through a slightly clean environment, commanding attention as if addressing a gathering with purpose and conviction.
Expressive emotional style	Infer the emotional tone from the transcription and situational tags, and describe the prosody that creates this feeling, capturing the speaker’s attitude and sentiment.	A voice filled with awe and admiration, conveyed through a rising pitch contour and deliberate pacing, expresses genuine astonishment, as if the speaker is witnessing something remarkable for the first time.
Linguistic pragmatic style	Analyze the transcription to identify linguistic features, and describe the pausing and stress patterns that reveal the speaker’s emphasis and intention.	The speech unfolds with a steady, measured rhythm, punctuated by deliberate pauses that emphasize key phrases, as the speaker carefully chooses words to convey their thoughts, creating a sense of sincerity and authenticity.
Perceptual listener-centric style	Describe the overall effect of the voice on the listener, using impressionistic intrinsic tags to capture the impression the voice leaves on those who hear it.	Listening to this voice is like being drawn into a conversation with a charismatic and confident individual, as the speaker’s authoritative tone and crisp delivery create a sense of trust and engagement.
Holistic creative synthesis	Combine all elements to create a rich, narrative-style caption that captures the essence of the speaking-style, weaving together the speaker’s inherent qualities, situational context, emotional tone, and linguistic features into a vivid and immersive portrait.	Imagine a female speaker with a high-pitched, crisp, and nasal voice, speaking with measured speed and authority, her American accent shining through as she expresses awe and admiration in a slightly clean environment, conveying confidence and charisma with every word.

Appendix

Table 7: Style-specific instructions and example generated captions.

Panel of Figure 6 — Figure 6: Distributions of emotion (a), accent (b) signal-to-noise (SNR) ratio (c), and speech duration (d) in StyleInstructCaps-MetaSet .

Panel of Figure 7 — Figure 7: Distributions of (a) accent (b) duration and (c) signal-to-noise (SNR) ratio in StyleInstructCaps In-domain balanced set (ID-Balanced).

Data split	Number of utterances	Number of hours
PSC/SIC train set	1031840	2681.54
PSC/SIC validation set	11967	26.29
PSC/SIC evaluation set	14756	33.04
StyleInstructCaps-MetaSet	434	1.02
In-domain balanced set	400	0.82
Unseen Data(UD) - VCTK	320	0.31

Appendix

Table 8: Summary of dataset splits.

Attack Type	Primary Target Metric	Failure Mode Tested	Count
Type 1: Attribute Inversion	Meta-Score, Hallucination	The caption preserves fluency and overall structure while directly contradicting core metadata attributes such as gender, accent, or pitch. This tests whether the judge anchors on stylistic fluency rather than factual correctness with respect to the provided metadata.	87
Type 2: Instruction Parroting	Instruction-Following Accuracy (IFA)	The caption directly restates the instruction as if it were a caption, without providing any grounded acoustic or stylistic content. This tests whether the judge rewards instruction-aligned surface form rather than true instruction compliance.	87
Type 3: Verbosity Masking	Meta-Score, Hallucination, IFA	The caption uses long, fluent, impressionistic prose filled with prosodic language while deliberately avoiding verifiable factual claims. This tests whether the judge exhibits verbosity bias by inflating scores for stylistically rich but ungrounded captions.	87
Type 4: Hedged Hallucination	Hallucination Severity	The caption introduces incorrect factual claims through epistemically hedged language such as “may suggest” or “could be interpreted as.” This tests whether the judge is overly lenient toward softened or non-committal hallucinations.	87
Type 5: Cross-Speaker Swap	Meta-Score	The caption uses a real oracle caption from a different speaker with mismatched demographic and emotional attributes. This tests whether the judge grounds evaluation in the provided metadata or simply rewards any fluent, high-quality caption.	86

Appendix

Table 9: Summary of the five adversarial attack categories used in the LLM judge calibration experiment. Each attack probes a distinct potential failure mode of the LLM-as-a-judge evaluation pipeline.

Figure 8: Calibration analysis of speaking-style captioning models across metadata groundedness, instruction-following and hallucination severity metrics.

Measure	Value
Krippendorff’s $\alpha$ / Fleiss’ $\kappa$	0.347 / 0.346
Mean pairwise agreement	67.3%
Split / lopsided / unanimous items	45 / 44 / 11
Attention-check pass rate (all 15 listeners)	83.3%

Appendix

Table 10: Inter-listener reliability and quality control for the human pairwise study (13 retained listeners, 100 items). Chance level for pairwise agreement is 50%.

Judge	$n$	Agreement (%)	Cohen’s $\kappa$	$p$
Ensemble	99	85.9 [77.8, 91.9]	0.717	$1\times 10^{-13}$
Qwen3-32B	65	83.1 [73.8, 92.3]	0.661	$6\times 10^{-8}$
GPT-OSS-20B	96	82.3 [74.0, 89.6]	0.646	$1\times 10^{-10}$
Gemma-4-31B-it	92	84.8 [76.1, 91.3]	0.696	$6\times 10^{-12}$

Appendix

Table 11: Agreement between the human majority vote and the preference of each LLM judge and of the reported ensemble. Items on which a judge assigns equal scores are excluded, leaving

n

items. Brackets give 95% confidence intervals.

p

tests agreement against a 50% chance level.

Bucket	$n$	Agreement	$p$
B1 (near-tie)	15	73.3%	0.12
B2	20	80.0%	0.012
B3	24	87.5%	$3\times 10^{-4}$
B4	25	92.0%	$2\times 10^{-5}$
B5 (large gap)	15	93.3%	$1\times 10^{-3}$

Appendix

Table 12: Agreement between the human majority and the ensemble judge as a function of the absolute ensemble score gap

|\Delta|

, partitioned into five equal-width buckets.

p

is a two-sided exact binomial test against 50%.

Judge	Spearman $\rho$	Kendall $\tau$
Ensemble	0.895	0.780
Qwen3-32B	0.873	0.707
GPT-OSS-20B	0.855	0.692
Gemma-4-31B-it	0.881	0.714

Appendix

Table 13: Rank correlation over the 14 systems between the ranking of a Bradley–Terry model fit to human votes alone and the ranking produced by each LLM judge.

Judge	Family / size	Role
Qwen3-32B (thinking)	Qwen, 32B dense	Panel judge; instruction-following
Gemma-4-31B-it (thinking)	Gemma, 31B dense	Panel judge
GPT-OSS-20B (reasoning)	OpenAI OSS, 20B MoE	Panel judge

Appendix

Table 14: LLM judges used in the evaluation panel.

Evaluation set	Meta. groundedness	Hallucination	Instr.-following
MetaSet	3 $\times$ each judge	3 $\times$ each judge	1 $\times$ Qwen3-32B
OOD (GigaSpeech, VCTK)	2 $\times$ each judge	2 $\times$ each judge	1 $\times$ Qwen3-32B

Appendix

Table 15: Number of runs per judge and metric.

Judge pair	Spearman $\rho$ ( $n=14$ )
GPT-OSS-20B $\leftrightarrow$ Gemma-4-31B-it	0.978
Qwen3-32B $\leftrightarrow$ GPT-OSS-20B	0.921
Qwen3-32B $\leftrightarrow$ Gemma-4-31B-it	0.864

Appendix

Table 16: Cross-judge agreement: Spearman rank correlation between judges over the 14 benchmarked systems ( MetaSet metadata groundedness, per-judge means).

Eval set (table)	Metric	$n$	Friedman $p$	Sig. pairs	%
MetaSet (Tab. 1 )	Groundedness $\uparrow$	15	$<10^{-4}$	101/105	96.2
	Hallucination $\downarrow$	15	$<10^{-4}$	99/105	94.3
	Instr.-following $\uparrow$	13	$<10^{-4}$	73/78	93.6
OOD (Tab. 3 )	Groundedness $\uparrow$	13	$<10^{-4}$	68/78	87.2
	Hallucination $\downarrow$	13	$<10^{-4}$	67/78	85.9
OOD (Tab. 4 )	Instr.-following $\uparrow$	14	$<10^{-4}$	79/91	86.8

Appendix

Table 17: Omnibus Friedman test and number of significantly different system pairs (

p<0.05

) out of all

\binom{n}{2}

pairs, per metric and evaluation set.

n

is the number of systems compared. The MetaSet groundedness and hallucination tests use all 15 systems of Table 1 (two oracle caption sets, 12 open-weight models, and GPT-Audio). Instruction-following excludes the PSC oracle and GPT-Audio, which have no instruction-following scores. The OOD groundedness and hallucination tests use the 13 systems of Table 3 . The large majority (86–96%) of pairs are statistically distinguishable; the remaining pairs are statistically close.

Comparison	$\Delta$	$d$	Wilcoxon $p$
SSC-tuned SALMONN (SIC-DB, MT) $>$ baseline SALMONN	$+2.67$	1.51	$<10^{-4}$
SSC-tuned SALMONN (SIC-DB, ST) $>$ baseline SALMONN	$+2.62$	1.53	$<10^{-4}$
Oracle (SIC-DB) $>$ best real model	$+2.51$	1.70	$<10^{-4}$
Best open model $>$ GPT-Audio (closed SOTA)	$+1.65$	0.87	$<10^{-4}$

Appendix

Table 18: Headline comparisons on MetaSet metadata groundedness (

\uparrow

;

N=434

utterances).

\Delta

is the mean per-utterance score gap. A Cohen’s

d

above 0.8 is conventionally a large effect, so these gaps are both significant and sizeable.

Panel of Figure 10 — Figure 10: Hallucination evaluation prompt for comparing ground-truth metadata against a generated caption.

Panel of Figure 11 — Figure 11: Instruction-following evaluation prompt (Part 1): context, inputs, and evaluation framework overview.

Panel of Figure 12 — Figure 12: Instruction-following evaluation prompt (Part 2): scoring rubric, edge cases, and required output format.

Panel of Figure 13 — Figure 13: Unseen instruction prompts used to probe diverse instruction-following behaviors for speaking-style captioning.

Caption set	HD-D	Self-BLEU	Distinct-2	Distinct-3	Mean caption length	STD of caption lengths
Oracle caption (PSC)	0.732	—	0.0425	0.0795	32.5	7.97
Oracle caption (SIC)	0.771	0.304	0.152	0.310	34.9	12.5
Audio-Flamingo-3	0.748	0.0881	0.138	0.248	31.8	23.0
Voxtral-Mini-3B	0.813	0.123	0.247	0.463	145	78.1
Voxtral-Small-24B	0.807	0.145	0.281	0.484	72.0	56.8
Qwen2-Audio-7B-Instruct	0.836	0.0362	0.231	0.439	86.2	59.5

Appendix

Table 20: Lexical diversity and caption-length statistics across different caption sets and models.

Audio Captioning

Audio captioning turns sound into natural-language descriptions of events, attributes, speakers, music, and their timing.

See all 23 papers

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech
Feb 27, 2026Jielin Qiu, Jianguo Zhang, Zixiang Chen +13Audio-Language Model Evaluation Audio Captioning
Salesforce AI Research
Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech
Oct 8, 2026Shiao Zhu, Lianbo Liu, Sizhen Lyu +3TTS Synthesis Controllable Speech Generation
Institute of Science Tokyo, Tokyo, Japan · Independent Researcher, Tokyo, Japan
How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech
Jun 18, 2026Nityanand Mathur, Hamees Sayed, Wasim Madha +4TTS Synthesis Audio Diffusion Models

Feb 27, 2026J/K move · Enter open · S save

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

Jielin Qiu, Jianguo Zhang, Zixiang Chen, Liangwei Yang+12

Salesforce AI Research

Audio-Language Model Evaluation Audio Captioning

Open paper AskPDF arXiv

We introduce AudioCapBench, a benchmark for evaluating audio captioning capabilities of large multimodal models. \method covers three distinct audio domains, including environmental sound, music, and speech, with 1,000 curated evaluation samples drawn from established datasets. We evaluate 13 models across two providers (OpenAI, Google Gemini) using both reference-based metrics (METEOR, BLEU, ROUGE-L) and an LLM-as-Judge framework that scores predictions on three orthogonal dimensions: accuracy (semantic correctness), completeness (coverage of reference content), and hallucination (absence of fabricated content). Our results reveal that Gemini models generally outperform OpenAI models on overall captioning quality, with Gemini3Pro achieving the highest overall score (6.00/10), while OpenAI models exhibit lower hallucination rates. All models perform best on speech captioning and worst on music captioning. We release the benchmark as well as evaluation code to facilitate reproducible audio understanding research.

Abstract

Audio Captioning

Explore similar work

AudioCapBench: Quick Evaluation on Audio Captioning across Sound, Music, and Speech

Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech

How Do Instructions Shape Speech? Cross-Attention Attribution for Style-Captioned Text-to-Speech