AcoustiClaim: A Numeric Claim Benchmark with Instrument Ground Truth

Sheng-Tse Lin; Siyuan Zhai; Chien-Liang Kuo; Massa Baali; Bhiksha Raj

cs.SDSep 24, 2026

AcoustiClaim: A Numeric Claim Benchmark with Instrument Ground Truth

Authors: Sheng-Tse Lin, Siyuan Zhai, Chien-Liang Kuo, Massa Baali, Bhiksha Raj

Organizations: Carnegie Mellon University

Abstract

Audio language models state numbers for acoustic quantities, and neither human opinion nor a judge model says whether such a number is true of the signal. AcoustiClaim extracts each numeric claim from free text, scores it against the instrument that defines the quantity, and classes each quantity by where its reference can be read. Four open-weight systems and one closed model, asked for ten quantities five ways on two corpora, fill 207 cells. Of these, 49 emit fewer than five distinct values, and eight of the 158 cells that can be ranked exceed a rank correlation of 0.3, the bar we set, three with an interval clear of it, five of them one closed model reading pitch. Error sits at or above a constant-predictor floor in every ranked cell but three. The reference decoder we train declines the five voice quantities in prose on 95% of mixtures, with nothing withheld, and states them on the clean twins, reproducing its targets' rule from audio alone. With a calibrated threshold, withholding lowers error on all ten quantities on the mixtures in the mean and on eight at every split, against at most 0.6% from a random selector. A linear baseline orders errors at least as well as ours. F0 s.d. and shimmer stay above the constant floor.

Figures & tables

Figure 1: One recording decoded twice by our system, on the two-talker mixture and on its clean twin.

Quantity	Tier	Case	Stated	Ours $\rho$ $\pm$ s.d.	Ridge $\rho$	Ours nMAE	Ridge nMAE
SRMR	Exact	(ii)	$1.00$	$+0.918\pm 0.003$	$+0.881$	[ $0.394$	[ $0.473$
SNR	Measurable	(ii)	$1.00$	$+0.972\pm 0.005$	$+0.944$	[ $0.198$	[ $0.274$
Speaking rate	Stem-only	(i)	$1.00$	$+0.497\pm 0.030$	$+0.650$	[ $0.881$	[ $0.802$
Pause count	Stem-only	(i)	$1.00$	$+0.504\pm 0.019$	$+0.465$	[ $0.812$	[ $1.059$ ]
Pause rate	Stem-only	(i)	$1.00$	$+0.409\pm 0.043$	$+0.470$	[ $0.888$	[ $0.902$
F0 mean ‡	Masked †	(ii)	$0.95$	$+0.434\pm 0.029$	$+0.482$	[ $0.876$	[ $0.829$

Table 1: Spearman correlation against the instrument, readout A on a 300-mixture, five-speaker Libri2Mix subset, beside the matched ridge. The five rows marked

\ddagger

are read on the clips’ clean twins, the mixtures set aside, and

\dagger

is a reference read outside the overlap windows detected on the mixture, so the two twin F0 rows carry the mixture’s own windows and Tier is the tier on the mixture. Case (i) sums and (ii) quotients, Eq. ( 1 ), Stated the share of the scored clips, the twins on

\ddagger

rows, the decoder speaks on, and

\pm

a sample s.d. over five training seeds. Marks, here and in Table 2 : a bracket an nMAE at or above the floor, a dash under 25 parsed claims, c under five distinct values, excl a reference we do not score on that corpus,

\lx@sectionsign

a cell read on part of the clips,

\lx@paragraphsign

two training seeds, and a red arrow a value over the bar. Praat’s two half-stem readings agree at

0.42

0.19

and

0.54

on jitter, shimmer and HNR,

0.60

0.32

and

0.70

stepped up to full length, which caps any correlate at

0.77

0.57

and

0.84

; jitter’s

+0.809

exceeds its cap and its within-speaker

+0.646

does not. The other seven are a mixing parameter, deterministic or masked, so split-half does not apply.

System		SRMR	SNR	Speaking	Pause	Pause	F0	F0	Jitter	Shimmer	HNR	$>0.3$
				rate	count	rate	mean	s.d.				/ cells
Libri2Mix, where the two F0 columns are masked references read on the mixture, these clips having no twin
	Qwen2-Audio-7B	+0.191	+0.024	−0.035	−0.045	+0.050	+0.094	+0.024	+0.012	+0.096	+0.052	0/23
	Qwen2.5-Omni-7B	−0.057	+0.087	+0.021	c	−0.086	+0.062	+0.009	+0.004	+0.065	+0.037	0/29
	Granite-Speech-8B	+0.047	c	−0.128	−0.097	−0.050	c	c	−0.104	−0.063	+0.031	0/17
	Audio-Flamingo-3	+0.124	$\uparrow$ +0.304	−0.049	+0.047	+0.005	+0.153	−0.041	+0.115	+0.117	+0.018	1/29

Table 2: Largest Spearman correlation over the rungs against the instrument. Every row is parsed emitted text, the panel pooled over eight samples by clip, mixtures only, our system trained on train-clean-100 and scored here on test clips it never saw. Marks are Table 1 ’s, and the last column counts cells over the bar at every rung, six arrows here for eight such cells. Our row carries three bases,

\lx@sectionsign

five voice cells over four seeds on the

6\%

of panel clips it speaks on, five unmarked at full coverage over five seeds, and

\lx@paragraphsign

the AMI row over two; shimmer’s

+0.412

spans

[0.145,0.623]

on Fisher-

z

Figure 2: One $9.44$ s mixture in four columns. Rows are the mixture, the second talker’s spans, the frames the instrument reads and our 58-token decomposition. Each box is the shortest window holding half the reference’s magnitude and each peak a frame’s height over an even split.

Figure 3: (a) Reduction in selective risk per quantity, each wedge thickening from

16

42\%

withheld, the stroke at the reported cut, the ticks the five held-out splits, the right-hand numbers each row’s reduction there, pooled over mixtures and twins, and SRMR a positive control, its reference Exact. (b) Rows as in (a), risk-coverage area (AURC), the selective risk integrated over coverage, of that residual ridge minus ours on the full test set, one mark per training seed, four at

G=11

and five at

G=39

G

the number of speaker clusters the area is computed over, the fifth seed scored at

G=39

only, each bar filled by whether the nine marks all favour one side, right of zero our win.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 206 papers

MixJudge: Learning a Language-Queryable Critic for Vocal Mixing
Oct 22, 2026Kuan-Hsun Ho, Yuan-Chiao Cheng, Yen-Tung Yeh +1Audio-Language Models Audio Editing
Independent Researcher · Music Informatics Group, Georgia Institute of Technology · Music and AI Lab, National Taiwan University +1
All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation
Apr 27, 2026Leonardo Haw-Yang Foo, Chih-Kai Yang, Chen-An Li +2Audio-Language Model Evaluation Modality Reliance in Multimodal Models
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)
TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence
Oct 15, 2026Giovana Morais, Yujin Kim, Atilay Kucukoglu +1Audio-Language Model Evaluation Audio-Language Models
Music and Audio Research Lab, NYU · Integrated Design and Media, NYU

Oct 22, 2026J/K move · Enter open · S save

MixJudge: Learning a Language-Queryable Critic for Vocal Mixing

Kuan-Hsun Ho, Yuan-Chiao Cheng, Yen-Tung Yeh, Yi-Hsuan Yang

Independent Researcher · Music Informatics Group, Georgia Institute of Technology · Music and AI Lab, National Taiwan University+1

Audio-Language Models Audio Editing Audio-Language Model Evaluation

Open paper AskPDF arXiv

Mixing requires not only applying processing but also assessing the resulting mix. Audio-language models can provide conversa- tional mixing advice, but existing evaluations do not separately test whether they can judge specific mixing issues. Since professional supervision for such judgments is scarce, we construct a curriculum of 11 vocal-mix issues at multiple severity levels and use it to train MixJudge, a language-queryable audio critic. MixJudge shows its ability to retrieve mixing issues, distinguish opposite issues, and order the severity of an issue. Moreover, its judgments transfer to menu selection: given a set of processing candidates, MixJudge often favors versions acoustically closer to a held-out professional version, demonstrating how explicit mixing judgment can support comparison and downstream mixing decisions.

Abstract

Audio-Language Model Evaluation

Explore similar work

MixJudge: Learning a Language-Queryable Critic for Vocal Mixing

All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation

TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence