Do Audio LLMs HEAR Pain? Tracing Acoustic Evidence from Speech to Language Reasoning

Wenqi Li; Xiangjue Dong; Sulaiman Vesal

cs.CLOct 28, 2026

Do Audio LLMs HEAR Pain? Tracing Acoustic Evidence from Speech to Language Reasoning

Authors: Wenqi Li, Xiangjue Dong, Sulaiman Vesal

Organizations: Microsoft

Abstract

Large audio language models (LALMs) can process speech directly, but it remains unclear how well they use the non-lexical acoustic information that speech carries. We study this question through pain, using TAME-Pain, which contains pain-neutral Harvard sentences and spoken pain ratings recorded at different self-reported pain levels. We introduce HEAR, a four-stage diagnostic framework that traces pain-related information from the speech signal to the model's prediction, and apply it to seven closed-source and open-weight LALMs. On the Harvard sentences, where pain can only be inferred from the voice, all models perform at chance level, and same-speaker references or few-shot examples improve balanced accuracy by no more than six points. On the spoken ratings, the same models reach 87.5-99.7% balanced accuracy. Layer-wise probing of open-weight LALMs shows that pain can be decoded from early audio representations (74.1% balanced accuracy for binary prediction and 45.0% for three-class severity), but that this information, along with most pain-associated acoustic cues, decreases in deeper layers. In an acoustic perception test, weaker models cannot distinguish even large changes in these cues, whereas stronger models can but still fail to infer pain. Finally, we recover pain-related acoustic cues and directly reinject them as structured text into the prompt, which improves pairwise pain prediction by up to 32 points. These results indicate that current LALMs encode paralinguistic information in early layers that is largely unavailable when the model produces its answer.

Figures & tables

Figure 1: LALMs recognize pain when it is stated in words, but fail when it must be inferred from the voice. Examples from TAME-Pain ( Dao et al., 2025 ) , recorded during a pain-inducing cold pressor task. (a) Speakers read fixed, pain-neutral Harvard sentences, so pain is detectable only from the acoustic signal. (b) Speakers state their pain rating aloud, conveying pain both lexically and acoustically.

Figure 2: Overview of our HEAR analysis framework. (RQ1) : we compare seven LALMs’ pain prediction under the neutral-sentence and spoken-rating settings. (RQ2) : we identify pain-associated acoustic cues and probe each layer of LALMs for pain representations. (RQ3) : we test whether LALMs perceive large, controlled contrasts in these cues. (RQ4) : we add acoustic cue values to the prompt as text.

Figure 3: RQ1 pain prediction task settings . Each recording is presented as audio, its transcript, or both. The figure is a schematic illustration and questions are simplified. The exact prompts are given in Appendix H .

Pain vs. No Pain (Balanced Accuracy, %)
		Neutral Harvard Sentence			Spoken Pain Rating			Neutral + Pairwise		Neutral + 20-shot
	Model	Text ∗	Audio	Text+Audio	Text	Audio	Text+Audio	Audio	Text+Audio	Audio	Text+Audio
Closed- Weight	gpt-realtime-mini	92.7	48.1	50.2	98.9	98.7	98.7	48.6	45.2	52.8	50.4
	gpt-realtime	93.7	49.6	50.7	98.7	98.9	98.9	50.0	50.0	54.0	48.5
	gemini-3.5-flash	99.1	49.9	50.0	98.8	99.2	99.2	40.9	43.3	47.6	47.2
	gemini-3.1-pro-preview	99.5	50.0	50.0	99.0	99.7	99.4	52.4	49.5	49.8	51.9

Table 1: Results on pain detection across 7 LALMs and 4 settings . Text, Audio, and Text+Audio denote the input modalities. All scores are balanced accuracy (%), for which chance level is 50.0 for binary detection and 33.3 for 3-class severity, except the Neutral Harvard Sentence / Text column ( ∗ ), which reports accuracy. Because neutral transcripts carry no pain information, all text-only items are labeled “no pain”. Kimi-Audio does not accept text-only input (–). Bold and underline mark the best and second-best results in each column.

Figure 4: Acoustic feature groups associated with pain (34 features with

q<.05

and

|\beta|\geq.10

; groups in Table 10 ).

|\beta|

: within-speaker standardized association strength. Filled dots: group mean (feature number in parentheses); open circles: individual features.

Figure 5: Pain and acoustic-feature information across the layers of Qwen2.5-Omni-7B. (a, b, d, e) Out-of-fold balanced accuracy of probes trained on each layer’s representation (speaker-grouped five-fold cross-validation; line = mean over three seeds, band =

\pm 1

SD). Y-axis ranges differ between rows. (c, f) Held-out

R^{2}

(on test speakers) of ridge regression recovering the pain-associated acoustic features from the same representations, averaged within each feature group (Table 10 ). The projector output (P) is the input to Thinker layer 0.

	Pain vs. No Pain		3-Class Severity
Probe	Layer	BA	Layer	BA
Linear	Block 1	68.8 $\pm$ 0.8	Block 1	45.0 $\pm$ 0.9
MLP	Block 1	67.1 $\pm$ 2.0	Input	43.1 $\pm$ 1.7
CRNN	Block 1	74.1 $\pm$ 1.6	Input	44.4 $\pm$ 3.5

Table 2: Supervised probing results on Qwen2.5-Omni-7B . Balanced accuracy (BA, %) is reported on the 10 held-out test speakers. Input denotes the audio encoder input and Block 1 the first encoder block. Results are reported as mean

\pm

standard deviation over three random seeds. Experiment setup is in Appendix A.1

Figure 6: LALM accuracy (%) on the perception test. Each item pairs two clips created from the same recording by shifting one acoustic feature in opposite directions; the model picks the clip higher on that feature (chance

=50

{}^{*}p<.05

{}^{**}p<.01

, two-sided binomial test against chance. † Human: pooled over three listeners,

n=20

pairs per feature, Fleiss’

\kappa=0.95

	Pain vs. No Pain						3-Class Pain Severity
	Pairwise			20-shot			Pairwise			20-shot
Model	Audio	Text+Audio	AF	Audio	Text+Audio	AF	Audio	Text+Audio	AF	Audio	Text+Audio	AF
gpt-realtime-mini	66.3 $\uparrow$ 17.7	64.9 $\uparrow$ 19.7	77.4	63.1 $\uparrow$ 10.3	60.1 $\uparrow$ 9.7	58.8	40.0 $\uparrow$ 6.8	38.2 $\uparrow$ 4.6	39.1	36.5 $\downarrow$ 2.4	35.9 $\uparrow$ 3.8	38.4
gpt-realtime	53.8 $\uparrow$ 3.8	51.9 $\uparrow$ 1.9	67.8	60.4 $\uparrow$ 6.4	63.7 $\uparrow$ 15.2	60.7	34.6 $\downarrow$ 0.5	34.7 $\uparrow$ 1.6	36.2	38.3 $\uparrow$ 3.3	37.9 $\uparrow$ 6.6	46.6
gemini-3.5-flash	71.6 $\uparrow$ 30.7	61.5 $\uparrow$ 18.2	84.6	54.5 $\uparrow$ 6.9	53.1 $\uparrow$ 5.9	54.4	40.0 $\uparrow$ 6.9	36.5 $\uparrow$ 4.9	38.7	29.4 $\downarrow$ 5.8	33.8 $\downarrow$ 2.1	37.5
gemini-3.1-pro-preview	82.2 $\uparrow$ 29.8	81.7 $\uparrow$ 32.2	89.4	60.7 $\uparrow$ 10.9	60.6 $\uparrow$ 8.7	58.8	33.3 $\uparrow$ 0.2	32.5 $\uparrow$ 0.8	42.6	41.9 $\uparrow$ 5.8	42.6 $\uparrow$ 4.0	43.0

Table 3: Effect of adding explicit acoustic cues to the prompt on the pairwise and 20-shot Neutral Harvard Sentence settings (balanced accuracy, %). Arrows give the change relative to the baseline in Table 1 . AF (acoustic features only input) has no baseline. Kimi-Audio does not accept text-only input (–). Best result per column in bold .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Test Speaker IDs
p10555	p12460	p18355	p40505	p50580
p52610	p59520	p77220	p79550	p91315

Appendix

Table 4: Held-out test speakers for all supervised probing experiments on TAME-Pain dataset.

Hyperparameter	Search space
Layer	Encoder input; encoder blocks 1, 7, 13, 19, 25, 32; projector; LM layers 5, 11, 16, 22, 28
Head	Linear, MLP, CRNN

Appendix

Table 5: Pain Classification Probing hyperparameter search space for Qwen2.5-Omni-7B.

Prompt	Setting	Input	Description	$N$
Pain vs. No Pain
P H.1	Spoken Pain Rating	Audio	Classify pain from a single recording.	856
P H.1	Neutral Harvard Sentence	Audio	Classify pain from a single recording.	3,802
P H.1	Spoken Pain Rating	Text	Classify pain from the transcript only.	856
P H.1	Neutral Harvard Sentence	Text	Classify pain from the transcript only.	3,802
P H.1	Spoken Pain Rating	Text+Audio	Classify pain from a recording and its transcript.	856

Appendix

Table 6: Task descriptions. P

n

refers to Prompt

n

in Appendix H .

N

denotes the number of model queries per task

Model	Version / Checkpoint	Decoding	Max Tokens
Closed-Source Models
gpt-realtime-mini ( OpenAI, 2025a )	API (Sep. 2026)	default	1024
gpt-realtime ( OpenAI, 2025b )	API (Sep. 2026)	default	1024
gemini-3.5-flash ( Google DeepMind, 2026b )	API (Sep. 2026)	$T=0$	4096
gemini-3.1-pro-preview ( Google DeepMind, 2026a )	API (Sep. 2026)	$T=0$	1024
Open-Weight Models

Appendix

Table 7: Evaluated LALMs. The Azure gpt-realtime API does not expose decoding hyperparameters or random seeds, so we use its default settings. All other models use temperature

T=0

. gemini-3.5-flash uses a larger output limit because shorter limits frequently produced truncated responses.

Figure 7: Layer-wise pain decoding with a linear head in Kimi-Audio. Balanced accuracy of the probe at each layer of the Whisper encoder, the LM trunk (text path), and the MIMO audio branch, which is drawn from the trunk layer it branches off (dotted green line). Lines show the mean over 3 seeds (each averaged over 5 folds); shaded bands show

\pm 1

SD across seeds. (a) Binary pain detection. (b) Three-class severity.

Neutral Harvard Sentence
	gpt-realtime-mini		gpt-realtime		gemini-3.5-flash		gemini-3.1-pro-preview		Qwen2.5-Omni-7B		Qwen3-Omni-30B-A3B		Kimi-Audio
Modality	Binary	3-class	Binary	3-class	Binary	3-class	Binary	3-class	Binary	3-class	Binary	3-class	Binary	3-class
Text	100.0	99.8	100.0	100.0	100.0	100.0	100.0	99.9	100.0	100.0	100.0	100.0	–	–
Audio	97.3	98.7	100.0	100.0	99.4	99.8	100.0	99.8	100.0	100.0	100.0	100.0	100.0	100.0
Text+Audio	100.0	100.0	100.0	100.0	99.8	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Spoken Pain Rating

Appendix

Table 8: Parsing success rate (%) of model outputs. A response counts as successfully parsed if it contains exactly one valid label. Unparsed responses are counted as incorrect in Table 1 .

Figure 8: Confusion matrices for every model and condition in Table 1 . Rows are true labels and columns are predicted labels; each cell gives the percentage of items with that true label, and outlined cells are correct predictions. Unparsed outputs (Table 8 ) are not shown but remain in the denominator, so a row may sum to slightly less than 100. In the pairwise condition, the binary labels are Clip 1 and Clip 2. ∗ All Neutral Harvard Sentence text-only items are labeled No pain (binary) or Mild (three-class), since neutral transcripts carry no pain information.

Feature	$\beta$	95% CI	$q$
Frequency
F0semitoneFrom27.5Hz_sma3nz_amean 1	0.05	[0.01, 0.08]	.011
F0semitoneFrom27.5Hz_sma3nz_stddevNorm	0.19	[0.15, 0.22]	$<$ .001
F0semitoneFrom27.5Hz_sma3nz_percentile20.0	0.00	[ $-$ 0.04, 0.03]	.846
F0semitoneFrom27.5Hz_sma3nz_percentile50.0	0.16	[0.13, 0.20]	$<$ .001
F0semitoneFrom27.5Hz_sma3nz_percentile80.0	0.15	[0.11, 0.18]	$<$ .001

Appendix

Table 9: Within-speaker association between all 88 eGeMAPSv02 ( Eyben et al., 2016 ) features obtained by openSMILE ( Eyben et al., 2010 ) and self-reported pain score on the training speakers, grouped by feature family. Pain and each feature are centered within speaker and normalized;

\beta

(range

-1

1

) is the linear-regression slope, equivalent to the within-speaker correlation. Positive

\beta

means pain is higher when the feature is above the speaker’s usual value; larger

|\beta|

means a stronger association. 95% CI: Wald interval of the fixed effect (

\beta

± 1.96 SE); an interval excluding 0 indicates a reliable association.

q

: Benjamini–Hochberg FDR-corrected

p

-value over all 88 features; smaller means less likely to be a false positive. Bold marks features with a significant and non-negligible association (

q<.05

and

|\beta|\geq.10

). Thresholds are applied before rounding. Feature names follow openSMILE eGeMAPSv02 output.

Group / eGeMAPSv02 feature	Group / eGeMAPSv02 feature
Pitch level	Formants
F0semitoneFrom27.5Hz_sma3nz_percentile50.0	F1frequency_sma3nz_stddevNorm
F0semitoneFrom27.5Hz_sma3nz_percentile80.0	F3amplitudeLogRelF0_sma3nz_stddevNorm
Pitch variability	Spectral flux
F0semitoneFrom27.5Hz_sma3nz_stddevNorm	spectralFlux_sma3_amean
F0semitoneFrom27.5Hz_sma3nz_pctlrange0-2	spectralFlux_sma3_stddevNorm

Appendix

Table 10: Feature groups in Figure 5 (c). Each line in the figure is the mean held-out

R^{2}

over the features listed for its group. The 34 features are those marked in bold in Table 9 .

Cue	Target feature	Manipulation method	Range	Target shift	Largest off-target feature	Off-target shift	Pairs
Spectral balance	alphaRatioV_sma3nz_amean	High-shelf filter at 1 kHz	$+$ 4 dB	0.89	mfcc1V_sma3nz_amean	1.46	760
	$-$ hammarbergIndexV_sma3nz_amean	(SciPy biquad)	$-$ 4 dB	$-$ 0.87	mfcc1V_sma3nz_amean	1.36
	slopeV0-500_sma3nz_amean
	slopeV500-1500_sma3nz_amean
Loudness	loudness_sma3_amean	Waveform gain	$+$ 1 dB	0.71	spectralFluxUV_sma3nz_amean	1.23	760
		(NumPy)	$-$ 2 dB	$-$ 1.28	spectralFluxUV_sma3nz_amean	2.10

Appendix

Table 11: Construction and validation of the audio perturbations. We select ten representative acoustic attributes spanning the four parameter groups across 88 features. Each manipulation aims to shift its target eGeMAPSv02 parameter(s) while leaving parameters of other feature families unchanged (parameters within one family are inherently correlated). Shifts are measured on 24 recordings, each from a different training speaker, as the perturbed recording minus its matching null resynthesis (same tool at zero strength), divided by that parameter’s mean within-speaker SD over the training speakers. Target shift is the median of this shift over the 24 recordings, oriented so that a leading

-

on a target parameter means it is expected to decrease, and averaged over the target parameters when a cue has several. Range : for each cue and direction, the candidate strength whose Target shift is closest to

\pm

1 SD. Largest off-target feature : the parameter with the largest median absolute shift after excluding the target’s own family (e.g., all F0 parameters for the pitch cues). Its ratio to

|

Target shift

|

is flagged when above 0.5 but is only reported, not used for filtering. RMS level is matched to the original recording for every manipulation except Loudness. Filtering. A setting passes calibration if

|

Target shift

|\in[0.5,1.5]

and none of the 24 recordings fails processing; passing settings are shown in bold, and non-bold settings are excluded. WORLD could not lower jitter, shimmer or breathiness (Target shift

\approx

0), so these cues were tested in the increase direction only, contrasting a 2

\times

against the 1

\times

strength. Pairs : test recordings (760 in total, from 10 held-out speakers) for which both perturbed versions of a cue were processed successfully; the effect of each bidirectional cue is estimated from the

+

versus

-

version of the same recording. Cues marked † were excluded from the analysis because more than 5% of their 1

\times

\times

pairs showed no achieved change in the target parameter (jitter 10.0%, shimmer 8.9%, breathiness 9.3%). In total we generated 18,240 test-set versions for the 17 passing settings and 7 null resyntheses (4 failed, all Pauses), 2,280 2

\times

WORLD versions, and 4,560 round-trip null resyntheses for the PSOLA and speech-rate cues. a Each sample is multiplied by

(e/\tilde{e})^{k-1}

, where

e

is the smoothed Hilbert envelope,

\tilde{e}

its median and

k

the exponent;

k>1

exaggerates and

k<1

flattens loudness fluctuations. b Non-speech stretches detected by WebRTC VAD are resampled to the given multiple of their length; speech is left unchanged. c One of the 24 calibration recordings also failed processing. d 760 minus 4 recordings whose perturbation failed and 1 recording whose

+

and

-

versions had identical target values.

Attribute	Effect	95% CI	Sym.	Null slopes	Verdict	Flip (+ / $-$ )
Spectral balance	0.62	[0.59, 0.66]	0.05	0.62 / –	credible	0.22 / 0.18
Loudness	0.13	[0.11, 0.14]	$-$ 0.03	0.13 / –	credible	0.03 / 0.04
Loudness variability	0.02	[0.01, 0.05]	0.01	0.02 / –	credible	0.01 / 0.01
Mean pitch	0.19	[0.04, 0.34]	$-$ 0.20	0.10 / 0.24	credible	0.11 / 0.18
Pitch variability	$-$ 0.23	[ $-$ 0.43, $-$ 0.001]	$-$ 0.23	0.06 / $-$ 0.17	no effect	0.19 / 0.12
Speech rate	$-$ 0.14	[ $-$ 1.03, 0.23]	0.57	$-$ 1.84 / 0.06	artifact	0.22 / 0.26

Appendix

Table 12: Effect of each perturbation on the binary pain logit of the CRNN probe at audio-encoder block 1 (760 held-out clips, 10 speakers; perturbations as in Table 11 ; ‡ 755 after one zero-dose pair). Effect:

(y^{+}-y^{-})/(s^{+}-s^{-})

averaged over clips and three seeds, where

y

is the pain logit in units of its SD on original test clips and

s

the achieved target shift in within-speaker SD; positive means raising the attribute pushes toward pain . CI: speaker bootstrap. Sym.:

(y^{+}+y^{-})/2-y^{0}

, the direction-independent change. Null slopes (standard / round-trip): one-sided slopes against the standard and round-trip null resynthesis (round-trip only for PSOLA and time-stretch; undefined for pauses because some clips had zero shift relative to the null). Credible: effect CI excludes 0 and the two null slopes agree in sign; artifact: effect CI includes 0 but the Sym. CI excludes 0. Flip: fraction of binary predictions that change relative to the original audio under the + and

-

perturbation.

Cue	Target feature(s)	Manipulation method	Low	High	$\Delta$ target	Pairs
Pitch level	F0semitoneFrom27.5Hz_sma3nz_amean	PSOLA F0 shift (Praat)	$-$ 8 st	$+$ 8 st	15.87	100
Pitch variability	F0semitoneFrom27.5Hz_sma3nz_stddevNorm	PSOLA F0-range scaling (Praat)	$\times$ 0.0	$\times$ 2.5	3.96	100
	F0semitoneFrom27.5Hz_sma3nz_pctlrange0-2
Loudness level	loudness_sma3_amean	Waveform gain	$-$ 20 dB	$+$ 20 dB	30.95	100
	equivalentSoundLevel_dBp
Loudness dynamics	loudness_sma3_stddevNorm	Envelope exponent (Hilbert)	0.3	2.0	10.62	100

Appendix

Table 13: Construction of the maximal low/high acoustic contrasts. Each of 100 neutral Harvard-sentence recordings from TAME-Pain is rendered once at the low and once at the high magnitude.

\Delta

target is the median high-minus-low difference in the target composite (in within-speaker SD units); every pair moved in the intended direction (100/100 per analysed cue). † Generated but excluded from analysis.

Feature	L1	L2	L3	Pooled
Pitch level	100	100	100	100.0
Loudness level	100	100	100	100.0
Spectral tilt	100	100	100	100.0
Pitch variability	100	90	100	96.7
Loudness dynamics	100	95	95	96.7
Overall	100	97	99	98.7

Appendix

Table 14: Per-listener accuracy (%) on the acoustic perception test (

n=20

pairs per feature). Fleiss’

\kappa=0.95

across the three listeners; all three agreed on 96 of 100 pairs, and the majority vote was correct on all 100.

Neutral + Pairwise
	gpt-realtime-mini		gpt-realtime		gemini-3.5-flash		gemini-3.1-pro-preview		Qwen2.5-Omni-7B		Qwen3-Omni-30B-A3B		Kimi-Audio
Modality	Binary	3-class	Binary	3-class	Binary	3-class	Binary	3-class	Binary	3-class	Binary	3-class	Binary	3-class
Audio	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
Text+Audio	100.0	100.0	100.0	100.0	99.5	99.6	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
AF	100.0	100.0	100.0	100.0	100.0	98.9	100.0	100.0	100.0	100.0	100.0	100.0	–	–
Neutral + 20-shot

Appendix

Table 15: Parsing success rate (%) of model outputs when acoustic cues are added to the prompt (settings as in Table 3 ). A response counts as successfully parsed if it contains exactly one valid label. Unparsed responses are counted as incorrect. – indicates not evaluated.

	Pain vs. No Pain						3-Class Pain Severity
	Neutral Harvard Sentence			Spoken Pain Rating			Neutral Harvard Sentence			Spoken Pain Rating
Model	Audio	Text+Audio	AF	Audio	Text+Audio	AF	Audio	Text+Audio	AF	Audio	Text+Audio	AF
gpt-realtime-mini	50.8 $\uparrow$ 2.7	50.3 $\uparrow$ 0.1	49.8	98.5 $\downarrow$ 0.2	98.2 $\downarrow$ 0.5	50.5	34.0 $\uparrow$ 0.3	33.7 $\uparrow$ 0.2	34.3	90.7 $\downarrow$ 5.3	88.3 $\downarrow$ 5.3	33.0
gpt-realtime	50.0 $\uparrow$ 0.4	51.7 $\uparrow$ 1.0	50.0	98.8 $\downarrow$ 0.1	98.9 $\pm$ 0.0	50.0	31.2 $\downarrow$ 1.6	33.4 $\downarrow$ 0.7	33.3	98.9 $\pm$ 0.0	98.8 $\downarrow$ 0.5	34.3
Qwen2.5-Omni-7B	50.0 $\uparrow$ 0.2	50.1 $\uparrow$ 0.1	50.4	98.1 $\downarrow$ 0.7	98.8 $\uparrow$ 0.2	48.4	33.4 $\pm$ 0.0	33.4 $\uparrow$ 0.3	33.9	93.6 $\downarrow$ 4.4	89.6 $\downarrow$ 3.7	33.4
Qwen3-Omni-30B	50.3 $\uparrow$ 0.1	49.9 $\downarrow$ 0.2	50.2	99.5 $\pm$ 0.0	99.2 $\uparrow$ 0.1	50.1	33.4 $\pm$ 0.0	33.3 $\pm$ 0.0	34.5	99.5 $\pm$ 0.0	99.0 $\uparrow$ 1.2	32.1

Appendix

Table 16: Effect of adding acoustic cues to the prompt in the zero-shot single-recording setting (balanced accuracy, %). We evaluate one closed-source model family (gpt-realtime, top) and one open-weight model family (Qwen-Omni, bottom) due to resource limitation. Audio and Text+Audio cells report results with acoustic cues; arrows give the change relative to the same model and modality without cues. AF (acoustic features only, no audio input) has no cue-free counterpart and is reported as-is. Best result per column in bold .

Figure 9: Within-group association between each emotion (one-vs-rest) and each acoustic feature group on the LISTEN neutral-text set. Open circles show

|r|

of individual features (within-group point-biserial

r

); filled dots show the group mean. Only features with

q<.05

and

|r|\geq.10

for that emotion are shown; empty rows mean no feature in the group passed. Rows follow the same order in every panel (mean

|r|

across emotions). Surprise is omitted because too few groups contain within-group variation for inference. Feature groups are listed in Table 17 .

Group / eGeMAPSv02 feature	Group / eGeMAPSv02 feature
Pitch level (4)	Formants (14)
F0semitoneFrom27.5Hz_sma3nz_amean †	F1frequency_sma3nz_amean †
F0semitoneFrom27.5Hz_sma3nz_percentile20.0	F1frequency_sma3nz_stddevNorm
F0semitoneFrom27.5Hz_sma3nz_percentile50.0	F1bandwidth_sma3nz_amean
F0semitoneFrom27.5Hz_sma3nz_percentile80.0	F1bandwidth_sma3nz_stddevNorm
Pitch variability (5)	F1amplitudeLogRelF0_sma3nz_amean

Appendix

Table 17: Acoustic feature groups for the emotion analysis on the LISTEN neutral-text set, used in Figure 9 . The 67 features are those with

q<.05

and

|r|\geq.10

(one-vs-rest, within-group point-biserial

r

) for at least one emotion; numbers in parentheses give the number of features per group. † marks the feature with the largest mean

|r|

across emotions in each group.

Model	Base	+Features	$\Delta$	Base	+Features	$\Delta$
	Neutral Audio (Pairwise)			Neutral Audio+Text (Pairwise)
gpt-realtime-mini	18.38	18.41	$\uparrow$ 0.03	17.80	19.79	$\uparrow$ 1.99
gpt-realtime	20.75	23.90	$\uparrow$ 3.15	18.83	22.37	$\uparrow$ 3.54
Qwen2.5	28.18	27.07	$\downarrow$ 1.11	18.47	25.99	$\uparrow$ 7.53
Qwen3	28.79	35.94	$\uparrow$ 7.15	25.93	33.72	$\uparrow$ 7.79

Appendix

Table 18: Effect of adding raw acoustic cues on LISTEN dataset in pairwise settings (accuracy, %).

\Delta

denotes the change relative to baseline (

\uparrow

for improvements,

\downarrow

for drops). Best results and highest gains per condition are in bold .

Cue	Question
Pitch level	Which clip has the higher mean pitch?
Pitch variability	Which clip has more pitch variation?
Loudness level	Which clip is louder overall?
Loudness dynamics	Which clip has greater loudness variation?
Spectral tilt	Which clip sounds brighter or sharper?

Appendix

Table 19: Cue-specific questions substituted into the {question} slot of Prompt H.4 . The same questions were shown to human listeners.

Audio-Language Model Evaluation

Audio-language model evaluation tests whether systems understand speech, music, and environmental sound reliably, fairly, and with evidence grounded in audio.

See all 208 papers

Mar 19, 2026

How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation

Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs), yet how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance remains unclear. We study this gap by comparing different LLMs under two text-only and one audio-grounded setting: (1) direct probing on AKB-2000, a curated benchmark testing the breadth and depth of auditory knowledge; (2) cascade evaluation, where LLMs reason over text descriptions from an audio captioner; and (3) audio-grounded evaluation, where each LLM is fine-tuned into a Large Audio Language Model (LALM) with an audio encoder. Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance. Our work provides empirical grounding for a comprehensive understanding of LLMs in audio research.

Ke-Han Lu, Szu-Wei Fu, Chao-Han Huck Yang +13

National Taiwan University, Taiwan · NVIDIA · Academia Sinica, Taiwan

Audio Understanding Audio-Language Model Evaluation LLM Evaluation

Ask PDF

Mar 6, 2026

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

Multimodal large language models can exhibit text dominance, over-relying on linguistic priors instead of grounding predictions in non-text inputs. One example is large audio-language models (LALMs) where decisive audio evidence can be under-utilized even when it contains important information. To address this issue we use mechanistic interpretability to identify a small set of audio-specialist attention heads whose audio attention yields a ``listening'' signal. We show that this signal increases when audio evidence affects the model's output, providing an indicator of audio engagement under standard prompting. Leveraging this localization, we construct an audio--silence steering direction and apply an inference-time activation intervention to the final representation, amplifying the model's audio effect. To demonstrate the utility of this intervention, we show on MMAU that this improves accuracy by up to +8.0 percentage points on two Qwen-based LALMs, without any parameter updates.

Neta Glazer, Lenny Aharon, Ethan Fetaya

Bar-Ilan University, Ramat Gan, Israel · Columbia University, New York, NY, USA

Audio-Language Models Audio-Language Model Evaluation Causal Interventions in Language Models

Ask PDF

Oct 15, 2026

TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence

We introduce TRACE (Trustworthy Reasoning from Acoustic Cues and Evidence), a benchmark for diagnosing the reliability of large audio-language models (LALMs) by revealing when, and under what acoustic conditions these models behave untrustworthily. Existing benchmarks primarily measure whether a model produces the correct answer, but correctness alone cannot determine whether the answer is grounded in acoustic evidence or instead reflects language and contextual priors. TRACE evaluates model behavior along three complementary dimensions: (1) task performance, whether the answer is correct; (2) evidence alignment, whether the model is attending to the audio regions required to answer the question; and (3) robustness, whether answers and attributed evidence respond coherently to controlled changes such as event order, overlap, signal-to-noise ratio, or background content. TRACE targets temporal ordering, counting, and event existence, three tasks that remain challenging for current models. We synthesize contextually informed acoustic scenes that allow us to test whether models hallucinate events associated with the context but absent from the audio. This synthesis also allows us to obtain a scalable benchmark for which we know the answers by design. For each scene, we create controlled variants in a programmatic way to stress-test model robustness. We introduce attribution-based metrics that quantify model attention at scale on target events and non-target events. Evaluation across five state-of-the-art models shows that current LALMs often detect where relevant sound evidence occurs, but they do not reliably interpret it, integrate it into the answer, or respond to controlled changes in that evidence. Our benchmark also shows that models still struggle the most with ordering and counting questions.

Giovana Morais, Yujin Kim, Atilay Kucukoglu +1

Music and Audio Research Lab, NYU · Integrated Design and Media, NYU

Audio-Language Model Evaluation Audio-Language Models Audio-Language Model Hallucination Detection

Ask PDF

Abstract

Audio-Language Model Evaluation

Explore similar work

How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation

Are Audio-Language Models Listening? Audio-Specialist Heads for Adaptive Audio Steering

TRACE: Trustworthy Reasoning from Acoustic Cues and Evidence