Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs

Song-ha Jo; Sehyun Lee; Soyoon Kim; Jaesik Choi; Sanghyuk Choi

cs.SDSep 5, 2026

Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs

Authors: Song-ha Jo, Sehyun Lee, Soyoon Kim, Jaesik Choi, Sanghyuk Choi

Organizations: Seoul National University · KAIST · NAVER Cloud

Abstract

Audio-conditioned language models often underuse acoustic cues such as prosody, emotion, and non-speech sounds, raising the question of whether ASR-supervised frontends discard this information before it reaches the LM. We test whether the frontend is responsible by comparing Whisper-Tiny and Whisper-Small with EnCodec, DAC-VAE, and WavTokenizer in a shared Qwen3.5-4B audio-LM pipeline on ASR, emotion recognition, and sound captioning. Encoder replacement alone does not resolve this underuse: Whisper variants remain strongest overall, including on emotion and environmental sound captioning. To localize the failure, we trace task-relevant information through the encoder, projector, LM layers, and LM head. Linear probes and geometric analyses show that discriminative acoustic structure remains recoverable at the final LM layer, even when MCQA accuracy trails probe accuracy by up to 83 points. Because the answer format and decoding procedure are controlled, this task-dependent gap points to content-specific readout failure rather than generic format bias. LogitLens analyses and a targeted LM head intervention support the conclusion that acoustic underuse is not explained solely by encoder-side information loss and that readout alignment can be a dominant bottleneck.

Figures & tables

Figure 1: Overview of our analysis. An audio-conditioned LLM (ALM) processes the audio input through an encoder–projector–LM pipeline. To trace where the acoustic information goes, we investigate each stage, after the encoder, projector, LM internals, and LM head, asking whether discriminative acoustic information remains recoverable or has been lost.

Encoder	Arch.	Obj.	Tokens/s	Params
EnCodec	Conv	Recon.	$75$	$7.43$ M
DACVAE	Conv+VAE	Recon. + WM	$25$	$27.55$ M
Whisper-Small	Tr	ASR	$50$	$88.15$ M
Whisper-Tiny	Tr	ASR	$50$	$8.21$ M
WavTokenizer	Conv	Recon.	$40$	$8.30$ M

Table 1: Audio encoder candidates. Arch.: Conv (SEANet-style stack), Tr (Transformer), VAE (variational bottleneck). Obj.: Recon. (reconstruction-based, e.g., neural codec), Recon. + WM (with acoustic watermarking), ASR (supervised speech recognition). Params count the active forward path. Per-encoder sampling rate, hop, latent dimensionality, pretrained URIs, and training-time settings are listed in Table 5 .

Encoder	ASR	Emotion	Captioning
	WER $\downarrow$	macro-F1 $\uparrow$	CIDEr $\uparrow$
Whisper-Small	$\mathbf{7.05}$	$\mathbf{0.462}$	$\mathbf{0.847}$
Whisper-Tiny	$\underline{8.84}$	$\underline{0.444}$	$\underline{0.685}$
EnCodec	$20.80$	$0.393$	$0.259$
WavTokenizer	$33.61$	$0.356$	$0.325$
DAC-VAE	$20.46$	$0.400$	$0.235$

Table 2: Per-encoder summary across task families, averaged within each family. ASR WER averages LibriSpeech test-clean, test-other, and GigaSpeech test. Emotion macro-F1 averages MELD and IEMOCAP-S5. Captioning CIDEr averages FSD50K, Clotho, AudioCaps, and AudioSet. Per-benchmark detail is in Table 10 .

Task	Whisper-Tiny	Whisper-Small	Encodec	WavTokenizer	DAC-VAE
CREMA-D	57/31 ( $-25$ )	59/28 ( $-31$ )	30/30 ( $+0$ )	34/22 ( $-12$ )	29/22 ( $-7$ )
IEMOCAP	65/53 ( $-12$ )	69/58 ( $-11$ )	56/52 ( $-5$ )	52/44 ( $-8$ )	52/45 ( $-6$ )
SpeechCmds	97/83 ( $-14$ )	98/66 ( $-31$ )	49/41 ( $-7$ )	25/47 ( $+22$ )	35/44 ( $+9$ )
AudioMNIST	100/88 ( $-12$ )	100/53 ( $-47$ )	85/17 ( $-68$ )	64/14 ( $-50$ )	77/10 ( $-67$ )
VocalSound	83/41 ( $-42$ )	88/52 ( $-36$ )	55/27 ( $-29$ )	35/29 ( $-6$ )	44/26 ( $-18$ )
ESC-10	93/37 ( $-56$ )	95/50 ( $-45$ )	76/44 ( $-33$ )	62/20 ( $-42$ )	66/20 ( $-47$ )

Table 3: Probe / MCQA accuracy (%) per task across encoders, with the probe

\to

MCQA transfer gap in parentheses. Tasks span speech emotion (CREMA-D, IEMOCAP), speech content (SpeechCmds, AudioMNIST), vocal sounds (VocalSound), environmental sound (ESC-10, ESC-50 (5cat), UrbanSound8K, TAU, FSD50K, AudioSet), and music (GTZAN, NSynth, IRMAS, MedleySolos); per-task class count, balance, and chance level are listed in Table 9 .

\dagger

FSD50K (200 cls,

n{=}10{,}231

) and AudioSet (632 cls,

n{=}3{,}429

aligned) use 4-way MCQA with random distractors (chance

25\%

), since per-task

K

-way is impractical at that class count.

Figure 2: Layer-wise probe accuracy across encoder (

E0

–

E4

), projector (

P0

–

P3

P_{o}

), and LM (

L0

L8

L15

L23

L31

) stages. Dashed horizontal lines indicate final LM MCQA accuracy for the same encoder color.

	ASR (WER $\downarrow$ )			Emotion (macro-F1 $\uparrow$ )			Captioning (CIDEr $\uparrow$ )
Encoder	align	FT	$\Delta$	align	FT	$\Delta$	align	FT	$\Delta$
Whisper-Tiny	$8.84$	$6.06$	$-2.78$	$0.444$	$0.431$	$-0.013$	$0.685$	$0.476$	$-0.209$
EnCodec	$20.80$	$10.65$	$\mathbf{-10.15}$	$0.393$	$0.402$	$+0.009$	$0.259$	$0.408$	$\mathbf{+0.149}$
DAC-VAE	$20.46$	$16.60$	$-3.86$	$0.400$	$0.382$	$-0.018$	$0.235$	$0.344$	$+0.109$

Table 4: Family-level effect of LoRA fine-tuning (FT) over align, averaged within each task family using the same metric pool as Table 2 .

\Delta

is FT

-

align; for ASR, negative

\Delta

is improvement. Per-task numbers and the best-by-rank-sum FT checkpoint per encoder are in Appendix Table 11 .

Figure 3: Per-stage distance ratios across the five encoders on IEMOCAP Session 5. (a) Across-/within-speaker distance ratio (same transcript, different speaker

/

same speaker, different transcript). (b) Across-speaker

/

random-pair distance ratio (same transcript, different speaker

/

different speaker, different transcript).

Figure 4: Layer-wise LogitLens heatmap of teacher-forced top-1 next-token match across the 32 LM blocks plus the input embedding ( $x=0$ ), for three encoders (Whisper-Tiny / EnCodec / DAC-VAE) in two conditions (align, fine-tuning).

Figure 5: Rank-delta heatmap of the audio effect: the increase in rank-1 letter accuracy (%) when audio is present versus a text-only baseline. Rows are six tasks; columns are five encoders.

Figure 6:

W_{U}

letter-row surgical recovery, averaged over the six analysis tasks. Bars compare MCQA accuracy before and after fine-tuning only the

n_{c}

rows of

W_{U}

indexing the choice letters (

n_{c}\times 2560\approx 10

–

26

k parameters per task), across five encoders; the

+\Delta

label above each After bar is the per-encoder absolute gain (mean across tasks).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Encoder	HF repository	SR (kHz)	Hop	Dim.	cutoff (tok)	Global batch	Rows / phase
Whisper-Small	openai/whisper-small.en	$16$	$320$	$768$	$3584$	$32$	${\sim}24.4$ M
Whisper-Tiny	openai/whisper-tiny.en	$16$	$320$	$384$	$3584$	$32$	${\sim}24.4$ M
EnCodec	facebook/encodec_24khz	$24$	$320$	$128$	$4096$	$32$	${\sim}27.9$ M
WavTokenizer	novateur/WavTokenizer-large-unify-40token	$24$	$600$	$512$	$3584$	$32$	${\sim}24.4$ M
DAC-VAE	facebook/dacvae-watermarked	$48$	$1920$	$128$	$3584$	$16$	${\sim}12.2$ M

Appendix

Table 5: Per-encoder spec, pretrained checkpoint, and training-time settings. Dim. is the encoder latent dimensionality (projector input). Global batch

=

8

(GPUs)

\times

per-device batch

\times

gradient accumulation. Rows / phase is computed from

100{,}000

steps, global batch, cutoff length, and the row-size weighted mean of

\approx 470

tokens per training row. EnCodec uses a larger cutoff (

4096

tokens) because its

75

tokens/s rate fills

2{,}250

audio frames for a

30

-second utterance and exceeds

3584

once the ChatML wrapper is added; DAC-VAE uses a halved global batch (

16

) because its

48

kHz input carries roughly

3\times

as many raw audio samples per utterance as the

16

24

kHz encoders, forcing a smaller per-step batch under fixed GPU memory.

Modality	Hours	Rows	Prob.	Epochs @ $100$ k
ASR (English)	$50.8$ k	$15.47$ M	$0.65$	$0.51\text{--}1.17$
Environmental sound	$4.31$ k	$691.8$ k	$0.25$	$4.41\text{--}10.1$
Emotion	$51$	$50.28$ k	$0.10$	$24.3\text{--}55.5$
Total	$55.1$ k	$16.22$ M	$1.00$	—

Appendix

Table 6: Multi-task training mixture. Hours are total source duration (raw, before the

30

-second per-utterance cap). Probabilities are row-level sampling weights for interleave_datasets . Epochs at

100

k is the per-modality exposure range across encoders (low end: DAC-VAE global batch

16

; high end: EnCodec global batch

32

with cutoff

4096

), computed as rows-consumed

\times

prob

/

rows.

Global
Optimizer	AdamW (FusedAdam)
Peak LR	$2\times 10^{-4}$ (align) / $2\times 10^{-5}$ (FT)
Weight decay	$0.01$
Gradient clip	$1.0$
LR schedule	warm-up, stable, decay
Warmup	$1000$ steps

Appendix

Table 7: Training hyperparameters. Only the peak learning rate differs between the audio-LM alignment phase (align) and the LoRA fine-tuning phase (FT). Per-encoder batch composition (per-device batch, gradient accumulation, effective global batch) is listed in Table 5 .

Speech emotion
Dataset	Balance	Classes	Notes
IEMOCAP-S5	unbal 197–613	angry / happy / neutral / sad
CREMA-D	unbal 40–51	angry / disgust / fearful / happy / neutral / sad
Speech content
Speech Commands	balanced (200/cls)	yes / no / up / down / left / right / on / off / stop / go	keyword spotting
AudioMNIST	balanced (200/cls)	digit_0 …digit_9	spoken digits

Appendix

Table 8: Dataset summary across speech, vocal, environmental sound, and music domains.

\dagger

FSD50K and AudioSet use 4-way MCQA with random distractors per sample because

K

-way over the full class set is impractical.

Task	N	Cls	Bal	Spch	Dur (s)	Chance
Speech emotion
CREMA-D	$264$	$6$	no	yes	$2.5$	$16.7$ %
IEMOCAP	$2{,}000$	$4$	yes	yes	$4.5$	$25.0$ %
Speech content
SpeechCmds	$2{,}000$	$10$	yes	yes	$1.0$	$10.0$ %
AudioMNIST	$2{,}000$	$10$	yes	yes	$0.7$	$10.0$ %

Appendix

Table 9: Per-task metadata for the linear-probe vs MCQA evaluation in Table 3 . N: number of test samples. Cls: number of classes. Bal: class-balanced (yes/no). Spch: speech content (yes/no). Dur: median utterance duration in seconds. Chance: uniform-prior chance accuracy.

\dagger

FSD50K and AudioSet use 4-way MCQA with random distractors per sample (chance

25\%

) because

K

-way over the full class set is impractical.

ASR
Benchmark	Metric	Whisper-Small	Whisper-Tiny	EnCodec	WavTokenizer	DAC-VAE
Encoder size	params	88.2 M	8.2 M	7.4 M	8.3 M	27.6 M
Representative ckpt	step	94,000	100,000	90,000	100,000	80,000
LibriSpeech test-clean	WER $\downarrow$	2.49	2.96	6.99	13.08	6.50
	CER $\downarrow$	0.97	1.19	3.37	7.00	2.98
LibriSpeech test-other	WER $\downarrow$	6.19	8.58	22.62	38.41	23.39

Appendix

Table 10: Per-benchmark encoder comparison. ASR reports WER / CER (%, lower is better). All other metrics are higher-is-better. Bold marks the per-row best, underline marks the second-best across encoders (computed only over evaluated cells). – = not evaluated. FSD50K and AudioSet are evaluated as captioning tasks (prompt “Describe what you hear in the audio.” , BLEU / CIDEr scored against AudioSet-ontology descriptions of the ground-truth labels as multi-reference) rather than vocab-parse F1, because caption-trained Stage-1 models emit free-form descriptions and a vocab parser yields artifact

\approx 0

that does not reflect actual capability. The same prompt and metric pipeline are used for AudioCaps and Clotho, so the four caption rows are directly comparable. MELD uses the official held-out test split. IEMOCAP Session 5 is a leakage-free cross-corpus emotion benchmark (4-class, exc

\to

hap merge,

n=1241

ASR (WER $\downarrow$ ; negative $\Delta$ is improvement)
		Whisper-Tiny			EnCodec			DAC-VAE
		(FT ckpt $30$ k)			(FT ckpt $70$ k)			(FT ckpt $40$ k)
Task	Metric	align	FT	$\Delta$	align	FT	$\Delta$	align	FT	$\Delta$
LibriSpeech test-clean	WER	$2.96$	$2.62$	$-0.34$	$6.99$	$3.72$	$-3.27$	$6.50$	$5.29$	$-1.21$
LibriSpeech test-other	WER	$8.58$	$7.21$	$-1.37$	$22.62$	$13.28$	$-9.34$	$23.39$	$20.05$	$-3.34$
GigaSpeech test	WER	$14.98$	$8.34$	$-6.64$	$32.80$	$14.95$	$-17.85$	$31.48$	$24.48$	$-7.00$

Appendix

Table 11: Per-task align vs LoRA-fine-tuning (FT) comparison for the three encoders evaluated under FT. align values are reproduced from Table 10 . FT values are evaluated at the per-encoder best-by-rank-sum checkpoint over the 9-metric pool: Whisper-Tiny ckpt

30{,}000

, EnCodec ckpt

70{,}000

, DAC-VAE ckpt

40{,}000

. Family averages of these per-task numbers appear in Table 4 .

Task	Probe	MCQA	Gap	Probe	MCQA	Gap	$\Delta$ Gap
	align			FT
IEMOCAP	$56.5$	$79.5$	$-23.0$	$56.0$	$77.0$	$-21.0$	$+2.0$
AudioMNIST	$99.5$	$84.0$	$+15.5$	$100.0$	$91.5$	$+8.5$	$\mathbf{-7.0}$
VocalSound	$80.3$	$43.9$	$+36.3$	$77.2$	$48.5$	$+28.7$	$\mathbf{-7.6}$
UrbanSound8K	$74.5$	$26.0$	$+48.5$	$76.0$	$32.0$	$+44.0$	$\mathbf{-4.5}$
GTZAN	$72.5$	$8.5$	$+64.0$	$71.0$	$20.5$	$+50.5$	$\mathbf{-13.5}$

Appendix

Table 12: Probe at

L31

vs MCQA accuracy (%) for Whisper-Tiny under align and fine-tuning, with the probe-to-MCQA gap (Probe

-

MCQA) and the change in gap (

\Delta

Gap

=

Gap

{}_{\text{FT}}-

Gap

{}_{\text{align}}

). Negative

\Delta

Gap (bold) marks gap reduction.

Figure 7: LogitLens rank-1 letter accuracy across all six tasks, decoded at five LM depths (

L0,L8,L15,L23,L31

of the 32-layer Qwen3.5-4B) via final RMSNorm

+\,W_{U}

and restricted to the candidate-letter tokens. Each line is one encoder family; the grey band marks the range of

E4

encoder-probe accuracies across the five families for that task. Speech tasks reach a high

L31

accuracy that tracks audio-on MCQA, while music tasks and UrbanSound8K stay near chance at every depth despite high probe accuracy, isolating the readout as the bottleneck.

Figure 8: Per-task

W_{U}

letter-row fine-tune recovery, the task

\times

encoder breakdown of Figure 6 . Hatched bar: MCQA accuracy before fine-tune (equivalent to the rank-1 letter accuracy from final-layer LogitLens). Filled bar: MCQA accuracy after fine-tuning only the

n_{c}

rows of

W_{U}

that index the choice letters, with everything else frozen; the

\pm

label above each After bar is the absolute change in pp. IEMOCAP shows near-zero recovery because the readout is already aligned; the other five tasks show large recoveries up to

+78

pp, with the gain pattern across encoders mirroring the LogitLens curves in Figure 7 .

Figure 9: MCQA accuracy across three readout interventions, averaged over five encoders. B0 : vanilla LM, full-vocab argmax. B1 : + Logit processor restricting argmax to

\{A,\dots,K\}

. B2 : +

W_{U}

letter-row fine-tune (Appendix D.2 ). Error bars:

\pm

1 SD across encoders. Dotted segment over each group: per-task chance. Across

30

cells, B1 changes B0 accuracy by only

+0.23

pp on average, while B2 recovers

+25

–

+54

pp on the five readout-bottleneck tasks; IEMOCAP shows a small negative B2 effect (mild overfitting on the

\sim

162

training samples; its readout is already approximately aligned).

ASR
Benchmark	Metric	Whisper-Small	DAC-VAE
Encoder size	params	88.2 M	27.7 M
Representative ckpt	step	90,000	90,000
LibriSpeech test-clean	WER $\downarrow$	4.83	8.63
	CER $\downarrow$	2.67	5.47
LibriSpeech test-other	WER $\downarrow$	8.21	19.77

Appendix

Table 13: Downstream benchmark performance of the two encoders paired with Ministral3-3B (Instruct), the second base LM. Best value per row in bold. Values are reported on the scale used in Table 10 : WER and CER as percentages, all other metrics in

[0,1]

Task	Whisper-Small			DAC-VAE
Task	MCQA	$W_{U}$ FT	Probe	MCQA	$W_{U}$ FT	Probe
AudioMNIST	$76.8$	$99.9$	$100.0$	$10.7$	$98.5$	$98.8$
NSynth	$8.3$	$62.4$	$63.8$	$11.6$	$63.1$	$63.1$
GTZAN	$10.0$	$77.9$	$78.6$	$10.0$	$75.6$	$76.9$
UrbanSound8K	$25.2$	$82.8$	$83.0$	$12.0$	$73.3$	$76.5$
IEMOCAP	$58.8$	$65.8$	$65.6$	$50.9$	$60.3$	$60.3$

Appendix

Table 14: Readout bottleneck under a second base LM. MCQA accuracy,

W_{U}

letter-row fine-tuning, and final-layer linear probe accuracy for two encoders paired with Ministral3-3B (Instruct).

Task	$n_{c}$	Train	Split protocol
AudioMNIST	10	2,000	native train/test
NSynth	10	2,000	native train/test
GTZAN	10	700	seed-fixed disjoint
UrbanSound8K	10	2,000	official folds
IEMOCAP	4	5,370	session held-out
VocalSound	6	1,200	native train/test

Appendix

Table 15: Data protocol for the readout-robustness ablations (Tables 16 and 17 ). Each task uses its full standard training split, distinct from the

200

-per-class evaluation subsample of Table 8 , and the reported accuracy is measured on that split’s evaluation partition, which is disjoint from the states used to fit the letter rows. Original MCQA and post-intervention accuracy are paired measurements on the same evaluation examples, computed from the same saved

L31

hidden states. Only the

n_{c}

rows of

W_{U}

indexing the choice letters receive gradient, giving

n_{c}\times d_{\text{model}}

updated scalars per task, that is

10

–

26

k for Qwen3.5-4B (

d_{\text{model}}{=}2560

) and up to

31

k for Ministral3-3B (

d_{\text{model}}{=}3072

); every other parameter, including the embedding matrix and the remaining vocabulary rows, stays frozen at its pretrained value.

Condition	AudioMNIST	NSynth	GTZAN	UrbanSound8K	IEMOCAP	VocalSound	Average
Baseline	$74.5$ / $99.6$	$14.7$ / $56.5$	$13.7$ / $76.6$	$35.2$ / $81.0$	$57.9$ / $63.1$	$54.0$ / $87.3$	$41.7$ / $77.4$
Letter-remap	$76.8$ / $99.2$	$11.8$ / $55.8$	$9.0$ / $74.6$	$16.7$ / $81.3$	$53.4$ / $61.8$	$35.0$ / $86.8$	$33.8$ / $76.6$
Order-shuffle	$71.1$ / $99.8$	$13.0$ / $55.6$	$13.0$ / $73.6$	$34.8$ / $80.3$	$58.9$ / $62.3$	$49.8$ / $86.5$	$40.1$ / $76.4$
Instr-C1	$63.8$ / $99.7$	$16.4$ / $57.6$	$14.4$ / $73.6$	$34.2$ / $82.5$	$56.9$ / $62.2$	$54.0$ / $87.5$	$40.0$ / $77.2$
Instr-C2	$64.3$ / $99.7$	$17.6$ / $55.3$	$14.7$ / $74.6$	$36.3$ / $81.5$	$56.1$ / $62.3$	$52.2$ / $87.3$	$40.2$ / $76.8$
Label-word	$99.0$ / $100.0$	$12.1$ / $58.9$	$17.1$ / $75.6$	$38.0$ / $82.8$	$51.2$ / $58.3$	$64.5$ / $88.8$	$47.0$ / $77.4$

Appendix

Table 16: Robustness of the

W_{U}

letter-row intervention across MCQA answer formats (Whisper-Small). Each cell reports original MCQA accuracy / accuracy after the letter-row fine-tune (%). Letter-remap changes the choice-letter mapping, Order-shuffle permutes the option order, Instr-C1 uses the instruction “Listen to the audio and answer.” and Instr-C2 uses “You are given an audio clip. Choose the option that best describes the sound you hear. Respond with only the option identifier.”, and Label-word replaces each choice letter with its candidate label word ( e.g., “A. piano” becomes “piano”), scored by the first-token logit among the candidate label words and verified collision-free across the six task vocabularies.

Method	Intervention	IEMOCAP	AudioMNIST	VocalSound	UrbanSound8K	NSynth	GTZAN
Original MCQA	–	$57.9$	$74.5$	$54.0$	$35.2$	$14.7$	$13.7$
Bias-only calibration	Bias update only	$60.7$	$79.0$	$54.0$	$40.2$	$16.7$	$14.0$
Letter-row fine-tune	Direction learning (letter rows)	$64.7$	$99.7$	$87.3$	$82.2$	$59.7$	$74.9$
Full-row update	Direction learning (full vocabulary)	$64.0$	$99.7$	$87.3$	$82.2$	$59.0$	$74.9$
Final-layer linear probe	Supervised classifier	$64.3$	$99.8$	$87.7$	$82.7$	$59.6$	$79.6$

Appendix

Table 17: Readout condition ablation (Whisper-Small, MCQA accuracy in %, same train/test split throughout). The probe row is an upper reference rather than an intervention. It fits an unconstrained classifier and does not operate through

W_{U}

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 303 papers

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models
Sep 1, 2026Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh +1Audio-Language Model Evaluation Audio-Language Models
Language Technologies Institute Carnegie Mellon University
Tracing Audio Grounding and Answer Selection in Audio LLMs
Sep 4, 2026Hyebin Cho, Suho Yoo, Jihoo Jung +1Audio-Language Models Audio QA
Korea Advanced Institute of Science and Technology, South Korea
Direct or Mediated? Task-Dependent Audio Information Routing in Large Audio Language Models
Aug 27, 2026Yizhou Zhang, Wangjin Zhou, Xin Gu +6Audio-Language Models Audio QA
Graduate School of Informatics, Kyoto University, Japan · WXG, Tencent, China

Sep 1, 2026J/K move · Enter open · S save

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh, Bhiksha Raj

Language Technologies Institute Carnegie Mellon University

Audio-Language Model Evaluation Audio-Language Models Mechanistic Interpretability Speech Prosody

Open paper AskPDF arXiv

Audio language models are designed to understand speech, yet it remains unclear whether they capture how something is said beyond what is said. We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5-Omni-7B, and Chroma-4B, using the Expresso dataset with controlled speaking styles. We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output. All models strongly encode speaking style in the late encoder, that is, the top third of the audio encoder's layers, but this information is consistently degraded before reaching the output. The projector reshapes representation geometry without removing information, while decoders differ in how much style they preserve depending on architecture and training objective. At the output level, models fall into two behaviors. Some are content driven, where predictions depend mainly on text. Others are acoustic driven, where predictions vary with speaking style. The leakage metric quantifies this difference, and qualitative results confirm it. Overall, we identify a gap between what models encode and what they use, highlighting a key limitation in current audio language models.

Abstract

Audio-Language Models

Explore similar work

Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

Tracing Audio Grounding and Answer Selection in Audio LLMs

Direct or Mediated? Task-Dependent Audio Information Routing in Large Audio Language Models