cs.CLNov 8, 2026

Beyond the Transcript: Measuring and Improving the Use of Acoustic Information in Audio Language Models

Authors: Linkai Peng, Baorian Nuchged, Yuyang Yao

Organizations: University of Connecticut · The University of Texas at Austin · University of Oxford

Abstract

Audio language model benchmarks provide comprehensive evaluation across diverse tasks. However, some tasks can be answered from the transcript or even the question alone, and benchmark scores can therefore confound acoustic understanding with language modeling ability. We distinguish lexical tasks, for which the transcript usually suffices, from global-attribute and within-recording-structure tasks, which require the model to analyse the recording beyond its words, either as a whole, for properties such as the speaker and emotion, or within it, for patterns such as stress and intonation. We evaluated 19 open models and Gemini 3.1 Pro on 92 tasks from 6 benchmarks and estimated the recording gain, the accuracy advantage of the original recording over references that retain its words, by comparing the recording with synthetic readings of its transcript and by adding it to the transcript. On attribute and structure tasks, the open models fall behind Gemini by 28.6 and 41.9 percentage points, and about half or more of each deficit lies in the recording gain. Nevertheless, linear probes in Gemma-4 12B and Qwen2.5-Omni-7B show that the acoustic information needed for answering is already readable in early language-model layers, while the readout of many cues declines in the layers that follow. We therefore constructed training data and trained LoRA adapters in these layers, with a content anchor, which raises attribute and structure accuracy by 12.8 to 44.4 points in the two models and keeps mean lexical accuracy within 1 point of the base model. Most of the improvement appears as larger recording gains. These results suggest that training improves the use of acoustic information that the models already encode.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs

    Sep 5, 2026Song-ha Jo, Sehyun Lee, Soyoon Kim +2Audio-Language ModelsAudio-Language Model Evaluation

  2. Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

    Sep 1, 2026Bhuvan Koduru, Dareen Safar B Alharthi, Rita Singh +1Audio-Language Model EvaluationAudio-Language Models

  3. Tracing Audio Grounding and Answer Selection in Audio LLMs

    Sep 4, 2026Hyebin Cho, Suho Yoo, Jihoo Jung +1Audio-Language ModelsAudio QA