Beyond the Transcript: Measuring and Improving the Use of Acoustic Information in Audio Language Models
Organizations: University of Connecticut · The University of Texas at Austin · University of Oxford
Abstract
Audio language model benchmarks provide comprehensive evaluation across diverse tasks. However, some tasks can be answered from the transcript or even the question alone, and benchmark scores can therefore confound acoustic understanding with language modeling ability. We distinguish lexical tasks, for which the transcript usually suffices, from global-attribute and within-recording-structure tasks, which require the model to analyse the recording beyond its words, either as a whole, for properties such as the speaker and emotion, or within it, for patterns such as stress and intonation. We evaluated 19 open models and Gemini 3.1 Pro on 92 tasks from 6 benchmarks and estimated the recording gain, the accuracy advantage of the original recording over references that retain its words, by comparing the recording with synthetic readings of its transcript and by adding it to the transcript. On attribute and structure tasks, the open models fall behind Gemini by 28.6 and 41.9 percentage points, and about half or more of each deficit lies in the recording gain. Nevertheless, linear probes in Gemma-4 12B and Qwen2.5-Omni-7B show that the acoustic information needed for answering is already readable in early language-model layers, while the readout of many cues declines in the layers that follow. We therefore constructed training data and trained LoRA adapters in these layers, with a content anchor, which raises attribute and structure accuracy by 12.8 to 44.4 points in the two models and keeps mean lexical accuracy within 1 point of the base model. Most of the improvement appears as larger recording gains. These results suggest that training improves the use of acoustic information that the models already encode.
Figures & tables
| Form | ||||
|---|---|---|---|---|
| Input | Acc. | Text | Speech | |
| Question and options only | ||||
| Whisper transcript | ||||
| Synthetic reading of it | ||||
| Original recording | ||||
| Transcript and recording | ||||
| Lexical | Global attribute | Within-recording structure | |||||
| Gemma-4 12B | |||||||
| Base | 78.3 | 34.7 | 35.4 | ||||
| Repaired | 79.1 | 53.6 | 66.2 | ||||
| Change | +0.8 | +18.9 | +30.8 | ||||
| Qwen2.5-Omni-7B | |||||||
| Lexical | Attribute | Structure | |||
| Gemma-4 12B | |||||
| Layers 10–25 | |||||
| No content anchor | |||||
| Layers 0–15 | |||||
| Layers 32–47 | |||||
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Part used | Answer format | Tasks | Items |
|---|---|---|---|---|
| AudioBench | Voice understanding (6 sets), spoken QA (4 sets) | Label, multiple choice, short answer | 10 | 4,779 |
| AIR-Bench | Foundation, speech tasks | Multiple choice | 8 | 1,161 |
| Dynamic-SUPERB Phase-2 | 11 speech tasks | Label | 11 | 2,035 |
| MMAU | test-mini, speech | Multiple choice | 10 | 332 |
| MMAU-Pro | Speech, one recording per item | Multiple choice | 6 | 865 |
| MMSU | All tasks | Multiple choice | 47 | 4,535 |
| Benchmark | Tasks |
|---|---|
| Lexical (36 tasks) | |
| AIR-Bench (2) | Speaker_Intent_Classification, Speech_Entity_Reconition |
| AudioBench (4) | cn_college_listen_mcq_test, dream_tts_mcq_test, public_sg_speech_qa_test, slue_p2_sqa5_test |
| Dynamic-SUPERB (2) | DialogueActClassification_DailyTalk, SuperbIC_SLURP-Intent |
| MMAU (7) | Conversational Fact Retrieval, Dissonant Emotion Interpretation, Event-Based Knowledge Retrieval, Key highlight Extraction, Multi Speaker Role Mapping, Phonemic Stress Pattern Analysis, Phonological Sequence Decoding |
| MMSU (21) | casual_reasoning, code_switch_question_answering, consonant_vowel_perception, content_grounding, continuation_writing, couplet_matching, deixis_resolution, dialogue_reasoning, homophone_based_reasoning, idiom_reasoning, intent_detection, logical_reasoning, long_speech_summarization, near_homophone_perception, polysemy_reasoning, prolonged_sound_reasoning, puns_interpretation, speech_act_classification, speech_translation, syllable_perception, syntactic_structure_matching |
| Model | Reference | Checkpoint |
|---|---|---|
| Qwen3-Omni | Xu et al.,2025b | Qwen/Qwen3-Omni-30B-A3B-Instruct |
| Qwen2.5-Omni-7B | Xu et al.,2025a | Qwen/Qwen2.5-Omni-7B |
| MiniCPM-o 4.5 | Cui et al.,2026 | openbmb/MiniCPM-o-4_5 |
| Qwen2.5-Omni-3B | Xu et al.,2025a | Qwen/Qwen2.5-Omni-3B |
| MiniCPM-o 2.6 | Yao et al.,2024 | openbmb/MiniCPM-o-2_6 |
| Kimi-Audio | KimiTeam et al.,2025 | moonshotai/Kimi-Audio-7B-Instruct |
| Gemma-4 12B | Qwen2.5-Omni-7B | ||||
|---|---|---|---|---|---|
| Benchmark | Task | Base | Repaired | Base | Repaired |
| Lexical (36 tasks) | |||||
| AIR-Bench | Speaker_Intent_Classification | 73.9 | 70.4 | 90.8 | 85.2 |
| AIR-Bench | Speech_Entity_Reconition | 67.1 | 67.8 | 77.6 | 73.4 |
| AudioBench | cn_college_listen_mcq_test | 89.6 | 91.4 | 92.6 | 92.6 |
| AudioBench | dream_tts_mcq_test | 77.0 | 78.2 | 89.9 | 89.1 |
Audio-Language Models
Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.
See all 324 papers