Audio language model benchmarks provide comprehensive evaluation across diverse tasks. However, some tasks can be answered from the transcript or even the question alone, and benchmark scores can therefore confound acoustic understanding with language modeling ability. We distinguish lexical tasks, for which the transcript usually suffices, from global-attribute and within-recording-structure tasks, which require the model to analyse the recording beyond its words, either as a whole, for properties such as the speaker and emotion, or within it, for patterns such as stress and intonation. We evaluated 19 open models and Gemini 3.1 Pro on 92 tasks from 6 benchmarks and estimated the recording gain, the accuracy advantage of the original recording over references that retain its words, by comparing the recording with synthetic readings of its transcript and by adding it to the transcript. On attribute and structure tasks, the open models fall behind Gemini by 28.6 and 41.9 percentage points, and about half or more of each deficit lies in the recording gain. Nevertheless, linear probes in Gemma-4 12B and Qwen2.5-Omni-7B show that the acoustic information needed for answering is already readable in early language-model layers, while the readout of many cues declines in the layers that follow. We therefore constructed training data and trained LoRA adapters in these layers, with a content anchor, which raises attribute and structure accuracy by 12.8 to 44.4 points in the two models and keeps mean lexical accuracy within 1 point of the base model. Most of the improvement appears as larger recording gains. These results suggest that training improves the use of acoustic information that the models already encode.
Figures & tables
Figure 1: Overview of the study, from the data and models (top) to the recording gain (a), the readout (b) and the repair (c). The tasks are grouped by where the evidence for the answer lies. Each class is shown with one example item, and the 14 mixed tasks are not shown. (a) One item asked with five inputs, and the class means of S and H in percentage points (pp) for the mean of the 19 open models and for Gemini 3.1 Pro. (b) Readout of three acoustic cues in the early, middle and late layers of Gemma-4 12B, and answer accuracy on the same items (chance 50%, 50% and 25%). (c) LoRA training of the layers where readout declines, with cross-entropy on acoustic questions and a KL term, and accuracy by class before and after training, with Gemini shown for reference.
Form
Input
Acc.
Text
Speech
Question and options only
N
Whisper transcript
A
✓
Synthetic reading of it
T
✓
Original recording
O
✓
Transcript and recording
AO
✓
✓
Table 1: The five input conditions.
Figure 2: Recording gains of the open models and Gemini 3.1 Pro on the three task classes. Each row summarizes a group of tasks within a class. On the left, the 19 columns represent the open models, ordered from lowest to highest overall accuracy with the original recording, and each cell shows that model’s mean of S and H . The right side reports accuracy with the original recording, O (%), and the two gains (pp), S and H , for the mean of the 19 open models and for Gemini 3.1 Pro. Purple marks positive gains and brown negative gains. Mixed tasks are not shown.
Figure 3: Layer-wise linear-probe readout and answer accuracy in Gemma-4 12B and Qwen2.5-Omni-7B. The dots show answer accuracy with the original recording. The green band marks the readout-decline layers of Gemma-4 12B (layers 10–25). Blue panels show five attribute tasks and orange panels four structure tasks.
Lexical
Global attribute
Within-recording structure
O
O
S
H
O
S
H
Gemma-4 12B
Base
78.3
34.7
−0.4
+5.2
35.4
+2.5
+5.2
Repaired
79.1
53.6
+12.6
+15.7
66.2
+29.6
+29.5
Change
+0.8
+18.9
+13.0
+10.5
+30.8
+27.0
+24.3
Qwen2.5-Omni-7B
Table 2: Accuracy and the two recording gains by task class for the base and repaired Gemma-4 12B and Qwen2.5-Omni-7B, with the mean of the 19 open models and Gemini 3.1 Pro. O is accuracy (%) with the original recording, and S , H and changes are in pp.
Figure 4: Accuracy gain by adapter position. Arrow width shows the share of the full adapter’s gain recovered when it is active only at audio or text positions.
Lexical
Attribute
Structure
ΔO
ΔO
ΔS
ΔO
ΔS
Gemma-4 12B
Layers 10–25
+0.8
+18.9
+13.0
+30.8
+27.0
No content anchor
−10.5
+16.5
+15.1
+35.2
+29.4
Layers 0–15
+1.1
+20.3
+13.7
+16.3
+11.2
Layers 32–47
+0.2
+8.2
+3.9
+11.4
+7.6
Table 3: Training variants of Gemma-4 12B and Qwen2.5-Omni-7B. ΔO and ΔS give changes from the base model in original-recording accuracy and S by task class (pp), and tinted rows mark the repaired models. We report only ΔS here, since S and H change in the same direction for both repaired models.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Part used
Answer format
Tasks
Items
AudioBench
Voice understanding (6 sets), spoken QA (4 sets)
Label, multiple choice, short answer
10
4,779
AIR-Bench
Foundation, speech tasks
Multiple choice
8
1,161
Dynamic-SUPERB Phase-2
11 speech tasks
Label
11
2,035
MMAU
test-mini, speech
Multiple choice
10
332
MMAU-Pro
Speech, one recording per item
Multiple choice
6
865
MMSU
All tasks
Multiple choice
47
4,535
Appendix
Table 4: Benchmark subsets used in the study. The table lists their answer formats, task counts and item counts (13,707 items in total).
Table 5: Tasks by evidence class and benchmark. Names follow the source benchmarks, and parentheses give task counts.
Model
Reference
Checkpoint
Qwen3-Omni
Xu et al.,2025b
Qwen/Qwen3-Omni-30B-A3B-Instruct
Qwen2.5-Omni-7B
Xu et al.,2025a
Qwen/Qwen2.5-Omni-7B
MiniCPM-o 4.5
Cui et al.,2026
openbmb/MiniCPM-o-4_5
Qwen2.5-Omni-3B
Xu et al.,2025a
Qwen/Qwen2.5-Omni-3B
MiniCPM-o 2.6
Yao et al.,2024
openbmb/MiniCPM-o-2_6
Kimi-Audio
KimiTeam et al.,2025
moonshotai/Kimi-Audio-7B-Instruct
Appendix
Table 6: Open end-to-end audio models and evaluated checkpoints.
Figure 5: Probe accuracy across relative layer depth on all tasks. Blue panels show attributes and orange panels constructed structure items, with solid lines and filled answer dots for Gemma-4 12B, dashed lines and open dots for Qwen2.5-Omni-7B, a grey dashed chance line, and a green band over Gemma-4 12B layers 10–25 (LT, DS, AB, AIR denote LibriTTS, Dynamic-SUPERB, AudioBench, AIR-Bench).
Figure 6: Layer-wise probe and answer accuracy before and after training. Upper panels show Gemma-4 12B and lower panels Qwen2.5-Omni-7B, with grey and green lines for probe accuracy before and after training and other markers as in Figure 3 . The accent panel shows AccentDB, and the 11 probed tasks averaged in Section 5 also include the VoxCeleb and MMSU accent tasks, which are not drawn.
Figure 7: Accuracy with the original recording by task group. Rows cover 19 open models, Gemini 3.1 Pro, and two repaired models, with purple cells for accuracy (%), green and brown cells for repaired-minus-base changes (pp), and lexical tasks grouped by benchmark.
Gemma-4 12B
Qwen2.5-Omni-7B
Benchmark
Task
Base
Repaired
Base
Repaired
Lexical (36 tasks)
AIR-Bench
Speaker_Intent_Classification
73.9
70.4
90.8
85.2
AIR-Bench
Speech_Entity_Reconition
67.1
67.8
77.6
73.4
AudioBench
cn_college_listen_mcq_test
89.6
91.4
92.6
92.6
AudioBench
dream_tts_mcq_test
77.0
78.2
89.9
89.1
Appendix
Table 7: Original-recording accuracy (%) before and after training on 92 tasks. Tasks follow the classes in Table 5 , with repaired cells colored by change from the base (darker green for larger gains, brown for declines, light grey within 2 pp) and shaded rows marking attribute and structure tasks without matching training formats.