Large Audio Language Models (LALMs) have demonstrated strong performance across a wide range of audio understanding tasks. However, they are typically evaluated on single, coherent audio segments, leaving their behavior under less familiar input configurations underexplored. We study this issue through a controlled setting in which two audio segments are concatenated into a single input. Across multiple LALMs, we observe a striking task-dependent robustness gap: automatic speech recognition (ASR) remains comparatively stable, whereas audio question answering (AQA) degrades substantially. To investigate the mechanisms underlying this disparity, we analyze how audio information is routed through LALM decoders using layer-wise attention knockout. The results reveal distinct task-dependent pathways. ASR relies primarily on direct retrieval from audio tokens by answer tokens, whereas AQA depends more strongly on a mediated route in which audio information is first integrated into prompt tokens and subsequently accessed during generation. We further probe prompt-token representations under audio concatenation and find that task-relevant audio attributes remain readily decodable, particularly in middle and later decoder layers, even when AQA performance deteriorates sharply. This dissociation indicates that the failure cannot be explained by complete loss of audio information from the decoder states and is instead consistent with a downstream bottleneck in retrieving or utilizing prompt-mediated information during answer generation. Together, our findings reveal task-dependent audio information routing in LALMs and highlight information utilization as a potential limitation on their generalization.
Figures & tables
SpeechCommands
GTZAN
ESC-50
WER
CER
Model
Single
Concat.
Single
Concat.
Single
Concat.
Single
Concat.
Single
Concat.
Audio-Flamingo-Next
96.67
43.67
93.83
35.33
97.67
54.33
1.68
1.86
0.49
0.57
MiMo-Audio
94.67
40.67
85.17
45.00
95.17
53.33
3.19
4.49
1.49
2.80
Step-Audio-R1
97.67
56.00
87.83
40.00
97.50
67.33
2.21
3.13
1.09
2.12
Qwen3-Omni
98.00
88.67
96.00
82.67
97.67
83.33
1.43
1.53
0.39
0.54
Table 1: Performance of four LALMs on single-segment attribute-recognition controls and concatenated-audio AQA, together with single- and concatenated-input ASR. AQA results are multiple-choice accuracies (%; higher is better), whereas LibriSpeech results are WER and CER (%; lower is better).
Figure 1: Layer-wise effects of attention knockout on three directed pathways across decoder depth. The top row reports the increase in WER on LibriSpeech ASR, and the bottom row reports the decrease in multiple-choice accuracy on MMAU test-mini AQA. Each point corresponds to a contiguous five-layer intervention window indexed by its starting layer. Positive values indicate performance degradation relative to clean inference.
Figure 2: Layer-wise linear probing accuracy for task-relevant audio attributes decodable from prompt-token representations under concatenated inputs. Rows correspond to SpeechCommands, GTZAN, and ESC-50, and columns correspond to the evaluated LALMs. For each decoder layer, separate probes predict the attributes of the front and back segments; their macro-average is also reported. Higher accuracy indicates greater linear decodability of the target attribute.
Audio Large Language Models (Audio LLMs) have advanced in audio understanding, yet they can still predict the answer by reasoning from textual cues or linguistic priors rather than the provided audio. A common remedy is to train models on data whose answers cannot be inferred from text alone. This approach can improve performance, but what changes within the model remains unclear. In this paper, we ask what must happen inside the model for the audio to actually determine the answer. Our findings are threefold. (1) Replacing the audio with silence or unrelated audio causes substantially larger performance degradation in the trained model than in the pretrained model. (2) Acoustic information most strongly shapes the model's representations of the answer choices in early-to-middle layers, while training mainly increases the influence of audio information on the final prediction in middle-to-late layers. (3) The weights learned during training have their largest impact in specific layer bands. Together, these results provide a mechanistic account of how training strengthens the use of acoustic evidence in Audio LLMs.
Hyebin Cho, Suho Yoo, Jihoo Jung +1
Korea Advanced Institute of Science and Technology, South Korea
Despite the strong performance of large audio language models (LALMs) in various tasks, exactly how and where they integrate acoustic features with textual context remains unclear. We adapt causal tracing to investigate the internal information flow of LALMs during audio comprehension. By conducting layer-wise and token-wise analyses across DeSTA, Qwen, and Voxtral, we evaluate the causal effects of individual hidden states. Layer-wise analysis identifies different fusion strategies, from progressive integration in DeSTA to abrupt late-stage fusion in Qwen. Token-wise analysis shows that the final sequence token acts as an informational bottleneck where the network decisively retrieves relevant information from the audio. We also observe an attention-like query mechanism at intermediate token positions that triggers the model to pull task-relevant audio context. These findings provide a clear characterization of when and where multi-modal integration occurs within LALMs.
Wei-Chih Chen, Chien-yu Huang, Hung-yi Lee
National Taiwan University, Taiwan · Carnegie Mellon University, United States
Large audio-language models (LALMs) perform strongly on individual audio tasks, but whether these capabilities can be reliably composed remains underexplored. We conduct a controlled diagnostic study of capability composition in LALMs, requiring models to integrate audio-attribute recognition, cue-conditioned segment selection, and downstream ASR or question answering. We construct two-utterance inputs with distinct acoustic cues to evaluate composition across environmental sound, gender, and emotion cues, with ASR, Math QA, and Factual QA as downstream tasks. Across four open-source LALMs, compositional QA accuracy decreases in 39 of 40 model-task-cue settings, by an average of 26.7 percentage points. ASR exhibits a similarly consistent degradation, with WER increasing in 39 of 40 settings by an average of 28.5 percentage points, while the magnitude of degradation varies across models, cue types, and cue salience. We further probe these failures through output format, positional preference, and chain-of-thought (CoT) analyses. Our study reveals a systematic gap between possessing individual audio capabilities and reliably composing them.
Chien-Feng Liu, Chih-Kai Yang, Bo-Han Feng +3
National Taiwan University · ASUS Open Cloud Infrastructure Software Center · NTU Artificial Intelligence Center of Research Excellence (NTU AI-CoRE)