Heard But Not Said: Prompting Understates Temporal Information of Musical Moments in Frozen Large Audio-Language Models

Eunsik Shin

cs.SDOct 21, 2026

Heard But Not Said: Prompting Understates Temporal Information of Musical Moments in Frozen Large Audio-Language Models

Authors: Eunsik Shin

Organizations: Music and Audio Research Group (MARG), Department of Intelligence and Information, Seoul National University

Abstract

Large audio-language models (LALMs) understand music, yet struggle to locate a described moment in time. We ask whether this reflects missing temporal information or a bottleneck in how the time span is extracted. Using a paired-query probe built from Slakh2100 stem activations, we attach a state query (e.g., only piano and bass are playing'') and a change query (e.g., the guitar drops out, leaving piano and bass'') to the same span and vary how much preceding audio the model is shown. Across frozen LALMs, three ways of extracting the span ask different questions: full-track prompting asking whether the model can say when; windowed yes/no scoring asking whether it verifies the moment when handed the segment, and whether it needs what came before; attention asking whether it can find the moment on its own. State queries score above baseline without context, while change queries improve with preceding context in most short-segment models; 30s padded encoders show no comparable recovery. Prompting and attention expose almost none of this. Prompting failures thus understate what LALMs can verify, though not what they can locate, by an amount set by query type and encoder.

Figures & tables

Figure 1: (a) Stem activity read from Slakh2100 MIDI. A sparse texture (only piano and bass) that begins right after a stem departure (guitar) becomes one target span with two queries sharing the same audio, so chance is identical. (b) In Exp. 2 the model sees a window that ends with the judged 2 s segment and begins

C

seconds earlier; at

C{=}0

the departure lies outside the window and cannot be verified.

Figure 2: Exp. 2, oracle-duration R@[email protected] versus preceding context

C

for state (blue) and change (red) queries on the 570 evaluation pairs. Change queries recover with context in three of the four short-segment models (left); the three 30 s padded models (right) degrade as

C

grows regardless of training.

		Exp. 2, standard R@[email protected]				Exp. 2, oracle-duration R@[email protected]				Exp. 1 R@[email protected]		Exp. 3
		State		Change		State		Change				attn. excess
Model	Encoder segment	$C{=}0$	$C{=}8$	$C{=}0$	$C{=}8$	$C{=}0$	$C{=}8$	$C{=}0$	$C{=}8$	state	change	state / change
Qwen2.5-Omni-3B	Whisper-v3 (block attn.)	0.221 ‡	0.240 ‡	0.105	0.219 ‡	0.279 ‡	0.219 ‡	0.119	0.165 ‡	0.011	0.011	$+$ 0.041 / $+$ 0.020 ∗
Qwen3-Omni-30B-A3B	AuT	0.247 ‡	0.298 ‡	0.102	0.312 ‡	0.242 ‡	0.265 ‡	0.118	0.268 ‡	0.042	0.025	$+$ 0.044 / $+$ 0.062 ∗
MOSS-Audio-4B	MOSS Audio Encoder †	0.232 ‡	0.237 ‡	0.163	0.233 ‡	0.256 ‡	0.204 ‡	0.160 ‡	0.211 ‡	0.030	0.019	n.s.
MOSS-Music-8B	MOSS Audio Encoder †	0.223 ‡	0.184 ‡	0.139	0.170	0.209 ‡	0.142 ‡	0.160 ‡	0.168 ‡	0.072	0.042	n.s.

Table 1: All results on the 570 evaluation pairs. Exp. 2: R@[email protected] with standard post-processing and R@[email protected] with oracle duration at

C{=}0

8

; Exp. 1: R@[email protected]; Exp. 3: span-specific in-span attention excess (

z

-units). ‡ 95% track-bootstrap CI lower bound exceeds chance (2,000 resamples, uncorrected; no Exp. 1 cell qualifies). ∗ CI excludes 0 for both conditions; n.s.: CI includes 0; † Input pipeline interleaves 2 s time markers.

Audio-Language Models

Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.

See all 300 papers

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs
Oct 7, 2026Pooneh Mousavi, Mirco Ravanelli, Cem SubakanAudio-Language Models Audio Reasoning
Concordia University · Mila – Quebec AI Institute · Laval University
On Temporal Binding in Large Audio Language Models
Sep 28, 2026Paul Primus, Gerhard WidmerLLM Interpretability Audio Reasoning
Institute of Computational Perception, Johannes Kepler University Linz, Austria · LIT Artificial Intelligence Lab, Johannes Kepler University Linz, Austria
Note-Level Temporal Grounding of Musical Concepts in Large Audio-Language Models
Aug 30, 2026Kun Fang, Ziyu Wang, Ichiro FujinagaMusic Understanding Audio-Language Models
Schulich School of Music, McGill University, Montr´eal, QC, Canada · CIRMMT (Centre for Interdisciplinary Research in Music Media and Technology), Montr´eal, Canada · Courant Institute, New York University, New York, USA +1

Oct 7, 2026J/K move · Enter open · S save

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

Pooneh Mousavi, Mirco Ravanelli, Cem Subakan

Concordia University · Mila – Quebec AI Institute · Laval University

Audio-Language Models Audio Reasoning Audio QA

Open paper AskPDF arXiv

Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.

Abstract

Audio-Language Models

Explore similar work

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

On Temporal Binding in Large Audio Language Models

Note-Level Temporal Grounding of Musical Concepts in Large Audio-Language Models