Context is central to human auditory perception, as listeners interpret sounds within their acoustic environment to resolve ambiguity. However, large-scale datasets that jointly capture sound events and acoustic scenes remain limited. We address this by augmenting AudioSet Strong with clip-level scene annotations under two taxonomies for joint sceneevent modeling. A corpus-scale analysis shows that scene identity is better captured by events distinctive to a scene than by those that merely occur frequently. Reweighting event co-occurrences toward scene-specific associations produces representations that better align with human-derived taxonomies, while also exposing scenes where the acoustic and semantic views diverge. This geometric divergence predicts classification errors, while supervised validation confirms that the proposed scene annotations provide meaningful complementary information for developing scalable context-aware machine listening systems.
Figures & tables
Fig. 1: We propose extending AudioSet Strong with scene labels to foster joint analysis of sound event and acoustic environment relationships.
Fig. 2: Overview of the acoustic scene label extraction pipeline: visual evidence extraction, cross-modal reasoning, ensemble voting, and taxonomy mapping.
Fig. 3: 2-D projections of representational views with highlighted neighborhoods of four scenes across acoustic, semantic, and lexical spaces.
View
Tax.
Acoustic
Semantic
Lexical ( AL )
Lexical ( AE )
Manual
AV
0.123
0.312
0.326
0.253
AC
0.183
0.336
0.277
0.192
Acoustic
AV
0.167
0.231
AC
0.225
0.258
Semantic
AV
0.313
0.285
AC
0.390
0.345
Table 1: Mantel test Spearman correlations ( rM ) between representational similarity matrices across views and taxonomies. Audiovisual (AV) and audio-centric (AC) results are reported. All correlations are significant ( p=0.0001 ).
Fig. 4: Heatmap of scene-level semantic shift, ωk(s)=1−RBOk(s) , across neighborhood scales ( k=1,…,25 ) for the audiovisual taxonomy. Scenes are ordered by instability; higher = greater acoustic–semantic mismatch.
Evaluation taxonomy
Classifier
Accuracy ↑
Macro F1 ↑
Audiovisual (AV)
NB Acoustic
0.1881
0.1282
NB Semantic
0.1787
0.1315
Audio-centric (AC)
NB Acoustic
0.2816
0.1847
NB Semantic
0.2576
0.1860
Table 2: Scene classification performance of the Naive Bayes classifiers based on acoustic and semantic views under the AV and AC evaluation taxonomies. Classifiers receive event labels as input and predict the corresponding scene.
Fig. 5: Intuition map relating neighborhood overlap and Naive Bayes vulnerability. It indicates stable and vulnerable regimes.
Automatic audio scene description turns a recording into a text account of a situation. One difficulty is deciding which elements of the audio should be kept, since a description cannot include them all. Annotators disagree about this, making a ground truth hard to obtain. In this work, we first define the ground truth, then generate the data. We focus on audio events and define sound salience with a simple rule: a sound is salient when a speaker audibly reacts to it. For scale and variety, a controlled set of scenarios fixes the ground truth, and a language model writes the dialogues. The resulting corpus, CARES, contains 10,000 two-speaker scenes. We then benchmark six audio-language models on three tasks: identifying the scene, tagging the sounds present, and classifying reactions. We show that these models hear the sounds but miss how the speakers react to them.
Marcel Gibier, Thomas Thebaud, Olivier Boëffard +1
Recent Large Audio Language Models (LALMs) have achieved remarkable progress in audio perceptual tasks across individual acoustic layers, including speech, sound, and music. However, existing benchmarks predominantly evaluate these layers in isolation, overlooking the complex contextual relationships that arise when multiple acoustic sources co-occur in real-world auditory scenes. Real-world auditory interpretation requires Context-Aware Auditory Scene Understanding (CASU): the ability to comprehend the holistic scene by integrating sound layers. To evaluate this capability, we introduce the CASU benchmark, which assesses whether Audio LLMs can interpret auditory scenes composed of speech, acoustic events (e.g., announcements), and background environments (e.g., traffic), and reason about the logical relationships between these layers. We propose a scalable pipeline for constructing time-accurate, semi-synthetic audio streams by composing real-world scene sounds with synthetic speech. Building on this data, we design four tasks that probe scene understanding: contextual question answering, entity extraction from the scene, speaker role inference, and counterfactual reasoning where scene is manipulated. Experiments across multiple LALMs demonstrate that effective auditory scene understanding requires integration over all auditory layers, rather than reliance on speech or sound alone, underscoring the necessity of CASU for advancing complex audio understanding in LALMs.
Pengfei Zhang, Hoang H Nguyen, Kazi Shaharair Sharif +6
University of California Irvine · University of Illinois Chicago · Kennesaw State University
Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we introduce Logbook, a benchmark for hour-scale audio understanding, with recordings ranging from ten minutes to six days. Given a continuous audio recording and an event label vocabulary, a system must predict a gap-free segmentation with an event label and a description per segment. We compare 52 systems, end-to-end and cascaded, and ablate fine-tuning, context length, and reasoning budget. We find the task tractable, though the best systems remain below the human reference. Also, over-segmentation is pervasive, and fine-tuning partially mitigates it. Finally, end-to-end are often better than cascaded systems, but degrades with longer context.