Hierarchical Interpretable Multitask Audio Representation Learning: An Analysis of Event-Scene Relationships through the Information Bottleneck

Claudia Montero-Ramírez; Alba Martínez-Serrano; Carmen Peláez-Moreno

eess.ASOct 15, 2026

Hierarchical Interpretable Multitask Audio Representation Learning: An Analysis of Event-Scene Relationships through the Information Bottleneck

Authors: Claudia Montero-Ramírez, Alba Martínez-Serrano, Carmen Peláez-Moreno

Abstract

Joint Acoustic Scene Classification (ASC) and Audio Tagging (AT) models traditionally rely on unconstrained multihead architectures. However, this design ignores the natural hierarchical dependency between sounds and acoustic scenes. In this work, we investigate a hierarchical (sequential) conditioning paradigm that predicts scene categories directly from intermediate event probabilities. Through a large-scale empirical benchmark with 66 full training runs across three state-of-the-art backbones (PaSST, PANNs, and BEATs) and different loss-weighting settings (beta in [0.0, 1.0]), we show that hierarchical conditioning achieves competitive classification performance while altering multitask optimization dynamics. Unlike multihead baselines, which sequentially solve the easier ASC task before optimizing AT, the hierarchical architecture enables synchronized dual-task learning for specific loss-weighting settings (beta in [0.6, 0.9]). Furthermore, we propose a multi-perspective interpretability framework that combines an analysis of the information bottleneck, manifold geometry, and Formal Concept Analysis (FCA) on a Term-Frequency Inverse-Document-Frequency (TF-IDF) event representation, providing complementary insights into event-scene relationships. Our findings reveal that hierarchical conditioning systematically yields lower conditional entropy, higher mutual information, and superior latent cluster separability (higher Silhouette scores), establishing a more structured and semantically faithful intermediate feature space.

Figures & tables

Fig. 1: a) Traditional architecture of multitask models with multiple heads [ 12 , 13 , 14 ] . b) Our proposed hierarchical architecture using an information bottleneck with a feed-forward network (FFN).

Fig. 2: Sound Event distribution for development and evaluation sets. a) Distribution regarding the total duration (in seconds) of each sound event label. b) Frequency of occurrence for each sound event based on unique audio file counts. Teal bars represent the development set, while light orange bars represent the evaluation set. All labels are sorted according to the development set duration for consistency.

Fig. 3: Acoustic scene distribution for development and evaluation sets. The bars represent the total number of unique audio recordings associated with each scene label. Teal and light orange bars denote the development and evaluation partitions, respectively.

Fig. 4: Event-scene relationships for the development (top row) and evaluation (bottom row) subsets. (a,d) Scene-event presence matrices, where each entry is the proportion of the scene duration in which the event is active. (b,e) FCA lattices of the formal contexts obtained by binarizing these matrices with a threshold of $\varphi=0.15$ . (c,f) FCA lattices obtained by applying the same threshold after TF-IDF weighting (smoothed IDF and per-scene L2 normalization), which down-weights events shared across scenes and highlights discriminative event-scene associations. In each lattice, blue nodes denote scenes (objects) and grey labels denote events (attributes); an event is associated with every scene reachable downwards from its label, and events at the bottom node are not associated with any scene.

Fig. 5: Exact network configuration of the models with a) multihead and b) hierarchical architectures. * BEATs and PaSST use 768-dimensional acoustic embeddings, whereas ** PANNs uses 2048-dimensional embeddings.

Fig. 6: The Patchout transformer (PaSST) adapted to a hierarchical architecture: ( a ) Overall architecture; C: classification token and D: distillation token [ 16 ] . ( b ) The architecture of the Self-attention layer + FFN (Feedforward network) [ 40 ] . (c) Hierarchical extension of the model to predict acoustic scenes from sound events.

Fig. 7: Summary diagram of the combined loss calculation.

Fig. 8: Overview of the proposed hierarchical information bottleneck framework, where the acoustic input

X

, comprising the data and ground truth labels

X_{\mathrm{gt}}=\{{X_{\mathrm{events}},X_{\mathrm{scene}}}\}

, is compressed into an interpretable sound event bottleneck

E

to predict the acoustic scene

S

Architecture	Metric	PaSST	PANNs	BEATs
Multihead	Accuracy	0.89	0.94	0.94
Multihead	Balanced Acc.	0.90	0.95	0.95
Hierarchical	Accuracy	0.95	0.95	0.91
Hierarchical	Balanced Acc.	0.95	0.96	0.92

TABLE I: Global ASC performance in terms of accuracy and balanced accuracy

Architecture	Metric	PaSST	PANNs	BEATs
Multihead	F1-macro	0.69	0.61	0.62
Multihead	ER	0.27	0.27	0.35
Hierarchical	F1-macro	0.63	0.63	0.66
Hierarchical	ER	0.28	0.38	0.31

TABLE II: Global AT performance in terms of F1-macro and ER

Arch.	Model	Metric	home	office	city	residential
					center	area
Multihead	PaSST	Precision	0.99	1.00	0.80	0.82
	PaSST	Recall	1.00	0.98	0.85	0.77
	PANNs	Precision	0.99	1.00	0.96	0.85
	PANNs	Recall	0.99	0.98	0.84	0.97
	BEATs	Precision	0.99	1.00	0.99	0.84

TABLE III: Class-wise ASC performance in terms of precision and recall

Architecture	Model	Metric	banging	impact	rustling	snapping	squeaking	bird	brakes	breathing	car
			(obj.)	(obj.)	(obj.)	(obj.)	(obj.)	singing	squeaking
Multihead	PaSST	F1-score	0.75	1.00	0.77	0.91	0.00	0.63	0.73	0.33	0.83
	PaSST	ER	0.25	0.00	0.25	0.17	1.00	0.51	0.28	0.33	0.27
	PANNs	F1-score	0.40	1.00	0.52	0.57	0.00	0.73	0.54	0.22	0.82
	PANNs	ER	0.00	0.00	0.25	0.00	1.00	0.32	0.23	0.78	0.29
	BEATs	F1-score	0.40	1.00	0.80	0.75	0.00	0.70	0.70	0.17	0.82

TABLE IV: Class-wise AT performance in terms of F1-score and ER

Fig. 9: Test performance trajectories of the multitask models across different task-loss weights

\beta

, represented by color, in the ASC accuracy – AT F1-macro space. The left column (a,c,e) shows the multihead architecture, while the right column (b,d,f) shows the Hierarchical architecture. From top to bottom, rows correspond to BEATs, PANNs, and PaSST, respectively. Each trajectory connects the test performance obtained for the different

\beta

values, with a

90\%

smoothing applied for visualization.

Fig. 10: Information-theoretic comparison of the learned event representations across BEATs, PANNs, and PaSST. Rows show mean conditional event entropy

\bar{H}(E\mid S)

, mutual information

I(E;S)

, and Kullback-Leibler divergence

D_{\mathrm{KL}}(P_{H}|P_{M})

, respectively. Colors distinguish the hierarchical and multihead architectures.

Architecture	PaSST	PANNs	BEATs
Multihead	0.451	0.426	0.529
Hierarchical	0.534	0.512	0.478

TABLE V: Silhouette Scores of the learned event representations across the three pretrained audio backbones.

Fig. 11: Two-dimensional t-SNE projections of the predicted event probability distributions $P(E\mid S)$ , where each point represents an audio segment, and colors indicate the ground-truth acoustic scene. The projections illustrate the spatial organization and cluster separability of the learned event representations across the different architectures and pretrained audio backbones.

Fig. 12: FCA lattices constructed from the TF-IDF-weighted event-scene relationship matrices predicted by the models, using a threshold of $\varphi=0.15$ . The top row (a-c) corresponds to the hierarchical architecture, while the bottom row (d-f) corresponds to the multihead architecture. From left to right, columns correspond to PANNs, PaSST, and BEATs, respectively.

Arch.	Model	Epoch	$\beta$	ASC	AT	AT
				Pool.	Pool.	Th.
Multihead	PaSST	108	0.9	None	max	CT
	PANNs	296	0.8	max ∗	max	ST
	BEATs	228	0.4	max ∗	max	ST
Hierarchical	PaSST	227	0.9	None	max	CT
	PANNs	187	0.7	max ∗	max	ST

TABLE VI: Optimal hyperparameter configurations selected per backbone model and architectural approach. Threshold strategies are abbreviated as CT (Class-specific Thresholding) and ST (Static Thresholding).

Model	Arch.	Scene	Top event
PaSST	Mult.	home	water tap running (+1.60)
		office	fan (+3.93)
		city center	car (+1.59)
		residential area	bird singing (+5.27)
	Hier.	home	car (+1.18)
		office	fan (+5.28)

TABLE VII: Most influential acoustic event for each scene according to mean SHAP values.

Fig. 13: SHAP value summary plot comparing feature importance for BEATs under Multihead and Hierarchical configurations.

Fig. 14: SHAP value summary plot comparing feature importance for PANNs under Multihead and Hierarchical configurations.

Fig. 15: SHAP value summary plot comparing feature importance for PaSST under Multihead and Hierarchical configurations.

Fig. 16: Normalized confusion matrices for ASC on the evaluation set. Each column corresponds to an acoustic model, namely (a,d) BEATs, (b,e) PANNs, and (c,f) PaSST, while the top and bottom rows (a-c) represent the multihead and (d-f) hierarchical architectures, respectively.

Fig. 17: Normalized co-occurrence matrices for AT on the evaluation set. Each column corresponds to an acoustic model, namely (a,d) BEATs, (b,e) PANNs, and (c,f) PaSST, while the top and bottom rows (a-c) represent the multihead and (d-f) hierarchical architectures, respectively.

Audio Classification

Audio classification assigns recordings to categories such as speech, music, animal calls, acoustic scenes, and health-related events.

See all 147 papers

Jun 26, 2026

Grammar-Guided Hierarchical Parsing for Long-form Audio Activity Recognition

Long-form audio exhibits an inherent hierarchy: fine-grained events form sub-activities, which in turn constitute higher-level activities. Prior work often models these levels separately, leading to cross-level inconsistencies and requiring supervision at multiple levels. We formulate the problem as hierarchical parsing from event-level evidence: given detected event segments with class posteriors, we infer an order-consistent Act-Sub-Event parse tree. We propose Hierarchical Activity Grammar, encoding hierarchical composition and temporal-order constraints, and perform grammar-guided decoding that combines event evidence with a grammar prior. This yields a temporally grounded parse tree from which sub-activity segmentation and activity classification are derived, without requiring sub-activity or activity labels for training. Experiments on the long-form MultiAct audio dataset demonstrate improved temporal-order consistency (Edit score) and produces interpretable hierarchies.

Peng Zhang, Qingyu Luo, Philip J. B. Jackson +1

Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, U.K.

Temporal Action Segmentation Human Activity Recognition Audio Understanding Grammar-Constrained Decoding

Ask PDF

Jul 2, 2026

A Multi-Branch Hierarchy-Aware Framework for Heterogeneous Audio Classification

This technical report describes our system for Task 1 of the DCASE 2026 Challenge, which aims to classify heterogeneous audio recordings according to the Broad Sound Taxonomy (BST). The task requires both accurate second-level prediction and consistency with the top-level taxonomy. Our system is built on CLAP-based audio-text representations and is improved along three strategies: expanding the training set with a filtered subset of BSD35k, enhancing acoustic modeling with feature-specific branches, and refining predictions using hierarchy-aware classifiers and KNN-based post-processing. Among the acoustic features considered, the log-STFT branch provides the strongest single-model performance. With KNN-based post-processing, our best single system achieves a hierarchical F1 score (Hier. F1) of 80.84% on the BSD10k-v1.2 set under the same evaluation protocol as the baseline. We further construct ensemble systems by combining models with complementary acoustic features and classification heads, achieving Hier. F1 scores of 81.25% and 81.18%, respectively.

Beile Ning, Jiayi Yu, Zitong Wang +5

Wuhan University, Wuhan, China · Harbin Engineering University, Harbin, China

Hierarchical Classification Audio Classification

Ask PDF

Jun 24, 2026

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

Recent Large Audio Language Models (LALMs) have achieved remarkable progress in audio perceptual tasks across individual acoustic layers, including speech, sound, and music. However, existing benchmarks predominantly evaluate these layers in isolation, overlooking the complex contextual relationships that arise when multiple acoustic sources co-occur in real-world auditory scenes. Real-world auditory interpretation requires Context-Aware Auditory Scene Understanding (CASU): the ability to comprehend the holistic scene by integrating sound layers. To evaluate this capability, we introduce the CASU benchmark, which assesses whether Audio LLMs can interpret auditory scenes composed of speech, acoustic events (e.g., announcements), and background environments (e.g., traffic), and reason about the logical relationships between these layers. We propose a scalable pipeline for constructing time-accurate, semi-synthetic audio streams by composing real-world scene sounds with synthetic speech. Building on this data, we design four tasks that probe scene understanding: contextual question answering, entity extraction from the scene, speaker role inference, and counterfactual reasoning where scene is manipulated. Experiments across multiple LALMs demonstrate that effective auditory scene understanding requires integration over all auditory layers, rather than reliance on speech or sound alone, underscoring the necessity of CASU for advancing complex audio understanding in LALMs.

Pengfei Zhang, Hoang H Nguyen, Kazi Shaharair Sharif +6

University of California Irvine · University of Illinois Chicago · Kennesaw State University

Audio-Language Model Evaluation Audio Understanding Audio-Language Models

Ask PDF

Abstract

Audio Classification

Explore similar work

Grammar-Guided Hierarchical Parsing for Long-form Audio Activity Recognition

A Multi-Branch Hierarchy-Aware Framework for Heterogeneous Audio Classification

From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models