A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning
Authors: Harunori Kawano, Takeshi Sasaki
Organizations: Faculty of Engineering and Information Technology, University of Technology Sydney · Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology
While self-supervised learning (SSL) has revolutionized audio representation, the excessive parameterization and quadratic computational cost of standard Transformers limit their deployment on resource-constrained devices. To address this bottleneck, we propose HEAR (Human-inspired Efficient Audio Representation), a novel decoupled architecture. Inspired by the human cognitive ability to isolate local acoustic features from global context, HEAR splits the processing pipeline into two dedicated modules: an Acoustic Model for local feature extraction and a Task Model for global semantic integration. Coupled with an Acoustic Tokenizer trained via knowledge distillation, our approach enables robust Masked Audio Modeling (MAM). Extensive experiments demonstrate that HEAR requires only 15M parameters and 9.47 GFLOPs for inference, operating at a fraction of the computational cost of conventional foundation models (which typically require 85M-94M parameters). Despite this high efficiency, HEAR achieves highly competitive performance across diverse audio classification benchmarks. The code and pre-trained models are available at https://github.com/HarunoriKawano/HEAR
Figures & tables
Figure 1: Overview of the pre-training pipeline for the proposed HEAR framework. (a) Knowledge-Distilled Tokenizer Training: An acoustic tokenizer is trained to encode continuous mel-spectrogram patches into discrete semantic tokens, optimized via signal reconstruction, codebook diversity, and knowledge distillation from a pre-trained teacher model. (b) MAM: Using the frozen tokenizer, the Acoustic Model is pre-trained to predict the discrete tokens of independent random masked patches, enabling robust local feature extraction.
Figure 2: Comprehensive flow of the downstream adaptation, illustrating the feature gating integration of the Acoustic Model outputs and raw power spectrum into the Task Model.
Module
Hidden Size
Intermediate Size
Heads
Layers
Tokenizer
128
512
4
6
Decoder
128
512
4
2
Acoustic Model
384
1536
4
6
Task Model
384
1536
4
1
Table 1: Detailed architectural hyperparameters for each module in the proposed framework.
Dataset
Batch Size
Learning Rate
Max Epochs
ESC-50
16
2×10−4
80
GSCv1
64
3×10−4
30
GSCv2
64
3×10−4
30
VoxCeleb
16
1×10−4
15
Table 2: Hyperparameter settings for fine-tuning on downstream tasks.
Model
Accuracy (%)
Efficiency
ESC-50
GSCv1
GSCv2
VoxCeleb
Params
GFLOPs a
RTF a
wav2vec 2.0 [ 2 ]
-
96.2 b
-
75.1 b
94M
69.6
0.510
HuBERT [ 15 ]
-
96.3 b
-
81.4 b
94M
69.6
0.510
AudioMAE [ 16 ]
94.1
96.9
98.3
94.8
85M
42.4
0.244
SSAST [ 12 ]
88.8
96.0
98.0
64.2
85M
42.4
0.244
BEATs [ 7 ]
95.6
97.7
98.3
-
85M
42.4
0.244
Table 3: Performance and inference efficiency comparison between HEAR and representative baseline models.
Setting
ESC-50
GSCv1
GSCv2
VoxCeleb
Base
84.9
94.3
95.1
87.9
Scratch
62.1
88.1
89.5
72.2
w/o Spectrum
84.7
94.7
95.0
87.6
Transfer
78.5
95.1
95.8
76.0
Table 4: Ablation study results (Accuracy %) on downstream tasks.