HELIX: Scaling Raw Audio Understanding with Hybrid Mamba-Attention Beyond the Quadratic Limit

Khushiyant; Param Thakkar

cs.SDMar 22, 2026

HELIX: Scaling Raw Audio Understanding with Hybrid Mamba-Attention Beyond the Quadratic Limit

Authors: Khushiyant, Param Thakkar

Organizations: Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany · Department of Computer Engineering and Information Technology, Veermata Jijabai Technological Institute, Mumbai, India

Abstract

Audio representation learning typically evaluates design choices such as input frontend, sequence backbone, and sequence length in isolation. We show that these axes are coupled, and conclusions from one setting often do not transfer to others. We introduce HELIX, a controlled framework comparing pure Mamba, pure attention, and a minimal hybrid with a single attention bottleneck. All models are parameter-matched at about 8.3M parameters to isolate architectural effects. Across six datasets, we find that the preferred input representation depends on the backbone, and that attention hurts performance on short, stationary audio but becomes important at longer sequence lengths. On a 5-minute speaker identification task with 30,000 tokens, pure attention fails with out-of-memory errors, while HELIX closes an 11.5-point gap over pure Mamba.

Figures & tables

Tokens	Pure Mamba	HELIX	Pure Attention
500 (5 s)	$\checkmark$	$\checkmark$	$\checkmark$
3,000 (30 s)	$\checkmark$	$\checkmark$	slow
30,000 (5 min)	$\checkmark$	$\checkmark$	OOM

Table 1: Computational feasibility at

{\sim}

8.3M parameters on a single RTX 6000 Pro (48 GB).

\checkmark

= trains normally; slow = feasible but

>

\times

wall time vs. Pure Mamba; OOM = out of memory.

Figure 1: Overview of design choice interactions. Left: The raw-vs-spectrogram accuracy gap (positive = raw better) depends on backbone. Pure Mamba prefers raw on ESC-50 but not Speech Commands; Pure Attention strongly prefers spectrograms. Center: All six Speech Commands variants. Right: As sequence length scales from 3k to 30k tokens, the HELIX advantage grows from 0 to 11.5 points; Pure Attention cannot run at 30k tokens.

Architecture	Raw	Spectrogram
Pure Mamba	55.10 $\pm$ 3.33	53.75 $\pm$ 2.92
HELIX	50.20 $\pm$ 3.71	51.85 $\pm$ 2.41
Pure Attention	44.60 $\pm$ 3.16	46.10 $\pm$ 2.97

Table 2: ESC-50 environmental sound classification (5-fold CV, 100 epochs). Best accuracy per column in bold .

Architecture	Raw	Spectrogram
Pure Mamba	90.27 † (76 ep)	92.36 † (86 ep)
HELIX	92.94 † (81 ep)	92.44 † (94 ep)
Pure Attention	82.43 (100 ep)	91.27 (100 ep)

Table 3: Speech Commands v2 (single split). † Terminated before 100 epochs due to compute limits. Ranking is consistent with trends observed in validation curves across partial runs. Best per column in bold .

Figure 2: ESC-50 results. Solid bars: raw waveform; hatched bars: spectrogram. Error bars show

\pm

1 std across 5 folds. Pure Mamba dominates; input preference flips between backbones.

Figure 3: Speech Commands v2 results. Epoch counts shown above bars (runs that hit compute limits did not reach 100 epochs). Pure Attention raw collapses; HELIX raw leads despite fewer epochs.

Figure 4: ESC-50 training curves (fold 1). Raw waveform models (left) learn slower but Pure Mamba eventually surpasses all spectrogram variants (right). Smoothed with a 7-epoch moving average; raw values shown underneath.

Figure 5: Speech Commands training curves. Left: On raw waveforms, Pure Mamba raw destabilizes after epoch 30 and never recovers, while HELIX raw climbs steadily to 92.9%. Pure Attention raw plateaus at

{\sim}82

%. Right: Spectrograms stabilize all architectures; the gap between variants shrinks to

{\sim}1

Architecture	Best Acc (%)	Final Acc (%)
HELIX Raw	91.31	90.44
Pure Mamba Raw	89.81	89.36

Table 4: Concatenated Speech Commands (

n{=}10

clips,

L{=}1{,}000

tokens). Label from first clip; pooling restricted to first 100 tokens. Both runs crashed at 24 epochs.

Figure 6: Long-sequence training curves. HELIX separates from Pure Mamba after epoch 15 and maintains the gap. Both runs were still improving when compute ran out.

Architecture	Best Acc (%)
Pure Mamba Raw	73.55
HELIX Raw	85.07

Table 5: Speaker identification on VoxPopuli (English), 300 s clips (1,313 speakers,

L{=}30{,}000

tokens). Pure Attention could not run due to OOM.

Figure 7: VoxPopuli 300 s training curves. HELIX pulls ahead early and reaches 85% while Pure Mamba plateaus near 73%. Dotted lines mark best accuracy. This is the largest gap in our study.

Audio Representation Learning

Audio representation learning builds reusable features that capture speech, music, and sound for recognition, retrieval, generation, and analysis.

See all 207 papers

A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning
Mar 27, 2026Harunori Kawano, Takeshi SasakiAudio Representation Learning Self-Supervised Pre-Training
Faculty of Engineering and Information Technology, University of Technology Sydney · Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology
MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audio
Jun 14, 2026Salman Hussain Ali, Umberto Cappellazzo, Mirco RavanelliMamba Speech Processing
1Universit´e de Montr´eal, Canada · 2Imperial College London, UK · 3Concordia University, Canada +1
Audio Token Attention Is Predictable Before the Language Model Runs
Sep 30, 2026Kyoungjun Park, Yunzhe Li, Lili QiuAudio Token Compression Audio Understanding
The University of Texas at Austin

Mar 27, 2026J/K move · Enter open · S save

A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning

Harunori Kawano, Takeshi Sasaki

Faculty of Engineering and Information Technology, University of Technology Sydney · Dept. of Electrical Engineering and Computer Science, Shibaura Institute of Technology

Audio Representation Learning Self-Supervised Pre-Training

Open paper AskPDF arXiv

While self-supervised learning (SSL) has revolutionized audio representation, the excessive parameterization and quadratic computational cost of standard Transformers limit their deployment on resource-constrained devices. To address this bottleneck, we propose HEAR (Human-inspired Efficient Audio Representation), a novel decoupled architecture. Inspired by the human cognitive ability to isolate local acoustic features from global context, HEAR splits the processing pipeline into two dedicated modules: an Acoustic Model for local feature extraction and a Task Model for global semantic integration. Coupled with an Acoustic Tokenizer trained via knowledge distillation, our approach enables robust Masked Audio Modeling (MAM). Extensive experiments demonstrate that HEAR requires only 15M parameters and 9.47 GFLOPs for inference, operating at a fraction of the computational cost of conventional foundation models (which typically require 85M-94M parameters). Despite this high efficiency, HEAR achieves highly competitive performance across diverse audio classification benchmarks. The code and pre-trained models are available at https://github.com/HarunoriKawano/HEAR

Abstract

Audio Representation Learning

Explore similar work

A Human-Inspired Decoupled Architecture for Efficient Audio Representation Learning

MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audio

Audio Token Attention Is Predictable Before the Language Model Runs