Scattering Transform for Auditory Attention Decoding

René Pallenberg; Fabrice Katzberg; Alfred Mertins; Marco Maass

eess.SPFeb 26, 2026

Scattering Transform for Auditory Attention Decoding

Authors: René Pallenberg, Fabrice Katzberg, Alfred Mertins, Marco Maass

Organizations: Institute for Signal Processing, University of Luebeck, 23562 Luebeck, Germany · Department for Vision and Machine Learning, FPI Food Processing Innovation GmbH & Co. KG, 23556 Luebeck, Germany · German Research Center for Artificial Intelligence (DFKI), AI for Assistive Health Technologies, 23562 Luebeck, Germany

Abstract

The use of hearing aids will increase in the coming years due to demographic change. One open problem that remains to be solved by a new generation of hearing aids is the cocktail party problem. A possible solution is electroencephalography-based auditory attention decoding. This has been the subject of several studies in recent years, which have in common that they use the same preprocessing methods in most cases. In this work, in order to achieve an advantage, the use of a scattering transform is proposed as an alternative to these preprocessing methods. The two-layer scattering transform is compared with a regular filterbank, the synchrosqueezing short-time Fourier transform and the common preprocessing. To demonstrate the performance, the known and the proposed preprocessing methods are compared for different classification tasks on two widely used datasets, provided by the KU Leuven (KUL) and the Technical University of Denmark (DTU). Both established and new neural-network-based models, CNNs, LSTMs, and recent Transformer/graph-based models are used for classification. Various evaluation strategies were compared, with a focus on the task of classifying speakers who are unknown from the training. We show that the two-layer scattering transform can significantly improve the performance for subject-related conditions, especially on the KUL dataset. However, on the DTU dataset, this only applies to some of the models, or when larger amounts of training data are provided, as in 10-fold cross-validation. This suggests that the scattering transform is capable of extracting additional relevant information.

Figures & tables

Fig. 1 : Scattering transform applied to audio (top) and EEG at T8 (bottom) from KUL dataset. Layer 2 captures fine-grained temporal modulations invisible in Layer 1, shown by temporal striations (0.0-0.5s) and differentiated patterns during acoustically similar events (3.5-4.5s). Original signal and ST layers 1-2 with, $F_{o}=8$ , $Q=8$ .

Panel of Figure 2 — Fig. 2 : Architecture of the models used. The input has $T$ time steps, $C_{\mathrm{eeg}}$ channels for the EEG, and $C_{\mathrm{aud}}$ channels for the audio signals. The inputs are on the left, and the output is on the right. LSTM-2 can only work with two speakers, while the LSTM-X can handle a variable number of speakers.

EEG $x_{e}\in\mathbb{R}^{T\times 64C_{e}},\text{Audio}_{1}x_{1},\text{Audio}_{2}x_{2}\in\mathbb{R}^{T\times C_{a}}$
Input	Steps	Output
$x_{e}$	AvgTime	$w_{e}\in\mathbb{R}^{1\times 64C_{e}}$
$w_{e}$	Dense(units=8, tanh), DO(0.5)	$w_{e}\in\mathbb{R}^{8}$
$w_{e}$	Dense(units=64 $C_{e}$ , tanh)	$w_{e}\in\mathbb{R}^{64C_{e}}$
$x_{e}$	$x_{e}\odot w_{e}$ ,	$x_{e}\in\mathbb{R}^{T\times 64C_{e}}$
	Concat( $x_{e},x_{1},x_{2}$ )	$x\in\mathbb{R}^{T\times(64C_{e}+2C_{a})}$

TABLE I : CNN-C1: Detailed layer configuration. AvgTime eliminates the time dimension by averaging over

T

. Convolutions use no padding and stride

1

EEG $x_{e}\in\mathbb{R}^{T\times 64C_{e}},\text{Audio}_{i}x_{i}\in\mathbb{R}^{T\times C_{a}}$
Input	Steps	Output
$x_{e}$	Conv1D(k=1, f=8)	$x_{e}\in\mathbb{R}^{T\times 8}$
$x_{e}$	Conv1D(k=3, f=16, ReLU)	$x_{e}\in\mathbb{R}^{(T-2)\times 16}$
$x_{e}$	Conv1D(k=3, f=16, ReLU, dil=3)	$x_{e}\in\mathbb{R}^{(T-8)\times 16}$
$x_{e}$	Conv1D(k=3, f=16, ReLU, dil=9)	$x_{e}\in\mathbb{R}^{(T-15)\times 16}$
$x_{i}$	BN, Conv1D(k=3, f=16, ReLU)	$x_{i}\in\mathbb{R}^{(T-2)\times 16}$

TABLE II : CNN-Dil: Detailed layer configuration. Strides are always 1. Dilation (dil) defaults to 1 unless specified. For signals with

T<16

, dil=1 in all convolutions. Kernel length (k) is at the first postion, number of filters (f) at the second.

Input	Steps	Output
EEG $\in\mathbb{R}^{T\times 64C_{e}}$	BN, LSTM 1 (units=15)	$x_{e}\in\mathbb{R}^{15}$
Audio ${}_{1}\in\mathbb{R}^{T\times C_{a}}$	BN, LSTM 2 (units=15)	$x_{1}\in\mathbb{R}^{15}$
Audio ${}_{2}\in\mathbb{R}^{T\times C_{a}}$	BN, LSTM 2 (units=15)	$x_{2}\in\mathbb{R}^{15}$
$(x_{e},x_{1},x_{2})$	Concat( $x_{e}+x_{1},\;x_{e}+x_{2}$ )	$x\in\mathbb{R}^{30}$
$x$	BN, DO(0.5)	$x\in\mathbb{R}^{30}$
$x$	Dense (units=40, ReLU)	$x\in\mathbb{R}^{40}$

TABLE III : LSTM-2: Detailed layer configuration

Input	Steps	Output
EEG $\in\mathbb{R}^{T\times 64C_{e}}$	BN, LSTM 1 (units=15)	$x_{e}\in\mathbb{R}^{15}$
Audio ${}_{i}\in\mathbb{R}^{T\times C_{a}}$	BN, LSTM 2 (units=15)	$x_{i}\in\mathbb{R}^{15}$
$x_{e},x_{i}$	Concat( $x_{e},x_{i}$ ), BN, DO(0.5)	$x_{i}\in\mathbb{R}^{30}$
$x_{i}$	Dense(units=20, ReLU)	$x_{i}\in\mathbb{R}^{20}$
$x_{i}$	BN, DO(0.5)	$x_{i}\in\mathbb{R}^{20}$
$x_{i}$	Dense(units=1, Sigmoid)	$y_{i}\in[0,1]$

TABLE IV : LSTM-X: Detailed layer configuration.

EEG $x_{e}\in\mathbb{R}^{T\times 64\times C_{e}}$
Input	Steps	Output
-	$A=A_{ds}+A_{learned}$	$A\in\mathbb{R}^{64\times 64}$
-	Normalized Laplacian $L=AD^{-1}$	$L\in\mathbb{R}^{64\times 64}$
$X_{e}$	ELU $(X_{e}L+X_{e})$	$x_{e}\in\mathbb{R}^{T\times 64\times C_{e}}$
$x_{e}$	Reshape	$x_{e}\in\mathbb{R}^{64\times TC_{e}}$
$x_{e}$	Dense(units=128), DO(0.5)	$x_{e}\in\mathbb{R}^{64\times 128}$

TABLE V : EEGEncoder: GCANet-NoEn.

$\text{Audio}_{i}x_{i}\in\mathbb{R}^{T\times C_{a}}$
Input	Steps	Output
$x_{i}$	Conv1D(filters=64, kernel=8), BN, ReLU	$x_{i}\in\mathbb{R}^{64\times C_{a}}$
$x_{i}$	Dense(units=128), BN, ReLU	$x_{i}\in\mathbb{R}^{64\times 128}$

TABLE VI : AudioEncoder: GCANet-NoEn.

$F_{o}$	$J$	$Q$	1	2	3	All	FLOPs	Lag
64	6	8	1	54	179	234	11M	0.06
32	7	8	1	62	237	300	12M	0.13
16	8	8	1	70	303	374	15M	0.25
8	9	8	1	78	377	456	20M	0.5
8	11	4	1	42	189	232	10M	0.5
8	11	2	1	23	108	132	7M	0.5

TABLE VII : Number of channels after the ST for one audio channel with 16384 Hz sampling rate for different

F_{o}

Q

and

J

. The number of channels in each of the three scattering layers is listed separately.

$F_{o}$	$J$	$Q$	1	2	3	All	FLOPs	Lag
64	1	8	1	7	0	8	16,173	0.06
32	2	8	1	7	0	8	8,354	0.13
16	3	8	1	14	9	24	36,734	0.25
8	4	8	1	22	27	50	69,724	0.5
8	4	4	1	14	14	29	38,219	0.5
8	4	2	1	8	11	20	27,766	0.5

TABLE VIII : Number of channels after the ST for one EEG channel with 128 Hz sampling rate for different

F_{o}

Q

and

J

. The number of channels in each of the three scattering layers is listed separately.

Panel of Figure 11 — Fig. 3 : Kernel density plots comparing baseline (base), ST (scat), and SSQ-STFT preprocessing on DTU and KUL datasets. Accuracy distributions for trial-wise $5\times 2$ cv across all subjects with $L_{x}=$ 2\text{,}\mathrm{s} $(CNN-C1, CNN-Dil) and $L_{x}=$1\text{\,}\mathrm{s}$ (others). ST parameters: $F_{o}=$ 8\text{,}\mathrm{Hz}$$ , $Q_{a}=Q_{e}=8$ .

Fig. 4 : Kernel density plots comparing baseline (base), ST (scat), and SSQ-STFT preprocessing for speaker-wise evaluation on the KUL dataset. Accuracy distributions for trial-wise

5\times 2

cv across all subjects with

L_{x}=

2\text{,}\mathrm{s}

(CNN-C1, CNN-Dil) and $L_{x}=$1\text{\,}\mathrm{s}

(others). ST parameters:

F_{o}=

8\text{,}\mathrm{Hz}$$ ,

Q_{a}=Q_{e}=8

Fig. 5 : Collection of kernel density plots to compare the results of the scattering pipeline for different

F_{o}

and the baseline pipeline for the speaker-wise evaluation on the KUL dataset, with

L_{x}=

2\text{,}\mathrm{s}$$ for the LSMT-2. Each plot shows the accuracy distribution of the different runs over all subjects.

Q_{a}=Q_{e}=8

for the scattering pipeline.

TABLE IX : Comparison of the tested values for the scattering parameters

Q_{a}

and

Q_{e}

for the speaker-wise evaluation on the KUL dataset. The heatmap shows the median accuracies for the different configurations

Q_{a}

and

Q_{e}

for each model.

L_{x}=

2\text{,}\mathrm{s}

and $F_{o}=$8\text{\,}\mathrm{Hz}

Fig. 6 : Comparison of the tested values for the scattering parameters

F_{o}

Q_{a}

, and

Q_{e}

for the speaker-wise evaluation on the KUL dataset.

L_{x}=

2\text{,}\mathrm{s}$$ .

Fig. 7 : Comparison of the results for the different speaker sets of the speaker-wise evaluation on the KUL dataset for different window lengths. Each of the kernel density plots shows the accuracies over all subjects for one of the speaker sets. The scattering pipeline with the LSTM-2 and

F_{o}=

8\text{,}\mathrm{Hz}$$ ,

Q_{a}=8

, and

Q_{e}=8

was applied.

Fig. 8 : Comparison of the two different splits for the speaker-wise evaluation on the KUL dataset for each subject, for

L_{x}=

2\text{,}\mathrm{s}

. Each line represents the accuracy for one run on one subject with one of the two speaker sets. The scattering pipeline with the LSTM-2 and $F_{o}=$8\text{\,}\mathrm{Hz}

Q_{a}=2

, and

Q_{e}=8

was applied.

Fig. 9 : Overview of the experimental results for the cross-subject tests on the KUL dataset for

L_{x}=

1\text{,}\mathrm{s}

. Each kernel density plot shows the accuracy distribution over all subjects and runs of the $5\times 2$ cv. The results for the models with the preprocessing piplines are compared. The scattering pipeline was applied with $F_{o}=$8\text{\,}\mathrm{Hz}

Q_{a}=8

and

Q_{e}=8

Dataset	Model	Mean $5\times 2$ cv	Mean 5-fold cv
DTU	GCANet	0.55	0.83 [ 8 ]
DTU	GCANet-NoEn	0.78	0.94
DTU	LSTM-2	0.6	0.84
KUL	GCANet	0.73	0.92 [ 8 ]
KUL	GCANet-NoEn	0.91	92
KUL	LSTM-2	0.86	0.92

TABLE X : Best results for the first test series of the subject-wise evaluation. It includes the maximum mean validation accuracies compared to the original published results

Model	Variant	FLOPs	Weights
GCANet	Baseline	70M	3M
GCANet-NoEn	Scat(8,8)	50M	1.3M
GCANet-NoEn	Scat(16,16)	50M	1.4M
GCANet-NoEn	SSQ-STFT	61M	1.3M
LSTM-X	Baseline	1.8M	7.0K
LSTM-X	Scat(8,8)	4M	237K

TABLE XI : Floating point operations and weights for 1-second inputs across different preprocessing methods. Baseline: one EEG signal (

64\times 128

) and two audio signals (

1\times 128

). GCANet: two EEG signals (

64\times 128

and

64\times 10

) and two audio signals (

1\times 16000

. ST configurations:

F_{o}\in\{8,16\}

Hz with

Q=8

for both audio and EEG.

Wavelet Scattering Transforms

See all 9 papers

FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding
Jun 12, 2026Ziwei Wang, Xingyi He, Tianwang Jia +2Hybrid CNN-Transformer Architectures Neural Decoding
Hubei Key Laboratory of Brain-inspired Intelligent Systems, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China
End-to-End Markov State Sequence Learning for Auditory Attention Decoding
Jul 21, 2026Yushan Yashengjiang, Jie Zhang, Miao Sun +3Markov Models Neural Decoding
NERC-SLIP, University of Science and Technology of China (USTC), Hefei 230027, China · School of Information and Communication Engineering, Guangzhou Maritime University, Guangzhou 510725, China · Artificial Intelligence Research Institute, iFLYTEK Company, Ltd., Hefei 230088, China +1
CA-TCN: A Causal-Anticausal Temporal Convolutional Network for Direct Auditory Attention Decoding
Mar 27, 2026Iñigo García-Ugarte, Rubén Eguinoa, Ricardo San Martín +2Neural Decoding Temporal Convolutional Networks
Department of Science, Universidad Pública de Navarra (UPNA), 31006 Pamplona, Spain · BCBL, Basque Center on Cognition Brain and Language, 20009 San Sebastián, Spain · Department of Statistics, Computer Sciences and Mathematics, Universidad Pública de Navarra (UPNA), 31006 Pamplona, Spain +2

Jun 12, 2026J/K move · Enter open · S save

FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding

Ziwei Wang, Xingyi He, Tianwang Jia, Hongbin Wang+1

Hubei Key Laboratory of Brain-inspired Intelligent Systems, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China

Hybrid CNN-Transformer Architectures Neural Decoding EEG Decoding Frequency-Domain Feature Learning Transformer Attention

Open paper AskPDF arXiv

Auditory attention decoding (AAD) aims to infer the attended speaker from neural responses in multi-speaker acoustic environments and is a key problem for neuro-steered hearing systems. Although recent studies have achieved encouraging progress, existing AAD models still do not fully exploit frequency domain electroencephalography (EEG) information. In particular, most approaches introduce multi-band information through handcrafted feature extraction or direct cross-band feature concatenation, which mainly exploit frequency information at a shallow level and may overlook band-specific patterns and cross-band interactions. To address these limitations, this paper proposes FAConformer, a frequency-aware CNN-Transformer framework for AAD that explicitly integrates band-specific encoding and adaptive cross-band interaction. Specifically, FAConformer first decomposes EEG signals into multiple frequency bands and assigns each band to an independent CNN-Transformer encoder for band-specific modeling. The resulting band-wise features are then adaptively fused by a carefully designed frequency-aware attention (FAA) module that models cross-band dependencies by treating band-wise features as tokens. Further, band-wise auxiliary supervision (BAS) is introduced to prevent weakly contributing branches from being under-optimized during joint training. In this way, FAConformer performs frequency-aware modeling that more effectively exploits frequency domain information. Extensive experiments on two public AAD datasets with three decision-window lengths demonstrated that FAConformer consistently outperformed 12 competitive baselines, surpassing the current state-of-the-art model by 4.9%. Further analyses of band importance, ablation, and parameter sensitivity verify the effectiveness, robustness, and interpretability of the proposed framework. Code is available at https://github.com/wzwvv/FAConformer.

Abstract

Wavelet Scattering Transforms

Explore similar work

FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding

End-to-End Markov State Sequence Learning for Auditory Attention Decoding

CA-TCN: A Causal-Anticausal Temporal Convolutional Network for Direct Auditory Attention Decoding