MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

K M Naimul Hassan; Ali Alavi; Donald S. Williamson

eess.ASSep 25, 2026

MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

Authors: K M Naimul Hassan, Ali Alavi, Donald S. Williamson

Organizations: Department of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210 USA · Center for Cognitive and Brain Sciences, The Ohio State University

Abstract

Humans rely on gaze, head movements, and visual cues to attend to speakers in noisy environments, yet auditory attention decoding (AAD) has been studied primarily using electroencephalography (EEG). We introduce the Multimodal Auditory-attention Egocentric Speech-TRacking Open (MAESTRO) corpus, the first AAD dataset to simultaneously record EEG, eye gaze, pupillometry, egocentric video, and head inertial measurement unit (IMU) data. MAESTRO includes four competing speakers and background noise across multiple signal-to-noise ratio (SNR) conditions, enabling attention decoding under realistic listening scenarios. Through a four-speaker attention decoding benchmark, we show that combining behavioral and physiological signals improves decoding performance over EEG-only approaches, enabling future advances in multimodal auditory attention decoding. These findings open the door to new applications, analyses, and methodological advances in multimodal AAD. The complete dataset is publicly available at https://huggingface.co/datasets/aspire-osu/maestro-eeg-dataset . The official code repository is available at https://github.com/ASPIRE-OSU/MAESTRO .

Figures & tables

Dataset	Subjects	EEG Ch.	Speech Sources	Noise Sources	Language	SNRs	Adtl. modalities	Duration (hours)	Azimuth	Room reverberation
KUL [ 3 ]	16	64	2	×	Dutch	0 dB	×	19.2	$\mathbf{\pm 90^{\circ}}$	×
DTU [ 4 ]	18	64	2	×	Danish	0 dB	×	19	$\mathbf{\pm 60^{\circ}}$	Simulated
NJU [ 8 ]	21	32	2	×	Mandarin	0 dB	×	22.4	$\mathbf{\pm\{15^{\circ},30^{\circ},45^{\circ}\}}$	Low
									$\mathbf{\pm\{60^{\circ},90^{\circ},120^{\circ},135^{\circ}\}}$
ESAA [ 11 ]	20	64	2	×	Mandarin	0 dB	×	12.7	$\mathbf{\pm 90^{\circ}}$	×
ASA [ 9 ]	20	64	2	×	Mandarin	0 dB	×	8	$\mathbf{\pm\{5^{\circ},30^{\circ}\}}$	×

TABLE I: Comparison of notable AAD datasets. Speech Sources denotes the number of competing speakers, while Noise Sources denotes dedicated background-noise sources separate from the speech streams. Adtl. Modalities refers to data streams recorded alongside EEG, and Duration reports the total recording time (hours). Azimuth indicates loudspeaker positions relative to the listener’s frontal axis. Room Reverberation describes the recording environment: Low indicates minimal reverberation, Simulated denotes artificially added reverberation, and Natural denotes real-world acoustic reflections.

Fig. 1: The spatial arrangement of loudspeakers around the participant. Four loudspeakers ( $S_{1}$ - $S_{4}$ ) were positioned relative to the participant’s frontal axis, while two noise speakers ( $N_{1}$ and $N_{2}$ ) were placed behind the participant. The participant wore an EEG headset and eye-tracking glasses.

Fig. 2: Sample gaze recording from Subject S02, Trial eval_022 . Panels (a)-(f) show six evenly spaced video frames with gaze points from a $\pm$ 1-second window overlaid as colored dots, whose border colors match the shaded regions in (g) and (h). Panel (g) shows horizontal (X) and vertical (Y) gaze coordinates over the trial, illustrating smooth pursuit, saccadic transitions, and occasional tracking loss; (h) shows left and right pupil diameters, bilaterally symmetric with transient dilation events.

Panel of Figure 4 — Fig. 3: Behavioral performance across participants based on comprehension-question responses. (a) Per-subject percentages of correct (blue), incorrect (amber), and “I could not pay attention” (gray) responses; the dashed line indicates the mean accuracy. (b) Per-subject accuracy during the first and second experimental sessions, showing improved performance for most participants in the second session.

Fig. 4: Mean gaze dispersion (a), dynamic acceleration RMS (b), and gyroscope magnitude RMS (c) across six equal time bins within a trial. Gaze dispersion peaks at trial onset and offset and is stable in between, indicating an initial orienting response followed by sustained attention. Head movement peaks at onset and stabilizes from Bin 3 onward with no rise at offset, indicating a postural adjustment followed by stillness during listening.

Fig. 5: Baseline network for the benchmark task. Each modality is encoded by a dilated convolutional encoder (kernel size 3, dilation rates $2^{0},\ldots,2^{4}$ , receptive field 0.98 s) into a 16-dimensional embedding; the $1\times 1$ spatial convolution in the encoder detail is applied to EEG only. The EEG embedding is compared with each speaker envelope by time-centered correlation, and every embedding is also pooled over time and classified over the four speaker positions with no audio input. The two scores are added, and the attended speaker is the one whose envelope occupies the highest-scoring slot.

	Within-subject					LOSO
Mode	5s	10s	15s	20s	30s	5s	10s	15s	20s	30s
EEG	43.63 $\pm$ 1.76	47.57 $\pm$ 4.08	53.17 $\pm$ 1.66	53.62 $\pm$ 2.67	59.06 $\pm$ 2.04	43.64 $\pm$ 3.18	50.19 $\pm$ 6.98	52.40 $\pm$ 7.02	55.31 $\pm$ 9.92	61.88 $\pm$ 11.71
Gaze	37.13 $\pm$ 1.16	37.58 $\pm$ 1.75	39.98 $\pm$ 2.10	37.95 $\pm$ 2.38	36.04 $\pm$ 2.79	33.99 $\pm$ 8.05	34.95 $\pm$ 9.38	36.08 $\pm$ 9.09	36.87 $\pm$ 8.98	35.46 $\pm$ 9.44
IMU	38.21 $\pm$ 2.80	38.75 $\pm$ 2.13	38.84 $\pm$ 2.44	34.85 $\pm$ 4.16	35.41 $\pm$ 4.14	37.07 $\pm$ 10.86	34.77 $\pm$ 9.46	38.48 $\pm$ 13.53	32.16 $\pm$ 11.29	35.46 $\pm$ 12.69
Video	43.71 $\pm$ 4.20	42.59 $\pm$ 4.90	44.33 $\pm$ 3.86	43.16 $\pm$ 2.44	43.37 $\pm$ 6.63	39.57 $\pm$ 12.71	43.56 $\pm$ 14.66	41.35 $\pm$ 13.65	41.56 $\pm$ 17.61	44.69 $\pm$ 14.52
EEG+Gaze	51.24 $\pm$ 3.89 ∗	55.27 $\pm$ 1.79 ∗	57.97 $\pm$ 1.61 ∗	59.73 $\pm$ 3.41 ∗	65.15 $\pm$ 3.83	50.54 $\pm$ 7.52 ∗	52.14 $\pm$ 10.16	55.90 $\pm$ 8.64	61.00 $\pm$ 10.49 ∗	62.45 $\pm$ 16.23

TABLE II: Attended-speaker accuracy (mean

\pm

std across folds, %) for the within-subject and leave-one-subject-out (LOSO) splits, across all 15 modality combinations and 5 window sizes. Bold marks the best-performing mode within each window-size column, computed separately per split. ∗ marks a configuration that significantly outperforms EEG alone (

p<0.05

, two-tailed paired

t

-test).

Fig. 6: Per-subject and per-window comparison under LOSO evaluation of EEG alone, the best of the seven multimodal combinations containing EEG, and the best of all eleven; the second set is a subset of the third, so the two often coincide. Each subject is shown as five stacked bars, one per window size (5, 10, 15, 20, 30 s). Gray marks the lowest of the three accuracies, and the segments above it are colored by which configuration attained each: red (EEG alone), orange (best with EEG), blue (best overall, no EEG), purple (best overall, contains EEG).

Fig. 7: Attended-speaker accuracy versus SNR bin under the LOSO protocol, averaged across all five window sizes. Lines show EEG alone, and the best-performing multimodal configuration overall, alongside comprehension-question accuracy averaged across subjects per SNR bin.

Fig. 8: Contribution of the EEG-containing configurations under LOSO evaluation, defined as accuracy minus the accuracy the same model reaches when the listener’s signals are permuted across test windows. Curves show EEG alone, the best EEG-containing pair and triple, and the full four-modality configuration; the pair and triple are selected by contribution at each window size, so their identity varies across the sweep.

EEG Decoding

Electroencephalography (EEG) decoding maps brain-signal patterns to mental states, stimuli, speech, or intended actions.

See all 205 papers

A multi-scenario EEG dataset for auditory attention decoding in naturalistic multi-talker environments
Oct 7, 2026Shu Peng, Rui Liu, Yufei Zhang +7EEG Decoding Speech Processing
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China
Merit: Making every modality earn its contribution in multimodal attention decoding
Oct 13, 2026Ali Alavi, Donald S. WilliamsonNeural Decoding EEG Decoding
Department of Computer Science and Engineering The Ohio State University Columbus, OH 43210, USA
AFA-Net: A Differential Attention Approach for Auditory Attention Detection
Sep 25, 2026Philip H. Lee, Shreeram Suresh Chandra, Karan Thakkar +1Differential Attention EEG Decoding
Independent Researcher, USA · Center for Language and Speech Processing (CLSP), Johns Hopkins University, USA · Laboratory for Computational Audio Perception (LCAP), Johns Hopkins University, USA +1

Oct 7, 2026J/K move · Enter open · S save

A multi-scenario EEG dataset for auditory attention decoding in naturalistic multi-talker environments

Shu Peng, Rui Liu, Yufei Zhang, Wenlong You+6

Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China

EEG Decoding Speech Processing Neural Decoding

Open paper AskPDF arXiv

Understanding how the brain selectively follows relevant speech amid competing voices is a central challenge in auditory neuroscience and a key step toward neuro-steered hearing technologies. However, most open-source Electroencephalography (EEG) datasets for Auditory Attention Decoding (AAD) use idealized single-competing-talker paradigms that oversimplify the acoustic, spatial, and semantic structure of everyday communication. To capture this ecological complexity, we introduce the SoundBubble-EEG dataset: a high-density 128-channel EEG resource comprising more than 25 hours of recordings from 30 participants. The paradigm requires listeners to selectively attend to a dynamic target speaker group, a designated "sound bubble", amid competing multi-speaker distractor bubbles across three realistic scenarios: a restaurant, a home TV viewing, and a meeting discussion. By bridging the gap between constrained laboratory protocols and real-world auditory scenes, this dataset enables investigations of multi-talker speech comprehension, neural speech tracking, and cross-scenario generalization. It also provides a benchmark for AAD algorithms under realistic acoustic and semantic variability and may support auditory neuroscience and the development of neuro-steered hearing technologies.

Abstract

EEG Decoding

Explore similar work

A multi-scenario EEG dataset for auditory attention decoding in naturalistic multi-talker environments

Merit: Making every modality earn its contribution in multimodal attention decoding

AFA-Net: A Differential Attention Approach for Auditory Attention Detection