Selective attention enhances cortical tracking of attended speech, but the role of ignored speech remains less clear. In particular, it is unknown whether attended and ignored speech contribute to cortical tracking in a fixed balance or whether this balance changes with listening conditions. We addressed this question across six electroencephalography (EEG) datasets spanning acoustic and attentional challenges, hearing impairment, and hearing-aid processing. We systematically varied the relative weighting of attended and competing speech and studied how their combined contribution affected cortical speech tracking. Across most listening conditions, cortical tracking was improved when we included competing speech, compared with attended speech alone. Importantly, the balance between attended and competing speech changed across conditions. Increasing acoustic difficulty increased both the benefit and the optimal contribution of competing speech, with the same general pattern observed in normal-hearing and hearing-impaired listeners. In the hearing-aid datasets examining noise-reduction conditions, the contribution of competing speech was more limited. Attentional challenges also changed the contribution of competing speech, but differently from acoustic challenges. These findings show that attended and competing speech do not contribute to cortical tracking in a fixed balance. Rather, their balance depends on the listening challenge. Although these results do not define a single optimal ratio, they provide a step toward hearing devices that adapt not only to the attended talker, but also to how competing speech is balanced in complex listening environments.
Figures & tables
Fig. 1 : Illustration of the experimental paradigms used in datasets I–VI.
I
II
III
IV
V
VI
Participants
18
23
21
31
35
17
Hearing
nh
nh
nh
hi
hi
hi
Hearing aids
No
No
No
Yes
Yes
Yes
Age group
Young–middle
Young–middle
Young–middle
Middle–older
Middle–older
Middle–older
bbn
No
Yes/No
No
Yes
Yes
Yes
Presentation
Insert
Insert/ ff
av ff
ff
ff
ff
TABLE I : Overview of the datasets and experimental conditions. ff : free-field; av : audiovisual; bbn : background babble noise; ha : hearing aid; nr : noise reduction.
Fig. 2 : Illustration of the curve classifications (inverted U-shape, decreasing, and flat), and the three derived components. The y -axis shows the Pearson correlation coefficient ( r ) between the reconstructed and reference speech envelopes, while the x -axis shows the ignored-speech gain factor ( k∈[0,1] ). At k=0 , no speech mixing is applied, corresponding to the attended-only ( Xatt ; dashed blue line) and ignored-only ( Xign ; dashed orange line) reference envelopes. The solid curves show decoding performance obtained from weighted mixtures of the attended and ignored speech envelopes. For inverted U-shaped relationships, Component 1 denotes the change in decoding correlation between k=1 and k=0 , and Components 2–3 quantify the absolute peak improvement, and the optimal gain factor ( kpeak ), respectively.
TABLE II : Curve classification. Significance: p<0.05 . Vertex inside [0.1,0.9] ensures the peak is actually interior, not just at an endpoint.
Fig. 3 : Attended-speech tracking across listening conditions. Mean attended-speech correlation across participants for each condition and dataset. Within each dataset, conditions are ordered from the highest (light green) to the lowest (dark blue) attended-speech correlation, providing the within-dataset ranking of listening challenge used in the subsequent analyses of attended-competing speech weighting.
Fig. 4 : Ignored-speech association in reconstructed speech envelopes. Partial correlations between the reconstructed and ignored speech envelopes were obtained from decoders trained exclusively on the attended speech envelope ( k=0 ), while controlling for the attended speech envelope. Boxplots show the participant-level distributions, with boxes representing the interquartile range, black horizontal lines indicating the median, and whiskers extending to 1.5 times the interquartile range. Red diamonds with whiskers indicate group-level estimates with 95% confidence intervals, calculated on the Fisher- z scale. Group-level estimates and confidence intervals were back-transformed to the correlation scale for visualization. Asterisks indicate statistical significance (* p<0.05 , ** p<0.01 , *** p<0.001 ) after Benjamini-Hochberg false-discovery-rate correction ( Benjamini and Hochberg, 1995 ) .
Fig. 5 : Attended-competing speech weighting across listening conditions. Mean cortical speech-tracking correlation as a function of the competing-speech weight k for each condition and dataset. At k=0 , the weighting contains attended speech alone; at k=1 , attended and competing speech have equal weights. Solid blue lines denote significant inverted-U relationships, solid orange lines denote significant decreasing relationships, and dashed grey lines denote functions that did not meet the significance criteria for either classification. Within each dataset, conditions follow the ordering established from attended-speech tracking in Figure 3 , with lighter to darker shading indicating increasing listening challenge.
Fig. 6 : Results for components 1–3 for each dataset and condition. Within each dataset, the conditions are ordered from the easiest (left) to the hardest (right). Marker shape and colour indicate the corresponding shape classification shown in Figure 5 .
Fig. 7 : Statistical analysis of the difference between the hardest and easiest conditions for components 1–3. To increase the sample size ( n ), the statistical analyses were performed on pooled data for the Datasets I, II, III, and VI. Note: Datasets IV–V (decreasing curve; n=65 ) showed no significant differences for any component ( p>0.05 ). Asterisks indicate statistical significance after Benjamini-Hochberg false-discovery-rate correction ( N=6 , two tests and three components), where * p<0.05 , ** p<0.01 , and *** p<0.001 .
Fig. 8 : Average decoding correlations for all speech-mixture combinations. Rows correspond to datasets and columns to experimental conditions, ordered from the easiest to the hardest according to the results presented in Figure 3 . Blue curves show mixtures of the attended and ignored speech envelopes (also shown in Figure 5 ), orange curves show mixtures of the ignored and attended speech envelopes, and the grey and green curves show the corresponding control mixtures with a random ( null ) speech envelope.
Understanding how the brain selectively follows relevant speech amid competing voices is a central challenge in auditory neuroscience and a key step toward neuro-steered hearing technologies. However, most open-source Electroencephalography (EEG) datasets for Auditory Attention Decoding (AAD) use idealized single-competing-talker paradigms that oversimplify the acoustic, spatial, and semantic structure of everyday communication. To capture this ecological complexity, we introduce the SoundBubble-EEG dataset: a high-density 128-channel EEG resource comprising more than 25 hours of recordings from 30 participants. The paradigm requires listeners to selectively attend to a dynamic target speaker group, a designated "sound bubble", amid competing multi-speaker distractor bubbles across three realistic scenarios: a restaurant, a home TV viewing, and a meeting discussion. By bridging the gap between constrained laboratory protocols and real-world auditory scenes, this dataset enables investigations of multi-talker speech comprehension, neural speech tracking, and cross-scenario generalization. It also provides a benchmark for AAD algorithms under realistic acoustic and semantic variability and may support auditory neuroscience and the development of neuro-steered hearing technologies.
Shu Peng, Rui Liu, Yufei Zhang +7
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong SAR, China · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China
Identifying which speaker a listener is attending to in a noisy room -- the cocktail-party problem -- is the missing ingredient for next-generation hearing aids and brain-computer interfaces: it tells the device whose voice to amplify. Auditory attention decoding (AAD) reads this answer from EEG, but the literature splits into disconnected pieces: directional-AAD classifies side but does not map side to stream; regression-based source-AAD ranks candidate streams by a single Pearson correlation that is intrinsically noisy at the 1-5 s windows real devices need; and envelope reconstruction has no native AAD rule. We argue the right object is not any single statistic but the conditional likelihood of the attended envelope given EEG, and we make this practical with NEUROTOKEN: a single network whose three heads share one EEG front-end, with a conditional flow-matching head (ATTUNEFLOW) that scores candidates by an integrated velocity-residual likelihood ratio. Two inference-time ensembles -- QUADTRACK (four complementary statistics) and ENV-FLOW (z-normalised QUADTRACK+ATTUNEFLOW) -- absorb per-statistic failure modes for free. On KU Leuven, DTU, and NJU at 5 s, ATTUNEFLOW lifts per-segment source-AAD by 9%-16% over the strongest non-generative baseline and shrinks across-subject variance by ~3x; trial-level fusion exceeds 93% on two of three datasets. In parallel reproductions we show that canonical 95-97% direction-AAD numbers collapse by 17%-45% under a strict trial-disjoint protocol, clarifying both the true ceiling and why a likelihood-based formulation is needed.
Ali Alavi, Donald S. Williamson
Department of Computer Science and Engineering Ohio State University Columbus, OH, 43210
Humans rely on gaze, head movements, and visual cues to attend to speakers in noisy environments, yet auditory attention decoding (AAD) has been studied primarily using electroencephalography (EEG). We introduce the Multimodal Auditory-attention Egocentric Speech-TRacking Open (MAESTRO) corpus, the first AAD dataset to simultaneously record EEG, eye gaze, pupillometry, egocentric video, and head inertial measurement unit (IMU) data. MAESTRO includes four competing speakers and background noise across multiple signal-to-noise ratio (SNR) conditions, enabling attention decoding under realistic listening scenarios. Through a four-speaker attention decoding benchmark, we show that combining behavioral and physiological signals improves decoding performance over EEG-only approaches, enabling future advances in multimodal auditory attention decoding. These findings open the door to new applications, analyses, and methodological advances in multimodal AAD. The complete dataset is publicly available at https://huggingface.co/datasets/aspire-osu/maestro-eeg-dataset . The official code repository is available at https://github.com/ASPIRE-OSU/MAESTRO .
K M Naimul Hassan, Ali Alavi, Donald S. Williamson
Department of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210 USA · Center for Cognitive and Brain Sciences, The Ohio State University