Merit: Making every modality earn its contribution in multimodal attention decoding

Ali Alavi; Donald S. Williamson

cs.LGOct 13, 2026

Merit: Making every modality earn its contribution in multimodal attention decoding

Authors: Ali Alavi, Donald S. Williamson

Organizations: Department of Computer Science and Engineering The Ohio State University Columbus, OH 43210, USA

Abstract

More than 1.5 billion people have hearing loss. When several people talk at once, hearing aids amplify every voice alike. Auditory attention decoding (AAD) aims to solve this by reading a listener's brain signals to find the attended talker, letting the device amplify only them. A real device will not see the brain alone; smart glasses and earbuds also track eyes, head, and scene. These behavioral signals help, as people often look at who they listen to. However, they show where a listener looks, not whom they hear. A decoder relying heavily on them fails when a listener cannot move their head or listens without looking - exactly when a brain-based device is needed. A multimodal decoder must use these signals without letting them replace the electroencephalogram (EEG). Accuracy alone cannot show whether it does. We present MERIT, a decoder and evaluation method asking: how much accuracy does each input earn? MERIT shuffles each input stream across test examples, measures the accuracy drop, and splits the result across streams using Shapley values. It also blocks two shortcuts: candidate audio is checked before training so it cannot reveal the answer, and the scoring function cannot beat chance if it ignores the brain. On a public dataset with EEG, eye tracking, head motion, and scene video, MERIT improves on the published benchmark, reaching 0.82 four-way accuracy within listeners and 0.74 on unseen listeners (chance is 0.25). The measurement reveals what accuracy hides: adding behavioral streams raises overall accuracy but almost halves the brain's share of it. On two other datasets, the same tests find a label imbalance in a widely used benchmark and flag results that accuracy alone would report as brain decoding.

Figures & tables

Panel of Figure 1 — Figure 1: Merit . The coupling branch compares the EEG with each candidate using a correlation over time with the mean removed. A brain encoder that outputs the same vector at every step therefore gives every candidate a score of zero, which is chance. The orientation branch gets no audio , so it cannot use audio cues. The model encodes once and scores separately, so a trained model can be re-scored under any shuffle of any stream without encoding again.

				Shapley credit $\phi_{m}$
Model	Accuracy	Null	Contrib.	EEG	gaze	head	flow	fovea
EEG alone	0.6915	0.2944	+0.3971	+0.3971	—	—	—	—
Merit (no hinge, no dropout)	0.8173	0.2733	+0.5439	+0.2217	+0.0194	+0.0039	+0.0123	+0.2866
No hinge, with dropout	0.8091	0.2793	+0.5298	+0.2229	+0.0206	+0.0079	+0.0128	+0.2656
With hinge, no dropout	0.8242	0.2685	+0.5557	+0.2169	+0.0340	+0.0074	+0.0295	+0.2679
With hinge, with dropout	0.7884	0.2774	+0.5111	+0.1600	+0.0362	+0.0144	+0.0247	+0.2758

Table 1: More streams give more accuracy, and less credit to the brain. Within-listener, matched , chance

0.25

. The four fused models share streams, data, folds, architecture and listener adaptation; they differ only in the contribution hinge and in modality dropout.

\phi_{m}

is each stream’s Shapley credit and the five add up to the contribution. The EEG-only model shows how much credit the brain earns on its own. Appendix J gives the same four models on unseen listeners.

	Accuracy	Null	Contribution
Random crops, one decision per window	0.4956	0.2982	+0.1974
+ one decision per trial	0.6676	0.3810	+0.2866
+ only delays after the sound	0.6502	0.3231	+0.3271
Merit , EEG only (+ per-listener correction)	0.6915	0.2944	+0.3971
control: only delays before the sound	0.6145	0.4017	+0.2127

Table 2: Improvement ladder , EEG only, within-listener, matched , chance

0.25

. Each row adds one change to the row above. The last row is a control, not a step: it limits the coupling to delays before the sound, where no brain response to that sound can be.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Construction		$K$	Probe	Excess over chance
Competing talkers	unmatched	4	0.5581	+0.3081
+ distribution matching	matched	4	0.2603	+0.0103
Same-talker temporal negatives	same-talker	3	0.3758	+0.0425
+ distribution matching	same-talker matched	3	0.3564	+0.0231
Same-talker temporal negatives	same-talker	2	0.5131	+0.0131
+ distribution matching	same-talker matched	2	0.5134	+0.0134

Appendix

Table 3: Audio-only acceptance probe : what a linear reader takes from the candidates alone, with no access to any recording. A shortcut-free construction must sit at chance.

Figure 2: Shapley credit per stream , stacked, for the four fused rows of Table 1 . Each bar’s total height is that model’s contribution

v(M)-v(\emptyset)

, decomposed exactly. The question the figure answers is not how tall the bars are but how much of each one is blue.

Corpus	Gaze	Head	Scene video	Presentation	Listener
DTU ( Fuglsang et al., 2018 )	6 EOG	—	—	simulated rooms, earphones	soundproof booth
Four-speaker ( Yan et al., 2024b )	—	—	—	4 loudspeakers, anechoic	head fixed by chinrest
AV-GC-AAD ( Geirnaert et al., 2024 )	4 EOG	—	—	earphones, simulated HRTFs	gaze controlled by design
MAESTRO ( Hassan et al., 2026 )	eye tracker	IMU	egocentric	6 loudspeakers, real room	unconstrained

Appendix

Table 4: Public AAD corpora. “EOG” denotes a few electro-oculography electrodes recorded for artifact rejection, which is not calibrated gaze and carries no head or scene information. The last column is the one that decides whether modality freeloading is observable at all.

Corpus	Listeners	Windows	Probe	Excess	Attended-role prior
KU Leuven	16	9,564	0.6947	+0.1947	0.800 track 1
DTU	18	9,720	0.5008	+0.0008	0.494 male

Appendix

Table 5: The probe transfers, and it is calibrated. Audio-only two-way accuracy on

10

s windows, content-disjoint folds, no EEG. The last column is the corpus’s prior on which talker is attended; the probe tracks it.

Corpus	$2$ s	$5$ s	$10$ s	$30$ s
KU Leuven, as released	+0.0945	+0.1384	+0.1947	+0.0729
distribution-matched	+0.0321	+0.0401	+0.0974	+0.0208
DTU, as released	+0.0010	+0.0001	+0.0008	-0.0019

Appendix

Table 6: The same probe at four decision windows , excess over chance. On DTU, where the attended-role prior is balanced, no cell exceeds

0.005

. The KU Leuven excess is present at every window; its

30

s cell rests on 4,608 windows drawn from eight stimulus pairs and is the least powered of the four.

	Accuracy	Contribution	Causal	Acausal	Folds+
KU Leuven, within, competing	0.7252	+0.2141	+0.2104	+0.2074	5/5
KU Leuven, within, same talker	0.8149	+0.3150	+0.3115	+0.3080	5/5
KU Leuven, LOSO, competing	0.7138	+0.1766	+0.1764	+0.1687	16/16
DTU, within, competing	0.7590	+0.2584	+0.2560	+0.2499	5/5
DTU, within, same talker	0.8170	+0.3175	+0.3164	+0.3145	5/5
DTU, LOSO, competing	0.7269	+0.2264	+0.2006	+0.2060	18/18

Appendix

Table 7: What the full battery says about two public datasets. Accuracy and contribution for the unconstrained coupling branch, then contribution with the branch confined to the causal (

+125\dots+1109

ms) and mirrored acausal (

-1109\dots-125

ms) lag bands. Folds+ is for the unconstrained branch. The last row is MAESTRO’s same-talker

K=2

result, the reference for what a directional response looks like.

		KU Leuven		DTU
Method	Decodes	own split	no leakage	no leakage
DARNet ( Yan et al., 2024a )	direction	$0.895$	$0.552$	—
ListenNet ( Fan et al., 2025 )	direction	$0.930$	$0.484$	—
DBPNet ( Ni et al., 2024 )	direction	$0.884$	$0.483$	—
DenseNet-3D ( Xu et al., 2024 )	direction	$0.864$	$0.698$	—
SWIM ( Zhang et al., 2025 )	direction	$0.644$	$0.353$	—

Appendix

Table 8: Two-talker accuracy at

5

s: with each paper’s own split, and without leakage. The first five rows decode the direction of attention from EEG alone, and are NeuroToken’s re-runs on KU Leuven listeners

1

–

4

( Alavi & Williamson, 2026 ) ; the headline numbers in those papers are higher still,

95

–

97\%

1

s. NeuroToken and Merit decode the attended talker from EEG and the candidate speech. NeuroToken holds out whole trials; Merit holds out whole stimuli and uses matched candidates. “Unseen” means listeners not seen in training.

Arm	Accuracy	$\phi_{\text{EEG}}$	Accuracy	$\phi_{\text{EEG}}$	Folds+
	Within-subject		Leave-one-listener-out
Hinge, modality dropout	0.7884	+0.1600	0.7469	+0.1149	16/16
No hinge, modality dropout	0.8091	+0.2229	0.7625	+0.1560	16/16
Merit (no hinge, no dropout)	0.8173	+0.2217	0.7438	+0.1762	16/16
Hinge, no dropout	0.8242	+0.2169	0.7219	+0.1479	16/16

Appendix

Table 9: The

2\times 2

of Table 1 , both protocols.

\phi_{\text{EEG}}

is the EEG’s Shapley credit; Folds+ counts leave-one-listener-out folds with a positive contribution. No arm is best in every column.

Streams	Accuracy	Null	Contribution	$p$	Folds+
EEG	0.4788	0.2882	+0.1906	0.0476	5/5
EEG + gaze	0.5295	0.2610	+0.2685	0.0476	5/5
EEG + head	0.4995	0.2589	+0.2405	0.0476	5/5
EEG + flow	0.5325	0.2854	+0.2471	0.0476	5/5
EEG + fovea	0.6909	0.2550	+0.4358	0.0476	5/5
Behaviour only	0.6552	0.2541	+0.4010	0.0476	5/5

Appendix

Table 10: Accuracy, null and contribution by stream set , within-subject (five content-disjoint folds), matched ,

W=10

s, chance

0.25

. Folds+ counts folds with a positive contribution. The leave-one-listener-out counterpart is Table 11 .

Streams	Accuracy	Null	Contribution	$p$	Folds+
EEG	0.4831	0.3130	+0.1701	0.0863	15/16
EEG + gaze	0.4844	0.2760	+0.2084	0.0506	16/16
EEG + head	0.4944	0.2782	+0.2162	0.0536	16/16
EEG + flow	0.5013	0.2767	+0.2245	0.0506	16/16
EEG + fovea	0.6700	0.2649	+0.4051	0.0476	16/16
Behaviour only	0.6300	0.2677	+0.3623	0.0476	16/16

Appendix

Table 11: Leave-one-listener-out , sixteen folds, each with a held-out listener on held-out content. matched ,

W=10

s, chance

0.25

Candidates	$K$	Streams	EEG visible at	Chance	Accuracy	Null	Contribution
same-talker matched	3	EEG	centred	0.333	0.5474	0.3318	+0.2156
same-talker matched	3	all five	centred	0.333	0.5578	0.3321	+0.2257
same-talker matched	2	EEG	centred	0.500	0.6981	0.4995	+0.1986
same-talker matched	2	all five	centred	0.500	0.6857	0.4988	+0.1869
matched	4	EEG	$+125\dots+1109$ ms	0.250	0.4990	0.2750	+0.2240
matched	4	EEG	$-1109\dots-125$ ms	0.250	0.4171	0.3204	+0.0967

Appendix

Table 12: Orienting-free decoding and neural directionality. EEG only unless stated, within-subject,

W=10

s. Same-talker candidates are the attended talker at different times, so slot index carries no spatial meaning and no orienting cue can help. The lag band is set by the encoder’s direction plus an eight-sample shift; a contribution as large in the acausal band would indicate a shared artifact rather than envelope tracking.

Figure 3: Contribution against decision-window length , within-listener, matched . Left: contribution for the four stream sets measured at every window; the eleven stream sets at

2

and

5

s are in Table 13 . Right: the full model’s shuffled accuracy, which is what the contribution subtracts, and the linear audio probe, both measured at each window. The model’s shuffled accuracy stays within

0.015

of chance throughout; the probe varies more, between

0.23

and

0.30

	$W=2$ s				$W=5$ s
Streams	Acc.	Null	Contrib.	$p$	Acc.	Null	Contrib.	$p$
EEG	0.3281	0.3018	+0.0263	0.0647	0.3954	0.3101	+0.0853	0.0418
gaze	0.3448	0.2304	+0.1144	0.0050	0.3349	0.2236	+0.1113	0.0050
head motion	0.3541	0.2389	+0.1151	0.0050	0.3800	0.2456	+0.1345	0.0050
optical flow	0.4109	0.2477	+0.1632	0.0050	0.4126	0.2458	+0.1668	0.0050
fovea (V-JEPA 2)	0.7130	0.2560	+0.4570	0.0050	0.7096	0.2566	+0.4530	0.0050

Appendix

Table 13: Every stream set at the two shortest decision windows , within-subject (content-disjoint, five folds), matched ,

K=4

, chance

0.25

, hop

=W/2

. Contribution is accuracy minus the model’s own permutation null;

p

is the permutation

p

-value, floored at

0.048

for twenty permutations. The first block is single streams, the second EEG paired with each behavioural stream.

				Shapley credit $\phi_{m}$
Model	Accuracy	Null	Contrib.	EEG	gaze	head	flow	fovea
EEG alone (reference)	0.4788	0.2882	+0.1906	+0.1906	—	—	—	—
No hinge, accuracy-selected	0.7154	0.2600	+0.4554	+0.1039	+0.0296	+0.0087	+0.0178	+0.2954
No hinge, credit-selected	0.7250	0.2639	+0.4611	+0.1182	+0.0307	+0.0109	+0.0197	+0.2816
Hinge, accuracy-selected	0.7087	0.2602	+0.4486	+0.0882	+0.0442	+0.0156	+0.0270	+0.2736
Hinge, credit-selected	0.7088	0.2614	+0.4473	+0.0931	+0.0446	+0.0186	+0.0258	+0.2653

Appendix

Table 14: Hinge, selection rule and modality dropout at

W=10

s , per-window decisions — the earlier operating point, kept because it varies the selection rule, which Table 1 does not. Identical streams, data and folds; within-subject, matched ,

W=10

s, chance

0.25

\phi_{m}

is Shapley credit and sums to the contribution by construction. The single-stream EEG model is shown for reference.

Item	Value
Sampling rate / window / hop	$64$ Hz / $10$ s / $5$ s for per-window results (sweep: $2,5,10,15,20,30$ s at $50\%$ hop)
Per-trial decisions ( Merit )	$10$ s crops of $30$ s trials: $8$ random crops per trial when training, $5$ fixed crops at test; window scores summed per trial
Stream channels	EEG $32$ ( $64$ on KU Leuven and DTU), gaze $6$ , head $6$ , flow $4$ , fovea $1024$ (frozen)
Descriptor width $D$ / layers	$16$ / $5$ (dilations $1,2,4,8,16$ ; RF $63$ samples $=0.98$ s)
Spatial filters (EEG)	$8$
Normalisation / activation	GroupNorm( $4$ groups) after every conv; no activation on the last

Appendix

Table 15: Settings. None were tuned; each was set once from first principles.

Neural Decoding

Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.

See all 182 papers

Sep 25, 2026

MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

Humans rely on gaze, head movements, and visual cues to attend to speakers in noisy environments, yet auditory attention decoding (AAD) has been studied primarily using electroencephalography (EEG). We introduce the Multimodal Auditory-attention Egocentric Speech-TRacking Open (MAESTRO) corpus, the first AAD dataset to simultaneously record EEG, eye gaze, pupillometry, egocentric video, and head inertial measurement unit (IMU) data. MAESTRO includes four competing speakers and background noise across multiple signal-to-noise ratio (SNR) conditions, enabling attention decoding under realistic listening scenarios. Through a four-speaker attention decoding benchmark, we show that combining behavioral and physiological signals improves decoding performance over EEG-only approaches, enabling future advances in multimodal auditory attention decoding. These findings open the door to new applications, analyses, and methodological advances in multimodal AAD. The complete dataset is publicly available at https://huggingface.co/datasets/aspire-osu/maestro-eeg-dataset . The official code repository is available at https://github.com/ASPIRE-OSU/MAESTRO .

K M Naimul Hassan, Ali Alavi, Donald S. Williamson

Department of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210 USA · Center for Cognitive and Brain Sciences, The Ohio State University

EEG Decoding Neural Decoding Multimodal Sensor Fusion

Ask PDF

Sep 10, 2026

RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection

Auditory attention decoding (AAD) identifies the attended speaker from physiological signals, supporting neuro-steered hearing devices and natural human-machine interaction. Electroencephalography (EEG) is the dominant modality for AAD but provides incomplete evidence in naturalistic audio-visual scenes, motivating EEG and electrooculography (EOG) fusion. Existing approaches remain limited by weak cross-modal interaction, inefficient temporal modeling, and low robustness to sample variations. To address the limitations, we propose RAMamba-Net, a reliability-aware Mamba-based multimodal fusion network for AAD. RAMamba-Net employs a Mamba-enhanced band-aware convolutional Transformer to capture band-specific EEG patterns and long-range temporal dynamics. A dual-branch temporal-spatial encoder models EOG temporal and inter-channel dependencies. Cross-modal attention enables explicit modality interaction. Then, a reliability-aware module is introduced to estimate sample-wise modality weights for feature and prediction consistency, thereby enhancing multimodal fusion. Experiments on two AAD benchmarks demonstrate that RAMamba-Net effectively exploits complementary EEG-EOG information, yielding accuracy gains of 5.76% over unimodal baselines, together with more robust decoding and discriminative representations. Further analyses show that explicit cross-modal interaction improves multimodal alignment, while the reliability-aware module suppresses unreliable modality evidence and is robust to signal perturbation and parameter variation.

Xingyi He, Ziwei Wang, Dongrui Wu

School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, China

Neural Decoding EEG Decoding Multimodal Fusion Reliability-Aware Multimodal Fusion

Ask PDF

Sep 30, 2026

NEUROTOKEN: Joint Source and Directional AAD with Envelope Decoding via Conditional Flow Matching

Identifying which speaker a listener is attending to in a noisy room -- the cocktail-party problem -- is the missing ingredient for next-generation hearing aids and brain-computer interfaces: it tells the device whose voice to amplify. Auditory attention decoding (AAD) reads this answer from EEG, but the literature splits into disconnected pieces: directional-AAD classifies side but does not map side to stream; regression-based source-AAD ranks candidate streams by a single Pearson correlation that is intrinsically noisy at the 1-5 s windows real devices need; and envelope reconstruction has no native AAD rule. We argue the right object is not any single statistic but the conditional likelihood of the attended envelope given EEG, and we make this practical with NEUROTOKEN: a single network whose three heads share one EEG front-end, with a conditional flow-matching head (ATTUNEFLOW) that scores candidates by an integrated velocity-residual likelihood ratio. Two inference-time ensembles -- QUADTRACK (four complementary statistics) and ENV-FLOW (z-normalised QUADTRACK+ATTUNEFLOW) -- absorb per-statistic failure modes for free. On KU Leuven, DTU, and NJU at 5 s, ATTUNEFLOW lifts per-segment source-AAD by 9%-16% over the strongest non-generative baseline and shrinks across-subject variance by ~3x; trial-level fusion exceeds 93% on two of three datasets. In parallel reproductions we show that canonical 95-97% direction-AAD numbers collapse by 17%-45% under a strict trial-disjoint protocol, clarifying both the true ceiling and why a likelihood-based formulation is needed.

Ali Alavi, Donald S. Williamson

Department of Computer Science and Engineering Ohio State University Columbus, OH, 43210

Neural Decoding EEG Decoding Speech Processing

Ask PDF

Abstract

Neural Decoding

Explore similar work

MAESTRO: a Multimodal Auditory-attention Egocentric Speech-TRacking Open corpus

RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection

NEUROTOKEN: Joint Source and Directional AAD with Envelope Decoding via Conditional Flow Matching