NEUROTOKEN: Joint Source and Directional AAD with Envelope Decoding via Conditional Flow Matching

Ali Alavi; Donald S. Williamson

cs.LGSep 30, 2026

NEUROTOKEN: Joint Source and Directional AAD with Envelope Decoding via Conditional Flow Matching

Authors: Ali Alavi, Donald S. Williamson

Organizations: Department of Computer Science and Engineering Ohio State University Columbus, OH, 43210

Abstract

Identifying which speaker a listener is attending to in a noisy room -- the cocktail-party problem -- is the missing ingredient for next-generation hearing aids and brain-computer interfaces: it tells the device whose voice to amplify. Auditory attention decoding (AAD) reads this answer from EEG, but the literature splits into disconnected pieces: directional-AAD classifies side but does not map side to stream; regression-based source-AAD ranks candidate streams by a single Pearson correlation that is intrinsically noisy at the 1-5 s windows real devices need; and envelope reconstruction has no native AAD rule. We argue the right object is not any single statistic but the conditional likelihood of the attended envelope given EEG, and we make this practical with NEUROTOKEN: a single network whose three heads share one EEG front-end, with a conditional flow-matching head (ATTUNEFLOW) that scores candidates by an integrated velocity-residual likelihood ratio. Two inference-time ensembles -- QUADTRACK (four complementary statistics) and ENV-FLOW (z-normalised QUADTRACK+ATTUNEFLOW) -- absorb per-statistic failure modes for free. On KU Leuven, DTU, and NJU at 5 s, ATTUNEFLOW lifts per-segment source-AAD by 9%-16% over the strongest non-generative baseline and shrinks across-subject variance by ~3x; trial-level fusion exceeds 93% on two of three datasets. In parallel reproductions we show that canonical 95-97% direction-AAD numbers collapse by 17%-45% under a strict trial-disjoint protocol, clarifying both the true ceiling and why a likelihood-based formulation is needed.

Figures & tables

Panel of Figure 1 — Figure 1 : NeuroToken architecture. Two EEG front-ends feed three heads. The direction-AAD spatial head reads raw EEG via a closed-form CSP $+$ band-power pipeline (no gradient shared with the front-ends); the envelope head reconstructs the attended speech envelope and the AttuneFlow head scores candidate envelopes via an integrated likelihood ratio — both backpropagate through the front-ends and so share learned features. A subject adapter rescales features and predicts a per-listener cortical-response lag. Stage 1 trains everything inside the dashed box; Stage 2 freezes it and trains AttuneFlow only. Layer-by-layer specification in App. G .

Symbol	Loss	Role	$w$ S1	$w$ S2
$\mathcal{L}_{\mathrm{sp}}$	Spatial CE	Direction logits $h_{s}(\mathbf{x})\!\to\!\{L,R\}$ , label-smooth 0.05	$2.0$	–
$\mathcal{L}_{\mathrm{warm}}$	Envelope warm-up Pearson ( 3.4 )	Maximise $\rho(\hat{\mathbf{e}},\tilde{\mathbf{e}}_{a})$	$3.0$	–
$\mathcal{L}_{\mathrm{margin}}^{\rho}$	Envelope $r$ -margin ( 3.4 )	Force $\rho(\hat{\mathbf{e}},\tilde{\mathbf{e}}_{a})>\rho(\hat{\mathbf{e}},\tilde{\mathbf{e}}_{b})$	$8.0$	–
$\mathcal{L}_{\mathrm{NCE}}^{\rho}$	Cross-batch envelope InfoNCE ( 5 )	Contrast attendeds across batch + same-trial unatt. negative	$4.0$	–
$\mathcal{L}_{\mathrm{spec}}$	Multi-resolution STFT magnitude	L1 magnitude of $\hat{\mathbf{e}}$ vs $\tilde{\mathbf{e}}_{a}$ at $\{8,16,32\}$ frames	$0.5$	–
$\mathcal{L}_{\mathrm{cons}}$	Spatial–envelope consistency	Encourage $h_{s}$ side $\equiv$ side of stronger $\rho$ (App. B.6 )	$0.1$	–

Table 1: Loss terms used in each training stage of NeuroToken , with the equation defining each term and the linear weight applied when summing into the total stage objective. “–” means the term is not part of that stage. Stage 1 unfreezes

\{f_{\theta},g_{\psi},h_{s},h_{e},\text{subject adapter}\}

; Stage 2 freezes all of those and unfreezes only

\{v_{\phi},p_{\phi}\}

Dataset	Win	env- $r$	env-multi	attune-flow	env-multi-flow	env-trial	env-trial-fused	spatial	spatial-trial
KU Leuven (15)	1s	$50.4_{\pm 0.9}$	$53.0_{\pm 1.2}$	$54.9_{\pm 1.5}$	$54.3_{\pm 1.2}$	$80.3_{\pm 10.4}$	$78.3_{\pm 9.4}$	$94.4_{\pm 2.6}$	$99.7_{\pm 1.3}$
KU Leuven (16)	5s	$59.5_{\pm 3.4}$	$76.1_{\pm 9.5}$	$86.4_{\pm 2.6}$	$85.5_{\pm 4.8}$	$96.2_{\pm 5.6}$	$99.1_{\pm 2.7}$	$90.4_{\pm 11.6}$	$95.0_{\pm 10.8}$
DTU (18)	1s	$51.0_{\pm 1.5}$	$53.0_{\pm 1.8}$	$53.7_{\pm 1.3}$	$53.7_{\pm 2.0}$	$68.4_{\pm 7.3}$	$58.8_{\pm 6.6}$	$71.5_{\pm 5.4}$	$82.7_{\pm 6.8}$
DTU (18)	5s	$60.1_{\pm 3.0}$	$74.4_{\pm 9.2}$	$83.6_{\pm 3.3}$	$83.7_{\pm 3.5}$	$87.2_{\pm 9.7}$	$86.6_{\pm 6.6}$	$67.9_{\pm 14.1}$	$73.3_{\pm 16.0}$
NJU (21)	1s	$50.0_{\pm 1.6}$	$51.9_{\pm 2.1}$	$55.8_{\pm 1.7}$	$54.1_{\pm 1.9}$	$64.8_{\pm 11.1}$	$68.1_{\pm 8.7}$	$84.2_{\pm 7.5}$	$94.9_{\pm 3.4}$
NJU (21)	5s	$59.6_{\pm 3.5}$	$70.1_{\pm 14.1}$	$86.0_{\pm 5.1}$	$82.7_{\pm 7.3}$	$83.3_{\pm 17.1}$	$93.9_{\pm 5.9}$	$75.6_{\pm 15.3}$	$79.7_{\pm 18.0}$

Table 2: Intra-subject single-fold AAD per dataset. Cells are mean

\,\pm\,

std (%) over the per-subject values. Subject counts: KU Leuven 1s:

n=15

, KU Leuven 5s:

n=16

, DTU 1s:

n=18

, DTU 5s:

n=18

, NJU 1s:

n=21

, NJU 5s:

n=21

. Per-subject breakdowns in App. C .

Score	1 s (%)	5 s (%)	1 $\to$ 5 s (pp)
env- $r$	$50.5$	$59.7$	$+9.2$
QuadTrack -Multi	$52.6$	$73.5$	$+20.9$
AttuneFlow -Flow	$54.8$	$\mathbf{85.3}$	$\mathbf{+30.5}$
Env-Multi-Flow	$54.0$	$\mathbf{84.0}$	$\mathbf{+30.0}$

Table 3: Decision-window scaling (per-segment,

n\!=\!55

subjects pooled across KUL

+

DTU

+

NJU). We compare the effect of adding the flow stage at different time windows. The flow score gains

\sim

\times

more from increasing the window length than the single-Pearson baseline. Numbers from Table 2 .

Dataset	$n_{\text{win}}$	per-seg Env-Multi-Flow	per-seg AttuneFlow -Flow	trial AttuneFlow -EM-OR
KU Leuven	6	$63.4_{\pm 3.2}$	$61.7_{\pm 2.9}$	$\mathbf{98.1_{\pm 1.9}}$
DTU	6	$59.2_{\pm 1.8}$	$54.0_{\pm 1.3}$	$\mathbf{83.1_{\pm 4.1}}$
NJU	7	$51.0_{\pm 1.8}$	$51.4_{\pm 1.3}$	$\mathbf{73.7_{\pm 5.2}}$

Table 4: Cross-subject trial-level AAD accuracy. 19 leave-3-subjects-out windows across the three datasets; trial-level OR-fused AttuneFlow -EM pools

\sim

50 s of evidence (App. F.7 ).

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Type	Backbone	Cons. audio	Gen. env.	Gen. audio
mTRF [ 9 ]	Reconstructive	Linear ridge	yes	yes	no
VLAAI [ 1 ]	Reconstructive	CNN	yes	yes	no
DECAF [ 36 ]	Reconstructive	CNN + margin	yes	yes	no
SSM2Mel [ 15 ]	Reconstructive	State-space	yes	yes	no
DMF2Mel [ 14 ]	Reconstructive	State-space	yes	yes	no
wav2vec align [ 11 ]	Alignment	Transformer (frozen)	yes	no	yes $\dagger$

Appendix

Table 5: Representative AAD methods. “Cons. audio” = the model takes audio as input at inference; “Gen. env.” = produces a reconstructed envelope; “Gen. audio” = produces a waveform.

\dagger

wav2vec-align [ 11 ] synthesises waveforms via a separate pre-trained vocoder applied to predicted wav2vec representations rather than producing audio directly from the brain decoder; we count this as “yes” for completeness.

Figure 2 : KU Leuven per-subject AAD across the headline scoring rules: Env-Multi ( QuadTrack ), AttuneFlow , Env-Multi-Flow (the z-normalised ensemble that drives most of our improvement), Env-Trial-Fused , and Spatial . Top: 5 s windows; bottom: 1 s windows.

Figure 3 : DTU per-subject AAD across the headline scoring rules: Env-Multi , AttuneFlow , Env-Multi-Flow , Env-Trial-Fused , and Spatial . Top: 5 s windows; bottom: 1 s windows.

Figure 4 : NJU per-subject AAD across the headline scoring rules: Env-Multi , AttuneFlow , Env-Multi-Flow , Env-Trial-Fused , and Spatial . Top: 5 s windows; bottom: 1 s windows.

Rule	Inputs	Train cost	Best regime
Env-r	$\hat{\mathbf{e}},\mathbf{e}_{a/b}$	none	easy subjects, long windows
Env-Multi	$\hat{\mathbf{e}},\mathbf{e}_{a/b}$	none	noisy single-statistic failure modes
AttuneFlow	$\mathbf{c},\mathbf{e}_{a/b}$	flow head	long windows, strong $\delta$ – $\theta$
Env-Multi-Flow	all of the above	flow head	default per-segment decision
Spatial	raw EEG	none (closed-form CSP)	wide-azimuth ( $\pm 90^{\circ}$ ) datasets
Spatial-Trial	per-trial Spatial votes	none	wide-azimuth, intra-subject

Appendix

Table 6: Summary of inference scoring rules. “Inputs” is what each rule consumes; “Train cost” is whether the rule needs anything beyond the Stage-1

+

Stage-2 model; “Best regime” is the deployment scenario where it dominates.

Dataset	$\rho$	$\rho_{S}$	$\rho_{X}$	$\kappa$	Env-Multi
KU Leuven	60.4	60.7	64.5	62.1	71.7
DTU	58.9	59.4	67.0	60.2	75.5
NJU	61.8	62.6	65.3	60.9	70.8

Appendix

Table 7: Per-measure single-fold AAD accuracy (%) at 5 s for the four QuadTrack components and their unweighted sum, averaged across all subjects of each dataset. The sum exceeds every single measure because the four measures’ errors are largely uncorrelated — a single segment that fools

\rho

may not fool

\kappa

Dataset	QuadTrack -Multi	AttuneFlow -Flow	Env-Multi-Flow
KU Leuven (16)	$\pm 9.5$	$\pm 2.6$	$\pm 4.8$
DTU (18)	$\pm 9.2$	$\pm 3.3$	$\pm 3.5$
NJU (21)	$\pm 14.1$	$\pm 5.1$	$\pm 7.3$
mean	$\pm 10.9$	$\mathbf{\pm 3.7}$	$\pm 5.2$

Appendix

Table 8: Across-subject standard deviation (%) of per-segment AAD accuracy at 5 s windows for the three envelope-side scoring rules, extracted from the std columns of Table 2 . AttuneFlow -Flow has

2

–

4\times

smaller spread than QuadTrack -Multi on every dataset and stays under

\pm 6

pp even on NJU.

Probe	Recipe	Env-multi	Spatial	Env-concat	Sp-trial
D1	CSP shrink $0.2$ , env-infonce $2.0$	$58.6$	$61.9$	$95.0$	$65.0$
D2	Baseline (shrink 0.3, margin 8)	$\mathbf{59.5}$	$61.8$	$95.0$	$65.0$
E1	D2 with GroupNorm (BN $\to$ GN) in SpatialConv	$57.9$	$60.2$	$90.0$	$55.0$
E1b	E1 + rel. margin + hard-NCE	$56.1$	$59.6$	$75.0$	$60.0$
E2	E1b + Env-SubjectAdapter (neuro stats)	$57.0$	$61.2$	$80.0$	$60.0$
F	D2 + Match-Mismatch head	$58.0$	$61.6$	$\mathbf{100.0}$	$60.0$

Appendix

Table 9: KU Leuven LOSO S1 ablation (single fold). “Change” =

\Delta

vs. the D2 baseline. The per-segment envelope-multi column is bounded between

55.9

and

59.5\,\%

across every intervention we tried, a

3.6

-pp spread; the trial-concatenated column saturates at 95–100 %.

Held-out subject	Env-multi	Spatial	Env-concat	Sp-trial
S1 (D2)	$59.5$	$61.8$	$95.0$	$65.0$
S16 (J)	$59.6$	$\mathbf{85.3}$	$80.0$	$\mathbf{75.0}$

Appendix

Table 10: D2 recipe on two different KU Leuven held-out subjects (single-fold LOSO, 5 s). The per-segment env-multi ceiling is stable (

59.5

vs.

59.6\,\%

) across a radically different subject; the spatial path, however, is highly subject-dependent — S16’s clean alpha-lateralisation topography transfers to the cross-subject CSP filters much better than S1’s.

Epoch	Total loss	$\mathcal{L}_{\mathrm{warm}}$	$\mathcal{L}_{\mathrm{margin}}^{\rho}$	$\mathcal{L}_{\mathrm{NCE}}^{\rho}$
0	$50.03$	$1.01$	$0.71$	$9.79$
1	$37.63$	$1.00$	$0.69$	$6.66$
2	$37.04$	$0.99$	$0.69$	$6.58$
3	$36.99$	$0.99$	$0.69$	$6.58$
4	$36.80$	$0.99$	$0.69$	$6.58$
5	$36.82$	$0.99$	$0.69$	$6.57$

Appendix

Table 11: 1 s training, KU Leuven LOSO S16 (D2 recipe). Training-set loss terms do not descend beyond chance on the envelope path. The training-loss trajectory alone answers the question; the run was OOM-killed at validation.

Dataset	$n_{\text{win}}$	env- $r$	env-multi	attune-flow	env-multi-flow	spatial	spatial-trial	em-trial-OR
KU Leuven	6	$52.2_{\pm 0.8}$	$59.1_{\pm 1.9}$	$61.7_{\pm 2.9}$	$\mathbf{63.4_{\pm 3.2}}$	$\mathbf{69.7_{\pm 10.6}}$	$\mathbf{69.7_{\pm 8.8}}$	$\mathbf{98.1_{\pm 1.9}}$
DTU	6	$52.4_{\pm 1.1}$	$59.6_{\pm 2.4}$	$54.0_{\pm 1.3}$	$\mathbf{59.2_{\pm 1.8}}$	$\mathbf{50.0_{\pm 1.8}}$	$\mathbf{48.5_{\pm 2.0}}$	$\mathbf{83.1_{\pm 4.1}}$
NJU	7	$51.6_{\pm 1.2}$	$50.5_{\pm 1.0}$	$51.4_{\pm 1.3}$	$\mathbf{51.0_{\pm 1.8}}$	$\mathbf{51.4_{\pm 1.9}}$	$\mathbf{52.9_{\pm 2.4}}$	$\mathbf{73.7_{\pm 5.2}}$

Appendix

Table 12: Cross-subject 3-holdout AAD accuracy (%) on KU Leuven, DTU and NJU. For each dataset we slide a 3-subject validation window across all subjects (KU 6 windows, DTU 6 windows, NJU 7 windows = 19 cells total), train the full NeuroToken pipeline (Stage 1

+

Stage 2) on the remaining subjects, and evaluate on the 3 held-out subjects jointly. Cells are mean

\,\pm\,

std over the windows. Spatial / spatial-trial use the closed-form CSP

+

BandPower head; em-trial-OR is the AttuneFlow envelope

+

flow trial-level OR ensemble.

Held-out	env- $r$	env-multi	attune-flow	env-multi-flow	spatial	spatial-trial	em-trial-OR
S1,S2,S3	51.0	60.1	61.3	64.8	76.7	70.0	96.7
S4,S5,S6	52.5	60.7	64.2	66.0	59.5	61.7	100.0
S7,S8,S9	53.2	56.3	56.2	57.2	62.6	66.7	95.0
S10,S11,S12	52.5	57.1	62.4	63.1	59.5	61.7	98.3
S13,S14,S15	52.4	60.1	62.4	64.1	75.2	73.3	98.3
S14,S15,S16	51.4	60.1	63.8	65.1	84.6	85.0	100.0

Appendix

Table 13: KU Leuven cross-subject 3-holdout per-window AAD accuracy (%).

Held-out	env- $r$	env-multi	attune-flow	env-multi-flow	spatial	spatial-trial	em-trial-OR
S1,S2,S3	52.6	59.0	54.7	58.4	47.7	45.0	83.9
S4,S5,S6	50.6	57.9	53.8	57.1	49.1	48.3	76.1
S7,S8,S9	53.0	63.0	54.4	61.4	49.6	49.4	87.8
S10,S11,S12	53.2	57.3	51.6	57.6	53.1	51.1	81.7
S13,S14,S15	53.3	62.1	54.9	61.4	49.8	48.3	86.1
S16,S17,S18	51.5	58.4	54.8	59.2	50.8	48.9	83.3

Appendix

Table 14: DTU cross-subject 3-holdout per-window AAD accuracy (%).

Held-out	env- $r$	env-multi	attune-flow	env-multi-flow	spatial	spatial-trial	em-trial-OR
S02,S03,S04	53.5	49.3	49.8	49.7	50.5	51.0	64.6
S06,S07,S08	52.5	51.0	52.2	52.0	52.5	55.2	72.9
S09,S12,S13	52.2	51.1	51.4	53.3	52.2	52.1	81.2
S14,S15,S16	50.4	49.6	52.7	51.8	54.3	56.2	77.1
S17,S18,S19	51.3	50.5	50.9	50.0	49.7	51.0	75.0
S21,S22,S23	50.0	49.7	49.7	48.2	51.8	54.4	74.4

Appendix

Table 15: NJU cross-subject 3-holdout per-window AAD accuracy (%).

Panel of Figure 5 — Figure 5 : Common EEG front-end $f_{\theta}$ : a temporal-then-spatial CNN with four parallel multi-scale Conv1d branches, residual fusion gated by Squeeze–Excite, and adaptive pooling to the shared 32 Hz feature rate.

Layer	Op	Output shape	#Params
SpatialConv
temporal_conv	Conv2d( $1{\to}16$ , $k{=}(1,25)$ , $p{=}(0,12)$ , no bias)	$(B,16,64,T)$	400
spatial_conv	Conv2d( $16{\to}16$ , $k{=}(64,1)$ , groups $=16$ , no bias)	$(B,16,1,T)$	1 024
norm	BatchNorm2d(16)	$(B,16,1,T)$	32
act	GELU; squeeze axis 2	$(B,16,T)$	—
Multi-scale temporal

Appendix

Table 21

Panel of Figure 6 — Figure 6 : Envelope-spatial front-end $g_{\psi}$ : a fixed cortical-tracking-band FIR, a learned spatial mixer, four residual dilated 1-D convolutions and a strided downsample to the shared 32 Hz feature rate.

Layer	Op	Output shape	#Params
fixed FIR bandpass	257 taps, frozen, 0.5–9 Hz	$(B,64,T)$	0
SpatialConv (8 filters)	Conv2d $1{\to}8$ $k{=}(1,25)$ ; Conv2d $8{\to}8$ $k{=}(64,1)$ groups $=8$ ; BN; GELU	$(B,8,T)$	728
temporal_proj	Conv1d( $8{\to}48$ , $k{=}1$ )	$(B,48,T)$	432
Residual dilated stack: 4 layers, kernel 15, dilations 1/2/4/8
layer 1 (d=1)	Conv1d(48,48,15,p=7); GELU; Drop(0.2); residual	$(B,48,T)$	34 608
layer 2 (d=2)	Conv1d(48,48,15,p=14,d=2); GELU; Drop(0.2); residual	$(B,48,T)$	34 608

Appendix

Table 23

Panel of Figure 7 — Figure 7 : Subject adapter: two zero-initialised MLPs reading per-channel raw-EEG statistics emit (top) a FiLM scale–shift applied channel-wise to the $f_{\theta}$ feature stream, and (bottom) a bounded scalar lag $\Delta$ that temporally shifts the envelope head’s prediction.

Layer	Op	Output shape	#Params
stats	cat([mean, std]) over time	$(B,128)$	—
encoder	Linear(128 $\to$ 32); GELU; Linear(32 $\to$ 128) (zero-init)	$(B,128)$	8 352
chunk	split into $(\gamma,\beta)$ , $\gamma\!\leftarrow\!1+\gamma$	$(B,64),(B,64)$	—
FiLM apply	$\mathbf{h}\!\leftarrow\!\mathbf{h}\!\cdot\!\gamma+\beta$	$(B,160,64)$	—
lag_predictor	Linear(128 $\to$ 32); GELU; Linear(32 $\to$ 1) (zero-init)	$(B,1)$	4 161
lag bound	$8\,\tanh(\cdot)$ frames	$(B,)$	—

Appendix

Table 25

Panel of Figure 8 — Figure 8 : Spatial head $h_{s}$ (CSP $+$ BandPower): three parallel feature paths — closed-form CSP log-variance, per-channel band-power, and alpha-band asymmetries — are concatenated, BatchNorm-standardised, and classified by a small MLP. CSP filters $W^{\mathrm{CSP}}$ are fit on the train split and frozen.

Layer	Op	Output shape	#Params
csp_bandpass	rfft mask 8–30 Hz; irfft	$(B,64,T)$	0
csp_filters (frozen)	einsum $W\!\in\!\mathbb{R}^{8\times 64}$	$(B,8,T)$	0 (buf, 512)
log-var	$\log\!\mathrm{Var}_{t}(\cdot)$	$(B,8)$	—
bp_bandpass (6 bands)	RFFT, mag 2 /T, $\log(1+\cdot)$	$(B,64,6)$	0
alpha key-pair asym	$(R{-}L)/(R{+}L)$ at 6 pairs	$(B,6)$	—
concat (CSP $\,\|\,$ BP)	—	$(B,398)$	—

Appendix

Table 27

Panel of Figure 9 — Figure 9 : Envelope head $h_{e}$ : a residual dilated 1-D CNN over the cortical-tracking $g_{\psi}$ stream and a linear mTRF (“backward model”) FIR over raw EEG are summed; the subject adapter’s predicted lag $\Delta$ then shifts the resulting envelope before it is compared to the audio target.

Layer	Op	Output shape	#Params
Branch A: residual dilated CNN over $g_{\psi}$ output
proj_in	Conv1d(48 $\to$ 48, $k{=}1$ )	$(B,48,160)$	— (Identity)
block d=1	Conv1d(48,48,9,p=4); GELU; Drop(0.2); residual	$(B,48,160)$	20 784
block d=2	Conv1d(48,48,9,p=8,d=2); GELU; Drop(0.2); residual	$(B,48,160)$	20 784
block d=4	Conv1d(48,48,9,p=16,d=4); GELU; Drop(0.2); residual	$(B,48,160)$	20 784
block d=8	Conv1d(48,48,9,p=32,d=8); GELU; Drop(0.2); residual	$(B,48,160)$	20 784

Appendix

Table 29

Panel of Figure 10 — Figure 10 : AttuneFlow flow head $v_{\phi}$ : a conditional velocity network. Inputs are the noisy envelope $\mathbf{x}_{t}\!=\!(1{-}t)\mathbf{x}_{0}\!+\!t\mathbf{e}$ , the EEG conditioning $\mathbf{c}\!=\!g_{\psi}(\mathbf{x})$ , and a sinusoidal embedding of the flow time $t$ . Output is the predicted velocity along the rectified-flow path. Eight residual blocks of two dilated Conv1ds each give a $\sim$ 780 ms per-block receptive field; the output projection is zero-initialised so the velocity field starts as the identity transport.

Layer	Op	Output shape	#Params
time embed	sin/cos to $\mathbb{R}^{32}$ , broadcast over $T_{\eta}$	$(B,T_{\eta},32)$	—
time_mlp	Linear(32 $\to$ 128); GELU; Linear(128 $\to$ 128)	$(B,T_{\eta},128)$	20 736
concat $[x_{t},c,t_{\text{emb}}]$	—	$(B,T_{\eta},177)$	—
input_proj	Conv1d(177 $\to$ 128, $k{=}1$ )	$(B,128,T_{\eta})$	22 784
8 stacked velocity blocks (each, identical)
conv_a	Conv1d(128,128,9,p=4,d=1)	$(B,128,T_{\eta})$	147 584

Appendix

Table 31

Panel of Figure 11 — Figure 11 : AttuneFlow contrastive projector $p_{\phi}$ : two parallel MLP towers map temporally pooled (8 bins) EEG conditioning and a candidate envelope into a shared 128-d L2-normalised embedding space, scored by InfoNCE during Stage 2.

Layer	Op	Output shape	#Params
EEG branch
adaptive_avg_pool1d; flatten	to 8 bins	$(B,384)$	—
eeg_mlp	Linear(384 $\to$ 256); GELU; Drop(0.1); Linear(256 $\to$ 128)	$(B,128)$	131 456
L2 normalise	$z_{\text{eeg}}$	$(B,128)$	—
Envelope branch
adaptive_avg_pool1d; flatten	to 8 bins	$(B,8)$	—

Appendix

Table 33

Panel of Figure 12 — Figure 12 : Audio gammatone envelope (parameter-free): cochlear-style 8-band analysis followed by half-wave rectification, low-pass to the cortical-tracking band, downsampling to the 32 Hz feature rate, and mean over bands to the single broadband envelope used as the supervision target.

Layer	Op	Output shape
gammatone bank	8 subbands, 150–2000 Hz, fixed FIR	$(B,8,T_{a})$
half-wave rectify	$\max(0,\cdot)$	$(B,8,T_{a})$
low-pass + downsample	to 32 Hz, $T_{\eta}\!=\!160$ frames	$(B,8,160)$
broadband mean	average over 8 bands	$(B,160,1)$
Total

Appendix

Table 35

Panel of Figure 13 — Figure 13 : Auxiliary band-power module: per-band FIR analysis ( $\theta/\alpha/\beta$ ) with a small set of learned spatial filters, squared and Hann-smoothed to a 12-d 32 Hz feature stream. Computed every forward pass; consumed only by the optional transformer-style spatial head, retained in the trainable graph for ablations.

Layer	Op	Output shape	#Params
3 fixed FIR bandpass	129-tap, theta/alpha/beta (frozen)	$(B,64,T)$ ea.	0
spatial_filters[3 $\times$ ]	Conv1d(64 $\to$ 4, $k{=}1$ , no bias)	$(B,4,T)$ ea.	768
square; Hann smooth (250 ms); $\log(1+\cdot)$	—	$(B,12,T)$	0
adaptive_avg_pool1d	to $T_{\eta}\!=\!160$	$(B,12,160)$	—
proj	Linear(12 $\to$ 12); GELU	$(B,160,12)$	156
Total			924

Appendix

Table 37

Module	Stage 1 trainable	Stage 2 trainable
$f_{\theta}$ (EEG common front-end, $d_{z}\!=\!64$ )	23 360	frozen
$g_{\psi}$ (envelope-spatial front-end, 0.5–9 Hz)	140 024	frozen
Subject adapter (FiLM $+$ lag, $\Delta_{\max}\!=\!8$ frames)	12 513	frozen
$h_{s}$ (CSP $+$ BandPower spatial head; CSP buf frozen)	30 071	frozen
$h_{e}$ (residual dilated CNN $+$ linear mTRF)	84 210	frozen
Auxiliary BandPower features (3 bands $\times$ 4 spatial)	924	frozen

Appendix

Table 38

Model	S1 P / S	S2 P / S	S3 P / S	S4 P / S	Mean P (%)	Mean S (%)	$\Delta$ (pp)
DARNet	94.0/39.6	84.0/68.8	97.0/68.8	83.0/43.8	89.5 $\pm\,7.0$	55.2 $\pm\,15.7$	$+$ 34.3
ListenNet	89.9/33.3	92.0/52.1	100.0/95.5	89.9/12.5	93.0 $\pm\,4.8$	48.4 $\pm\,35.3$	$+$ 44.6
DBPNet	83.0/10.4	86.4/50.0	96.6/92.1	87.5/40.6	88.4 $\pm\,5.8$	48.3 $\pm\,33.7$	$+$ 40.1
DenseNet-3D	82.4/76.7	91.8/88.8	95.1/73.3	76.4/40.2	86.4 $\pm\,8.6$	69.8 $\pm\,20.8$	$+$ 16.6
SWIM	62.5/4.2	61.1/49.4	76.4/87.5	57.6/0.0	64.4 $\pm\,8.2$	35.3 $\pm\,41.4$	$+$ 29.2

Appendix

Table 16: Per-subject baseline reproduction on KU Leuven S1–S4 at 5 s windows. “P” = each paper’s exact published split (with all data leaks); “S” = trial-disjoint strict counterpart with no train/test window overlap and CSP/EA/std-divide fit on training trials only.

\Delta

is the leaky-minus-strict gap on the per-subject mean.

Explore similar work

End-to-End Markov State Sequence Learning for Auditory Attention Decoding
Jul 21, 2026Yushan Yashengjiang, Jie Zhang, Miao Sun +3Auditory Attention
NERC-SLIP, University of Science and Technology of China (USTC), Hefei 230027, China · School of Information and Communication Engineering, Guangzhou Maritime University, Guangzhou 510725, China · Artificial Intelligence Research Institute, iFLYTEK Company, Ltd., Hefei 230088, China +1
FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding
Jun 12, 2026Ziwei Wang, Xingyi He, Tianwang Jia +2Auditory Attention Neural Audio
Hubei Key Laboratory of Brain-inspired Intelligent Systems, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China
RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection
Sep 12, 2026Xingyi He, Ziwei Wang, Dongrui WuAuditory Attention Multimodal Fusion
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, China

Jul 21, 2026, cs.SDJ/K move · Enter open · S save

End-to-End Markov State Sequence Learning for Auditory Attention Decoding

Yushan Yashengjiang, Jie Zhang, Miao Sun, Huadong Liang+2

NERC-SLIP, University of Science and Technology of China (USTC), Hefei 230027, China · School of Information and Communication Engineering, Guangzhou Maritime University, Guangzhou 510725, China · Artificial Intelligence Research Institute, iFLYTEK Company, Ltd., Hefei 230088, China+1

Auditory Attention

Open paper AskPDF arXiv

Auditory attention decoding (AAD) identifies the speaker a listener attends to from neural responses like electroencephalography (EEG), making it a key algorithm in neuro-steered hearing aids. However, most neural AAD models are trained as independent short-window classifiers, despite auditory attention being a temporally persistent cognitive state and short-window EEG--audio evidence often being noisy and ambiguous. We propose an end-to-end Markov AAD framework based on conditional random field (CRF) that trains window-level neural emissions under a two-state attention prior. The framework treats the logits of any AAD backbone as Markov emissions, learns the transition rate from a standard HMM initialization, and jointly optimizes cross-entropy and CRF objectives, allowing temporal continuity to guide representation learning rather than merely smoothing predictions after training. We also introduce ESCNet, an EEG--speech correlation backbone that preserves time-aligned features and converts the difference between two mean Pearson correlations into state logits. We evaluate the framework with four emission backbones spanning correlation-based, convolutional, recurrent, and attention-based designs. On the dynamic AVGC dataset, CRF training generally outperforms post-hoc HMM smoothing; with ESCNet, it achieves

86.5\%

causal and

92.4\%

non-causal accuracy using

1

s windows. On the static KUL and USTC datasets, it improves causal decoding over fixed-rate post-hoc HMM baselines by

5.6\%

and

2.0\%

, respectively, showing the superiority of learning AAD as attention state sequence over isolated-window classification.

Abstract

Explore similar work

End-to-End Markov State Sequence Learning for Auditory Attention Decoding

FAConformer: Frequency-Aware Convolutional Transformer for Auditory Attention Decoding

RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection