Teaching neural decoders with language

Xuan Wen; Yonghao Song; Yuhong Zhang; Guoliang Li; Tzyy-Ping Jung; Xiaorong Gao

cs.AIOct 9, 2026

Teaching neural decoders with language

Authors: Xuan Wen, Yonghao Song, Yuhong Zhang, Guoliang Li, Tzyy-Ping Jung, Xiaorong Gao

Organizations: Tsinghua University · University of California, San Diego

Abstract

Brain-computer interfaces decode neural activity, yet class-label supervision (CLS) leaves relationships among classes implicit. We propose Language as Supervision (LaS), which aligns neural features with natural-language class descriptions through learned projections and predicts classes using candidate text prototypes. We evaluate seven encoder architectures with independently trained models for visual, motor, and speech decoding from scalp and intracranial recordings. LaS improves speech-decoding accuracy by up to 17.15~pp over matched CLS baselines. Its representations support candidate-set reuse and label-supervised subject and session adaptation, while LaS enables conditional unseen-frequency decoding in steady-state visual evoked potential (SSVEP) responses. Correct class-description correspondence supports this generalization, and dual-sided attribution visualization shows neural and textual contributions to the same class-matching score. These findings suggest that text descriptions can serve as general-purpose references for learning and interpretation in neural decoding.

Figures & tables

Figure 1: Language supervision across neural decoders. (a) CLS uses a task-specific classifier; LaS matches projected neural features to text prototypes. (b) Mean accuracy across ten task–encoder settings in MI, SSVEP, and speech decoding. Task sectors use different accuracy scales; radii and polygon areas are not comparable across tasks. Conformer denotes EEG-Conformer. Full comparisons appear in Table 1 .

Figure 2: The overall architecture of LaS. (a) Prompt $P_{c}$ is the complete class description supplied to the text encoder. Each task–encoder model trains its neural encoder and projections $W_{n}$ and $W_{t}$ for neural–text alignment, then predicts the highest-scoring candidate. (b) The evaluated tasks comprise finger MI and SSVEP decoding from scalp EEG, and overt Chinese word reading from intracranial sEEG. Speech class descriptions use English translations. (c) Neural input representations follow the selected encoder. Subject-specific sEEG tokenizers feed a backbone shared within the same model.

Task	Method/encoder	Params (M)		Duration (s)	Performance		$\Delta$ Acc. (pp)
Task	Method/encoder	CLS	LaS	Duration (s)	CLS	LaS	$\Delta$ Acc. (pp)
MI	FBCSP-LDA	—	—	5.0	32.47 $\pm$ 5.94 ( $\kappa$ =0.100)		—
	EEGNet	0.002	0.30		38.39 $\pm$ 4.76 ( $\kappa$ =0.179)	38.68 $\pm$ 7.08 ( $\kappa$ =0.182)	+0.28
	DeepConvNet	0.29	0.79		38.72 $\pm$ 6.05 ( $\kappa$ =0.183)	40.54 $\pm$ 8.65 ( $\kappa$ =0.207)	+1.82 *
	Conformer	0.29	0.48		41.67 $\pm$ 8.77 ( $\kappa$ =0.222)	43.02 $\pm$ 9.29 ( $\kappa$ =0.240)	+1.36 **
SSVEP RW $\to$ IW 1	EEGNet	0.04	0.32	0.6	26.76 $\pm$ 11.55	26.98 $\pm$ 10.91	+0.21

Table 1: Decoding performance across tasks and neural encoders. CLS–LaS comparisons; FBCSP-LDA is a separate MI reference. Performance: subject-mean accuracy (%)

\pm

sample SD; mean MI

\kappa

in parentheses.

\Delta=\mathrm{LaS}-\mathrm{CLS}

in percentage points. Blue/gray: highest/second-highest means within each MI or Speech metric across CLS and LaS. 1 SSVEP: RW training

\to

IW evaluation. Stars: Holm-corrected paired

t

-tests within encoder (Section 4.1 ; Appendix B.6 ).

Figure 4: Label-supervised adaptation and LaS zero-shot decoding. Neural models in (a,b) are initialized from LaS and adapted using CLS. (a) SSVEP subject adaptation: mean validation accuracy with $K$ target-subject calibration blocks (1-s inputs; six leave-one-block-out rounds). (b) MI session adaptation: adapted-model accuracy on eight later (“Fine”) test runs minus direct-inference accuracy on eight Base runs per subject (mean paired differences across 21 subjects; 95% $t$ intervals). (c) LaS SSVEP decoding restricted to five held-out frequency classes, with the same 35 subjects used for training and evaluation (mean $\pm$ SD across eight class splits; 20% chance). The broken axis omits 25–60%; its two segments use different scales.

Figure 5: Training prompt configurations and inference settings. (a) Closed-set decoding. (b) Five-class SSVEP unseen-only decoding with one 1-s initial window; dashed line: 20% chance. Panels (a,b) use fixed prompts at inference. (c) P0 and P2 are evaluated with an ensemble of five templates held out from both training configurations. Units in (a–c): subjects (21 MI, 35 SSVEP, 12 Speech) or eight ZSL class splits. Boxes: medians and IQRs; whiskers follow the $1.5\times\mathrm{IQR}$ rule; hollow circles: outliers; colored symbols: means. Brackets: paired comparisons with P0. (d) EmbeddingGemma $-$ BERT accuracy differences (pp); best configurations are selected independently of the Frozen–LoRA pairs (Figure B.1 ). Holm-corrected paired tests (Section 4.1 ).

Figure 6: Dual-sided attribution visualization with Integrated Gradients. Training-trial averages: (a) MI, (b) Speech, and (c) SSVEP. Blue denotes low nonnegative neural display values, not negative attribution; green denotes positive mean word contributions. Separate panel scales preclude absolute comparisons.

Figure 7: Speech training objectives across batch sizes. Du-IN with (a) InfoNCE, (b) Diagonal SigLIP, and (c) Class-aware SigLIP. Colored dots: each seed’s minimum epoch-average full training loss (left axis); gray diamonds: mean accuracy (right axis). Lines show the balanced-class lower bounds in Equation ( 5 ), extended continuously to

m=B/61

. Pale bands and upward point offsets are display aids, not uncertainty. † At

B=1024

in (c), four loss observations are shown; accuracy includes all five seeds. The full three-task comparison appears in Figure B.6 .

Figure 8: Directional InfoNCE ablation. Single-direction minus bidirectional accuracy (pp): (a) MI, $n=21$ ; (b) Speech, $n=12$ . Dots: subject differences; diamonds: means; bars: unadjusted 95% paired-difference $t$ intervals. Stars use Holm-corrected tests (Section 4.1 ).

Appendix figures & tables49 assets

Supplementary material from the paper’s appendix.

Appendix

Preferred term	Meaning and scope
Language as Supervision (LaS)	Language-supervised decoding by aligning neural representations with natural-language class descriptions.
Class-label supervision (CLS)	Neural-classification comparator trained with discrete labels and a task-specific head; LaS also uses class labels to establish pairs.
One-hot targets; cross-entropy (CE)	Target encoding and loss function, respectively.
Motor imagery (MI)	Finger motor-imagery decoding task.
SSVEP	Steady-state visual evoked potential decoding of visual stimulus targets.
Speech	Speech-decoding task using overt word reading.

Appendix

Table A.1: Terminology and naming conventions.

Item	Setting
Trial split	Class-stratified five-fold split of physical trials within each of 21 participants; four folds train and one fold is held out. All windows of a trial stay in one fold.
Model input	32 channels at 250 Hz; 1.0-s window, 0.2-s stride, and 5.0-s complete trial.
Candidate classes	Four finger motor-imagery classes.
Aggregation	Combine held-out trial predictions within each participant, then average participant metrics equally.

Appendix

Table A.2: MI protocol for the main performance comparison.

Module	Sub-module	Parameter	Value
EEGNet	Common boundary	Position; shape	After separable-convolution ELU; $16\times 25$
	CLS readout	Average pool	Kernel/stride $(1,8)$ / $(1,8)$
		Dropout	$p=0.5$
		Flatten	$16\times 3\rightarrow 48$
	CLS head	Linear	$48\rightarrow C$
DeepConvNet 1s	Common boundary	Position; shape	After fourth convolutional block; $200\times 1\times 6$

Appendix

Table A.3: MI encoders: CLS readout and classification paths.

Module	Sub-module	Parameter	Value
EEGNet	Common boundary	Position; shape	After separable-convolution ELU; $16\times 25$
	LaS readout	Flatten	$16\times 25\rightarrow 400$
	Neural projection	Linear	$400\rightarrow 256$
	Text projection	Linear	$768\rightarrow 256$
DeepConvNet 1s	Common boundary	Position; shape	After fourth convolutional block; $200\times 1\times 6$
	LaS readout	Flatten	$200\times 1\times 6\rightarrow 1200$

Appendix

Table A.4: MI encoders: LaS readout and projection paths.

ID	Template text
0 (P0)	The participant is imagining repetitive flexion and extension of the right {finger}.
1	The person is performing motor imagery of moving the right {finger}.
2	This EEG corresponds to imagined flexion and extension of the right {finger}.
3	The participant is mentally rehearsing repetitive movement of the right {finger}.
4	The intended action is flexion and extension of the right {finger}, imagined without physical movement.
S0	an EEG of imagined right {finger} motion.

Appendix

Table A.5: MI P0/P2 training and held-out templates.

Encoder	Duration (s)	Params (M)		Forward GFLOPs
Encoder	Duration (s)	CLS	LaS	CLS	LaS
EEGNet	5.0	0.002	0.30	0.22	0.23
DeepConvNet1s	5.0	0.29	0.79	0.55	0.56
EEG-Conformer	5.0	0.29	0.48	0.85	0.85

Appendix

Table A.6: MI parameter counts and classification forward FLOPs.

Item	Setting
Participant split	Seven folds; each has 25 training, five validation, and five evaluation participants.
Training input	RW of 0.6, 0.8, or 1.0 s; temporal masking at ratio 0.3 for the applicable time-domain input.
Evaluation input	IW of the corresponding length, starting 0.14 s after stimulus onset.
Candidate classes	All 40 stimulus classes.

Appendix

Table A.7: SSVEP protocol for the paired encoder comparison in Table 1.

Module	Sub-module	Parameter	Value
EEGNet	Common boundary	Position	After final dropout
	CLS readout	Flattened width $d$	384/576/672
	CLS head	Linear	$d\rightarrow C$
SSVEPFormer	Common boundary	Position	After frequency-domain encoder
	CLS readout	Flatten	10080
	CLS hidden	Linear	$10080\rightarrow 240$

Appendix

Table A.8: SSVEP encoders: CLS readout and classification paths.

Module	Sub-module	Parameter	Value
EEGNet	Common boundary	Position	After final dropout
	LaS readout	Flattened width $d$	384/576/672
	Neural projection	Linear	$d\rightarrow 128$
	Text projection	Linear	$768\rightarrow 128$
SSVEPFormer	Common boundary	Position	After frequency-domain encoder
	LaS readout	Avg + max pooling	Elementwise sum; kernel/stride 10/10

Appendix

Table A.9: SSVEP encoders: LaS readout and projection paths.

ID	Template text
0 (P0)	A neural SSVEP signal at {frequency} Hz.
1	A {frequency}-Hz SSVEP.
2	SSVEP signal, frequency {frequency} Hz.
3	Steady-state visual evoked potential at {frequency} Hz.
4	The neural response SSVEP corresponds to a {frequency}-Hz rhythm
S0	An EEG response at {frequency} Hz.

Appendix

Table A.10: SSVEP P0/P2 training and held-out templates.

Encoder	Duration (s)	Params (M)		Forward GFLOPs
Encoder	Duration (s)	CLS	LaS	CLS	LaS
EEGNet	0.6	0.04	0.32	0.04	0.06
	0.8	0.05	0.37	0.05	0.07
	1.0	0.05	0.39	0.07	0.08
MTSNet	0.6	9.92	7.37	0.33	0.34
	0.8	10.87	8.12	0.36	0.37

Appendix

Table A.11: SSVEP parameter counts and classification forward FLOPs.

Encoder	Sampling rate	Samples per trial and evaluation unit
Du-IN	1000 Hz	3000; one prediction per trial
EEGNet	250 Hz	750; one prediction per trial
EEG-Conformer	250 Hz	750; one prediction per trial
CNN-BiGRU	30 Hz	90; one prediction per trial

Appendix

Table A.12: Three-second Speech trial inputs for the main performance comparison.

Module	Sub-module	Parameter	Value
EEGNet	Common boundary	Position	Before final pooling
	Backbone end	Average pool	Kernel/stride $(1,8)$ / $(1,8)$
		Dropout	$p=0.5$
	CLS readout	Flatten	144
	CLS head	Linear; output	$144\rightarrow C$ ; raw logits
CNN-BiGRU	Common boundary	Position	Bidirectional GRU output

Appendix

Table A.13: Speech encoders: CLS readout and classification paths.

Module	Sub-module	Parameter	Value
EEGNet	Common boundary	Position	Before final pooling
	Backbone end	Average pool	Kernel/stride $(1,8)$ / $(1,8)$
		Dropout	$p=0.1$
	LaS readout	Flatten	144
	Neural projection	Linear	$144\rightarrow 128$
	Text projection	Linear	$768\rightarrow 128$

Appendix

Table A.14: Speech encoders: LaS readout and projection paths.

Module	Sub-module	Parameter	Value
Du-IN	Input	Shape	$10\times 3000$ (channels $\times$ samples)
	Segmentation	Output	30 segments of $10\times 100$
	Temporal Conv1d 1	Channels; kernel; stride; padding	$10\rightarrow 128$ ; 19; 10; 9
	Normalization 1	BatchNorm1d	128 channels; $\epsilon=10^{-5}$ ; momentum 0.1
	Temporal Conv1d 2	Channels; kernel; stride; padding	$128\rightarrow 128$ ; 3; 1; 1
	Normalization 2	BatchNorm1d	128 channels; $\epsilon=10^{-5}$ ; momentum 0.1

Appendix

Table A.15: Subject-specific input frontends and shared-backbone boundaries for Speech decoding.

ID	Template text
0 (P0)	The participant is speaking the word: {label}.
1	This sEEG recording corresponds to the spoken word: {label}.
2	The intended speech content is: {label}.
3	The neural activity was recorded while the participant said: {label}.
4	The target speech label for this intracranial neural signal is: {label}.
S0	An sEEG response for spoken {label}.

Appendix

Table A.16: Speech P0/P2 training and held-out templates.

Encoder	Duration (s)	Params (M)		Forward GFLOPs
Encoder	Duration (s)	CLS	LaS	CLS	LaS
EEGNet	3.0	0.02	0.13	0.01	0.02
CNN-BiGRU	3.0	0.62	0.84	0.03	0.05
EEG-Conformer	3.0	4.18	3.96	0.45	0.47
Du-IN	3.0	5.05	4.94	0.27	0.29

Appendix

Table A.17: Speech parameter counts and classification forward FLOPs.

Table B.1: SSVEP subject-mean accuracy (%) and estimated ITR (bits/min) across five initial-window durations. Stars indicate unadjusted paired-test significance:

{}^{*}p<.05

{}^{**}p<.01

, and

{}^{***}p<.001

Subject	FBCSP	EEGNet C	EEGNet L	DeepConv C	DeepConv L	Conformer C	Conformer L
S01	32.00	41.17	38.67	39.17	42.17	42.33	42.00
S02	36.83	42.33	37.00	39.83	44.33	44.17	46.83
S03	27.17	35.17	32.67	43.83	44.67	44.00	44.50
S04	28.91	39.84	36.25	38.44	38.59	36.88	35.94
S05	28.91	38.91	40.63	37.97	40.47	40.63	43.91
S06	30.67	36.00	37.00	35.33	34.33	37.00	38.33

Appendix

Table B.2: MI four-class trial accuracy by subject (%). C and L denote supervised CLS and LaS.

Subject	FBCSP	EEGNet C	EEGNet L	DeepConv C	DeepConv L	Conformer C	Conformer L
S01	0.09	0.22	0.18	0.19	0.23	0.23	0.23
S02	0.16	0.23	0.16	0.20	0.26	0.26	0.29
S03	0.03	0.14	0.10	0.25	0.26	0.25	0.26
S04	0.05	0.20	0.15	0.18	0.18	0.16	0.15
S05	0.05	0.19	0.21	0.17	0.21	0.21	0.25
S06	0.08	0.15	0.16	0.14	0.12	0.16	0.18

Appendix

Table B.3: MI four-class Cohen’s

\kappa

by subject.

Subject	0.6 C	0.6 L	0.8 C	0.8 L	1.0 C	1.0 L
0	30.42	28.33	45.42	41.25	53.33	52.08
1	33.75	28.75	40.83	39.58	50.00	54.17
2	48.75	47.08	58.33	57.92	75.00	72.08
3	36.67	39.58	64.58	55.83	70.00	57.50
4	20.42	19.17	34.58	30.00	43.75	38.75
5	27.50	32.08	51.67	47.50	66.25	59.17

Appendix

Table B.4: SSVEP subject accuracy (%) for EEGNet after RW training and IW evaluation.

Subject	0.6 C	0.6 L	0.8 C	0.8 L	1.0 C	1.0 L
0	35.83	35.83	54.58	48.33	62.08	61.67
1	56.67	55.42	72.92	73.75	82.92	85.83
2	65.00	69.58	87.08	87.92	94.17	93.33
3	62.08	63.33	82.92	75.42	87.50	88.75
4	35.83	39.58	70.42	63.75	77.50	77.50
5	31.25	33.75	56.25	63.75	72.08	68.75

Appendix

Table B.5: SSVEP subject accuracy (%) for MTSNet after RW training and IW evaluation.

Subject	0.6 C	0.6 L	0.8 C	0.8 L	1.0 C	1.0 L
0	35.42	42.50	44.58	53.75	62.50	66.67
1	51.25	54.17	74.17	75.00	80.83	83.33
2	62.92	69.17	80.83	84.17	91.67	94.17
3	62.50	52.50	75.00	78.75	88.75	87.50
4	37.50	35.00	60.42	50.83	75.83	72.92
5	31.25	34.58	57.50	60.83	67.08	75.42

Appendix

Table B.6: SSVEP subject accuracy (%) for SSVEPFormer after RW training and IW evaluation.

Subject	EEGNet C	EEGNet L	BiGRU C	BiGRU L	Conformer C	Conformer L	Du-IN C	Du-IN L
S001	9.69	24.06	51.56	70.94	60.31	73.75	73.75	77.81
S002	8.49	18.90	32.05	41.92	44.11	46.03	46.58	47.95
S003	6.69	4.46	25.63	38.72	32.03	35.93	38.44	45.68
S004	8.20	19.02	42.62	66.56	59.02	69.84	68.85	76.72
S005	11.48	22.62	50.82	69.18	61.31	73.11	73.11	77.38
S006	3.72	8.31	29.80	46.70	39.26	40.40	47.56	50.72

Appendix

Table B.7: Speech subject accuracy (%) across four neural encoders.

Subject	1–4 C	1–4 L	2–3 C	2–3 L	1–3–4 C	1–3–4 L	1–2–3 C	1–2–3 L
S01	76.00	76.00	61.67	63.33	54.22	52.89	43.11	40.22
S02	74.00	76.33	66.00	65.33	44.44	46.67	45.33	42.67
S03	70.00	65.00	53.00	50.67	50.22	42.00	40.89	36.89
S04	73.44	69.38	52.19	53.44	47.92	47.08	45.83	42.71
S05	79.06	74.69	55.63	51.88	53.13	51.46	41.25	41.67
S06	69.33	67.33	59.33	56.67	47.11	47.56	42.67	44.00

Appendix

Table B.8: MI candidate-set reuse: EEGNet subject accuracy (%).

Subject	1–4 C	1–4 L	2–3 C	2–3 L	1–3–4 C	1–3–4 L	1–2–3 C	1–2–3 L
S01	63.33	76.33	55.67	59.67	50.67	55.11	39.56	47.78
S02	67.33	76.00	66.00	67.00	48.22	51.11	40.89	50.67
S03	67.00	75.00	45.67	54.33	55.33	58.67	39.11	47.56
S04	69.69	75.63	48.13	52.50	48.54	51.46	35.00	40.83
S05	71.56	78.75	52.50	52.81	50.83	53.96	39.58	42.71
S06	55.33	64.00	52.67	54.67	44.44	44.89	37.33	43.56

Appendix

Table B.9: MI candidate-set reuse: DeepConvNet1s subject accuracy (%).

Subject	1–4 C	1–4 L	2–3 C	2–3 L	1–3–4 C	1–3–4 L	1–2–3 C	1–2–3 L
S01	77.00	76.67	60.67	65.33	56.44	52.00	46.00	46.89
S02	73.33	76.67	64.33	67.33	51.56	53.78	45.33	53.11
S03	77.67	77.67	56.67	59.67	58.67	60.00	45.33	46.00
S04	69.38	72.19	54.69	53.44	46.88	48.75	39.17	39.79
S05	78.75	82.81	54.38	56.56	55.83	58.54	44.79	43.13
S06	70.67	71.33	60.67	61.33	44.44	49.33	43.56	43.11

Appendix

Table B.10: MI candidate-set reuse: EEG-Conformer subject accuracy (%).

Method	3-class	2-class
Direct inference	46.43 $\pm$ 9.14	64.24 $\pm$ 9.56
Linear Probe	48.69 $\pm$ 10.08	66.63 $\pm$ 9.80
LP++	48.67 $\pm$ 9.81	65.76 $\pm$ 11.08
ProLIP	49.13 $\pm$ 9.67	65.55 $\pm$ 11.48
DTS-ProLIP	49.94 $\pm$ 9.36	65.91 $\pm$ 11.93
MLP Probe Encoder	50.23 $\pm$ 10.39	67.01 $\pm$ 11.06

Appendix

Table B.11: MI cross-session adaptation: subject-mean trial accuracy

\pm

between-subject SD (%).

Family	Method	K=1	K=2	K=3	K=4	K=5
LaS	DTS-ProLIP	80.38	81.85	82.81	83.45	84.17
LaS	ProLIP	79.46	80.14	80.70	80.94	81.25
LaS	Linear Probe	71.29	79.01	81.37	82.11	82.56
LaS	LP++	75.44	77.58	79.01	79.71	80.26
LaS	MLP Probe Encoder	72.89	82.77	85.93	87.55	88.93
LaS	Full Finetune Encoder	62.92	77.26	82.88	85.57	88.07

Appendix

Table B.12: SSVEP K-block adaptation: leave-one-block validation accuracy (%).

Class	Hz	MTSNet	SSVEPFormer	EEGNet	FBCCA
0	8.0	72.38	82.86	78.57	85.71
1	8.2	81.90	83.33	83.81	86.19
2	8.4	84.76	79.05	79.05	86.19
3	8.6	85.24	85.24	75.24	85.24
4	8.8	82.86	78.57	81.90	82.38
5	9.0	72.86	72.38	70.48	85.24

Appendix

Table B.13: Five-class unseen-only accuracy (%) by stimulus frequency with 0.6 s inputs.

Class	Hz	MTSNet	SSVEPFormer	EEGNet	FBCCA
0	8.0	89.52	92.86	84.29	93.81
1	8.2	90.48	90.95	87.62	94.76
2	8.4	92.86	90.95	88.10	90.48
3	8.6	93.81	88.57	86.19	91.43
4	8.8	90.95	82.86	89.05	90.95
5	9.0	84.76	86.19	85.24	90.95

Appendix

Table B.14: Five-class unseen-only accuracy (%) by stimulus frequency with 0.8 s inputs.

Class	Hz	MTSNet	SSVEPFormer	EEGNet	FBCCA
0	8.0	90.95	92.86	92.38	96.67
1	8.2	96.67	96.67	93.81	98.57
2	8.4	91.43	91.43	91.43	95.24
3	8.6	95.24	93.81	92.86	94.29
4	8.8	92.38	91.90	92.38	94.76
5	9.0	83.33	92.38	89.52	92.86

Appendix

Table B.15: Five-class unseen-only accuracy (%) by stimulus frequency with 1.0 s inputs.

Figure B.1: Paired Frozen and LoRA comparisons for BERT-base-uncased and EmbeddingGemma-300M. Lines connect the matched run units specified in the text.

Task	Template	P0	P2
MI closed-set	S0	40.20 $\pm$ 9.66	42.65 $\pm$ 9.59
	S1	38.81 $\pm$ 9.91	42.43 $\pm$ 9.48
	S2	39.23 $\pm$ 9.74	42.54 $\pm$ 9.36
	S3	40.19 $\pm$ 9.71	42.52 $\pm$ 9.45
	S4	35.88 $\pm$ 10.06	42.32 $\pm$ 9.47
	Five-template ensemble	38.94 $\pm$ 9.96	42.51 $\pm$ 9.40

Appendix

Table B.16: Accuracy (%, mean

\pm

SD) under held-out short templates.

Figure B.2: MI training attribution for thumb and index, subject S01; 120 trials per class.

Figure B.3: MI training attribution for middle and pinky, subject S01; 120 trials per class.

Figure B.4: Speech training attribution for four word classes, subject S001; class supports are 38, 40, 40, and 37.

Figure B.5: SSVEP training attribution at 8.0, 10.6, 13.2, and 15.8 Hz, subject S01; six original trials per class.

Figure B.6: Full training-objective by batch-size comparison across MI, SSVEP, and Speech. The left and right axes show training loss and classification accuracy, respectively; reference curves are theoretical loss baselines.

Figure B.7: Cumulative MI trial accuracy with logit averaging, probability averaging, and majority voting. Each time point uses all windows available up to that time.

Task	Encoder (window)	$n$	$\Delta$ (pp)	$t(df)$	$p_{\mathrm{Holm}}$	$d_{z}$
MI	EEGNet	21	$+0.28$	$0.33\ (20)$	$0.747607843$	$+0.07$
MI	DeepConvNet	21	$+1.82$	$2.74\ (20)$	$0.0125586274$	$+0.60$
MI	EEG-Conformer	21	$+1.36$	$3.23\ (20)$	$0.00423557447$	$+0.70$
SSVEP	EEGNet (0.6 s)	35	$+0.21$	$0.35\ (34)$	$0.727554094$	$+0.06$
SSVEP	EEGNet (0.8 s)	35	$-0.65$	$-1.04\ (34)$	$0.613050642$	$-0.18$
SSVEP	EEGNet (1.0 s)	35	$-1.76$	$-2.02\ (34)$	$0.154051070$	$-0.34$

Appendix

Table B.17: Paired statistical comparisons for Table 1.

n

is the number of subjects,

\Delta

is the mean accuracy difference in percentage points, and

p_{\mathrm{Holm}}

is corrected within the comparison families defined above.

EEG encoder	$L$	CLS	LaS	$\Delta$
EEGNet	0.6 s	26.76	26.98	+0.21
	0.8 s	41.07	40.42	$-0.65$
	1.0 s	51.62	49.86	$-1.76$
MTSNet	0.6 s	34.32	36.56	+2.24 ∗∗
	0.8 s	53.27	55.17	+1.89
	1.0 s	65.39	66.79	+1.39

Appendix

Table D.1: Forty-class accuracy (%) after RW training and IW evaluation.

\Delta

is LaS minus CLS in percentage points.

Window length $L$	10,000 random test windows per fold	Physical IW ( $\delta=0$ )
0.6 s	48.1171	36.5595
0.8 s	63.4886	55.1667
1.0 s	74.7371	66.7857

Appendix

Table D.2: Closed-set accuracy (%) for RW-trained LaS (MTSNet) under random-window and IW evaluation.

Method	0.6 s	0.7 s	0.8 s	0.9 s	1.0 s
FBCCA	28.25 ∗∗∗	38.20 ∗∗∗	48.54 ∗∗∗	56.90 ∗∗∗	63.75 ∗∗∗
EEGNet (CLS)	45.24 ∗∗∗	52.51 ∗∗∗	58.38 ∗∗∗	63.52 ∗∗∗	65.45 ∗∗∗
SSVEPFormer (CLS)	55.25	63.46	70.33	75.65	77.74
MTSNet (CLS)	52.06 ∗	62.61	68.61 ∗∗	73.98 ∗	78.01
LaS (MTSNet)	55.18	64.46	71.93	76.42	79.50

Appendix

Table D.3: Forty-class IW accuracy (%, subject mean) across five window lengths. Values and significance marks reproduce Appendix B.1 .

Method	0.6 s	0.7 s	0.8 s	0.9 s	1.0 s
FBCCA	41.0 ∗∗∗	60.4 ∗∗∗	80.0 ∗∗∗	94.5 ∗∗∗	104.7 ∗∗∗
EEGNet (CLS)	84.6 ∗∗∗	96.8 ∗∗∗	105.1 ∗∗∗	111.2 ∗∗∗	108.6 ∗∗∗
SSVEPFormer (CLS)	114.4	129.6	139.5	145.5	141.2 ∗
MTSNet (CLS)	105.7 ∗	126.1	134.7 ∗∗	140.6 ∗	142.3
LaS (MTSNet)	114.8	132.4	144.8	147.6	146.4

Appendix

Table D.4: Estimated ITR (bits/min, subject mean) for the same IW conditions. Values and significance marks reproduce Appendix B.1 .

IR trials / approximate training windows	IW accuracy
No IR / 6,000	34.7738
50 / 50,000	71.1667
100 / 100,000	73.3810
200 / 200,000	74.6905
300 / 300,000	76.3571

Appendix

Table D.5: IR training volume and IW closed-set accuracy (%). The no-IR condition uses approximately 6,000 training windows.

Mask ratio $\rho$	IW accuracy
0.0	72.8452
0.1	74.8571
0.2	76.0238
0.3	76.3571
0.4	75.3095
0.5	73.2500

Appendix

Table D.6: Temporal mask ratio and IW closed-set accuracy (%) with approximately 300,000 IR windows. The epoch budgets differ among mask conditions.

Figure D.1: Forty-class accuracy of a single 1.0-s Physical window at six offsets from the valid analysis onset

t_{0}=0.14

s. The neural models are trained with RW; FBCCA directly classifies each window.

Text encoder	State	0.6 s	0.8 s	1.0 s
BERT-base-uncased	Frozen	27.0200	42.3800	40.0100
BERT-base-uncased	LoRA	34.1500	59.3900	55.8700
EmbeddingGemma-300M	Frozen	34.3333	65.7857	65.6071
EmbeddingGemma-300M	LoRA	32.0119	58.0952	60.2619

Appendix

Table D.7: Five-class unseen-only ZSL accuracy (%) after IR training.

Neural Decoding

Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.

See all 184 papers

Sep 23, 2026

Brain-to-Language Decoding: Paradigms, Signals, Methods, Evaluation, Practical Use and Beyond

Brain-to-language decoding translates neural activity associated with language production, internal speech and perception into linguistic or expressive outputs. It offers a route to restoring communication after speech loss and a means of studying how the brain represents language. Advances in neural recording and representation learning have expanded the field from constrained recognition and acoustic reconstruction to text generation, streaming personalised speech and facial animation. This survey synthesises these developments across invasive and non-invasive measurements. We connect Articulated, Inner and Perceived paradigms to the neural populations they engage, the representations available to decoders and the outputs those representations can support. We examine model development, public resources and the evolution of evaluation, and compare published performance and communication costs within their reported protocols. The synthesis identifies complementary routes to progress: phonetic, acoustic and semantic targets preserve different aspects of a message; shared representations support reuse across recording conditions and tasks; and online communication increasingly depends on calibration, feedback and user control alongside decoding accuracy. Shared benchmarks enable algorithmic comparisons, while longitudinal studies reveal the demands of sustained use. We discuss these developments and their remaining limitations, then outline a prospective five-level trajectory from commands and language to meaning, scenarios and bidirectional cognitive exchange.

Yiqian Yang, Yiqun Duan, Chenyu Liu +4

Uploading Inc · Human-centric Artificial Intelligence Centre, University of Technology Sydney · Stanford Institute for Human-Centered Artificial Intelligence (HAI), Stanford University

Brain-Computer Interfaces Neural Decoding Brain-to-Speech Decoding

Ask PDF

Sep 25, 2026

Integrating Language Models into Listened and Imagined Speech Decoding from MEG

Decoding imagined speech is an important goal for brain-computer interfaces but remains challenging due to weak neural responses, low signal-to-noise ratio, and limited imagined-speech datasets. Language models provide strong contextual cues for text prediction, but how much they can help neural decoding and whether their contribution differs for decoding perceived and imagined speech remains unclear. To investigate this, we use a paired listened-imagined MEG dataset and incorporate language-model information at two stages. First, we train a contrastive neural decoder that aligns MEG representations with acoustic and contextual language representations, improving cross-subject word decoding for both listened and imagined speech. Second, at inference, we introduce a neural-constrained beam-search framework that combines neural evidence with language-model next-word probabilities. We find that imagined-speech decoding benefits more from the language model than listened-speech decoding. For Imagined speech, the best-performing balance between neural and language-model evidence shifts toward the language model, and the gain over neural-only decoding is larger. Together, these results suggest that language priors are most useful when neural evidence is weaker, making them particularly valuable for imagined-speech BCIs.

Maryam Maghsoudi, Sai Samrat Kankanala, Shihab A. Shamma +1

Department of Electrical and Computer Engineering, University of Maryland, College Park, MD, USA · LEAP Lab, Electrical Engineering, Indian Institute of Science, Bengaluru, India.

Neural Decoding Brain-to-Speech Decoding Brain-Computer Interfaces

Ask PDF

Apr 22, 2026

MoDAl: Self-Supervised Neural Modality Discovery via Decorrelation for Speech Neuroprosthesis

Speech neuroprosthesis systems decode intended speech from neural activity in the absence of audible output, offering a path to restoring communication for individuals with speech-impairing conditions. Current approaches decode predominantly from motor cortical areas, discarding others -- such as area 44, part of Broca's area -- that may encode complementary linguistic information. We introduce MoDAl (Modality Decorrelation and Alignment), a framework that discovers complementary neural modalities through the interplay of two objectives in a shared projection space. A contrastive loss aligns each of several parallel brain encoders with the text embeddings of a pretrained large language model (LLM), while a decorrelation loss prevents the encoders from coalescing to duplicative representations. We prove that these objectives are in productive tension: Contrastive alignment induces transitive modality coalescence, which decorrelation must counteract for the framework to discover diverse neurolinguistic modalities. On the Brain-to-Text Benchmark '24, MoDAl reduces word error rate (WER) from 26.3% to 21.6% compared to the previous best end-to-end method, with the gain from incorporating previously discarded area 44 signals arising entirely from the decorrelation mechanism. Analysis of the discovered modalities reveals functional specialization: Encoders receiving area 44 input capture structural and syntactic properties (sentence length, grammatical voice, wh-words), consistent with the neurolinguistic understanding of Broca's area.

Yuanhao Chen, Peter Chin

Dartmouth College Hanover, NH, USA

Brain-Computer Interfaces Neural Decoding Cross-Modal Alignment Neural Encoding Multimodal Contrastive Learning

Ask PDF

Abstract

Neural Decoding

Explore similar work

Brain-to-Language Decoding: Paradigms, Signals, Methods, Evaluation, Practical Use and Beyond

Integrating Language Models into Listened and Imagined Speech Decoding from MEG

MoDAl: Self-Supervised Neural Modality Discovery via Decorrelation for Speech Neuroprosthesis