ALIGN: Adversarial Learning for Generalizable Speech Neuroprosthesis

Zhanqi Zhang; Shun Li; Bernardo L. Sabatini; Mikio Aoi; Gal Mishne

cs.LGMar 18, 2026

ALIGN: Adversarial Learning for Generalizable Speech Neuroprosthesis

Authors: Zhanqi Zhang, Shun Li, Bernardo L. Sabatini, Mikio Aoi, Gal Mishne

Organizations: Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Boston, Massachusetts, USA · Halıcıo˘glu Data Science Institute, Department of Electrical and Computer Engineering, Department of Computer Science and Engineering, University of California San Diego, La Jolla, California, USA

Abstract

Intracortical brain-computer interfaces (BCIs) can decode speech from neural activity with high accuracy when trained on data pooled across recording sessions. In realistic deployment, however, models must generalize to new sessions without labeled data, and performance often degrades due to cross-session nonstationarities (e.g., electrode shifts, neural turnover, and changes in user strategy). In this paper, we propose ALIGN, a session-invariant learning framework based on multi-domain adversarial neural networks for semi-supervised cross-session adaptation. ALIGN trains a feature encoder jointly with a phoneme classifier and a domain classifier operating on the latent representation. Through adversarial optimization, the encoder is encouraged to preserve task-relevant information while suppressing session-specific cues. We evaluate ALIGN on intracortical speech decoding and find that it generalizes consistently better to previously unseen sessions, improving both phoneme error rate and word error rate relative to baselines. These results indicate that adversarial domain alignment is an effective approach for mitigating session-level distribution shift and enabling robust longitudinal BCI decoding.

Figures & tables

Figure 1: Dataset and domain shift across sessions. a. Attempted speech dataset. Intracortical neural activity was recorded from a participant with amyotrophic lateral sclerosis (ALS) while they attempted to speak sentences presented on a screen. Although vocalizations were produced, the speech was not intelligible. b. Example neural activity corresponding to the same attempted sentence recorded in two different sessions, illustrating session-dependent neural dynamics variabilities. c. Illustration of distribution shift in source sessions and target sessions caused by nonstationarities in neural dynamics and the session-invariant distributions with adaptation. d. t-SNE visualization of the latent embeddings used for phoneme decoding (input to the phoneme classifier), showing source sessions (blue) and target sessions (red) before and after ALIGN.

Figure 2: ALIGN model architecture. Our ALIGN model consists of three major modules. The feature encoder

f

and phoneme classifier

p

backbone shown here is instantiated with a Transformer-based decoder ( Feghhi et al. [2025] ). We added a domain classifier

d

, which takes the transformer encoder embedding as input and the multi-head classifiers are trained to predict whether the input is coming from source or target sessions. A gradient reversal layer is used to make the encoder learn session-invariant features.

Figure 3: Embedding visualization before and after ALIGN. ( a ) PCA projections of the intermediate latent embeddings produced by the feature encoder in the transformer baseline decoder for T12 (top) and in ALIGN (bottom). Twelve source sessions are shown in color, along with three target sessions shown in gray. ( b ) Corresponding embeddings of final latent embeddings in both model.

Figure 4: ALIGN with and without TTA. Test WER of GRU baseline (green) without TTA, transformer baseline (blue) and ALIGN model (orange) with and without TTA (dark and light shades), where TTA is trained from the first target session (left) or from the first test session (right). All models are tested on three different train-test partition for 5 seeds.

	WER (%) (mean $\pm$ std)
Days from last train	GRU Baseline	ALIGN
TTA from First Test Session
71	51.28 $\pm$ 0.95	45.40 $\pm$ 0.00
84	86.71 $\pm$ 2.29	65.65 $\pm$ 0.19
86	99.11 $\pm$ 0.11	71.79 $\pm$ 0.68

Table 1: T15 Test Word Error Rate (WER) (across 5 seeds)

Ablation	Validation PER (%)
ALIGN	$26.30\pm 0.52$
w/o adversarial loss	$31.95\pm 0.42$
w/o temporal stretch augmentation	$27.70\pm 0.67$

Table 2: Ablation study on validation phoneme error rate (PER). Models are trained with 12 source sessions and 4 target sessions. 4 seeds were ran for each condition. Validation PER are shown in mean percentage

\pm

std.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Per-day phoneme error rate (PER). Mean and std of PER of the transformer baseline (blue) and ALIGN (orange) on T12 12–8–3 dataset. The first eight sessions correspond to validation of target sessions, and the last three correspond to held-out test sessions.

Figure 6: Dataset Partition. Left: In the official Brain-to-Text Benchmark ’24 partition, each recording day is internally divided into train, validation, and competitionHoldOut (test) blocks (except for a few sessions that lack a test block), so most sessions contribute data to every partition. Right: We reorganize data by session into source (train), unlabeled target (for unsupervised alignment), labeled validation (for model selection via validation PER), and fully unseen test sessions to evaluate cross-session generalization.

Figure 7: ALIGN on target and test sessions. Validation and test WER (%) of GRU baseline (green) without TTA, transformer baseline (blue) and ALIGN model (orange) with and without TTA (dark and light shades), where TTA is trained from the first target session (left) or from the first test session (right). All models are tested on three different train-test partition for 5 seeds.

Figure 8: Embedding analysis of TSA-trained decoder. Feature-wise Wasserstein distance (WD) between augmented embeddings and a held-out set as a function of the stretch factor. Left: WD computed between source embeddings and target embeddings (average across the first 10 embedding dimensions); Right: WD computed between validation embeddings and target embeddings (average across the first 10 embedding dimensions). Thick lines show the mean across days and shaded bands denote

\pm

SEM; faint lines show individual day traces. Gray indicates the decoder trained on data without TSA, and green indicates decoder trained on data with TSA.

Figure 9: ALIGN reduces the session distributional gap. Top-

k

most session-distinct embedding dimensions before (left) and after ALIGN (right).

Parameter	Value
Optimizer	AdamW
Learning rate	0.001
Weight decay	$1\times 10^{-5}$
Batch size	64
Epochs	500
Scheduler	Multistep (milestone=300)

Appendix

Table 3: Transformer Decoder Training and Domain Adaptation Hyperparameters

Parameter	T12	T15
Optimizer	Adam	AdamW
Learning rate scheduler	None	Cosine
Learning rate warmup batches	–	1000
Training	73 epochs	120000 batches
Batch size	64	64
$\beta_{0},\beta_{1}$	(0.9, 0.999)	(0.9, 0.999)

Appendix

Table 4: GRU Decoder Training and Domain Adaptation Hyperparameters for T12 and T15

Brain-Computer Interfaces

Brain-computer interfaces translate neural activity into commands or feedback for communication, movement, rehabilitation, and sensory restoration.

See all 128 papers

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding
Mar 4, 2026Tommaso Boccato, Michal Olak, Matteo FerranteBrain-to-Speech Decoding Brain-Computer Interfaces
Tether Evo
From Neurons to Conversation: Speech Brain-Computer Interfaces
Sep 29, 2026Moein Khajehnejad, Forough Habibollahi, Tommaso Boccato +4Brain-Computer Interfaces Neural Decoding
Tether Evo · Turner Institute for Brain and Mental Health, Monash University · The University of Rome Tor Vergata
Stable Neural Decoding Across Sessions via Task-Conditioned Latent Alignment for Brain-Machine Interfaces
Sep 23, 2026Canyang Zhao, Bolin Peng, J. Patrick Mayo +2Brain-Computer Interfaces Neural Decoding
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Departments of Ophthalmology and Bioengineering, University of Pittsburgh, Pittsburgh, USA · School of Science and Engineering, Chinese University of Hong Kong, Shenzhen, China

Mar 4, 2026J/K move · Enter open · S save

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding

Tommaso Boccato, Michal Olak, Matteo Ferrante

Tether Evo

Brain-to-Speech Decoding Brain-Computer Interfaces Neural Decoding Automatic Speech Recognition

Open paper AskPDF arXiv

Decoding continuous speech from intracortical recordings is a central challenge for brain-computer interfaces (BCIs), with transformative potential for individuals with conditions that impair their ability to speak. While recent microelectrode array (MEA) decoders achieve impressive accuracy, their performance is fundamentally limited by the small size of existing datasets, they remain brittle to session-to-session variability, and their ability to generalize across participants remains unexplored. We introduce BrainWhisperer, a neural speech decoder that integrates high-resolution MEA recordings with a large pretrained automatic speech recognition (ASR) model. Building on interpretability findings showing that Whisper's encoder learns phoneme-selective representations with localized attention, we train a customized version of Whisper, modified to process neural features, using a hybrid objective that combines CTC loss on phonemes--predicted from the third encoder layer--and cross-entropy loss on word tokens. We introduce domain-informed modifications including windowed self-attention to capture articulatory continuity, hierarchical month/day-specific low-rank projections to address non-stationarity, and subject-specific embedders enabling cross-subject training. Evaluated on a publicly available MEA dataset (Card et al.), BrainWhisperer matches or outperforms prior state-of-the-art decoders. Critically, cross-dataset training improves performance even on individual datasets without fine-tuning, demonstrating unprecedented generalization. The model supports dual decoding paths: a high-accuracy phoneme-based path with external language model rescoring, and a fast direct text generation path enabling sub-100ms inference with minimal hardware requirements.

Abstract

Brain-Computer Interfaces

Explore similar work

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding

From Neurons to Conversation: Speech Brain-Computer Interfaces

Stable Neural Decoding Across Sessions via Task-Conditioned Latent Alignment for Brain-Machine Interfaces