cs.CLMar 10, 2026

Decoding the decoder: Contextual sequence-to-sequence modeling for intracortical speech decoding

Authors: Michal Olak, Tommaso Boccato, Matteo Ferrante

Organizations: Tether Evo

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Speech brain--computer interfaces require decoders that translate intracortical activity into linguistic output while remaining robust to limited data and day-to-day variability. While prior high-performing systems have largely relied on framewise phoneme decoding combined with downstream language models, it remains unclear what contextual sequence-to-sequence decoding contributes to sublexical neural readout, robustness, and interpretability. We evaluated a multitask Transformer-based sequence-to-sequence model for attempted speech decoding from area 6v intracortical recordings. The model jointly predicts phoneme sequences, word sequences, and auxiliary acoustic features. To address day-to-day nonstationarity, we introduced the Neural Hammer Scalpel (NHS) calibration module, which combines global alignment with feature-wise modulation. We further analyzed held-out-day generalization and attention patterns in the encoder and decoders. On the Willett et al. dataset, the proposed model achieved a state-of-the-art phoneme error rate of 14.3%. Word decoding reached 25.6% WER with direct decoding and 19.4% WER with candidate generation and rescoring. NHS substantially improved both phoneme and word decoding relative to linear or no day-specific transform, while held-out-day experiments showed increasing degradation on unseen days with temporal distance. Attention visualizations revealed recurring temporal chunking in encoder representations and distinct use of these segments by phoneme and word decoders. These results indicate that contextual sequence-to-sequence modeling can improve the fidelity of neural-to-phoneme readout from intracortical speech signals and suggest that attention-based analyses can generate useful hypotheses about how neural speech evidence is segmented and accumulated over time.

Figures & tables

Figure 1: Model overview. Spike counts and spike-band power are processed in parallel convolutional branches, merged into a shared latent sequence, and reweighted by a lightweight content-dependent gate. The NHS day-specific transform then aligns the sequence by combining a global affine transformation ( hammer ) with FiLM-style feature-wise modulation ( scalpel ) through a learned gate. The calibrated representation is encoded by a 6-layer Transformer and decoded by three heads: an autoregressive phoneme decoder, a BART-based word decoder, and an auxiliary MFCC prediction head attached to an intermediate encoder layer.
Table 1: Comparison of phoneme- and word-level decoding performance. Published reference systems are shown separately from controlled internal baselines and day-specific transform ablations. Our results are reported as mean ±\pm SD across 5 random seeds. Published results are taken from the respective papers and are not directly seed-matched to our runs.
Figure 2: Held-out-day generalization. PER/WER deltas on held-out days 22–24 relative to a 24-day model evaluated with day-specific transforms (0%). Blue curves ( seen data + proximal ) show the same model re-evaluated on these days using the transform from day 21. Green curves ( unseen data + proximal ) show a 21-day model evaluated on days 22–24 with the day-21 transform, after subtracting the mean 21- vs. 24-day performance gap on days 1–21. Mean over 5 seeds, shaded regions denote 95% bootstrap confidence intervals.
Figure 3: Attention analysis. Left: encoder self-attention for layers 1, 3, and 6. Middle: cross-attention for the BART and phoneme decoders. Right: decoder self-attention. Dotted vertical lines align with salient envelope events. Late encoder layers exhibit clean diagonals and “boxed” segments that decoders repeatedly attend to; phoneme self-attention is local, while BART self-attention is broadly triangular. The trial text is “for a man of his mass he was curiously short” ; target words are approximately aligned to envelope features, as no ground-truth timing annotations are available in this dataset.
Table 2: Inference time on the test set (880 sentences; 5,166 words). Times are wall-clock.
Figure 4: Scaling behavior and power-law extrapolation. Left: Phoneme (PER) and word (WER) error rates for six dataset fractions {0.05, 0.10, 0.25, 0.50, 0.75, 1.00}\{0.05,\,0.10,\,0.25,\,0.50,\,0.75,\,1.00\} , plotted as corresponding training-set sizes, with 95% bootstrap confidence intervals across seeds. Right: Power-law models fitted to the multi-seed mean curve (excluding the 0.05 fraction) with bootstrap 95% confidence bands and extrapolated performance up to 100,000 training trials.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Table 3: Example word-level predictions of Transformer+MFCC+BART model (no rescoring) from 10th, 50th (median) and 90th WER percentiles.
Table 4: Performance comparison between different strategies of BART initialization and training. Results reported as mean ± SD across 5 global seeds
Table 5: Word-level performance comparison between strategies of candidate selection. Score-based results are obtained by selection of candidates generated using beam search and stochastic sampling strategies. All results reported as mean ± SD across 5 global seeds
Figure 5: Self and cross attention: Usage of temporal chunking. Each cell shows vertically stacked 1) Encoder self-attention in the last layer, 2) Phoneme Decoder cross-attention, 3) BART word decoder cross-attention, 4) Audio envelope, obtained from Transformer Seq2Seq + MFCC + BART model, visualized for 20 randomly chosen validation trials. The pattern of temporal chunking of neural signal in the encoder and the downstream usage of those chunks by both decoders is prevalent across all samples - as well as the temporal lag of word decoder relative to the phoneme decoder, indicating evidence accumulation.
Figure 6: Self-attention: Temporal chunking. Self-attention map of the last (6th) layer of encoder across model variants (columns) visualized for random 15 trials (rows). The representation pattern of temporal chunking of the neural signal into ”boxes” is prevalent across the trials regardless of the auxiliary objectives
Table 6: Training and model parameters for the two-stage training process of Transformer+MFCC+BART model. Stage 1 focuses on pre-training the ECoG encoder with an auxiliary MFCC prediction task alongside the main phoneme decoding task. Stage 2 introduces a BART-based word decoder head and fine-tunes the entire model jointly on phoneme and text decoding. The Stage 2 column only lists values in Multi-Task Setup section, as only then they differ from Stage 1.
More in this topic

Sequence-to-Sequence Models

See all 11 papers

Explore similar work

CardsList
Mar 4, 2026

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding

Decoding continuous speech from intracortical recordings is a central challenge for brain-computer interfaces (BCIs), with transformative potential for individuals with conditions that impair their ability to speak. While recent microelectrode array (MEA) decoders achieve impressive accuracy, their performance is fundamentally limited by the small size of existing datasets, they remain brittle to session-to-session variability, and their ability to generalize across participants remains unexplored. We introduce BrainWhisperer, a neural speech decoder that integrates high-resolution MEA recordings with a large pretrained automatic speech recognition (ASR) model. Building on interpretability findings showing that Whisper's encoder learns phoneme-selective representations with localized attention, we train a customized version of Whisper, modified to process neural features, using a hybrid objective that combines CTC loss on phonemes--predicted from the third encoder layer--and cross-entropy loss on word tokens. We introduce domain-informed modifications including windowed self-attention to capture articulatory continuity, hierarchical month/day-specific low-rank projections to address non-stationarity, and subject-specific embedders enabling cross-subject training. Evaluated on a publicly available MEA dataset (Card et al.), BrainWhisperer matches or outperforms prior state-of-the-art decoders. Critically, cross-dataset training improves performance even on individual datasets without fine-tuning, demonstrating unprecedented generalization. The model supports dual decoding paths: a high-accuracy phoneme-based path with external language model rescoring, and a fast direct text generation path enabling sub-100ms inference with minimal hardware requirements.

Tommaso Boccato, Michal Olak, Matteo Ferrante

Tether Evo

Brain-to-Speech DecodingBrain-Computer InterfacesNeural DecodingAutomatic Speech Recognition
AskPDF
May 23, 2026

End-to-End Intracortical Speech Decoding from Neural Activity

Current high-performing intracortical speech neuroprostheses achieve low word error rates but typically rely on external language models during inference, increasing memory, computation, and latency. In this work, we investigate whether meaningful character-level decoding is achievable without such models. We propose an end-to-end Conformer-based neural decoder trained directly on intracortical recordings from a participant with amyotrophic lateral sclerosis (ALS). Without any external language model, the system achieves a character error rate (CER) of 23.80% on held-out validation data. Analysis shows that performance variability is driven by inter-session signal degradation, while dominant errors arise from incorrect word boundary segmentation. These results demonstrate that effective character-level decoding is possible in a fully end-to-end framework, providing a strong neural signal for downstream linguistic processing.

Owais Mujtaba Khanday, Jose A. Gonzalez-Lopez, Marc Ouellet +2

1Dpt. of Signal Theory, Telematics and Communications, University of Granada, Spain · 2Brain, Mind, and Behavior Research Center, University of Granada, Spain · 3Grupo CSUR de Epilepsia Refractaria, Hospital Virgen de Las Nieves (Granada), Spain

Brain-Computer InterfacesNeural DecodingAutomatic Speech RecognitionConformerBrain-to-Speech Decoding
AskPDF
Mar 18, 2026

ALIGN: Adversarial Learning for Generalizable Speech Neuroprosthesis

Intracortical brain-computer interfaces (BCIs) can decode speech from neural activity with high accuracy when trained on data pooled across recording sessions. In realistic deployment, however, models must generalize to new sessions without labeled data, and performance often degrades due to cross-session nonstationarities (e.g., electrode shifts, neural turnover, and changes in user strategy). In this paper, we propose ALIGN, a session-invariant learning framework based on multi-domain adversarial neural networks for semi-supervised cross-session adaptation. ALIGN trains a feature encoder jointly with a phoneme classifier and a domain classifier operating on the latent representation. Through adversarial optimization, the encoder is encouraged to preserve task-relevant information while suppressing session-specific cues. We evaluate ALIGN on intracortical speech decoding and find that it generalizes consistently better to previously unseen sessions, improving both phoneme error rate and word error rate relative to baselines. These results indicate that adversarial domain alignment is an effective approach for mitigating session-level distribution shift and enabling robust longitudinal BCI decoding.

Zhanqi Zhang, Shun Li, Bernardo L. Sabatini +2

Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Boston, Massachusetts, USA · Halıcıo˘glu Data Science Institute, Department of Electrical and Computer Engineering, Department of Computer Science and Engineering, University of California San Diego, La Jolla, California, USA

Brain-Computer InterfacesNeural DecodingDomain-Adversarial LearningAdversarial Training
AskPDF