cs.HCSep 29, 2026

From Neurons to Conversation: Speech Brain-Computer Interfaces

Authors: Moein Khajehnejad, Forough Habibollahi, Tommaso Boccato, Margarida Sousa, Michal Olak, Francesco Jamal Sheiban, Matteo Ferrante

Organizations: Tether Evo · Turner Institute for Brain and Mental Health, Monash University · The University of Rome Tor Vergata

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Speech brain-computer interfaces (BCIs) aim to restore communication by transforming neural activity related to speech, language, or communicative intent into external outputs such as text, synthesized voice, or avatar control. Recent advances in intracortical and electrocorticographic recording, deep sequence models, and language-model-assisted decoding have enabled rapid progress, including high-performance attempted-speech decoding and increasingly naturalistic speech synthesis. Yet these achievements also reveal that speech BCIs are not simply neural-to-text decoders. They are adaptive clinical systems in which neural representations, recording hardware, decoding architectures, language priors, feedback, and user learning interact over time. Here, we synthesize speech BCI research from a system-level perspective. We first examine the neural substrates of speech and language, emphasizing their hierarchical, distributed, temporally structured, and non-stationary organization. We then examine recording and decoding choices, closed-loop adaptation, evaluation, clinical translation, and ethics. Across these domains, we highlight recurring trade-offs between signal resolution and invasiveness, low-level motor and high-level semantic targets, decoder accuracy and user agency, and language-model fluency and faithful neural evidence. We argue the next generation of speech BCIs should be evaluated not only by offline accuracy, but also by robustness across sessions, calibration burden, latency, uncertainty, usability, and safeguards against unintended decoding. By reframing speech BCIs as adaptive, user-centred systems, we outline the interdisciplinary priorities spanning speech neuroscience, neural engineering, machine learning, clinical practice, and neuroethics needed to move from proof-of-concept decoding toward reliable, expressive, and controllable communication neuroprostheses.

Figures & tables

Figure 1: \sharpfont Speech BCI system stack: from neural sources to closed-loop communication. Speech BCIs can be viewed as a layered system linking neural sources, recording interfaces, decoding models, and user feedback. Bottom layer: neural sources and speech-related representations. Speech-related activity is distributed across cortical regions including ventral sensorimotor cortex (vSMC), associated with articulatory and motor-intent representations, and superior temporal gyrus (STG), associated with phonetic and acoustic processing. Middle layer: signal acquisition. Neural activity can be captured using recording technologies with different spatial, temporal, and clinical trade-offs, including penetrating microelectrode arrays (MEAs), electrocorticography (ECoG) grids, and non-invasive neuroimaging such as fMRI for whole-brain semantic mapping. Top layer: computational transformation. Recorded signals are preprocessed and passed through neural decoders, often combined with language or semantic priors, to generate communication outputs such as intended text or synthesized speech/audio. The dashed feedback pathway illustrates closed-loop operation, in which the user perceives the decoded output and adapts subsequent neural activity, strategy, and control over time.
Figure 2: \sharpfont Neuroanatomical, temporal, and representational structure of speech processing relevant to BCI. A. Schematic anatomical hierarchy of the speech production network. Speech-related processing spans high-level conceptual and executive control, lexical–semantic retrieval, phonological and auditory speech representations, speech planning and sequencing, sensorimotor integration, and low-level articulatory motor execution. Numbered regions include prefrontal cortex, posterior middle temporal gyrus (pMTG), posterior superior temporal gyrus/superior temporal sulcus (pSTG/STS), Area Spt, inferior frontal gyrus/Broca’s area (IFG), ventral premotor cortex (vPMC), supplementary and pre-supplementary motor areas (SMA/pre-SMA), and ventral sensorimotor cortex (vSMC). Arrows indicate primary production flow together with auditory and somatosensory/articulatory feedback pathways. B. Temporal progression through the speech hierarchy. Speech-related neural dynamics evolve from early acoustic–phonological analysis and auditory monitoring toward partially overlapping stages of auditory–motor interfacing, lexical access, phonological selection, sequencing, articulatory planning, and later contextual or conceptual stabilization. The schematic timeline emphasizes that these stages overlap rather than forming a strictly serial cascade. C. Conceptual latent spaces across the speech hierarchy. Different representational levels may support distinct decodable geometries in neural population activity: semantic/conceptual representations may separate categories such as animals versus tools or actions versus objects; phonological/auditory–motor representations may separate speech sounds such as /ba/ and /ga/; and motor representations in vSMC may separate articulatory classes such as labial versus coronal/dental gestures. These latent spaces are schematic rather than data-derived and are intended to illustrate how the choice of decoding target selects different levels of the speech hierarchy.
Figure 3: \sharpfont Methods and evolution of speech BCI decoding. A. Taxonomy of decoding models and training strategies. Speech BCI pipelines span three complementary levels. Machine-learning and probabilistic approaches provide feature-based mappings from neural activity to speech-related targets, including regression and classification models, Bayesian decoders, HMMs and state-space models, and Kalman filters. Deep learning approaches learn representations directly from neural time series using CNNs, recurrent and temporal convolutional networks, transformers or conformers, and encoder–decoder architectures. Training and inference strategies determine how neural activity is aligned with speech outputs and how prior information is incorporated, including frame-wise supervision, CTC, sequence-to-sequence or autoregressive decoding, language-model fusion, Bayesian inference, and transfer or self-supervised learning. Importantly, these strategies are distinct from the underlying model architecture; for example, CTC and language-model fusion specify alignment or inference mechanisms rather than decoder architectures. B. Evolving landscape of speech BCI research. Representative studies are positioned by approximate publication year and recording modality, including electrocorticography (ECoG), intracortical microelectrode-array systems (MEA), and non-invasive approaches such as fMRI, EEG, and MEG. The timeline highlights major methodological and decoding milestones, from articulatory, phonetic, acoustic, and semantic mapping to speech synthesis, large-vocabulary text decoding, rapid calibration, streaming speech or audio generation, and foundation-model-based approaches. Background shading distinguishes an earlier mapping and feasibility period from the more recent LLM and foundation-model era , reflecting a broader shift toward high-throughput, clinically oriented, scalable, and language-aware communication systems. The timeline is selective rather than exhaustive and summarizes major trends across the field.
Table 1: Comprehensive Empirical Landscape: Anatomical Targets, Modalities, and Data Granularity
Figure 4: \sharpfont Closed-loop co-adaptation in speech BCIs. A. Conventional one-way decoding treats the decoder as a feedforward mapping from neural activity to speech output, with the user reacting only after the output is produced. In contrast, interactive closed-loop speech BCIs embed decoding within a feedback loop: decoded text or synthesized speech is perceived by the user and can shape subsequent neural activity, behavior, and control strategy. B. Co-adaptive operation arises from coupled brain/user adaptation and decoder adaptation. Brain/user adaptation includes neural plasticity, strategy adaptation, reinforcement-driven learning, and neural re-tuning, whereas decoder adaptation includes online learning, recalibration, drift tracking, uncertainty-aware updates, and pseudo-labeling/self-training. These processes interact through a shared system state defined by latency, feedback modality, decoder uncertainty, user burden, and stability. C. Feedback latency and feedback quality/modality richness shape distinct operating regimes. Low-latency, informative feedback supports stable and learnable interaction, shown as the green region. High-quality but delayed feedback can create high-burden regimes, shown in yellow, because rich feedback may still be difficult to use when temporal credit assignment is disrupted. Delayed and low-quality feedback can produce unstable regimes, shown in red, where control becomes unreliable and learning may degrade. The blank lower-left region represents fast but low-richness feedback: such feedback may be usable for simple correction or coarse control, but is not assumed to provide sufficient information for robust, stable speech learning. D. Across longer timescales, neural drift, electrode degradation, and biological signal changes can degrade performance when decoders remain fixed. Periodic recalibration and co-adaptive updating can help maintain performance during long-term or lifelong BCI operation.
Table 2: \sharpfont Evaluation and clinical translation considerations for speech BCIs
More in this topic

Brain-Computer Interfaces

Brain-computer interfaces translate neural activity into commands or feedback for communication, movement, rehabilitation, and sensory restoration.

See all 128 papers

Explore similar work

CardsList
Mar 4, 2026

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding

Decoding continuous speech from intracortical recordings is a central challenge for brain-computer interfaces (BCIs), with transformative potential for individuals with conditions that impair their ability to speak. While recent microelectrode array (MEA) decoders achieve impressive accuracy, their performance is fundamentally limited by the small size of existing datasets, they remain brittle to session-to-session variability, and their ability to generalize across participants remains unexplored. We introduce BrainWhisperer, a neural speech decoder that integrates high-resolution MEA recordings with a large pretrained automatic speech recognition (ASR) model. Building on interpretability findings showing that Whisper's encoder learns phoneme-selective representations with localized attention, we train a customized version of Whisper, modified to process neural features, using a hybrid objective that combines CTC loss on phonemes--predicted from the third encoder layer--and cross-entropy loss on word tokens. We introduce domain-informed modifications including windowed self-attention to capture articulatory continuity, hierarchical month/day-specific low-rank projections to address non-stationarity, and subject-specific embedders enabling cross-subject training. Evaluated on a publicly available MEA dataset (Card et al.), BrainWhisperer matches or outperforms prior state-of-the-art decoders. Critically, cross-dataset training improves performance even on individual datasets without fine-tuning, demonstrating unprecedented generalization. The model supports dual decoding paths: a high-accuracy phoneme-based path with external language model rescoring, and a fast direct text generation path enabling sub-100ms inference with minimal hardware requirements.

Tommaso Boccato, Michal Olak, Matteo Ferrante

Tether Evo

Brain-to-Speech DecodingBrain-Computer InterfacesNeural DecodingAutomatic Speech Recognition
AskPDF
Mar 18, 2026

ALIGN: Adversarial Learning for Generalizable Speech Neuroprosthesis

Intracortical brain-computer interfaces (BCIs) can decode speech from neural activity with high accuracy when trained on data pooled across recording sessions. In realistic deployment, however, models must generalize to new sessions without labeled data, and performance often degrades due to cross-session nonstationarities (e.g., electrode shifts, neural turnover, and changes in user strategy). In this paper, we propose ALIGN, a session-invariant learning framework based on multi-domain adversarial neural networks for semi-supervised cross-session adaptation. ALIGN trains a feature encoder jointly with a phoneme classifier and a domain classifier operating on the latent representation. Through adversarial optimization, the encoder is encouraged to preserve task-relevant information while suppressing session-specific cues. We evaluate ALIGN on intracortical speech decoding and find that it generalizes consistently better to previously unseen sessions, improving both phoneme error rate and word error rate relative to baselines. These results indicate that adversarial domain alignment is an effective approach for mitigating session-level distribution shift and enabling robust longitudinal BCI decoding.

Zhanqi Zhang, Shun Li, Bernardo L. Sabatini +2

Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Boston, Massachusetts, USA · Halıcıo˘glu Data Science Institute, Department of Electrical and Computer Engineering, Department of Computer Science and Engineering, University of California San Diego, La Jolla, California, USA

Brain-Computer InterfacesNeural DecodingDomain-Adversarial LearningAdversarial Training
AskPDF
Oct 16, 2026

Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling

Recent speech brain-computer interfaces (BCIs) increasingly rely on large models and complex multi-stage pipelines for neural decoding. However, in speech BCI, where training data are scarce, noisy, and expensive to collect, the benefits of model scaling remain unclear. This work revisits a fundamental question: are large models truly necessary for accurate neural decoding? We propose BEST (Brain-Encoding-Speech-Text), a two-stage framework for efficient neural-to-text decoding. Instead of relying on large generative models or phoneme-centric pipelines, BEST introduces hierarchical intermediate representations and a compact speech-oriented decoder that directly maps neural activity to text. This design preserves essential structure while significantly reducing computational cost, enabling robust generalization in low-resource settings and practical deployment under constrained resources. On the Brain-to-Text '24 and '25 benchmarks, BEST achieves word error rates (WERs) of 6.19% and 1.62%, respectively, using only 3.85% of the parameters required by comparable methods. Beyond these results, we provide three key insights: (1) decoder scaling yields non-monotonic gains; (2) cross-modal alignment is more critical than model size; and (3) overfitting degards sequence-level performance. Representation analysis further reveals a consistent neural-audio-text alignment pathway.

Zehui Feng, Weichuan Wang, Xiaohan Chen +2

Shanghai Jiao Tong University · Nanyang Technological University · Zhejiang University

Neural DecodingBrain-to-Speech DecodingLanguage Model ScalingCross-Modal Representation Learning
AskPDF