Organizations: Computer Science and Technology, Zhejiang University, Hangzhou, Zhejiang, China. · Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, China.
Speech production and perception are the main ways humans communicate daily. Prior brain-to-text decoding studies have largely focused on a single modality and alphabetic languages. Here, we present a unified brain-to-sentence decoding framework for both speech production and perception in Mandarin Chinese. The framework exhibits strong generalization ability, enabling sentence-level decoding when trained only on single-character data and supporting characters and syllables unseen during training. In addition, it allows direct and controlled comparison of neural dynamics across modalities. Mandarin speech is decoded by first classifying syllable components in Hanyu Pinyin, namely initials and finals, from neural signals, followed by a post-trained large language model (LLM) that maps sequences of toneless Pinyin syllables to Chinese sentences. To enhance LLM decoding, we designed a three-stage post-training and two-stage inference framework based on a 7-billion-parameter LLM, achieving overall performance that exceeds larger commercial LLMs with hundreds of billions of parameters or more. In addition, several characteristics were observed in Mandarin speech production and perception: speech production involved neural responses across broader cortical regions than auditory perception; channels responsive to both modalities exhibited similar activity patterns, with speech perception showing a temporal delay relative to production; and decoding performance was broadly comparable across hemispheres. Our work not only establishes the feasibility of a unified decoding framework but also provides insights into the neural characteristics of Mandarin speech production and perception. These advances contribute to brain-to-text decoding in logosyllabic languages and pave the way toward neural language decoding systems supporting multiple modalities.
Figures & tables
Figure 1: Overview of our brain-to-sentence framework. a,b, Data acquisition paradigm. a, Task alternating between listening to and speaking individual Chinese characters. b, Task alternating between listening to and speaking complete Mandarin sentences. c,e, Training pipeline of the decoding framework. c, the brain decoder is trained to classify syllable components (initials and finals) from neural signals. We adopted NeuroSketch, a 2D-CNN-based neural decoder developed in our concurrent work. e, The LLM is post-trained to generate full sentences from syllable sequences. d,f, Inference pipeline of the decoding framework. d, Beam search is applied to the probabilistic outputs of the brain decoder to generate multiple syllable sequence candidates, from which the top-20 candidates are retained. f, The top-20 candidates are fed into the post-trained LLM, which performs a two-stage inference process to produce the final decoded sentence. g, The proposed framework exhibits hierarchical, character-level, and syllable-level generalization abilities.
Figure 2: Channel contribution analysis. a, Histogram showing the distribution of relative improvement over chance level for all channels in the speaking and listening tasks. Three improvement thresholds, 0%, 50%, and 100%, are indicated, corresponding to low-, medium-, and high-contribution channels, respectively. b, Spatial distributions of high-contribution channels identified during speech production and perception. c, Violin plots comparing the decoding performance between speaking and listening tasks for channels that were highly responsive to both speech production and perception. d, For the subset of channels in (c) exhibiting stable temporal relationships between modalities, a heatmap illustrating the maximum correlation between speaking- and listening-evoked neural signals and the corresponding time lags. e, Box plots showing the distribution of time lags across different syllables for each channel with stable speaking–listening delays. f, Violin plots comparing decoding performance between left and right hemispheric channels across different contribution thresholds.
Figure 3: Decoding of initials and finals. a, Bar plots showing the initial and final classification accuracies of NeuroSketch and three other brain decoders in the speaking and listening tasks. For each task, results are shown for participants with available high-contribution channels (12 out of 12 for speaking, 10 out of 12 for listening). Complete results for all participants are provided in the Supplementary Tables. b, Bar plots illustrating syllable generalization performance of NeuroSketch. Classification accuracies of initials and finals are compared between in-domain (ID) and out-of-domain (OOD) syllables for participants evaluated on the second corpus.
Figure 4: Sentence decoding. a, Proportion distributions of SER for the top-20 syllable-sequence candidates generated by NeuroSketch and the three other brain decoder architectures in the speaking (12 participants) and listening (10 participants) tasks. Complete results of our model for all participants are provided in the Supplementary Tables. b, Scatter plot illustrating the relationship between the relative improvement in initial and final classification accuracies and the relative improvement in EMP among the top-20 candidates. c,d, Results for sentence-level decoding. To ensure that the input candidate sets contain a minimal threshold of reliable information for sentence decoding, we focused on participants whose average proportion of high-quality candidates (SER <0.3 ) across the decoding corpus exceeded 10% (7 out of 12 for speaking, 6 out of 12 for listening). Complete results of our model for all participants are provided in the Supplementary Tables. c, Bar plots comparing CER of our LLM and other LLMs on the syllable-to-sentence decoding task, including small-sized LLM (Qwen2.5-7B-Instruct), medium-sized LLMs (Qwen2.5-72B-Instruct), and large-scale commercial LLMs (Qwen-3 Max, Deepseek-v3.2-exp, Doubao-1.6, GPT-5-chat-latest, Grok-4-fast, Llama-3.1). d, Bar plots presenting ablation results of our syllable-to-sentence decoding framework. The Baseline method used an IME-style decoder based on lexicon- and language-model–driven search. The Top-1, Top-3, and Top-20 correction groups performed direct correction of the top-k beam search candidates. Auxiliary component ablations, labeled “w/o translation” and “w/o vocabulary,” were created by removing the respective components from the full framework.
Figure 5: Decoding of tones. a, Bar plots showing tone classification accuracy in the speaking and listening tasks. b, Proportion distributions of syllable error rates for the top-20 syllable sequence candidates with and without tone information. c, Bar plots comparing exact match probability under decoding with and without tone information.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Spatial distribution of sEEG electrodes and their contributions across the brain. Electrodes from 12 subjects are visualized from three brain perspectives. Heatmaps indicate each channel’s F1 score in the speaking and listening tasks during initial, final, and tone decoding.
Figure 7: Architectures of the four brain decoders. The input to each brain decoder is an sEEG signal with C channels and L time steps. For each stage of the decoder, we indicate the number of output channels, the number of time steps, and the corresponding number of blocks.
Figure 8: Initial-final classification results . Bar plots illustrate the initial and final classification F1 scores achieved by four different brain decoders for speaking and listening tasks.
Figure 9: Tone classification results . Bar plots illustrate the tone classification F1 scores achieved by four different brain decoders for speaking and listening tasks.
Brain-to-language decoding translates neural activity associated with language production, internal speech and perception into linguistic or expressive outputs. It offers a route to restoring communication after speech loss and a means of studying how the brain represents language. Advances in neural recording and representation learning have expanded the field from constrained recognition and acoustic reconstruction to text generation, streaming personalised speech and facial animation. This survey synthesises these developments across invasive and non-invasive measurements. We connect Articulated, Inner and Perceived paradigms to the neural populations they engage, the representations available to decoders and the outputs those representations can support. We examine model development, public resources and the evolution of evaluation, and compare published performance and communication costs within their reported protocols. The synthesis identifies complementary routes to progress: phonetic, acoustic and semantic targets preserve different aspects of a message; shared representations support reuse across recording conditions and tasks; and online communication increasingly depends on calibration, feedback and user control alongside decoding accuracy. Shared benchmarks enable algorithmic comparisons, while longitudinal studies reveal the demands of sustained use. We discuss these developments and their remaining limitations, then outline a prospective five-level trajectory from commands and language to meaning, scenarios and bidirectional cognitive exchange.
Yiqian Yang, Yiqun Duan, Chenyu Liu +4
Uploading Inc · Human-centric Artificial Intelligence Centre, University of Technology Sydney · Stanford Institute for Human-Centered Artificial Intelligence (HAI), Stanford University
The loss of speech limits communication for individuals with paralysis. Direct neural-to-speech synthesis is challenging due to the limited availability of neural data for training speech brain-computer interfaces. Most existing systems rely on cascaded neural-to-text-to-speech pipelines, which increase inference latency and propagate errors across stages. We present Brain2Speech-Net, a single-stage neural-to-speech generation framework without intermediate text decoding. We use a differentiable phoneme bottleneck and a deep-HMM alignment mechanism to map long neural recordings into the latent space of a text-to-speech (TTS) model, enabling high-quality speech synthesis. Brain2Speech-Net is the only system in our comparison that produces intelligible speech while generating faster than real time.
Shreeram Suresh Chandra, Zexin Cai, Yu Tsao +2
Johns Hopkins University, USA · Academia Sinica, Taiwan · University of Edinburgh, UK
Decoding imagined speech is an important goal for brain-computer interfaces but remains challenging due to weak neural responses, low signal-to-noise ratio, and limited imagined-speech datasets. Language models provide strong contextual cues for text prediction, but how much they can help neural decoding and whether their contribution differs for decoding perceived and imagined speech remains unclear. To investigate this, we use a paired listened-imagined MEG dataset and incorporate language-model information at two stages. First, we train a contrastive neural decoder that aligns MEG representations with acoustic and contextual language representations, improving cross-subject word decoding for both listened and imagined speech. Second, at inference, we introduce a neural-constrained beam-search framework that combines neural evidence with language-model next-word probabilities. We find that imagined-speech decoding benefits more from the language model than listened-speech decoding. For Imagined speech, the best-performing balance between neural and language-model evidence shifts toward the language model, and the gain over neural-only decoding is larger. Together, these results suggest that language priors are most useful when neural evidence is weaker, making them particularly valuable for imagined-speech BCIs.
Maryam Maghsoudi, Sai Samrat Kankanala, Shihab A. Shamma +1
Department of Electrical and Computer Engineering, University of Maryland, College Park, MD, USA · LEAP Lab, Electrical Engineering, Indian Institute of Science, Bengaluru, India.