Organizations: Computer Science and Technology, Zhejiang University, Hangzhou, Zhejiang, China. · Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences, Shanghai, China.
Speech production and perception are the main ways humans communicate daily. Prior brain-to-text decoding studies have largely focused on a single modality and alphabetic languages. Here, we present a unified brain-to-sentence decoding framework for both speech production and perception in Mandarin Chinese. The framework exhibits strong generalization ability, enabling sentence-level decoding when trained only on single-character data and supporting characters and syllables unseen during training. In addition, it allows direct and controlled comparison of neural dynamics across modalities. Mandarin speech is decoded by first classifying syllable components in Hanyu Pinyin, namely initials and finals, from neural signals, followed by a post-trained large language model (LLM) that maps sequences of toneless Pinyin syllables to Chinese sentences. To enhance LLM decoding, we designed a three-stage post-training and two-stage inference framework based on a 7-billion-parameter LLM, achieving overall performance that exceeds larger commercial LLMs with hundreds of billions of parameters or more. In addition, several characteristics were observed in Mandarin speech production and perception: speech production involved neural responses across broader cortical regions than auditory perception; channels responsive to both modalities exhibited similar activity patterns, with speech perception showing a temporal delay relative to production; and decoding performance was broadly comparable across hemispheres. Our work not only establishes the feasibility of a unified decoding framework but also provides insights into the neural characteristics of Mandarin speech production and perception. These advances contribute to brain-to-text decoding in logosyllabic languages and pave the way toward neural language decoding systems supporting multiple modalities.
Figures & tables
Figure 1: Overview of our brain-to-sentence framework. a,b, Data acquisition paradigm. a, Task alternating between listening to and speaking individual Chinese characters. b, Task alternating between listening to and speaking complete Mandarin sentences. c,e, Training pipeline of the decoding framework. c, the brain decoder is trained to classify syllable components (initials and finals) from neural signals. We adopted NeuroSketch, a 2D-CNN-based neural decoder developed in our concurrent work. e, The LLM is post-trained to generate full sentences from syllable sequences. d,f, Inference pipeline of the decoding framework. d, Beam search is applied to the probabilistic outputs of the brain decoder to generate multiple syllable sequence candidates, from which the top-20 candidates are retained. f, The top-20 candidates are fed into the post-trained LLM, which performs a two-stage inference process to produce the final decoded sentence. g, The proposed framework exhibits hierarchical, character-level, and syllable-level generalization abilities.
Figure 2: Channel contribution analysis. a, Histogram showing the distribution of relative improvement over chance level for all channels in the speaking and listening tasks. Three improvement thresholds, 0%, 50%, and 100%, are indicated, corresponding to low-, medium-, and high-contribution channels, respectively. b, Spatial distributions of high-contribution channels identified during speech production and perception. c, Violin plots comparing the decoding performance between speaking and listening tasks for channels that were highly responsive to both speech production and perception. d, For the subset of channels in (c) exhibiting stable temporal relationships between modalities, a heatmap illustrating the maximum correlation between speaking- and listening-evoked neural signals and the corresponding time lags. e, Box plots showing the distribution of time lags across different syllables for each channel with stable speaking–listening delays. f, Violin plots comparing decoding performance between left and right hemispheric channels across different contribution thresholds.
Figure 3: Decoding of initials and finals. a, Bar plots showing the initial and final classification accuracies of NeuroSketch and three other brain decoders in the speaking and listening tasks. For each task, results are shown for participants with available high-contribution channels (12 out of 12 for speaking, 10 out of 12 for listening). Complete results for all participants are provided in the Supplementary Tables. b, Bar plots illustrating syllable generalization performance of NeuroSketch. Classification accuracies of initials and finals are compared between in-domain (ID) and out-of-domain (OOD) syllables for participants evaluated on the second corpus.
Figure 4: Sentence decoding. a, Proportion distributions of SER for the top-20 syllable-sequence candidates generated by NeuroSketch and the three other brain decoder architectures in the speaking (12 participants) and listening (10 participants) tasks. Complete results of our model for all participants are provided in the Supplementary Tables. b, Scatter plot illustrating the relationship between the relative improvement in initial and final classification accuracies and the relative improvement in EMP among the top-20 candidates. c,d, Results for sentence-level decoding. To ensure that the input candidate sets contain a minimal threshold of reliable information for sentence decoding, we focused on participants whose average proportion of high-quality candidates (SER <0.3 ) across the decoding corpus exceeded 10% (7 out of 12 for speaking, 6 out of 12 for listening). Complete results of our model for all participants are provided in the Supplementary Tables. c, Bar plots comparing CER of our LLM and other LLMs on the syllable-to-sentence decoding task, including small-sized LLM (Qwen2.5-7B-Instruct), medium-sized LLMs (Qwen2.5-72B-Instruct), and large-scale commercial LLMs (Qwen-3 Max, Deepseek-v3.2-exp, Doubao-1.6, GPT-5-chat-latest, Grok-4-fast, Llama-3.1). d, Bar plots presenting ablation results of our syllable-to-sentence decoding framework. The Baseline method used an IME-style decoder based on lexicon- and language-model–driven search. The Top-1, Top-3, and Top-20 correction groups performed direct correction of the top-k beam search candidates. Auxiliary component ablations, labeled “w/o translation” and “w/o vocabulary,” were created by removing the respective components from the full framework.
Figure 5: Decoding of tones. a, Bar plots showing tone classification accuracy in the speaking and listening tasks. b, Proportion distributions of syllable error rates for the top-20 syllable sequence candidates with and without tone information. c, Bar plots comparing exact match probability under decoding with and without tone information.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Spatial distribution of sEEG electrodes and their contributions across the brain. Electrodes from 12 subjects are visualized from three brain perspectives. Heatmaps indicate each channel’s F1 score in the speaking and listening tasks during initial, final, and tone decoding.
Figure 7: Architectures of the four brain decoders. The input to each brain decoder is an sEEG signal with C channels and L time steps. For each stage of the decoder, we indicate the number of output channels, the number of time steps, and the corresponding number of blocks.
Figure 8: Initial-final classification results . Bar plots illustrate the initial and final classification F1 scores achieved by four different brain decoders for speaking and listening tasks.
Figure 9: Tone classification results . Bar plots illustrate the tone classification F1 scores achieved by four different brain decoders for speaking and listening tasks.
Uploading Inc · Human-centric Artificial Intelligence Centre, University of Technology Sydney · Stanford Institute for Human-Centered Artificial Intelligence (HAI), Stanford University
Department of Electrical and Computer Engineering, University of Maryland, College Park, MD, USA · LEAP Lab, Electrical Engineering, Indian Institute of Science, Bengaluru, India.