Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling

Zehui Feng; Weichuan Wang; Xiaohan Chen; Cuntai Guan; Ting Han

cs.CLOct 16, 2026

Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling

Authors: Zehui Feng, Weichuan Wang, Xiaohan Chen, Cuntai Guan, Ting Han

Organizations: Shanghai Jiao Tong University · Nanyang Technological University · Zhejiang University

Abstract

Recent speech brain-computer interfaces (BCIs) increasingly rely on large models and complex multi-stage pipelines for neural decoding. However, in speech BCI, where training data are scarce, noisy, and expensive to collect, the benefits of model scaling remain unclear. This work revisits a fundamental question: are large models truly necessary for accurate neural decoding? We propose BEST (Brain-Encoding-Speech-Text), a two-stage framework for efficient neural-to-text decoding. Instead of relying on large generative models or phoneme-centric pipelines, BEST introduces hierarchical intermediate representations and a compact speech-oriented decoder that directly maps neural activity to text. This design preserves essential structure while significantly reducing computational cost, enabling robust generalization in low-resource settings and practical deployment under constrained resources. On the Brain-to-Text '24 and '25 benchmarks, BEST achieves word error rates (WERs) of 6.19% and 1.62%, respectively, using only 3.85% of the parameters required by comparable methods. Beyond these results, we provide three key insights: (1) decoder scaling yields non-monotonic gains; (2) cross-modal alignment is more critical than model size; and (3) overfitting degards sequence-level performance. Representation analysis further reveals a consistent neural-audio-text alignment pathway.

Figures & tables

Figure 2: Overall illustration of BEST . The framework consists of four core modules: (1) Hierarchical Transformer Encoder (Stage 1): it employs frequency-channel attention and iTransformer layers to encode neural activity into phoneme/character representations with hierarchical temporal masking; (2) Phoneme-Character Fusion Block (Stage 1): a lightweight convolutional fusion module aggregates hierarchical neural representation to bridge neural and linguistic spaces; (3) Lightweight Speech Translator (Stage 2): a lightweight speech translator generates autoregressive text sequences from fused embeddings; and (4) KenLM Reranking (Stage 2): a custom 4-gram language model resolves ambiguities from noisy neural signals for final selection.

Model	#Param, RAM	Brain-to-text ’24 T12				Brain-to-text ’25 T15
Model	#Param, RAM	PER $\downarrow$	CER $\downarrow$	Valid WER $\downarrow$	WER $\downarrow$	PER $\downarrow$	CER $\downarrow$	Valid WER $\downarrow$	WER $\downarrow$
RNN(Baseline) [ 48 ]	5.70B, 300GiB	0.2122	0.2561	0.2271	0.0976	0.1310	0.2075	0.1127	0.0667
Transformer [ 11 ]	14.79B, 320GiB	0.1628	0.1732	0.1121	0.0818	0.1081	0.1573	0.0934	0.0612
DCoND [ 23 ]	10.50B, 320GiB	0.1534	-	-	0.0860	-	-	-	-
BIT [ 58 ]	1.53B, 32GiB	0.1439	-	0.1607	0.1567	0.0712	-	0.0879	0.1106
BIT [ 58 ]	1.53B, 350GiB	-	-	-	0.0635	-	-	-	0.0406

Table 1: We compare the speech-BCI framework on the Brain-to-Text ’24 and ’25 hold-out set. Gray background indicates the pipeline without the language model. Some results lack detailed data as the corresponding code is unavailable; see the competition leaderboard for more details.

\star

\diamond

\bullet

denote the most lightweight network, the most robust network, and the most practical network, respectively.

Decoder	#Params	Dimension	Valid WER $\downarrow$
Decoder	#Params	Dimension	Full fine-tuning	Partial fine-tuning	Frozen
Whisper-tiny [ 39 ]	34M	384	0.3678	0.2845	0.2996
Whisper-base [ 39 ]	74M	512	0.2409	0.2183	0.2235
DistilGPT2 [ 41 ]	82M	768	0.2091	0.1930	0.2006
Whisper-large [ 39 ]	1.55B	1280	0.1461	0.1460	0.1572
Whisper-medium [ 39 ]	769M	1024	0.1351	0.1309	0.1366

Table 2: Performance comparison across different decoders with parameter counts and fine-tuning strategies on T15. Gray background indicates the audio-based LLM decoders.

Neural encoder	Whisper-small	Audio-LLM	Decoder gap
BEST encoder	0.0681	0.0989	0.0308
Transformer encoder	0.0862	0.1217	0.0355
GRU encoder	0.1054	0.1462	0.0408

Table 3: Encoder–decoder factorization on Brain-to-Text ’25 validation set.

Figure 3: (a) Self-RSA on character, phoneme, fusion embedding. (b) Cross-RSA between Phoneme Embedding-Audio Embedding, Phoneme Embedding-Audio Embedding, and Fusion Embedding-Text Embedding. (c) Temporal activation patterns of GRU, transformer, and proposed BEST .

Panel of Figure 4 — Figure 4: Cross-day generalization under temporal shift. Models are trained on first 15 days (left) and 18 days (right), and evaluated on subsequent days.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
GRU Layers / Units	5 / 768
GRU Dropout	0.4
Patch Size / Stride	14 / 4
Input Dropout	0.2
Optimizer	Adam ( $\beta_{1}$ =0.9, $\beta_{2}$ =0.999)
Weight Decay	$1\times 10^{-3}$

Appendix

Table 10: Hyperparameters of the RNN Baseline Model

Panel of Figure 5 — Figure 5: Validation curves of CTC loss and CER for the Transformer and proposed BEST on participant T12.

Figure 6: Phoneme decoding probability heatmaps under well-fitted (top) and overfitted (bottom) conditions. Overfitting produces fragmented and overconfident activations, indicating representation collapse despite lower phoneme-level error.

Figure 7: Top-2 phoneme and character confusion matrices on Brain-to-Text ’24 and ’25 benchmarks.

Figure 8: Phoneme and character transition matrices (previous $\rightarrow$ current) on Brain-to-Text ’24 and ’25 benchmarks. The matrices capture empirical transition patterns in neural decoding at both phonological and orthographic levels.

Neural Decoding

Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.

See all 179 papers

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding
Mar 4, 2026Tommaso Boccato, Michal Olak, Matteo FerranteBrain-to-Speech Decoding Brain-Computer Interfaces
Tether Evo
From Neurons to Conversation: Speech Brain-Computer Interfaces
Sep 29, 2026Moein Khajehnejad, Forough Habibollahi, Tommaso Boccato +4Brain-Computer Interfaces Neural Decoding
Tether Evo · Turner Institute for Brain and Mental Health, Monash University · The University of Rome Tor Vergata
Brain2Speech-Net: Fast and Intelligible Brain-to-Speech Synthesis Without Text Decoding
Sep 3, 2026Shreeram Suresh Chandra, Zexin Cai, Yu Tsao +2Brain-Computer Interfaces Neural Decoding
Johns Hopkins University, USA · Academia Sinica, Taiwan · University of Edinburgh, UK

Mar 4, 2026J/K move · Enter open · S save

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding

Tommaso Boccato, Michal Olak, Matteo Ferrante

Tether Evo

Brain-to-Speech Decoding Brain-Computer Interfaces Neural Decoding Automatic Speech Recognition

Open paper AskPDF arXiv

Decoding continuous speech from intracortical recordings is a central challenge for brain-computer interfaces (BCIs), with transformative potential for individuals with conditions that impair their ability to speak. While recent microelectrode array (MEA) decoders achieve impressive accuracy, their performance is fundamentally limited by the small size of existing datasets, they remain brittle to session-to-session variability, and their ability to generalize across participants remains unexplored. We introduce BrainWhisperer, a neural speech decoder that integrates high-resolution MEA recordings with a large pretrained automatic speech recognition (ASR) model. Building on interpretability findings showing that Whisper's encoder learns phoneme-selective representations with localized attention, we train a customized version of Whisper, modified to process neural features, using a hybrid objective that combines CTC loss on phonemes--predicted from the third encoder layer--and cross-entropy loss on word tokens. We introduce domain-informed modifications including windowed self-attention to capture articulatory continuity, hierarchical month/day-specific low-rank projections to address non-stationarity, and subject-specific embedders enabling cross-subject training. Evaluated on a publicly available MEA dataset (Card et al.), BrainWhisperer matches or outperforms prior state-of-the-art decoders. Critically, cross-dataset training improves performance even on individual datasets without fine-tuning, demonstrating unprecedented generalization. The model supports dual decoding paths: a high-accuracy phoneme-based path with external language model rescoring, and a fast direct text generation path enabling sub-100ms inference with minimal hardware requirements.

Abstract

Neural Decoding

Explore similar work

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding

From Neurons to Conversation: Speech Brain-Computer Interfaces

Brain2Speech-Net: Fast and Intelligible Brain-to-Speech Synthesis Without Text Decoding