LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses

Ebrahim Feghhi; Junlin Hu; Nima Hadidi; Jonathan C. Kao

cs.HCMar 14, 2026

LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses

Authors: Ebrahim Feghhi, Junlin Hu, Nima Hadidi, Jonathan C. Kao

Organizations: Neuroscience Interdepartmental Program, University of California Los Angeles · Department of Electrical & Computer Engineering, University of California Los Angeles · Department of Computer Science, University of California Los Angeles

Abstract

A promising pathway for restoring communication in patients with dysarthria and anarthria is speech neuroprostheses, which directly decode speech from cortical neural activity. Two benchmarks, Brain-to-Text '24 and '25, released intracranial recordings from patients with dysarthria along with a baseline algorithm trained with Connectionist Temporal Classification (CTC). Despite significant innovation on these benchmarks, all leading published prior work relies on a WFST-based CTC decoder that requires ${\sim}$ 320 GB of RAM. These memory requirements limit accessibility for both patients and researchers. Here, we propose LightBeam, a non-WFST based CTC decoder that requires only ${\sim}$ 10 GB of RAM and achieves state-of-the-art performance on both benchmarks. LightBeam achieves this by integrating an LLM into the beam-search process via delayed fusion, obviating the prior need for using a large N-gram LM. LightBeam is implemented in Python and is open-source.

Figures & tables

\multirow 2* Method	B2T ’24				B2T ’25
	WER	RAM	VRAM	RTF	Pub. WER	Priv. WER	RAM	VRAM	RTF
WFST-Based (Orig.)	9.76	-	-	-	6.67	7.00	-	-	-
WFST-Based (Re-impl.)	9.71 $\pm$ 0.08	322.9	9.0	0.05	6.31 $\pm$ 0.12	6.72 $\pm$ 0.05	317.8	18.1	0.01
LightBeam (Ours)	9.37 ∗† $\pm$ 0.08	9.4	5.6	0.14	5.77 ∗† $\pm$ 0.12	6.47 ∗† $\pm$ 0.05	11.6	5.8	0.06

Table 1: Combined performance and computational efficiency on B2T ’24 and ’25 benchmark test sets when using the baseline GRU encoder. Peak RAM and VRAM are reported in GB. Processing speed is measured in RTF using an GeForce RTX 5090 GPU. Lower is better for all metrics. Mean

\pm

SEM across

n=10

seeds. WFST-based (Orig.) is a single seed result provided by original authors, and re-implementation (re-impl) values are generated in this study. ( ∗ sig. better than Orig., † sig. better than Re-impl.)

\multirow 2* Method	B2T ’24		B2T ’25
	WER	RTF	Pub. WER	Priv. WER	RTF
WFST-Based	8.36 $\pm$ .12	0.02	4.95 $\pm$ .26	4.81 $\pm$ .13	0.01
LightBeam	8.08 † $\pm$ .12	0.10	4.44 † $\pm$ .21	4.59 † $\pm$ .14	0.05

Table 2: Word Error Rate (%) comparison on the B2T ’24 (WER) and B2T ’25 (Pub. and Priv. WER) benchmarks when using the causal time-masked Transformer with bounded KV cache. Mean

\pm

SEM over

n=10

seeds. ( † sig. better than WFST decoder).

Figure 1: LightBeam continues to match or outperform baseline WFST decoder across all benchmarks when both methods are paired with generative error correction. Results are shown with time-masked Transformer. Asterisks indicate significant improvement when performing paired t-test across

n=10

seeds.

Configuration	B2T ’24 WER		B2T ’25 WER
LightBeam (Full)	14	.17 $\pm$ 0.08	6	.36 $\pm$ 0.05
No LLM first-pass rescore	17	.17 † $\pm$ 0.05	8	.31 † $\pm$ 0.07
No NWP Fine-tuning	15	.89 † $\pm$ 0.09	7	.26 † $\pm$ 0.04
No Phoneme Variants	14	.05 $\pm$ 0.07	6	.36 $\pm$ 0.05

Table 3: Ablation study on B2T ’24 and B2T ’25 validation sets with baseline GRU. Values are Mean WER (%)

\pm

SEM (

n=10

seeds).

\dagger

denotes significant degradation relative to Full model.

Figure 2: Impact of LLM rescore interval on validation WER and RTF across both datasets. An encoder frame is output every 100 ms in B2T ’24, and every 80 ms in B2T ’25. Shaded regions indicate SEM across

n=10

seeds. LLM rescore interval was set to 10 and 15 throughout the study for B2T ’24 and ’25, respectively.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Blocks	Sentences	Split
Switchboard	182	7,672	Train, Val/Test
Freq. Words	48	2,050	Train, Val/Test
50-word Vocab	25	1,091	Train Only
Openwebtext	5	172	Val/Test only
Harvard	3	98	Val/Test only
Random	2	79	Val/Test only

Appendix

Table A1: Distribution of data sources across training and evaluation splits.

Hyperparameter	Value
Patch Length	4 (0.08 ms)
Patch Width	512
Transformer Dimension	384
Layers	5
Heads	6
Head Dimension	64

Appendix

Table A2: Transformer architecture and training hyperparameters for B2T ’25. Feed-forward network (FFN) multiplier indicates the increase in dimensionality in the FFN module.

Figure A1: Phoneme Error Rate (PER) as a function of left-context window size for B2T’24 and B2T ’25 on validation set.

Hyperparameter	B2T’ 24	B2T’ 25
Blank Penalty	log(7)	log(90)
Acoustic Scale ( $\alpha$ )	0.5	0.325
Beam	18	17
Max Active States	7000	7000
Min Active States	200	200

Appendix

Table A3: WFST decoding hyperparameters for B2T ’24 and ’25 benchmarks during first-pass rescoring. These hyperparameters were tuned by the original authors using the baseline GRU encoder, and the same set of hyperparmeters were used for analyses with the time-masked Transformer. Variable notation is copied from original authors ( [ 23 , 2 ] ).

Hyperparameter	B2T’ 24	B2T’ 25
Acoustic Scale ( $\alpha$ )	0.5	0.325
N-gram LM weight ( $\beta$ )	0.5	0.45
LLM weight	0.5	0.55
N-best beam size	100	100
Used quantized LLM	Yes	No

Appendix

Table A4: WFST decoding hyperparameters for B2T’24 and ’25 benchmarks during second-pass rescoring. These hyperparameters were tuned by the original authors using the baseline GRU encoder, and the same set of hyperparmeters were used for analyses with the time-masked Transformer.Variable notation is copied from original authors ( [ 23 , 2 ] ).

Hyperparameter	B2T’ 24	B2T’ 25
LLM rescore interval (Encoder Frames)	10	15
LLM Weight ( $\phi$ )	1.2	1.2
N-gram LM Weight ( $\omega$ )	0.8	1.0
Acoustic Scale ( $\alpha$ )	0.6	0.4
Beam Size ( k )	1000	900
Beam Prune Threshold ( $\theta$ )	22.0	18.0

Appendix

Table A5: LightBeam decoding hyperparameters for B2T ’24 and B2T ’25. These hyperparameters were tuned using the GRU encoder, and the same set of hyperparmeters were used for analyses with the time-masked Transformer.

Hyperparameter	Value
Base Precision	BFloat16
Optimizer	AdamW
Learning Rate	$2\times 10^{-4}$
Weight Decay	$10^{-3}$
Batch Size	16
Epochs	3

Appendix

Table A6: Fine-tuning hyperparameters for the next-word prediction task

Hyperparameter	Value
Quantization	4-bit NF4 (bitsandbytes)
Optimizer	AdamW-8bit
Learning Rate	$2\times 10^{-4}$
Batch Size	16
Epochs	1
Data Scheduling	[val, train, val]

Appendix

Table A7: Fine-tuning hyperparameters for the generative error correction

Figure A2: RTF Distributions of LightBeam across all trials on the test set of B2T ’24 and B2T ’25. The scale of the y-axis is logarithmic.

LLM	B2T ’24			B2T ’25
LLM	WER	RTF	VRAM	WER	RTF	VRAM
llama-3.2-1B	14.17 $\pm$ .22	0.18	6.14	6.36 $\pm$ .14	0.06	5.60
llama-3.2-3B	13.82 † $\pm$ .13	0.26	9.82	6.24 $\pm$ .16	0.08	9.64

Appendix

Table A8: Word Error Rate (%) comparison on the B2T ’24 and B2T ’25 Validation set when using GRU encoder and different LLMs for Delayed Fusion. Mean

\pm

SEM over

n=10

seeds. ( † sig. better than 1B Model).

Connectionist Temporal Classification

See all 14 papers

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding
Mar 4, 2026Tommaso Boccato, Michal Olak, Matteo FerranteBrain-to-Speech Decoding Brain-Computer Interfaces
Tether Evo
Jev Matches 7B Language Models for Speech-Neuroprosthesis Rescoring
Sep 27, 2026Gabriele CinàBrain-Computer Interfaces Neural Decoding
Brain2Speech-Net: Fast and Intelligible Brain-to-Speech Synthesis Without Text Decoding
Sep 3, 2026Shreeram Suresh Chandra, Zexin Cai, Yu Tsao +2Brain-Computer Interfaces Neural Decoding
Johns Hopkins University, USA · Academia Sinica, Taiwan · University of Edinburgh, UK

Mar 4, 2026J/K move · Enter open · S save

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding

Tommaso Boccato, Michal Olak, Matteo Ferrante

Tether Evo

Brain-to-Speech Decoding Brain-Computer Interfaces Neural Decoding Automatic Speech Recognition

Open paper AskPDF arXiv

Decoding continuous speech from intracortical recordings is a central challenge for brain-computer interfaces (BCIs), with transformative potential for individuals with conditions that impair their ability to speak. While recent microelectrode array (MEA) decoders achieve impressive accuracy, their performance is fundamentally limited by the small size of existing datasets, they remain brittle to session-to-session variability, and their ability to generalize across participants remains unexplored. We introduce BrainWhisperer, a neural speech decoder that integrates high-resolution MEA recordings with a large pretrained automatic speech recognition (ASR) model. Building on interpretability findings showing that Whisper's encoder learns phoneme-selective representations with localized attention, we train a customized version of Whisper, modified to process neural features, using a hybrid objective that combines CTC loss on phonemes--predicted from the third encoder layer--and cross-entropy loss on word tokens. We introduce domain-informed modifications including windowed self-attention to capture articulatory continuity, hierarchical month/day-specific low-rank projections to address non-stationarity, and subject-specific embedders enabling cross-subject training. Evaluated on a publicly available MEA dataset (Card et al.), BrainWhisperer matches or outperforms prior state-of-the-art decoders. Critically, cross-dataset training improves performance even on individual datasets without fine-tuning, demonstrating unprecedented generalization. The model supports dual decoding paths: a high-accuracy phoneme-based path with external language model rescoring, and a fast direct text generation path enabling sub-100ms inference with minimal hardware requirements.

Abstract

Connectionist Temporal Classification

Explore similar work

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding

Jev Matches 7B Language Models for Speech-Neuroprosthesis Rescoring

Brain2Speech-Net: Fast and Intelligible Brain-to-Speech Synthesis Without Text Decoding