iPhoneme: Brain-to-Text Communication for ALS Using ConformerXL Decoding

Yoonmin Cha; Dawit Chun; Sung Park

cs.SDApr 7, 2026

iPhoneme: Brain-to-Text Communication for ALS Using ConformerXL Decoding

Authors: Yoonmin Cha, Dawit Chun, Sung Park

Organizations: School of Data Science and Artificial Intelligence, Taejae University

Abstract

Brain-computer interfaces (BCIs) for speech restoration hold transformative potential for the approximately 173,000--232,500 individuals worldwide with ALS-related dysarthria. Despite recent progress, high-performance speech BCIs have been demonstrated in only 22--31 patients globally, largely due to limitations in neural decoding accuracy and practical input interfaces. We present iPhoneme, a brain-to-text communication system that jointly addresses these challenges through integrated modeling and interaction design. The system combines a deep learning phoneme decoder based on a modified Conformer architecture (ConformerXL, 192.9M parameters) with a gaze-assisted phoneme input interface that mitigates the Midas touch problem in eye-tracking systems. The acoustic model incorporates a temporal prenet with multi-scale dilated convolutions and bidirectional GRU for neural jitter correction, temporal subsampling for CTC stability, and Pre-RMSNorm stabilization across 12 encoder blocks, trained with AdamW and cosine scheduling. On the interaction side, iPhoneme introduces a chorded gaze-plus-silent-speech paradigm that replaces dwell-time selection, enabling more efficient input. We evaluate the system on the T15 dataset (45 sessions, 8,071 trials) of 256-channel intracranial EEG from speech motor cortex regions. A 6-gram phoneme language model trained on 3.1M sequences, combined with WFST beam search (beam=128), achieves 92.14% phoneme accuracy (7.86% PER) and 73.39% word accuracy (26.61% WER), approximately 3% above prior state-of-the-art. The system operates on CPU with 180 ms latency, demonstrating real-time, high-accuracy brain-to-text communication for ALS.

Figures & tables

Fig. 1 : T15 dataset overview showing iEEG waveforms, cortical topography maps for theta, alpha, beta, and gamma frequency bands, and electrode placement on speech motor cortex regions.

Range	Brain Area	Measurement
0–64	Ventral 6v	Threshold crossings
65–128	Area 4 (M1)	Threshold crossings
129–192	55b	Threshold crossings
193–256	Dorsal 6v	Threshold crossings
257–320	Ventral 6v	Spike band power
321–384	Area 4 (M1)	Spike band power

TABLE I : Electrode Channel Mapping: 512 Input Features

Fig. 2 : Input feature heatmap for a representative trial. The y-axis shows 512 channels organized by brain region and measurement type; the x-axis shows time steps. The upper half (channels 0–256) shows threshold crossings; the lower half (257–512) shows spike-band power.

Fig. 3 : Dataset split composition. Top: overall distribution. Bottom: per-session breakdown showing train (green), validation (yellow), and test (red) trials across all 45 sessions.

Fig. 4 : Data processing pipeline. 3D surface plots show the 512-channel signal at each stage for a 0.5s trial (T=0.5s). Bandpass filtering removes drift and HF noise; CAR subtracts common-mode artifacts; temporal binning compresses 600 $\times$ ; z-score normalization standardizes across sessions.

Component	Standard Conformer	ConformerXL (Ours)	Status	Parameter	Standard	Ours
Prenet				d_model	512	384
Structure	2 $\times$ Conv2D + Linear	Dilated Conv + BiGRU	Novel	num_layers	12–17	12
Convolutions	Single scale	Multi-scale (d=1, d=2)	Novel	num_heads	8	6
Temporal modeling	None	Bidirectional GRU	Novel	head_dim	64	64
Kernel size	3	5	Novel	ff_expansion	4	4
Normalization	LayerNorm	RMSNorm	Novel	conv_kernel	31	15

TABLE II : Architectural Comparison: Standard Conformer vs. ConformerXL. Components marked “Novel” are new to this work; “Modified” indicates adapted components with changed parameters.

Fig. 5 : End-to-end pipeline. The ConformerXL decodes iEEG to phonemes; an LLM encodes phonemes to sentences.

Fig. 6 : ConformerXL architecture. The temporal prenet extracts multi-scale features via dilated convolutions and BiGRU before 8 $\times$ subsampling compresses the sequence. Twelve Pre-RMSNorm Conformer blocks process encoded features, followed by linear projection to 42 phoneme classes.

Fig. 7 : Temporal prenet stage-by-stage processing. Dilated convolution d=1 captures fast transitions; d=2 captures sustained patterns; BiGRU fuses bidirectional context; linear projection maps to model dimension.

Fig. 8 : BiGRU temporal jitter correction. Forward and backward GRU outputs converge at the true phoneme event center, correcting for 10–50 ms timing variability in neural signals.

Fig. 9 : Single ConformerXL block with Pre-RMSNorm before each sub-layer. FFN uses GELU + 0.5 $\times$ residual; convolution uses pointwise-depthwise-pointwise with GLU and GroupNorm(32).

Fig. 10 : CTC training. Multiple alignment paths through blank and phoneme tokens collapse to the same output. Right: optimizer and regularization configuration.

Fig. 11 : Language model correction. Greedy decoding produces “Wether”; the 6-gram LM applies phonotactic constraints to correct to “Whether” given preceding context.

Fig. 12 : WFST graph structure. Nodes are states; edges are LM-weighted phoneme transitions. Green: highest-scoring path.

Fig. 13 : Beam search visualization (simplified to width 3). Hypotheses expand, score, and prune at each step. Red: pruned; green: retained best path.

Standard Method	Conventional Usage	Our Application	Origin
Beam search algorithm	Greedy CTC decoding in ASR	Applied to BCI/neural decoding without acoustic dictionary	Freitag (2017)
N-gram language model	Word-level LM or none	Phoneme-level 6-gram with Kneser-Ney + confusion matrix	Freitag / Chen [ 6 ]
CTC decoding output	CTC $\to$ greedy argmax	Combined with phoneme LM rescoring via recursive backoff	Graves [ 10 ]
Length normalization	Fixed penalty or none	Bayesian optimization via Optuna (150 trials)	Akiba [ 1 ]
WFST graph structure	Word-segmented decoding	Continuous speech without word boundaries	Mohri [ 21 ]

TABLE III : Decoding Approach: Standard Methods vs. Our BCI Application

Fig. 14 : iPhoneme: Slide to Unlock. Gaze swipe + silent speech phoneme trigger replaces dwell-time, providing chorded intentional selection.

Fig. 15 : Gaze-phoneme interaction modalities. Top: swipe navigation. Bottom: text drag-and-copy.

Fig. 16 : Phoneme frequency analysis from LibriSpeech (365M phonemes). Left: frequency ranking. Right: top-10 trigger candidates by rarity.

Fig. 17 : Safety vs. accuracy trade-off. Ideal triggers (upper-left) combine high precision with low frequency. DH, HH, TH, Y, W consistently occupy the optimal region.

Method	PER (%)	Acc. (%)	Latency
Greedy (argmax)	10.62	89.38	45 ms
+ 6-gram LM (0.8)	9.98	90.02	65 ms
+ WFST beam ( $K$ =128)	7.86	92.14	180 ms

TABLE IV : Decoding Performance Across Pipeline Stages

Beam	LM Wt.	Len. Pen.	Acc. (%)
128	1.0	0.9	92.14
96	1.0	0.9	92.11
128	1.0	1.0	92.08
64	1.0	0.9	92.06

TABLE V : WFST Beam Search Top-4 (150 Optuna Trials)

Study	Year	Acc. (%)
Wairagkar et al. [ 29 ]	2024	56
Willett et al. [ 31 ]	2024	82
Card et al. [ 5 ]	2025	89
This Work	2025	92.14

TABLE VI : Comparison with Prior Work

Fig. 18 : Per-phoneme confusion matrices (train left, validation right). Red boxes highlight confusion clusters among articulatorily similar phonemes.

Fig. 19 : Left: per-phoneme accuracy ranking (avg. 85.7%). Right: trigger candidate scores with top-5 (DH, HH, Y, W, TH) highlighted.

Component	Challenge	Mitigation
Accuracy (+3% PER)	Phoneme hallucinations may misrepresent intent in medical/legal contexts	Confidence gating; top- $k$ candidates shown for correction
ConformerXL + WFST	Large models fail out-of-distribution; latency harms daily use	WFST+LM gives stable behavior; CPU deployment removes GPU dependency
Gaze + phoneme UI	Decoding errors erode trust in cognitively intact but physically dependent users	Confidence visualization; chorded input prevents accidental actions
Eye-tracking + iEEG	One-size UX excludes users with varying neural profiles or disease progression	Adaptive LM weighting; personalized trigger calibration per session

TABLE VII : Ethical Impact Analysis

Conformer

See all 12 papers

Brain2Speech-Net: Fast and Intelligible Brain-to-Speech Synthesis Without Text Decoding
Sep 3, 2026Shreeram Suresh Chandra, Zexin Cai, Yu Tsao +2Brain-Computer Interfaces Neural Decoding
Johns Hopkins University, USA · Academia Sinica, Taiwan · University of Edinburgh, UK
BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding
Mar 4, 2026Tommaso Boccato, Michal Olak, Matteo FerranteBrain-to-Speech Decoding Brain-Computer Interfaces
Tether Evo
End-to-End Intracortical Speech Decoding from Neural Activity
May 23, 2026Owais Mujtaba Khanday, Jose A. Gonzalez-Lopez, Marc Ouellet +2Brain-Computer Interfaces Neural Decoding
1Dpt. of Signal Theory, Telematics and Communications, University of Granada, Spain · 2Brain, Mind, and Behavior Research Center, University of Granada, Spain · 3Grupo CSUR de Epilepsia Refractaria, Hospital Virgen de Las Nieves (Granada), Spain

Sep 3, 2026J/K move · Enter open · S save

Brain2Speech-Net: Fast and Intelligible Brain-to-Speech Synthesis Without Text Decoding

Shreeram Suresh Chandra, Zexin Cai, Yu Tsao, Simon King+1

Johns Hopkins University, USA · Academia Sinica, Taiwan · University of Edinburgh, UK

Brain-Computer Interfaces Neural Decoding Brain-to-Speech Decoding

Open paper AskPDF arXiv

The loss of speech limits communication for individuals with paralysis. Direct neural-to-speech synthesis is challenging due to the limited availability of neural data for training speech brain-computer interfaces. Most existing systems rely on cascaded neural-to-text-to-speech pipelines, which increase inference latency and propagate errors across stages. We present Brain2Speech-Net, a single-stage neural-to-speech generation framework without intermediate text decoding. We use a differentiable phoneme bottleneck and a deep-HMM alignment mechanism to map long neural recordings into the latent space of a text-to-speech (TTS) model, enabling high-quality speech synthesis. Brain2Speech-Net is the only system in our comparison that produces intelligible speech while generating faster than real time.

Abstract

Conformer

Explore similar work

Brain2Speech-Net: Fast and Intelligible Brain-to-Speech Synthesis Without Text Decoding

BrainWhisperer: Leveraging Large-Scale ASR Models for Neural Speech Decoding

End-to-End Intracortical Speech Decoding from Neural Activity