Non-invasive decoding of inner speech faces a fundamental data problem: a corpus pairing brain activity with a person's spontaneous inner monologue cannot be collected, and the available proxy paradigms (cued repetitive and retrospectively reported generative inner speech) are slow to acquire, poorly time-locked, and subject compliance is unverifiable. We therefore treat silent reading as a scalable proxy task and ask how much lexical and semantic information a contrastive decoder can extract from it. We report an open-vocabulary analysis of approximately 240,000 word presentations recorded from a single densely-sampled participant across 393 runs (ca. 49 h) of 19-channel dry-electrode EEG. Words from continuous narrative text were presented in rapid serial visual presentation, with typography randomised on every trial to partially decorrelate word identity from low-level visual form. A convolutional EEG encoder, optionally followed by a causal transformer, was trained with a CLIP-style contrastive objective to align short EEG windows with hidden-state embeddings of the presented word taken from a large language model. Decoding, evaluated as word-grouped top-10 retrieval against permutation baselines, was reliably above chance, extended to mid-frequency and rare words, and scaled log-linearly with training-data volume with no sign of saturation. Removing occipital and posterior-temporal electrodes reduced the word-level gain by roughly one third but left context tracking unchanged. Control analyses separate word-level decoding from narrative context tracking and from a non-neural positional prior introduced by the transformer's positional embedding. These results establish that open-vocabulary word-level information is recoverable from EEG during silent reading, and that decoding is data-limited rather than saturated.
Figures & tables
Figure 1: Paradigm and decoding pipeline. Full caption on the following page.
Figure 2: The three evaluation regimes, and why the within-run gain is the conservative one. (a-c) Pool construction and null for each regime; trials are coloured by recording run. (a) Overall retrieval gain : pools are random samples of 512 validation trials, so most of a trial’s competitors come from other runs and therefore other text passages; the null permutes predictions within the pool. (b) Context-tracking gain : the same pools, but each trial’s EEG prediction is replaced by that of another trial from the same run. Run identity is preserved while the EEG-to-word pairing is destroyed, so whatever score survives was carried by passage-level context rather than by the current word. Hence, the context component is measured, not assumed. (c) Within-run retrieval gain , the primary metric and the checkpoint-selection criterion: pools are consecutive trials from a single run and the permutation null shuffles predictions within that run, so every candidate shares the passage’s topic. (d) Worked example for a hypothetical decoder that recovers only which text passage is being read and nothing about the current word. In a mixed pool, it ranks every same-run candidate above the rest, and so scores above chance on the overall gain. That score is unchanged by the run-mate swap, which is what makes the swap a measurement of the context-tracking component; and in a single-run pool it has nothing left to exploit and falls back to the permutation null. A positive within-run gain therefore cannot be produced by topic tracking. Pools are drawn with six candidates rather than 512 for legibility; the argument does not depend on pool size.
Configuration
Within-run gain (pp)
Overall gain (pp)
Context-tracking gain (pp)
Context-independent gain (pp)
Non-contextual target, no sequence model (L0, no TF)
7.5±2.0
7.9±1.9
1.0±0.5
6.9±1.9
Non-contextual target, transformer (L0, TF)
6.2±2.2
7.9±2.3
1.9±0.8
6.0±2.2
Contextual target, no sequence model (L20, no TF)
5.5±2.5
11.0±3.2
4.6±1.6
6.3±3.0
Contextual target, transformer (L20, TF)
7.8±3.3
19.8±6.1
5.9±1.5
13.9±5.4
Table 1: Retrieval-gain decomposition by target type and architecture (Sweep 1; n=144 fits per cell; mean ± SD; gains in percentage points, pp).
Figure 3: Retrieval-gain decomposition by target type and architecture. Word-level information is recoverable, and it is not restricted to narrative topic tracking. Grouped bars give the three evaluation regimes of Figure 2 , i.e. within-run (the primary metric), overall, and context-tracking, for the four crossed configurations of target type × architecture, in (a) Sweep 1 (576 fits, 144 per configuration) and (b) Sweep 3 (192 fits, 48 per configuration). Configurations are ordered left to right by how much narrative context is available to the fit: neither the target nor the architecture carries it (L0, no TF), one of them does (L0, TF and L20, no TF), or both do (L20, TF). Bars are means over fits ±1 SD, points are individual fits, the number above each bar is n , and every quantity is referenced to an empirical permutation baseline, so chance is 0 pp on the y-axis. The diamond marks the context-independent gain (overall − context-tracking); it is drawn as a derived marker rather than a fourth bar because the three measured gains come from different pooling regimes and therefore do not sum. The decomposition behaves as designed: the context-tracking bar rises with available context (Sweep 1: 1.0→1.9→4.6→5.9 pp; Sweep 3: 0.7→1.5→4.8→6.7 pp), an ordering predicted by construction. Contextual targets and models (L20, TF) result in by far the largest overall gain, and some of this gain is context tracking.
Configuration
Within-run gain (pp)
Position- only (pp)
Preceding- EEG (pp)
Current- trial (pp)
Position- corrected (pp)
Dispersion
L0, no TF
7.5±2.0
0.0±0.0
0.0±0.0
7.5±2.0
7.5±2.0
0.000±0.000
L0, TF
6.2±2.2
0.0±0.1
0.0±0.2
6.1±2.2
6.1±2.2
0.167±0.041
L20, no TF
5.5±2.5
0.0±0.0
0.0±0.0
5.5±2.5
5.5±2.5
0.000±0.000
L20, TF
7.8±3.3
2.7±0.7
1.9±0.8
3.1±2.7
5.1±3.0
0.091±0.048
Table 2: Position probes: the within-run gain decomposed into a positional prior, a preceding-EEG contribution, and a current-trial contribution (Sweep 1; n=144 fits per cell; mean ± SD; gains in percentage points, pp). Unlike Table 1 , these terms do sum: position-only + preceding-EEG + current-trial = within-run gain, and position-corrected = within-run gain − position-only. Position-only is the only non-neural term. Dispersion is a validity check on the position-only null; a dispersion of zero indicates representation collapse when the model receives masked EEG data and can only use the positional embedding for its prediction. Hence, a near-zero position-only gain is evidence only where dispersion is above zero. (In the transformer-free cells, a dispersion of exactly zero is expected by construction and does not indicate collapse.) For more details, see Supplementary S6.1 .
Figure 4: Position probes: the within-run gain is not produced by the positional embedding. The transformer’s learned positional embedding is the only trial-varying input to the model that is not neural, and contextual (layer-20) targets drift systematically along a run, so in principle a within-run gain could be earned from position alone. The position probes measure this potential confound by re-running the trained sequence model with per-trial encoder outputs replaced by the learnable [MASK] token (Section 2.9 ). (a) and (b) The probe results for the four target-type × architecture configurations, in Sweeps 1 ( (a) ) and 3 ( (b) ). The three terms are nested (position only; position + preceding-word EEG; + the current word’s own EEG) and sum to the within-run gain. The preceding-EEG segment is a signal, not a nuisance: The model receives no text, so exploiting local context at all requires having decoded preceding words from EEG. Only the position-only segment is non-neural. (c) Raw and position-corrected within-run gain per word-frequency bin. A potential concern is that the decoding performance in some frequency bins might be overwhelmingly driven by exploiting position information. But the results survive position correction in all word frequency bins. (d) Scaling of the position probe terms across a decade of training data (Sweep 2). The position-only term stays flat near zero (L0, no TF) or grows at a lower rate than the corrected component (L20, TF). Bars are means over fits ±1 SD, and chance is zero on every gain axis.
Configuration
Rare
Mid
Frequent
L0, no TF
3.2±1.5
3.2±1.4
8.1±2.2
L0, TF
2.4±1.5
2.4±1.4
6.7±2.4
L20, no TF
3.5±2.0
3.6±2.1
5.8±2.5
L20, TF
6.0±3.5
6.0±3.3
8.1±3.3
Table 3: Within-run retrieval gain by word-frequency bin (Sweep 1; n=144 per cell; gains in percentage points, pp).
Figure 5: Word-frequency profile. The decodable signal is not confined to frequent words. (a, b) Within-run retrieval gain per word-frequency bin. Validation words were split into rare, mid and frequent thirds by their trial count in the validation set, each scored against its own empirical permutation baseline, for each of the four target-type × architecture configurations, in (a) Sweep 1 (144 fits per configuration) and (b) Sweep 3 (48 per configuration). The solid inner bar is the within-run gain (the primary metric, word-level decoding); the wider, paler bar behind it is the overall gain from random cross-run pools (including a context-tracking component). Bars are means over fits ±1 SD, points are individual fits, and chance is zero for every bar. The frequent bin carries the largest gain in every configuration, as expected; frequent words are seen more often in training, and are better resolved by the metric (more validation trials). But the rare and mid bins are also positive throughout. A gain confined to function words would appear here as a rare and mid bin at zero, and it does not. (c) Ratio of the rare-bin to the frequent-bin within-run gain per configuration, computed from the cell means for both sweeps. The dotted line at 1.0 marks where the rare bin would decode as well as the frequent one. The gap narrows where narrative context is available, from 0.40 in the purely non-contextual cell to 0.74 in the contextual-plus-transformer cell in Sweep 1, and from 0.46 to 0.87 in Sweep 3. Contextual targets may genuinely improve decoding of rare content words, and / or a model tracking the discourse state may have an advantage when ranking the topically distinctive rare words well, compared to a model that has no context and can only decode them from their own neural response.
Contextual (L20 + TF)
Non-contextual (L0, no TF)
Metric
Slope, pp/decade ( R2 )
10% → 100% (pp)
Slope, pp/decade ( R2 )
10% → 100% (pp)
Within-run gain (primary)
+8.7 (0.98)
6.2→15.0
+4.8 (0.99)
5.0→10.1
Overall gain
+24.5 (0.96)
8.9→33.7
+5.0 (0.99)
4.7→9.7
Context-tracking gain
+5.6 (0.94)
1.7→7.3
+0.6 (0.85)
0.2→0.7
Within-run gain, rare bin
+9.0
4.5→13.3
+2.3
1.2→4.4
Within-run gain, mid bin
+8.8
4.0→13.0
+2.6
0.9→3.6
Table 4: Log-linear scaling fits, gain∼a+b⋅log10(training-data ratio) . Slopes ( b ) are in top-10 percentage points (pp) per decade, with the R2 of the log-linear fit in parentheses; the endpoint columns give the gain at 10% and at 100% of training runs, in pp. One fit per ratio. The probe rows decompose the within-run gain (Section 2.9 ); “flat” means the trace had no variance across the range, which is the desired outcome for the position-only probe.
Figure 6: Data scaling: decoding grows log-linearly with training-data volume and shows no saturation at ∼49 h. Sweep 2, ten fits per configuration, using 10%, 20%, … up to 100% of the training data subset. The lower and upper x-axes show the same information on different metrics (ratio of retained training runs and number of training trials after artefact rejection, respectively). The same, full validation set is used at every ratio. (a) Within-run gain against training volume, with log-linear fit annotated, separately for the most contextual (L20, TF) and least contextual (L0, no TF) conditions. Both configurations roughly double their gain across one decade of data. (b) All three evaluation regimes for most and least contextual configurations (colour = configuration, line style = evaluation regime). Importantly, in the non-contextual condition, the context-tracking trace stays flat against zero ( 0.2→0.7 pp, +0.6 pp/decade) while its within-run trace climbs, so the additional data buys word-level decoding rather than progressively better passage recognition. In the contextual condition, the overall gain grows fastest of all ( +24.5 pp/decade) but its context-tracking component grows with it ( +5.6 pp/decade). (c, d) Per-bin scaling, one panel per configuration. (c) Non-contextual, (d) contextual. Growth reaches the rare and mid bins, at 45–51% of the frequent bin’s rate in the non-contextual arm and at or slightly above the frequent bin’s rate in the contextual one. Chance is zero on every gain axis.
Contrast
All 19 channels
O1/O2/T5/T6 removed
Relative change
Within-run gain (marginal)
9.2±2.7
6.3±2.3
−32%
Overall gain (marginal)
15.2±8.6
11.8±8.1
−22%
Context-tracking gain (marginal)
3.5±2.7
3.3±2.6
−4%
Position-corrected within-run gain
8.5±2.4
5.6±1.8
−34%
Within-run gain, non-contextual target
8.8±1.6
5.5±1.3
−37%
Within-run gain, contextual target
9.7±3.4
7.0±2.8
−28%
Table 5: Effect of removing occipital/posterior-temporal channels (Sweep 3; n=96 per marginal cell, 48 per interaction cell; gains in percentage points, pp).
Figure 7: Channel ablation: Removing occipital and posterior-temporal electrodes reduces the word-level component by about a third, but leaves passage-level tracking untouched. All panels are Sweep 3 (192 fits; 96 per channel set, 48 per interaction bar), the only sweep in which the channel set was varied. (a) The 19-electrode 10-20 layout, with the four ablated electrodes (O1, O2, T5, T6; visual cortex and ventral visual stream) marked. (b) The three evaluation regimes with the channel set in the hue, and the relative change of the ablated cell annotated above each pair. The within-run component falls by 32% ( 9.2→6.3 pp) and the overall gain by 22%, while the context-tracking component moves by only −4% ( 3.5→3.3 pp), well within the dispersion. Whatever supports passage-level tracking is therefore not primarily measured at occipital electrodes, whatever supports word-level discrimination partly is. (c) The ablation cost by target type and by architecture, each bar marginal over the remaining swept factors. The cost is slightly larger in the non-contextual and no-transformer cells ( −37% and −36% ) than in the contextual and transformer cells ( −28% and −29% ), consistent with the contextual configurations having an additional, non-visual source of signal to fall back on. (d) The ablation cost by word-frequency bin: roughly two thirds of the gain survives in every bin (rare −29% , frequent −32% ), so the loss is not concentrated in one part of the vocabulary. Bars are means over fits ±1 SD, points are individual fits, numbers above bars are n , and chance is zero throughout. The channel ablation is not a clean localisation; removing 4 of 19 channels removes 21% of the input, and EEG has ill-defined spatial resolution so occipital sources still project to the remaining electrodes (Section 3.5 ).
Factor
Level
Within-run gain (pp)
Sweep
Window lock
Onset
6.8±2.9
1
Centre
6.9±2.8
1
Offset
6.5±2.5
1
Window length
0.5 s
6.9±2.7
3
0.7 s
8.3±2.8
3
0.9 s
8.1±2.9
3
Table 6: Within-run retrieval gain by analysis-window parameters (gains in percentage points, pp).
Figure 8: Temporal properties of the informative signal. The lock point barely matters, window length helps up to ∼0.7 s, and training-time jitter is harmful. (a) Within-run gain by window lock point (Sweep 1, 0.5 s windows; 192 fits per level). Onset-, centre- and offset-locked windows fall within 0.4 pp of each other with overlapping distributions. The decoder is not critically sensitive to window alignment at this granularity. (b) Within-run gain by window length (Sweep 3; 64 fits per level). Extending the window from 0.5 s to 0.7 s raises the gain by roughly 20% ( 6.9→8.3 pp); 0.9 s is no better than 0.7 s (8.1 pp). (c) Within-run gain by training-time window jitter ( ±100 ms was crossed in Sweep 1, ±50 ms in Sweep 3). Jitter is harmful and dose-dependently so: −13% at ±50 ms and −39% at ±100 ms. This is consistent either with the informative activity being precisely time-locked to word onset, so that jitter smears a phase-locked response, or with the model failing to learn a shift-invariant decoding. In either case, temporal-shift augmentation does not help as a data augmentation strategy in this pipeline. (d) Stimulus timeline and the three onset-locked analysis windows. Depending on the word duration (0.5 to 0.9 s) and on the window lock (onset, offset, centre), the 0.7 s and 0.9 s window lengths can contain the onset of the next word’s presentation. The absence of any further improvement from 0.7 s to 0.9 s in panel (b) is consistent with the word-specific activity being largely complete within ∼0.7 s of onset. Bars in (a), (b), and (c) are means over fits ±1 SD, points are individual fits, and chance is zero on every gain axis.
Figure 9: Optimisation factors, and how much of each difference is a training-budget artefact rather than an optimum. Five factors, one column each. (a) Top row: within-run gain per level. (b) Bottom row: the epoch at which that fit’s checkpoint was selected, with the fixed 100-epoch budget drawn as a dashed ceiling. A bar sitting against the ceiling identifies a comparison in which the losing level was plausibly still improving when training stopped, so its gain is a lower bound and the difference should not be read as an intrinsic optimum. Sweeps are separate, learning rate and batch size are marginals of Sweep 3, normalisation, within-run negative masking and trainable temperature marginals of Sweep 1, so every comparison is a clean marginal of one fully crossed grid. Bars are means over fits ±1 SD, points are individual fits, numbers above bars are n , chance is zero. Learning rate : 1e-4 clearly beats 1e-5 (9.4 vs 6.1 pp), but the 1e-5 fits select their checkpoint at epoch 91±7 out of 100 against 80±15 for 1e-4, i.e. the difference is partly an incomplete budget. Batch size : 4 runs beats 8 (8.2 vs 7.3 pp), with the same caveat ( 83±13 vs 87±13 epochs) plus two structural confounds; doubling the batch halves the number of optimiser steps per epoch, and because a batch item is a complete recording run it also doubles the number of in-batch contrastive negatives. Normalisation : simple amplitude scaling beats scaling with pre-stimulus baseline subtraction (7.2 vs 6.2 pp). Within-run negative masking : masking same-run pairs out of the contrastive loss reduces the gain ( 8.0→5.5 pp), the opposite of its motivation, so discriminating between adjacent words is the useful part of the training signal rather than a source of confusion. Trainable temperature : no detectable effect on any gain, which is the expected and desired null. The word-grouped metrics are temperature-dependent, so a null here is evidence they are undistorted.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Paper term
Pool construction
Null / baseline
What it measures
Overall retrieval gain
Random samples of 512 validation trials (mixes recording runs)
Permutation of predictions within the pool
Total decoding signal, including passage-level context
Within-run retrieval gain (primary)
Consecutive trials from a single recording run, chunked to ≤512
Permutation within the same run (within-run EEG shuffle)
Discrimination among words in the same passage; excludes cross-run topic and run-identity drift
Context-tracking gain
Random cross-run pools (as overall)
Each trial’s EEG prediction replaced by a run-mate’s
The passage-tracking component of the overall gain
Context-independent gain
n/a
n/a
Overall gain − context-tracking gain
Headroom-normalised accuracy
as overall
as overall
(accuracy − baseline) / (1 − baseline)
Rare / mid / frequent bin gains
as parent regime
per-bin empirical permutation rate
Frequency-resolved profile
Appendix
Table S1: Metric glossary.
Factor
Levels
LLM embedding layer
0, 20
Window lock
onset, offset, centre
ISI subset
{0 ms}, {100 ms}, {0, 100 ms}
Sequence model
none, transformer
Normalisation
scale-only, baseline subtraction + scale
Temporal-shift augmentation
0, ±100 ms
Appendix
Table S2: Sweep 1, 576 fits =2×3×3×2×2×2×2×2 .
Factor
Levels
LLM embedding layer
0, 20
Window length
0.5, 0.7, 0.9 s
Temporal-shift augmentation
0, ±50 ms
Channel set
all 19; O1/O2/T5/T6 removed
Sequence model
none, transformer
Batch size
4, 8
Appendix
Table S3: Sweep 3, 192 fits =2×3×2×2×2×2×2 .
Configuration
n (gain)
n (abs)
Gain (pp)
norm-acc
Implied empirical chance
Implied model top-10
V (distinct words / pool)
10/V reference
all fits
576
576
11.64
0.1352
14.31%
25.96%
299
3.34%
noTF_L0
144
144
7.92
0.0930
15.09%
23.01%
299
3.34%
noTF_L20
144
144
10.95
0.1272
14.19%
25.14%
300
3.34%
TF_L0
144
144
7.90
0.0921
14.42%
22.32%
299
3.34%
TF_L20
144
144
19.81
0.2286
13.54%
33.35%
300
3.34%
Appendix
Table S4: Sweep 1, random cross-run pools: absolute accuracies. Across all 576 fits: mean V=299 distinct core words per pool, so the naive uniform-over-words reference is 10/V=3.34% , against a measured empirical chance of 14.31% and a model top-10 of 25.96%.
Table S6: Sweep 3, random cross-run pools: absolute accuracies. Across all 192 fits: mean V=301 distinct core words per pool, so the naive uniform-over-words reference is 10/V=3.32% , against a measured empirical chance of 15.75% and a model top-10 of 29.28%.
Table S8: Sweep 1: decomposition by factor (mean ± SD). n=288 per level for two-level factors, 192 for three-level factors. Gains in percentage points (pp); norm-acc is a ratio.
Factor
Level
Rare
Mid
Frequent
Embedding layer
0
2.81±1.53
2.80±1.44
7.43±2.39
20
4.75±3.10
4.81±2.96
6.92±3.16
Window lock
onset
3.27±2.58
3.41±2.51
7.31±3.00
offset
3.98±2.52
3.92±2.37
6.88±2.56
centre
4.08±2.72
4.07±2.68
7.34±2.85
ISI subset
0 ms
4.30±2.56
4.29±2.53
7.26±2.79
Appendix
Table S9: Sweep 1: within-run gain by frequency bin. Gains in percentage points (pp).
Factor
Level
Within-run gain (pp)
Overall gain (pp)
Context- tracking gain (pp)
Context- independent gain (pp)
norm-acc
val-loss
Embedding layer
0
7.15±2.17
7.52±2.35
1.06±0.48
6.46±2.19
0.0901
7.969
20
8.38±3.36
19.55±8.20
5.76±1.60
13.79±6.81
0.2264
7.134
Window length
0.5 s
6.86±2.73
12.24±8.06
3.26±2.52
8.97±5.89
0.1434
7.612
0.7 s
8.32±2.85
14.14±8.78
3.41±2.68
10.74±6.44
0.1651
7.542
0.9 s
8.11±2.90
14.22±8.69
3.57±2.73
10.66±6.31
0.1664
7.499
Temporal shift
0
8.31±2.88
13.93±8.41
3.30±2.56
10.63±6.19
0.1625
7.565
Appendix
Table S10: Sweep 3: decomposition by factor (mean ± SD). Gains in percentage points (pp); norm-acc is a ratio. Distribution across all 192 fits: within-run gain 7.76±2.89 pp (median 7.52, min 1.99, max 16.53); overall gain 13.54±8.52 pp; context-tracking gain 3.41±2.64 pp.
Factor
Level
Rare
Mid
Frequent
Embedding layer
0
3.39±1.26
3.51±1.27
7.63±2.32
20
7.05±4.17
6.85±3.81
8.56±3.30
Window length
0.5 s
3.64±2.75
3.80±2.71
7.26±2.78
0.7 s
5.90±3.56
5.70±3.28
8.65±2.85
0.9 s
6.11±3.84
6.04±3.41
8.38±2.87
Temporal shift
0
5.62±3.74
5.61±3.41
8.65±2.87
Appendix
Table S11: Sweep 3: within-run gain by frequency bin. Gains in percentage points (pp).
Crossed factor
Level
All 19 channels
O1/O2/T5/T6 removed
Embedding layer
0
8.76±1.61
5.54±1.26
20
9.72±3.36
7.03±2.81
Sequence model
none
8.58±2.22
5.51±1.67
transformer
9.91±2.91
7.06±2.57
Window length
0.5 s
8.36±2.49
5.35±2.08
0.7 s
9.77±2.64
6.87±2.28
Appendix
Table S12: Sweep 3: channel-ablation interactions (within-run gain). Gains in percentage points (pp). The ordering of every crossed factor is preserved across channel sets; the ablation is additive rather than interactive in sign, though its relative magnitude is larger in the non-contextual and no-transformer cells.
Level
n
Within-run gain (pp)
Position- only (pp)
Preceding- EEG (pp)
Current- trial (pp)
Position- corrected (pp)
Dispersion
all fits
576
6.74
0.70
0.49
5.55
6.04
0.0645
no transformer
288
6.48
0.00
0.00
6.48
6.48
0.0000
transformer
288
7.00
1.40
0.98
4.62
5.60
0.1290
Appendix
Table S13: Sweep 1: probes by architecture (mean over fits).
Level
n
Rare
Mid
Frequent
all fits
576
3.20
3.24
6.46
L0, no TF
144
3.22
3.17
8.12
L20, no TF
144
3.51
3.64
5.75
L0, TF
144
2.40
2.42
6.69
L20, TF
144
3.69
3.72
5.26
Appendix
Table S14: Sweep 1: position-corrected within-run gain per word-frequency bin.
Level
n
Within-run gain (pp)
Position- only (pp)
Preceding- EEG (pp)
Current- trial (pp)
Position- corrected (pp)
Dispersion
all fits
192
7.76
0.73
0.58
6.46
7.03
0.0642
L0, no TF
48
7.37
0.00
0.00
7.37
7.37
0.0000
L20, no TF
48
6.71
0.00
0.00
6.71
6.71
0.0000
L0, TF
48
6.93
0.07
−0.02
6.88
6.86
0.1495
L20, TF
48
10.04
2.85
2.34
4.86
7.20
0.1071
Appendix
Table S15: Sweep 3: probes by target type and architecture.
Level
n
Within-run gain (pp)
Position- only (pp)
Preceding- EEG (pp)
Current- trial (pp)
Position- corrected (pp)
Dispersion
all 19 channels
96
9.24
0.77
0.56
7.92
8.48
0.0659
O1/O2/T5/T6 removed
96
6.28
0.69
0.60
4.99
5.59
0.0624
Appendix
Table S16: Sweep 3: probes by channel set. The position-only term is essentially unchanged by the ablation while the current-trial term drops by ∼37% : the ablation removes neural signal, not positional information, which is why the position-corrected gain falls in step with the raw within-run gain (Table 5 ).
Level
n
Rare
Mid
Frequent
all fits
192
4.67
4.66
7.34
L0, no TF
48
3.63
3.82
7.84
L20, no TF
48
5.24
5.17
6.91
L0, TF
48
3.19
3.20
7.33
L20, TF
48
6.62
6.46
7.28
all 19 channels
96
5.53
5.58
8.85
Appendix
Table S17: Sweep 3: position-corrected within-run gain per word-frequency bin.
Crossed factor
Level
Masking off
Masking on
Embedding layer
0
7.17±2.21
6.49±2.18
20
8.86±2.79
4.42±1.39
Sequence model
none
7.41±2.02
5.54±2.51
transformer
8.62±3.05
5.37±1.59
Temporal shift
0
9.88±1.93
6.88±1.72
±100 ms
6.15±1.84
4.03±1.34
Appendix
Table S18: Sweep 1: within-run gain by within-run negative masking.
Figure S1: Training dynamics: The validation-loss minimum and the within-run-gain maximum do not coincide, which is why checkpoints are selected on the gain. (a, b) Sweep 1; (c, d) Sweep 3. Left column: within-run retrieval gain against epoch. Right column: validation loss against epoch. Lines are means across fits within each of the four target-type × architecture configurations, bands are ±1 SD, and the extremum of each group-mean curve is marked ( ▲ gain maximum, ▼ loss minimum). Epoch 0 is the pre-training validation pass, so every gain curve starts at ∼0 by construction. The Sweep 1 row is restricted to fits with within-run negative masking off: masking changes the number of terms in the contrastive softmax, so validation loss is on a different scale in the two masking conditions and averaging across them would be meaningless. The gain panel is restricted identically so both panels in the row describe the same set of fits. Sweep 3 holds masking off throughout, so no filter applies there. In the non-contextual configurations the validation loss reaches a minimum around epoch 7–13 and rises thereafter while the within-run gain keeps improving. Selecting checkpoints on validation loss would therefore pick early, weak checkpoints.
Figure S2: Distribution of the primary metric across all fits, against its pre-training null. (a) Within-run retrieval gain at the selected checkpoint for all 768 fits of Sweeps 1 and 3, stacked by sweep (colour), overlaid on the pre-training (epoch 0) values of the same fits (grey). The grey distribution is the empirical null for this metric: the model has been initialised but not trained, so any departure from zero is the metric’s own noise given five evaluation pool draws and ten permutations per pool. The two distributions do not overlap: the pre-training values lie within ±0.2 pp of zero, while the selected-checkpoint values run from 1.4 to 16.5 pp, with medians of 6.3 pp (Sweep 1) and 7.5 pp (Sweep 3). The best single fit of each sweep is marked (15.4 pp in Sweep 1, 16.5 pp in Sweep 3); both are maxima over fits as well as over epochs and should be read as upper bounds. (b) The same values split by target type × architecture as box plots (median, quartiles, whiskers at 1.5× IQR) with individual fits jittered behind. No fit in any of the four configurations falls at or below chance, so the decoding effect is present everywhere in the explored hyperparameter space and the sweeps determine its magnitude rather than its existence. Chance is zero for every value shown.
Factor
Level
Fits
Presented
Included
Excluded
Inclusion rate
Sweep
Window lock
onset
64
240,141
220,787 (218,530–223,044)
19,354
91.0–92.9%
Sweep 1
centre
64
240,141
221,334 (219,018–223,651)
18,806
91.2–93.1%
Sweep 1
offset
64
240,141
221,832 (219,670–223,995)
18,308
91.5–93.3%
Sweep 1
Temporal shift
0
96
240,141
223,563 (223,044–223,995)
16,578
92.9–93.3%
Sweep 1
±100 ms
96
240,141
219,073 (218,530–219,670)
21,068
91.0–91.5%
Sweep 1
Window length
0.5 s
64
240,141
222,157 (220,498–223,790)
17,984
91.8–93.2%
Sweep 3
Appendix
Table S19: Word presentations surviving the 100 μ V amplitude criterion, by analysis-window parameter (mean across the fits in each cell; range in parentheses where it varies).
In a previous report we showed that a decoder can recover open-vocabulary word-level information from 19-channel EEG recorded during silent reading, by aligning short EEG windows with hidden-state embeddings of the presented word taken from a large language model (LLM). That analysis contrasted two of the model's 33 layers: the context-free input embedding and one mid-depth layer. Here we ask how EEG-decodable structure is distributed over the full depth of the language model. We trained one decoder per layer of Llama-3.1-8B, with and without a causal transformer over the sequence of EEG windows. We distinguish word-level decodability from a context-tracking component. Word-level decodability peaks in early layers and declines steadily towards the last layer. In contrast, the context-tracking component of the retrieval gain rises with depth, both with and without a causal transformer. Unexpectedly, a decoder that sees a single 0.5-s window at a time tracks the text passage too. Two new controls dissect this component. A target-geometry control, a synthetic predictor made of the true target plus noise calibrated to each decoder's word-level accuracy, shows that context tracking is not a by-product of word decoding and target geometry. A control that permutes predictions across runs from either the same or from different sessions shows that most of the observed context-tracking is not caused by an EEG recording session signature. Instead, context tracking mostly follows the text. A single EEG window therefore carries passage-level information that a contextual target can absorb. The results locate word identity in the early layers of the language model, discourse state in the deeper layers, and make the choice of target layer a design dial for EEG language decoders.
Decoding natural language from non-invasive electroencephalography (EEG) remains constrained by low signal-to-noise ratio and limited information bandwidth. This raises a central question: can sentence-level language be reliably recovered from such signals? Under realistic information constraints, this direct-recovery assumption may be too strong. We introduce a semantic compression hypothesis: non-invasive EEG may preserve recoverable semantic anchors rather than the full lexical--syntactic form of a sentence. From this perspective, direct sentence reconstruction is overly fine-grained relative to the recoverable information scale of EEG. To address this mismatch, we propose Brain-CLIPLM, a two-stage framework that decomposes EEG-to-text decoding into semantic-anchor recovery and anchor-guided sentence reconstruction. Stage 1 uses contrastive learning to align word-level EEG evidence with a fixed keyword vocabulary and recover ordered semantic anchors. Stage 2 uses a retrieval-grounded large language model with chain-of-thought reasoning prompts to reconstruct sentence meaning from these anchors, following a granularity matching principle that aligns decoding complexity with the recoverable neural information scale. On the combined Zurich Cognitive Language Processing (ZuCo) benchmark, Brain-CLIPLM achieves 67.6% Top-5 and 85.0% Top-25 sentence retrieval accuracy, with the strongest performance at intermediate anchor granularity. Control analyses show that EEG-derived anchors carry sentence-specific information beyond language-model priors. Within the constrained ZuCo sentence pool and fixed keyword-vocabulary settings, these findings suggest that EEG-to-text decoding is better framed as recovering compressed semantic content before anchor-guided sentence reconstruction.
Xiaoli Yang, Huiyuan Tian, Yurui Li +3
College of Computer Science and Technology, Zhejiang University, Hangzhou, China · State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China
Imagined speech decoding from EEG offers a promising route toward naturalistic brain-computer interfaces. Traditional closed-set classifiers treat semantic categories as independent labels and do not explicitly exploit relationships among linguistic intents, limiting output flexibility. We adapt a multimodal contrastive learning framework to imagined speech, aligning EEG trials with language representations in the embedding space of a frozen pretrained text encoder. An EEG encoder is trained with a composite objective combining semantically reweighted EEG-sentence alignment and text-prototype-based class discrimination, encouraging semantic structure while preserving separability among 39 intent categories. We evaluate the approach on the Chisco imagined speech dataset across five subjects, reporting Top-k accuracy, macro-F1, and balanced pairwise matching accuracy (2v2). We also investigate maximum EEG-text cosine similarity (MaxCosine) as an intrinsic indicator of decoding reliability and probe semantic structure beyond category labels using five evaluation-time semantic axes and a qualitative Semantic Gallery Probe (SGP) with previously unseen candidate sentences. The contrastive framework achieved decoding performance comparable to a task-optimized closed-set classifier while mapping EEG into a frozen language embedding space. MaxCosine identified subsets of held-out trials with higher Top-1 and Top-5 accuracy across subjects. Evaluation-time semantic axes showed mean agreement above the 50% chance reference across all five dimensions, and the SGP qualitatively illustrated direct comparison with unseen candidate sentences. Together, these findings suggest that EEG-language alignment can retain comparable fixed-intent decoding while supporting similarity-based retrieval, intrinsic reliability ranking, and evaluation of broader semantic structure.
Shahryar G. Namdari, Francesco Iacomi, Riccardo Barbieri +1
Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milano, 20133, Italy