cs.LGOct 16, 2026
SaveDeep LLM layers expose passage-level context in EEG during silent reading
Organizations: nubrain
Abstract
In a previous report we showed that a decoder can recover open-vocabulary word-level information from 19-channel EEG recorded during silent reading, by aligning short EEG windows with hidden-state embeddings of the presented word taken from a large language model (LLM). That analysis contrasted two of the model's 33 layers: the context-free input embedding and one mid-depth layer. Here we ask how EEG-decodable structure is distributed over the full depth of the language model. We trained one decoder per layer of Llama-3.1-8B, with and without a causal transformer over the sequence of EEG windows. We distinguish word-level decodability from a context-tracking component. Word-level decodability peaks in early layers and declines steadily towards the last layer. In contrast, the context-tracking component of the retrieval gain rises with depth, both with and without a causal transformer. Unexpectedly, a decoder that sees a single 0.5-s window at a time tracks the text passage too. Two new controls dissect this component. A target-geometry control, a synthetic predictor made of the true target plus noise calibrated to each decoder's word-level accuracy, shows that context tracking is not a by-product of word decoding and target geometry. A control that permutes predictions across runs from either the same or from different sessions shows that most of the observed context-tracking is not caused by an EEG recording session signature. Instead, context tracking mostly follows the text. A single EEG window therefore carries passage-level information that a contextual target can absorb. The results locate word identity in the early layers of the language model, discourse state in the deeper layers, and make the choice of target layer a design dial for EEG language decoders.
Figures & tables
Figure 1: Design. (a) Two readers. A human participant reads text one word at a time while EEG is recorded. An LLM processes the same text one token at a time. Each token updates the LLM’s hidden states. We extract a hidden state from each layer of the LLM. Early layers represent token / word identity, whereas deeper layers are more contextual. Our EEG decoder learns to align a 0.5-s EEG window with the LLM’s hidden state for that word. We separately train one EEG decoder per layer (0–32), without and with a transformer over the sequence of EEG windows. (b) Lag-dependent run-mate swap. We have previously observed that our EEG decoder tracks the narrative context of text, and that this context tracking component is distinct from word-level decoding ( Marquardt et al., 2026 ) . Here, we conduct further tests on the context tracking. The lag-dependent run-mate swap probes how far the information shared between EEG trials reaches. Each trial’s own prediction is replaced by the prediction of a run-mate k trials earlier or later before computing the gain in retrieval accuracy over a permutation baseline. (c) Narrative neighbours across sessions. The context-tracking gain may be confounded by an EEG session signature that is unrelated to the text. To test this, we permute predictions within sessions and across sessions.
Figure 2: Word-level decodability across layers. (a) Within-run retrieval gain (candidates from the same passage; the primary metric) against the layer of Llama-3.1-8B the target embedding was taken from, for the decoder without a sequence model (orange) and with the transformer over EEG windows (blue). Lines and bands are the mean and SD over the five train/validation splits, and faint lines are the individual splits. (b) The same after subtracting the transformer’s position-only probe (without a sequence model (a) and (b) are identical because there is no positional embedding). Chance is 0 pp. The two decoders are indistinguishable at layer 0, where the target carries no context; word-level decodability peaks one or two blocks into the model and declines steadily with depth, and the shape is the same in every split. (b) shows that the profile survives removal of the transformer’s positional shortcut, indicating that the profile is the result of EEG-to-LLM alignment, and not a modelling confound.
Figure 3: Overall gain and its context-tracking share across layers. (a) Overall gain in random cross-run pools (where the correct word has to be selected among candidates from many different runs / text passages). (b) Its context-tracking component, measured by replacing every trial’s prediction with a randomly sampled prediction from the same run. (c) The context-independent remainder (overall minus context-tracking). (d) The context-tracking share of the overall gain. Conventions as in Figure 2 . Panel (a) shows how much the decoder gains when candidates come from many passages, which is far more with the sequence model. (b) shows that the part of that gain earned by recognising the passage rises with depth for both decoders, i.e. also for the one without a sequence model that only sees a single EEG window (discussed in Section 3.5 ). (c) shows that the remaining context-independent gain (after subtracting the context-tracking component) follows the word-level metric’s slow decline towards the deep layers. (d) shows that the passage becomes a larger share of what the decoder reads with every layer.
Figure 4: Word-frequency bins across layers. Within-run gain for rare, mid-frequency and frequent words (tertiles by validation trial count, each against its own baseline) against layer, (a) without and (b) with the sequence model. (c) Rare-bin gain divided by frequent-bin gain; 1.0 would mean rare words decode as well as frequent ones. Means and SD over splits. Panels (a, b) show that frequent words drive the early peak, whereas rare and mid-frequency words keep improving further into the middle of the network; (c) makes the comparison directly: relative to frequent words, rare words are decoded worst against the context-free layer 0 and best against mid-depth targets.
Figure 5: Position probes and the transformer’s advantage across layers. (a) The transformer’s within-run gain decomposed into the position-only (non-neural), preceding-EEG and current-trial terms, per layer (means over splits; error bars are the SD of the within-run gain). (b) Transformer minus no-sequence-model, paired within split, for the overall gain, the within-run gain and the position-corrected within-run gain. Panel (a) shows that the non-neural position-only term is flat across depth, so the depth profile of Figure 2 is explained by the EEG terms; (b) shows that the transformer’s advantage is large on the overall gain, where it is passage tracking, small on the position-corrected word-level metric, and slightly negative at layer 0, where the target carries no context for it to exploit.
Figure 6: Lag-dependent run-mate swap and the target-geometry control. (a, b) Cross-run gain after replacing each trial’s prediction with that of the run-mate trials away (both directions averaged), for five layers, (a) without and (b) with the sequence model; dashed lines are the target-geometry control at the same layers. The profile shows how far along the run the information shared between trials reaches: a flat profile is a component that the whole run shares, a decaying one a local component, and the target-geometry control marks what word decoding with no context in the EEG would produce at each lag. Both without (a) and with (b) a sequence model, the profiles lie far above the control, so the decoder’s predictions share something that the geometry of the targets does not explain. Without a sequence model the profile is comparatively flat (note the different y-axis scales in (a) and (b)), so the shared component is run-level. With the sequence model the profile falls steeply with the lag of the swapped trial, so its context is local. (c, d) Against layer: the swap gain for a run-mate 4 and 256 trials away, the random-run-mate control of report 1, and the target-geometry control’s random swap. These panels show that the ordering of (a, b) holds at every layer, and that the run-level component grows with depth while the control stays at 1 pp or less without a sequence model and below 2 pp with it. Means and SD over splits.
Figure 7: Probing potential confounds that could explain the observed context-tracking gain. The context-tracking might be caused by a recording-session signature (mechanism 3 in 2.3 ) or by the similarity of target embeddings for adjacant words (mechanism 1 in 2.3 ). (a–d) Fraction of a validation run’s cross-run gain retained when its trials take the predictions of the validation run 2–8 places away in the narrative, for neighbours recorded in the same (green) or in a different (purple) session, with the partner’s trials position-matched (a, c) or scrambled (b, d), without (a, b) and with (c, d) the sequence model. Panels (a, c) test whether what neighbouring runs share follows the text or the recording session: a session signature would make the different-session curve collapse to zero, whereas it stays within about a quarter of the same-session curve at every layer. Panels (b, d) show how much of the shared component is tied to position within the run, by scrambling the partner’s trial order: little without a sequence model, most of it with the transformer, whose positional embedding the probes of Figure 5 already measure. (e, f) Standardised coefficients of the cosine similarity between two validation runs’ mean predictions regressed on the similarity of their mean targets, a same-session indicator and the log gap in recording time, over all pairs of validation runs. These panels weigh the three candidate sources of run-to-run similarity against each other: the text carries most of the weight, the same-session indicator little on this standardised scale (Supplementary Table S3 ), and the time gap a negative weight that is present even at layer 0, where the targets are context-free, and is therefore a slow drift along the recording axis rather than passage tracking through the targets. Means and SD over splits.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Training dynamics per layer. Within-run gain against epoch for every layer, averaged over splits, (a) without and (b) with the sequence model. Tick marks give the mean selected epoch; the budget was 100 epochs. The figure shows that every fit converged within the budget, with the selected epochs well before the ceiling at every layer.
Figure S2: Diagnostics per layer. (a) Selected checkpoint epoch against the 100-epoch ceiling. (b) Validation loss at the selected epoch; not comparable across layers, because the contrastive loss depends on the geometry of the projected targets. (c) Final learned temperature (clamp 100). (d, e) Headroom-normalised accuracy in mixed and within-run pools. The figure shows that checkpoint selection was not truncated by the epoch ceiling and that no fit approached the temperature clamp; (d, e) restate the gains of Figures 2 and 3 as accuracy relative to headroom. The headroom is the maximum gain a decoder could reach. Even a perfect EEG model that correctly identifies each word on each trial would not have a gain of 1.0 because the permutation baseline is subtracted.
Figure S3: Lag profiles for every layer, decoder without sequence model. Swap gain against the lag of the swapped run-mate (mean and SD over splits), the random-run-mate control (horizontal line) and the target-geometry control (dashed). The figure shows that the comparatively flat profile without sequence model, far above the target-geometry control, holds at every layer above 0.
Figure S4: Lag profiles for every layer, transformer decoder. As Supplementary Figure S3 , for the decoder with the sequence model. The steep decay beyond a lag of a few dozen trials holds at every layer, so the sequence model’s context is local throughout the depth.
Figure S5: Restricted candidate set across layers. The validation predictions of the main text, scored on pools that hold one trial of each of 50 fixed words (Section S2 ): the 50 most frequent content words ( a–c ) and the 50 most frequent words of any kind ( d–f ). (a, d) Top-1 and (b, e) top-10 accuracy of the 50-way identification against the target layer, for both decoders (solid, mean and SD over splits), with uniform chance (2% and 20%) as the horizontal line. The dotted and dashed lines are the two controls scored on the same pools: every prediction replaced by that of the run-mate one trial earlier (local discourse state and position, no current word) or of a random run-mate. At layer 0 the targets carry no context and both controls sit within a few points of chance (Table S10 ), so the accuracy there is word identification; the small excess of the controls over chance comes from the run-level component of the predictions, which is present even at layer 0 (Section S2 ). With depth the earlier-neighbour control rises with the accuracy itself: in a cross-run pool a contextual target is found through the discourse state, and the word-level part of the closed-set accuracy is the distance between the solid and the dotted line. (c, f) Report 1’s within-run gain restricted to the list words (pools are the list trials of one run), where candidates from other passages are absent; chance is 0 pp. For the transformer these values are not position-corrected and are upper bounds on word-level decoding (Section S2 ).
| Metric | Decoder | Peak | Value at peak | Peak per seed | Seed agreement | L0 | L20 | L32 | (layer) |
| within-run gain | noTF | L1 | 11.8 0.3 | 1/1/2/1/1 | 0.95 | 10.1 | 9.7 | 7.6 | -0.94 |
| TF | L2 | 16.8 0.2 | 2/2/2/2/2 | 0.97 | 9.7 | 14.0 | 10.4 | -0.81 | |
| position-corrected within-run gain | noTF | L1 | 11.8 0.3 | 1/1/2/1/1 | 0.95 | 10.1 | 9.7 | 7.6 | -0.94 |
| TF | L2 | 14.0 0.7 | 2/1/2/5/2 | 0.88 | 9.6 | 11.6 | 8.1 | -0.79 | |
| overall gain | noTF | L19 | 20.0 0.9 | 19/16/16/19/16 | 0.78 | 10.3 | 19.6 | 17.3 | 0.06 |
| TF | L9 | 38.5 1.9 | 9/9/8/5/6 | 0.85 | 11.0 | 34.7 | 27.9 | -0.59 |
Table S1: Where each metric peaks along the depth axis. Peak layer and value of the seed-mean curve (mean SD over the seeds at the peak layer), each seed’s own peak layer, the mean pairwise Spearman correlation between the per-seed curves (agreement on the shape of the depth profile), the values at layers 0, 20 and 32, and the Spearman correlation between layer index and value (a monotonic-trend descriptor). Gains in percentage points (pp).
| Decoder | Layer | Overall | Random run-mate | Geometry control, random | Lag 4 | Lag 256 | Geometry control, lag 4 |
| noTF | 0 | 10.3 0.3 | 1.0 0.1 | 0.1 0.1 | 1.1 0.1 | 1.0 0.1 | 0.1 0.0 |
| 1 | 17.0 0.6 | 3.8 0.3 | 0.4 0.0 | 5.2 0.3 | 3.5 0.3 | 0.6 0.1 | |
| 2 | 17.5 0.7 | 3.9 0.4 | 0.4 0.1 | 5.4 0.4 | 3.6 0.4 | 0.7 0.1 | |
| 4 | 18.1 1.0 | 4.7 0.5 | 0.5 0.1 | 6.4 0.6 | 4.4 0.5 | 0.8 0.1 | |
| 8 | 18.2 0.6 | 5.4 0.4 | 0.6 0.1 | 7.3 0.3 | 4.9 0.4 | 1.2 0.1 | |
| 12 | 18.2 0.8 | 5.7 0.6 | 0.7 0.1 | 7.8 0.5 | 5.3 0.6 | 1.3 0.0 |
Table S2: Lag-dependent run-mate swap at selected layers. Every column is a cross-run gain in the random (mixed) pools of the overall gain, mean SD over seeds, in pp; noTF and TF denote the decoder without and with the sequence model. Overall : the gain of the decoder’s own predictions, with no swap. Random run-mate : the gain after replacing each trial’s prediction with that of a random trial of the same run (report 1’s context-tracking control). Lag 4 and Lag 256 : the gain after replacing each trial’s prediction with that of the run-mate 4 or 256 trials away (earlier and later partner averaged). Geometry control, random and Geometry control, lag 4 : the random and the lag-4 swap applied to the target-geometry control, a synthetic predictor made of the true target plus noise calibrated to the decoder’s within-run gain, which has word-level information but none from the EEG; its swap gain is what the geometry of the targets alone produces, and the decoder’s excess over it requires context in the EEG.
| Decoder | Partner trials | Same session | Different session |
| noTF | matched | 0.32 0.03 | 0.24 0.02 |
| scrambled | 0.25 0.03 | 0.19 0.02 | |
| TF | matched | 0.47 0.04 | 0.36 0.02 |
| scrambled | 0.15 0.03 | 0.10 0.01 |
Table S3: Narrative neighbours across session boundaries. Top: fraction of a validation run’s cross-run gain that is retained when its trials take the predictions of the validation run 2–8 places away in the narrative, with the partner’s trials matched by position or scrambled, for neighbours recorded in the same or in a different session. Bottom: the regression of the run-centroid prediction similarity on target similarity, the same-session indicator and the log recording-time gap, over all pairs of validation runs: the standardised coefficients; the same-session term on its own 0/1 scale and as a direct contrast between same-session pairs and same-day pairs from different sessions after regressing on target similarity alone; the time-gap term with a same-passage dummy and the log narrative distance added, and within passages with the narrative distance alongside; and a block-permutation null for the same-session term (session sizes kept, their order shuffled over the time-ordered runs, 1,000 draws per checkpoint). Both parts: mean SD over layers 1–32 of the seed-mean values.
| Layer | within-run noTF | within-run TF | position-corr. noTF | position-corr. TF | overall noTF | overall TF | context-tracking noTF | context-tracking TF | context-indep. noTF | context-indep. TF |
| 0 | 10.1 0.3 | 9.7 0.1 | 10.1 0.3 | 9.6 0.2 | 10.3 0.3 | 11.0 0.1 | 1.0 0.1 | 1.6 0.0 | 9.2 0.3 | 9.4 0.1 |
| 1 | 11.8 0.3 | 15.9 0.3 | 11.8 0.3 | 13.4 0.3 | 17.0 0.6 | 33.4 1.5 | 3.8 0.3 | 6.5 0.6 | 13.2 0.4 | 26.9 1.0 |
| 2 | 11.6 0.3 | 16.8 0.2 | 11.6 0.3 | 14.0 0.7 | 17.5 0.7 | 35.0 1.7 | 3.9 0.4 | 6.3 0.7 | 13.6 0.4 | 28.7 1.0 |
| 3 | 11.2 0.3 | 16.2 0.2 | 11.2 0.3 | 12.7 0.6 | 17.4 0.6 | 34.1 1.0 | 4.3 0.4 | 6.3 0.3 | 13.2 0.3 | 27.8 0.7 |
| 4 | 11.1 0.3 | 15.8 0.3 | 11.1 0.3 | 12.7 0.8 | 18.1 1.0 | 34.6 1.8 | 4.7 0.5 | 6.7 0.7 | 13.4 0.6 | 27.9 1.2 |
| 5 | 11.0 0.3 | 16.3 0.2 | 11.0 0.3 | 13.3 0.5 | 18.6 0.7 | 36.7 1.4 | 5.0 0.4 | 7.2 0.7 | 13.6 0.5 | 29.5 0.8 |
Table S4: Per-layer retrieval-gain decomposition, mean SD over seeds (pp).
| Layer | rare noTF | rare TF | mid noTF | mid TF | frequent noTF | frequent TF | rare / frequent noTF | rare / frequent TF |
| 0 | 3.56 0.47 | 3.29 0.50 | 4.00 0.17 | 3.48 0.24 | 10.84 0.37 | 10.44 0.15 | 0.33 0.04 | 0.32 0.05 |
| 1 | 5.19 0.49 | 8.44 0.70 | 6.27 0.57 | 9.99 0.57 | 12.53 0.34 | 16.75 0.36 | 0.41 0.03 | 0.50 0.04 |
| 2 | 5.90 0.54 | 10.00 0.35 | 6.66 0.29 | 12.00 0.82 | 12.30 0.39 | 17.53 0.25 | 0.48 0.05 | 0.57 0.02 |
| 3 | 6.20 0.39 | 10.55 0.72 | 6.68 0.27 | 12.65 0.64 | 11.79 0.36 | 16.71 0.24 | 0.53 0.05 | 0.63 0.05 |
| 4 | 6.18 0.75 | 11.18 0.91 | 7.21 0.57 | 12.70 0.51 | 11.62 0.30 | 16.31 0.35 | 0.53 0.07 | 0.69 0.06 |
| 5 | 6.58 0.63 | 12.25 0.45 | 7.49 0.58 | 14.04 0.76 | 11.52 0.26 | 16.65 0.25 | 0.57 0.05 | 0.74 0.03 |
Table S5: Per-layer within-run gain by word-frequency bin (pp) and the rare-to-frequent ratio, mean SD over seeds.
| Layer | within-run | position-only | preceding-EEG | current-trial | position-corr. | dispersion |
| 0 | 9.7 0.1 | 0.0 0.2 | 0.0 0.1 | 9.6 0.2 | 9.6 0.2 | 0.156 0.014 |
| 1 | 15.9 0.3 | 2.5 0.4 | 3.4 0.3 | 10.0 0.2 | 13.4 0.3 | 0.121 0.009 |
| 2 | 16.8 0.2 | 2.9 0.6 | 4.0 0.4 | 10.0 0.3 | 14.0 0.7 | 0.117 0.014 |
| 3 | 16.2 0.2 | 3.5 0.4 | 3.2 0.4 | 9.5 0.3 | 12.7 0.6 | 0.118 0.012 |
| 4 | 15.8 0.3 | 3.2 0.7 | 3.3 0.6 | 9.4 0.6 | 12.7 0.8 | 0.101 0.011 |
| 5 | 16.3 0.2 | 3.0 0.6 | 3.5 0.6 | 9.8 0.2 | 13.3 0.5 | 0.096 0.009 |
Table S6: Per-layer position probes of the transformer decoder: the within-run gain and its nested decomposition into position-only, preceding-EEG and current-trial terms (pp), which sum to the within-run gain, and the position-corrected gain (within-run minus position-only). The last column, dispersion, is a validity check on the position-only probe (report 1): the mean cosine distance of the position-only predictions from their own centroid (a cosine distance, not a percentage). A dispersion of zero would mean that the probe’s predictions had collapsed onto a single vector when the EEG is masked and only the positional embedding is available, in which case a near-zero position-only gain would be uninformative. It is clearly above zero at every layer, so the position-only terms are valid estimates; at layer 0 the term is zero despite a positive dispersion, as expected when the target carries no context. Mean SD over seeds.
| Layer | random noTF | geom. control noTF | lag 4 noTF | lag 256 noTF | random TF | geom. control TF | lag 4 TF | lag 256 TF |
| 0 | 1.0 0.1 | 0.1 0.1 | 1.1 0.1 | 1.0 0.1 | 1.6 0.0 | 0.1 0.0 | 1.8 0.1 | 1.7 0.0 |
| 1 | 3.8 0.3 | 0.4 0.0 | 5.2 0.3 | 3.5 0.3 | 6.5 0.6 | 1.4 0.0 | 20.7 1.3 | 0.7 0.3 |
| 2 | 3.9 0.4 | 0.4 0.1 | 5.4 0.4 | 3.6 0.4 | 6.3 0.7 | 1.3 0.1 | 22.0 1.6 | 0.1 0.3 |
| 3 | 4.3 0.4 | 0.4 0.1 | 5.8 0.5 | 3.9 0.4 | 6.3 0.3 | 1.3 0.1 | 21.9 0.9 | 0.2 0.1 |
| 4 | 4.7 0.5 | 0.5 0.1 | 6.4 0.6 | 4.4 0.5 | 6.7 0.7 | 1.3 0.1 | 22.2 1.2 | 0.5 0.3 |
| 5 | 5.0 0.4 | 0.6 0.1 | 6.8 0.4 | 4.6 0.4 | 7.2 0.7 | 1.4 0.1 | 24.1 1.2 | 0.6 0.3 |
Table S7: Per-layer lag-dependent run-mate swap for both decoders (pp, mean SD over seeds): the cross-run gain after replacing each trial’s prediction with that of a random run-mate (report 1’s context-tracking control), the same random swap applied to the target-geometry control (a pure word decoder with no information from the EEG, calibrated to the decoder’s within-run gain), and the swap with the run-mate 4 or 256 trials away (earlier and later averaged). The table is the full-depth version of Table S2 and Figure 6 . The random-swap gain lies far above the geometry control at every layer above 0, so the context-tracking gain is not a by-product of word decoding and target geometry. Without a sequence model the gain at lag 256 is close to the gain at lag 4, so the component that the trials of a run share is run-level; with the sequence model the gain is large at lag 4 and collapses at lag 256, so its context is local. At layer 0, where the target is context-free, all columns are within 2 pp of zero.
| Layer | noTF trials matched same session | noTF trials matched different session | noTF trials scrambled same session | noTF trials scrambled different session | TF trials matched same session | TF trials matched different session | TF trials scrambled same session | TF trials scrambled different session |
| 0 | 0.08 0.02 | 0.08 0.03 | 0.07 0.02 | 0.08 0.02 | 0.13 0.02 | 0.11 0.01 | 0.12 0.02 | 0.11 0.01 |
| 1 | 0.25 0.04 | 0.21 0.03 | 0.19 0.04 | 0.16 0.03 | 0.38 0.04 | 0.32 0.03 | 0.10 0.02 | 0.08 0.02 |
| 2 | 0.25 0.03 | 0.21 0.02 | 0.19 0.04 | 0.16 0.03 | 0.41 0.04 | 0.36 0.03 | 0.10 0.02 | 0.08 0.02 |
| 3 | 0.26 0.04 | 0.22 0.03 | 0.20 0.04 | 0.17 0.03 | 0.42 0.04 | 0.36 0.04 | 0.11 0.02 | 0.08 0.02 |
| 4 | 0.28 0.04 | 0.22 0.04 | 0.22 0.04 | 0.18 0.04 | 0.44 0.05 | 0.36 0.07 | 0.12 0.02 | 0.09 0.03 |
| 5 | 0.29 0.04 | 0.22 0.03 | 0.23 0.04 | 0.17 0.03 | 0.44 0.04 | 0.35 0.04 | 0.12 0.02 | 0.08 0.02 |
Table S8: Per-layer retained fraction of a validation run’s cross-run gain when its trials take the predictions of a narrative neighbour, a validation run 2–8 places away in the text (Section 2.4 ; definition in Section S0 ), mean SD over seeds. Columns: the decoder (noTF, TF); how the neighbour’s trials are assigned ( trials matched : trial of the run takes trial of the neighbour run, which preserves any component that depends on position within the run; trials scrambled : a random trial of the neighbour run); and whether the neighbour run was recorded in the same session as the run ( same session ) or in a different one ( different session ). The table is the full-depth version of Table S3 and Figure 7 a–d. A recording-session signature would make the different-session fraction collapse towards zero; instead it stays at roughly three quarters of the same-session fraction at every layer above 0, so most of what neighbouring runs share follows the text. Scrambling the neighbour’s trials removes about a fifth of the fraction without a sequence model and about two thirds of it with the transformer, whose positional embedding carries the within-run position component measured by the probes of Table S6 .
| Layer | selected epoch noTF | selected epoch TF | validation loss noTF | validation loss TF | temperature noTF | temperature TF | norm. acc. (mixed) noTF | norm. acc. (mixed) TF | norm. acc. (within-run) noTF | norm. acc. (within-run) TF |
| 0 | 83 8 | 78 13 | 8.52 0.01 | 8.55 0.01 | 16.09 0.01 | 16.69 0.02 | 0.120 0.003 | 0.128 0.001 | 0.121 0.004 | 0.115 0.001 |
| 1 | 63 16 | 91 8 | 8.34 0.03 | 7.33 0.04 | 15.94 0.03 | 17.05 0.02 | 0.196 0.006 | 0.377 0.017 | 0.142 0.004 | 0.191 0.004 |
| 2 | 58 8 | 86 6 | 8.34 0.05 | 7.23 0.03 | 16.01 0.02 | 17.08 0.01 | 0.201 0.008 | 0.396 0.019 | 0.141 0.005 | 0.203 0.003 |
| 3 | 72 12 | 83 9 | 8.38 0.03 | 7.30 0.06 | 16.02 0.03 | 17.04 0.01 | 0.200 0.007 | 0.386 0.012 | 0.135 0.004 | 0.194 0.003 |
| 4 | 74 16 | 81 9 | 8.38 0.03 | 7.34 0.09 | 15.99 0.03 | 17.00 0.01 | 0.208 0.011 | 0.391 0.020 | 0.134 0.004 | 0.191 0.004 |
| 5 | 61 7 | 92 6 | 8.35 0.05 | 7.27 0.05 | 15.93 0.02 | 16.98 0.03 | 0.213 0.008 | 0.414 0.015 | 0.133 0.003 | 0.196 0.003 |
Table S9: Per-layer training and evaluation diagnostics for both decoders, mean SD over seeds: the epoch of the selected checkpoint (budget 100 epochs), the validation loss at that epoch (not comparable across layers, because the contrastive loss depends on the geometry of the projected targets), the final learned temperature (logit scale; clamp 100), and the headroom-normalised top-10 accuracy in mixed and in within-run pools, that is, the gain over the permutation baseline divided by the largest gain a perfect decoder could reach (1 minus the baseline), so that 0 is chance and 1 is perfect retrieval. The table is the numerical version of Supplementary Figure S2 . The selected epochs lie below the 100-epoch ceiling at every layer and the temperatures far below the clamp, so no fit was cut short by the budget or ran away in temperature. The normalised accuracies restate the gains of Figures 2 and 3 on one scale: in mixed pools the decoders capture 12–13% of the headroom at layer 0 and at most 23% (noTF) and 43% (TF) deeper in the model, and within run 9–20%, with the within-run values peaking at layers 1–2 and the mixed-pool values at mid depth.
| List | Decoder | Layer | Top-1 | Top-1, lag-1 swap | Top-1, random swap | Top-10 | Top-10, lag-1 swap | Within-run gain (pp) |
| 50 content words | noTF | 0 | 5.1 0.7 | 2.7 0.2 | 2.9 0.1 | 33.8 1.2 | 23.8 1.5 | 8.0 0.6 |
| 1 | 12.8 0.5 | 7.2 0.5 | 5.6 0.3 | 59.5 0.9 | 45.5 1.4 | 13.2 0.7 | ||
| 2 | 13.0 0.9 | 7.2 0.6 | 5.7 0.4 | 60.2 1.8 | 46.6 1.5 | 13.3 0.9 | ||
| 4 | 14.0 1.1 | 8.5 0.4 | 6.4 0.5 | 61.7 2.2 | 50.0 1.5 | 12.1 0.8 | ||
| 8 | 13.6 1.0 | 9.6 0.5 | 6.9 0.6 | 62.1 2.3 | 53.5 2.6 | 11.1 0.4 | ||
| 16 | 15.6 0.5 | 11.3 0.4 | 8.3 0.6 | 64.5 0.8 | 56.8 1.9 | 10.9 0.8 |
Table S10: Restricted candidate set at selected layers. The validation predictions of the main text, scored on trials whose word belongs to a fixed list of 50 words (Supplement Section S2 ). Balanced pools hold one trial of every list word, so a pool is a 50-way identification task with uniform chance (top-1 / top-10: 50 content words: 2.0% / 20.0%; 50 most frequent words: 2.0% / 20.0%; the empirical permutation baseline reproduces these). Two controls are scored on the same pools: every prediction replaced by that of the run-mate one trial earlier (the local discourse state and position without the current word) or of a random run-mate (report 1’s context-tracking control). Within-run: report 1’s within-run gain restricted to the list words, in pp. Accuracies in percent; mean SD over seeds.
| 50 content words, 1–25 | 50 content words, 26–50 | 50 most frequent words, 1–25 | 50 most frequent words, 26–50 |
| 1. said (1,206) | 26. left (218) | 1. the (14,112) | 26. on (1,387) |
| 2. man (590) | 27. say (213) | 2. and (6,503) | 27. said (1,206) |
| 3. like (414) | 28. thought (211) | 3. of (5,896) | 28. this (1,147) |
| 4. did (404) | 29. tell (211) | 4. to (5,887) | 29. from (1,128) |
| 5. time (394) | 30. night (204) | 5. a (5,756) | 30. be (1,114) |
| 6. know (357) | 31. make (202) | 6. he (4,064) | 31. my (1,103) |
Table S11: The two candidate word lists of the restricted-candidate-set analysis, with the number of presentations of each word in the full corpus of the participant (409 runs, 248,826 presentations, 15,352 word types; target events excluded). Content words: the 50 most frequent words that are alphabetic, not in the English stop-word list of scikit-learn 1.9.0, and not proper names (capitalised in more than 90% of presentations); they account for 5.3% of all presentations. Most frequent words: the 50 most frequent words of any kind (44.0% of presentations). Words are the punctuation-stripped, lower-cased forms used for word grouping throughout.
| Component | Detail | No sequence model | With transformer |
| EEG encoder | global pathway, four convolutional blocks | 551,328 | 551,328 |
| global pathway, attention pooling | 385 | 385 | |
| local pathway, three convolutional blocks | 181,536 | 181,536 | |
| head, two fully-connected layers and output layer | 1,182,720 | 1,182,720 | |
| [MASK] token | 256-dimensional | 256 | 256 |
| Input layer normalisation | 512 | 512 |
Table S12: Number of trainable parameters by component, for the decoder without and with the sequence model (19 channels; the architecture of every fit in this report, identical at every target layer). Parameter tensors are counted at their full size, including the affine parameters of batch and layer normalisation; the temperature is the single learned log-scale scalar of the contrastive loss.
More in this topic
EEG Decoding
Electroencephalography (EEG) decoding maps brain-signal patterns to mental states, stimuli, speech, or intended actions.
See all 198 papers