NeuroLens: Learning Latent Embeddings of Neural Semantics from Chronic Recordings
Authors: Hanrui Lyu, Baiyuan Chen, Tianshu Tan, Matthew R. Whiteway, Maxwell D. Melin, Ji Xia, Linyang He, Bradly C. Stadie, Anne Churchland, Liam Paninski, Yizi Zhang
Organizations: Northwestern University · University of Cambridge · Johns Hopkins University · Columbia University · University of California, Los Angeles · Stanford University
Understanding how neural activity represents higher-order cognition and how these representations evolve over time has long been a central pursuit in neuroscience. However, current analytical tools cannot easily distinguish representational plasticity from recording instability in chronic neural recordings. Here, we introduce NeuroLens (Latent Embeddings of Neural Semantics), a self-supervised model based on the Joint-Embedding Predictive Architecture (JEPA) framework that learns denoised, semantically informative latents from chronic neural recordings. An adaptive encoder maps changing neural populations into a common latent space, while a temporal predictor learns structure that supports prediction of future latent states. By predicting in latent space, NeuroLens captures temporally predictable structure and reduces sensitivity to transient, recording-specific variability. Across chronic intracortical data in mice and humans, the learned representations improve decoding of decision-making and semantic task variables. Multi-day pretraining enables generalization to future sessions, rapid few-shot adaptation to unseen neural populations, and more stable decoding over time than state-of-the-art baselines. Together, these results establish NeuroLens as a new paradigm for studying how neural representations change during learning and over long timescales.
Figures & tables
Figure 1: NeuroLens framework. (A) Model overview. A day-adaptive encoder maps spike counts at each time bin to latent representations, and an autoregressive (AR) predictor shared across days predicts future latents from past latents. The model is trained using a latent prediction loss together with SIGReg loss to prevent representation collapse. (B-C) Day-adaptive cross-attention encoder based on calibration data. At each time bin, an MLP independently projects each neuron’s spike count into a neuron token. These tokens serve as the keys and values of the cross-attention module. Because attention is permutation-invariant, we distinguish neuron identities using calibration trials sampled from each recording day. From these trials, we compute firing-rate statistics, autocorrelograms (ACGs), and population coupling to generate a signature for each neuron. A second MLP maps these signatures to neuron embeddings, which are concatenated with brain-region embeddings and added to the corresponding neuron tokens. Learned queries then aggregate the variable-length sequence of neuron tokens through cross-attention, producing a fixed number of latent embeddings for the AR predictor. Processing each time bin independently prevents temporal leakage, while calibration-based neuron embeddings enable adaptation to previously unseen neural populations across recording days.
Figure 2: NeuroLens improves decoding of decision-related and semantic information from mouse and human intracortical recordings. (A) Chronic multi-area Neuropixels recordings from a mouse reporting left or right stimulus location by turning a wheel ( choice ). (B) Across 14 test sessions, NeuroLens achieves lower latent effective rank, better choice decoding, and comparable movement decoding relative to causal NDT. Both models outperform decoding using raw neural activity ( Spike ). (C) Latent trajectories for left- and right-choice trials, projected onto the first three principal components (PCs), from April to June. Dark (light) lines show trial averages (individual trials). As the animal learns the task, trajectories for the two choices become increasingly separated, with clearer separation in NeuroLens than in causal NDT. We apply linear discriminant analysis (LDA) to the first three PCs to quantify this separation with balanced classification accuracy. (D) Multi-area Utah array recordings from two human participants as they attempted to speak displayed sentences. (E) NeuroLens improves sentence-embedding decoding over causal NDT with comparable phoneme decoding. Lower phoneme error rate (PER) and higher top-5 sentence retrieval accuracy indicate better decoding performance. Boxplots summarize five random seeds. We report average top-5 accuracy across held-out days. Each day has a candidate pool of 288 ∼ 448 sentences, which keeps chance level low (1 ∼ 2%). (F) For each trial, we average neural latents from causal NDT and NeuroLens across time bins, project them onto the first three PCs, and compare them with ground-truth sentence embeddings from all-MiniLM-L6-v2 ( Reimers and Gurevych, 2019 ) . Trials are colored by semantic topic; NeuroLens shows clearer semantic separation than causal NDT, quantified by LDA balanced accuracy.
Figure 3: NeuroLens learns neural representations that generalize across time, enabling rapid adaptation to unseen neural populations and more stable decoding than SOTA baselines. (A) The mouse makes increasingly accurate decisions about the visual stimulus location from April to June, while the number of simultaneously recorded neurons decreases over time. We pretrain on the first 39 days and test generalization on the held-out last 14 days. (B) Across 14 held-out days, NeuroLens outperforms baselines in choice and movement decoding, demonstrating better generalization across time. (C) Comparison of predicted movement-related behavior averaged across all trials (top) and predicted behavior from single trials (bottom), shown to demonstrate that the baselines can predict trial-to-trial variability in behaviors. The R2 values for both trial-average and single-trial behavior are shown. The dashed line shows the stimulus onset. (D) T12 and T15 show clear day-to-day distribution shifts in neural activity. We hold out the last 5 and 9 days, respectively, for testing and pretrain on the remaining days. (E) NeuroLens (FT, full fine-tuning) consistently outperforms baselines in phoneme and sentence embedding decoding across 5 (T12) and 9 (T15) held-out days, reflecting better generalization over time. Lower phoneme error rate (PER) and higher top-5 sentence retrieval accuracy indicate better decoding performance. (F) Latent trajectories for different phoneme onsets, projected onto the first three PCs, across held-out days in T15. Dark (light) lines show averaged (individual) time windows. NeuroLens shows clearer phoneme separation than causal NDT, indicating better held-out adaptation, quantified by LDA balanced classification accuracy.
Figure 4: NeuroLens learns denoised, stable latents with more consistent representations across days and stable effective rank over time. (A) We measure representational similarity between each pair of recording days for NeuroLens latents and PCA latents of raw spikes using 1− RDM similarity (Appendix E.4 ). Error bars denote one standard error of the dissimilarity metrics across day pairs. As the temporal gap between recording days increases, NeuroLens latents exhibit greater temporal consistency than raw spikes in both the IBL and human (T15) datasets. (B) Figure 3 A shows that the number of recorded neurons decreases over time, as reflected by the decreasing effective rank of the causal NDT latents. In contrast, NeuroLens maintains a more compact and stable latent rank over time in both the IBL and T15 datasets.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Movement Behavior ( R2 ) ↑
Choice (Balanced Accuracy) ↑
Session
Single-day
39-day pretrained
Single-day
39-day pretrained
2023-05-18
0.42
0.47
0.72
0.76
2023-05-22
0.41
0.48
0.63
0.76
2023-05-24
0.35
0.44
0.83
0.61
2023-05-30
0.38
0.47
0.63
0.69
2023-06-09
0.33
0.42
0.67
0.82
Appendix
Table F.1: Pretraining on 39 days vs. training on the target day alone, on 5 held-out sessions drawn at random. Both baselines under comparison use the same 20% / 10% / 70% adaptation/validation/test split for few-shot finetuning and the same architecture; they differ in whether NeuroLens is initialized with a 39-day pretrained backbone. Movement behavior R2 is the mean over the 5 time-bin-level variables.
Movement Behavior ( R2 ) ↑
Choice (Balanced Accuracy) ↑
Session (Day) / Neuron ID
✓
×
✓
×
2023-05-24
0.44
0.43
0.88
0.82
2023-06-05
0.38
0.36
0.95
0.93
2023-06-12
0.38
0.36
0.94
0.93
Appendix
Table H.1: Ablation of calibration-based neuron-identity embeddings on 3 held-out IBL sessions. For both behavioral variables, higher R2 and balanced accuracy correspond to better decoding performance.
PER (Phoneme) (%) ↓
Top-5 Accuracy (Sentence) (%) ↑
Held-Out Day / Per-Day FiLM
✓
×
✓
×
2022-08-11
27.23
27.50
51.39
40.62
2022-08-13
26.90
27.51
48.61
44.10
2022-08-18
35.19
32.39
34.38
32.81
2022-08-23
34.60
36.48
35.27
30.80
2022-08-25
36.83
38.78
31.47
30.13
Appendix
Table I.1: Ablation of calibration-based channel drift adaptation ( per-day FiLM ) on the T12 dataset. Values are reported as mean ± 1 standard error across 5 held-out days. Lower PER and higher sentence retrieval accuracy indicate better decoding performance.
IBL
Human
Stage 1
39 training days, with labels used only for supervised baselines
Pooled T12 and T15 training data, with labels used only for supervised baselines
Stage 2
Backbone frozen; only update the read-in layers required to map new neural populations to the latent space, and the task-specific decoder heads
Full end-to-end finetuning (updating the backbone, read-in, and decoder heads)
Appendix
Table J.1: Two-stage training setup used for both the IBL and human chronic datasets, across all baselines, for evaluating generalization across time.
Shared Transformer Backbone
Backbone Parameters
Hyperparameter
IBL
Human
Model
IBL
Human
Layers
6
6
NeuroLens
3.58M
7.29M
Embedding dimension
192
384
Causal NDT
3.28M
7.29M
Attention heads
8
6
POSSM
3.57M
7.27M
FFN dim
1024
768
SPINT
3.55M
—
Activation
GELU
GELU
CEBRA
0.31M
0.44M
Appendix
Table K.1: Shared Transformer backbone architecture and backbone size for the IBL and human chronic datasets. Backbone size includes the causal self-attention blocks but excludes model-specific encoders that project spikes to latents before the backbone, decoder heads, and other model-specific components.
Hyperparameter
IBL
Human
Input representation
Binned spike counts
5-bin time patches
Input dimension
N neurons
1280 (T12); 2560 (T15)
Latent dimension
192
384
Attention heads
8 (predictor) & 1 (encoder)
6
FFN dimension
1024 (predictor) & 256 (encoder)
768
Dropout
0.1
0.1
Appendix
Table K.2: Model-specific hyperparameters used for NeuroLens .
Hyperparameter
IBL
Human
Input representation
Binned spike counts
5-bin time patches
Latent dimension
192
384
# of attention heads
8
6
FFN dimension
1024
768
Dropout
0.2
0.2
Context window size
59 preceding bins
All preceding patches
Appendix
Table K.3: Model-specific hyperparameters used for Causal NDT.
Hyperparameter
IBL
Human
Input representation
Spike events
5-bin time patch
Latent dimension
256
192
# of latents per time step
1
1
# of cross-attention heads
1
1
Cross-attention head dimension
32
64
GRU depth
2
8
Appendix
Table K.4: Model-specific hyperparameters used for POSSM.
Hyperparameter
IBL
Input representation
Binned spike counts
Latent dimension
256
Cross-attention depth
2
# of attention heads & head dimension
4 & 64
FFN dimension
2944
Activation
ReLU
Appendix
Table K.5: Model-specific hyperparameters used for SPINT, which is only applied to IBL data.
Hyperparameter
IBL
Human
Input representation
Binned spike counts
5-bin time patch
Encoder
offset10-model
MLP read-in + 3 conv blocks
Embedding dimension
192
192
Conv block latent dimension
128
192
Receptive field
10 time bins
10 time patches
Positional embedding
None
None
Appendix
Table K.6: Model-specific settings used for CEBRA. Parameter counts refer to the encoder, which serves as the backbone for this model.
Self-supervised time series representation learning uses unlabeled sequences to learn features that transfer to forecasting, classification, and anomaly detection.
The brain maintains stable cognition despite continuously changing neural activity. How to extract stable cognitive states from variable neural observations remains a central problem in neural decoding. Existing neural decoding methods map neural observations to predefined external labels based on the stimulus-response principle, often capturing recording-specific spurious correlations. Inspired by how the brain infers the world, and specifically by Bayesian brain theory, we recast neural decoding as cognitive inference constrained by brain-intrinsic priors, yielding high-level meta-neural semantic representations. In decoding experiments spanning five neural recording modalities and three cognitive domains (motor, perception and internal mentation), our cognitive inference method reorganized the geometry of neural observation representations, yielding meta-neural semantic representations that exhibited consistent geometric relationships across cognitive tasks and enabled the recovery of stable cognitive states from variable neural observations. Our work provides an account of how the brain maintains relatively stable cognition despite continual changes in the external environment. Cognitive stability is sustained through cognitive inference from changing neural activity, without requiring fixed neural activity patterns.
Yi Guo, Changhong Jing, Yong Hu +4
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China. · University of Chinese Academy Sciences, Beijing, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong. +1
Achieving stable long-term neural decoding in invasive brain-machine interfaces (BMIs) remains challenging due to variations in recorded neural populations across sessions. Current latent alignment approaches may overlook task-dependent structure during cross-session adaptation. We propose Task-Conditioned Latent Alignment (TCLA), a framework that stabilizes neural decoding by learning a shared latent space. TCLA learns a low-dimensional source representation using neural reconstruction and continuous behavioral supervision. During target-session adaptation, the shared representation is fixed, while target neural activity is mapped into the source latent space by aligning source and target distributions separately for each task condition. We evaluated TCLA on seven nonhuman primate datasets spanning multiple tasks. In long-term cross-session evaluation, TCLA achieved a mean R2 of 0.476±0.014 with a negative R2 failure rate of only 6.8%. Across 1,356 within-subject session pairs, TCLA achieved a mean R2 of 0.371±0.009 with a failure rate of 6.8%. Across 2,134 cross-subject session pairs, TCLA achieved a mean R2 of 0.218±0.004 with a failure rate of 12.9%, substantially better than those of the comparison methods. These results demonstrate that by preserving behaviorally relevant and task-dependent latent structure, TCLA improves the robustness of neural decoding across recording sessions and subjects. The source code is publicly available at https://github.com/FAMD-CASIA/TCLA.
Canyang Zhao, Bolin Peng, J. Patrick Mayo +2
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Departments of Ophthalmology and Bioengineering, University of Pittsburgh, Pittsburgh, USA · School of Science and Engineering, Chinese University of Hong Kong, Shenzhen, China
Electroencephalography (EEG) is a widely used tool for studying brain function, with applications in clinical neuroscience, diagnosis, and brain-computer interfaces (BCIs). Recent EEG foundation models trained on large unlabeled corpora aim to learn transferable representations, but their effectiveness remains unclear; reported improvements over smaller task-specific models are often modest, sensitive to downstream adaptation and fine-tuning strategies, and limited under linear probing. We hypothesize that one contributing factor is the reliance on signal reconstruction as the primary self-supervised learning (SSL) objective, which biases representations toward high-variance artifacts rather than task-relevant neural structure. To address this limitation, we explore an SSL paradigm based on Joint Embedding Predictive Architectures (JEPA), which learn by predicting latent representations instead of reconstructing raw signals. We introduce Laya, the first EEG foundation model based on LeJEPA. We show that latent prediction yields representations that encode semantic structure in EEG: Laya embeddings track clinically meaningful state changes such as seizure onset, are resilient to noise, and achieve the strongest mean clinical accuracy under frozen linear probing, with particular gains on tasks where relevant neural patterns are subtle and easily obscured by artifacts. Controlled ablations against matched MAE variants confirm that the choice of pretraining objective, rather than architecture or data, is the primary driver of these gains.
Saarang Panchavati, Uddhav Panchavati, Hiroki Nariai +2
Department of Medical Informatics UCLA Los Angeles, USA · Department of Cognitive Science UCSD San Diego, USA · David Geffen School of Medicine UCLA Mattel Children’s Hospital Los Angeles, USA