NeuroLens: Learning Latent Embeddings of Neural Semantics from Chronic Recordings

Hanrui Lyu; Baiyuan Chen; Tianshu Tan; Matthew R. Whiteway; Maxwell D. Melin; Ji Xia; Linyang He; Bradly C. Stadie

cs.LGOct 2, 2026

NeuroLens: Learning Latent Embeddings of Neural Semantics from Chronic Recordings

Authors: Hanrui Lyu, Baiyuan Chen, Tianshu Tan, Matthew R. Whiteway, Maxwell D. Melin, Ji Xia, Linyang He, Bradly C. Stadie,

Organizations: Northwestern University · University of Cambridge · Johns Hopkins University · Columbia University ·

Abstract

Understanding how neural activity represents higher-order cognition and how these representations evolve over time has long been a central pursuit in neuroscience. However, current analytical tools cannot easily distinguish representational plasticity from recording instability in chronic neural recordings. Here, we introduce NeuroLens (Latent Embeddings of Neural Semantics), a self-supervised model based on the Joint-Embedding Predictive Architecture (JEPA) framework that learns denoised, semantically informative latents from chronic neural recordings. An adaptive encoder maps changing neural populations into a common latent space, while a temporal predictor learns structure that supports prediction of future latent states. By predicting in latent space, NeuroLens captures temporally predictable structure and reduces sensitivity to transient, recording-specific variability. Across chronic intracortical data in mice and humans, the learned representations improve decoding of decision-making and semantic task variables. Multi-day pretraining enables generalization to future sessions, rapid few-shot adaptation to unseen neural populations, and more stable decoding over time than state-of-the-art baselines. Together, these results establish NeuroLens as a new paradigm for studying how neural representations change during learning and over long timescales.

Figures & tables

Figure 1: NeuroLens framework. (A) Model overview. A day-adaptive encoder maps spike counts at each time bin to latent representations, and an autoregressive (AR) predictor shared across days predicts future latents from past latents. The model is trained using a latent prediction loss together with SIGReg loss to prevent representation collapse. (B-C) Day-adaptive cross-attention encoder based on calibration data. At each time bin, an MLP independently projects each neuron’s spike count into a neuron token. These tokens serve as the keys and values of the cross-attention module. Because attention is permutation-invariant, we distinguish neuron identities using calibration trials sampled from each recording day. From these trials, we compute firing-rate statistics, autocorrelograms (ACGs), and population coupling to generate a signature for each neuron. A second MLP maps these signatures to neuron embeddings, which are concatenated with brain-region embeddings and added to the corresponding neuron tokens. Learned queries then aggregate the variable-length sequence of neuron tokens through cross-attention, producing a fixed number of latent embeddings for the AR predictor. Processing each time bin independently prevents temporal leakage, while calibration-based neuron embeddings enable adaptation to previously unseen neural populations across recording days.

Figure 2: NeuroLens improves decoding of decision-related and semantic information from mouse and human intracortical recordings. (A) Chronic multi-area Neuropixels recordings from a mouse reporting left or right stimulus location by turning a wheel ( choice ). (B) Across 14 test sessions, NeuroLens achieves lower latent effective rank, better choice decoding, and comparable movement decoding relative to causal NDT. Both models outperform decoding using raw neural activity ( Spike ). (C) Latent trajectories for left- and right-choice trials, projected onto the first three principal components (PCs), from April to June. Dark (light) lines show trial averages (individual trials). As the animal learns the task, trajectories for the two choices become increasingly separated, with clearer separation in NeuroLens than in causal NDT. We apply linear discriminant analysis (LDA) to the first three PCs to quantify this separation with balanced classification accuracy. (D) Multi-area Utah array recordings from two human participants as they attempted to speak displayed sentences. (E) NeuroLens improves sentence-embedding decoding over causal NDT with comparable phoneme decoding. Lower phoneme error rate (PER) and higher top-5 sentence retrieval accuracy indicate better decoding performance. Boxplots summarize five random seeds. We report average top-5 accuracy across held-out days. Each day has a candidate pool of 288 $\sim$ 448 sentences, which keeps chance level low (1 $\sim$ 2%). (F) For each trial, we average neural latents from causal NDT and NeuroLens across time bins, project them onto the first three PCs, and compare them with ground-truth sentence embeddings from all-MiniLM-L6-v2 ( Reimers and Gurevych, 2019 ) . Trials are colored by semantic topic; NeuroLens shows clearer semantic separation than causal NDT, quantified by LDA balanced accuracy.

Figure 3: NeuroLens learns neural representations that generalize across time, enabling rapid adaptation to unseen neural populations and more stable decoding than SOTA baselines. (A) The mouse makes increasingly accurate decisions about the visual stimulus location from April to June, while the number of simultaneously recorded neurons decreases over time. We pretrain on the first 39 days and test generalization on the held-out last 14 days. (B) Across 14 held-out days, NeuroLens outperforms baselines in choice and movement decoding, demonstrating better generalization across time. (C) Comparison of predicted movement-related behavior averaged across all trials (top) and predicted behavior from single trials (bottom), shown to demonstrate that the baselines can predict trial-to-trial variability in behaviors. The $R^{2}$ values for both trial-average and single-trial behavior are shown. The dashed line shows the stimulus onset. (D) T12 and T15 show clear day-to-day distribution shifts in neural activity. We hold out the last 5 and 9 days, respectively, for testing and pretrain on the remaining days. (E) NeuroLens (FT, full fine-tuning) consistently outperforms baselines in phoneme and sentence embedding decoding across 5 (T12) and 9 (T15) held-out days, reflecting better generalization over time. Lower phoneme error rate (PER) and higher top-5 sentence retrieval accuracy indicate better decoding performance. (F) Latent trajectories for different phoneme onsets, projected onto the first three PCs, across held-out days in T15. Dark (light) lines show averaged (individual) time windows. NeuroLens shows clearer phoneme separation than causal NDT, indicating better held-out adaptation, quantified by LDA balanced classification accuracy.

Figure 4: NeuroLens learns denoised, stable latents with more consistent representations across days and stable effective rank over time. (A) We measure representational similarity between each pair of recording days for NeuroLens latents and PCA latents of raw spikes using $1-$ RDM similarity (Appendix E.4 ). Error bars denote one standard error of the dissimilarity metrics across day pairs. As the temporal gap between recording days increases, NeuroLens latents exhibit greater temporal consistency than raw spikes in both the IBL and human (T15) datasets. (B) Figure 3 A shows that the number of recorded neurons decreases over time, as reflected by the decreasing effective rank of the causal NDT latents. In contrast, NeuroLens maintains a more compact and stable latent rank over time in both the IBL and T15 datasets.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	Movement Behavior ( $R^{2}$ ) $\uparrow$		Choice (Balanced Accuracy) $\uparrow$
Session	Single-day	39-day pretrained	Single-day	39-day pretrained
2023-05-18	0.42	0.47	0.72	0.76
2023-05-22	0.41	0.48	0.63	0.76
2023-05-24	0.35	0.44	0.83	0.61
2023-05-30	0.38	0.47	0.63	0.69
2023-06-09	0.33	0.42	0.67	0.82

Appendix

Table F.1: Pretraining on 39 days vs. training on the target day alone, on 5 held-out sessions drawn at random. Both baselines under comparison use the same 20% / 10% / 70% adaptation/validation/test split for few-shot finetuning and the same architecture; they differ in whether NeuroLens is initialized with a 39-day pretrained backbone. Movement behavior

R^{2}

is the mean over the 5 time-bin-level variables.

	Movement Behavior ( $R^{2}$ ) $\uparrow$		Choice (Balanced Accuracy) $\uparrow$
Session (Day) / Neuron ID	$\checkmark$	$\times$	$\checkmark$	$\times$
2023-05-24	0.44	0.43	0.88	0.82
2023-06-05	0.38	0.36	0.95	0.93
2023-06-12	0.38	0.36	0.94	0.93

Appendix

Table H.1: Ablation of calibration-based neuron-identity embeddings on 3 held-out IBL sessions. For both behavioral variables, higher

R^{2}

and balanced accuracy correspond to better decoding performance.

	PER (Phoneme) (%) $\downarrow$		Top-5 Accuracy (Sentence) (%) $\uparrow$
Held-Out Day / Per-Day FiLM	$\checkmark$	$\times$	$\checkmark$	$\times$
2022-08-11	27.23	27.50	51.39	40.62
2022-08-13	26.90	27.51	48.61	44.10
2022-08-18	35.19	32.39	34.38	32.81
2022-08-23	34.60	36.48	35.27	30.80
2022-08-25	36.83	38.78	31.47	30.13

Appendix

Table I.1: Ablation of calibration-based channel drift adaptation ( per-day FiLM ) on the T12 dataset. Values are reported as mean

\pm

1 standard error across 5 held-out days. Lower PER and higher sentence retrieval accuracy indicate better decoding performance.

	IBL	Human
Stage 1	39 training days, with labels used only for supervised baselines	Pooled T12 and T15 training data, with labels used only for supervised baselines
Stage 2	Backbone frozen; only update the read-in layers required to map new neural populations to the latent space, and the task-specific decoder heads	Full end-to-end finetuning (updating the backbone, read-in, and decoder heads)

Appendix

Table J.1: Two-stage training setup used for both the IBL and human chronic datasets, across all baselines, for evaluating generalization across time.

Shared Transformer Backbone			Backbone Parameters
Hyperparameter	IBL	Human	Model	IBL	Human
Layers	6	6	NeuroLens	3.58M	7.29M
Embedding dimension	192	384	Causal NDT	3.28M	7.29M
Attention heads	8	6	POSSM	3.57M	7.27M
FFN dim	1024	768	SPINT	3.55M	—
Activation	GELU	GELU	CEBRA	0.31M	0.44M

Appendix

Table K.1: Shared Transformer backbone architecture and backbone size for the IBL and human chronic datasets. Backbone size includes the causal self-attention blocks but excludes model-specific encoders that project spikes to latents before the backbone, decoder heads, and other model-specific components.

Hyperparameter	IBL	Human
Input representation	Binned spike counts	5-bin time patches
Input dimension	$N$ neurons	1280 (T12); 2560 (T15)
Latent dimension	192	384
Attention heads	8 (predictor) & 1 (encoder)	6
FFN dimension	1024 (predictor) & 256 (encoder)	768
Dropout	0.1	0.1

Appendix

Table K.2: Model-specific hyperparameters used for NeuroLens .

Hyperparameter	IBL	Human
Input representation	Binned spike counts	5-bin time patches
Latent dimension	192	384
# of attention heads	8	6
FFN dimension	1024	768
Dropout	0.2	0.2
Context window size	59 preceding bins	All preceding patches

Appendix

Table K.3: Model-specific hyperparameters used for Causal NDT.

Hyperparameter	IBL	Human
Input representation	Spike events	5-bin time patch
Latent dimension	256	192
# of latents per time step	1	1
# of cross-attention heads	1	1
Cross-attention head dimension	32	64
GRU depth	2	8

Appendix

Table K.4: Model-specific hyperparameters used for POSSM.

Hyperparameter	IBL
Input representation	Binned spike counts
Latent dimension	256
Cross-attention depth	2
# of attention heads & head dimension	4 & 64
FFN dimension	2944
Activation	ReLU

Appendix

Table K.5: Model-specific hyperparameters used for SPINT, which is only applied to IBL data.

Hyperparameter	IBL	Human
Input representation	Binned spike counts	5-bin time patch
Encoder	offset10-model	MLP read-in + 3 conv blocks
Embedding dimension	192	192
Conv block latent dimension	128	192
Receptive field	10 time bins	10 time patches
Positional embedding	None	None

Appendix

Table K.6: Model-specific settings used for CEBRA. Parameter counts refer to the encoder, which serves as the backbone for this model.

Hyperparameter	IBL
Input representation	Spike events
Latent dimension	192
# of latents per time step	2
Latent grid step	1 bin (16.7 ms)
# of cross-attention heads & head dimension	2 & 32
Time positional embedding	RoPE

Appendix

Table K.7: Model-specific settings used for POYO.

Self-Supervised Time Series Representation Learning

Self-supervised time series representation learning uses unlabeled sequences to learn features that transfer to forecasting, classification, and anomaly detection.

See all 144 papers

Neural Decoding as Cognitive Inference
Oct 8, 2026Yi Guo, Changhong Jing, Yong Hu +4Neural Decoding Cognitive Modeling
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China. · University of Chinese Academy Sciences, Beijing, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong. +1
Stable Neural Decoding Across Sessions via Task-Conditioned Latent Alignment for Brain-Machine Interfaces
Sep 23, 2026Canyang Zhao, Bolin Peng, J. Patrick Mayo +2Brain-Computer Interfaces Neural Decoding
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Departments of Ophthalmology and Bioengineering, University of Pittsburgh, Pittsburgh, USA · School of Science and Engineering, Chinese University of Hong Kong, Shenzhen, China
Laya: A LeJEPA Approach to EEG via Latent Prediction over Reconstruction
Mar 17, 2026Saarang Panchavati, Uddhav Panchavati, Hiroki Nariai +2EEG Foundation Models Self-Supervised Time Series Representation Learning
Department of Medical Informatics UCLA Los Angeles, USA · Department of Cognitive Science UCSD San Diego, USA · David Geffen School of Medicine UCLA Mattel Children’s Hospital Los Angeles, USA

Oct 8, 2026J/K move · Enter open · S save

Neural Decoding as Cognitive Inference

Yi Guo, Changhong Jing, Yong Hu, Yan Liu+3

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China. · University of Chinese Academy Sciences, Beijing, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong.+1

Neural Decoding Cognitive Modeling Bayesian Inference

Open paper AskPDF arXiv

The brain maintains stable cognition despite continuously changing neural activity. How to extract stable cognitive states from variable neural observations remains a central problem in neural decoding. Existing neural decoding methods map neural observations to predefined external labels based on the stimulus-response principle, often capturing recording-specific spurious correlations. Inspired by how the brain infers the world, and specifically by Bayesian brain theory, we recast neural decoding as cognitive inference constrained by brain-intrinsic priors, yielding high-level meta-neural semantic representations. In decoding experiments spanning five neural recording modalities and three cognitive domains (motor, perception and internal mentation), our cognitive inference method reorganized the geometry of neural observation representations, yielding meta-neural semantic representations that exhibited consistent geometric relationships across cognitive tasks and enabled the recovery of stable cognitive states from variable neural observations. Our work provides an account of how the brain maintains relatively stable cognition despite continual changes in the external environment. Cognitive stability is sustained through cognitive inference from changing neural activity, without requiring fixed neural activity patterns.

Abstract

Self-Supervised Time Series Representation Learning

Explore similar work

Neural Decoding as Cognitive Inference

Stable Neural Decoding Across Sessions via Task-Conditioned Latent Alignment for Brain-Machine Interfaces

Laya: A LeJEPA Approach to EEG via Latent Prediction over Reconstruction