Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI

Serwan Jassim; Victoria Bosch; Daniel Anthes; Elia Bruni; Tim Kietzmann

cs.LGOct 29, 2026

Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI

Authors: Serwan Jassim, Victoria Bosch, Daniel Anthes, Elia Bruni, Tim Kietzmann

Organizations: Institute of Cognitive Science University of Osnabrück

Abstract

Decoding sensory-evoked functional magnetic resonance imaging (fMRI) data is a central analytical tool in cognitive neuroscience. Here we ask whether self-supervised learning (SSL), which has been scarcely explored in this domain, can yield representations that support strong downstream performance. To do so, we adapt a joint embedding predictive architecture (JEPA) to train participant-specific encoders on brain data alone. The encoders never see the stimulus images or any representation derived from them. Applied to the Natural Scenes Dataset (NSD), we evaluate our learned brain embeddings on the downstream tasks of brain captioning and image reconstruction, as well as cross-participant generalization. Our self-supervised approach achieves competitive performance with parameter-rich supervised pipelines that are trained end-to-end with access to image and language data. We further find that encoders trained independently for different participants can be aligned fully unsupervised, using neither images nor stimuli shared between them. After alignment, a neural decoding head trained on one participant can be applied to another without fine-tuning and retains most of its within-subject performance on image reconstruction and brain captioning. By removing the need for images and shared stimuli, our approach is a step towards pooling fMRI data across participants and, eventually, across datasets.

Figures & tables

Figure 1: Overview of our self-supervised brain encoder training. (1) A subject’s brain response to a stimulus image is recorded across repeated presentations; we sample two of the three repetitions, $r$ and $r^{\prime}$ , one for the context encoder path and one for the target encoder path. After Schaefer parcellation, each trial’s brain response is represented as a sequence of parcels. (2) Reliability weights are applied to the raw vertices within each parcel. (3) Mask positions are sampled, and the corresponding tokens are dropped from the context encoder input sequence. (4) The context encoder $f_{\theta}$ produces context representations using per-parcel MLPs, followed by transformer layers. The target encoder $f_{\bar{\theta}}$ , updated as an exponential moving average of the context encoder, encodes the unmasked complementary trial; no gradient flows through this path. (5) Using learnable mask tokens, the predictor $g_{\phi}$ predicts the missing positions from the context representations. (6) Predictor outputs at masked positions, as well as at context positions, are supervised against the corresponding target representations under an $L_{2}$ loss.

Method	BLEU				METEOR	ROUGE-L	CIDEr	CLIP-S	RefCLIP-S
	1	2	3	4
CorText †	0.628	0.440	0.300	0.204	0.202	0.462	0.679	0.649	0.716
Ours: within-subject
Full sequence	0.625	0.438	0.303	0.212	0.201	0.464	0.673	0.635	0.705
Visual readout	0.623	0.439	0.304	0.212	0.200	0.463	0.674	0.633	0.703
Control, within-subject

Table 1: Caption generation from brain activity on the NSD test set, averaged across subjects 1, 2, 5, and 7. Higher is better for all metrics; the best result in each column and group (within-subject vs. cross-subject) is shown in bold. All variants of our method use frozen pre-trained brain encoders within the CorText pipeline. Visual readout encodes all 500 parcels but exposes only visual cortex tokens to the decoding head, while Full sequence uses all 500 encoded parcels. For cross-subject decoding, a head trained on one subject is applied to the aligned representations of another, averaged over all 12 ordered subject pairs. Control conditions remove stimulus correspondence, brain input, or cross-subject alignment, respectively.

{\dagger}

CorText is trained fully supervised.

Method	Low-Level				High-Level				Retrieval
	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Image $\uparrow$	Brain $\uparrow$
MindEye2 †	0.278	0.328	95.2	99.0	96.4	94.5	0.622	0.343	98.8	98.3
Ours, within-subject
Full sequence	0.262	0.354	94.9	98.4	94.5	92.9	0.647	0.363	97.2	97.3
Visual readout	0.260	0.353	95.3	98.5	94.7	93.0	0.642	0.363	97.2	97.5
Control, within-subject

Table 2: Image reconstruction from brain activity on the NSD test set, scores averaged across subjects 1, 2, 5, and 7. Arrows indicate the direction of improvement; the best result in each column and group (within-subject vs. cross-subject) is shown in bold. All variants of our method use frozen pre-trained brain encoders within the MindEye2 pipeline. Visual readout , Full sequence , the cross-subject protocol, and the control conditions are defined as in Table 1 . Retrieval uses sets of 300 candidates sampled from the full test set, with scores averaged over 30 independent draws. Brain retrieval is undefined for the zeroed-betas control, where every trial yields the same embedding, and is therefore omitted.

{\dagger}

MindEye2 is trained fully supervised and uses all other subjects to pre-train the decoder.

Method	Mean rank $\downarrow$		Recall@1 $\uparrow$
	FS	VS	FS	VS
Betas (no encoder)	356.68	197.76	0.01	0.09
JEPA default (same-trial, $\mathcal{L}_{\text{pred}}$ only)	392.51	362.69	0.01	0.01
+ cross-trial pairing	237.40	116.27	0.02	0.13
+ context loss $\mathcal{L}_{\text{ctx}}$	33.28	18.44	0.39	0.61
+ reliability weighting	31.48	16.53	0.39	0.63

Table 3: Pre-training ablation, measured by within-subject cross-trial retrieval. One repetition of a stimulus is ranked by cosine similarity against the embeddings of another repetition over the full 1000-stimulus test set, with no trained decoder; chance is mean rank

500.5

and Recall@1

0.001

. Results are averaged over subjects 1, 2, 5, and 7. FS reads the full sequence of 500 parcels, VS the 66-parcel visual subset selected at the encoder output. The first row uses no encoder and ranks the betas directly. Each subsequent row adds one component to the row above.

Method	Mean rank $\downarrow$	Recall@1 $\uparrow$
Betas (no encoder) + PCA	429.07	0.00
JEPA default (same-trial, $\mathcal{L}_{\text{pred}}$ only)	452.40	0.00
+ cross-trial pairing	367.17	0.05
+ context loss $\mathcal{L}_{\text{ctx}}$	15.00	0.45
+ reliability weighting	13.11	0.49
+ whitening (post-training)	7.45	0.59

Table 4: Pre-training and alignment ablation, measured by cross-subject retrieval. For each pair of subjects, an orthogonal map is fitted per parcel token, from each subject’s own training embeddings alone and with no shared stimuli or labels; a subject’s repetition-averaged response is then mapped into the other’s frame and ranked by cosine similarity against that subject’s embeddings over the full 1000-stimulus test set. Chance is mean rank

500.5

and Recall@1

0.001

. Results are averaged over the 12 ordered pairs of subjects 1, 2, 5, and 7. The first row uses no encoder and aligns dimensionality-reduced betas directly. Each subsequent row adds one component to the row above, except the last, which leaves pre-training unchanged and whitens the embeddings after training as part of the alignment procedure. All rows use the 66-parcel visual subset selected at the encoder output.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Architecture
Encoder transformer depth	2
Encoder attention heads	8
Parcellation	Schaefer-500
Encoder embedding dimension	512
Embedding MLP hidden dimension	256

Appendix

Table 5: Pre-training hyperparameters for all four participants.

Parameter	Value
Optimizer	AdamW
Learning rate schedule	cosine
Peak learning rate	$3\times 10^{-4}$
Minimum learning rate	0
Weight decay	0.01
Batch size	64

Appendix

Table 6: Image reconstruction optimization parameters used within the MindEye2 pipeline.

Parameter	Value
Optimizer	AdamW (8-bit)
Learning rate schedule	cosine
Stage-1 learning rate	$2\times 10^{-3}$
Stage-2 learning rate	$2\times 10^{-5}$
Stage-1 weight decay	0.05
Stage-2 weight penalty	0.001

Appendix

Table 7: Brain captioning optimization parameters used within the CorText pipeline.

Parameter	Value
Initialization
$k$ -means clusters	40
Anchor restarts	100
Random seeds	5
Refinement
ICP iterations	100

Appendix

Table 8: Cross-subject alignment hyperparameters. The map is fitted for each ordered pair of participants from their own training embeddings alone, with no shared stimuli, no paired samples, and no labels using the framework introduced by Marcos-Manchón et al. (2026) .

		BLEU				METEOR	ROUGE-L	CIDEr	CLIP-S	RefCLIP-S
Subject	Method	1	2	3	4
1	Full sequence	0.630	0.443	0.310	0.217	0.201	0.465	0.689	0.642	0.711
	Visual readout	0.622	0.436	0.301	0.210	0.199	0.461	0.681	0.639	0.709
	Trial permutation	0.388	0.171	0.071	0.033	0.088	0.291	0.050	0.391	0.471
	Zeroed betas	0.181	0.079	0.026	0.006	0.094	0.191	0.001	0.351	0.414
2	Full sequence	0.629	0.445	0.311	0.218	0.205	0.469	0.683	0.634	0.704

Appendix

Table 9: Per-subject caption generation scores from brain activity, within-subject. Higher is better for all metrics; the best result in each column and subject is shown in bold. The final block averages over subjects and corresponds to the within-subject rows of Table 1 .

		BLEU				METEOR	ROUGE-L	CIDEr	CLIP-S	RefCLIP-S
Pair	Method	1	2	3	4
$2\rightarrow 1$	Full sequence	0.526	0.337	0.208	0.133	0.160	0.403	0.392	0.573	0.644
	Visual readout	0.557	0.358	0.228	0.153	0.169	0.421	0.468	0.583	0.656
	w/o alignment	0.002	0.000	0.000	0.000	0.001	0.002	0.000	0.532	0.591
$5\rightarrow 1$	Full sequence	0.574	0.383	0.252	0.172	0.180	0.432	0.543	0.605	0.675
	Visual readout	0.579	0.383	0.252	0.169	0.180	0.434	0.542	0.608	0.679

Appendix

Table 10: Per-pair caption generation from brain activity, cross-subject. A head trained on one subject is applied to the aligned representations of another. Higher is better for all metrics; the best result in each column and pair is shown in bold. The final block averages over the 12 ordered pairs and corresponds to the cross-subject rows of Table 1 .

		Low-Level				High-Level				Retrieval
Subject	Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Image $\uparrow$	Brain $\uparrow$
1	Full sequence	0.309	0.369	97.3	99.2	95.3	93.7	0.631	0.352	99.9	100.0
	Visual readout	0.313	0.364	97.2	99.2	95.7	93.8	0.625	0.349	99.9	99.9
	Trial permutation	0.016	0.286	48.1	49.3	49.8	49.6	0.973	0.643	0.2	0.2
	Zeroed betas	0.018	0.255	49.8	50.9	49.7	49.7	0.977	0.700	0.4	–
2	Full sequence	0.276	0.357	96.8	99.2	95.4	93.1	0.632	0.351	99.9	99.9

Appendix

Table 11: Per-subject image reconstruction and retrieval, within-subject. Arrows indicate the direction of improvement; the best result in each column and subject is shown in bold. The final block averages over subjects and corresponds to the within-subject rows of Table 2 . Brain retrieval is undefined for the zeroed-betas control and is omitted.

		Low-Level				High-Level				Retrieval
Pair	Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Image $\uparrow$	Brain $\uparrow$
$2\rightarrow 1$	Full sequence	0.235	0.355	93.8	97.4	90.0	87.5	0.731	0.405	94.0	97.0
	Visual readout	0.242	0.382	91.8	96.0	87.6	85.2	0.759	0.435	79.4	90.0
	w/o alignment	-0.008	0.213	55.4	59.4	54.1	54.5	0.972	0.627	0.8	0.8
$5\rightarrow 1$	Full sequence	0.172	0.339	87.7	94.7	90.4	88.0	0.739	0.419	57.1	62.4
	Visual readout	0.205	0.374	87.1	93.2	88.8	86.6	0.753	0.436	35.0	49.4

Appendix

Table 12: Per-pair image reconstruction and retrieval, cross-subject. A head trained on one subject is applied to the aligned representations of another. Arrows indicate the direction of improvement; the best result in each column and pair is shown in bold. The final block averages over the 12 ordered pairs and corresponds to the cross-subject rows of Table 2 .

Figure 2: Image reconstruction from brain activity, within-subject. Reconstructions use the refined MindEye2 pipeline, whereas Table 2 reports metrics on the unrefined outputs. Full sequence , Visual readout and the control conditions are defined as in Table 1 . Stimuli were drawn at random from the 1000-image test set with a fixed seed.

Figure 3: Image reconstruction from brain activity, cross-subject. Reconstructions use the refined MindEye2 pipeline, whereas Table 2 reports metrics on the unrefined outputs. The w/o alignment control transfers the head between encoders without the alignment transformation. The same stimuli are shown as in Figure 2 .

fMRI Representation Learning

See all 17 papers

Statistical Learning for Latent Embedding Alignment with Application to Brain Encoding and Decoding
Mar 22, 2026Shuoxun Xu, Zhanhao Yan, Lexin LiLatent Space Alignment Neural Decoding
University of California at Berkeley
Platonic Representations in the Human Brain: Unsupervised Recovery of Universal Geometry
May 19, 2026Pablo Marcos-Manchón, Rishi Jha, Lluís FuentemillaNeural Representation Geometry Unsupervised Representation Learning
Department of Cognition, Development and Education Psychology, University of Barcelona · Institute of Neurosciences, University of Barcelona · Department of Computer Science, Cornell University +1
Learning Robust and Task-Invariant Functional Representation from fMRI through Siamese Self-Supervised Learning
May 27, 2026Jiyao Wang, Peiyu Duan, Nicha C. Dvornek +4Self-Supervised Learning fMRI Representation Learning
Department of Biomedical Engineering, Yale University, New Haven, CT, USA · Radiology & Biomedical Imaging, Yale School of Medicine, New Haven, CT, USA · Electrical Engineering, Yale University, New Haven, CT, USA +1

Mar 22, 2026J/K move · Enter open · S save

Statistical Learning for Latent Embedding Alignment with Application to Brain Encoding and Decoding

Shuoxun Xu, Zhanhao Yan, Lexin Li

University of California at Berkeley

Latent Space Alignment Neural Decoding Neural Encoding fMRI-Based Visual Decoding Meta-Learning Semi-Supervised Learning

Open paper AskPDF arXiv

Brain encoding and decoding aims to understand the relationship between external stimuli and brain activities, and is a fundamental problem in neuroscience. In this article, we study latent embedding alignment for brain encoding and decoding, with a focus on improving sample efficiency under limited fMRI-stimulus paired data and substantial subject heterogeneity. We propose a lightweight alignment framework equipped with two statistical learning components: inverse semi-supervised learning that leverages abundant unpaired stimulus embeddings through inverse mapping and residual debiasing, and meta transfer learning that borrows strength from pretrained models across subjects via sparse aggregation and residual correction. Both methods operate exclusively at the alignment stage while keeping encoders and decoders frozen, allowing for efficient computation, modular deployment, and rigorous theoretical analysis. We establish finite-sample generalization bounds and safety guarantees, and demonstrate competitive empirical performance on the large-scale fMRI-image reconstruction benchmark data.

Abstract

fMRI Representation Learning

Explore similar work

Statistical Learning for Latent Embedding Alignment with Application to Brain Encoding and Decoding

Platonic Representations in the Human Brain: Unsupervised Recovery of Universal Geometry

Learning Robust and Task-Invariant Functional Representation from fMRI through Siamese Self-Supervised Learning