AI-Driven Neural Surrogates for In Silico Design of Cognitive-Affective Neuromodulation Targets

Marco Rothermel; Madleen Stenger; Soroush Daftarian; Svenja Jule Francke; Bita Shariatpanahi; José C. García Alanis; Mohammad-Ali Nikouei Mahani; Stefan G. Hofmann

q-bio.NCSep 23, 2026

AI-Driven Neural Surrogates for In Silico Design of Cognitive-Affective Neuromodulation Targets

Authors: Marco Rothermel, Madleen Stenger, Soroush Daftarian, Svenja Jule Francke, Bita Shariatpanahi, José C. García Alanis, Mohammad-Ali Nikouei Mahani, Stefan G. Hofmann,

Organizations: Department of Psychiatry and Psychotherapy, University of Marburg, Germany. · Center for Mind, Brain, and Behavior (CMBB), University of Marburg, Germany. · Department of Psychology, University of Marburg, Germany. · Faculty of Medicine, University of British Columbia, Canada.

Abstract

In neuropsychiatry, the primary goal is often not only to decode brain activity but to change it, for example to lessen a negative affective bias or an overly salient memory. Motivated by control theory, we develop an AI-driven neural-surrogate framework that proposes candidate representational changes and tests their predicted perceptual effects from snapshots of stimulus-evoked fMRI activity, without physical stimulation. The framework combines fMRI decoding, deep generative modeling, and constrained latent-space steering. Valence and memorability are used only as worked examples. Using more than 36,000 image-fMRI observations from four deeply sampled Natural Scenes Dataset participants, subject-specific models recovered coarse generative structure from visually responsive cortex (two-way identification, 0.79-0.88; chance, 0.5). Graded perturbations were reconstructed as images and evaluated with automated scorers and human ratings from 7,200 trials by 18 participants. In the primary VDVAE model, valence shifted from -0.61 to +1.03 SD and memorability from -1.34 to +1.45 SD; a later Versatile Diffusion refinement reduced or altered these effects. Across five perturbation levels, human valence ratings moved in the predicted direction under the linear time-correction model (mean slope, 0.038 SD per unit of alpha; 95 percent CI, 0.003-0.074; positive in 16 of 18 participants). Perceived memorability did not change reliably. Baseline agreement with the automated assessor was suggestive for valence (r = 0.30) and weak for memorability (r = 0.10). Extreme perturbations drifted from the original stimulus, so intended change must be weighed against loss of fidelity. These findings provide a falsifiable upstream method for designing and behaviorally testing candidate representational targets for future neuromodulation in psychiatry, while marking the limits of the present static approximation.

Figures & tables

Figure 1: Conceptual Illustration of Surrogate-Guided In Silico Cognitive-Affective Neuromodulation Design. Reconstructed visual stimuli illustrate the predicted consequences of surrogate-guided representational perturbations targeting emotional valence. Numerical scores below each image summarize automated valence estimates from EmoNet ( Goetschalckx et al., 2019 ) and independent human valence ratings; they do not represent cognitive changes directly induced through brain stimulation. Warmer hues and pleasant scenes are associated with positive-direction perturbations, whereas cooler and more aversive content is associated with negative-direction perturbations. These examples are illustrative and should not be used to infer effect size or consistency; baseline human–assessor correspondence and perturbation effects are evaluated in Sections 4.1 and 4.2 , respectively. Additional examples are available at https://www.neuralsurrogate.com/ .

Figure 2: Framework for AI-Driven In Silico Perturbation Design. Left: Subject-specific ridge regression maps stimulus-evoked 7T fMRI patterns from the Natural Scenes Dataset Allen et al. (2022) into the latent space of a pretrained VDVAE Child (2020) , enabling reconstruction of the viewed stimuli. Valence and memorability are quantified using EmoNet Goetschalckx et al. (2019) and MemNet Khosla et al. (2015) . Right: Empirical attribute directions are applied at graded strengths to fMRI-decoded latent representations. The perturbed representations are reconstructed using a CLIP-conditioned diffusion pipeline Radford et al. (2021) ; Ozcelik and VanRullen (2023) and evaluated using model-based scores, independent human ratings, and image-fidelity measures. The fMRI-space projection visualizes candidate activity-pattern directions under the learned decoder; the pipeline does not derive physical stimulation parameters.

Variable	Category	Count	Percent
Age group	21–29	12	66.7
	30–39	6	33.3
Gender	Female	10	55.6
	Male	8	44.4
Mental-health history	No, never	12	66.7
	Yes, in the past	2	11.1

Table 1: Participant characteristics for the independent behavioral evaluation (

N=18

Figure 3: Independent human evaluation of reconstructed stimuli. (a) Participants completed the PHQ-9 and a demographic survey, followed by task instructions and practice. The main experiment contained one original-image block and nine reconstruction blocks, with 40 trials per block. The reconstruction conditions comprised the unperturbed baseline ( $\alpha=0$ ) and valence- or memorability-targeted perturbations at $\alpha\in\{-4,-2,2,4\}$ . Each image was shown for 350 ms and followed by ratings of arousal, valence, and perceived memorability. (b) Before each block, a fixed demonstration screen illustrated the valence–arousal space using OASIS images ( Kurdi et al., 2017 ) ; memorability was demonstrated separately. Demonstration images were not drawn from NSD, were not generated by the surrogate pipeline, and were excluded from the analyses. Participants rated only the perceptual consequences of the in silico perturbations; no neurostimulation was administered.

Property	Comparison	Evaluator	$n$	$r$ [95% CI]	$p$
Valence	Score preservation	EmoNet	4 fMRI subjects	0.63 [0.61, 0.65]	$<0.001$
	Human–assessor agreement	Human ratings–EmoNet	40 images	0.30 [ $-0.01$ , 0.56]	0.058
Memorability	Score preservation	MemNet	4 fMRI subjects	0.77 [0.75, 0.79]	$<0.001$
	Human–assessor agreement	Human ratings–MemNet	40 images	0.10 [ $-0.22$ , 0.40]	0.551

Table 2: Baseline score preservation and human–assessor agreement. Automated score preservation was quantified by correlating assessor scores at the unperturbed baseline with scores from the corresponding final-diffusion image-to-image reconstructions. Correlations were Fisher-

z

averaged across the four fMRI subjects, with

p

values Holm-adjusted across the 20 preservation tests. Human–assessor agreement was calculated across the 40 source-image indices in the unperturbed behavioral condition by correlating mean human ratings with standardized

\alpha=0

Versatile Diffusion assessor scores averaged across the four fMRI subjects, rather than with the separate image-to-image reference scores. This comparison is source-index matched; the fMRI-subject provenance of the displayed images was not recorded. Human–assessor confidence intervals use the Fisher-

z

transformation, and their two-sided

p

values are unadjusted.

Figure 4: Attribute-score changes across latent perturbation strengths. (a–b) Changes relative to the unperturbed condition ( $\alpha=0$ ), expressed in baseline standard-deviation units, for valence and memorability. Individual observations are shown as unconnected points; lines connect group means, and error bars denote 95% confidence intervals. The symmetric-logarithmic axes have a linear region within $\pm 0.10$ SD. (c–d) Participant- and assessor-level slopes estimated across the full range from $\alpha=-4$ to $\alpha=+4$ , in SD units per unit of $\alpha$ . Error bars denote 95% confidence intervals, and the symmetric-logarithmic axes have a linear region within $\pm 0.01$ SD per unit of $\alpha$ . (e–f) Descriptive kernel-density estimates and rug plots of the same full-range slopes. These distributions are shown for visualization rather than inferential comparison, particularly because the automated estimates contain four fMRI subjects, compared with 18 human participants. Human estimates incorporate the leave-one-participant-out, assessor-neutral linear fatigue correction. The shared legend identifies Human Scores, VDVAE Scores, and Versatile Diffusion Scores.

Figure 5: Trade-offs between attribute-score modulation and visual fidelity. (a–b) Pearson correlations between assessor scores at each perturbation level and the corresponding scores of the unperturbed reconstructions ( $\alpha=0$ ), shown for the VDVAE and Versatile Diffusion stages. At $\alpha=0$ , the correlations quantify score preservation between the unperturbed reconstruction and its image-to-image reconstruction. (c–d) PixCorr and CLIP image-embedding similarity relative to the corresponding $\alpha=0$ reconstruction. Fidelity measures are oriented so that higher values indicate better preservation. Small, semi-transparent symbols show the four fMRI subjects and are not connected. Lines join only group means; large symbols and error bars show Fisher- $z$ mean correlations with 95% confidence intervals in panels a–b , and means $\pm$ SEM in panels c–d . Color denotes the targeted attribute: valence (blue) or memorability (magenta). To improve readability, the valence and memorability series are slightly horizontally offset; this visual jitter has no numerical meaning. In the fidelity panels, filled squares denote PixCorr and open circles denote CLIP similarity.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Regression target	Output per image	Ridge penalty $\lambda$	Maximum iterations
VDVAE latents	91,168	50,000	10,000
CLIP text embeddings	$77\times 768$	100,000	50,000
CLIP vision embeddings	$257\times 768$	60,000	50,000

Appendix

Table A1: Parameters of the subject-specific fMRI-to-feature regressions. All models included an intercept and used training-set fMRI normalization.

Resolution	Coordinates	S1	S2	S5	S7
$1\times 1$	32	0.252	0.230	0.243	0.200
$4\times 4$	1,024	0.187	0.149	0.117	0.117
$8\times 8$	8,192	$-0.011$	$-0.012$	$-0.015$	$-0.013$
$16\times 16$	65,536	$-0.022$	$-0.021$	$-0.019$	$-0.019$
$32\times 32$	16,384	$-0.023$	$-0.022$	$-0.019$	$-0.019$

Appendix

Table B1: Layer-resolved held-out

R^{2}

of the fMRI-to-VDVAE mapping. Values are reported relative to a training-mean latent predictor. Coordinates denote the number of latent variables within each spatial-resolution group.

Measure	Direct $\boldsymbol{\theta}$	Neural $RR^{\top}\boldsymbol{\theta}$	Permuted/random routing
Cosine with $\boldsymbol{\theta}$	$1.00$	$0.30$ – $0.37$	–
Energy in $1\times 1$ and $4\times 4$ blocks	$3\%$ – $9\%$	$23\%$ – $45\%$	${<}\,3\%$
PixCorr with $\alpha=0$	$0.60$ – $0.86$	$0.13$ – $0.24$	$0.92$ – $0.98$
Memorability: images with intended $0\rightarrow+2$ change	$65\%$ – $92\%$	$0\%$ – $16\%$	Not consistently observed
Subjects with monotonic memorability	$4/4$	$0/4$	$0/4$

Appendix

Table B2: Comparison of direct and neural-routed latent perturbations. Ranges summarize the four NSD subjects and the two cognitive attributes where applicable. PixCorr was computed relative to the corresponding unperturbed reconstruction.

Figure C1: Held-out validation of the empirical steering directions. (a) Distribution of held-out VDVAE latent projections onto the EmoNet valence direction for images in the bottom 15%, central 70%, and top 15% of held-out EmoNet scores. Dashed lines mark the mean projections of the low- and high-scoring groups. (b) Held-out EmoNet scores versus projection onto the valence direction ( $r=0.812$ ). (c) Corresponding projection distributions for the MemNet memorability direction. (d) Held-out MemNet scores versus projection onto the memorability direction ( $r=0.834$ ). The panels show fMRI subject 1; each direction was estimated from training-image latents only. Pearson correlations quantify held-out geometric correspondence with the frozen assessor; they do not constitute human or neural validation.

Figure D1: Illustrative valence and memorability perturbation sequences. (a) Two valence-targeted Versatile Diffusion reconstruction sequences from one fMRI subject. (b) Two memorability-targeted sequences from the same reconstruction pipeline. The first column shows the original NSD image; the remaining columns show reconstructions at $\alpha\in\{-4,-2,0,+2,+4\}$ . The dark border identifies the unperturbed $\alpha=0$ reconstruction. Negative and positive values indicate the intended directions of decreasing and increasing automated attribute scores, respectively. The examples were selected for visibly interpretable trajectories and are not intended to represent the distribution of effects across all images. They illustrate both the intended modulation and the increasing generative drift observed at extreme perturbation strengths. Additional examples are available at https://www.neuralsurrogate.com/ .

Figure E1: Surrogate-predicted neural patterns under extreme cognitive-attribute perturbations. Predicted cortical beta-pattern differences relative to the unperturbed condition ( $\alpha=0$ ). Panels a and b show negative ( $\alpha=-4$ ) and positive ( $\alpha=+4$ ) valence perturbations, respectively; panels c and d show low- ( $\alpha=-4$ ) and high-memorability ( $\alpha=+4$ ) perturbations. Yellow indicates predicted increases and cyan indicates predicted decreases relative to $\alpha=0$ . These maps are descriptive outputs of the fMRI-derived surrogate and do not represent independently measured activation changes or validated stimulation targets. No inferential statistical tests were performed.

Figure F1: Behavioral time-course and negative-control diagnostics. (a,b) Changes in valence and perceived-memorability ratings across the ten fixed block positions, expressed relative to the unperturbed reconstruction block ( $\alpha=0$ , block 2). Small points show individual participants; lines connect only the group means, and error bars denote 95% confidence intervals. (c) Participant-specific within-block rating slopes for valence, perceived memorability, and arousal. Slopes were estimated across the 40 trial positions within each block. (d) Arousal-rating changes across block position, used as a negative-control outcome because arousal was not directly targeted. The between-block curves are descriptive rather than causal estimates because block position, image composition, and perturbation condition were not independently randomized.

Figure F2: Sensitivity of the human effects to alternative detrending specifications. (a,b) Mean valence and perceived-memorability changes under the raw, baseline-composition, within-block-only, assessor-neutral linear-time, and assessor-neutral quadratic-time specifications. Changes are expressed relative to $\alpha=0$ in participant-specific baseline-SD units; error bars denote 95% confidence intervals. (c,d) Participant-level observations after the primary leave-one-participant-out assessor-neutral linear correction. Small points are unconnected participant estimates; lines join only group means. The comparison shows that the valence inference depends principally on correction of the between-block trend, whereas perceived memorability remains close to zero under every specification.

	Valence		Perceived memorability
Analysis	Mean slope [95% CI]	Positive	Mean slope [95% CI]	Positive
Raw	$\phantom{-}0.006$ [ $-0.021$ , $\phantom{-}0.034$ ]	11/18	$-0.004$ [ $-0.021$ , $\phantom{-}0.013$ ]	7/18
Baseline composition	$\phantom{-}0.014$ [ $-0.015$ , $\phantom{-}0.044$ ]	12/18	$-0.004$ [ $-0.021$ , $\phantom{-}0.013$ ]	7/18
Within-block only	$\phantom{-}0.006$ [ $-0.021$ , $\phantom{-}0.034$ ]	11/18	$-0.004$ [ $-0.021$ , $\phantom{-}0.013$ ]	7/18
Assessor-neutral linear (primary)	$\phantom{-}0.038$ [ $\phantom{-}0.003$ , $\phantom{-}0.074$ ]	16/18	$-0.011$ [ $-0.028$ , $\phantom{-}0.007$ ]	5/18
Assessor-neutral quadratic	$\phantom{-}0.010$ [ $-0.020$ , $\phantom{-}0.040$ ]	11/18	$-0.009$ [ $-0.027$ , $\phantom{-}0.008$ ]	6/18

Appendix

Table F1: Full-range participant-slope sensitivity across behavioral analyses. Slopes were estimated across

\alpha=-4,-2,0,+2,+4

and are expressed in participant-specific baseline SD per unit of

\alpha

. Confidence intervals summarize the participant-level slopes. Exact-source slopes were calculated from the corresponding participant-specific matched-image condition means.

Figure F3: Exact-source matched behavioral sensitivity analysis. (a) Valence changes when each nonzero- $\alpha$ image was paired with the same source image in the $\alpha=0$ block. (b) Corresponding analysis for perceived memorability. Cyan squares show all available exact-source pairs; magenta diamonds additionally restrict the analysis to assessor-neutral images with an absolute Versatile Diffusion assessor change no larger than 0.50 baseline SD. Small points show individual participants; lines connect only group means, and error bars denote 95% confidence intervals. Values are displayed on a signed symmetric-logarithmic scale. Exact matching controls source-image identity but does not remove the fixed-order confound; the leave-one-participant-out linear time correction was therefore retained.

Neural Surrogate Modeling

Neural surrogate modeling uses learned models to approximate costly simulations, enabling faster prediction, design, control, and inference.

See all 418 papers

EmoMind: Decoding Affective Captions from Human Brain fMRI
May 16, 2026Bilal A. Mohammed, Lin Gu, Ruogu FangImage Captioning Neural Decoding
Department of Biomedical Engineering Vanderbilt University Nashville, TN · Research Institute of Electrical Communication Tohoku University Sendai, Japan · Department of Biomedical Engineering University of Florida Gainesville, FL
Predicting Neuromodulation Outcome for Parkinson's Disease with Generative Virtual Brain Model
Mar 31, 2026Siyuan Du, Siyi Li, Shuwei Bai +10Clinical Outcome Prediction Parkinson's Disease
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China · Shanghai AI Laboratory, Shanghai, China · University of Science and Technology of China, Anhui, China +6
Activation-Guided Neuron Intervention to Induce Alzheimer's-Related Computational Language Phenotypes in a Large Language Model
Aug 4, 2026Rui He, Ercong Nie, Hong Jiang +3Alzheimer's Disease LLM Interpretability
Grammar and Cognition Lab, Department of Translation & Language Sciences, Universitat Pompeu Fabra, Barcelona, Spain. · School of Foreign Languages, Shanghai Jiao Tong University, Shanghai, China. · Zhuhai People’s Hospital (The Affiliated Hospital of Beijing Institute of Technology, Zhuhai Clinical Medical College of Jinan University), Zhuhai, China. +4

May 16, 2026J/K move · Enter open · S save

EmoMind: Decoding Affective Captions from Human Brain fMRI

Bilal A. Mohammed, Lin Gu, Ruogu Fang

Department of Biomedical Engineering Vanderbilt University Nashville, TN · Research Institute of Electrical Communication Tohoku University Sendai, Japan · Department of Biomedical Engineering University of Florida Gainesville, FL

Image Captioning Neural Decoding

Open paper AskPDF arXiv

Decoding visual experience from brain activity has advanced substantially, but current brain-to-text systems largely recover semantic content while discarding affect. Additionally, language models can generate emotional text when prompted with categorical labels, but such labels collapse rich inter-subject variability into coarse discrete bins. We present EmoMind, the first end-to-end pipeline for decoding affective captions directly from fMRI signals. EmoMind first retrieves a semantically grounded neutral scene description from brain-decoded visual features, then rewrites it using a continuous 34-dimensional emotion vector decoded from the same fMRI recording. To control the balance between content preservation and affective expression, we train the rewriter with classifier-free guidance against an identity-preserving null branch, enabling smooth interpolation between semantic fidelity and affective expressivity. We evaluate affective caption generation with a three-axis validation framework spanning subject-specificity, structural geometry, and causal control. We further augment this framework with a synthetic-brain substitution test that probes robustness to the measurement apparatus, and we benchmark each axis against GPT-4 prompted with brain-decoded top-5 emotion labels as a strong discrete baseline. Across two independent emotion fMRI datasets, EmoMind significantly outperforms label-prompted GPT-4 on all three axes, with the largest gains on metrics that require person-specific affective structure rather than population-level emotion aggregation. These results establish continuous brain-decoded affect as a viable control signal for individualized affective caption generation and open new directions for studying individual affective brain organisation.

Abstract

Neural Surrogate Modeling

Explore similar work

EmoMind: Decoding Affective Captions from Human Brain fMRI

Predicting Neuromodulation Outcome for Parkinson's Disease with Generative Virtual Brain Model

Activation-Guided Neuron Intervention to Induce Alzheimer's-Related Computational Language Phenotypes in a Large Language Model