cs.CLSep 28, 2026

Cognitive Expert Language Models Better Align with the Corresponding Brain Systems

Authors: Zhivar Sourati, Mengxuan Helen Wu, Nona Ghazizadeh, Jonas Kaplan, Morteza Dehghani, Samuel A. Nastase

Organizations: Department of Computer Science, University of Southern California · Center for Computational Language Sciences, University of Southern California · Department of Psychology, University of Southern California

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) can predict human brain activity across a variety of brain regions during natural language comprehension. Typically, however, LLM-brain alignment is measured using one model for different regions of the brain, and then model performance is summarized across regions. This one-model-fits-all approach ignores the functional specialization of brain regions. In this study, we assess whether a model oriented toward a particular cognitive domain aligns better with the brain system dedicated to that domain. Through prompting and fine-tuning, we first build expert LLM variants for six domains: sensory, spatial, numerical, reasoning, social, and abstract processing. We then examine whether each expert best predicts activity in the brain region associated with the corresponding cognitive domain. Consistent with our hypotheses, each expert's representations align more closely with the brain system most associated with the matching domain than do other experts. This holds under both prompting and fine-tuning, across three base models and three fMRI datasets. In a series of control analyses, we show that this model-brain alignment is specific to cognitive domain interventions; non-cognitive and surface-level interventions do not result in comparable alignment. Specializing models shifts regional alignment while leaving aggregate prediction accuracy largely unchanged, suggesting that summarizing alignment across regions may obscure regional differences in performance for specific models.

Figures & tables

Figure 1: Measuring alignment with specialized brain systems. (a) A voxelwise encoding model predicts each voxel’s response to a stimulus from a language model’s features and is scored on held-out data, giving an alignment map over the cortex (here for the base model, in one LeBel participant). (b) We build one expert per cognitive domain by prompting or fine-tuning, pass each through the same steps, and ask whether each domain’s Neurosynth region is predicted best by its own expert; the matrix ranks the six experts in each region for prompted LLaMA on the LeBel dataset, the running example of Figure 3 (displacement is computed per participant, then averaged).
Figure 2: Behavioral validation of the experts. LLaMA. (a) Fine-tuning: change in perplexity (%) relative to the base model, each expert (row) on each domain’s held-out text (column). (b) Prompting: change in the share of generated words from each domain’s lexicon (percentage points). (c) Prompting: each expert’s rank on each content kind by a blind judge (1 = highest). Outlined cells mark the matched expert; each column compares the six experts on one domain.
Figure 3: Expert models are aligned with the corresponding brain systems. Prompted LLaMA on LeBel. (a) The six regions on the exemplar subject’s cortical surface, each filled with its own expert’s color where that expert ranks above chance (the other two subjects are in Figure 20 ). (b) Each expert’s rank within each region (row; 1 = best, chance = 3.5) on its accuracy inside minus outside the region relative to the family mean, averaged over subjects ( Table 1 averages per-subject displacement instead). Outlined cells are the matched experts; bars give their rank relative to chance.
ModelInterventionDatasetnnDisplacementppvs nullΔr×103\Delta r\times 10^{3}ppRegions
LLaMAPromptingLeBel3+1.56.003100th+1.25.0116/6
  LPP49+0.42.017100th+0.39<< .0015/6
  Pereira10+0.54.017100th+6.51.0134/5
  all pooled62+0.49<< .001 +1.42.002 
 Fine-tuningLeBel3+0.89.01999th+0.77.1173/6
  LPP49+0.30.01498th+0.41<< .0014/6
Table 1: Regional alignment for every model family, intervention, and dataset. Displacement: 3.5 minus the mean rank of each region’s own expert (3.0 on Pereira; 0 = chance), with its exact relabeling pp . vs null: percentile within the seed-only control family’s displacement distribution at the same setting. Δr\Delta r : the matched expert’s accuracy inside its region minus the other experts’, tested across subjects. Regions: regions whose own expert ranks above chance. Bold: pp < .05 (vs null: beyond the 95th percentile). Pooled rows give the mean over all 62 subjects, with pp from a test that standardizes each subject’s displacement by its own permutation null ( Appendix D ).
Figure 4: Comparing the two interventions. (a) Dissimilarity of the six experts’ stimulus representations (1 −- mean pairwise CKA) by layer, for every family on LPP and for the surface-form control. (b) Dissimilarity of each fine-tuned family’s six prediction maps (1 −- mean pairwise correlation) against its displacement, one point per model, dataset and layer. (c) Each expert’s own-region displacement under each intervention (rings 0.5 apart), averaged over the narrative datasets and base models; dots are single configurations, and values below chance sit at the center.
Figure 5: Cognitive domain experts yield stronger regional alignment than control interventions. LLaMA on LeBel, each intervention at its Table 1 setting. Bands: the central 90% of a control family’s displacement over all 720 assignments of its members to the regions. Points: the expert family, each expert paired with its own domain’s region, with exact permutation pp .
Figure 6: Specialization improves regional alignment without changing overall alignment. (a) Turning LLaMA into experts: change in prediction accuracy against displacement, one point per LPP subject; the base model, read as six identical copies, has displacement 0. (b) Correlation across a family’s six experts between accuracy and (left) rank over all six regions or (right) how much higher an expert ranks in its own region than in the others; mean and 95% interval over 62 subjects. Only the right-hand quantity enters displacement.
Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

DomainInstructionDistilledLong-form additionsTrainValidation
Sensory17,8433,673none21,516557
Spatial14,9163,668Wikipedia geography 2,85021,434647
Numerical35,7413,670Wikipedia biographies 2,852; GovReport 1,89844,1611,172
Reasoning12,1473,674Mixture-of-Thoughts 2,849; LongWriter 1,33620,006660
Social47,0393,674SODA 2,844; HH-RLHF helpful 1,90655,4631,402
Abstract2,8113,667PG-19 5,711; Gutenberg 1,765; GoodWiki philosophy 1,60715,561821
Appendix
Table 2: Composition of each domain’s fine-tuning set. Rows after deduplication. Instruction corpora are the filtered public sets; distilled are the teacher-generated domain commentaries.
Figure 7: The six regions in every LeBel subject. Left: the Neurosynth regions for the six domains on each subject’s flattened cortical surface, with the dataset’s localizer and atlas regions outlined. Right: the prediction accuracy of the prompted LLaMA family (mean over its six experts), each subject on its own color scale. Right (RH) and left (LH) hemispheres are shown.
Figure 8: The perplexity matrix of every fine-tuned family. Percentage change in perplexity relative to the base model, each expert (rows) on each domain’s held-out text (columns), for the three base models at three adapter ranks; the white dot marks each column’s strongest improvement.
Figure 9: Lexicon share of generated text for every base model. As Figure 2 b, for LLaMA, Qwen and Phi-4.
Figure 10: The blind judge’s ranking for every base model. Each cell is the expert’s rank on that content kind (1 = the judge scores this expert’s continuations highest of the six, relative to the base model); the outlined diagonal is the matched cell.
Figure 11: LeBel, LLaMA fine-tuning and Qwen prompting. Each row is one configuration in the format of Figure 3 . Left: every region is outlined and labeled, and filled with its own expert’s color when that expert beats chance inside it, with the expert’s rank printed beside the region name. Right: the expert ×\times region matrix, where each cell prints that expert’s rank within the region’s row and the region’s own expert is outlined, and beside it the matched expert’s rank against chance.
Figure 12: LeBel, Qwen fine-tuning and Phi-4 prompting. Conventions as in Figure 11 .
Figure 13: LeBel, Phi-4 fine-tuning. Conventions as in Figure 11 .
Figure 14: Le Petit Prince, LLaMA prompting and fine-tuning. Conventions as in Figure 11 .
Figure 15: Le Petit Prince, Qwen prompting and fine-tuning. Conventions as in Figure 11 .
Figure 16: Le Petit Prince, Phi-4 prompting and fine-tuning. Conventions as in Figure 11 .
Figure 17: Pereira, LLaMA prompting and fine-tuning. Conventions as in Figure 11 .
Figure 18: Pereira, Qwen prompting and fine-tuning. Conventions as in Figure 11 .
Figure 19: Pereira, Phi-4 prompting and fine-tuning. Conventions as in Figure 11 .
Figure 20: The running example in the other two LeBel subjects. Each row as Figure 3 , for UTS01 and UTS03, with each subject’s own ranks: the region map, the region ×\times expert matrix, and each matched expert’s rank relative to chance.
Figure 21: Per-expert differential maps (UTS01, LeBel, prompted LLaMA). Each panel: one expert’s encoding accuracy minus the family mean, on the cortical surface (smoothed for display), with that expert’s own region outlined; for display, the color scale is centered at the expert’s cortex-wide level. The printed values are the means of the unsmoothed map inside and outside the region, and the check mark records which is larger.
Figure 22: The same panels for UTS02 (top) and UTS03 (bottom). Conventions as in Figure 21 .
Figure 23: Per-subject region ×\times expert matrices for the six LeBel configurations. Each configuration at its Table 1 setting; conventions as in Figure 3 b, one panel per subject.
Figure 24: The network by expert matrices behind the Pereira numbers. Left, LLaMA prompting; right, the surface-form control on the same base model. Rows are the five functional networks, columns the five experts with a matched network, and each cell prints that expert’s rank within the network’s row, with the network’s own expert outlined. The surface-form control has no matched expert, so its outlined cells are one arbitrary assignment of its members, shown on identical conventions.
Figure 25: The result under every way of scoring it. Eight scoring statistics ×\times three voxel sets, for all eighteen configurations. Bars: how many cognitive configurations exceed their matched control (surface-form for fine-tuning, random-prompt for prompting, same dataset), each cognitive configuration at the layer and rank chosen under that statistic as for Table 1 and its control, scored in its fixed member order, at the same setting; the dashed line is the even-odds level, against which the region statistics’ worst result (13/18) has binomial pp = .048 and their best (16/18) pp = .0007. Right margin: every combination finds every computable configuration positive, with the one exception noted in the text, which sits at or just below zero. The whole-cortex statistics need continuous Neurosynth maps, which exist for LeBel and Le Petit Prince (twelve configurations); Pereira provides binary networks only.
Figure 26: Δr\Delta r standardized across subjects for every configuration. Cohen’s d of the per-subject Δr\Delta r , one point per model ×\times intervention ×\times dataset configuration.
Figure 27: Every intervention contrast. Which intervention has the larger (stronger) value on each of the 45 properties, in each of the six base model ×\times dataset combinations; shaded rows are consistent across all six.
Figure 28: The intervention difference per expert. Prompting minus fine-tuning own-region displacement per expert, pooled over the two narrative datasets and three base models (bar) with the six individual configurations (open LeBel, filled Le Petit Prince; circle LLaMA, square Qwen, diamond Phi-4).
Figure 29: Where every configuration falls inside the null. (a) Each of the eighteen configurations of Table 1 as its percentile within the seed-only control family’s assignment distribution; asterisks mark individually significant configurations. (b) The same eighteen values on one axis; the shaded region is the lower half of the distribution, where a random assignment falls with probability one half.
Figure 30: The size of a change and the place it lands. (a) The size of the change, SS : the members’ contrasts in absolute value, averaged over members, regions and subjects, with no ranking and no pairing to regions. (b) Displacement, DD , under the theory-given pairing with its exact permutation pp , for the expert families at their Table 1 settings (circle LLaMA, square Qwen, diamond Phi-4), against the central 90% of the control families at the same settings (grey).
DatasetModelInstructionppVocabularyppInstruction ahead
LeBelLLaMA+1.56.003+0.22.342yes
LeBelQwen+0.89.007+1.22.004no
LeBelPhi-4+1.17.003−- 0.17.654yes
LPPLLaMA+0.42.017+0.12.242yes
LPPQwen+0.07.321+0.18.151no
LPPPhi-4+0.17.214+0.16.226yes
Appendix
Table 3: Instruction against bare vocabulary in every base model ×\times dataset comparison. Own-region displacement of the cognitive prompting family (instruction) and of the keyword family (vocabulary), both read at the expert family’s Table 1 setting, with each family’s exact relabeling pp (bold: displacement with pp below .05).
Figure 31: The same cortical map for cognitive experts and for a control. Left, the running example (LLaMA, prompting) on the conventions of Figure 3 ; right, the random-prompt control at the same setting under one arbitrary assignment. A region is filled when its assigned expert ranks above chance inside it and hatched when it does not.
More in this topic

Cognitive Modeling

Cognitive modeling uses computational theories to explain and predict how people perceive, learn, remember, reason, and decide.

See all 119 papers

Explore similar work

CardsList
Jun 10, 2026

Beyond representational alignment with brain-guided language models for robust reasoning

The correspondence between large language models (LLMs) and the neural mechanisms underlying human higher-order cognition remains insufficiently characterized. Given that language and reasoning in the human brain appear dissociable, an open question is whether LLMs align with neural signals from reasoning-related regions and whether such signals can improve them. Here, focusing on deductive reasoning, we show that LLM internal representations are not only partially aligned with task-fMRI activity but can also be directly enhanced by these signals. Using a neural-predictivity metric, we find that LLMs explain a substantial fraction of the explainable variance in reasoning-related regions at the aggregate level, whereas predictivity within specific reasoning types is lower, indicating both alignment and divergence. Building on this, we propose a brain-guided framework: we steer model representations along directions induced by the joint structure of model and brain representations, applying intervention at inference and fine-tuning during training. We demonstrate that task-evoked brain signals can directly enhance LLM reasoning, yielding gains orthogonal to language-only supervision across 10 LLMs (1.5B-72B), with transfer across reasoning types and up to 13% absolute accuracy gain. Our results advance LLM-brain correspondences from correlation to guidance, establishing a brain-signal-driven pathway toward more robust and cognitively aligned AI.

Mingqing Xiao, Kai Du, Zhouchen Lin

1State Key Lab of General AI, School of Intelligence Science and Technology, Peking University. · 3Microsoft Research Asia. · Department of Psychological and Cognitive Sciences, Tsinghua University.

Logical ReasoningLanguage ModelingLLM Reasoning
AskPDF
May 20, 2026

Cross-lingual brain-language model alignment is robust but challenges hierarchical and computational accounts

Brain-language model alignment is often interpreted as evidence that transformer models implement computations similar to those of the human brain. This assumes that neural predictivity reflects internal computational properties of large language models (LLMs), such as hierarchical contextual processing, predictive coding, or representational compression. An alternative possibility is that brain scores primarily reflect stable lexical-semantic correspondences shared by language models and the brain. Here we tested these interpretations using whole-brain encoding models across Mandarin, English, and French. Across all three languages, transformer representations significantly predicted activity in a distributed network spanning classical language regions, transmodal cortical systems, and subcortical structures. These spatial patterns showed substantial cross-linguistic overlap and remained remarkably stable across layers, providing little evidence that model depth systematically maps onto cortical processing hierarchies. Likewise, contextual transformer embeddings did not consistently outperform static lexical embeddings, despite providing some unique predictive variance. Finally, neither surprisal nor intrinsic dimensionality reproduced the layer-wise profile of brain scores, arguing against prediction and information compression as primary explanations for brain-LLM alignment. Together, these findings suggest that brain-LLM alignment is more robust across languages, transformer depth, and model architectures than previously appreciated, but less informative about shared computational mechanisms. Our results are more consistent with neural predictivity reflecting stable representational structure preserved across model transformations than with a one-to-one correspondence between their underlying computations.

Ni Yang, Rui He, Philipp Homan +3

Grammar and Cognition Lab, Department of Translation & Language Sciences, Universitat Pompeu Fabra, Barcelona, Spain. · Department of Adult Psychiatry and Psychotherapy, University of Zurich, Zurich, Switzerland. · Neuroscience Center Zurich, University of Zurich and ETH Zurich, Zurich, Switzerland. +3

Neural Encoding
AskPDF
Mar 24, 2026

When Language Models Lose Their Mind: The Consequences of Brain Misalignment

While brain-aligned large language models (LLMs) have garnered attention for their potential as cognitive models and for potential for enhanced safety and trustworthiness in AI, the role of this brain alignment for linguistic competence remains uncertain. In this work, we investigate the functional implications of brain alignment by introducing brain-misaligned models--LLMs intentionally trained to predict brain activity poorly while maintaining high language modeling performance. We evaluate these models on over 200 downstream tasks encompassing diverse linguistic domains, including semantics, syntax, discourse, reasoning, and morphology. By comparing brain-misaligned models with well-matched brain-aligned counterparts, we isolate the specific impact of brain alignment on language understanding. Our experiments reveal that brain misalignment substantially impairs downstream performance, highlighting the critical role of brain alignment in achieving robust linguistic competence. These findings underscore the importance of brain alignment in LLMs and offer novel insights into the relationship between neural representations and linguistic processing.

Gabriele Merlin, Mariya Toneva

MPI-SWS, Saarbrücken, Germany

LLM Alignment
AskPDF