Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?

Yuxuan Gu; Lunjun Liu; Xiaocheng Feng; Kun Zhu; Weihong Zhong; Lei Huang; Bing Qin

cs.CLMar 29, 2026

Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?

Authors: Yuxuan Gu, Lunjun Liu, Xiaocheng Feng, Kun Zhu, Weihong Zhong, Lei Huang, Bing Qin

Organizations: Harbin Institute of Technology.

Abstract

An essential problem in artificial intelligence is whether LLMs can simulate human cognition or merely imitate surface-level behaviors, while existing datasets suffer from either synthetic reasoning traces or population-level aggregation, failing to capture authentic individual cognitive patterns. We introduce a benchmark grounded in the longitudinal research trajectories of 217 researchers across diverse domains of artificial intelligence, where each author's scientific publications serve as an externalized representation of their cognitive processes. To distinguish whether LLMs transfer cognitive patterns or merely imitate behaviors, our benchmark deliberately employs a cross-domain, temporal-shift generalization setting. A multidimensional cognitive alignment metric is further proposed to assess individual-level cognitive consistency. Through systematic evaluation of state-of-the-art LLMs and various enhancement techniques, we provide a first-stage empirical study on the questions: (1) How well do current LLMs simulate human cognition? and (2) How far can existing techniques enhance these capabilities?

Figures & tables

Figure 1: Motivating case of human simulation. The memory panel presents two representative cases of the author’s prior research with reasoning traits: (1) an explicit awareness of shallow heuristic modeling in LLMs and a deliberate effort to avoid such shortcuts, and (2) a problem-solving style that starts from modeling the research objectives . Upper-right Panel : LLMs can correctly interpret the task objective but seem to rely on directly imitating specific methods from the research memory. Lower-right Panel : In contrast, the authentic author shares a consistent abstraction and reasoning trajectory across problem scenarios. This comparison illustrates a central question motivating our study: whether LLMs can transcend behavioral imitation to internalize and reproduce human-like cognitive processes.

Figure 2: Overview of the construction pipeline for our benchmark. It consists of two separate lines: Expert Annotation , which involves fine-grained verification and systematic quality filtering at each stage to ensure data reliability, and Automatic Annotation , which utilizes OCR and LLM-based extraction for expandable data collection. The Segmentation Protocol shows a splitting strategy based on the average publication dates of research topics, enabling cognitive evolution from the test set to the train set.

Figure 3: Research topics in our benchmark.

Figure 4: Characterization of temporal and semantic variance between train and test sets across scholars. (A) Distribution of temporal shifts, quantified as the difference in average publication year (test minus train). Almost all authors’ test sets are more recent than their train sets, with the distribution peaking at about

2

years. (B) Distribution of semantic gaps, measured by the normalized centroid L2 distance between each author’s train and test set embeddings. The dashed line indicates the baseline distance between the global centroids of all train and test sets. It reveals substantial within-author semantic distance concentrated between

0.7

and

0.9

Figure 5: Overview of our evaluation framework.

Statistic	Expert	Automatic	Total
# Scholars	100	117	217
# Total Papers	1,875	1,547	3,422
Papers per Author	18.8	13.2	15.8
Clusters per Author	2.3	2.3	2.3
Tokens per Component
Introduction	614.9	637.2	625.0

Table 1: Statistics of our benchmark.

Method	Lexical		Semantic	Cognitive
Method	ROUGE-1/2/L	BertScore	Semantic	P.	S.	L.	K.	T.
LLaMA-3.1	$22.12/3.91/12.02$	$81.39$	$3.22$	$3.03$	$3.15$	$3.00$	$3.05$	$3.20$
- Other	$23.11/3.94/12.70$	$81.63$	$2.94$	$2.74\!\,\color[rgb]{0.5,0,0}{\downarrow}$	$2.95\!\,\color[rgb]{0.5,0,0}{\downarrow}$	$2.73\!\,\color[rgb]{0.5,0,0}{\downarrow}$	$2.89\!\,\color[rgb]{0.5,0,0}{\downarrow}$	$3.09\!\,\color[rgb]{0.5,0,0}{\downarrow}$
- Self	$24.10/4.45/13.10$	$81.89$	$3.13$	$3.02$	$3.23\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$2.98$	$3.06$	$3.32\!\,\color[rgb]{0,0.5,0}{\uparrow}$
- SelfAkin	$25.65/5.93/14.12$	$82.36$	$3.44$	$3.55\!\,\color[rgb]{0,0.5,0}{\mathrel{\scalebox{0.8}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.8}[1]{$ \uparrow $}}}$	$3.77\!\,\color[rgb]{0,0.5,0}{\mathrel{\scalebox{0.8}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.8}[1]{$ \uparrow $}}}$	$3.41\!\,\color[rgb]{0,0.5,0}{\mathrel{\scalebox{0.8}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.8}[1]{$ \uparrow $}}}$	$3.34\!\,\color[rgb]{0,0.5,0}{\mathrel{\scalebox{0.8}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.8}[1]{$ \uparrow $}}}$	$3.82\!\,\color[rgb]{0,0.5,0}{\mathrel{\scalebox{0.8}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.8}[1]{$ \uparrow $}}}$
- Leaky	$26.70/6.79/14.75$	$82.66$	$3.63$	$3.77\!\,\color[rgb]{0,0.5,0}{\mathrel{\scalebox{0.7}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.7}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.7}[1]{$ \uparrow $}}}$	$3.86\!\,\color[rgb]{0,0.5,0}{\mathrel{\scalebox{0.7}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.7}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.7}[1]{$ \uparrow $}}}$	$3.55\!\,\color[rgb]{0,0.5,0}{\mathrel{\scalebox{0.7}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.7}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.7}[1]{$ \uparrow $}}}$	$3.45\!\,\color[rgb]{0,0.5,0}{\mathrel{\scalebox{0.7}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.7}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.7}[1]{$ \uparrow $}}}$	$3.92\!\,\color[rgb]{0,0.5,0}{\mathrel{\scalebox{0.7}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.7}[1]{$ \uparrow $}\mkern 0.0mu\scalebox{0.7}[1]{$ \uparrow $}}}$

Table 2: Main results of simulating human cognition across different LLMs. We evaluate five scenarios: (1) Base Model represents the original output without any cognition history conditioning. (2) Other uses cognition histories randomly sampled from other authors, serving as an interference control group. (3) Self incorporates the target author’s cognition history from the train set, representing our standard cross-domain setting. (4) SelfAkin leverages cognition histories from the same author’s test set, examining in-domain behavioral leakage. (5) Leaky provides oracle access to the author’s authentic cognition trajectory, serving as an upper bound. We report ROUGE-1/2/L and BertScore for lexical coverage, LLM-as-a-Judge for semantic similarity, and five cognitive consistency dimensions: Foundational P remise, S trategic Preference, L ogic Mapping, K eyword Style, and T aboo Avoidance. See Appendix E for more LLMs.

Method	A	B	C	D
Qwen-3-235B - Self	$11\%$	$58\%$	$31\%$	$\phantom{0}0\%$
- SelfAkin	$\phantom{0}3\%$	$35\%$	$60\%$	$\phantom{0}1\%$
GPT-5.1 - Self	$\phantom{0}9\%$	$51\%$	$38\%$	$\phantom{0}2\%$
- SelfAkin	$\phantom{0}3\%$	$26\%$	$67\%$	$\phantom{0}4\%$

Table 3: Human Evaluation. Three senior Ph.D. students evaluate 100 papers per method (one randomly sampled per scholar). Annotation scales: A is no relation to author’s cognition; B is simple imitation of historical research; C is attempted but failed cognitive simulation, remaining behavioral; D is successful thinking pattern transfer. Inter-annotator agreement (Fleiss’ kappa):

0.43

Method	Lexical		Semantic	Cognitive
Method	ROUGE-1/2/L	BertScore	Semantic	P.	S.	L.	K.	T.
LLaMA-3.1	$22.12/3.91/12.02$	$81.39$	$3.22$	$3.03$	$3.15$	$3.00$	$3.05$	$3.20$
- Self	$24.10/4.45/13.10$	$81.89$	$3.13$	$3.02$	$3.23$	$2.98$	$3.06$	$3.32$
- PersonalityTrait	$21.74/3.82/11.77$	$81.54$	$3.21$	$3.10\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$3.35\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$3.07\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$3.16\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$3.36\!\,\color[rgb]{0,0.5,0}{\uparrow}$
- CognitiveGuide	$19.36/3.48/10.45$	$81.64$	$3.24$	$3.23\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$3.47\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$3.24\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$3.21\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$3.50\!\,\color[rgb]{0,0.5,0}{\uparrow}$
- TrainIndividual	$22.49/3.94/12.19$	$81.42$	$3.21$	$3.05$	$3.31\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$3.00$	$3.15\!\,\color[rgb]{0,0.5,0}{\uparrow}$	$3.28\!\,\color[rgb]{0,0.5,0}{\uparrow}$

Table 4: Evaluation of enhancement techniques for human cognitive simulation. We assess the effectiveness of various augmentation strategies on two open-source LLMs: LLaMA-3.1-8B and Qwen-3-8B . The base model output and the Self condition are baselines. We evaluate two categories of enhancement techniques: (1) Prompting techniques, including PersonalityTrait that incorporates personality traits as psychological priors, and CognitiveGuide which explicitly instructs LLMs with cognitive theories; (2) Learning-based methods, including TrainIndividual which trains independent parameters for each scholar, and TrainUnified that extracts profiles for scholars as their identifiers to train a unified model. The evaluation metrics are shared with those used in Table 2 .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic	Expert	Automatic	Total
# Scholars	100	117	217
# Total Papers	1,875	1,547	3,422
Papers per Author	18.8	13.2	15.8
Clusters per Author	2.3	2.3	2.3
Tokens per Component
Introduction	614.9	637.2	625.0

Appendix

Table 5: Statistics of our benchmark.

Alignment Metric	Count
Overlapping Authors	21
Total Papers Compared	224
Valid Papers (Success)	199
Invalid Papers (Failed)	25

Appendix

Table 6: Statistics of the overlapping subset used for quality alignment check on automatic annotation.

Component	Precision	Recall	F1 Score
Background	0.8834	0.6749	0.7318
Motivation	0.6788	0.5412	0.5684
Approach	0.7596	0.6085	0.6405
Solution	0.7557	0.5661	0.6047

Appendix

Table 7: Token-level agreement between the expert annotation and the automated extraction.

Agreement Metric	Value
Sample Size	30 Authors
Raters	1 Human + 2 LLMs
Absolute Agreement (3/3)	70.0% (21/30)
Fleiss’ Kappa ( $\kappa$ )	0.44 (Moderate)
Gwet’s AC1	0.69 (Substantial)

Appendix

Table 8: Inter-rater reliability for the automated topic clustering validation. Gwet’s AC1 is reported as the primary metric due to its robustness against prevalence paradox in high-quality data.

Model A: Qwen3-8B
Hyperparameter	Unified	Individual
Rank ( $r$ )	32	16
Alpha ( $\alpha$ )	64	32
Learning Rate	5e-5	2e-5
Target Modules	Attn Only ‡	All Linear
Max Seq Length	8,192	2,048

Appendix

Table 9: Hyperparameter Settings. Optimal configurations for each strategy. Note that the Qwen-Unified model utilizes an efficient Attention-Only approach.

Dimension	Assessment Protocol
Foundational Premise	Focus: The theoretical lens and problem formulation style. Target (Score 5): The solution must accurately reconstruct how the author typically frames problems (e.g., framing optimization as probabilistic inference) and their foundational knowledge base.
Strategic Preference	Focus: Value trade-offs and research paradigms. Target (Score 5): The solution must mirror the author’s persistent choices, such as favoring interpretable frameworks over black-box performance, or theoretical modeling over engineering hacks.
High-Level Logic	Focus: Argumentation architecture and challenge-response patterns. Target (Score 5): The reasoning flow must follow the author’s deductive style (e.g., “Challenge A $\rightarrow$ Innovation B”) rather than generic logical templates.
Keyword Style	Focus: Domain-specific terminology and linguistic habits. Target (Score 5): The text naturally embeds the author’s unique vocabulary (e.g., using ”manifold” instead of ”space”) without mechanical forcing.
Taboo Avoidance [Inverse Metric]	Focus: Forbidden methodologies, assumptions, or baselines. Target (Score 5): The solution successfully avoids or actively critiques approaches the author historically rejects (e.g., explicitly excluding brute-force data fitting).

Appendix

Table 10: Detailed Definition of Cognitive Dimensions. This matrix guides the evaluator to map specific inherited traits to the generated solution.

Method	Lexical		Semantic	Cognitive
Method	ROUGE-1/2/L	BertScore	Semantic	P.	S.	L.	K.	T.
LLaMA-3-8B	$22.28/3.86/12.27$	$81.46$	$3.16$	$2.98$	$3.17$	$2.90$	$3.11$	$3.18$
- Other	$23.86/3.85/13.13$	$81.57$	$2.70$	$2.56$	$2.83$	$2.58$	$2.89$	$3.05$
- Self	$24.95/4.36/13.48$	$81.89$	$3.01$	$2.95$	$3.19$	$2.87$	$3.03$	$3.42$
- SelfAkin	$26.73/6.12/14.72$	$82.40$	$3.38$	$3.46$	$3.69$	$3.33$	$3.31$	$3.82$
- Leaky	$27.70/6.66/15.05$	$82.55$	$3.54$	$3.59$	$3.77$	$3.42$	$3.37$	$3.86$

Appendix

Table 11: Main results of simulating human cognition across different LLMs. We evaluate each model under five situations: (1) Base Model represents the original output without any cognition history conditioning. (2) Other uses cognition histories randomly sampled from other authors, serving as an interference control group. (3) Self incorporates the target author’s cognition history from the train set, representing our standard cross-domain setting. (4) SelfAkin leverages cognition histories from the same author’s test set, examining in-domain behavioral leakage. (5) Leaky provides oracle access to the author’s authentic cognition trajectory, serving as an upper bound. We report ROUGE-1/2/L and BertScore for lexical coverage, LLM-as-a-Judge for semantic similarity, and five cognitive consistency dimensions: Foundational P remise, S trategic Preference, L ogic Mapping, K eyword Style, and T aboo Avoidance.

Method	Lexical		Semantic	Cognitive
Method	ROUGE-1/2/L	BertScore	Semantic	P.	S.	L.	K.	T.
GLM-4-9B	$23.06/3.56/12.41$	$80.82$	$3.19$	$2.92$	$3.03$	$2.73$	$3.04$	$3.19$
- Other	$21.94/3.34/12.14$	$80.98$	$2.98$	$2.72$	$2.89$	$2.67$	$2.93$	$3.15$
- Self	$22.58/3.59/12.30$	$81.12$	$3.14$	$2.92$	$3.07$	$2.81$	$3.02$	$3.40$
- SelfAkin	$23.68/4.54/13.01$	$81.47$	$3.37$	$3.18$	$3.38$	$3.07$	$3.14$	$3.63$
- Leaky	$25.31/5.27/13.83$	$81.73$	$3.61$	$3.39$	$3.49$	$3.23$	$3.19$	$3.86$

Appendix

Table 12: Main results of simulating human cognition across different LLMs.

Method	Lexical		Semantic	Cognitive
Method	ROUGE-1/2/L	BertScore	Semantic	P.	S.	L.	K.	T.
Grok-4	$20.48/2.83/\phantom{0}9.13$	$79.44$	$3.53$	$3.88$	$4.05$	$3.91$	$3.73$	$3.92$
- Other	$19.99/2.37/\phantom{0}9.00$	$80.97$	$3.16$	$3.53$	$3.77$	$3.58$	$3.45$	$3.68$
- Self	$20.77/2.81/\phantom{0}9.31$	$81.39$	$3.37$	$3.81$	$4.02$	$3.83$	$3.61$	$3.94$
- SelfAkin	$22.22/3.50/\phantom{0}9.94$	$81.62$	$3.65$	$4.27$	$4.41$	$4.19$	$3.87$	$4.23$
- Leaky	$23.15/4.11/10.34$	$81.96$	$3.84$	$4.42$	$4.54$	$4.31$	$3.98$	$4.35$

Appendix

Table 13: Main results of simulating human cognition across different LLMs.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Mar 10, 2026

Are they human? Detecting large language models by probing human memory constraints

The validity of online behavioral research relies on study participants being human rather than machine. In the past, it was possible to detect machines by posing simple challenges that were easily solved by humans but not by machines. General-purpose agents based on large language models (LLMs) can now solve many of these challenges, threatening the validity of online behavioral research. Here we explore the idea of detecting humanness by using tasks that machines can solve too well to be human. Specifically, we probe for the existence of an established human cognitive constraint: limited working memory capacity. We show that cognitive modeling on a standard serial recall task can be used to distinguish online participants from LLMs even when the latter are specifically instructed to mimic human working memory constraints. Our results demonstrate that it is viable to use well-established cognitive phenomena to distinguish LLMs from humans.

Simon Schug, Brenden M. Lake

Department of Computer Science, Princeton University · Department of Psychology, Princeton University

Working Memory in Language Models

Ask PDF

Sep 28, 2026

AwarenessBench: Assessing Cognitive Capabilities of Language Models

As language models (LMs) exhibit increasingly consciousness-like behaviors, evaluating their cognitive abilities becomes essential. We introduce AwarenessBench, the first comprehensive benchmark for assessing the cognitive abilities of LMs in four dimensions: metacognition, self-awareness, social awareness, and situational awareness, covering 15 cognitive functions and 14,381 samples. Evaluating 18 state-of-the-art LMs, we find that all consistently surpass random baselines, with more advanced models performing better. We further compare LMs with human performance across three demographic groups, where the best-performing model surpasses human averages overall, but most still fall markedly short in metacognition and self-awareness. Finally, we show that awareness is a distinct capability: progress in language modeling or reasoning does not necessarily translate into improved cognition.

Xiaojian Li, Rongwu Xu, Tianyun Zhang +9

Tsinghua University · Shanghai Qi Zhi Institute · Fangcun AI +5

LLM Evaluation Evaluation Awareness in Language Models Metacognition in Language Models

Ask PDF

Mar 8, 2026

Evaluating LLMs as Human Surrogates in Controlled Experiments

Large language models (LLMs) are increasingly used to simulate human responses in behavioral research, yet it remains unclear when LLM-generated data support the same experimental inferences as human data. We evaluate this by directly comparing off-the-shelf LLM-generated responses with human responses from a canonical survey experiment on accuracy perception. Each human observation is converted into a structured prompt, and models generate a single 0--10 outcome variable without task-specific training; identical statistical analyses are applied to human and synthetic responses. We find that LLMs reproduce several directional effects observed in humans, but effect magnitudes and moderation patterns vary across models. Off-the-shelf LLMs therefore capture aggregate belief-updating patterns under controlled conditions but do not consistently match human-scale effects, clarifying when LLM-generated data can function as behavioral surrogates.

Adnan Hoq, Tim Weninger

University of Notre Dame Notre Dame, Indiana, USA

Human Behavior Simulation Large Language Model-Based Social Simulation LLM Evaluation Belief Updating in LLMs

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Are they human? Detecting large language models by probing human memory constraints

AwarenessBench: Assessing Cognitive Capabilities of Language Models

Evaluating LLMs as Human Surrogates in Controlled Experiments