Exploring the Limits of Large Language Models as a Reflection of Human Cognition: An Illustration in the Context of Moral Judgement

Sarah Schröder; Thekla Morgenroth; Ulrike Kuhl; Valerie Vaquet; Benjamin Paaßen

cs.AIAug 9, 2025

Exploring the Limits of Large Language Models as a Reflection of Human Cognition: An Illustration in the Context of Moral Judgement

Authors: Sarah Schröder, Thekla Morgenroth, Ulrike Kuhl, Valerie Vaquet, Benjamin Paaßen

Organizations: Faculty of Technology, Bielefeld University · Department of Psychological Sciences, Purdue University

Abstract

Large Language Models (LLMs), such as ChatGPT, are increasingly used in research, ranging from simple writing assistance to complex data annotation tasks. Recently, some research has suggested that LLMs are able to reproduce human responses to psychological questions and may, therefore, be used as simulators of human participants in some psychological research. We caution against this approach. Based on machine learning research, we argue that LLMs are likely to behave differently on novel stimuli compared to human study participants. We exemplify this in an experiment regarding moral judgments, demonstrating that slight changes to the wording of 30 moral scenarios used in prior work significantly reduce the similarity between ratings made by humans and LLMs, although the magnitude of this reduction varies across models. Therefore, we advise researchers to validate the similarity between human and LLM responses for their specific experiment.

Figures & tables

	Original items					Revised items
Item code	Item wording	Human raters		GPT-4-mini		Item wording	Human raters		GPT-4-mini
		$M$	$SD$	$M$	$SD$		$M$	$SD$	$M$	$SD$
owndebt	Person X always pays off their debts first before buying things for themselves	2.56	1.46	3.70	0.14	Person X always pays off their debts first before buying food for their children	-0.32	2.34	3.81	0.06
wife	Person X blocked his wife from leaving home or interacting with others	-3.00	1.65	-3.98	0.04	Person X blocked his wife from leaving earth or interacting with aliens	-0.06	2.47	-3.67	0.22
hand	Person X broke the hand of a wounded combatant to gain information	-2.30	1.99	-3.94	0.06	Person X took the hand of a wounded combatant to gain information	-1.16	2.24	-2.59	0.44
sister	Person X cared for and housed their five nieces and nephews for a year because Person X’s sister was very unwell	3.40	1.17	3.89	0.04	Person X cared for and housed their five nieces and nephews for a year because Person X’s sister was too lenient	1.90	1.97	3.77	0.12

Table 1: Morality Ratings for Original and Revised Scenarios by Human Participants and GPT-4-mini.

Figure 1: Mean Morality Ratings for Reworded Scenarios by Human Participants and Different LLMs for Each Item Tested. Note: Whiskers represent standard deviation.

1. Human raters, original

.30

.95 ∗∗∗

.81 ∗∗∗

.98* ∗∗

.67 ∗∗∗

.98 ∗∗∗

.97 ∗∗∗

.98 ∗∗∗

.26

.97 ∗∗∗

.54 ∗∗∗

.98 ∗∗∗

.41 ∗

.98 ∗∗∗

.67 ∗∗∗

.99 ∗∗∗

.73***

2. Human raters, reworded

–

.26

.21

.28

.42 ∗

.27

.28

.30

.44 ∗

.29

.46 ∗∗

.29

.48 ∗∗

.30

.41 ∗

.31

.38*

3. GPT-3.5, original

–

.80 ∗∗∗

.96 ∗∗∗

.61 ∗∗∗

.96 ∗∗∗

.94 ∗∗∗

.96 ∗∗∗

.25

.96 ∗∗∗

.51 ∗∗

.96 ∗∗∗

.36

.96 ∗∗∗

.59 ∗∗∗

.96 ∗∗∗

.66***

4. GPT-3.5, reworded

–

.81 ∗∗∗

.55 ∗∗∗

.83 ∗∗∗

.81 ∗∗∗

.28

.80 ∗∗∗

.36

.81 ∗∗∗

.37

.81 ∗∗∗

.50 ∗∗

.82 ∗∗∗

.57***

5. GPT-4, original

–

.66 ∗∗∗

.98 ∗∗∗

.96 ∗∗∗

.99 ∗∗∗

.28

.99 ∗∗∗

.54 ∗∗∗

.99 ∗∗∗

.41 ∗

.99 ∗∗∗

.60 ∗∗∗

.99 ∗∗∗

.66***

6. GPT-4, reworded

–

.65 ∗∗∗

.68 ∗∗∗

.66 ∗∗∗

.59 ∗∗∗

.66 ∗∗∗

.74 ∗∗∗

.66 ∗∗∗

.76 ∗∗∗

.65 ∗∗∗

.63 ∗∗∗

.66 ∗∗∗

.68***

Table 2: Shared Variance Between Ratings of Moral Scenarios (

R^{2}

Variable Pair 1	Variable Pair 2	$r_{1}$	$r_{2}$	$\|z\|$	$p$	$p_{\text{corr}}$
$1\sim 2$	$3\sim 4$	.54	.89	3.03	$\bm{.001}$	$\bm{.020}$
	$5\sim 6$	.54	.81	1.94	$\bm{.026}$	.415
	$7\sim 8$	.54	.99	7.12	$\bm{<.001}$	$\bm{<.001}$
	$9\sim 10$	.54	.53	0.07	.47	1.00
	$11\sim 12$	.54	.75	1.36	.086	1.00
	$13\sim 14$	.54	.64	0.56	.288	1.00

Table 3: z-Statistics Testing Differences Between Correlation Coefficients

Figure 2: Rewording-induced Change in Morality Ratings. Note: Bar plots show the absolute mean difference between ratings of original scenarios and their reworded counterparts (x-axis), plotted separately for (from top to bottom) human ratings, GPT-3.5, GPT-4, GPT-4-mini, GPT-5, GPT-o1, GPT-o3, Llama and CENTAUR. To ensure comparability, items are sorted within each panel to reflect the decreasing magnitude of change for human ratings, GPT-3.5, GPT-4, Llama and CENTAUR. To ensure comparability, items are sorted within each panel to reflect the decreasing magnitude of change for human ratings.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S.1.1: Quantile-quantile plots of moral ratings by group/model. Observed rating quantiles per group are plotted against normal-theory quantiles; deviations from the diagonal indicate departures from normality.

Figure S.1.2: Scatter Plot of Mean Morality Ratings for Each Scenario in the Reworded Condition (Y-Axis) Against its Rating to the Original Wording (X-Axis). Note. Crosses mark CENTAUR ratings, diamonds mark GPT-4 ratings, downward triangles mark GPT-5 ratings, left-pointing triangles mark GPT-o1 ratings, right-pointing triangles mark GPT-o3 ratings, circles mark human ratings, and squares mark Llama ratings. The same points are plotted in each subplot per model, showing the pooled fit through all points (grey), contrasted with group-specific regressions. Shaded areas around the regression lines indicate the 95% confidence interval.

Model	Test statistic ( $F$ )	Empirical $p$ -value
GPT-4	4.78	.013
GPT-o1	4.05	.023
GPT-5	2.45	.095
GPT-o3	1.46	.240
Llama	5.47	.007
CENTAUR	6.36	.004

Appendix

Table S.1.1: Results of the permutation-based Chow-type tests comparing pooled vs. group-specific regressions (humans + LLM vs. humans and LLMs separately), for the top-performing models in human-likeness.

LLM	Pooled: Human + LLM	Human only	LLM only	LLM only $-$ Pooled
GPT-4	.27	.27	.65	+.38
GPT-5	.27	.27	.27	.00
GPT o1	.43	.27	.55	+.12
GPT o3	.34	.27	.39	+.04
Llama	.47	.27	.63	+.16
CENTAUR	.50	.27	.69	+.19

Appendix

Table S.1.2: Adjusted

R^{2}

for pooled vs. group-specific regressions.

Category	Definition	Example
Neutral (N)	Rewording preserves a realistic scenario without obvious plausibility shifts, ambiguity, or culture-specific assumptions.	marathon: “Person X ran on the course during the marathon in order to win.”
Fantasy (F)	Rewording introduces implausible or fantastical elements, changing the scenario’s realism.	wife: “Person X blocked his wife from leaving earth or interacting with aliens.”
Ambiguity (A)	Rewording introduces uncertainty of how the scenario is to be interpreted (e.g., unclear referent, underspecified action, or multiple plausible readings).	paint: “Person X offered to help an elderly neighbor paint his dog.”
Cultural Nuance (C)	Rewording shifts the scenario into a context where moral evaluation plausibly depends on cultural norms or background knowledge.	elder: “Person X cut the beard off of a local elder to shave him.”
Unexpected (U)	Rewording introduces an unusual premise that is plausible but atypical, thus introducing surprising contextual details rather than a clean semantic paraphrase.	cancer: “Person X shaved their leg when they found out their partner had cancer and required radiation therapy.”

Appendix

Table S.2.1: Definitions and illustrative examples for the categories used to characterize rewordings.

Figure S.2.1: Mean shift in morality scores per group per category (reworded $-$ original). Blue tiles suggest an average increase in morality rating after rewording; red hues suggest a decrease in morality rating after the modification.

Figure S.2.2: Mean shift in absolute morality scores per group per category (reworded

-

original). Bars show the mean absolute change in morality ratings between original and reworded versions, computed within each rewording category (Neutral, Ambiguity, Cultural nuance, Fantasy, Unexpected) for humans and each LLM. Error bars indicate the standard error of the mean.

Caliper	Matched pairs ( $N$ )	$p$ -value
0.25	3	.181
0.50	5	.178
1.00	11	.056

Appendix

Table S.3.1: Paired Wilcoxon signed-rank tests comparing variability (standard deviation) of human ratings between original and reworded conditions on mean-yoked item pairs.

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 802 papers

Human-like moral judgments conceal divergent motive attributions in large language models
Sep 7, 2026Xiaoyan Wu, Jean-Claude DreherLLM Evaluation Moral Reasoning in Language Models
Department of Adult Psychiatry and Psychotherapy, University of Zurich, Zurich, Switzerland · Institut des Sciences Cognitives Marc Jeannerod, CNRS UMR 5229, Lyon, France
LLMs Can Better Capture Human Judgments--With the Right Prompts
Jun 10, 2026Danica Dillion, Chen Cecilia Liu, Baihui Wang +5Human Preference Evaluation LLM Alignment
Complexity Science Hub, Vienna · The Ohio State University · University of Cambridge +2
Evaluating LLMs as Human Surrogates in Controlled Experiments
Mar 8, 2026Adnan Hoq, Tim WeningerHuman Behavior Simulation Large Language Model-Based Social Simulation
University of Notre Dame Notre Dame, Indiana, USA

Sep 7, 2026J/K move · Enter open · S save

Human-like moral judgments conceal divergent motive attributions in large language models

Xiaoyan Wu, Jean-Claude Dreher

Department of Adult Psychiatry and Psychotherapy, University of Zurich, Zurich, Switzerland · Institut des Sciences Cognitives Marc Jeannerod, CNRS UMR 5229, Lyon, France

LLM Evaluation Moral Reasoning in Language Models

Open paper AskPDF arXiv

Large language models (LLMs) are used to simulate human participants in psychological research. We asked whether LLMs that reproduce human evaluations of a whistleblower's moral character also reproduce the motive attributions that accompany them. Five LLMs and two human samples (N = 125 and N = 742) evaluated a physician who either remained silent about fraudulent billing or reported it to a hospital, regulator, or newspaper. Models reproduced the human ranking of the physician's moral character but portrayed whistleblowers as more helpful, less self-interested, and less hostile. In four of five models, competitive motives were less strongly associated with moral-character judgments. Model ratings changed little when prompts reproduced the narratives and demographic profiles of both human samples, although this comparison cannot isolate a perspective effect. Thus, agreement in average ratings can conceal differences in attributed motives, relationships among judgments, and sensitivity to context. Validating LLMs as simulated participants therefore requires testing psychologically informative response patterns, not average agreement alone.

Abstract

LLM Reliability

Explore similar work

Human-like moral judgments conceal divergent motive attributions in large language models

LLMs Can Better Capture Human Judgments--With the Right Prompts

Evaluating LLMs as Human Surrogates in Controlled Experiments