AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities

Yibai Li; Xiaolin Lin; Zhenghui Sha; Zhiye Jin; Xiaobing Li

cs.AIMar 11, 2026

AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities

Authors: Yibai Li, Xiaolin Lin, Zhenghui Sha, Zhiye Jin, Xiaobing Li

Organizations: The University of Scranton · California State University, Sacramento · The University of Texas at Austin · Marywood University

Abstract

The immense number of parameters and deep neural networks make large language models (LLMs) rival the complexity of human brains, which also makes them opaque ``black box'' systems that are challenging to evaluate and interpret. AI Psychometrics is an emerging field that aims to tackle these challenges by applying psychometric methodologies to evaluate and interpret the psychological traits and processes of artificial intelligence (AI) systems. This paper investigates the application of AI Psychometrics to evaluate the psychological reasoning and overall psychometric validity of four prominent LLMs: GPT-3.5, GPT-4, LLaMA-2, and LLaMA-3. Using the Technology Acceptance Model (TAM), we examined convergent, discriminant, predictive, and external validity across these models. Our findings reveal that the responses from all these models generally met all validity criteria. Moreover, higher-performing models like GPT-4 and LLaMA-3 consistently demonstrated superior psychometric validity compared to their predecessors, GPT-3.5 and LLaMA-2. These results help to establish the validity of applying AI Psychometrics to evaluate and interpret large language models.

Figures & tables

Item	Description
Perceived Usefulness (PU)
PU1	Product recommendations make my shopping experience more interesting.
PU2	Product recommendations make my shopping experience more enjoyable.
PU3	Product recommendations help me make more informed purchase decisions.
PU4	Product recommendations help me make more accurate purchase decisions.
PU5	Product recommendations meet my needs adequately in my purchase decisions.

Table 1: Measurement Items

	GPT-3.5	GPT-4o	LLaMA-2	LLaMA-3	Human
PU1	0.90	0.93	0.70	0.86	0.90
PU2	0.89	0.92	0.72	0.89	0.88
PU3	0.91	0.92	0.72	0.84	0.89
PU4	0.89	0.92	0.73	0.86	0.93
PU5	0.92	0.94	0.75	0.82	0.90
EOU1	0.85	0.90	0.76	0.83	0.84

Table 2: Factor Loadings

	GPT-3.5	GPT-4o	LLaMA-2	LLaMA-3	Human
PU	0.94	0.96	0.77	0.91	0.83
EOU	0.90	0.92	0.68	0.86	0.86
PI	0.85	0.91	0.41	0.79	0.83

Table 3: Cronbach’s Alpha

	GPT-3.5	GPT-4o	LLaMA-2	LLaMA-3	Human
PU	0.96	0.97	0.84	0.93	0.95
EOU	0.93	0.94	0.80	0.90	0.90
PI	0.90	0.93	0.90	0.87	0.89

Table 4: Composite Reliability

	GPT-3.5	GPT-4o	LLaMA-2	LLaMA-3	Human
PU	0.81	0.86	0.52	0.73	0.81
EOU	0.77	0.80	0.51	0.70	0.70
PI	0.70	0.78	0.36	0.62	0.67

Table 5: Average Variance Extracted

	PU	EOU	PI
PU	0.88 / 0.90
EOU	0.23/0.54	0.83 / 0.88
PI	0.33/0.53	0.42/0.62	0.90 / 0.93

Table 6: Construct Correlations and Square Roots of AVEs from GPT-3.5 and GPT-4o

	PU	EOU	PI
PU	0.70 / 0.84
EOU	0.36/0.51	0.60 / 0.79
PI	0.50/0.67	0.40/0.59	0.72 / 0.85

Table 7: Construct Correlations and Square Roots of AVEs from LLaMA-2 and LLaMA-3

	PU	EOU	PI
PU	0.84
EOU	0.75	0.82
PI	0.48	0.53	0.90

Table 8: Construct Correlations and Square Roots of AVEs from Human Participants

	GPT-3.5	GPT-4o	LLaMA-2	LLaMA-3	Human
PU $\rightarrow$ PI	$0.39^{***}$	$0.46^{***}$	$0.30^{***}$	$0.46^{***}$	$0.22^{***}$
EOU $\rightarrow$ PI	$0.11^{*}$	$0.30^{***}$	$0.21^{***}$	$0.19^{***}$	$0.65^{***}$
$R^{2}$ of PI	18.4%	44.3%	19.7%	37.3%	59.9%

Table 9: Path Coefficients and the

R^{2}

of Purchase Intention

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact
Jun 18, 2026Jelena Meyer, David Garcia, Dirk U. WulffLLM Evaluation Personality Modeling in Language Models
Max Planck Institute for Human Development. · University of Konstanz. · Barcelona Supercomputing Center. +1
Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling
Sep 19, 2026Yu Sha, Junqi Tao, Dixin Zhou +10Personality Modeling in Language Models Multilingual Language Model Evaluation
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong, 518172, China. · Key Laboratory of Quark & Lepton Physics and Institute of Particle Physics, Central China Normal University, Wuhan, 430079, China. · Shandong Science and Technology Press, Shandong Publishing Group, Jinan, 250003, China. +6
Benchmarking Psychological Dynamics in Generative Agents
Oct 3, 2026Sumer S. Vaid, Ashley V. WhillansHuman Behavior Simulation Large Language Model-Based Social Simulation
AI Institute Harvard Business School · Negotiation, Organizations & Markets Unit Harvard Business School

Jun 18, 2026J/K move · Enter open · S save

Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact

Jelena Meyer, David Garcia, Dirk U. Wulff

Max Planck Institute for Human Development. · University of Konstanz. · Barcelona Supercomputing Center.+1

LLM Evaluation Personality Modeling in Language Models Psychometric Validation

Open paper AskPDF arXiv

Psychological instruments designed for humans are increasingly used to assign large language models (LLMs) stable psychological profiles that affect their usability, safety assessment, and use as proxies for human participants in research. Using a formal psychometric framework, we show that these profiles are largely a measurement artifact. Administering a battery of personality and risk-preference instruments spanning self-reports and behavioral tasks to 56 instruction-tuned LLMs alongside large human reference samples, we report four findings. First, differences between models are driven not by the traits an instrument targets but by a directional response bias, a tendency to respond toward one end of the scale, or one labeled option, regardless of item content; a variance decomposition attributes 81-90% of between-model variation to this bias, against 9-16% in humans. Second, the bias declines with model capability but is not eliminated by it. Third, because bias rather than trait drives responding, an instrument's apparent reliability is almost entirely predicted by its response orthogonality, a term we coin for the proportion of items for which trait and bias point in opposite directions. Fourth, the profile a model appears to have shifts with the items used and can be manufactured through item selection. These results demonstrate that the apparent psychological profiles of LLMs are artifacts of the instrument used to measure them, not properties of the models themselves. As instruments borrowed from human psychology are rarely fully orthogonal and may inherently lack validity for LLMs, we call for dedicated assessments centered on response orthogonality.

Abstract

LLM Evaluation

Explore similar work

Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact

Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling

Benchmarking Psychological Dynamics in Generative Agents