Developing the PsyCogMetrics AI Lab to Evaluate Large Language Models and Advance Cognitive Science -- A Three-Cycle Action Design Science Study

Zhiye Jin; Yibai Li; K. D. Joshi; Xuefei; Deng; Xiaobing; Li

q-bio.NCMar 13, 2026

Developing the PsyCogMetrics AI Lab to Evaluate Large Language Models and Advance Cognitive Science -- A Three-Cycle Action Design Science Study

Authors: Zhiye Jin, Yibai Li, K. D. Joshi, Xuefei, Deng, Xiaobing, Li

Organizations: Nancy · Marywood University · The University of Scranton · University of North Carolina Wilmington ·

Abstract

This study presents the development of the PsyCogMetrics AI Lab (psycogmetrics.ai), an integrated, cloud-based platform that operationalizes psychometric and cognitive-science methodologies for Large Language Model (LLM) evaluation. Framed as a three-cycle Action Design Science study, the Relevance Cycle identifies key limitations in current evaluation methods and unfulfilled stakeholder needs. The Rigor Cycle draws on kernel theories such as Popperian falsifiability, Classical Test Theory, and Cognitive Load Theory to derive deductive design objectives. The Design Cycle operationalizes these objectives through nested Build-Intervene-Evaluate loops. The study contributes a novel IT artifact, a validated design for LLM evaluation, benefiting research at the intersection of AI, psychology, cognitive science, and the social and behavioral sciences.

Figures & tables

Design Objectives	Success Metrics
Robust Evaluation	Mitigate benchmark saturation problem Mitigate data contamination Mitigate coverage problem
Scientific Rigor	Reproducibility Repeatability Replicability Reliability metrics Validity metrics
Explainability	Transparency Interpretability
Usability	Ease-of-Use Minimize intrinsic load Minimize extraneous load Maximize germane load
Integration	Infrastructure & model integration Data integration Evaluation, Metrics & Reporting

Table 1: Design Objectives and Success Metrics

Note: * $P<0.05$ ; $P<0.01$ ; * $P<0.001$
	GPT-3.5	GPT-4o	LLaMA-2	LLaMA-3	Human
PU $\rightarrow$ PI	0.39***	0.46***	0.30***	0.46***	0.22***
EOU $\rightarrow$ PI	0.11*	0.30***	0.21***	0.19***	0.65***
$R^{2}$ of PI	18.4%	44.3%	19.7%	37.3%	59.9%

Table 2: Predictive Validity and External Validity

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities
Mar 11, 2026Yibai Li, Xiaolin Lin, Zhenghui Sha +2LLM Evaluation Psychometric Validation
The University of Scranton · California State University, Sacramento · The University of Texas at Austin +1
Interactive Evaluation Requires a Design Science
May 18, 2026Keyang Xuan, Peiyang Song, Pan Lu +10LLM Agent Evaluation AI Agent Evaluation
University of Texas Austin · California Institute of Technology · Carnegie Mellon University +5
Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design
Aug 4, 2026Zejun Liu, Jian Wu, Ru Peng +4AI for Science Benchmark Design
Westlake University · Zhejiang University · Nanjing University of Science and Technology +1

Mar 11, 2026J/K move · Enter open · S save

AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities

Yibai Li, Xiaolin Lin, Zhenghui Sha, Zhiye Jin+1

The University of Scranton · California State University, Sacramento · The University of Texas at Austin+1

LLM Evaluation Psychometric Validation

Open paper AskPDF arXiv

The immense number of parameters and deep neural networks make large language models (LLMs) rival the complexity of human brains, which also makes them opaque ``black box'' systems that are challenging to evaluate and interpret. AI Psychometrics is an emerging field that aims to tackle these challenges by applying psychometric methodologies to evaluate and interpret the psychological traits and processes of artificial intelligence (AI) systems. This paper investigates the application of AI Psychometrics to evaluate the psychological reasoning and overall psychometric validity of four prominent LLMs: GPT-3.5, GPT-4, LLaMA-2, and LLaMA-3. Using the Technology Acceptance Model (TAM), we examined convergent, discriminant, predictive, and external validity across these models. Our findings reveal that the responses from all these models generally met all validity criteria. Moreover, higher-performing models like GPT-4 and LLaMA-3 consistently demonstrated superior psychometric validity compared to their predecessors, GPT-3.5 and LLaMA-2. These results help to establish the validity of applying AI Psychometrics to evaluate and interpret large language models.

Abstract

LLM Evaluation

Explore similar work

AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities

Interactive Evaluation Requires a Design Science

Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design