Psychometric Validation

Follow this topic Ask about this topic

Psychometric validation asks whether scores from tests, surveys, rubrics, or AI systems measure what they are meant to measure. Researchers use reliability checks, factor analysis, item response theory, and comparisons with independent outcomes to assess score consistency, construct validity, and fairness across groups or settings. In AI evaluation, these methods are used to audit benchmarks and automated judges, and to test whether language models can serve as synthetic respondents or psychological measurement tools. A central challenge is that plausible or human-like scores may reflect response bias, prompt wording, or benchmark design rather than the intended ability or trait.

Latest papers

67 papers7 papers in the last four weeks, up 40% on the four weeks before. 0.0% of all new papers.

Falsifying Discriminant Validity of Predictive Algorithms
Jan 23, 2026Amanda CostonAlgorithmic Auditing Psychometric Validation
Statistics Department University of California, Berkeley Berkeley, CA, USA
Psychometric Comparability of LLM-Based Digital Twins
Dec 22, 2025Yufei Zhang, Zhihao MaLLM Evaluation Personality Modeling in Language Models
Computational Communication Collaboratory, School of Journalism and Communication, Nanjing University, Nanjing 210023, China
Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
Oct 26, 2025Peiyu Li, Xiuxiu Tang, Si Chen +4LLM Evaluation Psychometric Validation
University of Notre Dame, Notre Dame, USA
Human Psychometric Questionnaires Mischaracterize LLM Behavior
Sep 12, 2025Woojung Song, Dongmin Choi, Yoonah Park +3Personality Modeling in Language Models Psychometric Validation
1Graduate School of Data Science, Seoul National University · Department of Communication, Interdisciplinary Program in Artificial Intelligence, Seoul National University
"Mirror" Large Language Model Evaluations of Depression are Criterion Contaminated
Aug 7, 2025Tong Li, Rasiq Hussain, Mehak Gupta +1Depression Detection Mental Health
Division of Computational & Data Sciences (DCDS), Washington University in St. Louis, St. Louis, MO · Department of Computer Science, SMU · Department of Psychological & Brain Sciences, Washington University in St. Louis
Automatic Item Generation for Personality Situational Judgment Tests with Large Language Models
Dec 10, 2024Chang-Jin Li, Jiyuan Zhang, Yun Tang +1Psychometric Validation Automated Test Generation
Faculty of Psychology, Beijing Normal University, Beijing, 100875, China · School of Psychology, Central China Normal University, Wuhan, Hubei, 430079, China · Key Laboratory of Adolescent Cyberpsychology and Behavior, Ministry of Education, Wuhan, Hubei, 430079, China +1
Interpretable factorization of clinical questionnaires to identify latent factors of psychopathology
Dec 12, 2023Ka Chun Lam, Francisco Pereira, Bridget W Mahony +1Latent Variable Models Non-Negative Matrix Factorization
Machine Learning Core National Institute of Mental Health, National Institutes of Health · Section on Developmental Neurogenomics, Human Genetics Branch National Institute of Mental Health, National Institutes of Health

Jan 23, 2026J/K move · Enter open · S save

Falsifying Discriminant Validity of Predictive Algorithms

Amanda Coston

Statistics Department University of California, Berkeley Berkeley, CA, USA

Algorithmic Auditing Psychometric Validation Statistical Hypothesis Testing

Open paper AskPDF arXiv

Empirical investigations into unintended model behavior often show that the algorithm is predicting another outcome than what was intended. These exposés highlight the need to identify when algorithms predict unintended quantities - ideally before deploying them into consequential settings. We propose a falsification framework that provides a principled statistical test for discriminant validity: the requirement that an algorithm predict intended outcomes better than impermissible ones. Drawing on falsification practices from causal inference, econometrics, and psychometrics, our framework compares calibrated prediction losses across outcomes to assess whether the algorithm exhibits discriminant validity with respect to a specified impermissible proxy. In settings where the target outcome is difficult to observe, multiple permissible proxy outcomes may be available; our framework accommodates both this setting and the case with a single permissible proxy. Throughout we use nonparametric hypothesis testing methods that make minimal assumptions on the data-generating process. We illustrate the method in an admissions setting, where the framework establishes discriminant validity with respect to gender but fails to establish discriminant validity with respect to race. This demonstrates how falsification can serve as an early validity check. We also provide analysis in a criminal justice setting, where we highlight the limitations of our framework and emphasize the need for complementary approaches to assess other aspects of construct validity and external validity.