Psychometric Validation

Ask about this topic

Overview

Psychometric validation asks whether scores from tests, surveys, rubrics, or AI systems measure what they are meant to measure. Researchers use reliability checks, factor analysis, item response theory, and comparisons with independent outcomes to assess score consistency, construct validity, and fairness across groups or settings. In AI evaluation, these methods are used to audit benchmarks and automated judges, and to test whether language models can serve as synthetic respondents or psychological measurement tools. A central challenge is that plausible or human-like scores may reflect response bias, prompt wording, or benchmark design rather than the intended ability or trait.

Latest papers

CardsList

67 papers7 papers in the last four weeks, up 40% on the four weeks before. 0.0% of all new papers.

  1. semanticfa: An R Package for Response-Free Semantic Factor Analysis of Psychometric Scales

    Oct 27, 2026Devon Yanitski, Chris WestburyPsychometric ValidationLatent Variable Models

    Department of Psychology, University of Alberta, Edmonton, Alberta, Canada

  2. When Respondents Are Scarce: LLM-Derived Semantic Priors for Bayesian Confirmatory Factor Analysis

    Oct 26, 2026Shaohuai Feng, Mohd Wira Mohd Shafiei, Mohd Sukri Shafie +1Latent Variable ModelsPsychometric Validation

    Centre for Global Sustainability Studies (CGSS), Universiti Sains Malaysia, Minden, Penang, Malaysia. · Centre for Education and Training in Renewable Energy, Energy Efficiency and Green Technology (CETREE), Universiti Sains Malaysia, Minden, Penang, Malaysia. · Centre for Instructional Technology and Multimedia, Universiti Sains Malaysia, Minden, Penang, Malaysia.

  3. Is the Platform Part of the Measurement? A Protocol and Simulation Study for Cross-Platform Equivalence of Machine-Labelled Policy Sentiment

    Oct 12, 2026Asad Ikram, Ella HaigSocial Media AnalysisPsychometric Validation

    Independent researcher. · School of Computing, Mathematics & Physics, University of Portsmouth, UK

  4. Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

    Oct 10, 2026Mantas Lukauskas, Viktorija ŠarkauskaitėSynthetic Data GenerationSynthetic Data Evaluation

    Hostinger; Kaunas University of Technology; AI Insight Lab · AI Insight Lab; Independent Researcher

  5. Fear Is Not Always Fear: Variation in Emotion Label Equivalence Across Datasets

    Oct 9, 2026Emily Öhman, Anna KoufakouEmotion RecognitionPsychometric Validation

    Waseda University, Japan · Florida Gulf Coast University, USA

  6. COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal

    Oct 5, 2026Baihan LinRepresentational Similarity AnalysisPsychometric Validation

    Department of Artificial Intelligence and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA +2

  7. AI-Enabled Quality Assurance for Multiple-Choice Assessment Items

    Oct 3, 2026Steven Moore, Nicholas DianaAutomated EvaluationEducational Assessment

    George Mason University · Colgate University

  8. Benchmarking Psychological Dynamics in Generative Agents

    Oct 3, 2026Sumer S. Vaid, Ashley V. WhillansHuman Behavior SimulationLarge Language Model-Based Social Simulation

    AI Institute Harvard Business School · Negotiation, Organizations & Markets Unit Harvard Business School

  9. Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty

    Oct 2, 2026Longwei Cong, Sonja Hahn, Sebastian Gombert +4LLM-as-a-JudgeAutomated Evaluation

    DIPF | Leibniz Institute for Research and Information in Education · Centre for International Student Assessment (ZIB) · Faculty of Computer Science, Goethe University Frankfurt +1

  10. Toward Embedding-Based Psychometrics: Structural Modeling of Assessment-Item Semantics With Contextual Scores

    Sep 25, 2026Jinsong Chen, Shi-Ting ChenEducational AssessmentPsychometric Validation

    Faculty of Education, The University of Hong Kong

  11. Artificial Societies Benchmark: A Validation Framework for Synthetic Research

    Sep 24, 2026Edoardo Chidichimo, Min Jun Jung, Felix P. S. Wallis +1Benchmark DesignComputational Social Science

    Artificial Societies University of Oxford, UK

  12. Measuring AI Leadership: Development and Validation of a Multidimensional Measure for AI-Native Organizations

    Sep 16, 2026Mustafa Akben, Leslie CoynePsychometric Validation

    Department of Management and Entrepreneurship Love School of Business, Elon University · Signalead LLC

  13. Bridging Network Psychometrics and Artificial Intelligence: An Ising-Potts Model with LLM-Derived Weights

    Sep 8, 2026Matthias von DavierInter-Rater ReliabilityPsychometric Validation

  14. Three Ways Classical Test Theory Can Mislead About LLM Judges

    Aug 31, 2026Louis Yiven ZhuLLM-as-a-JudgePsychometric Validation

    University of Oxford

  15. One Capability or Many? Structural and Predictive Tests of Benchmark Validity Disagree About Economic Benchmarks for Frontier AI

    Aug 29, 2026Louis Yiven ZhuBenchmark ValidityBenchmark Design

    University of Oxford

  16. A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

    Aug 25, 2026Jianlin Chen, Wenhui Chen, Ziyao Lin +1LLM-as-a-JudgePsychometric Validation

    South China University of Technology · University of Macau

  17. What Reaches Expert Review? Representation, Structural Screening, and Candidate-Form Dependence in AI-Assisted Item Development

    Aug 24, 2026Christopher BrooksAutomated EvaluationPsychometric Validation

    School of Information, University of Michigan

  18. When Persona Simulations Are Informative: Graph-Structured Signals for Pluralistic Opinion Sensing

    Aug 23, 2026Taehyeon An, Jaehyeong Park, Donghyuk ShinLarge Language Model-Based Social SimulationPersona-Conditioned Generation

    Korea Advanced Institute of Science and Technology Seoul, Republic of Korea

  19. Do Assessment Instruments Measure the Same Thing for Humans and LLMs? A Latent Structure Analysis

    Aug 16, 2026Alona Strugatski, Licol Zeinfeld, Giora AlexandronLLM EvaluationPsychometric Validation

    Weizmann Institute of Science

  20. Automated Generation of Complexity-Validated Decision Scenarios Using Large Language Models

    Aug 9, 2026Abdalla Doleh, Toni Somers, Ratna Babu ChinnamSynthetic Data GenerationPsychometric Validation

    Department of Industrial & Systems Engineering, Wayne State University, Detroit, MI, 48202, USA · Mike Ilitch School of Business, Wayne State University, Detroit, MI, 48202, USA

  21. Natural Language Processing Psychometrics

    Aug 7, 2026Edoardo Sebastiano De Duro, Emma Franchino, Massimo StellaClinical Outcome PredictionPsychometric Validation

    CogNoscoLab, Department of Psychology and Cognitive Science, University of Trento, Italy

  22. Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

    Aug 3, 2026Xiao Fei, Yang Zhang, Sarah Almeida Carneiro +1LLM EvaluationPsychometric Validation

    École Polytechnique, Institut Polytechnique de Paris, France · Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates

  23. Comparative Validation of GPT-4o-mini and Teacher Mean Scores for Automated Scoring of Music Analysis Responses: Single-Pass Deployment, Repeatability, and Strategy-Specific Bias

    Aug 3, 2026Baicheng Lin, Lingxi Jin, Kyung-Seok MinLLM-as-a-JudgeEducational Technology

    Department of Education, Sejong University, South Korea · Department of Educational Technology, Ewha Womans University, South Korea

  24. Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

    Aug 1, 2026William CabanInter-Rater ReliabilityAI Agent Evaluation

    Red Hat, Inc., Raleigh, North Carolina, USA · Alma Mater Europaea University, Vienna, Austria

  25. Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset

    Jul 29, 2026Mayank Sharma, Savira Nadela, Tyler MattesonLLM EvaluationPsychometric Validation

    Stanford University

  26. Developing and Validating the Spanish Version of the Large Language Models Dependency Scale (LLM-D12-SP)

    Jul 24, 2026Tran Gia Bao, Mo El-Haj, Sameha Al-Shakhsi +3Human-AI InteractionPsychometric Validation

    VinUniversity · Lancaster University · Hamad Bin Khalifa University +2

  27. Measuring How Students Rely on Generative AI in Academic Writing: Development and Multi-Source Validation of the Generative AI Reliance Types Scale (GenAI-RTS)

    Jul 15, 2026Shahin Hossain, Tukhbita Afroz NawmiGenerative AI in EducationPsychometric Validation

    School of Education University of Maryland Baltimore County Baltimore, Maryland, USA · Department of Counseling, School and Educational Psychology, Graduate School of Education University at Buffalo, Buffalo, New York, USA

  28. Bringing Back Rule Induction to Fluid Intelligence Research? An Initial Validation of the ARC-AGI Benchmark in Humans

    Jul 13, 2026Jasmin Thelen, Oliver WilhelmPsychometric Validation

    Institute of Psychology and Education, Ulm University, Germany

  29. Evaluating Nonuniform Dependability Across Response Conditions: A Conditional Generalizability Framework Illustrated in Automated Essay Scoring

    Jul 13, 2026Yi GuiAutomated Essay ScoringPsychometric Validation

    Measurement Incorporated

  30. When the Target Domain Changes: AI-Mediated Construct Drift in High-Stakes English Language AssessmenW

    Jul 13, 2026Yi GuiEducational AssessmentGenerative AI in Education

    Measurement Incorporated