Item Response Theory

Latest papers 28

2 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Rubric Rewards from Item Response Theory
Sep 28, 2026Milad Yazdani, Yaser Souri, Xiren Zhou +4Reward Modeling Rubric-Based RL
Department of Electrical and Computer Engineering, University of British Columbia · Microsoft · School of Biomedical Engineering, University of British Columbia
Toward Embedding-Based Psychometrics: Structural Modeling of Assessment-Item Semantics With Contextual Scores
Sep 25, 2026Jinsong Chen, Shi-Ting ChenEducational Assessment Psychometric Validation
Faculty of Education, The University of Hong Kong
Dynamical Non-compensatory Multidimensional IRT Model Using Variational Approximation
Sep 9, 2026Hiroshi Tamano, Daichi MochihashiLatent Dynamics Modeling Knowledge Tracing
the graduate university for advanced studies, sokendai · the institute of statistical mathematics
Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation
Sep 1, 2026Kefeng Duan, Dewu Zheng, Yanlin Wang +7Software Engineering Benchmarks LLM Agent Evaluation
School of Software Engineering, Sun Yat-sen University, Zhuhai, China · Huawei Cloud Computing Technologies Co., Ltd., China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
LLM Evaluation on Unseen Questions: Contextual Multidimensional IRT Model
Aug 23, 2026Ergan Shang, Weijing Tang, Yinqiu HeLLM Evaluation Language Model Generation Evaluation
Department of Statistics and Data Science, Carnegie Mellon University · Department of Statistics, University of Madison
Multimodal Item Parameter Estimation using Simulated Response Probabilitie
Aug 10, 2026Christopher Ormerod, YoungKoung KimEducational Assessment Multimodal Large Language Models
College Board
Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks
Aug 3, 2026Xiao Fei, Yang Zhang, Sarah Almeida Carneiro +1LLM Evaluation Psychometric Validation
École Polytechnique, Institut Polytechnique de Paris, France · Mohamed bin Zayed University of Artificial Intelligence, United Arab Emirates
Laplace-PSN-IRT: Uncertainty Quantification for Neural Item Response Theory Models of LLM Benchmarks
Jul 28, 2026Juan Francisco, Mandujano ReyesLLM Evaluation Language Model Calibration
Independent Research
Can We Trust Item Response Theory for AI Evaluation?
Jul 16, 2026Han Jiang, Sunbeom Kwon, Jinwen Luo +2LLM Evaluation Item Response Theory
1Johns Hopkins University · University of Illinois Urbana-Champaign · University of California, Los Angeles
From Text to Parameters: Predicting Item Parameters from Embedding Regularization with Reliability and Design Ceilings
Jul 8, 2026Shi-Ting Chen, Jinsong ChenText Embedding Evaluation Benchmark Validity
Faculty of Education, The University of Hong Kong
DualEval: Joint Model-Item Calibration for Unified LLM Evaluation
Jun 24, 2026Aaron J. Li, Hao Huang, Youngmin Park +6LLM Evaluation Language Model Generation Evaluation
University of California, Berkeley · Arena · University of California, Los Angeles
PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded Feedback
Jun 18, 2026Wei Xia, Jin Wu, Haoran Shi +2Educational Assessment Automated Essay Scoring
Department of Educational Psychology, East China Normal University · 2Shanghai Institute of Artificial Intelligence for Education, East China Normal University · School of Computer Science and Technology, East China Normal University
Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation
Jun 14, 2026Gili Lior, Tzviel Frostig, Gabriel Stanovsky +1Multilingual Language Model Evaluation Cross-Cultural Language Model Evaluation
Google Research · The Hebrew University of Jerusalem · PhaseV Trials
Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
May 29, 2026Sang Truong, Yuheng Tu, Rylan Schaeffer +1Language Model Scaling Laws Test-Time Scaling
1Stanford University · University of California, Los Angeles
Efficient Safety Benchmarking via Item Response Theory
May 26, 2026Fabio Spagliardi, Mírian Silva, Ayan Datta +3LLM Safety Benchmarks Language Model Safety Evaluation
Supervised Program for Alignment Research (SPAR) · Yale University · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
Response-free item difficulty modelling for multiple-choice items with fine-tuned transformers: Component-wise representation and multi-task learning
May 16, 2026Jan Netík, Patrícia MartinkováTransformer Educational Assessment
Faculty of Education, Charles University, Prague, Czech Republic · Institute of Computer Science of the Czech Academy of Sciences, Prague, Czech Republic
The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains
May 11, 2026Jung Min KangBenchmark Design Benchmark Validity
Independent Researcher · Seoul, South Korea
An Interpretable and Scalable Framework for Evaluating Large Language Models
May 7, 2026Xinhao Qu, Qiang Heng, Hao Zeng +1LLM Evaluation Item Response Theory
Department of Statistics, University of California, Riverside · School of Statistics and Data Science, Southeast University · Department of Statistics and Data Science, Southern University of Science and Technology
Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
Apr 30, 2026Longwei Cong, Sonja Hahn, Sebastian Gombert +3LLM Evaluation Automated Grading
DIPF | Leibniz Institute for Research and Information in Education · Faculty of Computer Science, Goethe University Frankfurt · 3Chemnitz University of Technology
MCQ Difficulty Prediction via Modeling Learner Heterogeneity Using Data-Driven Cognitive Profiling
Apr 13, 2026Dhriti Krishnan, Jaromir SavelkaEducational Assessment Educational Data Mining
Carnegie Mellon University, Pittsburgh, PA, USA
Cost-Efficient Estimation of General Abilities Across Benchmarks
Apr 1, 2026Michael Krumdick, Adam Wiemerslage, Seth Ebner +2LLM Evaluation Item Response Theory
Kensho Technologies · MIT
Agent psychometrics: Task-level performance prediction in agentic coding benchmarks
Apr 1, 2026Chris Ge, Daria Kryvosheieva, Daniel Fried +2LLM Agent Evaluation AI Agent Evaluation
MIT · Fulcrum · Carnegie Mellon University
Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory
Mar 3, 2026Shunki Uebayashi, Kento Masui, Kyohei Atarashi +5Multimodal Model Evaluation Multimodal Reasoning
Kyoto University · CyberAgent. · The Institute of Statistical Mathematics +1
CodeGENCAT: Generative Computerized Adaptive Testing for Open-ended Coding Problems
Feb 23, 2026Wanyong Feng, Alexander Scarlatos, Ruochen Sun +1Educational Assessment Item Response Theory
University of Massachusetts Amherst · Independent Researcher
Capabilities Ain't All You Need: Measuring Propensities in AI
Feb 20, 2026Daniel Romero-Alvarado, Fernando Martínez-Plumed, Lorenzo Pacchiardi +11LLM Evaluation Item Response Theory
Valencian Research Institute of Artificial Intelligence, Universitat Polit`ecnica de Val`encia, Valencia, Spain · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Existential Risk Observatory, Amsterdam, Netherlands +5
Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores
Jan 20, 2026Esma Balkır, Alice Pernthaller, Marco Basaldella +2LLM Evaluation Item Response Theory
Trismik · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Universitat Politècnica de València +1
Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length
Dec 7, 2025Zhiyu Xu, Jia Liu, Yixin Wang +1LLM Evaluation Latent Variable Models
Department of Statistics, Columbia University. · Department of Statistics, University of Michigan
Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
Oct 26, 2025Peiyu Li, Xiuxiu Tang, Si Chen +4LLM Evaluation Psychometric Validation
University of Notre Dame, Notre Dame, USA

Sep 28, 2026J/K move · Enter open · S save

Rubric Rewards from Item Response Theory

Milad Yazdani, Yaser Souri, Xiren Zhou, Pranit Chawla+3

Department of Electrical and Computer Engineering, University of British Columbia · Microsoft · School of Biomedical Engineering, University of British Columbia

Reward Modeling Rubric-Based RL Reward Design for RL Item Response Theory

Open paper AskPDF arXiv

Many language tasks have no single answer that can be checked automatically. Rubrics provide criteria for judging responses to these tasks. For reinforcement learning, the resulting verdicts must be combined into a scalar reward. A common approach sums the points assigned to satisfied criteria. Distinct verdict patterns can thus receive the same reward, and the fixed points encode how much each criterion should count, not how strongly its verdict distinguishes the current rollouts. Beyond this aggregation problem, judging the full rubric needs more judge requests as the criterion count grows. To address these limitations, Rubric Response Theory (RRT) measures quality and selects criteria when rubric criteria are monotone indicators of a shared target. Rather than adding assigned points, RRT uses a two parameter item response model that treats the verdict pattern as evidence about scalar quality specific to the rubric. Under this model, its likelihood score maximizes the local signal-to-noise ratio for quality. Its Response Parameter Network (RPN) reads the prompt and criterion text to predict criterion difficulty and discrimination. As the policy distribution changes during training, RRT uses online expectation maximization to update the RPN from current rollout verdicts. With Qwen3.5-4B as the policy, RRT's macro criterion score across Medical, Science, Rubrics as Rewards Science, and RubricBench is 1.7 points above that of group relative policy optimization (GRPO). On hard and very hard criteria in Medical and Science, RRT gains 2.8 to 5.6 points over GRPO. At half the criterion budget, adaptive Fisher selection with a frozen RPN keeps the macro criterion score across four datasets within 0.1 points of GRPO with full judging. These results show RRT can reduce judge requests while remaining competitive with GRPO.