Item Response Theory

Follow this topic Ask about this topic

Item Response Theory (IRT) uses patterns of correct, incorrect, or graded responses to estimate unobserved traits such as learner skill or model capability, while accounting for item difficulty and discrimination. Researchers apply it in education and increasingly in AI evaluation, where adaptive item selection, multidimensional and multimodal models, and response-level analysis can reduce testing costs and reveal differences hidden by average scores. Neural and contextual extensions also aim to predict performance on unseen questions, analyze open-ended responses, and incorporate signals such as answer choices or reasoning trajectories. Key challenges include whether estimates and rankings remain reliable with few evaluated systems, and whether models generalize beyond the tasks and settings used for calibration.

Latest papers

33 papers3 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores
Jan 20, 2026Esma Balkır, Alice Pernthaller, Marco Basaldella +2LLM Evaluation Computerized Adaptive Testing
Trismik · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Universitat Politècnica de València +1
Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length
Dec 7, 2025Zhiyu Xu, Jia Liu, Yixin Wang +1LLM Evaluation Latent Variable Models
Department of Statistics, Columbia University. · Department of Statistics, University of Michigan
Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
Oct 26, 2025Peiyu Li, Xiuxiu Tang, Si Chen +4LLM Evaluation Psychometric Validation
University of Notre Dame, Notre Dame, USA

Jan 20, 2026J/K move · Enter open · S save

Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores

Esma Balkır, Alice Pernthaller, Marco Basaldella, José Hernández-Orallo+1

Trismik · Leverhulme Centre for the Future of Intelligence, University of Cambridge · Universitat Politècnica de València+1

LLM Evaluation Computerized Adaptive Testing Item Response Theory

Open paper AskPDF arXiv

Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation increasingly relies on generation tasks where outputs are scored continuously rather than marked correct/incorrect. We present a principled extension of IRT-based adaptive testing to continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge) by replacing the Bernoulli response distribution with a heteroskedastic normal distribution. Building on this, we introduce an uncertainty aware ranker with adaptive stopping criteria that achieves reliable model ranking while testing as few items and as cheaply as possible. We validate our method on five benchmarks spanning n-gram-based, embedding-based, and LLM-as-judge metrics. Our method improves ranking correlation by 0.13

τ

over random sampling and has 99% accuracy on confident predictions while using 2% of the items after a one-time calibration step.

Latest papers

Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores

Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks