Item Response Theory

Item Response Theory (IRT) uses patterns of correct, incorrect, or graded responses to estimate unobserved traits such as learner skill or model capability, while accounting for item difficulty and discrimination. Researchers apply it in education and increasingly in AI evaluation, where adaptive item selection, multidimensional and multimodal models, and response-level analysis can reduce testing costs and reveal differences hidden by average scores. Neural and contextual extensions also aim to predict performance on unseen questions, analyze open-ended responses, and incorporate signals such as answer choices or reasoning trajectories. Key challenges include whether estimates and rankings remain reliable with few evaluated systems, and whether models generalize beyond the tasks and settings used for calibration.