Item Difficulty Estimation

Follow this topic Ask about this topic

Latest papers

17 papers2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language
Oct 1, 2026Peng Cui, Qiaoyuan Zheng, Rudolf Debelak +1Item Difficulty Estimation
Department of Computer Science ETH Zurich · Machine Learning and Optimization Lab EPFL
PSM: Dataset Distillation Based on Precise Statistical Matching by Difficulty
Sep 28, 2026Hongxu Ma, Guang Li, Shijie Wang +6Distribution Matching Dataset Distillation
Zhejiang University · Hokkaido University · The University of Queensland +3
What Makes Software Issue Resolution Tasks Difficult for Agents?
Aug 18, 2026Ebtesam Al-Haque, Brittany JohnsonItem Difficulty Estimation Software Engineering
Department of Computer Science, George Mason University, Fairfax, VA
Predicting Task Difficulty Without Rollouts
Aug 6, 2026Stefan Krsteski, Charlotte MeyerLong-Horizon Agent Evaluation AI Agent Benchmarks
Andromede AI, EPFL
Representing Visual Evidence for Item Difficulty Prediction: Visual Textualization and Image-Native Modeling
Aug 5, 2026Han Chen, Ming Li, Hong Jiao +1Vision-Language Models Educational Assessment
Mohamed bin Zayed University of Artificial Intelligence · University of Maryland
Disentangling Curriculum Learning in NLP: Towards a Unifying Taxonomy
Jul 21, 2026Vanessa Toborek, Florian Seiffarth, Sebastian Müller +1Curriculum Learning Item Difficulty Estimation
University of Bonn, Germany · Lamarr Institute, Germany · Fraunhofer IAIS, Germany
Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction
Jun 26, 2026Chenguang Wang, Ming Li, Xinyue Zeng +4Cognitive Modeling Educational Assessment
1Virginia Tech · University of Maryland · 3MBZUAI +1
Hard or Just Unreached? Diagnosing the Sampling Blind Spot in Math-Reasoning Difficulty Estimation
Jun 17, 2026Luca Zhou, Sajel Shah, Emanuele Rodolà +1Mathematical Reasoning Benchmarks LLM Evaluation
1Sapienza University of Rome · 2Not Diamond · 3Paradigma
Structure-Aware Modeling of Multiple-Choice Questions Improves Automatic Difficulty Estimation
Jun 8, 2026Gabriel Ortega, Abelino Jiménez, Séverin Lions +1Educational Assessment Item Difficulty Estimation
Centro de Investigación Avanzada en Educación (CIAE) Instituto de Estudios Avanzados en Educación (IE) Universidad de Chile. · Departamento de Evaluación Medición y Registro Educacional (DEMRE) Universidad de Chile. · Centro de Modelamiento Matemático (CMM) Universidad de Chile. +1
Estimating Item Difficulty with Large Language Models as Experts
May 18, 2026Diana Kolesnikova, Kirill Fedyanin, Abe D. Hofman +2LLM Evaluation LLM-as-a-Judge
Department of Methodology and Statistics, Tilburg University, Tilburg, Netherlands · Smart Business Technologies, Belgrade, Serbia · Department of Psychological Methods, University of Amsterdam, Amsterdam, Netherlands +2
Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring
May 12, 2026Jamshid Mozafari, Bhawna Piryani, Adam JatowtLLM Evaluation Item Difficulty Estimation
University of Innsbruck, Innsbruck, Austria
MCQ Difficulty Prediction via Modeling Learner Heterogeneity Using Data-Driven Cognitive Profiling
Apr 13, 2026Dhriti Krishnan, Jaromir SavelkaItem Difficulty Estimation Educational Assessment
Carnegie Mellon University, Pittsburgh, PA, USA
Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation using Knowledge Graphs and Large Language Models
Apr 12, 2026Mehmet Can Şakiroğlu, H. Altay Güvenir, Kamer KayaMultiple-Choice Question Generation Automated Test Generation
Computer Engineering Department, Bilkent University, Ankara, Turkey · Faculty of Engineering and Natural Sciences, Sabanci University · VERIM, Center of Excellence in Data Analytics, Sabanci University
RoMathExam: A Longitudinal Dataset of Romanian Math Exams (1895-2025) with a Seven-Decade Core (1957-2025)
Mar 28, 2026Luca-Ncolae Cuclea, Sabin-Codrut Badea, Adrian-Marius DumitranItem Difficulty Estimation Educational Assessment
University of Bucharest, Faculty of Mathematics and Computer Science, Bucharest, Romania · Cu Drag si Sport SRL, Bucharest, Romania · Softbinator Technologies, Bucharest, Romania
4OPS: Structural Difficulty Modeling in Integer Arithmetic Puzzles
Mar 26, 2026Yunus E. ZeytuncuItem Difficulty Estimation Mathematical Reasoning
Level Up: Defining and Exploiting Transitional Problems for Curriculum Learning
Mar 14, 2026Amogh Inamdar, Zhenwei Tang, Ashton Anderson +1Item Difficulty Estimation Curriculum Learning
Department of Computer Science Columbia University · Department of Computer Science University of Toronto
Using Vision + Language Models to Predict Item Difficulty
Mar 4, 2026Samin KhanItem Difficulty Estimation Educational Assessment
Stanford Graduate School of Education

Oct 1, 2026J/K move · Enter open · S save

What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language

Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan

Department of Computer Science ETH Zurich · Machine Learning and Optimization Lab EPFL

Item Difficulty Estimation

Open paper AskPDF arXiv

Difficulty is one of the most fundamental properties of a question: it determines whether the question can meaningfully discriminate between models of differing ability. Although a variety of methods can now estimate or predict difficulty automatically, they yield only a single descriptive number, with no account of the underlying factors that make a question difficult in the first place. In this work, we propose a data-driven approach that automatically generates and validates natural-language hypotheses explaining what makes one question harder than another. We first estimate each item's difficulty from the responses of a large pool of LLMs using Item Response Theory. We then sample contrasting sets of easy and hard questions and prompt an LLM to propose candidate explanations of the difference, which are subsequently validated and selected on held-out questions. Experimental results across three datasets spanning mathematical, logical, and commonsense reasoning show that our method produces interpretable and predictive hypotheses. On their own, they predict the difficulty of unseen questions competitively with, or better than, advanced black-box difficulty regressors; used as additional features, they further improve those regressors, implying that they discover difficulty signals that existing models fail to capture. Moreover, we demonstrate that editing questions according to a hypothesis can shift their measured difficulty in the expected direction, indicating that the discovered hypotheses are causally valid difficulty factors rather than post-hoc descriptions. Our approach thus turns a purely descriptive difficulty score into actionable statements.