cs.CLFeb 24, 2025

Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations

Authors: Dong-Ho Lee, Hyundong Cho, Jonathan May, Jay Pujara

Organizations: Information Sciences Institute, University of Southern California

Abstract

Asking questions is fundamental for deeper understanding, yet evaluating and generating questions that improve downstream outcomes remains a challenging problem. To address this gap, we introduce QUEST (Question Utility Estimation with Simulated Tests), a framework that offers a new outcome-driven paradigm for question evaluation and generation, instead of indirect metrics that are based on the underlying content of the question and its answer. QUEST uses language models to simulate learners that ask questions while studying a textbook chapter and quantifies the utility of a question as its direct contribution to understanding as measured by the learner's exam performance. To analyze the effectiveness of QUEST, we curate TEXTBOOK-EXAM, a benchmark that aligns textbook sections with exam questions across five academic disciplines. Using QUEST, we filter for high-utility questions and fine-tune question generators via rejection sampling and show that questions generated by QUEST-trained models improve simulated exam scores by 18.5% compared to the next strongest baseline. Furthermore, we demonstrate the practical real-world value of QUEST by showing that simulation-computed utility of a question is moderately positively correlated with its usefulness as perceived by human evaluators.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Ask-E: An Environment for Calibrated Question Generation

    Aug 7, 2026Sarah Pratt, Jae Sung Park, Scott Geng +1Mathematical Reasoning BenchmarksSynthetic Benchmark Generation

  2. KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration

    Feb 23, 2026Mohammad Amanlou, Erfan Shafiee Moghaddam, Yasaman Amou Jafari +3Multiple-Choice Question GenerationLLM Evaluation

  3. Humanity's Last Exam

    Jan 24, 2025Long Phan, Alice Gatti, Ziwen Han +1178LLM EvaluationLanguage Model Reasoning Evaluation