Which Questions Improve Learning the Most? Utility Estimation of Questions with LM-based Simulations
Organizations: Information Sciences Institute, University of Southern California
Abstract
Asking questions is fundamental for deeper understanding, yet evaluating and generating questions that improve downstream outcomes remains a challenging problem. To address this gap, we introduce QUEST (Question Utility Estimation with Simulated Tests), a framework that offers a new outcome-driven paradigm for question evaluation and generation, instead of indirect metrics that are based on the underlying content of the question and its answer. QUEST uses language models to simulate learners that ask questions while studying a textbook chapter and quantifies the utility of a question as its direct contribution to understanding as measured by the learner's exam performance. To analyze the effectiveness of QUEST, we curate TEXTBOOK-EXAM, a benchmark that aligns textbook sections with exam questions across five academic disciplines. Using QUEST, we filter for high-utility questions and fine-tune question generators via rejection sampling and show that questions generated by QUEST-trained models improve simulated exam scores by 18.5% compared to the next strongest baseline. Furthermore, we demonstrate the practical real-world value of QUEST by showing that simulation-computed utility of a question is moderately positively correlated with its usefulness as perceived by human evaluators.
Figures & tables
| Subject | # | Split | # / | % w/ A | # / |
|---|---|---|---|---|---|
| Microbiology | 20 | Train | 12.4 | 64% | 16.4 |
| 5 | Test | 13.4 | 58% | 17.0 | |
| Chemistry | 20 | Train | 14.2 | 51% | 11.0 |
| 5 | Test | 16.2 | 49% | 6.4 | |
| Economics | 20 | Train | 12.2 | 23% | 14.1 |
| 5 | Test | 12.2 | 23% | 14.4 |
| Microbiology | Chemistry | Economics | Sociology | US History | Average | |
| No-study | 45.8 1.0 | 9.2 0.9 | 0.3 1.1 | 61.2 0.8 | 3.2 1.2 | 23.9 0.7 |
| Zero-shot | 62.3 0.4 (+16.5) | 39.7 0.3 (+30.5) | 40.3 0.5 (+40.0) | 60.7 0.2 (-0.5) | 19.4 0.1 (+16.2) | 44.5 0.4 (+20.6) |
| Few-shot | 61.6 0.5 (+15.8) | 45.6 0.2 (+36.4) | 46.8 0.3 (+46.5) | 62.4 0.1 (+1.2) | 15.8 0.5 (+12.6) | 46.4 0.2 (+22.5) |
| Chain-of-thought | 61.4 0.3 (+15.6) | 45.4 0.5 (+36.2) | 45.9 0.2 (+45.6) | 61.1 0.1 (-0.1) | 18.6 0.4 (+15.4) | 46.5 0.3 (+22.6) |
| Bloom-based | 57.3 0.1 (+11.5) | 36.6 0.5 (+27.4) | 29.4 0.2 (+29.1) | 61.5 0.4 (+0.3) | 21.7 0.3 (+18.5) | 41.3 0.5 (+17.4) |
| SFT (Subject-Specific) | 65.4 0.2 (+19.6) | 23.9 0.3 (+14.7) | 46.1 0.1 (+45.8) | 64.2 0.4 (+3.0) | 20.2 0.5 (+17.0) | 44.0 0.2 (+20.1) |
| Metric 1 | Metric 2 | Correlation | -value |
|---|---|---|---|
| Utility | Salience | 0.097 | 0.003 |
| EIG | 0.512 | ||
| Exam Embedding Sim. | 0.250 | 0.001 | |
| Exam ROUGE | 0.040 | 0.010 | |
| Bloom’s Taxonomy | 0.120 | 0.001 |
| Sal. | Util. | Question |
|---|---|---|
| High | High | How does the combination of elements result in properties that are different from those of the individual elements in their uncombined states, and can you provide more examples of such transformations? |
| Low | High | What are the ‘Twelve Principles of Green Chemistry’? |
| High | Low | What happens to the charge of an atom when it gains or loses electrons? |
| Microbiology | Chemistry | Economics | Sociology | US History | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Filter | E | Sal. | EIG | E | Sal. | EIG | E | Sal. | EIG | E | Sal. | EIG | E | Sal. | EIG |
| 76 | 4.27 | .18 | 46 | 4.65 | .20 | 58 | 4.70 | .04 | 65 | 4.49 | .02 | 31 | 4.65 | .01 | |
| 73 | 4.42 | .24 | 39 | 4.46 | .22 | 46 | 4.66 | .08 | 64 | 4.49 | .03 | 23 | 4.68 | .02 | |
| 61 | 4.21 | .17 | 32 | 4.40 | .09 | 47 | 4.65 | .01 | 62 | 4.46 | .01 | 21 | 4.65 | .01 | |
| QG ( ) | AG ( ) | RS ( ) | Microbiology | Chemistry | Economics | Sociology | US History | |
|---|---|---|---|---|---|---|---|---|
| Zero-shot | 4o-mini | 4o-mini | 4o-mini | 62.0 | 41.4 | 39.8 | 60.9 | 23.3 |
| 4o | 4o-mini | 4o-mini | 68.1 | 45.7 | 46.6 | 63.4 | 18.0 | |
| 4o-mini | 4o | 4o-mini | 68.2 | 42.2 | 48.0 | 63.4 | 23.2 | |
| 4o-mini | 4o-mini | 4o | 71.0 | 17.3 | 47.6 | 56.4 | 26.3 | |
| Quest | 4o-mini | 4o-mini | 4o-mini | 75.6 | 45.7 | 58.2 | 64.9 | 31.1 |
| EIG | Salience | Utility | |
|---|---|---|---|
| Useful. | 0.219 | 0.007 | 0.321 |
| Useful. | 3.00 0.10 | 2.97 0.09 | 3.96 0.09 ∗∗ |
| Interesting. | 3.13 0.08 | 3.23 0.09 | 3.26 0.09 |
| Rank | 2.25 0.06 | 2.04 0.05 | 1.71 0.05 ∗ |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Subject | Input Tokens | Output Tokens | Total Tokens | API Calls | Cost |
|---|---|---|---|---|---|
| US History | 1.91M | 0.40M | 2.31M | 5,479 | $0.53 |
| Microbiology | 3.59M | 0.74M | 4.33M | 10,456 | $0.98 |
| Subject | Salience Utility Question |
|---|---|
| Chemistry | High High How does the combination of elements result in properties that are different from those of the individual elements in their uncombined states, and can you provide more examples of such transformations? Low High What are the ‘Twelve Principles of Green Chemistry’? High Low What happens to the charge of an atom when it gains or loses electrons? |
| Economics | High High Why does inelastic demand allow the government to pass the tax burden onto consumers without significantly affecting the quantity demanded? Low High What are the three different combinations of labor and physical capital mentioned for cleaning up a park? High Low How does José evaluate the trade-off between giving up the third T-shirt and purchasing two additional movies in terms of marginal utility, and what does this reveal about his preferences for consumption? |
| Sociology | High High How does the concept of carrying capacity relate to the tragedy of the commons as described by Garrett Hardin and William Forster Lloyd? Low High What are some reasons people often forget where their old electronics are disposed of or stored? High Low How do the challenges faced by students from low socioeconomic backgrounds, as described in the example, impact their ability to succeed in the educational system compared to their peers from higher social classes? |
| U.S. History | High High What were the main differences in the motivations and goals of the English migrants who settled in the Chesapeake Bay colonies compared to those who settled in New England? Low High What challenges did Venetian sailors face when transporting goods along the old Silk Road? High Low What were some of the challenges and dangers faced by serfs and their families in their daily lives during the Middle Ages? |
| Microbiology | High High What roles do NAD+, NADP+, and FAD play in cellular metabolism, and how do their oxidized and reduced forms differ in function? Low High What evidence did Robert Remak provide to support the idea that cells originate from other cells? High Low What was the significance of Dmitri Ivanovski’s use of the Chamberland filter in discovering the cause of tobacco mosaic disease? |