cs.CLOct 1, 2026

What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language

Authors: Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan

Organizations: Department of Computer Science ETH Zurich · Machine Learning and Optimization Lab EPFL

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Difficulty is one of the most fundamental properties of a question: it determines whether the question can meaningfully discriminate between models of differing ability. Although a variety of methods can now estimate or predict difficulty automatically, they yield only a single descriptive number, with no account of the underlying factors that make a question difficult in the first place. In this work, we propose a data-driven approach that automatically generates and validates natural-language hypotheses explaining what makes one question harder than another. We first estimate each item's difficulty from the responses of a large pool of LLMs using Item Response Theory. We then sample contrasting sets of easy and hard questions and prompt an LLM to propose candidate explanations of the difference, which are subsequently validated and selected on held-out questions. Experimental results across three datasets spanning mathematical, logical, and commonsense reasoning show that our method produces interpretable and predictive hypotheses. On their own, they predict the difficulty of unseen questions competitively with, or better than, advanced black-box difficulty regressors; used as additional features, they further improve those regressors, implying that they discover difficulty signals that existing models fail to capture. Moreover, we demonstrate that editing questions according to a hypothesis can shift their measured difficulty in the expected direction, indicating that the discovered hypotheses are causally valid difficulty factors rather than post-hoc descriptions. Our approach thus turns a purely descriptive difficulty score into actionable statements.

Figures & tables

Figure 1: We first estimate question difficulty from LLMs’ responses using an IRT model. We then present an LLM with contrasting pairs or groups of hard and easy questions and prompt it to propose candidate hypotheses that explain the difference between them. Finally, we retain the hypotheses that best explain the difficulty of held-out questions.
#Top-5 Generated HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
GSM8K
1The problem requires multistep bookkeeping in which intermediate results feed subsequent calculations.1.498[1.043,1.959]1.498_{\scriptscriptstyle[1.043,\,1.959]}0.899 ‡51.0
2The solver must derive a rate, ratio, or percentage rather than directly apply one.2.035[1.341,2.742]2.035_{\scriptscriptstyle[1.341,\,2.742]}1.174 ‡9.0
3The solver must set up and solve an algebraic equation rather than perform only sequential arithmetic.1.428[0.520,2.320]1.428_{\scriptscriptstyle[0.520,\,2.320]}0.790 †4.0
4The reference answer contains a logical error or contradiction that conflicts with the correct solution.3.696[3.043,4.351]3.696_{\scriptscriptstyle[3.043,\,4.351]}2.063 ‡1.0
5The solver must aggregate multiple time intervals or convert between time units to obtain a final duration.1.057[−0.238,2.519]1.057_{\scriptscriptstyle[-0.238,\,2.519]}0.584 †6.5
Table 1: Top hypotheses by our method on three datasets. SEP{\rm SEP} is the difference in difficulty between matched and unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized effect size, where ‡ marks a large effect ( d≥0.8d\geq 0.8 ) and † a medium effect ( d≥0.5d\geq 0.5 ) ( Cohen, 2013 ) ; Coverage is the matched-item percentage. The full set of hypotheses by all methods are in Appendix E .
Figure 2: We fit the test set using hypotheses generated by different methods as features to test their explanatory power, and report the adjusted R2R^{2} for varying numbers of hypotheses. We rank the features by greedily picking, at each step, the hypothesis that yields the largest incremental R2R^{2} .
ApproachesGSM8KBBH-structuredWinogrande
R2↑\mathrm{R^{2}}\uparrowRMSE↓\mathrm{RMSE}\downarrowR2↑\mathrm{R^{2}}\uparrowRMSE↓\mathrm{RMSE}\downarrowR2↑\mathrm{R^{2}}\uparrowRMSE↓\mathrm{RMSE}\downarrow
RoBerta-base finetuned0.3620.362 0.021.454 0.020.646 0.010.698 0.01-0.038 0.091.711 0.07
Gemma-3-4B last-token0.191 0.001.637 0.000.554 0.000.783 0.00-0.019 0.001.697 0.00
Qwen3-Embedding-4B0.200 0.001.629 0.000.533 0.000.802 0.00-0.000 0.001.681 0.00
Gemini-3.1-pro 10-shot0.346 0.031.472 0.040.139 0.1721.085 0.107-0.134 0.141.788 0.11
Claude-sonnet-5 10-shot0.041 0.061.782 0.050.043 0.2841.139 0.1650.080 0.031.618 0.03
Table 2: Performance of different difficulty prediction methods on three datasets. All methods are trained on G∪V\mathcal{G}\cup\mathcal{V} , and we report R2\mathrm{R^{2}} and RMSE\mathrm{RMSE} on T\mathcal{T} . Results of RoBerta-base are averaged over 5 seeds, and prompting of LLMs over 3 set of demonstrations. The number of hypotheses ∣H∣|\mathcal{H}| for the baselines is selected on the verification pool over a sweep of {10,20,50}\{10,20,50\} ; for Edisa , we filtered out clusters with ≤5\leq 5 members. Subscripts denote standard deviations across seeds. Lasso and OLS are deterministic given the data, so no s.d.\mathrm{s.d.} is reported for them. Best results are in bold and second-best are underlined .
Figure 3: Comparison of R2R^{2} gains from incorporating hypotheses generated by different methods into different regression models (Base-model performance is shown below).
Figure 6
HypothesisRequires tracking and calculating values for four or more distinct entities or categories.
Increase: originalMishka bought 3 pairs of shorts, 3 pairs of pants, and 3 pairs of shoes. One pair of shorts costs 16.50.Onepairofpantscosts16.50. One pair of pants costs 22.50 and one pair of shoes costs 42.HowmanydollarsdidMishkaspendonalltheclothingitems?Ans.42. How many dollars did Mishka spend on all the clothing items? Ans. 243
Increase: 
editedMishka bought 3 pairs of shorts, 3 pairs of pants, 3 pairs of shoes, and 3 shirts . One pair of shorts costs 16.50.Onepairofpantscosts16.50. One pair of pants costs 22.50, one pair of shoes costs 42,andoneshirtcosts42, and one shirt costs 15 . How many dollars did Mishka spend on all the clothing items? Ans. $288
Decrease: originalBernie has 4 dogs. They each need a certain amount of exercise per day. The first needs to walk 1 mile. The second needs to walk 4 miles. The third needs to walk 3 miles. On average, they need to walk 3 miles per day. How many miles does the last dog need? Ans. 4 miles
Decrease: editedBernie has 3 dogs . They each need a certain amount of exercise per day. The first needs to walk 1 mile. The second needs to walk 4 miles. On average, they need to walk 3 miles per day. How many miles does the last dog need? Ans. 4 miles
Table 3: Bidirectional editing for a hypothesis with an explicit entity-count threshold. Red text marks additions or replacements; the decrease rewrite also removes the third dog’s stated distance. The increase edit expands three clothing categories to four, whereas the decrease edit reduces four dogs to three. Both preserve the task and answer units.
 GSM8KBBH-structured
InterventionOrig.EditedΔAccH{\Delta}\mathrm{Acc_{\mathcal{H}}}SRH\mathrm{SR}_{\mathcal{H}}Orig.EditedΔAccH{\Delta}\mathrm{Acc_{\mathcal{H}}}SRH\mathrm{SR}_{\mathcal{H}}
Increase difficulty45.0129.75−15.27-15.2770.9% (100/141)60.6336.75−23.88-23.8877.6% (52/67)
Decrease difficulty25.6157.79+32.18+32.1892.2% (47/51)36.8265.34+28.52+28.5285.5% (47/55)
Table 4: Hypothesis-guided rewriting results on both datasets. Orig. and Rewrite are mean accuracies (%) across pairs and models; ΔAccH\mathrm{\Delta Acc}_{\mathcal{H}} and SRH\mathrm{SR_{\mathcal{H}}} are the accuracy shift and success rate (counts in parentheses) averaged over all hypotheses, respectively (Eq. 4 ).
Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

  SplitIRT difficulty dd
Dataset#Models∣G∣|\mathcal{G}|∣V∣|\mathcal{V}|∣T∣|\mathcal{T}|MinMean ±\pm SDMax
GSM8K5,000719400200−3.40-3.401.13±1.851.13\pm 1.857.297.29
BBH-structured3,811558558280−2.83-2.830.14±1.190.14\pm 1.194.864.86
Winogrande5,000759254254−3.92-3.92−0.70±1.67-0.70\pm 1.675.095.09
Appendix
Table 5: Dataset statistics. #Models denotes the number of models whose responses are used to estimate IRT difficulty. G\mathcal{G} , V\mathcal{V} , and T\mathcal{T} denote our hypothesis-generation, validation, and test sets, respectively. Difficulty statistics summarize unstandardized 1PL IRT difficulty dd over G∪V∪T\mathcal{G}\cup\mathcal{V}\cup\mathcal{T} . SD denotes the sample standard deviation across items.
DatasetSeedQuestion IDs
GSM8K03344, 3268, 2478, 3542, 3198, 2561, 2761, 3407, 3132, 2894
12823, 2876, 3727, 3336, 3078, 3406, 2555, 2866, 2806, 3139
22946, 2602, 3689, 2950, 3273, 3149, 3272, 3306, 3005, 2880
BBH-structured036661, 34883, 36010, 36121, 34851, 36773, 35797, 36779, 35888, 35503
136235, 36068, 35911, 37867, 36206, 36670, 37800, 37794, 35495, 34780
236664, 37919, 36648, 35527, 35981, 35567, 34958, 36780, 35475, 34813
Appendix
Table 6: Demonstrations used by the prompted judges, listed in increasing order of measured difficulty.
  Increase-difficulty editsDecrease-difficulty edits
#ModelOrig.EditedΔAcc.\bm{\Delta}\mathbf{Acc.}Orig.EditedΔAcc.\bm{\Delta}\mathbf{Acc.}
GSM8K
1TinyLlama-1.1B ( Zhang et al., 2024 )1.41.4++ 0.00.00.0++ 0.0
2Gemma-2B-IT ( Mesnard et al., 2024 )4.36.4++ 2.12.025.5++ 23.5
3Qwen1.5-0.5B-Chat ( Bai et al., 2023 )10.62.8−- 7.80.013.7++ 13.7
4Phi-1.5 ( Li et al., 2023 )13.519.9++ 6.43.943.1++ 39.2
Appendix
Table 7: Counterfactual evaluation of hypothesis-guided question edits on GSM8K and BBH-structured across open-weight language models. Values are accuracies (%); ΔAcc.\Delta{\rm Acc.} is rewritten-item accuracy minus original-item accuracy in percentage points. For increase-difficulty edits, a negative ΔAcc.\Delta{\rm Acc.} is the intended outcome; for decrease-difficulty edits, a positive ΔAcc.\Delta{\rm Acc.} is intended. GSM8K contains 141 increase and 51 decrease edits; BBH-structured contains 67 increase and 55 decrease edits.
 IncreaseDecrease
#Q✓ →\to ✗✗ →\to ✓ΔAcch\Delta\mathbf{Acc}_{h}SRh\mathbf{SR}_{h}Q✓ →\to ✗✗ →\to ✓ΔAcch\Delta\mathbf{Acc}_{h}SRh\mathbf{SR}_{h}
GSM8K
111469−- 19.775.9000——
211485−- 22.9983.4000——
393711−- 16.9968.02010++ 29.4129.4
4289740−- 11.9768.35225++ 27.0636.5
Appendix
Table 8: Hypothesis-level rewriting outcomes. QQ counts accepted rewrites; transition counts pool 17 models on GSM8K and 16 on BBH-structured. For each hypothesis hh , let nCCn_{CC} , nCWn_{CW} , nWWn_{WW} , and nWCn_{WC} denote correct-to-correct, correct-to-incorrect, incorrect-to-incorrect, and incorrect-to-correct counts, respectively, and let N=nCC+nCW+nWW+nWCN=n_{CC}+n_{CW}+n_{WW}+n_{WC} . ΔAcch=100(nWC−nCW)/N\Delta\mathrm{Acc}_{h}=100(n_{WC}-n_{CW})/N is rewritten minus original accuracy on the percentage scale, not a relative percentage change. The success rate SRh\mathrm{SR}_{h} is 100(nCW+nWW)/N100(n_{CW}+n_{WW})/N for Increase and 100(nWC+nCC)/N100(n_{WC}+n_{CC})/N for Decrease, including unchanged outcomes. Overall pools all observations within each dataset and direction. A dash denotes no accepted rewrites. Original outcomes come from historical RouterEval evaluations.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
GSM8K
1Questions involving counter-intuitive operations or implicit variable relationships, such as returned items or equating frequency of work to earnings, are more likely to be rated as difficulty level 3.2.036[1.079,2.975]2.036_{\scriptscriptstyle[1.079,\,2.975]}1.1546.0
2Questions that require tracking and aggregating multiple entities or sub-totals over several steps tend to be rated as moderate to high difficulty.1.442[0.957,1.905]1.442_{\scriptscriptstyle[0.957,\,1.905]}0.85540.5
3The inclusion of negative numbers, unit conversions, or complex ratios significantly increases the difficulty level of a question.1.052[0.263,1.883]1.052_{\scriptscriptstyle[0.263,\,1.883]}0.58410.5
4Questions that require tracking and combining multiple sub-totals before reaching the final answer are more difficult than those with a single linear calculation path.1.291[0.798,1.774]1.291_{\scriptscriptstyle[0.798,\,1.774]}0.75341.0
5Problems that involve implicit conversions or understanding of units, such as ’dozen’ or ’a week’, often have a higher difficulty level.0.718[−0.130,1.607]0.718_{\scriptscriptstyle[-0.130,\,1.607]}0.39611.5
Appendix
Table 9: All 20 final HypoGenic hypotheses on GSM8K, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
BBH-structured
1Questions where the final answer is explicitly stated verbatim in the prompt text are easier than those requiring implicit logical deduction.−0.949[−1.219,−0.672]-0.949_{\scriptscriptstyle[-1.219,\,-0.672]}-0.86530.2
2Questions that require tracking and ordering five or more entities are generally harder than those involving fewer entities, unless the answer is explicitly stated.0.824[0.555,1.096]0.824_{\scriptscriptstyle[0.555,\,1.096]}0.74344.0
3Logical deduction tasks that require combining multiple relative constraints to deduce a complete sequence are typically rated at difficulty level 3.0.946[0.634,1.253]0.946_{\scriptscriptstyle[0.634,\,1.253]}0.85928.4
4Questions that require sorting numerical values or combining multiple relational constraints to find an intermediate position tend to be rated at the highest difficulty level.1.322[1.029,1.616]1.322_{\scriptscriptstyle[1.029,\,1.616]}1.26121.5
5Questions that require deducing the relative order of objects from multiple constraints are generally more difficult than questions that explicitly state the answer or involve simple counting.0.170[−0.112,0.450]0.170_{\scriptscriptstyle[-0.112,\,0.450]}0.14448.4
Appendix
Table 10: All 20 final HypoGenic hypotheses on BBH-structured, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Statistics exclude missing labels (275 of 280 test items have usable labels). Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
Winogrande
1Questions requiring the model to infer physical constraints or spatial relationships, such as a bulky object breaking a band or a minuscule ring not fitting a gem, are typically harder.0.258[−0.400,0.933]0.258_{\scriptscriptstyle[-0.400,\,0.933]}0.15310.6
2Increased syntactic distance or the use of complex clause structures between the target blank and the contextual clues increases the difficulty level.0.2630.263 [—]{}_{[\text{---}]}—0.4
3Questions containing explicit causal conjunctions like ’because’ or ’so’ that directly link an action to its immediate outcome are more likely to be rated as difficulty level 1.0.058[−0.385,0.486]0.058_{\scriptscriptstyle[-0.385,\,0.486]}0.03465.4
4The presence of negation, sarcasm, or counter-intuitive scenarios increases the difficulty level of the question.0.070[−0.748,0.947]0.070_{\scriptscriptstyle[-0.748,\,0.947]}0.0414.7
5Questions that rely on direct, common-sense word associations, such as ’chainsaw’ and ’tree cutter’, are easier (level 1) than those requiring multi-step logical deductions.−0.919[−1.573,−0.138]-0.919_{\scriptscriptstyle[-1.573,\,-0.138]}-0.5507.1
Appendix
Table 11: All 20 final HypoGenic hypotheses on Winogrande, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
GSM8K
1no coherent item-level property Trigger: the final digit of the last number in an arithmetic expression, immediately preceding the equals sign.——100.0
2no coherent item-level property Trigger: The space token immediately following the ’####’ marker before the final answer in GSM8K solutions.−0.579[−1.576,0.434]-0.579_{\scriptscriptstyle[-1.576,\,0.434]}-0.31894.0
3The question involves specific times of day formatted with a colon. Trigger: The colon ’:’ used in time formats (e.g., 5:00, 10:00).1.523[0.546,2.600]1.523_{\scriptscriptstyle[0.546,\,2.600]}0.8558.0
4no coherent item-level property Trigger: A space character immediately preceding a number.1.072[0.561,1.576]1.072_{\scriptscriptstyle[0.561,\,1.576]}0.61239.0
5no coherent item-level property Trigger: The equals sign (’ =’) immediately preceding a calculator annotation (’ ¡¡’) in the reference solution.0.136[−0.356,0.624]0.136_{\scriptscriptstyle[-0.356,\,0.624]}0.07453.0
Appendix
Table 12: The 20 explained HypotheSAEs features on GSM8K. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Matching means SAE activation >0>0 , not LLM-judged agreement with the explanation. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
BBH-structured
1The reference solution involves calculating a sum, average, or identifying a specific numerical quantity. Trigger: The space or first digit of a number that represents the result of an arithmetic calculation or a specific numerical value in the reasoning process.0.522[0.277,0.769]0.522_{\scriptscriptstyle[0.277,\,0.769]}0.45331.1
2The reference solution contains a list of relative ordering constraints or properties separated by commas, culminating in an ’and’. Trigger: The comma separating the penultimate and final items in a list of sequential logical deductions or conditions, typically right before ’and’.——100.0
3The question asks to sort items by alphabetic order or find the second heaviest item. Trigger: The token ’abetic’ in the phrase ’alphabetic order’ or the token ’ second’ in the phrase ’The second heaviest’.0.869[0.109,1.707]0.869_{\scriptscriptstyle[0.109,\,1.707]}0.7453.6
4The question has at least six multiple-choice options (reaching option F or beyond). Trigger: The open parenthesis ’(’ for option F, or the name ’Gwen’ in option D.0.520[0.234,0.816]0.520_{\scriptscriptstyle[0.234,\,0.816]}0.45027.9
5The question involves logical deduction over ordered objects, specifically identifying the position of an object in a sequence. Trigger: the word ’the’ in phrases like ’from the left’ or ’from the right’ within multiple-choice options0.326[−0.185,0.828]0.326_{\scriptscriptstyle[-0.185,\,0.828]}0.27893.2
Appendix
Table 13: The 20 explained HypotheSAEs features on BBH-structured. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Matching means SAE activation >0>0 , not LLM-judged agreement with the explanation. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
Winogrande
1The reference solution explains the reasoning by attributing a state of being or property to one of the characters (e.g., being blind, Jewish, hungry, skilled, poor). Trigger: The copula verbs ’is’, ’are’, ’was’, ’be’ or the noun ’kids’ when describing a property or state of a person in the explanation of a Winograd schema.0.297[−0.528,1.064]0.297_{\scriptscriptstyle[-0.528,\,1.064]}0.17693.3
2The question involves a scenario set in a workplace, business, or commercial context. Trigger: Tokens related to businesses, workplaces, or commercial entities (like ’factory’, ’plant’, ’supermarket’, ’company’, ’companies’, ’merchant’, ’office’) and their preceding determiners/modifiers.−0.208[−1.070,0.651]-0.208_{\scriptscriptstyle[-1.070,\,0.651]}-0.1232.4
3no coherent item-level property Trigger: The feature fires on nouns or names that are the object of a preposition or part of a noun phrase in the explanation section of a Winograd schema problem.−0.383[−1.182,0.563]-0.383_{\scriptscriptstyle[-1.182,\,0.563]}-0.2272.4
4The reference solution uses a complex sentence structure with relative clauses, modal verbs, or infinitive phrases to explain the reasoning. Trigger: Tokens like ’would’, ’which’, ’to’, ’that’, ’can’, or commas that introduce a relative clause, modal verb, or infinitive phrase explaining a causal or conditional relationship in the reasoning.0.263[−0.774,1.286]0.263_{\scriptscriptstyle[-0.774,\,1.286]}0.1564.3
5no coherent item-level property Trigger: Tokens related to quantity, degree, or specific actions (e.g., ’any’, ’his’, ’a’, ’many’, ’less’, ’cure’, ’pose’, ’of’) in the context of reasoning explanations.−0.020[−1.275,1.636]-0.020_{\scriptscriptstyle[-1.275,\,1.636]}-0.0122.0
Appendix
Table 14: The 20 explained HypotheSAEs features on Winogrande. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Matching means SAE activation >0>0 , not LLM-judged agreement with the explanation. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
GSM8K
1requires calculating an average; specifically, the question asks for the mean or average of a set of values1.514[0.413,2.500]1.514_{\scriptscriptstyle[0.413,\,2.500]}0.8352.5
2involves unit conversions; specifically, the problem requires converting between different units of measurement (e.g., minutes to hours, kb to Mb, yards to miles)0.781[−0.225,1.926]0.781_{\scriptscriptstyle[-0.225,\,1.926]}0.4308.0
3contains a narrative with physical actions causing subsequent events; specifically, the text describes an action (like throwing an object) that directly causes another event (like more objects falling)−1.026-1.026 [—]{}_{[\text{---}]}—0.5
4involves multi-step arithmetic with more than three operations; specifically, the solution requires performing four or more distinct mathematical operations to reach the final answer1.574[1.114,2.040]1.574_{\scriptscriptstyle[1.114,\,2.040]}0.95446.0
5uses decimal numbers; specifically, the text contains non-integer numerical values such as 1.5, 1.25, or 2.50 in the problem description0.045[−0.630,0.765]0.045_{\scriptscriptstyle[-0.630,\,0.765]}0.0259.0
Appendix
Table 15: All 20 final NLParam hypotheses on GSM8K, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
BBH-structured
1requires determining the position of an object relative to others; specifically, the question asks for the position of an object (e.g., ’third from the left’, ’rightmost’) based on a set of relative ordering constraints0.309[0.009,0.609]0.309_{\scriptscriptstyle[0.009,\,0.609]}0.26427.5
2involves logical deduction of a sequence; specifically, the text requires ordering a set of objects based on a series of relative constraints−0.048[−0.315,0.223]-0.048_{\scriptscriptstyle[-0.315,\,0.223]}-0.04154.3
3requires multi-step logical deduction to determine a sequence; specifically, the text provides a set of constraints about the relative order or age of objects and asks to identify the position of a specific object0.134[−0.142,0.411]0.134_{\scriptscriptstyle[-0.142,\,0.411]}0.11449.3
4includes an example sentence explaining how to read the data; specifically, the text contains a phrase like ’−0.021[−0.627,0.624]-0.021_{\scriptscriptstyle[-0.627,\,0.624]}-0.0186.4
5has a multiple-choice format with more than five options; specifically, the text presents a question followed by six or more lettered options0.454[0.158,0.737]0.454_{\scriptscriptstyle[0.158,\,0.737]}0.39337.5
Appendix
Table 16: All 20 final NLParam hypotheses on BBH-structured, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
Winogrande
1requires reasoning about inverse relationships; specifically, the text requires understanding that an increase in one quantity (e.g., eating more) implies a decrease or lack in another (e.g., having a smaller lunch or being hungry)0.096[−0.864,1.203]0.096_{\scriptscriptstyle[-0.864,\,1.203]}0.0573.9
2involves negation in the premise; specifically, the text contains words like ’not’, ’didn’t’, or ’unlike’ to establish a negative condition for one of the subjects−0.225[−0.666,0.233]-0.225_{\scriptscriptstyle[-0.666,\,0.233]}-0.13418.5
3uses dimensional or spatial adjectives; specifically, the sentence containing the blank uses words like ’small’, ’big’, ’broad’, or ’confined’ to describe the missing entity0.300[−0.399,1.014]0.300_{\scriptscriptstyle[-0.399,\,1.014]}0.17810.2
4requires inferring the cause of a state or action from a contrasting pair of individuals; specifically, the text contrasts two people and requires deducing which one is responsible for a specific outcome or state based on their described characteristics−0.229[−0.635,0.200]-0.229_{\scriptscriptstyle[-0.635,\,0.200]}-0.13628.7
5involves contrasting actions or states between two individuals; specifically, the text describes two people doing different things or having different characteristics, and the blank must be resolved by matching the correct person to the concluding action or state−0.115[−0.525,0.308]-0.115_{\scriptscriptstyle[-0.525,\,0.308]}-0.06836.2
Appendix
Table 17: All 20 final NLParam hypotheses on Winogrande, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
GSM8K
1The question requires calculating a total by summing multiple intermediate values derived from rates, prices, or percentages0.197[−0.345,0.751]0.197_{\scriptscriptstyle[-0.345,\,0.751]}0.10826.0
2The question is a math word problem that can be solved using exactly two basic arithmetic operations−1.583[−2.095,−1.054]-1.583_{\scriptscriptstyle[-2.095,\,-1.054]}-0.92722.0
3The question involves calculating the age of one or more individuals based on relative age differences and time shifts0.802[0.346,1.254]0.802_{\scriptscriptstyle[0.346,\,1.254]}0.4391.0
4The question requires calculating a final value by determining and combining at least two intermediate quantities1.222[0.751,1.693]1.222_{\scriptscriptstyle[0.751,\,1.693]}0.70761.0
5The question is a multi-step math word problem that requires accounting for implicit entities, rounding rules, or multi-period summations3.144[0.823,5.391]3.144_{\scriptscriptstyle[0.823,\,5.391]}1.7712.0
Appendix
Table 18: All 44 hypotheses exported by the Edisa baseline on GSM8K with a maximum of 50 hypotheses, in exported order. Repeated hypothesis texts are retained. Statistics use all 200 test items. Negative SEP indicates lower mean difficulty among matched items. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
BBH-structured
1The question asks for the position of an object whose position is explicitly stated in the provided text, requiring no actual logical deduction−0.987[−1.274,−0.703]-0.987_{\scriptscriptstyle[-1.274,\,-0.703]}-0.87212.1
2The question requires deducing the left-to-right order of a sequence of books based on relative position clues−0.014[−0.574,0.568]-0.014_{\scriptscriptstyle[-0.574,\,0.568]}-0.0125.4
3The question asks to identify the available time gap in a person’s schedule when they could have visited a specific location0.144[−0.087,0.377]0.144_{\scriptscriptstyle[-0.087,\,0.377]}0.12220.7
4The question asks for the position of an object, and the correct answer is explicitly stated verbatim in the provided text, requiring no actual logical deduction−0.987[−1.271,−0.700]-0.987_{\scriptscriptstyle[-1.271,\,-0.700]}-0.87212.1
5asks for the color of the left-most item in a list of objects−1.099[−1.317,−0.882]-1.099_{\scriptscriptstyle[-1.317,\,-0.882]}-0.9360.7
Appendix
Table 19: All 47 hypotheses exported by the Edisa baseline on BBH-structured with a maximum of 50 hypotheses, in exported order. Repeated hypothesis texts are retained. Statistics use all 280 test items. Negative SEP indicates lower mean difficulty among matched items. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
Winogrande
1The question asks the model to choose between two options to fill in a blank or resolve a pronoun in a given sentence based on contextual clues——100.0
2The question asks the user to choose the correct entity from two options to fill in a blank in a given sentence——100.0
3The question requires resolving a missing word or pronoun in a sentence by choosing between two provided options——100.0
4The question requires resolving an ambiguous pronoun or blank in a sentence to one of two given noun options——100.0
5The question requires resolving a pronoun or blank in a sentence by choosing between two provided options, similar to a Winograd Schema Challenge——100.0
Appendix
Table 20: All 37 hypotheses exported by the Edisa baseline on Winogrande with a maximum of 50 hypotheses, in exported order. Repeated hypothesis texts are retained. Statistics use all 254 test items. Negative SEP indicates lower mean difficulty among matched items. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
GSM8K
1The question contains implicit information or requires the solver to infer a missing value from a combination of given constraints (e.g., working backwards from a total or a difference).1.112[0.546,1.703]1.112_{\scriptscriptstyle[0.546,\,1.703]}0.62315.0
2The question involves multi-step bookkeeping where intermediate quantities must be calculated and then used as inputs for subsequent calculations, rather than a simple linear sequence of operations.1.498[1.043,1.954]1.498_{\scriptscriptstyle[1.043,\,1.954]}0.89951.0
3The reference answer contains a logical error or contradiction, such as ignoring a stated value and summing the wrong numbers, making it difficult for a correct model to match the expected output.3.696[3.037,4.361]3.696_{\scriptscriptstyle[3.037,\,4.361]}2.0631.0
4The question requires more than three distinct arithmetic operations to reach the final answer.0.735[0.185,1.311]0.735_{\scriptscriptstyle[0.185,\,1.311]}0.40826.0
5Requires at least five distinct arithmetic operations to reach the final answer.1.147[0.567,1.732]1.147_{\scriptscriptstyle[0.567,\,1.732]}0.64820.0
Appendix
Table 21: The 40 hypotheses selected by our main-table model on GSM8K, in selection order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
BBH-structured
1The question asks for a fact, property, or position that is explicitly stated in the text (e.g., ’The plums are the cheapest’, ’What color is the textbook?’), requiring only direct lookup rather than logical deduction.−1.278[−1.521,−1.032]-1.278_{\scriptscriptstyle[-1.521,\,-1.032]}-1.20620.0
2The question asks to identify an entity at a specific non-extreme ordinal position (e.g., ’second-to-last’, ’third from the right’) rather than at an extreme end (e.g., ’leftmost’, ’cheapest’).0.565[0.257,0.875]0.565_{\scriptscriptstyle[0.257,\,0.875]}0.49228.2
3The question requires sorting a list of strings (e.g., names) alphabetically to determine the answer, rather than performing a simple numerical minimum/maximum operation or direct lookup.2.214[0.757,3.670]2.214_{\scriptscriptstyle[0.757,\,3.670]}1.9050.7
4The question requires identifying an object based on its relative position to another specific object (e.g., ’directly to the right of’, ’furthest from’) rather than its absolute position in the sequence (e.g., ’left-most’, ’right-most’).0.505[0.118,0.888]0.505_{\scriptscriptstyle[0.118,\,0.888]}0.4326.1
5The question requires reasoning about relative spatial or temporal positions (e.g., ’to the right of’, ’between what times’) rather than absolute attributes (e.g., ’is the jug mauve?’).0.738[0.462,1.009]0.738_{\scriptscriptstyle[0.462,\,1.009]}0.65862.1
Appendix
Table 22: The 12 hypotheses selected by our main-table model on BBH-structured, in selection order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.
#HypothesesSEP[95%CI]\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}Cohen’s ddCov. (%)
Winogrande
1The sentence structure involves a complex causal chain where the blank refers to an entity that is the cause of a state described earlier in the sentence, requiring the model to infer the cause from the effect.0.676[−0.262,1.729]0.676_{\scriptscriptstyle[-0.262,\,1.729]}0.4024.3
2The easier question can be solved using strong lexical associations, common collocations, or direct causal links (e.g., ’tough stain’, ’IOS’ to ’iPhone’, ’lazy’ to ’failed’).−0.889[−1.583,−0.069]-0.889_{\scriptscriptstyle[-1.583,\,-0.069]}-0.5327.5
3The question requires multi-step causal reasoning to connect the premise to the resolution, where the solver must infer an unstated intermediate state or action (e.g., hating a food means eating less of it, which leaves more to harvest; or accepting a challenge implies believing one can win), rather than relying on direct semantic associations or attribute matching (e.g., ice is cold, or an animal rights activist dislikes leather).0.916[0.449,1.393]0.916_{\scriptscriptstyle[0.449,\,1.393]}0.55926.4
4The question can be resolved using basic selectional restrictions or inherent physical properties of the entities (e.g., a cloth is soft, a video is boring, a color is bright) rather than requiring causal reasoning about the events described.−0.872[−1.456,−0.220]-0.872_{\scriptscriptstyle[-1.456,\,-0.220]}-0.52411.8
5The harder question involves resolving a pronoun based on an action that caused regret or a negative emotional reaction (e.g., feeling bad and vowing not to do it again).——0.0
Appendix
Table 23: The 44 hypotheses selected by our main-table model on Winogrande, in selection order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s dd is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and dd are omitted when either group has fewer than two items.

Explore similar work

CardsList

  1. Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring

    May 12, 2026Jamshid Mozafari, Bhawna Piryani, Adam JatowtPhysical Plausibility

    University of Innsbruck, Innsbruck, Austria

  2. Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

    Jun 26, 2026Chenguang Wang, Ming Li, Xinyue Zeng +4LLM Reasoning StrategiesLarge Reasoning Models

    1Virginia Tech · University of Maryland · 3MBZUAI +1

  3. Estimating Item Difficulty with Large Language Models as Experts

    May 18, 2026Diana Kolesnikova, Kirill Fedyanin, Abe D. Hofman +2Item Response TheoryRater

    Department of Methodology and Statistics, Tilburg University, Tilburg, Netherlands · Smart Business Technologies, Belgrade, Serbia · Department of Psychological Methods, University of Amsterdam, Amsterdam, Netherlands +2