What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language

Peng Cui; Qiaoyuan Zheng; Rudolf Debelak; Mrinmaya Sachan

cs.CLOct 1, 2026

What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language

Authors: Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan

Organizations: Department of Computer Science ETH Zurich · Machine Learning and Optimization Lab EPFL

Abstract

Difficulty is one of the most fundamental properties of a question: it determines whether the question can meaningfully discriminate between models of differing ability. Although a variety of methods can now estimate or predict difficulty automatically, they yield only a single descriptive number, with no account of the underlying factors that make a question difficult in the first place. In this work, we propose a data-driven approach that automatically generates and validates natural-language hypotheses explaining what makes one question harder than another. We first estimate each item's difficulty from the responses of a large pool of LLMs using Item Response Theory. We then sample contrasting sets of easy and hard questions and prompt an LLM to propose candidate explanations of the difference, which are subsequently validated and selected on held-out questions. Experimental results across three datasets spanning mathematical, logical, and commonsense reasoning show that our method produces interpretable and predictive hypotheses. On their own, they predict the difficulty of unseen questions competitively with, or better than, advanced black-box difficulty regressors; used as additional features, they further improve those regressors, implying that they discover difficulty signals that existing models fail to capture. Moreover, we demonstrate that editing questions according to a hypothesis can shift their measured difficulty in the expected direction, indicating that the discovered hypotheses are causally valid difficulty factors rather than post-hoc descriptions. Our approach thus turns a purely descriptive difficulty score into actionable statements.

Figures & tables

Figure 1: We first estimate question difficulty from LLMs’ responses using an IRT model. We then present an LLM with contrasting pairs or groups of hard and easy questions and prompt it to propose candidate hypotheses that explain the difference between them. Finally, we retain the hypotheses that best explain the difficulty of held-out questions.

#	Top-5 Generated Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
GSM8K
1	The problem requires multistep bookkeeping in which intermediate results feed subsequent calculations.	$1.498_{\scriptscriptstyle[1.043,\,1.959]}$	0.899 ‡	51.0
2	The solver must derive a rate, ratio, or percentage rather than directly apply one.	$2.035_{\scriptscriptstyle[1.341,\,2.742]}$	1.174 ‡	9.0
3	The solver must set up and solve an algebraic equation rather than perform only sequential arithmetic.	$1.428_{\scriptscriptstyle[0.520,\,2.320]}$	0.790 †	4.0
4	The reference answer contains a logical error or contradiction that conflicts with the correct solution.	$3.696_{\scriptscriptstyle[3.043,\,4.351]}$	2.063 ‡	1.0
5	The solver must aggregate multiple time intervals or convert between time units to obtain a final duration.	$1.057_{\scriptscriptstyle[-0.238,\,2.519]}$	0.584 †	6.5

Table 1: Top hypotheses by our method on three datasets.

{\rm SEP}

is the difference in difficulty between matched and unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized effect size, where ‡ marks a large effect (

d\geq 0.8

) and † a medium effect (

d\geq 0.5

) ( Cohen, 2013 ) ; Coverage is the matched-item percentage. The full set of hypotheses by all methods are in Appendix E .

Figure 2: We fit the test set using hypotheses generated by different methods as features to test their explanatory power, and report the adjusted

R^{2}

for varying numbers of hypotheses. We rank the features by greedily picking, at each step, the hypothesis that yields the largest incremental

R^{2}

Approaches	GSM8K		BBH-structured		Winogrande
Approaches	$\mathrm{R^{2}}\uparrow$	$\mathrm{RMSE}\downarrow$	$\mathrm{R^{2}}\uparrow$	$\mathrm{RMSE}\downarrow$	$\mathrm{R^{2}}\uparrow$	$\mathrm{RMSE}\downarrow$
RoBerta-base finetuned	$0.362$ 0.02	1.454 0.02	0.646 0.01	0.698 0.01	-0.038 0.09	1.711 0.07
Gemma-3-4B last-token	0.191 0.00	1.637 0.00	0.554 0.00	0.783 0.00	-0.019 0.00	1.697 0.00
Qwen3-Embedding-4B	0.200 0.00	1.629 0.00	0.533 0.00	0.802 0.00	-0.000 0.00	1.681 0.00
Gemini-3.1-pro 10-shot	0.346 0.03	1.472 0.04	0.139 0.172	1.085 0.107	-0.134 0.14	1.788 0.11
Claude-sonnet-5 10-shot	0.041 0.06	1.782 0.05	0.043 0.284	1.139 0.165	0.080 0.03	1.618 0.03

Table 2: Performance of different difficulty prediction methods on three datasets. All methods are trained on

\mathcal{G}\cup\mathcal{V}

, and we report

\mathrm{R^{2}}

and

\mathrm{RMSE}

\mathcal{T}

. Results of RoBerta-base are averaged over 5 seeds, and prompting of LLMs over 3 set of demonstrations. The number of hypotheses

|\mathcal{H}|

for the baselines is selected on the verification pool over a sweep of

\{10,20,50\}

; for Edisa , we filtered out clusters with

\leq 5

members. Subscripts denote standard deviations across seeds. Lasso and OLS are deterministic given the data, so no

\mathrm{s.d.}

is reported for them. Best results are in bold and second-best are underlined .

Figure 3: Comparison of

R^{2}

gains from incorporating hypotheses generated by different methods into different regression models (Base-model performance is shown below).

Hypothesis	Requires tracking and calculating values for four or more distinct entities or categories.
Increase: original	Mishka bought 3 pairs of shorts, 3 pairs of pants, and 3 pairs of shoes. One pair of shorts costs $16.50. One pair of pants costs$ 22.50 and one pair of shoes costs $42. How many dollars did Mishka spend on all the clothing items? Ans.$ 243
Increase:
edited	Mishka bought 3 pairs of shorts, 3 pairs of pants, 3 pairs of shoes, and 3 shirts . One pair of shorts costs $16.50. One pair of pants costs$ 22.50, one pair of shoes costs $42, and one shirt costs$ 15 . How many dollars did Mishka spend on all the clothing items? Ans. $288
Decrease: original	Bernie has 4 dogs. They each need a certain amount of exercise per day. The first needs to walk 1 mile. The second needs to walk 4 miles. The third needs to walk 3 miles. On average, they need to walk 3 miles per day. How many miles does the last dog need? Ans. 4 miles
Decrease: edited	Bernie has 3 dogs . They each need a certain amount of exercise per day. The first needs to walk 1 mile. The second needs to walk 4 miles. On average, they need to walk 3 miles per day. How many miles does the last dog need? Ans. 4 miles

Table 3: Bidirectional editing for a hypothesis with an explicit entity-count threshold. Red text marks additions or replacements; the decrease rewrite also removes the third dog’s stated distance. The increase edit expands three clothing categories to four, whereas the decrease edit reduces four dogs to three. Both preserve the task and answer units.

Intervention	Orig.	Edited	${\Delta}\mathrm{Acc_{\mathcal{H}}}$	$\mathrm{SR}_{\mathcal{H}}$	Orig.	Edited	${\Delta}\mathrm{Acc_{\mathcal{H}}}$	$\mathrm{SR}_{\mathcal{H}}$
	GSM8K				BBH-structured
Increase difficulty	45.01	29.75	$-15.27$	70.9% (100/141)	60.63	36.75	$-23.88$	77.6% (52/67)
Decrease difficulty	25.61	57.79	$+32.18$	92.2% (47/51)	36.82	65.34	$+28.52$	85.5% (47/55)

Table 4: Hypothesis-guided rewriting results on both datasets. Orig. and Rewrite are mean accuracies (%) across pairs and models;

\mathrm{\Delta Acc}_{\mathcal{H}}

and

\mathrm{SR_{\mathcal{H}}}

are the accuracy shift and success rate (counts in parentheses) averaged over all hypotheses, respectively (Eq. 4 ).

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	#Models	$\|\mathcal{G}\|$	$\|\mathcal{V}\|$	$\|\mathcal{T}\|$	Min	Mean $\pm$ SD	Max
		Split			IRT difficulty $d$
GSM8K	5,000	719	400	200	$-3.40$	$1.13\pm 1.85$	$7.29$
BBH-structured	3,811	558	558	280	$-2.83$	$0.14\pm 1.19$	$4.86$
Winogrande	5,000	759	254	254	$-3.92$	$-0.70\pm 1.67$	$5.09$

Appendix

Table 5: Dataset statistics. #Models denotes the number of models whose responses are used to estimate IRT difficulty.

\mathcal{G}

\mathcal{V}

, and

\mathcal{T}

denote our hypothesis-generation, validation, and test sets, respectively. Difficulty statistics summarize unstandardized 1PL IRT difficulty

d

over

\mathcal{G}\cup\mathcal{V}\cup\mathcal{T}

. SD denotes the sample standard deviation across items.

Dataset	Seed	Question IDs
GSM8K	0	3344, 3268, 2478, 3542, 3198, 2561, 2761, 3407, 3132, 2894
	1	2823, 2876, 3727, 3336, 3078, 3406, 2555, 2866, 2806, 3139
	2	2946, 2602, 3689, 2950, 3273, 3149, 3272, 3306, 3005, 2880
BBH-structured	0	36661, 34883, 36010, 36121, 34851, 36773, 35797, 36779, 35888, 35503
	1	36235, 36068, 35911, 37867, 36206, 36670, 37800, 37794, 35495, 34780
	2	36664, 37919, 36648, 35527, 35981, 35567, 34958, 36780, 35475, 34813

Appendix

Table 6: Demonstrations used by the prompted judges, listed in increasing order of measured difficulty.

GSM8K
		Increase-difficulty edits			Decrease-difficulty edits
#	Model	Orig.	Edited	$\bm{\Delta}\mathbf{Acc.}$	Orig.	Edited	$\bm{\Delta}\mathbf{Acc.}$
1	TinyLlama-1.1B ( Zhang et al., 2024 )	1.4	1.4	$+$ 0.0	0.0	0.0	$+$ 0.0
2	Gemma-2B-IT ( Mesnard et al., 2024 )	4.3	6.4	$+$ 2.1	2.0	25.5	$+$ 23.5
3	Qwen1.5-0.5B-Chat ( Bai et al., 2023 )	10.6	2.8	$-$ 7.8	0.0	13.7	$+$ 13.7
4	Phi-1.5 ( Li et al., 2023 )	13.5	19.9	$+$ 6.4	3.9	43.1	$+$ 39.2

Appendix

Table 7: Counterfactual evaluation of hypothesis-guided question edits on GSM8K and BBH-structured across open-weight language models. Values are accuracies (%);

\Delta{\rm Acc.}

is rewritten-item accuracy minus original-item accuracy in percentage points. For increase-difficulty edits, a negative

\Delta{\rm Acc.}

is the intended outcome; for decrease-difficulty edits, a positive

\Delta{\rm Acc.}

is intended. GSM8K contains 141 increase and 51 decrease edits; BBH-structured contains 67 increase and 55 decrease edits.

	Increase					Decrease
#	Q	✓ $\to$ ✗	✗ $\to$ ✓	$\Delta\mathbf{Acc}_{h}$	$\mathbf{SR}_{h}$	Q	✓ $\to$ ✗	✗ $\to$ ✓	$\Delta\mathbf{Acc}_{h}$	$\mathbf{SR}_{h}$
GSM8K
1	11	46	9	$-$ 19.7	75.9	0	0	0	—	—
2	11	48	5	$-$ 22.99	83.4	0	0	0	—	—
3	9	37	11	$-$ 16.99	68.0	2	0	10	$+$ 29.41	29.4
4	28	97	40	$-$ 11.97	68.3	5	2	25	$+$ 27.06	36.5

Appendix

Table 8: Hypothesis-level rewriting outcomes.

Q

counts accepted rewrites; transition counts pool 17 models on GSM8K and 16 on BBH-structured. For each hypothesis

h

, let

n_{CC}

n_{CW}

n_{WW}

, and

n_{WC}

denote correct-to-correct, correct-to-incorrect, incorrect-to-incorrect, and incorrect-to-correct counts, respectively, and let

N=n_{CC}+n_{CW}+n_{WW}+n_{WC}

\Delta\mathrm{Acc}_{h}=100(n_{WC}-n_{CW})/N

is rewritten minus original accuracy on the percentage scale, not a relative percentage change. The success rate

\mathrm{SR}_{h}

100(n_{CW}+n_{WW})/N

for Increase and

100(n_{WC}+n_{CC})/N

for Decrease, including unchanged outcomes. Overall pools all observations within each dataset and direction. A dash denotes no accepted rewrites. Original outcomes come from historical RouterEval evaluations.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
GSM8K
1	Questions involving counter-intuitive operations or implicit variable relationships, such as returned items or equating frequency of work to earnings, are more likely to be rated as difficulty level 3.	$2.036_{\scriptscriptstyle[1.079,\,2.975]}$	1.154	6.0
2	Questions that require tracking and aggregating multiple entities or sub-totals over several steps tend to be rated as moderate to high difficulty.	$1.442_{\scriptscriptstyle[0.957,\,1.905]}$	0.855	40.5
3	The inclusion of negative numbers, unit conversions, or complex ratios significantly increases the difficulty level of a question.	$1.052_{\scriptscriptstyle[0.263,\,1.883]}$	0.584	10.5
4	Questions that require tracking and combining multiple sub-totals before reaching the final answer are more difficult than those with a single linear calculation path.	$1.291_{\scriptscriptstyle[0.798,\,1.774]}$	0.753	41.0
5	Problems that involve implicit conversions or understanding of units, such as ’dozen’ or ’a week’, often have a higher difficulty level.	$0.718_{\scriptscriptstyle[-0.130,\,1.607]}$	0.396	11.5

Appendix

Table 9: All 20 final HypoGenic hypotheses on GSM8K, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
BBH-structured
1	Questions where the final answer is explicitly stated verbatim in the prompt text are easier than those requiring implicit logical deduction.	$-0.949_{\scriptscriptstyle[-1.219,\,-0.672]}$	-0.865	30.2
2	Questions that require tracking and ordering five or more entities are generally harder than those involving fewer entities, unless the answer is explicitly stated.	$0.824_{\scriptscriptstyle[0.555,\,1.096]}$	0.743	44.0
3	Logical deduction tasks that require combining multiple relative constraints to deduce a complete sequence are typically rated at difficulty level 3.	$0.946_{\scriptscriptstyle[0.634,\,1.253]}$	0.859	28.4
4	Questions that require sorting numerical values or combining multiple relational constraints to find an intermediate position tend to be rated at the highest difficulty level.	$1.322_{\scriptscriptstyle[1.029,\,1.616]}$	1.261	21.5
5	Questions that require deducing the relative order of objects from multiple constraints are generally more difficult than questions that explicitly state the answer or involve simple counting.	$0.170_{\scriptscriptstyle[-0.112,\,0.450]}$	0.144	48.4

Appendix

Table 10: All 20 final HypoGenic hypotheses on BBH-structured, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Statistics exclude missing labels (275 of 280 test items have usable labels). Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
Winogrande
1	Questions requiring the model to infer physical constraints or spatial relationships, such as a bulky object breaking a band or a minuscule ring not fitting a gem, are typically harder.	$0.258_{\scriptscriptstyle[-0.400,\,0.933]}$	0.153	10.6
2	Increased syntactic distance or the use of complex clause structures between the target blank and the contextual clues increases the difficulty level.	$0.263$ ${}_{[\text{---}]}$	—	0.4
3	Questions containing explicit causal conjunctions like ’because’ or ’so’ that directly link an action to its immediate outcome are more likely to be rated as difficulty level 1.	$0.058_{\scriptscriptstyle[-0.385,\,0.486]}$	0.034	65.4
4	The presence of negation, sarcasm, or counter-intuitive scenarios increases the difficulty level of the question.	$0.070_{\scriptscriptstyle[-0.748,\,0.947]}$	0.041	4.7
5	Questions that rely on direct, common-sense word associations, such as ’chainsaw’ and ’tree cutter’, are easier (level 1) than those requiring multi-step logical deductions.	$-0.919_{\scriptscriptstyle[-1.573,\,-0.138]}$	-0.550	7.1

Appendix

Table 11: All 20 final HypoGenic hypotheses on Winogrande, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
GSM8K
1	no coherent item-level property Trigger: the final digit of the last number in an arithmetic expression, immediately preceding the equals sign.	—	—	100.0
2	no coherent item-level property Trigger: The space token immediately following the ’####’ marker before the final answer in GSM8K solutions.	$-0.579_{\scriptscriptstyle[-1.576,\,0.434]}$	-0.318	94.0
3	The question involves specific times of day formatted with a colon. Trigger: The colon ’:’ used in time formats (e.g., 5:00, 10:00).	$1.523_{\scriptscriptstyle[0.546,\,2.600]}$	0.855	8.0
4	no coherent item-level property Trigger: A space character immediately preceding a number.	$1.072_{\scriptscriptstyle[0.561,\,1.576]}$	0.612	39.0
5	no coherent item-level property Trigger: The equals sign (’ =’) immediately preceding a calculator annotation (’ ¡¡’) in the reference solution.	$0.136_{\scriptscriptstyle[-0.356,\,0.624]}$	0.074	53.0

Appendix

Table 12: The 20 explained HypotheSAEs features on GSM8K. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Matching means SAE activation

>0

, not LLM-judged agreement with the explanation. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
BBH-structured
1	The reference solution involves calculating a sum, average, or identifying a specific numerical quantity. Trigger: The space or first digit of a number that represents the result of an arithmetic calculation or a specific numerical value in the reasoning process.	$0.522_{\scriptscriptstyle[0.277,\,0.769]}$	0.453	31.1
2	The reference solution contains a list of relative ordering constraints or properties separated by commas, culminating in an ’and’. Trigger: The comma separating the penultimate and final items in a list of sequential logical deductions or conditions, typically right before ’and’.	—	—	100.0
3	The question asks to sort items by alphabetic order or find the second heaviest item. Trigger: The token ’abetic’ in the phrase ’alphabetic order’ or the token ’ second’ in the phrase ’The second heaviest’.	$0.869_{\scriptscriptstyle[0.109,\,1.707]}$	0.745	3.6
4	The question has at least six multiple-choice options (reaching option F or beyond). Trigger: The open parenthesis ’(’ for option F, or the name ’Gwen’ in option D.	$0.520_{\scriptscriptstyle[0.234,\,0.816]}$	0.450	27.9
5	The question involves logical deduction over ordered objects, specifically identifying the position of an object in a sequence. Trigger: the word ’the’ in phrases like ’from the left’ or ’from the right’ within multiple-choice options	$0.326_{\scriptscriptstyle[-0.185,\,0.828]}$	0.278	93.2

Appendix

Table 13: The 20 explained HypotheSAEs features on BBH-structured. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Matching means SAE activation

>0

, not LLM-judged agreement with the explanation. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
Winogrande
1	The reference solution explains the reasoning by attributing a state of being or property to one of the characters (e.g., being blind, Jewish, hungry, skilled, poor). Trigger: The copula verbs ’is’, ’are’, ’was’, ’be’ or the noun ’kids’ when describing a property or state of a person in the explanation of a Winograd schema.	$0.297_{\scriptscriptstyle[-0.528,\,1.064]}$	0.176	93.3
2	The question involves a scenario set in a workplace, business, or commercial context. Trigger: Tokens related to businesses, workplaces, or commercial entities (like ’factory’, ’plant’, ’supermarket’, ’company’, ’companies’, ’merchant’, ’office’) and their preceding determiners/modifiers.	$-0.208_{\scriptscriptstyle[-1.070,\,0.651]}$	-0.123	2.4
3	no coherent item-level property Trigger: The feature fires on nouns or names that are the object of a preposition or part of a noun phrase in the explanation section of a Winograd schema problem.	$-0.383_{\scriptscriptstyle[-1.182,\,0.563]}$	-0.227	2.4
4	The reference solution uses a complex sentence structure with relative clauses, modal verbs, or infinitive phrases to explain the reasoning. Trigger: Tokens like ’would’, ’which’, ’to’, ’that’, ’can’, or commas that introduce a relative clause, modal verb, or infinitive phrase explaining a causal or conditional relationship in the reasoning.	$0.263_{\scriptscriptstyle[-0.774,\,1.286]}$	0.156	4.3
5	no coherent item-level property Trigger: Tokens related to quantity, degree, or specific actions (e.g., ’any’, ’his’, ’a’, ’many’, ’less’, ’cure’, ’pose’, ’of’) in the context of reasoning explanations.	$-0.020_{\scriptscriptstyle[-1.275,\,1.636]}$	-0.012	2.0

Appendix

Table 14: The 20 explained HypotheSAEs features on Winogrande. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Matching means SAE activation

>0

, not LLM-judged agreement with the explanation. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
GSM8K
1	requires calculating an average; specifically, the question asks for the mean or average of a set of values	$1.514_{\scriptscriptstyle[0.413,\,2.500]}$	0.835	2.5
2	involves unit conversions; specifically, the problem requires converting between different units of measurement (e.g., minutes to hours, kb to Mb, yards to miles)	$0.781_{\scriptscriptstyle[-0.225,\,1.926]}$	0.430	8.0
3	contains a narrative with physical actions causing subsequent events; specifically, the text describes an action (like throwing an object) that directly causes another event (like more objects falling)	$-1.026$ ${}_{[\text{---}]}$	—	0.5
4	involves multi-step arithmetic with more than three operations; specifically, the solution requires performing four or more distinct mathematical operations to reach the final answer	$1.574_{\scriptscriptstyle[1.114,\,2.040]}$	0.954	46.0
5	uses decimal numbers; specifically, the text contains non-integer numerical values such as 1.5, 1.25, or 2.50 in the problem description	$0.045_{\scriptscriptstyle[-0.630,\,0.765]}$	0.025	9.0

Appendix

Table 15: All 20 final NLParam hypotheses on GSM8K, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
BBH-structured
1	requires determining the position of an object relative to others; specifically, the question asks for the position of an object (e.g., ’third from the left’, ’rightmost’) based on a set of relative ordering constraints	$0.309_{\scriptscriptstyle[0.009,\,0.609]}$	0.264	27.5
2	involves logical deduction of a sequence; specifically, the text requires ordering a set of objects based on a series of relative constraints	$-0.048_{\scriptscriptstyle[-0.315,\,0.223]}$	-0.041	54.3
3	requires multi-step logical deduction to determine a sequence; specifically, the text provides a set of constraints about the relative order or age of objects and asks to identify the position of a specific object	$0.134_{\scriptscriptstyle[-0.142,\,0.411]}$	0.114	49.3
4	includes an example sentence explaining how to read the data; specifically, the text contains a phrase like ’	$-0.021_{\scriptscriptstyle[-0.627,\,0.624]}$	-0.018	6.4
5	has a multiple-choice format with more than five options; specifically, the text presents a question followed by six or more lettered options	$0.454_{\scriptscriptstyle[0.158,\,0.737]}$	0.393	37.5

Appendix

Table 16: All 20 final NLParam hypotheses on BBH-structured, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
Winogrande
1	requires reasoning about inverse relationships; specifically, the text requires understanding that an increase in one quantity (e.g., eating more) implies a decrease or lack in another (e.g., having a smaller lunch or being hungry)	$0.096_{\scriptscriptstyle[-0.864,\,1.203]}$	0.057	3.9
2	involves negation in the premise; specifically, the text contains words like ’not’, ’didn’t’, or ’unlike’ to establish a negative condition for one of the subjects	$-0.225_{\scriptscriptstyle[-0.666,\,0.233]}$	-0.134	18.5
3	uses dimensional or spatial adjectives; specifically, the sentence containing the blank uses words like ’small’, ’big’, ’broad’, or ’confined’ to describe the missing entity	$0.300_{\scriptscriptstyle[-0.399,\,1.014]}$	0.178	10.2
4	requires inferring the cause of a state or action from a contrasting pair of individuals; specifically, the text contrasts two people and requires deducing which one is responsible for a specific outcome or state based on their described characteristics	$-0.229_{\scriptscriptstyle[-0.635,\,0.200]}$	-0.136	28.7
5	involves contrasting actions or states between two individuals; specifically, the text describes two people doing different things or having different characteristics, and the blank must be resolved by matching the correct person to the concluding action or state	$-0.115_{\scriptscriptstyle[-0.525,\,0.308]}$	-0.068	36.2

Appendix

Table 17: All 20 final NLParam hypotheses on Winogrande, in exported order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
GSM8K
1	The question requires calculating a total by summing multiple intermediate values derived from rates, prices, or percentages	$0.197_{\scriptscriptstyle[-0.345,\,0.751]}$	0.108	26.0
2	The question is a math word problem that can be solved using exactly two basic arithmetic operations	$-1.583_{\scriptscriptstyle[-2.095,\,-1.054]}$	-0.927	22.0
3	The question involves calculating the age of one or more individuals based on relative age differences and time shifts	$0.802_{\scriptscriptstyle[0.346,\,1.254]}$	0.439	1.0
4	The question requires calculating a final value by determining and combining at least two intermediate quantities	$1.222_{\scriptscriptstyle[0.751,\,1.693]}$	0.707	61.0
5	The question is a multi-step math word problem that requires accounting for implicit entities, rounding rules, or multi-period summations	$3.144_{\scriptscriptstyle[0.823,\,5.391]}$	1.771	2.0

Appendix

Table 18: All 44 hypotheses exported by the Edisa baseline on GSM8K with a maximum of 50 hypotheses, in exported order. Repeated hypothesis texts are retained. Statistics use all 200 test items. Negative SEP indicates lower mean difficulty among matched items. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
BBH-structured
1	The question asks for the position of an object whose position is explicitly stated in the provided text, requiring no actual logical deduction	$-0.987_{\scriptscriptstyle[-1.274,\,-0.703]}$	-0.872	12.1
2	The question requires deducing the left-to-right order of a sequence of books based on relative position clues	$-0.014_{\scriptscriptstyle[-0.574,\,0.568]}$	-0.012	5.4
3	The question asks to identify the available time gap in a person’s schedule when they could have visited a specific location	$0.144_{\scriptscriptstyle[-0.087,\,0.377]}$	0.122	20.7
4	The question asks for the position of an object, and the correct answer is explicitly stated verbatim in the provided text, requiring no actual logical deduction	$-0.987_{\scriptscriptstyle[-1.271,\,-0.700]}$	-0.872	12.1
5	asks for the color of the left-most item in a list of objects	$-1.099_{\scriptscriptstyle[-1.317,\,-0.882]}$	-0.936	0.7

Appendix

Table 19: All 47 hypotheses exported by the Edisa baseline on BBH-structured with a maximum of 50 hypotheses, in exported order. Repeated hypothesis texts are retained. Statistics use all 280 test items. Negative SEP indicates lower mean difficulty among matched items. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
Winogrande
1	The question asks the model to choose between two options to fill in a blank or resolve a pronoun in a given sentence based on contextual clues	—	—	100.0
2	The question asks the user to choose the correct entity from two options to fill in a blank in a given sentence	—	—	100.0
3	The question requires resolving a missing word or pronoun in a sentence by choosing between two provided options	—	—	100.0
4	The question requires resolving an ambiguous pronoun or blank in a sentence to one of two given noun options	—	—	100.0
5	The question requires resolving a pronoun or blank in a sentence by choosing between two provided options, similar to a Winograd Schema Challenge	—	—	100.0

Appendix

Table 20: All 37 hypotheses exported by the Edisa baseline on Winogrande with a maximum of 50 hypotheses, in exported order. Repeated hypothesis texts are retained. Statistics use all 254 test items. Negative SEP indicates lower mean difficulty among matched items. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
GSM8K
1	The question contains implicit information or requires the solver to infer a missing value from a combination of given constraints (e.g., working backwards from a total or a difference).	$1.112_{\scriptscriptstyle[0.546,\,1.703]}$	0.623	15.0
2	The question involves multi-step bookkeeping where intermediate quantities must be calculated and then used as inputs for subsequent calculations, rather than a simple linear sequence of operations.	$1.498_{\scriptscriptstyle[1.043,\,1.954]}$	0.899	51.0
3	The reference answer contains a logical error or contradiction, such as ignoring a stated value and summing the wrong numbers, making it difficult for a correct model to match the expected output.	$3.696_{\scriptscriptstyle[3.037,\,4.361]}$	2.063	1.0
4	The question requires more than three distinct arithmetic operations to reach the final answer.	$0.735_{\scriptscriptstyle[0.185,\,1.311]}$	0.408	26.0
5	Requires at least five distinct arithmetic operations to reach the final answer.	$1.147_{\scriptscriptstyle[0.567,\,1.732]}$	0.648	20.0

Appendix

Table 21: The 40 hypotheses selected by our main-table model on GSM8K, in selection order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
BBH-structured
1	The question asks for a fact, property, or position that is explicitly stated in the text (e.g., ’The plums are the cheapest’, ’What color is the textbook?’), requiring only direct lookup rather than logical deduction.	$-1.278_{\scriptscriptstyle[-1.521,\,-1.032]}$	-1.206	20.0
2	The question asks to identify an entity at a specific non-extreme ordinal position (e.g., ’second-to-last’, ’third from the right’) rather than at an extreme end (e.g., ’leftmost’, ’cheapest’).	$0.565_{\scriptscriptstyle[0.257,\,0.875]}$	0.492	28.2
3	The question requires sorting a list of strings (e.g., names) alphabetically to determine the answer, rather than performing a simple numerical minimum/maximum operation or direct lookup.	$2.214_{\scriptscriptstyle[0.757,\,3.670]}$	1.905	0.7
4	The question requires identifying an object based on its relative position to another specific object (e.g., ’directly to the right of’, ’furthest from’) rather than its absolute position in the sequence (e.g., ’left-most’, ’right-most’).	$0.505_{\scriptscriptstyle[0.118,\,0.888]}$	0.432	6.1
5	The question requires reasoning about relative spatial or temporal positions (e.g., ’to the right of’, ’between what times’) rather than absolute attributes (e.g., ’is the jug mauve?’).	$0.738_{\scriptscriptstyle[0.462,\,1.009]}$	0.658	62.1

Appendix

Table 22: The 12 hypotheses selected by our main-table model on BBH-structured, in selection order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

#	Hypotheses	$\mathbf{SEP}_{\scriptscriptstyle[95\%\,\mathrm{CI}]}$	Cohen’s $d$	Cov. (%)
Winogrande
1	The sentence structure involves a complex causal chain where the blank refers to an entity that is the cause of a state described earlier in the sentence, requiring the model to infer the cause from the effect.	$0.676_{\scriptscriptstyle[-0.262,\,1.729]}$	0.402	4.3
2	The easier question can be solved using strong lexical associations, common collocations, or direct causal links (e.g., ’tough stain’, ’IOS’ to ’iPhone’, ’lazy’ to ’failed’).	$-0.889_{\scriptscriptstyle[-1.583,\,-0.069]}$	-0.532	7.5
3	The question requires multi-step causal reasoning to connect the premise to the resolution, where the solver must infer an unstated intermediate state or action (e.g., hating a food means eating less of it, which leaves more to harvest; or accepting a challenge implies believing one can win), rather than relying on direct semantic associations or attribute matching (e.g., ice is cold, or an animal rights activist dislikes leather).	$0.916_{\scriptscriptstyle[0.449,\,1.393]}$	0.559	26.4
4	The question can be resolved using basic selectional restrictions or inherent physical properties of the entities (e.g., a cloth is soft, a video is boring, a color is bright) rather than requiring causal reasoning about the events described.	$-0.872_{\scriptscriptstyle[-1.456,\,-0.220]}$	-0.524	11.8
5	The harder question involves resolving a pronoun based on an action that caused regret or a negative emotional reaction (e.g., feeling bad and vowing not to do it again).	—	—	0.0

Appendix

Table 23: The 44 hypotheses selected by our main-table model on Winogrande, in selection order. SEP is the mean IRT difficulty of matched minus unmatched test items (95% bootstrap CI); Cohen’s

d

is the standardized difference, and coverage is the matched-item percentage. Undefined statistics are shown as —; CI and

d

are omitted when either group has fewer than two items.

Explore similar work

May 12, 2026cs.CL

Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring

Estimating question difficulty is a critical component in evaluating and improving large language models (LLMs) for question answering (QA). Existing approaches often rely on readability formulas, retrieval-based signals, or popularity statistics, which may not fully capture the reasoning challenges posed to modern LLMs. In this paper, we introduce Q-DAPS (Question Difficulty based on Answer Plausibility Scores) method, a novel approach that estimates question difficulty by computing the entropy of plausibility scores over candidate answers. We systematically evaluate Q-DAPS across four prominent QA datasets-TriviaQA, NQ, MuSiQue, and QASC-demonstrating that it consistently outperforms baselines. Moreover, Q-DAPS shows strong robustness across hyperparameter variations and question types. Extensive ablation studies further show that Q-DAPS remains robust across different plausibility estimation paradigms, model sizes, and realistic settings. Human evaluations further confirm strong alignment between Q-DAPS's difficulty estimates and human judgments of question difficulty. Overall, Q-DAPS provides an interpretable, scalable, and bias-resilient approach to question difficulty estimation in modern QA systems.

Jamshid Mozafari, Bhawna Piryani, Adam Jatowt

University of Innsbruck, Innsbruck, Austria

Physical Plausibility

Ask PDF

Jun 26, 2026cs.CL

Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

Predicting human item difficulty is central to educational assessment, where reliable estimates support fairness and effective test construction. Existing methods often depend on costly human calibration or item-level textual representations, providing limited evidence about the cognitive processes that make items difficult. We argue that difficulty should be viewed not only as a property of item text, but also as an observable consequence of the problem-solving burden an item induces. Large Reasoning Models (LRMs) offer scalable process evidence through reasoning traces, but such evidence must be structured to support interpretable modeling. To this end, we introduce Epi2Diff (Episode to Difficulty), a framework that maps LRM reasoning traces into cognitively grounded episode sequences. These episodes group trace segments into functional problem-solving states, enabling difficulty to be modeled through reasoning scale, effort allocation, and state transitions. Epi2Diff extracts compact episode-dynamic features and combines them with semantic item representations for human difficulty prediction. Experiments on four real-world human difficulty datasets show that Epi2Diff consistently outperforms strong baselines, including fine-tuned small language models, LLM in-context learning, and supervised LLM adaptation. On SAT-derived classification benchmarks, Epi2Diff achieves an 8.1% average relative gain over supervised LLM fine-tuning baselines. Further analyses show that harder items induce more effortful, iterative, and implementation-centered episode dynamics, rather than merely longer responses. These results demonstrate that cognitive episodes in LRM reasoning traces provide a predictive and interpretable process representation for human item difficulty, offering a new lens for educational measurement with reasoning models.

Chenguang Wang, Ming Li, Xinyue Zeng +4

1Virginia Tech · University of Maryland · 3MBZUAI +1

LLM Reasoning Strategies Large Reasoning Models Item Response Theory Semantic Representations Episodic Memory

Ask PDF

May 18, 2026stat.ME

Estimating Item Difficulty with Large Language Models as Experts

Accurate estimates of item difficulty are essential for valid assessment and effective adaptive learning. However, for newly created tasks, response data are typically unavailable. Pretesting and expert judgement can be costly and slow, while machine learning methods often require large labelled training datasets. Recent work suggests that large language models (LLMs) may help. However, there is limited evidence on the elicitation procedures and prompt configurations used to emulate experts for difficulty estimation. This study addresses this gap by evaluating three off-the-shelf LLMs as difficulty raters for newly created items without access to response data. Using an item bank from an online learning system, the study examined 6 domains of primary-school mathematics, with empirical difficulty estimates treated as empirical reference. The study used a full factorial design crossing three factors: judgement format (absolute vs pairwise), decision type (hard decisions vs token-probability-based estimates), and prompting strategy (zero-shot vs few-shot). LLM-derived difficulty estimates were compared with empirical difficulties using Spearman rank correlations. Across domains, LLM-based estimates exhibited moderate to strong positive correlations with empirical item difficulties. For simpler arithmetic tasks, some configurations approached the upper end of the accuracy range reported for human experts in previous research. Pairwise comparison consistently outperformed absolute judgement in the absence of additional refinements. However, when token-level probabilities were incorporated and examples of items with known empirical difficulty were provided, the absolute judgement configuration likewise demonstrated moderate-to-high alignment. The study positions LLMs as a promising tool for initial item calibration and offers insights into effective workflow configuration.

Diana Kolesnikova, Kirill Fedyanin, Abe D. Hofman +2

Department of Methodology and Statistics, Tilburg University, Tilburg, Netherlands · Smart Business Technologies, Belgrade, Serbia · Department of Psychological Methods, University of Amsterdam, Amsterdam, Netherlands +2

Item Response Theory Rater Experts Token-Level Uncertainty

Ask PDF