Finding Common Mistakes In Modelling With Mathematical Formalisms Using LLMs

Lilian Killich; Marko Schmellenkamp; Fabian Vehlken; Thomas Zeume

cs.CYSep 15, 2026

Finding Common Mistakes In Modelling With Mathematical Formalisms Using LLMs

Authors: Lilian Killich, Marko Schmellenkamp, Fabian Vehlken, Thomas Zeume

Organizations: Ruhr University Bochum Bochum, Germany

Abstract

Modelling with mathematical formalisms like logical formulas, mathematical equations, or regular expressions is an important yet challenging task for students of computer science and other STEM disciplines. Identifying common mistakes occurring in this context is an important step towards helping struggling students by providing targeted high-quality feedback, e.g. in interactive learning systems. We present a tool-supported workflow that allows to (1) identify candidates for common mistakes that explain many student mistakes in large educational data sets, (2) cluster candidates according to similarities, and (3) visualize resulting clusters for instructors and CS education researchers. The visualization is designed to help researchers to identify common modelling mistakes. The candidates for common mistakes are represented by bug fixing transformations that translate incorrect formalizations into correct formalizations; they are generated by an LLM and validated algorithmically. We show that this approach works well by reproducing common mistakes in propositional logic modelling that were identified by hand in the literature; showing that, unlike other algorithmic approaches, the LLM-based approach is suitable for very large sets of data; and applying it to multiple other formalisms to showcase it generalizes beyond propositional logic.

Figures & tables

Panel of Figure 1 — Figure 1. Illustration of the workflow for finding bug fixing transformations, clustering them, and visualizing the clusters.

Hand-picked mistake / transformations	#	%	#dst	%dst	example of (incorrect, correct) formula pairs
Mistake: disjunction instead of conjunction	334	5.47%	61	3.88%
Transformations combined	330	98.80%	58	95.08%
$\lnot\left(Y_{1}\land\ldots\land Y_{n}\right)$ $\rightsquigarrow$ $\lnot Y_{1}\land\ldots\land\lnot Y_{n}$	201	60.18%	23	37.70%	$\lnot(A\land B\land C),$	$\lnot A\land\lnot B\land\lnot C$
$\lnot\left(Y_{1}\land Y_{2}\right)$ $\rightsquigarrow$ $\lnot Y_{1}\land\lnot Y_{2}$	153	45.81%	16	26.23%	$\lnot(A\land B),$	$\lnot A\land\lnot B$
$\lnot\left(Y_{1}\land Y_{2}\right)\rightarrow Y_{3}$ $\rightsquigarrow$ $\left(\lnot Y_{1}\land\lnot Y_{2}\right)\rightarrow Y_{3}$	150	44.91%	15	24.59%	$\lnot(A\land B)\to C,$	$(\lnot A\land\lnot B)\to C$
$Y_{1}\lor\ldots\lor Y_{n}$ $\rightsquigarrow$ $Y_{1}\land\ldots\land Y_{n}$	128	38.32%	34	55.74%	$A\lor B\lor C,$	$A\land B\land C$

Table 1. Exemplary excerpt of the table of hierarchical bug-fixing transformation clusters that cover most of the pairs explained by the common mistake that a disjunction was erroneously used instead of a conjunction (as identified in ( Schmellenkamp et al., 2023 ) ). For each transformation, a simple representative example is provided. For the mistake as a whole, each transformation, and the entire cluster, the number # and distinct number #dst of pairs are shown. The common mistake occurrs in

5.47

% of all pairs in the full data set (

3.88

% distinct). Of these, the complete bug fixing transformation cluster found by our workflow explains

330

(

98.8

%). The fraction of the

334

pairs each transformation explains is also shown (columns % and %dst, respectively). A table with all bug fixing transformation clusters found by our workflow is presented in the (anonymously published) supplementary material ( Authors, 2026 ) .

	PL	ML	CTL
# total formula pairs	$6106$	$12482$	$7210$
# modelling contexts	$11$	$6$	$6$
# statements to be modelled	$58$	$28$	$20$
# total clusters	248	1585	271
% total explained	$84.44$ %	$79.39$ %	$35.89$ %
# clusters explaining $\geq 0.1$ %	78	132	62

Table 2. Overview of the data sets, number of clusters found by our workflow, and the percentage of pairs explained by clusters for the propositional logic (PL), modal logic (PL) and computation tree logic (CTL) data sets. For each data set, we provide the number of (incorrect, correct) formula pairs, modelling contexts (i.e. different assignments), and statements to be modelled.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Hand-picked mistake / Transformations	#	%	#dst	%dst
Unexplained by ALL hand errors	48	100.00%	26	100.00%
Transformations combined	13	27.08%	12	46.15%
T1: $Y_{1}\land Y_{2}\land\lnot\left(Y_{3}\land Y_{4}\right)$ $\rightsquigarrow$ $Y_{1}\land Y_{2}\land\lnot Y_{3}\land\lnot Y_{4}$	4	8.33%	3	11.54%
T2: $\left(Y_{1}\land Y_{2}\right)\leftrightarrow\lnot\left(Y_{3}\land Y_{4}\right)$ $\rightsquigarrow$ $Y_{1}\land Y_{2}\land\lnot Y_{3}\land\lnot Y_{4}$	1	2.08%	1	3.85%
T3: $\left(Y_{1}\land Y_{2}\right)\rightarrow\left(\lnot Y_{3}\land\lnot Y_{4}\right)$ $\rightsquigarrow$ $Y_{1}\land Y_{2}\land\lnot Y_{3}\land\lnot Y_{4}$	1	2.08%	1	3.85%
T4: $\left(Y_{1}\land Y_{2}\right)\leftrightarrow\left(\lnot Y_{3}\land\lnot Y_{4}\right)$ $\rightsquigarrow$ $Y_{1}\land Y_{2}\land\lnot Y_{3}\land\lnot Y_{4}$	1	2.08%	1	3.85%

Appendix

Table 3. Hand-picked errors and transformations for linguistic operator(s) AND_AND_NONE_OF

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_ADD_OPERAND_TO_CONJUNCTION	19	65.52%	4	30.77%
Transformations combined	0	0.00%	0	0.00%
BUGGY_REPLACE_DISJUNCTION_BY_CONJUNCTION	2	6.90%	2	15.38%
Transformations combined	2	100.00%	2	100.00%
T1: $Y_{1}\lor\ldots\lor Y_{n}$ $\rightsquigarrow$ $Y_{1}\land\ldots\land Y_{n}$	1	50.00%	1	50.00%
T4: $\left(Y_{1}\lor Y_{2}\right)\land Y_{3}$ $\rightsquigarrow$ $Y_{3}\land Y_{1}\land Y_{2}$	1	50.00%	1	50.00%

Appendix

Table 4. Hand-picked errors and transformations for linguistic operator(s) AND_AND

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	12	8.89%	1	1.69%
Transformations combined	12	100.00%	1	100.00%
T2: $\left(Y_{1}\rightarrow Y_{2}\right)\land Y_{3}$ $\rightsquigarrow$ $\left(Y_{2}\rightarrow Y_{1}\right)\land Y_{3}$	12	100.00%	1	100.00%
BUGGY_REPLACE_IMPLICATION_BY_CONJUNCTION	10	7.41%	3	5.08%
Transformations combined	8	80.00%	1	33.33%
T4: $Y_{1}\rightarrow\left(Y_{2}\rightarrow Y_{3}\right)$ $\rightsquigarrow$ $\left(Y_{2}\rightarrow Y_{3}\right)\land Y_{1}$	8	80.00%	1	33.33%

Appendix

Table 5. Hand-picked errors and transformations for linguistic operator(s) AND_IF

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_EQUIVALENCE_BY_CONJUNCTION	1	100.00%	1	100.00%
Transformations combined	1	100.00%	1	100.00%
T1: $Y_{1}\leftrightarrow Y_{2}$ $\rightsquigarrow$ $Y_{1}\land Y_{2}$	1	100.00%	1	100.00%
Unexplained by ALL hand errors	0	0.00%	0	0.00%
Transformations combined	0	0.00%	0	0.00%
Whole file	1	100.00%	1	100.00%

Appendix

Table 6. Hand-picked errors and transformations for linguistic operator(s) AND_NOT_NOT

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_DISJUNCTION_BY_CONJUNCTION	4	80.00%	1	50.00%
Transformations combined	4	100.00%	1	100.00%
T1: $Y_{1}\lor\lnot Y_{2}$ $\rightsquigarrow$ $Y_{1}\land\lnot Y_{2}$	4	100.00%	1	100.00%
BUGGY_REPLACE_IMPLICATION_BY_CONJUNCTION	1	20.00%	1	50.00%
Transformations combined	1	100.00%	1	100.00%
T2: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{1}\land Y_{2}$	1	100.00%	1	100.00%

Appendix

Table 7. Hand-picked errors and transformations for linguistic operator(s) AND_NOT

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_DISJUNCTION_BY_CONJUNCTION	50	78.12%	6	40.00%
Transformations combined	50	100.00%	6	100.00%
$\lceil$ $=$ T1: $Y_{1}\lor\ldots\lor Y_{n}$ $\rightsquigarrow$ $Y_{1}\land\ldots\land Y_{n}$	50	100.00%	6	100.00%
$\lfloor$ $=$ T2: $Y_{1}\lor Y_{2}$ $\rightsquigarrow$ $Y_{1}\land Y_{2}$	50	100.00%	6	100.00%
BUGGY_REPLACE_IMPLICATION_BY_CONJUNCTION	7	10.94%	3	20.00%
Transformations combined	7	100.00%	3	100.00%

Appendix

Table 8. Hand-picked errors and transformations for linguistic operator(s) AND

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_NAND_BY_DISJUNCTION	2	40.00%	1	25.00%
Transformations combined	2	100.00%	1	100.00%
T1: $\lnot\left(Y_{1}\land Y_{2}\land Y_{3}\land Y_{4}\right)$ $\rightsquigarrow$ $Y_{1}\lor Y_{2}\lor Y_{3}\lor Y_{4}$	2	100.00%	1	100.00%
BUGGY_ADD_OPERAND_TO_DISJUNCTION	1	20.00%	1	25.00%
Transformations combined	0	0.00%	0	0.00%
Unexplained by ALL hand errors	2	40.00%	2	50.00%

Appendix

Table 9. Hand-picked errors and transformations for linguistic operator(s) AT_LEAST_NEUTRAL

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_DISJUNCTION_BY_CONJUNCTION	3	1.46%	2	1.61%
Transformations combined	3	100.00%	2	100.00%
T10: $\left(Y_{1}\land Y_{2}\right)\lor\left(Y_{1}\land Y_{3}\right)\lor Y_{2}\lor Y_{3}$ $\rightsquigarrow$ $\left(Y_{1}\land Y_{2}\right)\lor\left(Y_{1}\land Y_{3}\right)\lor\left(Y_{2}\land Y_{3}\right)$	3	100.00%	2	100.00%
BUGGY_REPLACE_CONJUNCTION_BY_DISJUNCTION	2	0.97%	2	1.61%
Transformations combined	0	0.00%	0	0.00%
BUGGY_REPLACE_NAND_BY_DISJUNCTION	2	0.97%	2	1.61%

Appendix

Table 10. Hand-picked errors and transformations for linguistic operator(s) AT_LEAST

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_DISJUNCTION_BY_XOR	91	50.00%	2	6.25%
Transformations combined	91	100.00%	2	100.00%
T1: $Y_{1}\lor Y_{2}$ $\rightsquigarrow$ $\lnot\left(Y_{1}\leftrightarrow Y_{2}\right)$	91	100.00%	2	100.00%
BUGGY_REPLACE_EQUIVALENCE_BY_XOR	21	11.54%	1	3.12%
Transformations combined	21	100.00%	1	100.00%
T2: $Y_{1}\leftrightarrow Y_{2}$ $\rightsquigarrow$ $\lnot\left(Y_{1}\leftrightarrow Y_{2}\right)$	21	100.00%	1	100.00%

Appendix

Table 11. Hand-picked errors and transformations for linguistic operator(s) EITHER_OR

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REMOVE_NEGATION	9	37.50%	3	21.43%
Transformations combined	9	100.00%	3	100.00%
T4: $Y_{1}\rightarrow\lnot Y_{2}$ $\rightsquigarrow$ $Y_{1}\rightarrow Y_{2}$	9	100.00%	3	100.00%
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	4	16.67%	2	14.29%
Transformations combined	4	100.00%	2	100.00%
T1: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	4	100.00%	2	100.00%

Appendix

Table 12. Hand-picked errors and transformations for linguistic operator(s) IF_AND_NEUTRAL

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_ADD_NEGATION	24	33.80%	6	15.00%
Transformations combined	24	100.00%	6	100.00%
T1: $\left(Y_{1}\land Y_{2}\right)\rightarrow Y_{3}$ $\rightsquigarrow$ $\left(Y_{1}\land Y_{2}\right)\rightarrow\lnot Y_{3}$	24	100.00%	6	100.00%
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	12	16.90%	5	12.50%
Transformations combined	12	100.00%	5	100.00%
T2: $\lnot Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow\lnot Y_{1}$	12	100.00%	5	100.00%

Appendix

Table 13. Hand-picked errors and transformations for linguistic operator(s) IF_AND_NOT

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_EQUIVALENCE	628	88.20%	39	52.00%
Transformations combined	626	99.68%	38	97.44%
T1: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\leftrightarrow Y_{1}$	344	54.78%	20	51.28%
T2: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{1}\leftrightarrow Y_{2}$	282	44.90%	18	46.15%
BUGGY_REPLACE_CONJUNCTION_BY_EQUIVALENCE	35	4.92%	10	13.33%
Transformations combined	35	100.00%	10	100.00%

Appendix

Table 14. Hand-picked errors and transformations for linguistic operator(s) IF_AND_ONLY_IF

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	6	25.00%	3	17.65%
Transformations combined	6	100.00%	3	100.00%
T1: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	6	100.00%	3	100.00%
BUGGY_REPLACE_EQUIVALENCE_BY_REVERSED_IMPLICATION	2	8.33%	2	11.76%
Transformations combined	2	100.00%	2	100.00%
T2: $Y_{1}\leftrightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	2	100.00%	2	100.00%

Appendix

Table 15. Hand-picked errors and transformations for linguistic operator(s) IF_AND

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_CONJUNCTION_BY_IMPLICATION	40	29.63%	2	4.00%
Transformations combined	40	100.00%	2	100.00%
T1: $Y_{1}\land Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	40	100.00%	2	100.00%
T16: $\left(Y_{1}\rightarrow Y_{2}\right)\land Y_{3}$ $\rightsquigarrow$ $Y_{3}\rightarrow\left(Y_{1}\rightarrow Y_{2}\right)$	13	32.50%	1	50.00%
BUGGY_REPLACE_EQUIVALENCE_BY_IMPLICATION	10	7.41%	2	4.00%
Transformations combined	9	90.00%	1	50.00%

Appendix

Table 16. Hand-picked errors and transformations for linguistic operator(s) IF_IF

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_DISJUNCTION_BY_CONJUNCTION	73	20.05%	3	3.16%
Transformations combined	73	100.00%	3	100.00%
$\lceil$ $=$ T1: $\lnot\left(Y_{1}\land Y_{2}\right)\rightarrow\lnot Y_{3}$ $\rightsquigarrow$ $\left(\lnot Y_{1}\land\lnot Y_{2}\right)\rightarrow\lnot Y_{3}$	65	89.04%	2	66.67%
$\|$ $=$ T2: $\lnot\left(Y_{1}\land Y_{2}\right)$ $\rightsquigarrow$ $\lnot Y_{1}\land\lnot Y_{2}$	65	89.04%	2	66.67%
$\|$ $=$ T3: $\lnot\left(Y_{1}\land Y_{2}\right)\rightarrow Y_{3}$ $\rightsquigarrow$ $\left(\lnot Y_{1}\land\lnot Y_{2}\right)\rightarrow Y_{3}$	65	89.04%	2	66.67%
$\lfloor$ $=$ T28: $\lnot\left(Y_{1}\land\ldots\land Y_{n}\right)$ $\rightsquigarrow$ $\lnot Y_{1}\land\ldots\land\lnot Y_{n}$	65	89.04%	2	66.67%

Appendix

Table 17. Hand-picked errors and transformations for linguistic operator(s) IF_NEITHER_NOR_NOT

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_DISJUNCTION_BY_CONJUNCTION	37	45.12%	2	6.25%
Transformations combined	37	100.00%	2	100.00%
T1: $\lnot\left(Y_{1}\land Y_{2}\right)\rightarrow Y_{3}$ $\rightsquigarrow$ $\left(\lnot Y_{1}\land\lnot Y_{2}\right)\rightarrow Y_{3}$	28	75.68%	1	50.00%
T2: $\left(\lnot Y_{1}\lor\lnot Y_{2}\right)\rightarrow Y_{3}$ $\rightsquigarrow$ $\left(\lnot Y_{1}\land\lnot Y_{2}\right)\rightarrow Y_{3}$	9	24.32%	1	50.00%
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	5	6.10%	2	6.25%
Transformations combined	5	100.00%	2	100.00%

Appendix

Table 18. Hand-picked errors and transformations for linguistic operator(s) IF_NEITHER_NOR

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_ADD_NEGATION	16	39.02%	2	12.50%
Transformations combined	16	100.00%	2	100.00%
T1: $\left(Y_{1}\land Y_{2}\right)\rightarrow Y_{3}$ $\rightsquigarrow$ $\left(Y_{2}\land\lnot Y_{1}\right)\rightarrow Y_{3}$	15	93.75%	1	50.00%
T11: $Y_{1}\lor\lnot\left(Y_{2}\land Y_{3}\right)$ $\rightsquigarrow$ $\left(Y_{3}\land\lnot Y_{2}\right)\rightarrow Y_{1}$	1	6.25%	1	50.00%
BUGGY_REMOVE_NEGATION	6	14.63%	2	12.50%
Transformations combined	6	100.00%	2	100.00%

Appendix

Table 19. Hand-picked errors and transformations for linguistic operator(s) IF_NOT_AND

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	137	47.24%	1	4.35%
Transformations combined	137	100.00%	1	100.00%
T1: $\lnot Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow\lnot Y_{1}$	137	100.00%	1	100.00%
BUGGY_REPLACE_EQUIVALENCE_BY_REVERSED_IMPLICATION	54	18.62%	2	8.70%
Transformations combined	54	100.00%	2	100.00%
T2: $\lnot Y_{1}\leftrightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow\lnot Y_{1}$	49	90.74%	1	50.00%

Appendix

Table 20. Hand-picked errors and transformations for linguistic operator(s) IF_NOT

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_CONJUNCTION_BY_DISJUNCTION	6	17.65%	2	9.09%
Transformations combined	6	100.00%	2	100.00%
T1: $Y_{1}\rightarrow\left(Y_{2}\land Y_{3}\right)$ $\rightsquigarrow$ $Y_{1}\rightarrow\left(Y_{2}\lor Y_{3}\right)$	6	100.00%	2	100.00%
BUGGY_REMOVE_NEGATION	5	14.71%	2	9.09%
Transformations combined	5	100.00%	2	100.00%
T2: $Y_{1}\rightarrow\left(Y_{2}\lor\lnot Y_{3}\right)$ $\rightsquigarrow$ $Y_{1}\rightarrow\left(Y_{2}\lor Y_{3}\right)$	4	80.00%	1	50.00%

Appendix

Table 21. Hand-picked errors and transformations for linguistic operator(s) IF_OR_NEUTRAL

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	18	52.94%	4	23.53%
Transformations combined	18	100.00%	4	100.00%
$\lceil$ $=$ T1: $\left(Y_{1}\lor Y_{2}\right)\rightarrow Y_{3}$ $\rightsquigarrow$ $Y_{3}\rightarrow\left(Y_{1}\lor Y_{2}\right)$	18	100.00%	4	100.00%
$\lfloor$ $=$ T9: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	18	100.00%	4	100.00%
BUGGY_REPLACE_CONJUNCTION_BY_DISJUNCTION	3	8.82%	2	11.76%
Transformations combined	3	100.00%	2	100.00%

Appendix

Table 22. Hand-picked errors and transformations for linguistic operator(s) IF_OR

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	61	58.65%	4	19.05%
Transformations combined	61	100.00%	4	100.00%
T1: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	61	100.00%	4	100.00%
BUGGY_REPLACE_EQUIVALENCE_BY_IMPLICATION	20	19.23%	4	19.05%
Transformations combined	20	100.00%	4	100.00%
T2: $Y_{1}\leftrightarrow Y_{2}$ $\rightsquigarrow$ $Y_{1}\rightarrow Y_{2}$	20	100.00%	4	100.00%

Appendix

Table 23. Hand-picked errors and transformations for linguistic operator(s) IF

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_DISJUNCTION_BY_CONJUNCTION	25	37.31%	7	23.33%
Transformations combined	25	100.00%	7	100.00%
T1: $\lnot\left(Y_{1}\land\ldots\land Y_{n}\right)$ $\rightsquigarrow$ $\lnot Y_{1}\land\ldots\land\lnot Y_{n}$	14	56.00%	2	28.57%
$\lceil$ $=$ T2: $Y_{1}\lor\ldots\lor Y_{n}$ $\rightsquigarrow$ $Y_{1}\land\ldots\land Y_{n}$	7	28.00%	2	28.57%
$\lfloor$ $=$ T7: $\lnot Y_{1}\lor\ldots\lor\lnot Y_{n}$ $\rightsquigarrow$ $\lnot Y_{1}\land\ldots\land\lnot Y_{n}$	7	28.00%	2	28.57%
T11: $\lnot Y_{1}\lor\lnot Y_{2}\lor\lnot Y_{3}$ $\rightsquigarrow$ $\lnot Y_{1}\land\lnot Y_{3}\land\lnot Y_{2}$	2	8.00%	1	14.29%

Appendix

Table 24. Hand-picked errors and transformations for linguistic operator(s) NEITHER_NOR

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REMOVE_NEGATION	52	100.00%	2	100.00%
Transformations combined	52	100.00%	2	100.00%
T1: $\lnot Y_{1}$ $\rightsquigarrow$ $Y_{1}$	52	100.00%	2	100.00%
Unexplained by ALL hand errors	0	0.00%	0	0.00%
Transformations combined	0	0.00%	0	0.00%
Whole file	52	100.00%	2	100.00%

Appendix

Table 25. Hand-picked errors and transformations for linguistic operator(s) NEUTRAL

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_DISJUNCTION_BY_CONJUNCTION	36	78.26%	7	46.67%
Transformations combined	36	100.00%	7	100.00%
T1: $\lnot\left(Y_{1}\land\ldots\land Y_{n}\right)$ $\rightsquigarrow$ $\lnot Y_{1}\land\ldots\land\lnot Y_{n}$	34	94.44%	5	71.43%
T7: $Y_{1}\lor\ldots\lor Y_{n}$ $\rightsquigarrow$ $Y_{1}\land\ldots\land Y_{n}$	2	5.56%	2	28.57%
BUGGY_REPLACE_NOR_BY_CONJUNCTION	4	8.70%	2	13.33%
Transformations combined	4	100.00%	2	100.00%

Appendix

Table 26. Hand-picked errors and transformations for linguistic operator(s) NONE_OF

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_PULL_NEGATION_CONJUNCTION	67	12.48%	13	7.78%
Transformations combined	67	100.00%	13	100.00%
$\lceil$ $=$ T3: $\lnot Y_{1}\land\lnot Y_{2}$ $\rightsquigarrow$ $\lnot\left(Y_{1}\land Y_{2}\right)$	61	91.04%	12	92.31%
$\lfloor$ $=$ T4: $\lnot Y_{1}\land\ldots\land\lnot Y_{n}$ $\rightsquigarrow$ $\lnot\left(Y_{1}\land\ldots\land Y_{n}\right)$	61	91.04%	12	92.31%
T17: $\lnot Y_{1}\land\ldots\land\lnot Y_{n}$ $\rightsquigarrow$ $\lnot\left(Y_{1}\land\ldots\land Y_{n}\right)$	6	8.96%	1	7.69%
BUGGY_REPLACE_XOR_BY_NAND	44	8.19%	5	2.99%

Appendix

Table 27. Hand-picked errors and transformations for linguistic operator(s) NOT_ALL_OF

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_ADD_NEGATION	1	100.00%	1	100.00%
Transformations combined	1	100.00%	1	100.00%
T1: $Y_{1}$ $\rightsquigarrow$ $\lnot Y_{1}$	1	100.00%	1	100.00%
Unexplained by ALL hand errors	0	0.00%	0	0.00%
Transformations combined	0	0.00%	0	0.00%
Whole file	1	100.00%	1	100.00%

Appendix

Table 28. Hand-picked errors and transformations for linguistic operator(s) NOT

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	446	50.06%	5	5.75%
Transformations combined	443	99.33%	3	60.00%
$\lceil$ $=$ T1: $\left(Y_{1}\land\ldots\land Y_{n}\right)\rightarrow Y_{3}$ $\rightsquigarrow$ $Y_{3}\rightarrow\left(Y_{1}\land\ldots\land Y_{n}\right)$	443	99.33%	3	60.00%
$\|$ $=$ T2: $\left(Y_{1}\land Y_{2}\right)\rightarrow Y_{3}$ $\rightsquigarrow$ $Y_{3}\rightarrow\left(Y_{1}\land Y_{2}\right)$	443	99.33%	3	60.00%
$\lfloor$ $=$ T3: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	443	99.33%	3	60.00%
BUGGY_REPLACE_EQUIVALENCE_BY_REVERSED_IMPLICATION	202	22.67%	4	4.60%

Appendix

Table 29. Hand-picked errors and transformations for linguistic operator(s) ONLY_IF_AND

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	277	45.26%	20	10.10%
Transformations combined	274	98.92%	18	90.00%
T3: $\left(Y_{1}\lor Y_{2}\right)\rightarrow Y_{3}$ $\rightsquigarrow$ $Y_{3}\rightarrow\left(Y_{1}\lor Y_{2}\right)$	239	86.28%	4	20.00%
T7: $\left(Y_{1}\lor Y_{2}\lor Y_{3}\right)\rightarrow Y_{4}$ $\rightsquigarrow$ $Y_{4}\rightarrow\left(Y_{1}\lor Y_{2}\lor Y_{3}\right)$	35	12.64%	14	70.00%
BUGGY_REPLACE_EQUIVALENCE_BY_REVERSED_IMPLICATION	33	5.39%	4	2.02%
Transformations combined	33	100.00%	4	100.00%

Appendix

Table 30. Hand-picked errors and transformations for linguistic operator(s) ONLY_IF_AT_LEAST

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	138	28.34%	33	15.21%
Transformations combined	138	100.00%	33	100.00%
T1: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	117	84.78%	20	60.61%
T4: $Y_{1}\rightarrow\lnot\left(Y_{2}\lor Y_{3}\right)$ $\rightsquigarrow$ $\left(\lnot Y_{2}\land\lnot Y_{3}\right)\rightarrow Y_{1}$	21	15.22%	13	39.39%
BUGGY_REPLACE_DISJUNCTION_BY_CONJUNCTION	47	9.65%	17	7.83%
Transformations combined	47	100.00%	17	100.00%

Appendix

Table 31. Hand-picked errors and transformations for linguistic operator(s) ONLY_IF_NEITHER_NOR

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	19	32.76%	2	11.11%
Transformations combined	19	100.00%	2	100.00%
T1: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	11	57.89%	1	50.00%
T7: $\lnot\left(Y_{1}\lor Y_{2}\right)\rightarrow\lnot Y_{3}$ $\rightsquigarrow$ $\lnot Y_{3}\rightarrow\left(\lnot Y_{1}\land\lnot Y_{2}\right)$	8	42.11%	1	50.00%
BUGGY_ADD_NEGATION	3	5.17%	2	11.11%
Transformations combined	3	100.00%	2	100.00%

Appendix

Table 32. Hand-picked errors and transformations for linguistic operator(s) ONLY_IF_NOT_AND_NOT_NOT

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	50	30.12%	10	12.05%
Transformations combined	50	100.00%	10	100.00%
T1: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	44	88.00%	7	70.00%
T11: $\lnot\left(Y_{1}\land Y_{2}\right)\rightarrow\lnot Y_{3}$ $\rightsquigarrow$ $\lnot Y_{3}\rightarrow\left(\lnot Y_{1}\lor\lnot Y_{2}\right)$	6	12.00%	3	30.00%
BUGGY_ADD_NEGATION	21	12.65%	10	12.05%
Transformations combined	21	100.00%	10	100.00%

Appendix

Table 33. Hand-picked errors and transformations for linguistic operator(s) ONLY_IF_NOT_OR_NOT_NOT

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	164	35.12%	1	3.70%
Transformations combined	164	100.00%	1	100.00%
T1: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	164	100.00%	1	100.00%
BUGGY_REPLACE_EQUIVALENCE_BY_IMPLICATION	156	33.40%	2	7.41%
Transformations combined	152	97.44%	1	50.00%
T2: $Y_{1}\leftrightarrow Y_{2}$ $\rightsquigarrow$ $Y_{1}\rightarrow Y_{2}$	152	97.44%	1	50.00%

Appendix

Table 34. Hand-picked errors and transformations for linguistic operator(s) ONLY_IF

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_EQUIVALENCE_BY_CONJUNCTION	5	13.89%	1	4.55%
Transformations combined	5	100.00%	1	100.00%
T1: $\left(Y_{1}\lor Y_{2}\right)\leftrightarrow\left(Y_{3}\lor Y_{4}\right)$ $\rightsquigarrow$ $\left(Y_{3}\lor Y_{4}\right)\land\left(Y_{1}\lor Y_{2}\right)$	5	100.00%	1	100.00%
BUGGY_REPLACE_CONJUNCTION_BY_DISJUNCTION	4	11.11%	1	4.55%
Transformations combined	4	100.00%	1	100.00%
T7: $Y_{1}\land Y_{2}\land Y_{3}$ $\rightsquigarrow$ $Y_{1}\land\left(Y_{2}\lor Y_{3}\right)$	4	100.00%	1	100.00%

Appendix

Table 35. Hand-picked errors and transformations for linguistic operator(s) OR_AND_OR

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_CONJUNCTION_BY_DISJUNCTION	30	83.33%	5	62.50%
Transformations combined	30	100.00%	5	100.00%
T3: $\left(Y_{1}\lor Y_{2}\right)\land Y_{3}$ $\rightsquigarrow$ $Y_{1}\lor Y_{3}\lor Y_{2}$	15	50.00%	1	20.00%
T4: $\left(Y_{1}\land Y_{2}\right)\lor Y_{3}$ $\rightsquigarrow$ $Y_{3}\lor Y_{1}\lor Y_{2}$	9	30.00%	2	40.00%
T1: $Y_{1}\land\ldots\land Y_{n}$ $\rightsquigarrow$ $Y_{1}\lor\ldots\lor Y_{n}$	5	16.67%	1	20.00%
T2: $Y_{1}\land\ldots\land Y_{n}$ $\rightsquigarrow$ $Y_{1}\lor\ldots\lor Y_{n}$	1	3.33%	1	20.00%

Appendix

Table 36. Hand-picked errors and transformations for linguistic operator(s) OR

Hand-picked mistake / Transformations	#	%	#dst	%dst
BUGGY_REPLACE_IMPLICATION_BY_REVERSED_IMPLICATION	1418	23.22%	105	6.68%
Transformations combined	1414	99.72%	102	97.14%
T169: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	1343	94.71%	80	76.19%
T184: $Y_{1}\rightarrow\left(Y_{2}\land Y_{3}\right)$ $\rightsquigarrow$ $\left(Y_{2}\land Y_{3}\right)\rightarrow Y_{1}$	172	12.13%	37	35.24%
T210: $Y_{1}\rightarrow\lnot\left(Y_{2}\lor Y_{3}\right)$ $\rightsquigarrow$ $\left(\lnot Y_{2}\land\lnot Y_{3}\right)\rightarrow Y_{1}$	55	3.88%	17	16.19%
T293: $\lnot\left(Y_{1}\lor Y_{2}\right)\rightarrow\lnot Y_{3}$ $\rightsquigarrow$ $\lnot Y_{3}\rightarrow\left(\lnot Y_{1}\land\lnot Y_{2}\right)$	8	0.56%	1	0.95%

Appendix

Table 37. Hand-picked errors and transformations for the ungrouped propositional data set.

Transformations	#	%	#dst	%dst
Whole data set	6106	100.00%	1572	100.00%
ANY transformation explained	5156	84.44%	955	60.75%
Cluster 1
T169: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{2}\rightarrow Y_{1}$	1343	21.99%	80	5.09%
T184: $Y_{1}\rightarrow\left(Y_{2}\land Y_{3}\right)$ $\rightsquigarrow$ $\left(Y_{2}\land Y_{3}\right)\rightarrow Y_{1}$	172	2.82%	37	2.35%
Cluster 2

Appendix

Table 38. Transformation hierarchy clusters

Transformations	#	%	#dst	%dst
Whole data set	12482	100.00%	3951	100.00%
ANY transformation explained	9910	79.39%	2333	59.05%
Cluster 1
T960: $Y_{1}\rightarrow Y_{2}$ $\rightsquigarrow$ $Y_{1}\rightarrow\left(\Diamond\left(\lnot Y_{1}\right)\lor\Diamond\left(\Diamond\left(\lnot Y_{1}\right)\right)\right)$	707	5.66%	162	4.10%
T1035: $Y_{1}\rightarrow\Diamond\left(\lnot Y_{1}\right)$ $\rightsquigarrow$ $Y_{1}\rightarrow\left(\Diamond\left(\lnot Y_{1}\right)\lor\Diamond\left(\Diamond\left(\lnot Y_{1}\right)\right)\right)$	32	0.26%	1	0.03%
T1758: $\Diamond\left(\lnot Y_{1}\right)\lor\Box\left(\lnot Y_{1}\right)$ $\rightsquigarrow$ $\Diamond\left(\lnot Y_{1}\right)\lor\Diamond\left(\Diamond\left(\lnot Y_{1}\right)\right)$	16	0.13%	1	0.03%

Appendix

Table 39. Transformation hierarchy clusters

Transformations	#	%	#dst	%dst
Whole data set	7210	100.00%	2695	100.00%
ANY transformation explained	2588	35.89%	409	15.18%
Cluster 1
T2: $Y_{1}$ $\rightsquigarrow$ $\mathbf{A}\mathbf{G}Y_{1}$	215	2.98%	9	0.33%
T304: $\lnot Y_{1}$ $\rightsquigarrow$ $\mathbf{A}\mathbf{G}\left(\lnot Y_{1}\right)$	36	0.50%	2	0.07%
T222: $\mathbf{E}\mathbf{F}\left(\lnot Y_{1}\right)$ $\rightsquigarrow$ $\mathbf{E}\mathbf{F}\left(\mathbf{A}\mathbf{G}\left(\lnot Y_{1}\right)\right)$	28	0.39%	1	0.04%

Appendix

Table 40. Transformation hierarchy clusters

Educational Data Mining

Educational Data Mining studies computational methods for analyzing learning data to understand learners, improve instruction, and support educational decisions.

See all 74 papers

Can LLMs Model Incorrect Student Reasoning? A Case Study on Distractor Generation
Mar 16, 2026Yanick Zengaffinen, Andreas Opedal, Donya Rooein +3LLM Prompting Generative AI in Education
1ETH Zürich · 2Bocconi University · University of Central Florida
Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification
Sep 14, 2026Runa Yoshida, Kosuke Nishida, Kyosuke NishidaLLM Prompting LLM Mathematical Reasoning
Human Informatics Labs., NTT, Inc.
Correct Diagnosis, Better Feedback: A Symbolic-Verifier for Faithful LLM Tutoring Feedback in Logic Proofs
Sep 18, 2026Tahreem Yasir, Arnav Mody, Xioayi Tian +1Intelligent Tutoring Systems Language Model Error Detection
North Carolina State University · Purdue University · Kennesaw State University

Mar 16, 2026J/K move · Enter open · S save

Can LLMs Model Incorrect Student Reasoning? A Case Study on Distractor Generation

Yanick Zengaffinen, Andreas Opedal, Donya Rooein, Kv Aditya Srivatsa+2

1ETH Zürich · 2Bocconi University · University of Central Florida

LLM Prompting Generative AI in Education User Modeling Text Generation LLM Reasoning

Open paper AskPDF arXiv

Modeling student misconceptions in a realistic manner is critical for AI in education. In this work, we examine how large language models (LLMs) reason about misconceptions when generating distractor answers for multiple-choice questions (MCQs), a task that requires producing answers that are incorrect, yet plausible. We introduce a taxonomy over reasoning strategies for distractor generation that is grounded in learning-science literature and empirical observation, which we apply to LLM-generated reasoning traces across math and science MCQs. On the math dataset, we find that models follow a misconception-based process with potentially high diagnostic value: they recover the correct solution, articulate student errors, simulate them, and select plausible candidates. On the science dataset, on the other hand, they tend to follow a less robust approach based on semantic similarity to the correct answer. We find the most frequent failure modes to be that the model is unable to generate a correct solution or that it discards plausible distractor candidates when performing selection. Providing the correct solution in the prompt yields a relative improvement of 6.4% in alignment with human-authored distractors, highlighting the critical role of anchoring distractor generation to the correct solution. Together, our findings offer an interpretable view of how LLMs model incorrect student reasoning.

Abstract

Educational Data Mining

Explore similar work

Can LLMs Model Incorrect Student Reasoning? A Case Study on Distractor Generation

Improving Mathematical Reasoning Capabilities in Large Language Models via Reasoning Process Error Classification

Correct Diagnosis, Better Feedback: A Symbolic-Verifier for Faithful LLM Tutoring Feedback in Logic Proofs