cs.CYMar 13, 2026

Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessments

Authors: Arne Vanhoyweghen, Vincent Holst, Melika Mobini, Lukas Van de Voorde, Tibo Vanleke, Bert Verbruggen, Brecht Verbeken, Andres Algaba, +4 more

Organizations: Data Analytics Lab, Vrije Universiteit Brussel, 1050 Brussel, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Providing timely and individualised feedback on handwritten student work is highly beneficial for learning but difficult to achieve at scale. This challenge has become more pressing as generative AI undermines the reliability of take-home assessments, shifting emphasis toward supervised, in-class evaluation. We present a scalable, end-to-end workflow for LLM-assisted grading of short, pen-and-paper assessments. The workflow spans (1) constructing solution keys, (2) developing detailed rubric-style grading keys used to guide the LLM, and (3) a grading procedure that combines automated scanning and anonymisation, multi-pass LLM scoring, automated consistency checks, and mandatory human verification. We deploy the system in two undergraduate mathematics courses using six low-stakes in-class tests. Empirically, LLM assistance reduces grading time by approximately 23% while achieving agreement comparable to, and in several cases tighter than, fully manual grading. Occasional model errors occur but are effectively contained by the hybrid design. Overall, our results show that carefully embedded human-in-the-loop LLM grading can substantially reduce workload while maintaining fairness and accuracy.

Figures & tables

Figure 1: Example grading key highlighting two design issues: assigning too many points to a single step and using imprecise terminology (“factorise”), which can cause the model to award credit for mathematically irrelevant operations.
QuestionD/MD/M
Bonus 3 – Q2B0.7051.323
Bonus 4 – Q2A0.9981.165
Bonus 5 – Q10.9070.696
Bonus 3 – Q2A0.5960.632
Bonus 4 – Q10.5770.675
Bonus 5 – Q2A0.6290.655
Table 1: Paired ratios of digital to manual grading time (D/M) for each question. Each row corresponds to one question graded by two annotators in opposite order: one graded manually first (M →\rightarrow D, shown in bold ), the other digitally first (D →\rightarrow M). Ratios below 1 indicate faster digital grading.
Figure 2: Quadratically weighted Cohen’s κ\kappa values comparing grading agreement across media, tests, and question types. A1 and A2 denote the two human annotators (e.g., A1 vs A2 indicates inter-annotator agreement). Colours indicate the bonus test, while marker shapes denote the question type. Across all but one question–annotator pairing, annotator–LLM agreement in the digital condition is comparable to, and in several cases higher than, inter-annotator agreement under manual grading.
QuestionManual: A1 vs A2Digital: A1 vs GPTDigital: A2 vs GPT
Bonus 3 – Q2A0.710.79 (0.71)0.50 (0.43)
Bonus 3 – Q2B0.700.87 (0.85)0.87 (0.85)
Bonus 4 – Q10.960.87 (0.94)0.91 (0.91)
Bonus 4 – Q2A0.890.89 (0.85)0.87 (0.86)
Bonus 5 – Q10.840.91 (0.89)0.89 (0.93)
Bonus 5 – Q2A0.820.90 (0.88)0.90 (0.88)
Table 2: Quadratically weighted Cohen’s κ\kappa values compare inter-annotator agreement under manual grading with annotator–LLM agreement under digital grading, shown for each question instance. In the teaching workflow, we used the maximum of five independent LLM evaluations as the provisional score (a conservative choice in the student’s favour). For analysis, we also report results using the median of five evaluations, which is less sensitive to outliers. In this table, the primary κ\kappa values (without parentheses) use the median of five LLM evaluations as the provisional grade, while the values in parentheses use the maximum LLM score instead. Both aggregation strategies yield similar agreement levels, with median-based aggregation typically producing slightly higher κ\kappa values. The low alignment on Bonus 3–Q2A was driven by differing interpretations of the grading key: both annotators applied an additional grading step that was not explicitly specified, underscoring the need for a detailed and comprehensive grading key.
Figure 3: Distributions of absolute score deviations for all grader pairings under manual and digital grading. The top row shows manual grading comparisons (human vs. human, human vs. GPT using the median assigned score), while the bottom row shows digital grading comparisons. Dashed vertical lines indicate median deviations and solid lines indicate mean deviations. In the manual setting, mean and median deviations coincide for both human vs. human and human vs. GPT comparisons, indicating similar grading behaviour. In the digital setting, median deviations are lower than the mean for both human vs. human and human vs. GPT comparisons, consistent with an anchoring effect in which the LLM grade acts as a stabilising reference that reduces typical disagreement while leaving mean deviation largely unchanged.
Figure 4: Relative positioning of LLM-assigned grades with respect to the two human annotators. Bars indicate the proportion of responses for which the LLM score lies below both human scores, between the two human scores, is equal to both, or exceeds both. Error bars report mean ±\pm standard deviation of absolute distance to the nearest human boundary. Across grading media, the LLM most frequently assigns intermediate or matching scores, with deviations outside the human range occurring less often.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Bonus testQuestionAnnotatorOrderTime (mm:ss)
3Q2BLM →\rightarrow D13:27
3Q2BLD →\rightarrow M9:29
3Q2BVD →\rightarrow M12:41
3Q2BVM →\rightarrow D16:47
4Q2ATM →\rightarrow D20:53
4Q2ATD →\rightarrow M20:51
Appendix
Table 3: Raw grading times for manual and LLM-assisted digital grading. Each row corresponds to a single annotator grading 30 responses for one question. The grading order of the mediums is by D →\rightarrow M and M →\rightarrow D whether manual grading (M) or digital grading (D) was performed first.
Figure 5: Relative positioning of LLM-assigned grades with respect to the two human annotators when the maximum of five LLM evaluations is used as the provisional grade. Compared to median aggregation (main text), maximum-based aggregation produces more positive deviations outside the human score range.
Figure 6: Ordered comparison of five LLM scores with the minimum and maximum human grades per submission, illustrating tight LLM score concentration within the human grading range.
Figure 7: OCR template used for in-class bonus tests. The structured layout enables reliable extraction of student identifiers and automated isolation of answer regions, ensuring that only anonymised handwritten responses are passed to the LLM for grading.
Figure 11
Figure 12
Figure 8: Illustrative graded examples for the same bonus-test question, showing three performance levels: incorrect (0/10), partially correct (3/10), and fully correct (10/10). All responses and student identifiers are synthetic and were created by the authors for demonstration purposes. In each case, the structured LLM-generated explanation follows the grading key and makes explicit how points are allocated across intermediate reasoning steps prior to human verification.

Explore similar work

CardsList

  1. Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs

    May 18, 2026Jacob Levine, Miguel Aenlle, Craig Zilles +2VLM EvaluationRubric-Based Evaluation

    University of Illinois Urbana-Champaign

  2. Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark

    Mar 1, 2026Zhiqi Yu, Xingping Liu, Haobin Mao +4Automated GradingGenerative AI in Education

    UC Irvine Irvine, USA · Independent Shanghai, China · UC Berkeley Berkeley, USA

  3. LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability

    Jun 1, 2026Aastha Sapkota, M. G. Sarwar MurshedLLM EvaluationLLM-as-a-Judge

    Computer Science Department University of Wisconsin - Green Bay, Green Bay, WI, USA