Human-in-the-Loop LLM Grading for Handwritten Mathematics Assessments
Authors: Arne Vanhoyweghen, Vincent Holst, Melika Mobini, Lukas Van de Voorde, Tibo Vanleke, Bert Verbruggen, Brecht Verbeken, Andres Algaba, +4 more
Organizations: Data Analytics Lab, Vrije Universiteit Brussel, 1050 Brussel, Belgium · School of Engineering and Applied Sciences, Harvard University, Cambridge, Massachusetts 02138, USA
Providing timely and individualised feedback on handwritten student work is highly beneficial for learning but difficult to achieve at scale. This challenge has become more pressing as generative AI undermines the reliability of take-home assessments, shifting emphasis toward supervised, in-class evaluation. We present a scalable, end-to-end workflow for LLM-assisted grading of short, pen-and-paper assessments. The workflow spans (1) constructing solution keys, (2) developing detailed rubric-style grading keys used to guide the LLM, and (3) a grading procedure that combines automated scanning and anonymisation, multi-pass LLM scoring, automated consistency checks, and mandatory human verification. We deploy the system in two undergraduate mathematics courses using six low-stakes in-class tests. Empirically, LLM assistance reduces grading time by approximately 23% while achieving agreement comparable to, and in several cases tighter than, fully manual grading. Occasional model errors occur but are effectively contained by the hybrid design. Overall, our results show that carefully embedded human-in-the-loop LLM grading can substantially reduce workload while maintaining fairness and accuracy.
Figures & tables
Figure 1: Example grading key highlighting two design issues: assigning too many points to a single step and using imprecise terminology (“factorise”), which can cause the model to award credit for mathematically irrelevant operations.
Question
D/M
D/M
Bonus 3 – Q2B
0.705
1.323
Bonus 4 – Q2A
0.998
1.165
Bonus 5 – Q1
0.907
0.696
Bonus 3 – Q2A
0.596
0.632
Bonus 4 – Q1
0.577
0.675
Bonus 5 – Q2A
0.629
0.655
Table 1: Paired ratios of digital to manual grading time (D/M) for each question. Each row corresponds to one question graded by two annotators in opposite order: one graded manually first (M → D, shown in bold ), the other digitally first (D → M). Ratios below 1 indicate faster digital grading.
Figure 2: Quadratically weighted Cohen’s κ values comparing grading agreement across media, tests, and question types. A1 and A2 denote the two human annotators (e.g., A1 vs A2 indicates inter-annotator agreement). Colours indicate the bonus test, while marker shapes denote the question type. Across all but one question–annotator pairing, annotator–LLM agreement in the digital condition is comparable to, and in several cases higher than, inter-annotator agreement under manual grading.
Question
Manual: A1 vs A2
Digital: A1 vs GPT
Digital: A2 vs GPT
Bonus 3 – Q2A
0.71
0.79 (0.71)
0.50 (0.43)
Bonus 3 – Q2B
0.70
0.87 (0.85)
0.87 (0.85)
Bonus 4 – Q1
0.96
0.87 (0.94)
0.91 (0.91)
Bonus 4 – Q2A
0.89
0.89 (0.85)
0.87 (0.86)
Bonus 5 – Q1
0.84
0.91 (0.89)
0.89 (0.93)
Bonus 5 – Q2A
0.82
0.90 (0.88)
0.90 (0.88)
Table 2: Quadratically weighted Cohen’s κ values compare inter-annotator agreement under manual grading with annotator–LLM agreement under digital grading, shown for each question instance. In the teaching workflow, we used the maximum of five independent LLM evaluations as the provisional score (a conservative choice in the student’s favour). For analysis, we also report results using the median of five evaluations, which is less sensitive to outliers. In this table, the primary κ values (without parentheses) use the median of five LLM evaluations as the provisional grade, while the values in parentheses use the maximum LLM score instead. Both aggregation strategies yield similar agreement levels, with median-based aggregation typically producing slightly higher κ values. The low alignment on Bonus 3–Q2A was driven by differing interpretations of the grading key: both annotators applied an additional grading step that was not explicitly specified, underscoring the need for a detailed and comprehensive grading key.
Figure 3: Distributions of absolute score deviations for all grader pairings under manual and digital grading. The top row shows manual grading comparisons (human vs. human, human vs. GPT using the median assigned score), while the bottom row shows digital grading comparisons. Dashed vertical lines indicate median deviations and solid lines indicate mean deviations. In the manual setting, mean and median deviations coincide for both human vs. human and human vs. GPT comparisons, indicating similar grading behaviour. In the digital setting, median deviations are lower than the mean for both human vs. human and human vs. GPT comparisons, consistent with an anchoring effect in which the LLM grade acts as a stabilising reference that reduces typical disagreement while leaving mean deviation largely unchanged.
Figure 4: Relative positioning of LLM-assigned grades with respect to the two human annotators. Bars indicate the proportion of responses for which the LLM score lies below both human scores, between the two human scores, is equal to both, or exceeds both. Error bars report mean ± standard deviation of absolute distance to the nearest human boundary. Across grading media, the LLM most frequently assigns intermediate or matching scores, with deviations outside the human range occurring less often.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Bonus test
Question
Annotator
Order
Time (mm:ss)
3
Q2B
L
M → D
13:27
3
Q2B
L
D → M
9:29
3
Q2B
V
D → M
12:41
3
Q2B
V
M → D
16:47
4
Q2A
T
M → D
20:53
4
Q2A
T
D → M
20:51
Appendix
Table 3: Raw grading times for manual and LLM-assisted digital grading. Each row corresponds to a single annotator grading 30 responses for one question. The grading order of the mediums is by D → M and M → D whether manual grading (M) or digital grading (D) was performed first.
Figure 5: Relative positioning of LLM-assigned grades with respect to the two human annotators when the maximum of five LLM evaluations is used as the provisional grade. Compared to median aggregation (main text), maximum-based aggregation produces more positive deviations outside the human score range.
Figure 6: Ordered comparison of five LLM scores with the minimum and maximum human grades per submission, illustrating tight LLM score concentration within the human grading range.
Figure 7: OCR template used for in-class bonus tests. The structured layout enables reliable extraction of student identifiers and automated isolation of answer regions, ensuring that only anonymised handwritten responses are passed to the LLM for grading.
Figure 11Figure 12
Figure 8: Illustrative graded examples for the same bonus-test question, showing three performance levels: incorrect (0/10), partially correct (3/10), and fully correct (10/10). All responses and student identifiers are synthetic and were created by the authors for demonstration purposes. In each case, the structured LLM-generated explanation follows the grading key and makes explicit how points are allocated across intermediate reasoning steps prior to human verification.
Automated grading systems have enabled scalable assessment for many response types, but handwritten mathematics remains a barrier due to the complexity of multi-step solutions. Vision-capable large language models (LLMs) offer new opportunities here, yet their reliability in authentic instructional settings remains poorly understood. We present an empirical evaluation of an LLM-based grader for handwritten mathematical work using instructor-defined rubrics. Extending a prior pipeline for typed responses, we integrate transcription and rubric-based evaluation of photographic submissions within a single LLM call, evaluating on student work from two university STEM courses. Comparing AI grading decisions against human-assigned ground truth at the rubric-item level, we observe high overall accuracy, with most errors -- 87% in the best model -- attributable to transcription failures rather than rubric misapplication. We categorize common error modes, including image quality issues, hallucinated content, and incorrect handling of equivalent expressions. These findings highlight both the promise and limitations of LLM-based grading for handwritten mathematics, providing guidance for system design, prompt refinement, and deployment in educational settings.
Grading in large undergraduate STEM courses often yields minimal feedback due to heavy instructional workloads. We present a large-scale empirical study of AI grading on real, handwritten single-variable calculus work from UC Irvine. Using OCR-conditioned large language models with structured, rubric-guided prompting, our system produces scores and formative feedback for thousands of free-response quiz submissions from nearly 800 students. In a setting with no single ground-truth label, we evaluate performance against official teaching-assistant grades, student surveys, and independent human review, finding strong alignment with TA scoring and a large majority of AI-generated feedback rated as correct or acceptable across quizzes. Beyond calculus, this setting highlights core challenges in OCR-conditioned mathematical reasoning and partial-credit assessment. We analyze key failure modes, propose practical rubric- and prompt-design principles, and introduce a multi-perspective evaluation protocol for reliable, real-course deployment. Building on the dataset and evaluation framework developed here, we outline a standardized benchmark for AI grading of handwritten mathematics to support reproducible comparison and future research.
Zhiqi Yu, Xingping Liu, Haobin Mao +4
UC Irvine Irvine, USA · Independent Shanghai, China · UC Berkeley Berkeley, USA
Open-ended mathematics exams are valuable because they assess reasoning, proof construction, algorithmic thinking, and communication of intermediate steps. They are also difficult to grade at scale because instructors must apply partial-credit rubrics consistently while giving feedback that helps students repair misconceptions. This paper evaluates six contemporary large language model (LLM) configurations, Gemini 3.1 Pro Extended, Gemini 3.5 Flash, ChatGPT 5.5 Pro Extended, ChatGPT 5.5 Thinking, Claude Pro Opus 4.7, and Claude Sonnet 4.6, as grading assistants for an undergraduate discrete mathematics examination. The study compares two grading policies. The BASELINE policy uses a stricter rubric-following prompt that emphasizes explicit evidence and complete justification. The LIBERAL policy was added after preliminary grading showed that the baseline condition sometimes applied harsh point deductions and failed to recognize valid partial reasoning. Agreement with human grading is measured at both the question and exam-total levels using mean absolute error, root mean squared error, normalized root mean squared error, Pearson correlation, and exact agreement. The results show that liberal partial-credit prompting reduces average question-level error for every evaluated model family. ChatGPT 5.5 Thinking (LIBERAL) has the lowest average question-level MAE (1.87) and RMSE (2.53), while Gemini 3.1 Pro Extended (LIBERAL) has the lowest total-score MAE (8.00) and RMSE (10.66). However, the strongest total-score Pearson correlation occurs under Gemini 3.1 Pro Extended (BASELINE) at 0.58, showing that point calibration and rank preservation remain distinct goals. We also report practical usability observations.
Aastha Sapkota, M. G. Sarwar Murshed
Computer Science Department University of Wisconsin - Green Bay, Green Bay, WI, USA