Large Scale AI Grading of Handwritten Physics Assessments: Score Agreement and Olympiad Team Selection Outcomes

Praveen Pathak; Siddharth Tiwary; Charudatt Kadolkar; Vijay Singh; David Rakestraw; Shirish Pathare; Anwesh Mazumdar

Dataset	Candidates/ students	Exam submissions	Exam components	Scanned pages	Main role
OE1	350	350	1	5600	Top-cohort identification
OE2	26	130	5	4164	Final team selection
QM	40	40	1	600	University class grade
Total	416	520	7	10 364	–

Table 1: Datasets used in the study. An exam component means one separately graded OE2 theory or experiment test section. An exam submission means one student’s answer booklet for one component. OE1 and QM each had one component per student, while OE2 candidates completed five components.

Model	Effort	Exams/runs included
GPT-5.5 Thinking	High	Main RI/RII grading: OE1, OE2, QM
GPT-5.5 Pro	Standard	OE2 full comparison and outcome check
GPT-5.5 Pro	Extended	OE2 pilot involving selected submissions

Table 2: AI models and effort settings used in the study.

Quantity	What it measures	Purpose
$r$	Pearson correlation between human and AI scores	Score tracking
$\rho$	Spearman rank correlation	Rank-order agreement
$D$	Mean of AI minus human scores. Positive values mean AI awarded more	Direction of over- or under-awarding
MAD	Mean absolute difference between AI and human scores	Size of the grading difference
Top- $k$ overlap	Fraction of human top- $k$ submissions also in the AI top- $k$	Top-group and team-selection checks
$d=\|\mathrm{AI}-\mathrm{Human}\|$	Raw point difference for one official question part	Difference bands for individual question parts

Table 3: Analysis parameters used in the study. Unless raw marks are explicitly stated,

D

and MAD are reported as percentages of the relevant maximum possible score.

Figure 1: AI versus human total scores in RI and RII, expressed as percentages of the assessment maximum. Each point represents one student. The diagonal line is perfect agreement. Blue points show RI and green points show RII.

r

,

\rho

,

D

, and MAD are shown inside each panel.

Figure 2: Residual total-score plots pooled across OE1, OE2 theory, OE2 experiment, and QM. The vertical axis is AI minus human total score, expressed as a percentage of the assessment maximum. Blue circles show RI and green squares show RII. The fitted line is a least-squares visual guide, and the horizontal zero line is perfect agreement. The figure mainly shows the direction and spread of AI–human differences: RI is more often positive, while RII is less shifted upward. The fitted slopes should be read cautiously because the human score is part of the plotted difference.

Figure 3: Outcome agreement. The OE1 and OE2 panels show the percentage of human top-

k

submissions or students also present in the AI top-

k

group; blue circles show RI and green squares show RII. The QM panel shows RII course-grade agreement across the nine released grade categories. Green bubbles show exact grade matches and orange bubbles show different grade pairs. The number inside each bubble is the number of students; AS is the highest grade.

		$d=0$		$0<d\leq 0.5$		$0.5<d\leq 1$		$d>1$
Set	Question parts	RI	RII	RI	RII	RI	RII	RI	RII
OE1	4200	61	71	15	12	9	9	15	8
OE2 theory	1482	70	75	13	13	9	6	8	6
OE2 experiment	416	45	56	21	17	9	13	25	14
QM	960	63	69	21	18	8	9	8	4
All official question parts	7058	63	70	16	14	9	9	13	7

Table 4: Agreement between AI and human scores for individual question parts. Here

d=|\mathrm{AI}-\mathrm{Human}|

in raw points. Values are rounded percentages. Bands sum to 100% before rounding.

Figure 4: AI behavior by official human score group. Each bar shows the distribution of AI scores within question parts where the official human score was zero, partial credit, or full credit. The partial-credit parts contain 40% of the available points.

Figure 5: Example of AI using the full working to interpret a summary-box transcription error. (a) The transferred expression in the summary answer box is off by a factor of two. (b) The detailed working of the same submission contains the correct self-energy factor and final expression. The AI comment stated: “ The summary box misses a factor, but the working sheet correctly derives

Q_{0}^{2}/(8\pi\epsilon_{0})(1/R_{0}-1/R)

. ” This matched the official full-credit mark.

Figure 6: Examples of visual reasoning in AI grading. (a) Correct reference shapes from the solution. (b) Full-credit example where the relevant sketches were outside the designated answer box. AI found them on later pages and commented: “ Detailed sketches show the required below-critical, critical, and above-critical effective-potential shapes with the expected qualitative behavior. ” (c) Limited-credit example where AI awarded the endpoint behavior specified in the rubric but withheld credit for the missing double-well and critical-transition shapes. The AI comment stated: “ Provides some qualitative potential curves and endpoint behavior, but the subcritical double-well and critical transition are not represented correctly. ”

Figure 7: Examples from masked OE2 experimental submissions. Panels (a) and (b) belong to the same data-table and graph-analysis task, and the AI and official scores agreed on both marking components. AI comments for these two components were: “ The transformed

1/a+1/b

values are tabulated for all

n

values and the linearization is written ” and “ Graph is present with plotted points, best-fit and limiting lines, and slope calculations for the equivalent

1/a+1/b

plot. ” Panel (c) shows a partial-credit graph. Both graders awarded partial credit. AI comment for this component was: “ Five points are plotted on an

R

-versus-

\sqrt{h}

graph, but no usable best-fit line, slope or uncertainty construction is shown. ”

Question type	Description
Conceptual written reasoning	Written physical explanation, evaluation of an answer choice, interpretation, or validity check.
Figure-diagram	Circuits, free-body diagrams, qualitative sketches, ray diagrams, or graph shapes where the visual structure is the answer.
Numerical	Calculation or substitution where the quantity being assessed is primarily a value, probability, time, energy, or angle.
Derivation	Symbolic proof, formula derivation, operator derivation, or multi-step algebraic reasoning.
Figure-data	Experimental measurements, data tables, plotted data, graph fitting, slope extraction, uncertainty, or data-based inference.

Table 5: Question-type labels used across the three assessments.

Figure 8: MAD by question type in RI and RII for the 7058 official question parts. MAD% is the mean absolute difference as a percentage of the maximum possible score for each part.

Question type	$N$	$D_{\mathrm{I}}$	$D_{\mathrm{II}}$	MAD I	MAD II	$r_{\mathrm{I}}$	$r_{\mathrm{II}}$
Conceptual written reasoning	706	+13.3	+4.3	16.6	10.7	0.83	0.89
Figure-diagram	1082	+3.8	+2.5	10.6	7.7	0.84	0.89
Numerical	1978	+4.7	+0.7	11.1	7.6	0.85	0.88
Derivation	2876	+4.8	+2.3	11.4	9.1	0.88	0.91
Figure-data	416	+3.3	+4.4	9.3	6.6	0.92	0.96

Table 6: Question-type metrics for official question parts.

N

is the number of question-part comparisons.

D

and MAD are percentages of the maximum possible score for each part.

Source	Description
Awarding credit too easily	AI sometimes awarded partial credit to work to which human examiners assigned zero points, especially when the answer contained plausible symbols, text, tables, or diagram features.
Over-valuing the final answer	A correct-looking final answer sometimes received too much credit even when the reasoning had serious errors.
Treatment of carried-forward errors	AI could recognize an earlier error but sometimes applied carried-forward-error rules differently from the human examiner.
Missing a required diagram feature	AI often read the diagram but could miss a specific feature required by the rubric, such as relative placement, curvature, scale, or a required label.
Experimental evidence chain	AI often identified tables, graphs, and calculations, but disagreements arose when the score depended on whether the data, fit, uncertainty, and conclusion supported each other.
Permitted scoring increments	In a few cases, AI awarded increments smaller than the intended scoring resolution. This should be stated clearly in the prompt.

Table 7: Recurring sources of disagreement between AI and official scores. These summarize patterns seen in reviewed cases; individual score differences are interpreted against the official scoring.

Target	Points	Responses	Main issue clarified
QM Q3b	3	40	Grover diagram and operator conditions
OE1 Q2	8	72	A thermodynamic reasoning based question
OE2 T2 Q1(f)	4	26	$T$ – $S$ graph shape, labels, and curvature

Table 8: Focused refinement targets selected after reviewing RI disagreements.

Figure 9: Refinement of the rubric for the OE2 question. (a) The intended

T

–

S

cycle from the solution. (b) Student sketch with labels and arrows but with the wrong curvature. In RI, AI marked this as a correct shape. The refined rubric emphasized the physical conditions of the cycle, with less reliance on labels and arrows alone.

Figure 10: Grover-rotation example from the QM exam. (a) Reference rotation diagram redrawn from the official solution. (b) Student response. In RII, the AI identified that the circuit-style sketch was not the required rotation diagram.

	RI	RII
Rubric	Correct diagram with all symbols explained: 1 point. Detailed mathematical description of the Grover operator: 2 points. Determine an appropriate point allocation when the response is incomplete.	0.5 points for axes labeled as $\|x_{0}\rangle$ and $\|x_{0}^{\perp}\rangle$ . 0.5 for showing rotation by $2\theta$ under $G$ . 0.5 for defining $\|x_{0}^{\perp}\rangle$ . 0.5 for the action of $O$ . 1 point for the action of $G=DO$ .
Comment by AI	Awarded 2/3. Correctly describes Grover as oracle plus diffusion and notes the oracle phase flip, with a supporting circuit-style diagram. Deducted 1 because the diffusion/reflection and amplitude-rotation geometry are not fully explained. Method partly follows the official scheme.	Awarded 0.5/3. Gives a qualitative oracle/diffusion description and a circuit-like oracle sketch, but not the required two-dimensional rotation diagram or mathematical action of $O$ and $G$ on $\cos(\alpha)\|x_{0}^{\perp}\rangle+\sin(\alpha)\|x_{0}\rangle$ .

Table 9: RI and RII rubric/comment comparison for the Grover example in Fig. 10 .

Figure 11: AI-minus-human score distributions for two focused refinement examples, shown as percentages of submissions. The horizontal axis gives the difference between AI and human scores. The Round I bars show the original scores, and the Refined bars show scores from the question-specific refinement run.

Figure 12: Confidence flags in RII grading. Medium- and low-confidence question parts have larger differences from official scores, so confidence helps identify parts for human review.

Large Scale AI Grading of Handwritten Physics Assessments: Score Agreement and Olympiad Team Selection Outcomes

Abstract

Figures & tables

Automated Grading

Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark

LLM-as-a-judge validity is strongly task-dependent across physics assessment formats

Designing Reliable LLM-Assisted Rubric Scoring for Constructed Responses: Evidence from Physics Exams

Abstract

Automated Grading

Explore similar work

Evaluating AI Grading on Real-World Handwritten College Mathematics: A Large-Scale Study Toward a Benchmark

LLM-as-a-judge validity is strongly task-dependent across physics assessment formats

Designing Reliable LLM-Assisted Rubric Scoring for Constructed Responses: Evidence from Physics Exams