Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

Ali Habibullah; Yazan Alshoibi; Mohammad Alshiekh; Salman Khan; Naeemullah Khan

cs.CLSep 24, 2026

Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

Authors: Ali Habibullah, Yazan Alshoibi, Mohammad Alshiekh, Salman Khan, Naeemullah Khan

Organizations: KAUST Academy Computer, Electrical & Mathematical Sciences & Engineering (CEMSE) King Abdullah University of Science and Technology (KAUST) Thuwal, Saudi Arabia · Visual Artificial Intelligence Laboratory Oxford Brookes University · KAUST Academy & CEMSE, KAUST · Lady Margaret Hall, University of Oxford

Abstract

One long-form exam in a large course costs hundreds of grader-hours, and qualified graders are scarce; LLM graders are a tempting alternative. To show its pitfalls we grade a practical Computer Vision exam ( $570$ dual-graded students) under $171$ configurations spanning closed and open-weights models; the best reaches mean absolute error $1.64/35$ , below the $2.61/35$ two human graders achieve against each other. The catch is the prompt: a short ''strict grader'' preamble drives $14$ of $17$ open-weights models out of the graded band ( $\text{MAE} \ge 8$ ), three stopping grading altogether. The damage traces to the preamble's two credit-withholding sentences, not to tone or model scale; one of them, ''never give partial credit'', alone makes two of three probed models stop grading. The closed flagships of three vendors shift calibration under it but stay in the band. In $162$ further configurations on a second, independent Machine Learning exam from another course ( $1{,}038$ dual-graded students), the preamble worsens ten models, moving three out of the band into collapse and one into refusal, yet improves seven whose neutral prompts over-mark: the vulnerability replicates, but its direction is exam-specific. Light LoRA fine-tuning repairs it: one adapter on the two exams' pooled $\sim 3{,}900$ graded examples brings five small open models to parity or better with a human grader in agreement with the grader pair, and sensitivity to the three harsh personas nearly vanishes ( $\le 0.32$ MAE). We release the anonymised dataset, full ablation grid, and grading, fine-tuning and analysis pipelines.

Figures & tables

	CV exam ( $n{=}114$ )				ML exam ( $n{=}208$ )
Model	base	CV	ML	pooled	base	CV	ML	pooled
Qwen2.5-Coder-7B	6.60	1.93	3.22	1.84	6.48	6.37	3.67	3.40
Qwen2.5-Coder-14B	4.06	1.89	3.29	1.89	7.70	5.41	3.35	3.29
Qwen3-Coder-30B-A3B	4.76	1.80	2.59	1.75	8.60	5.22	3.67	3.45
Gemma-4-E4B	4.29	1.88	3.79	1.88	4.66	4.79 ‡	3.51	3.53
Llama-3.1-8B	7.99	1.99	4.95	2.01	12.30	5.02	3.48	3.37

Table 1: Held-out MAE vs. the grader average by training set (marks recipe; bd in Table 17 , Appendix E ). The cross columns are transfer; pooled matches or beats each specialist on its own exam.

\ddagger

: the one negative transfer cell (Section 8.5 ).

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

CV exam
Run	Model	Persona	$n$	Q $1$ $\rho$	Q $2$ $\rho$	Q $3$ $\rho$	Total $\rho$
D01	gemini-3-flash-preview	neutral	570	0.92	0.90	0.91	0.95
D02	gemini-3.1-pro-preview	neutral	570	0.92	0.90	0.90	0.94
F02	gemini-3.1-pro-preview	lenient	570	0.92	0.90	0.91	0.94
B03	gemini-flash-lite + thinking	neutral	570	0.89	0.88	0.87	0.92
A01	gemini-flash-lite	neutral	570	0.77	0.65	0.83	0.84

Appendix

Table 2: Item-level agreement for both exams’ headline configurations: Spearman

\rho

between the AI mark and the grader-average mark, per question and at total level, over the same students each run’s MAE uses. The total is the optimistic number: across the

40

CV and

38

ML runs measured here (the rows below plus each exam’s

17

matched neutral/strict pairs; Table 3 , Section 5.4 ), it exceeds the weakest question’s

\rho

in all

37

CV and

37

ML runs where every correlation is defined — median gap

0.20

(CV),

0.12

(ML), and at least

0.15

30

CV and

12

ML runs. The other three CV and one ML runs give every student the same mark, leaving their correlations undefined rather than zero. The weak item is the same within an exam: Q

2

on CV (

30

37

), Q

1

on ML (

27

37

). Pearson

r

differs from

\rho

by a median of

0.02

on both exams. Generated by analysis/checks/per_question_rank_correlation.py .

Run	Model	Family	Params	Neutral	Strict	Ratio	Behaviour
Open weights — ordered by total parameters
L-M05	Qwen2.5-Coder-7B	Qwen	7B	5.68	7.92	$\times 1.39$	graded
L-U02	Llama-3.1-8B	Llama	8B	7.31	26.04	$\times 3.56$	refusal
L-W02	GLM-4-9B	GLM	9B	4.31	25.48	$\times 5.91$	near-refusal
L-F02	Gemma-3-12B	Gemma	12B	5.20	8.77	$\times 1.69$	collapse
L-M07	Qwen2.5-Coder-14B	Qwen	14B	3.51	24.52	$\times 6.99$	collapse

Appendix

Table 3: The strict persona against each model’s own neutral baseline at an identical prompt configuration. MAE in points out of

35

; ratio is strict

\div

neutral; behaviour classes as in Section 5.1 ; a refusal’s MAE is a ceiling artifact and must not be read as severity. Every row is

n=570

. Generated by analysis/computer_vision_make_paper_tables.py .

Model	Family	Params	Neutral	strict	rigorous	exacting
Flash-Lite	Gemini	—	3.34	5.75	3.92	4.58
Llama-3.1-8B	Llama	8B	7.31	26.04	10.31	6.89
GLM-4-9B	GLM	9B	4.31	25.48	4.53	5.18
Gemma-3-12B	Gemma	12B	5.20	8.77	5.45	5.10
DeepSeek-Coder-V2-Lite	DeepSeek	16B	5.98	12.09	7.01	6.69
Mistral-Small-24B	Mistral	24B	3.66	26.03	6.15	7.04

Appendix

Table 4: The three strict-flavoured presets on the models that received all of them. Bold marks a run outside the graded band. These presets are not minimal pairs: only strict carries the two policy sentences (Section 5.2 ); rigorous and exacting ask for rubric precision with neither. Qwen2.5-Coder-

7

B, Qwen3-Coder-

30

B-A

3

B and Qwen3-Coder-Next received only strict in the full-cohort grid; Section 8.4 covers the first two.

	headword varied, policy fixed					policy varied, frame fixed
Model	STRICT	RIGOROUS	FAIR	spread	nharsh	frame	$+$ S1	$+$ S2	both
Qwen2.5-Coder-32B	16.41	17.74	18.55	2.15	8.25	9.37	15.55	12.92	20.26
Llama-3.3-70B	12.67	11.37	11.41	1.30	4.60	—	—	—	—
Gemma-3-27B	10.49	9.23	8.60	1.89	4.56	—	—	—	—
GLM-4-32B	7.02	7.43	6.41	1.02	2.88	—	—	—	—
Mistral-Small-24B	25.99	25.87	25.05	0.94	4.23	5.62	10.33	26.04	26.03

Appendix

Table 5: Attribution of the collapse to the policy sentences rather than the headword. Left: S1 and S2 held verbatim, only the adjective varies; the nharsh control is the neutral frame with its one adjective swapped to “harsh”. Right: the HARSH frame held fixed, crossing S1 with S2 (both quoted in Section 5.2 ). All rows

n\geq 569

, default prompt configuration,

t=0

. Spreads are max

-

min of the unrounded MAEs, so they can differ from the difference of the printed cells by

0.01

. Bold marks cells whose damage matches the full preset’s.

blanket zeroing — consistent with refusal
Run	Model	Family	MAE	Q $1=0$	…of which Q $2>0$
L-U02	Llama-3.1-8B	Llama	26.04	570 / 570 (100%)	0 (0%)
L-Z02	Mistral-Small-24B	Mistral	26.03	570 / 570 (100%)	0 (0%)
L-W02	GLM-4-9B	GLM	25.48	570 / 570 (100%)	29 (5%)
L-M07	Qwen2.5-Coder-14B	Qwen	24.52	533 / 570 (94%)	31 (6%)
L-J02	GLM-4.5-Air	GLM	21.02	361 / 570 (63%)	55 (15%)

Appendix

Table 6: How each matched strict run fails. Q

1=0

is the count of students awarded zero on Q

1

; the next column is how many of those still received a non-zero Q

2

. Generated by analysis/computer_vision_make_paper_tables.py .

Run	Q $1$ score = 0	…with prose $>0$	mean contradicted prose-credit
L-C01 (Qwen2.5-Coder-32B + strict)	437 / 570 (77%)	86 (20%)	2.60 pts (max 10.0)
L-M01 (Qwen3-Coder-30B-A3B + strict)	357 / 570 (63%)	152 (43%)	2.75 pts (max 8.5)

Appendix

Table 7: Self-contradiction between the structured Q

1

score and the prose breakdown, on the two strict-persona runs whose rationales we annotated by hand. A row is self-contradicting when the structured score is

0

but the rationale itemises positive partial credit. Both runs sit in the selective-field-collapse group of Table 6 and match its Q

1=0

counts; the second column differs because this one requires reading the prose, which we did not do at scale.

Model	Run	MAE [CI]	Bias	Zero	Behaviour
Qwen2.5-Coder-32B	paper neutral (L-A $32$ )	8.02 [7.38, 8.68]	$-7.96$	0%	collapse
	paper strict (L-C $01$ )	21.24 [20.31, 22.14]	$-21.24$	18%	collapse
	decomposed, strict	20.00 [19.10, 20.93]	$-20.00$	3%	collapse
	arbitrated, strict	9.29 [8.58, 9.98]	$-9.27$	0%	collapse
GLM-4-32B	paper neutral (L-X $01$ )	2.24 [1.91, 2.59]	$-0.17$	0%	graded
	paper strict (L-X $02$ )	8.90 [7.81, 10.03]	$-8.81$	5%	collapse

Appendix

Table 8: Prompt-only repairs under the strict persona, CV exam, first

100

students,

35

-point base scale; the paper’s strict and neutral runs are restricted to the same students. MAE with

95\%

bootstrap CI (

2000

resamples); zero = share of students awarded a total of

0

; behaviour classes as in Section 5.1 . Human floor on these students:

3.43

Model	Persona	CV exam (floor $2.61$ )		ML exam (floor $5.13$ )
		MAE [95% CI]	bias	MAE [95% CI]	bias
claude-opus-5	neutral	$3.54$ $[3.31,\,3.77]$	$-3.24$	$4.37$ $[4.13,\,4.63]$	$-2.64$
	strict	$5.17$ $[4.90,\,5.45]$	$-5.02$	—	—
gemini-3.1-pro-preview	neutral	$1.86$ $[1.70,\,2.02]$	$-0.82$	$3.40$ $[3.16,\,3.65]$	$+1.18$
	strict	$2.75$ $[2.51,\,2.99]$	$-2.09$	$3.67$ $[3.43,\,3.94]$	$-1.00$
	lenient	$1.79$ $[1.62,\,1.95]$	$+0.82$	$4.48$ $[4.22,\,4.75]$	$+3.46$

Appendix

Table 9: Closed models from three vendors under neutral , strict and lenient on both exams (Section 6.1 ): MAE against the grader average with

95\%

bootstrap CI, and bias. Gemini rows are grid runs (D

02

03

02

and A

01

01

02

on the CV exam; IG

08

/IG

23

/IG

14

and IG

01

/IG

05

/IG

06

on the ML exam, with their

n

as in Tables 24 and 25 ). OpenAI and Anthropic rows were graded on the full cohorts through the vendors’ batch APIs with the identical prompt, reasoning off and temperature

0

(OpenAI) or thinking disabled (Anthropic, whose API exposes no temperature); every reply parsed. Bold marks a run outside the graded band; no zero-total rate exceeds

1.1\%

. Generated by analysis/closed_vendor_table.py .

Run	Model	$n$	MAE	$95\%$ CI	Bias
Human inter-grader floor (Section 4 )
—	G 1 vs. G 2	570	2.61	$[2.37,\ 2.85]$	—
Full-cohort configurations
D01	gemini-3-flash-preview, neutral	570	1.64	$[1.51,\ 1.79]$	$+0.01$
F02	gemini-3.1-pro-preview + lenient	570	1.79	$[1.62,\ 1.95]$	$+0.82$
D02	gemini-3.1-pro-preview, neutral	570	1.86	$[1.70,\ 2.02]$	$-0.82$

Appendix

Table 10: The six full-cohort configurations, and five first-

100

runs, whose point-estimate MAE meets or undercuts the human inter-grader floor, with

95\%

bootstrap CIs (

2000

resamples), plus reference rows. The G-series and M02 rows cover only the first

\sim 100

students, an easier subset (the unmodified D01 recipe scores

1.23

there vs.

1.64

on the full cohort; Appendix C ); they are not comparable to the full-cohort rows, and M02’s CI upper bound crosses the floor.

Run	Configuration	$\|\mathrm{AI}{-}\mathrm{G}_{\mathrm{avg}}\|$	$\|\mathrm{AI}{-}\mathrm{G}_{1}\|$	$\|\mathrm{AI}{-}\mathrm{G}_{2}\|$	$\overline{d}$ [ $95\%$ CI]	Verdict
D01	3-flash-preview, neutral	1.64	2.23	1.92	$-0.54$ $[-0.71,-0.36]$	better
F02	3.1-pro-preview $+$ lenient	1.79	2.32	2.15	$-0.37$ $[-0.56,-0.18]$	better
D02	3.1-pro-preview, neutral	1.86	2.53	1.96	$-0.37$ $[-0.55,-0.18]$	better
B03	flash-lite $+$ thinking	2.05	2.55	2.29	$-0.19$ $[-0.39,+0.01]$	$=$ grader
O03	gpt-5.5 $+$ lenient (Section 6.1 )	2.25	2.59	2.62	$-0.01$ $[-0.22,+0.21]$	$=$ grader
O01	gpt-5.5 , neutral (Section 6.1 )	2.43	3.10	2.51	$+0.19$ $[-0.02,+0.41]$	$=$ grader

Appendix

Table 11: The paired third-grader test (Eq. ( 1 )) applied to the zero-shot configurations, on the full

570

-student CV cohort: the same statistic and decision rule as Table 19 , the standard the fine-tuned models face. better means the whole

95\%

CI of

\overline{d}

sits below zero, “

=

grader” that it contains zero, worse that it lies above. CIs are the normal approximation used in Table 19 ; a

2000

-resample percentile bootstrap gives the same verdict in every row. With the floor at

2.61

\overline{d}=\tfrac{1}{2}(|\mathrm{AI}-\mathrm{G}_{1}|+|\mathrm{AI}-\mathrm{G}_{2}|)-2.61

exactly. The two single-grader distances differ by up to

0.63

here, so both are shown. Generated by analysis/checks/paired_test_zero_shot_cv.py .

Run	Change vs. A01	MAE	Bias
A01	— (baseline)	3.34	$-1.78$
B01	drop reference solution	4.13	$+3.30$
B02	drop guidelines	3.34	$-1.42$
B03	thinking on (dynamic)	2.05	$-0.15$
B04	drop rubric breakdown	3.22	$-1.45$

Appendix

Table 12: Prompt-component ablations on gemini-flash-lite . B03 vs. A01 measures dynamic-budget thinking against the baseline’s disabled thinking ( thinking_budget

=0

; Appendix A ).

n=570

except B01 (

567

) and B02/B04 (

569

each), which exclude students with a permanently failed grading call.

Run	Change vs. baseline	MAE	$95\%$ CI
D01 (restricted)	— (baseline)	1.23	$[1.02,\ 1.49]$
G04	drop rubric breakdown	1.25	$[1.01,\ 1.51]$
G03	thinking on (dynamic)	1.39	$[1.18,\ 1.61]$
G02	drop guidelines	1.60	$[1.30,\ 1.93]$
G01	drop reference solution	1.63	$[1.32,\ 2.02]$

Appendix

Table 13: G-series on gemini-3-flash-preview , all on the first-

100

subset (

n=100

, except G01/G02 at

98

99

after excluding permanently failed grading calls). D01-restricted is the baseline recipe on the same students. Bootstrap CI on

2000

resamples.

Run	Model	Params	MAE	$95\%$ CI	Bias
L-S01	DeepSeek-Coder-V2-Lite (MoE)	16B	5.98	$[5.69,\ 6.29]$	$-3.44$
L-W01	GLM-4-9B	9B	4.31	$[4.02,\ 4.63]$	$+1.68$
L-X01	GLM-4-32B	32B	2.85	$[2.66,\ 3.05]$	$+0.50$
L-J01	GLM-4.5-Air (MoE)	106B	5.66	$[5.38,\ 5.95]$	$-5.37$
L-F01	Gemma-3-12B	12B	5.20	$[4.94,\ 5.48]$	$-2.75$
L-Y01	Gemma-3-27B	27B	4.32	$[4.12,\ 4.55]$	$-1.52$

Appendix

Table 14: Open-weights baselines on the default prompt (neutral persona,

t=0

), ordered by family then total parameters. MAE in points out of

35

with

95\%

bootstrap CI; bias is mean signed error (AI

-

grader-avg). Six vendors and both dense and mixture-of-experts architectures. All rows

n=570

Run	Model	MAE	Bias
A01	gemini-flash-lite (no few-shot)	3.34	$-1.78$
K01	gemini-flash-lite + few-shot	3.03	$+1.09$
L-A30M	Qwen3-Coder-30B-A3B (no few-shot)	4.50	$-2.70$
L-K01	Qwen3-Coder-30B-A3B + few-shot	3.34	$-1.77$

Appendix

Table 15: Two worked examples per question, drawn from the highest-agreement grader pair and labelled with D

01

’s scores and rationales (in-context distillation, not human exemplars); the five demonstration students remain in the evaluated cohort (Appendix J ), though excluding them moves the uplift by at most

0.01

( analysis/checks/fewshot_demo_leakage.py ). Few-shot moves Gemini Flash-Lite by

\approx 0.3

MAE and Qwen3-Coder-

30

B-A

3

B by

\approx 1.2

MAE points.

	CV exam		ML exam
Model	base	pooled	base	pooled
Qwen2.5-Coder-7B	$+5.40$	$+0.02$	$+0.22$	$-0.04$
Qwen2.5-Coder-14B	$+19.75$	$-0.02$	$+1.13$	$+0.32$
Qwen3-Coder-30B-A3B	$+12.51$	$+0.07$	$-1.26^{\dagger}$	$+0.29$
Gemma-4-E4B	$+5.81$	$+0.05$	$+6.53$	$+0.30$
Llama-3.1-8B	$+17.28$	$-0.05$	$+18.62$	$+0.03$

Appendix

Table 16: Worst MAE drift over the three harsh personas ( strict , rigorous , exacting ): the largest signed difference persona

-

neutral; when all three are negative it is the least negative, which is why Qwen3-Coder-

30

B-A

3

B’s ML entry of

-1.26

is its rigorous drift while strict sits at

-2.26

(marks recipe; full grid in Table 18 ).

\dagger

: negative drift: the persona accidentally improved the over-grading base (Section 8.4 ). lenient , swept after tuning, is in Table 18 : the pooled marks adapters drift

+0.05

+0.39

except Qwen3-Coder-

30

B-A

3

B (

+1.02

CV,

+1.81

ML); bd

+0.20

+1.64

Graded on	base	CV-trained	ML-trained	pooled
CV exam, marks	5.54	1.90	3.57	1.87
CV exam, bd	4.73	2.13	3.52	2.01
ML exam, marks	7.95	5.36	3.54	3.41
ML exam, bd	8.76	6.16	4.24	4.15

Appendix

Table 17: Held-out MAE vs. the grader average, mean over the five models, both recipes (the per-model marks grid is Table 1 ). The pooled adapter is best or tied in all eight columns.

		CV exam					ML exam
Model	Arm	neut.	strict	rigor.	exact.	len.	neut.	strict	rigor.	exact.	len.
Qwen2.5-7B	base	6.60	12.00	9.07	6.07	4.81	6.48	6.70	6.04	6.15	7.50
	pooled	1.84	1.84	1.85	1.81	2.00	3.40	3.36	3.30	3.31	3.52
Qwen2.5-14B	base	4.06	23.81	3.65	7.31	3.79	7.70	8.83	7.71	6.33	11.18
	pooled	1.89	1.85	1.87	1.81	2.00	3.29	3.61	3.23	3.28	3.50
Qwen3-30B	base	4.76	17.28	4.96	5.76	4.49	8.60	6.34 †	7.34	6.30	13.00

Appendix

Table 18: The full persona grid (MAE, marks recipe; per-persona bias and the bd grid are in the released workbooks).

\dagger

: the sign-inconsistency cell of Section 8.4 — strict improves the over-grading base. len. is the inflating direction, swept after tuning: the pooled marks adapters drift

+0.05

+0.39

except Qwen3-

30

B (

+1.02

CV,

+1.81

ML, the latter above the

5.19

floor); every pooled lenient drift is positive (bias

+1.0

+4.4

). bd worst drifts: base up to

+18.5

on either exam; pooled

\leq+0.62

under the three harsh wordings, with Llama-bd- strict at

2.67

(down from

4.08

under the CV-exam-only adapter); under lenient the bd adapters drift

+0.20

+1.64

(Qwen3-

30

B ML

4.07\to 5.71

, Gemma-

4

4.22\to 5.51

Cell	$n$	$\|\mathrm{AI}{-}\mathrm{G}_{\mathrm{avg}}\|$	$\|\mathrm{AI}{-}\mathrm{G}_{1}\|$	$\|\mathrm{AI}{-}\mathrm{G}_{2}\|$	$\overline{d}$ [ $95\%$ CI]	Verdict
7B marks, CV	114	1.84	2.35	2.35	$-0.48$ $[-0.90,-0.05]$	better
14B marks, CV	114	1.89	2.38	2.35	$-0.46$ $[-0.89,-0.044]$	better
30B marks, CV	114	1.75	2.38	2.17	$-0.56$ $[-0.98,-0.13]$	better
Gemma marks, CV	114	1.88	2.41	2.34	$-0.45$ $[-0.89,-0.020]$	better
Llama marks, CV	114	2.01	2.54	2.34	$-0.39$ $[-0.85,+0.07]$	$=$ grader
7B marks, ML	208	3.40	4.19	3.96	$-1.11$ $[-1.73,-0.49]$	better

Appendix

Table 19: Paired third-grader test (Eq. ( 1 )) for the pooled adapters. Floors:

2.83

(CV),

5.19

(ML). better : the whole

95\%

CI of

\overline{d}

below zero; “

=

grader”: it contains zero. Both single-grader distances are shown:

|\mathrm{AI}-\mathrm{G}_{1}|

is the larger in all ten ML rows and nine of ten CV rows, by up to

0.22

(CV) and

0.57

(ML). No paired-bootstrap difference excludes zero, and one ML student whose G 1 slot reads

0.0

against a real G 2 mark (Appendix H ) alone accounts for

0.21

–

0.25

of that gap, so we read the asymmetry as the grader-slot artifact, not a systematic preference. Three CV upper bounds fall inside

\pm 0.05

and are printed to three decimals; all stay below zero under both the normal CI used here and a

2000

-resample percentile bootstrap, though 7B bd, CV is marginal (Section 10 ). Recomputed by analysis/checks/ftpaired_marginal_verdicts.py and ftpaired_symmetry_guard.py .

Run	Configuration	$n$	$b_{1}$	$b_{2}$	$b_{3}$	Total	$\sum_{q}\|b_{q}\|$	$C$
CV exam — $35$ -point base scale (Q $1$ $12$ , Q $2$ $11$ , Q $3$ $12$ )
D01	3-flash-preview, neutral	570	$-0.15$	$+0.08$	$+0.08$	$+0.01$	0.30	0.29
D02	3.1-pro-preview, neutral	570	$-0.44$	$-0.07$	$-0.31$	$-0.82$	0.82	0.00
F02	3.1-pro-preview $+$ lenient	570	$-0.01$	$+0.28$	$+0.55$	$+0.82$	0.84	0.02
B03	flash-lite $+$ thinking	570	$-0.50$	$+0.04$	$+0.30$	$-0.15$	0.85	0.69
A01	flash-lite, neutral (baseline)	570	$-0.28$	$-2.12$	$+0.63$	$-1.78$	3.03	1.25

Appendix

Table 20: Per-question mean signed error for the headline runs of both exams, over exactly the students each run’s published metric uses.

b_{q}

is the mean of

\mathrm{AI}_{q}-\bar{\mathrm{G}}_{q}

on question

q

(CV exam:

35

-point base scale, Q

4

bonus excluded; ML exam: score plus bonus,

\approx 65

-point scale). Total is the run’s published bias.

C=\sum_{q}|b_{q}|-|\sum_{q}b_{q}|

is the cancellation the total hides, zero exactly when every question is pushed the same way. At two decimals a row can miss

\sum_{q}b_{q}={}

Total by

0.01

through rounding. Generated by analysis/checks/per_question_signed_error.py .

Pair	$n$	Human MAE	D $01$ AI MAE	D $01$ AI bias
G 6 / G 16	56	0.85	1.86	$+1.18$
G 9 / G 19	55	1.87	1.43	$-0.46$
G 5 / G 15	57	2.12	1.95	$-0.90$
G 4 / G 14	56	2.25	1.40	$-0.88$
G 7 / G 17	57	2.48	1.84	$+1.11$
G 8 / G 18	58	2.85	1.83	$-0.34$

Appendix

Table 21: Per-pair human disagreement vs. D

01

’s per-pair AI MAE, sorted by human MAE.

model	neutral	strict	strict CI	ratio	$\times$ floor	zero-rate	bias	behaviour
GLM-4.5-Air	5.60	18.49	[17.96, 19.02]	$\times 3.30$	3.6	37.8%	$-18.31$	collapse
Qwen2.5-Coder-32B	4.57	13.24	[12.79, 13.69]	$\times 2.90$	2.6	10.5%	$-13.07$	graded
Llama-3.1-8B	13.16	29.58	[28.70, 30.45]	$\times 2.25$	5.8	100.0%	$-29.58$	refusal
Llama-3.3-70B	4.70	10.17	[9.77, 10.58]	$\times 2.16$	2.0	9.0%	$-9.63$	graded
Mistral-Small-24B	8.96	17.32	[16.77, 17.87]	$\times 1.93$	3.4	32.2%	$-17.20$	collapse
Qwen3-Coder-Next 80B	4.47	8.63	[8.26, 9.00]	$\times 1.93$	1.7	2.3%	$-8.12$	graded

Appendix

Table 22: ML exam,

1{,}038

dual-graded students: matched neutral/strict pairs for all

17

open-weights models, sorted by damage ratio. MAE against the grader average on a

\approx 65

-point scale,

95\%

bootstrap CI on the strict run, strict MAE as a multiple of the ML exam’s human floor (

5.13

, defined as in Section 4 ). Behaviour classes are those of Section 5.1 and describe the strict run, at this exam’s floor-matched band

\text{MAE}\geq 15.7

(

3.07\times

its

5.13

floor, the multiple the CV exam’s

8

fixes); four strict runs and one neutral baseline cross it. A fixed

\text{MAE}\geq 8

would be only

1.56\times

this floor and would label ten strict runs and eleven neutral baselines collapse. A refusal’s MAE is a ceiling artifact, not a severity measurement. Ratios below

1

are real improvements, examined below. Generated by analysis/introduction_to_ai_make_paper_tables.py .

model (strict)	Q $1=0$	of those, Q $2>0$	share	reading
Llama-3.1-8B	1038 / 1038 (100%)	0	0%	blanket
GLM-4-9B	669 / 1038 (64%)	255	38%	selective
GLM-4.5-Air	521 / 1038 (50%)	110	21%	blanket
GLM-4-32B	477 / 1038 (46%)	295	62%	n/a (in band)
Mistral-Small-24B	375 / 1038 (36%)	21	6%	blanket
Qwen2.5-Coder-14B	196 / 1038 (19%)	13	7%	n/a (in band)

Appendix

Table 23: ML exam, strict runs: blanket versus selective zeroing, all

17

models. Q

1=0

is the count of students awarded zero on Q

1

out of those the run graded; the next column is how many of those still received a non-zero Q

2

. A run is read only if it left the graded band and zeroed Q

1

on at least

5\%

of the students it graded (the CV exam applies the same volume gate at

10\%

, MECH_ZERO_RATE ; on these runs any cut between

4.8\%

and

13.3\%

gives the same partition): below that, the zeros are ordinary non-attempts and the share is noise on a handful of students. Generated by analysis/introduction_to_ai_make_paper_tables.py .

Run	Model	Persona	Prompt	$t$	$n$	MAE	$95\%$ CI	Bias	Behaviour
Closed-model configurations (Gemini: A–P series; OpenAI: O; Anthropic: N)
A01	flash-lite	neutral	SGB	$0$	$570$	$3.34$	$[3.15,\ 3.53]$	$-1.78$	graded
B01	flash-lite	neutral	GB	$0$	$567$	$4.13$	$[3.83,\ 4.42]$	$+3.30$	graded
B02	flash-lite	neutral	SB	$0$	$569$	$3.34$	$[3.16,\ 3.52]$	$-1.42$	graded
B03	flash-lite	neutral	SGBR	$0$	$570$	$2.05$	$[1.89,\ 2.21]$	$-0.15$	graded
B04	flash-lite	neutral	SG	$0$	$569$	$3.22$	$[3.03,\ 3.43]$	$-1.45$	graded

Appendix

Table 24: Every grader configuration in the study, one row per run: the

33

closed-model configurations (

25

Gemini via Vertex,

6

OpenAI and

2

Anthropic via their batch APIs; Section 6.1 ) followed by the

138

open-weights (L-) configurations, each sorted by run id. Prompt lists the components present (S reference solution, G grading guidelines, B rubric breakdown, R thinking enabled, F few-shot demonstrations); personas prefixed mp are the mechanism probes of Section 5.2 : mpstrict , mprigorous and mpfair keep the strict preset’s two policy sentences and swap only its headword; mpframe is the HARSH frame alone, mpnoclause the frame with S1, mps2only the frame with S2, mpharsh the frame with both (the strict text, rerun on the current stack); mpnharsh is the neutral frame with “harsh” in place of “strict”. L-MP

04

and L-PS

03

are one physical run ( qwen2.5-coder-32b , mpnoclause ), listed under both the headword series and the policy

2\times 2

n

is the number of students scored against the grader average; MAE and bias are on the

35

-point base scale, the

95\%

bootstrap CI on MAE; behaviour classes are defined in Section 5.1 . A refusal’s MAE is a ceiling artifact, not a severity measurement. D

03

ran with thinking active regardless of its flag ( gemini-2.5-pro cannot disable it; Appendix A ). claude-opus-5 exposes no sampling temperature (thinking disabled; shown as —). The final block lists the

8

prompt-only baselines of Appendix B (first

100

students; the persona column names the strategy), which are not configurations of the paper’s grader and are excluded from every count; their CIs use

2000

resamples. Model keys match the released spreadsheets. Generated by analysis/computer_vision_make_appendix_table.py .

Run	Model	Persona	Prompt	$t$	$n$	MAE	$95\%$ CI	Bias	Behaviour
Closed-model configurations (Gemini IG-, OpenAI IO-, Anthropic IN-)
IG01	flash-lite	neutral	SB	$0$	$1013$	$7.53$	$[7.20,\ 7.87]$	$+5.98$	graded
IG02	flash-lite	neutral	B	$0$	$1032$	$12.46$	$[12.03,\ 12.89]$	$+12.16$	graded
IG03	flash-lite	neutral	SBR	$0$	$1037$	$5.37$	$[5.09,\ 5.66]$	$-4.04$	graded
IG04	flash-lite	neutral	S	$0$	$1016$	$6.88$	$[6.57,\ 7.20]$	$+5.27$	graded
IG05	flash-lite	strict	SB	$0$	$1008$	$9.02$	$[8.61,\ 9.43]$	$-7.83$	graded

Appendix

Table 25: Every second-exam grader configuration, one row per run: the

30

closed-model configurations (

23

Gemini IG-,

6

OpenAI IO- and

1

Anthropic IN-; Section 6.1 ) followed by the

132

open-weights (IA-) configurations, each in run-id order. Prompt lists the components present (S reference solution, B rubric breakdown, R thinking enabled, F few-shot demonstrations); this exam has no student-facing guidelines document, so the G component of Table 24 never appears. Personas prefixed mp are the mechanism probes of Section 5.2 , replayed on this exam (keys decoded in Table 24 ’s caption).

n

is the number of students scored against the grader average (

\ast

: IG

18

’s

693

are the survivors of prompt-length failures, an easier subsample whose MAE is not comparable to full-cohort rows; Appendix H ); MAE and bias are on the

\approx 65

-point scale (score plus bonus, matching how its graders record marks), the

95\%

bootstrap CI (

50,000

resamples) on MAE. Behaviour classes are those of Section 5.1 , at this exam’s floor-matched collapse band (

\text{MAE}\geq 15.7

), and describe a run’s error level rather than persona damage: DeepSeek-Coder-V2-Lite over-marks badly enough to leave it unprompted (IA

11

, IA

13

). A refusal’s MAE is a ceiling artifact, not a severity measurement. IG

15

–IG

20

ran on the first

100

students; IG

10

–IG

12

, IA

92

and IA

136

are repeated-sampling probes whose scores are the per-student mean across five reruns (Appendix H ). claude-opus-5 exposes no sampling temperature (thinking disabled; shown as —). Run ids match the released tracker and spreadsheets. Generated by analysis/introduction_to_ai_make_appendix_table.py .

Explore similar work

Jun 1, 2026cs.CY

LLMs as Teaching Assistants for Mathematics Exam Grading: Reliability, and Practical Usability

Open-ended mathematics exams are valuable because they assess reasoning, proof construction, algorithmic thinking, and communication of intermediate steps. They are also difficult to grade at scale because instructors must apply partial-credit rubrics consistently while giving feedback that helps students repair misconceptions. This paper evaluates six contemporary large language model (LLM) configurations, Gemini 3.1 Pro Extended, Gemini 3.5 Flash, ChatGPT 5.5 Pro Extended, ChatGPT 5.5 Thinking, Claude Pro Opus 4.7, and Claude Sonnet 4.6, as grading assistants for an undergraduate discrete mathematics examination. The study compares two grading policies. The BASELINE policy uses a stricter rubric-following prompt that emphasizes explicit evidence and complete justification. The LIBERAL policy was added after preliminary grading showed that the baseline condition sometimes applied harsh point deductions and failed to recognize valid partial reasoning. Agreement with human grading is measured at both the question and exam-total levels using mean absolute error, root mean squared error, normalized root mean squared error, Pearson correlation, and exact agreement. The results show that liberal partial-credit prompting reduces average question-level error for every evaluated model family. ChatGPT 5.5 Thinking (LIBERAL) has the lowest average question-level MAE (1.87) and RMSE (2.53), while Gemini 3.1 Pro Extended (LIBERAL) has the lowest total-score MAE (8.00) and RMSE (10.66). However, the strongest total-score Pearson correlation occurs under Gemini 3.1 Pro Extended (BASELINE) at 0.58, showing that point calibration and rank preservation remain distinct goals. We also report practical usability observations.

Aastha Sapkota, M. G. Sarwar Murshed

Computer Science Department University of Wisconsin - Green Bay, Green Bay, WI, USA

Grading Preliminary Study Assistant Spearman Correlation

Ask PDF

Jul 2, 2026cs.AI

Automated grading of Linux/bash examinations using large language models: a four-level cognitive taxonomy approach

Scalable and reliable grading of command-line examinations remains a challenge in computing education, where rising enrolments make manual marking difficult and rule-based autograders cannot handle partial credit, equivalent solutions, or syntactic variation. This paper evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The study adopts a four-level cognitive taxonomy that combines cognitive complexity and operational impact, ranging from information retrieval (L1) and basic file manipulation (L2) to structural operations (L3) and advanced system management (L4). The models were tested with two prompt variants, a minimal baseline and a rubric-enhanced version, on 1200 real responses from second-year Computer Engineering students independently graded by three expert instructors. Gemini~3.0 Pro with rubric-guided prompting achieved the highest human-AI agreement (ICC(3,1) = 0.888, MAE = 0.10, Bland-Altman bias = -0.014). Agreement declined consistently as taxonomy level increased, with the largest discrepancies at higher levels. Across all models, rubric quality had a larger effect than provider choice, with structured prompts consistently improving agreement. These results show that question complexity is a reliable predictor of the difficulty LLMs face in grading accurately, and they establish a principled, taxonomy-based framework for determining which questions are suitable for AI-assisted grading and which require human review, while also providing a transferable evaluation protocol and prompt templates.

Manuel Alonso-Carracedo, Ruben Fernandez-Boullon, Pedro Celard +2

aUniversidade de Vigo, Department of Computer Science, ESEI-Higher School of Computer Engineering, Edificio Politécnico, Campus Universitario As Lagoas s/n, Ourense, 32004, Spain · bIFCAE-Institute for Research in Physics, Computing and Aerospace Science, Universidade de Vigo, Ourense, 32004, Spain

Grading Assessment Taxonomy Cognitive Science Robotwin

Ask PDF

Sep 13, 2026cs.CV

Vision-Language Models for Criterion-Level Grading of Handwritten Examinations in Outcome-Based Education

Criterion-level grading connects examination performance to learning outcomes, but manual marking introduces workload and variation between markers. This study evaluates vision-language models (VLMs) for handwritten outcome-based assessment across five dimensions: accuracy, human agreement, repeated-run reliability, error concentration, and explanation quality. Using 1,982 criterion-level records from 485 undergraduate examination answers, we compare 20 configurations spanning Qwen2.5-VL, InternVL3, Pixtral, a Donut baseline, and a cascade ensemble. Evaluation setups include zero-shot prompting, few-shot prompting, partial fine-tuning, and Low-Rank Adaptation (LoRA). Two independent faculty markers regraded all 291 test criteria, providing a human agreement baseline on the same assessment materials. Qwen2.5-VL with LoRA achieved Quadratic Weighted Kappa (QWK) of 0.727 and mean absolute error of 0.435 marks against the examiner, compared with mean human-pair QWK of 0.551. This comparison reflects calibration to the examiner's training marks. LoRA outperformed partial fine-tuning for all three instruction-tuned VLMs, while few-shot prompting reduced QWK in every configuration with valid prompted scores. Aggregate reliability and exact repeatability diverged: intraclass correlations ranged from 0.790 to 0.874, yet 50.2-63.6% of criteria changed marks across five sampled runs. Attention-guided deletion showed no statistically significant advantage over random masking, and four faculty reviewers reached no consensus on explanation usefulness. These findings highlight the need for rubric-specific calibration, repeatable scoring, review of consequential errors, and separate validation of explanations. The released evaluation protocol supports criterion-level assessment research and grading tools with teacher oversight.

Asif Hasan Tonmoy, Saad Ahmed, Md Khalid Syfullah +1

Grading Inter-Annotator Agreement Assessment

Ask PDF