CHiL(L)Grader: Calibrated Human-in-the-Loop Short-Answer Grading

Pranav Raikote; Korbinian Randl; Ioanna Miliou; Athanasios Lakes; Panagiotis Papapetrou

cs.CLMar 12, 2026

CHiL(L)Grader: Calibrated Human-in-the-Loop Short-Answer Grading

Authors: Pranav Raikote, Korbinian Randl, Ioanna Miliou, Athanasios Lakes, Panagiotis Papapetrou

Organizations: Department of Computer and Systems Sciences, Stockholm University, 164 25 Kista, Sweden

Abstract

Scaling educational assessment with large language models requires not just accuracy, but the ability to recognize when predictions are trustworthy. Instruction-tuned models tend to be overconfident, and their reliability deteriorates as curricula evolve, making fully autonomous deployment unsafe in high-stakes settings. We introduce CHiL(L)Grader, the first automated grading framework that incorporates calibrated confidence estimation into a human-in-the-loop workflow. Using post-hoc temperature scaling, confidence-based selective prediction, and continual learning, CHiL(L)Grader automates only high-confidence predictions while routing uncertain cases to human graders, and adapts to evolving rubrics and unseen questions. Across three short-answer grading datasets, CHiL(L)Grader automatically scores 35-65% of responses at expert-level quality (QWK >= 0.80). A QWK gap of 0.347 between accepted and rejected predictions confirms the effectiveness of the confidence-based routing. Each correction cycle strengthens the model's grading capability as it learns from teacher feedback. These results show that uncertainty quantification is key for reliable AI-assisted grading.

Figures & tables

Figure 1: The CHiL(L)Grader loop over two iterations for similar responses to the same question. In Iteration I, the model predicts 3/5 with low confidence (40%), triggering teacher review; the corrected 4/5 grade is used for fine - tuning. In Iteration II, the updated model predicts 4/5 with high confidence (80%), enabling automatic acceptance.

Figure 2: CHiL(L)Grader architecture. Historical exams are used to train the instruction - tuned model. Prior - year exams calibrate its confidence. During the current exam, low - confidence cases are sent to human review, whose corrections, combined with replay samples, guide conservative model updates and recalibration.

Panel of Figure 3 — Algorithm 1 Calibrated Human-in-the-Loop ASAG

Dataset	Domain	Train	Cal	Test	MaxGrade	Eval Type
DAMI	Data Mining	3,770	260	261	5/8/10	UA + UQ
SciEntsBank	Science (K–6)	4,969	540	733	4	UQ
EngSAF	Engineering	3,650	405	1,743	2	UA + UQ

Table 1: Dataset statistics. MaxGrade indicates the grading scale. UA = Unseen Answers (questions in train), UQ = Unseen Questions (questions not in train).

Method	Model	QWK
Zero-shot	Llama-3.1-8B	0.289
Few-shot (k=3)	Llama-3.1-8B	0.587
Few-shot (k=5)	Llama-3.1-8B	0.603
RAG only	Llama-3.1-8B	0.443
RAG + Few-shot (k=3)	Llama-3.1-8B	0.491
Fine-tuning (LoRA)	Llama-3.1-8B	0.677 †

Table 2: Baseline results on DAMI. † Llama-3.1-8B achieves the highest QWK but with severe systematic overgrading (

+1.87

mean grade offset), disqualifying it for deployment. Fine-tuned models substantially outperform all prompt-based baselines.

Dataset	Baseline	Calibrated	Optimal	ECE
	ECE	ECE	T*	Reduction
DAMI	0.270	0.094	0.337	-65%
SciEntsBank	0.089	0.084	1.170	-6%
EngSAF	0.097	0.046	1.816	-53%

Table 3: Calibration results. Temperature scaling reduces ECE by 53–65% on DAMI and EngSAF while leaving QWK unchanged. SciEntsBank requires minimal adjustment as the base model is already near-calibrated.

Dataset	Split	Iteration	Coverage	Baseline	Acc.	$\Delta$	Rej.
			(%)	QWK	QWK		QWK
DAMI	$\mathcal{D}_{21}$	0	97.7	—	0.721	—	—
DAMI	$\mathcal{D}_{22}$	1	35.1	0.458	0.882	+0.424	0.535
SciEntsBank	$\mathcal{D}_{21}$	0	70.1	—	0.436	—	0.000
	$\mathcal{D}_{22}$	1	36.1	0.426	0.764	+0.338	0.253
	$\mathcal{D}_{23}$	2	43.2	0.390	0.410	+0.020	0.327

Table 4: Progressive HiL learning across correction cycles. Baseline QWK is computed on each split’s accepted subset prior to corrections; Acc. QWK is computed after adapter fine-tuning;

\Delta

= Acc. QWK

-

Baseline QWK. Rej. QWK is the model’s performance on human-routed samples.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Prompt	Zero	FS-1	FS-3	FS-5
Llama-3.1-8B	basic	0.289	0.549	0.587	0.603
	detailed	0.477	0.482	0.537	0.524
	json_strict	0.316	0.490	0.560	0.541
	rubric	0.425	0.532	0.507	0.506
Llama-3.2-3B	basic	0.256	0.383	0.324	0.337
	detailed	0.365	0.416	0.388	0.355

Appendix

Table 5: Full prompt engineering results on DAMI . Bold indicates the best configuration per model.

Figure 3: Exact Match and Off-by-1 accuracy for Qwen-2.5-7B across DAMI , SciEntsBank , and EngSAF . The improvement across datasets reflects scale granularity rather than model quality, Off-by-1 on EngSAF (

G\in\{0,1,2\}

) is a much coarser tolerance than on DAMI (

G\in\{0,\ldots,10\}

Model	Type	$n$	QWK	EM (%)	Off-by-1 (%)	MAE
Baseline	UA	89	0.705	41.6	61.8	1.427
Baseline	UQ	42	0.365	19.0	47.6	2.310
Post-HiL	UA	89	0.719	30.3	66.3	1.472
Post-HiL	UQ	42	0.474	23.8	47.6	2.024

Appendix

Table 6: Performance breakdown by question type on DAMI . UA = Unseen Answers (questions seen during training); UQ = Unseen Questions (questions not seen during training). Baseline is the fine-tuned model on the full test set; Post-HiL is evaluated on

\mathcal{D}_{22}

after one correction cycle.

Figure 4: Confusion matrices for baseline models on DAMI , SciEntsBank , and EngSAF . DAMI errors concentrate within $\pm 1$ of the diagonal; SciEntsBank shows systematic under-prediction of grade 4; EngSAF confirms the relative simplicity of 3-way grading.

Figure 5: Coverage–quality curves for DAMI , SciEntsBank , and EngSAF . Each point corresponds to a specific

\tau

value; selected operating points are marked. Restricting coverage to high-confidence predictions consistently improves accepted-set QWK across all three datasets.

#	Model	$\tau$	Replay	QWK ( $\mathcal{D}_{21}$ )	QWK ( $\mathcal{D}_{22}$ )	Coverage ( $\mathcal{D}_{22}$ )
1	Llama-3.2-3B	0.5	✓	0.854	$-0.071$	10.7%
2	Llama-3.1-8B	0.5	$\times$	0.863	0.025	39.7%
3	Qwen-2.5-7B	0.5	✓	0.925	0.916	17.6%
4	Qwen-2.5-7B	0.4	✓	0.721	0.882	35.1%

Appendix

Table 7: Design configurations evaluated on DAMI leading to the final CHiL(L)Grader setup. Each row isolates one design decision: model capacity (1), replay buffer (2), and threshold selection (3 vs 4). Scenario 4 is the configuration reported in the main paper.

Figure 6: HiL progression across four correction cycles for SciEntsBank (left) and EngSAF (right). Full-set QWK (blue, left axis) and automated coverage (green dashed, right axis) are shown at each split. SciEntsBank shows a non-monotonic but ultimately recovering QWK trajectory; EngSAF maintains high coverage throughout with steady QWK improvement.

Figure 7: Grade scale distribution across train and test splits of DAMI . The

G=8

scale is substantially underrepresented (

133

train,

7

test samples), motivating the scale-aware replay buffer in CHiL(L)Grader which explicitly balances representation across

G

categories during HiL adaptation.

	Original scale	Normalized (3-class)
Val QWK	0.826	0.768
Test QWK	0.705	0.430
Val $\to$ Test gap	$-$ 0.121	$-$ 0.338
$\mathcal{D}_{22}$ coverage	35.1%	4.6%
$\mathcal{D}_{22}$ accepted QWK	0.882	0.519

Appendix

Table 8: Grade granularity ablation on DAMI . The normalized variant collapses to

G\in\{0,1,2\}

Panel of Figure 8 — Figure 8: Basic prompt template. Minimal context; used as the default for instruction-tuning and all cross-dataset experiments.

Panel of Figure 9 — Figure 9: Detailed prompt template. Adds explicit grading criteria (correctness, completeness, clarity) and per-endpoint scale descriptors.

Panel of Figure 10 — Figure 10: JSON-strict prompt template. Enforces strict output formatting with explicit constraints against free-text generation.

Panel of Figure 11 — Figure 11: Rubric-based prompt template. Provides explicit percentage-based scoring descriptors applied proportionally to the grading scale.

Question (truncated)	Student Response (truncated)	Grade
To what extent do the following methods allow constituent models to be generated in parallel: Bagging, Random Forests, Boosting, Stacking?	Generated in parallel: Bagging and Random Forests. Some steps can be generated in parallel: Stacking. Not generated in parallel: Boosting.	5/10
The standard K-Means algorithm loads all data into memory. Data instead arrives in a streaming manner. What modification would you suggest?	Not necessary to modify the algorithm, just do each part individually with a constant stream of information. Use threads to treat, analyse, and send the data…	1/10
Consider a CNN with three consecutive 2 $\times$ 2 conv. layers (stride=1, no pooling). How many original pixels activate a single neuron in the 2nd non-image layer? What if stride=2?	Layer 1: $2{\times}2{=}4$ . Striding right: $+2$ , down: $+2$ , diagonal: $+1$ . Total $=9$ . With stride $=$ 2: $4{+}4{+}4{+}4{=}16$ .	5/5
Graph $G$ has 20 nodes and 190 edges (complete). Compute the indegree. What is the PageRank of any node? If half the edges are removed at random, what is the new PageRank?	Indegree $=n{-}1=19$ . Initial PageRank $=1$ (complete graph). After removing half the edges, probability halves: PageRank $=0.5$ .	2.5/5

Appendix

Table 9: Representative examples from the DAMI dataset illustrating the difficulty and subjectivity of graduate-level short-answer grading. Questions span multiple topics from a Master’s-level data mining course; grades use either a 5-point or 10-point scale depending on question complexity.

Automated Grading

Automated grading uses machine learning and language models to score student work against answer keys or rubrics and provide feedback.

See all 88 papers

When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment
Mar 31, 2026Robinson Ferrer, Damla Turgut, Zhongzhou Chen +1Confidence Estimation in Language Models Language Model Calibration
University of Central Florida
Confidence Estimation in Automatic Short Answer Grading with LLMs
Apr 30, 2026Longwei Cong, Sonja Hahn, Sebastian Gombert +3Uncertainty Quantification Selective Prediction
DIPF | Leibniz Institute for Research and Information in Education, 60323 Frankfurt am Main, Germany · Faculty of Computer Science, Goethe University Frankfurt, 60629 Frankfurt am Main, Germany · Chemnitz University of Technology, 09111 Chemnitz, Germany
Creating and Evaluating K-12 GenAI Assessment Graders Through Context Engineering
May 8, 2026Zewei Tian, Alex Liu, Lief Esbenshade +6Educational Assessment Generative AI in Education
University of Washington · Colleague AI

Mar 31, 2026J/K move · Enter open · S save

When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment

Robinson Ferrer, Damla Turgut, Zhongzhou Chen, Shashank Sonkar

University of Central Florida

Confidence Estimation in Language Models Language Model Calibration Automated Grading Selective Prediction LLM-as-a-Judge

Open paper AskPDF arXiv

Large Language Models (LLMs) show promise for automated grading, but their outputs can be unreliable. Rather than improving grading accuracy directly, we address a complementary problem: predicting when an LLM grader is likely to be correct. This enables selective automation where high-confidence predictions are processed automatically while uncertain cases are flagged for human review. We compare three confidence estimation methods (self-reported confidence, self-consistency voting, and token probability) across seven LLMs of varying scale (4B to 120B parameters) on three educational datasets: RiceChem (long-answer chemistry), SciEntsBank, and Beetle (short-answer science). Our experiments reveal that self-reported confidence consistently achieves the best calibration across all conditions (avg ECE 0.166 vs 0.229 for self-consistency). Surprisingly, self-consistency remains 38% worse despite requiring 5

\times

the inference cost. Larger models exhibit substantially better calibration though gains vary by dataset and method (e.g., a 28% ECE reduction for self-reported), with GPT-OSS-120B achieving the best calibration (avg ECE 0.100) and strong discrimination (avg AUC 0.668). We also observe that confidence is strongly top-skewed across methods, creating a ``confidence floor'' that practitioners must account for when setting thresholds. These findings suggest that simply asking LLMs to report their confidence provides a practical approach for identifying reliable grading predictions. Code is available here.

Abstract

Automated Grading

Explore similar work

When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment

Confidence Estimation in Automatic Short Answer Grading with LLMs

Creating and Evaluating K-12 GenAI Assessment Graders Through Context Engineering