Confusion-Aware Rubric Optimization for LLM-based Automated Grading

Yucheng Chu; Hang Li; Kaiqi Yang; Yasemin Copur-Gencturk; Joseph Krajcik; Namsoo Shin; Jiliang Tang

cs.AIFeb 28, 2026

Confusion-Aware Rubric Optimization for LLM-based Automated Grading

Authors: Yucheng Chu, Hang Li, Kaiqi Yang, Yasemin Copur-Gencturk, Joseph Krajcik, Namsoo Shin, Jiliang Tang

Organizations: Michigan State University · University of Southern California

Abstract

Accurate and unambiguous guidelines are critical for large language model (LLM) based graders, yet manually crafting these prompts is often sub-optimal as LLMs can misinterpret expert guidelines or lack necessary domain specificity. Consequently, the field has moved toward automated prompt optimization to refine grading guidelines without the burden of manual trial and error. However, existing frameworks typically aggregate independent and unstructured error samples into a single update step, resulting in "rule dilution" where conflicting constraints weaken the model's grading logic. To address these limitations, we introduce Confusion-Aware Rubric Optimization (CARO), a novel framework that enhances accuracy and computational efficiency by structurally separating error signals. CARO leverages the confusion matrix to decompose monolithic error signals into distinct modes, allowing for the diagnosis and repair of specific misclassification patterns individually. By synthesizing targeted "fixing patches" for dominant error modes and employing a diversity-aware selection mechanism, the framework prevents guidance conflict and eliminates the need for resource-heavy nested refinement loops. Empirical evaluations on teacher education and STEM datasets demonstrate that CARO significantly outperforms existing SOTA methods. These results suggest that replacing mixed-error aggregation with surgical, mode-specific repair yields robust improvements in automated assessment scalability and precision.

Figures & tables

Figure 1: Framework of the CARO algorithm.

Panel of Figure 2 — Figure 2: Example Reflector prompt for analyzing the dominant error mode $(0\to 1)$ . The prompt provides: (1) global context via the full confusion matrix, (2) local error examples with model reasoning traces, and (3) contrastive correct examples to establish decision boundaries.

Panel of Figure 3 — Figure 3: Structured diagnosis output from the Reflector for mode $(0\to 1)$ . The output identifies root causes, misleading patterns, and proposes targeted rule modifications with safety considerations.

Panel of Figure 4 — Figure 4: Refiner prompt structure showing how cross-mode awareness is incorporated. The Refiner receives information about all active error modes to generate compatible rule fixes.

Panel of Figure 5 — Figure 5: Two-phase rule consolidation. Phase 1 generates per-mode rules with cross-mode awareness. Phase 2 synthesizes them into a priority-weighted consolidated ruleset with explicit conflict resolution directives.

Dataset	Domain	#Samples	Scores
$\mathcal{D}_{I}$	Interaction	252	{0,1}
$\mathcal{D}_{T}$	Teacher Education	224 $\sim$ 236	{0,1,2}
$\mathcal{D}_{E}$	Elementary	199 $\sim$ 434	{0,1,2,3,4}

Table 1: A summary of statistics of the datasets

Type	Task	Naive		GradeOpt		CARO
		Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$
KSMT	T1	0.40	0.20	0.61	0.41	0.67	0.47
KSMT	T2	0.58	0.42	0.74	0.61	0.79	0.69
KSMT	T3	0.39	0.09	0.51	0.25	0.55	0.30
KSMT	Avg	0.46	0.24	0.62	0.42	0.67	0.49
KMT	T4	0.30	0.05	0.55	0.28	0.76	0.57

Table 2: Performance Comparison for Teacher Education Dataset

\mathcal{D}_{T}

Category	Naive		GradeOpt		CARO
	Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$
1	0.94	0.79	0.96	0.84	0.98	0.92
2	0.86	0.71	0.88	0.76	0.93	0.87
3	0.82	0.52	0.85	0.61	0.95	0.89
4	0.52	0.19	0.64	0.36	0.81	0.62
5	0.62	0.18	0.95	0.72	0.96	0.78

Table 3: Performance Comparison for Interaction Dataset

\mathcal{D}_{I}

Task	Naive		GradeOpt		CARE
	Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$
T1	0.16	-0.08	0.63	0.35	0.63	0.39
T2	0.49	0.30	0.74	0.64	0.75	0.66
T3	0.57	0.25	0.79	0.67	0.81	0.70
T4	0.48	0.20	0.71	0.57	0.74	0.62
T5	0.50	0.04	0.70	0.46	0.73	0.56

Table 4: Performance Comparison for EIR Dataset

\mathcal{D}_{E}

Dataset	Naive		GradeOpt		CARO
	Acc	$\kappa$	Acc	$\kappa$	Acc	$\kappa$
$\mathcal{D}_{I}$ (avg)	0.58	0.25	0.75	0.43	0.90	0.55
$\mathcal{D}_{T}$ (avg)	0.51	0.27	0.65	0.44	0.72	0.53
$\mathcal{D}_{E}$ (avg)	0.44	0.14	0.71	0.54	0.73	0.59
Overall	0.51	0.22	0.70	0.47	0.78	0.56

Table 5: Summary of average performance across datasets.

Dataset	Per-Mode	Consolidated	Best $\kappa$
$\mathcal{D}_{I}$	93.9%	6.1%	Per-Mode
$\mathcal{D}_{T}$	91.7%	8.3%	Per-Mode
$\mathcal{D}_{E}$	93.3%	6.7%	Per-Mode
Avg	93.0%	7.0%	-

Table 6: Ablation: Per-mode vs. consolidated rule selection. Values show the percentage of final best prompts using each strategy.

Panel of Figure 6 — Figure 6: Convergence comparison on T4 ( $\mathcal{D}_{I}$ ). CARO achieves faster convergence and higher final accuracy.

Method	Rounds	API Calls	Est. Cost
GradeOpt	6	$\sim$ 40,000	$\sim$ $5
CARO	6	$\sim$ 8,000	$\sim$ $2
Reduction	–	80%	60%

Table 7: API cost comparison (average per task). Costs estimated using GPT-4o-mini pricing (

0.15/1M input,

0.60/1M output tokens).

Round 0	Round 2	Round 4
$\begin{bmatrix}2&1&0&0&1\\ 7&41&0&0&0\\ 1&\pagecolor{red!30}\mathbf{24}&0&0&0\\ 0&\pagecolor{red!30}\mathbf{8}&2&0&0\\ 0&1&0&0&0\end{bmatrix}$	$\begin{bmatrix}\pagecolor{green!25}4&0&0&0&0\\ 4&35&\pagecolor{orange!25}9&0&0\\ 0&\pagecolor{green!25}4&\pagecolor{green!25}\mathbf{21}&0&0\\ 0&0&10&0&0\\ 0&0&1&0&0\end{bmatrix}$	$\begin{bmatrix}2&1&0&0&1\\ 3&\pagecolor{green!25}\mathbf{40}&\pagecolor{green!25}5&0&0\\ 0&4&\pagecolor{green!25}\mathbf{21}&0&0\\ 0&0&9&\pagecolor{green!25}1&0\\ 0&0&1&0&0\end{bmatrix}$
Acc = 0.49, $\kappa$ = 0.02	Acc = 0.68, $\kappa$ = 0.49	Acc = 0.73, $\kappa$ = 0.55

Figure 7: Confusion matrix progression on T5 (

\mathcal{D}_{E}

). Rows represent true labels; columns represent predictions. CARO’s iterative refinement first resolves the dominant

2\to 1

collapse (Round 0–2), then addresses the emergent

1\to 2

over-correction (Round 2–4), achieving balanced diagonal improvement.

Automated Grading

Automated grading uses machine learning and language models to score student work against answer keys or rubrics and provide feedback.

See all 89 papers

Optimizing In-Context Demonstrations for LLM-based Automated Grading
Feb 28, 2026Yucheng Chu, Hang Li, Kaiqi Yang +4Education Automated Grading
Michigan State University · University of Southern California
EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Grading
Jun 4, 2026Zhihao Wu, Linhai Zhang, Taiyi Wang +4LLM-as-a-Judge Rubric-Based Evaluation
1King’s College London · University of Cambridge · 3AQA +1
REC-CBM: Rubric-Aware Error-Correction Concept Bottleneck Models for Trustworthy Open-Ended Grading
Apr 24, 2026Chengshuai Zhao, Fan Zhang, Kumar Satvik Chaudhary +4Concept Bottleneck Models Automated Essay Scoring
School of Computing and Augmented Intelligence, Arizona State University, USA · Mary Lou Fulton Teachers College, Arizona State University, USA · Department of Computer Science, National Yang Ming Chiao Tung University, TW

Feb 28, 2026J/K move · Enter open · S save

Optimizing In-Context Demonstrations for LLM-based Automated Grading

Yucheng Chu, Hang Li, Kaiqi Yang, Yasemin Copur-Gencturk+3

Michigan State University · University of Southern California

Education Automated Grading In-Context Example Selection AI in Education Rubric-Based Evaluation

Open paper AskPDF arXiv

Automated assessment of open-ended student responses is a critical capability for scaling personalized feedback in education. While large language models (LLMs) have shown promise in grading tasks via in-context learning (ICL), their reliability is heavily dependent on the selection of few-shot exemplars and the construction of high-quality rationales. Standard retrieval methods typically select examples based on semantic similarity, which often fails to capture subtle decision boundaries required for rubric adherence. Furthermore, manually crafting the expert rationales needed to guide these models can be a significant bottleneck. To address these limitations, we introduce GUIDE (Grading Using Iteratively Designed Exemplars), a framework that reframes exemplar selection and refinement in automated grading as a boundary-focused optimization problem. GUIDE operates on a continuous loop of selection and refinement, employing novel contrastive operators to identify "boundary pairs" that are semantically similar but possess different grades. We enhance exemplars by generating discriminative rationales that explicitly articulate why a response receives a specific score to the exclusion of adjacent grades. Extensive experiments across datasets in physics, chemistry, and pedagogical content knowledge demonstrate that GUIDE significantly outperforms standard retrieval baselines. By focusing the model's attention on the precise edges of rubric, our approach shows exceptionally robust gains on borderline cases and improved rubric adherence. GUIDE paves the way for trusted, scalable assessment systems that align closely with human pedagogical standards.

Abstract

Automated Grading

Explore similar work

Optimizing In-Context Demonstrations for LLM-based Automated Grading

EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Grading

REC-CBM: Rubric-Aware Error-Correction Concept Bottleneck Models for Trustworthy Open-Ended Grading