TeachGrade: Training-Free Short Answer Scoring with Teacher Evidence

Jinhao Yan; Zhihong Shen; Shengjie Wang; Yiting Ren; Wentao Zhang; Yilei Wang; Peichao Lai

cs.CLOct 10, 2026

TeachGrade: Training-Free Short Answer Scoring with Teacher Evidence

Authors: Jinhao Yan, Zhihong Shen, Shengjie Wang, Yiting Ren, Wentao Zhang, Yilei Wang, Peichao Lai

Organizations: Fuzhou University · Peking University

Abstract

Multi-subject, multi-step scoring requires consistent partial credit across grading conventions. We introduce TeachGrade, a training-free framework using teacher-graded examples. The Task Adapt module builds two frozen memories: Skill captures contextual teacher preferences beyond the rubric; CaseBank supplies local evidence. PreJudge provides retrieval cues; FinalJudge regrades the answer without preliminary scores. On 3,829 SAS-Bench answers across 12 tasks, official macro collaborative consistency score (CCS) rises from Native's 67.99 to 81.32 (+13.33 points), and pooled step exact match from 54.85% to 62.71%. Macro quadratic weighted kappa (QWK) reaches 84.71, exceeding Native, OpenJudge, ProofGrader, and Trojsten by 9.52, 3.50, 11.37, and 17.43 points, respectively, and leading on 8 tasks. Local Qwen3.8-27B gains 8.83 CCS and 4.08 QWK points, improving CCS on all 12 tasks. QWK gains over Native span 6.38--14.65 points on Portuguese automatic essay scoring (AES) and 2.16--2.24 on ASAP-SAS across label views, with lower mean absolute error. These results support reuse across languages, response formats, and backbones without weight updates.

Figures & tables

Figure 1: Fine-grained scoring and cross-subject applicability. Top: evaluated workflows. Bottom: macro CCS/QWK and five largest subject-wise QWK gains over the best external adapter (one task per subject). Native directly scores current-question inputs without teacher cases, Skill, or retrieval; all use DeepSeek-V4-Flash. Tables 2 – 3 cover all 12 tasks.

Figure 2: Offline adaptation and online evidence-guided grading. The Task Adapt module specifies both memories; PreJudge guides retrieval; FinalJudge regrades without preliminary scores. Native directly scores current-question inputs without teacher cases, Skill, or retrieval. Model outputs use DeepSeek-V4-Flash. The mathematics example improves step credit at an unchanged total; teacher labels are post-hoc audit data.

Information	Content and online role
Rubric	Current-question requirements, conditions, and points; governs both judgments.
Skill $S_{t}$	Teacher-derived condition–action preferences; shared guidance for both Judges.
CaseBank $\mathcal{R}_{t}$	Local teacher observations of completion and boundaries; evidence for FinalJudge.

Table 1: Complementary scoring requirements and teacher memories.

Task	$n$	Native ( 2026 )	Trojsten ( 2025 )	ProofGrader ( 2026 )	OpenJudge ( 2025 )	TeachGrade
Physics S.A.	100	62.98	28.44	43.30	68.05	69.83
History S.A.	620	89.30	86.54	90.67	90.18	90.07
Physics M.C.	90	82.19	38.35	80.34	83.88	92.72
Geography S.A.	120	62.55	64.87	59.18	61.78	64.94
Biology F.B.	556	89.43	83.34	87.21	93.08	94.13
Chinese F.B.	119	90.93	85.98	77.77	94.14	97.48

Table 2: Task-level SAS-Bench QWK. Baselines: Native ( Lai et al., 2026 ) , Trojsten ( Zahradník and Šuppa, 2025 ) , ProofGrader ( Ma et al., 2026 ) , and OpenJudge ( The OpenJudge Team, 2025 ) . Bold indicates the highest point estimate per row. Avg. is the unweighted task mean. Question types are defined in Section 4.1 ; baseline configurations in Section 4.2 .

Metric	Native	TeachGrade
CCS $\uparrow$	67.99	81.32
QWK $\uparrow$	75.19	84.71
MAE $\downarrow$	1.548	1.248
RMSE $\downarrow$	2.301	1.907
Exact $\uparrow$	37.61	45.92

Table 3: Equal-task averages for Native and TeachGrade. MAE, RMSE, and exact match refer to totals; CCS uses the official implementation.

Setting	QWK	CCS	MAE	RMSE	Exact
Full TeachGrade	80.18	75.39	1.4448	2.2006	32.66
Raw teacher few-shot	73.33	66.89	1.8042	2.6259	25.54
Retrieved examples	76.03	70.34	1.6240	2.4178	28.01
Without rubric	78.07	74.66	1.5864	2.3647	30.76
Total-only teacher labels	75.39	74.82	1.5394	2.3333	27.91

Table 4: Input and supervision controls on four tasks and 1,306 answers. Full is the complete TeachGrade workflow; task-level results and call counts appear in the appendix.

Dataset/view	$n$	QWK	Gain	MAE
Portuguese all-raters	216	57.69 64.07	+6.38	173.704 132.963
Portuguese median-gold	72	58.39 73.04	+14.65	158.889 104.444
ASAP-SAS Score1 Avg.	1504	52.74 54.90	+2.16	0.401 0.380
ASAP-SAS Score2 Avg.	1504	50.94 53.18	+2.24	0.424 0.393

Table 5: Cross-benchmark comparisons. Each metric cell lists Native above TeachGrade; bold marks the better value. Native scores directly without teacher memories. Portuguese all-raters contains 216 ratings of 72 essays. Gain is the QWK difference.

Task	$n$	QWK		CCS
		Native	Teach Grade	Native	Teach Grade
Physics S.A.	100	63.60	66.07	45.09	60.83
History S.A.	620	91.54	92.03	83.29	91.07
Physics M.C.	90	84.95	95.99	82.60	97.03
Geography S.A.	120	60.75	61.01	55.72	59.41
Biology F.B.	556	91.91	95.00	75.82	91.12

Table 6: Local Qwen3.8-27B on the same evaluation cohort. Native is direct base-model scoring without teacher memories. Bold marks the better value in each metric pair; Avg. gives the equal-task mean.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Full	Raw few-shot	Retrieved	No rubric	Total only
Physics S.A.	69.83	60.96	65.86	62.73	68.77
Math S.A.	86.14	85.18	85.38	86.79	83.72
Politics S.A.	92.24	84.34	89.39	90.42	93.86
English F.B.	72.52	62.83	63.49	72.36	55.19

Appendix

Table A1: Task-level QWK for the four-task controls. Bold marks the best value per task.

Task	Full	Raw few-shot	Retrieved	No rubric	Total only
Physics S.A.	60.14	52.55	58.29	59.52	67.39
Math S.A.	80.73	70.99	74.36	81.08	73.32
Politics S.A.	86.30	78.34	84.27	85.42	85.70
English F.B.	74.37	65.68	64.45	72.61	72.85

Appendix

Table A2: Task-level CCS for the four-task controls under the official frozen-order protocol.

System	Knowledge K	QWK $\uparrow$	CCS $\uparrow$	MAE $\downarrow$
SAS-Bench Native	0	81.26	66.63	–
Adaptive K10	10	84.93	78.16	1.307
Adaptive K20	20	86.80	80.88	1.163
Adaptive K30	30	85.91	80.60	1.179
Adaptive K40	40	86.03	80.64	1.219

Appendix

Table A3: Teacher-pool sensitivity on a shared mathematics development cohort, not pooled with the final mathematics evaluation.

Task	$n$	N QWK	O QWK	A QWK	N CCS	O CCS	A CCS
Physics S.A.	100	62.98	58.68	69.83	52.26	58.54	60.14
History S.A.	620	89.30	87.81	90.07	80.98	88.20	89.74
Physics M.C.	90	82.19	91.37	92.72	86.96	90.97	93.68
Geography S.A.	120	62.55	61.07	64.94	56.15	60.65	62.85
Biology F.B.	556	89.43	93.86	94.13	72.74	90.02	90.07
Chinese F.B.	119	90.93	98.22	97.48	86.07	95.78	94.85

Appendix

Table A4: Complete Native (N), Original (O), and final Adaptive (A) comparisons. Adaptive is the final TeachGrade system.

System	QWK $\uparrow$	CCS $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$	Bias $\to 0$	Exact $\uparrow$
Native	75.19	67.99	1.548	2.301	-0.300	37.61
Original TeachGrade	81.01	80.03	1.460	2.194	-0.818	45.70
Adaptive TeachGrade	84.71	81.32	1.248	1.907	-0.458	45.92

Appendix

Table A5: Complete Original/final macro comparison. Negative Bias denotes underestimation.

Figure A1: Task-level QWK differences between Original and the final adaptive system. Positive and negative values indicate improvement and decline. Values correspond to Table A4 .

Group	System	QWK $\uparrow$	CCS $\uparrow$	MAE $\downarrow$	RMSE $\downarrow$
Open humanities	Native	80.91	68.81	1.999	2.764
Open humanities	Original	73.57	73.51	2.442	3.421
Open humanities	Adaptive	81.49	78.35	1.945	2.687
Structured science	Native	73.98	65.84	1.125	1.852
Structured science	Original	91.00	88.73	0.778	1.327
Structured science	Adaptive	91.14	88.09	0.744	1.305

Appendix

Table A6: Internal development comparison of open-ended humanities and structured-science tasks.

Evidence in the student answer	Teacher	Original	Adaptive
Counters attempts to discredit heroes using sound values, historical facts, and core values	6	4	6
Defends heroes by reconstructing history, identifying misinformation, and promoting the national spirit	6	4	6
Calls for learning from and practicing the spirit of heroes	3	2	3
Total	15	10	15

Appendix

Table A7: Step evidence and scores for Political_ShortAns_79. Text summarizes the original answer’s meaning. Bold highlights corrected scores matching the teacher.

Dimension	Original TeachGrade	Adaptive TeachGrade
Credit unit	Generic task-level point-based guidance	Defines key principles in the current reference as credit owners
Equivalence	Allows similar meanings without evidence-specific scope	Credits equivalent statements; excludes incorrect principles or confused concepts
Contrasts	Cases primarily record the teacher’s awarded points	Records higher/lower boundaries , supporting and contrasting cases, and confidence markers
Duplication and dependency	Generic Skill warns against duplicate credit	Counts repeated owners once ; propagates earlier errors only with supporting evidence
Score scale	Historical points may implicitly anchor the current score	Transfers completion fractions under the current budget

Appendix

Table A8: Original and adaptive scoring representations; neither changes the question-specific rubric. Bold highlights the adapted representation’s key distinctions.

Metric	PreJudge	FinalJudge	Gain and 95% interval
Avg. QWK	83.69	84.71	$1.02\ [-0.28,2.35]$
Avg. total MAE	1.3378	1.2479	$0.09\ [0.04,0.13]$
Avg. total exact match	43.38	45.92	$2.54\ [0.83,4.23]$
Pooled step MAE	0.6176	0.6012	$0.0164$ (see Table A10 )
Pooled step exact match	60.09	62.71	$2.62\ [1.87,3.34]$

Appendix

Table A9: Full paired regrading audit. Gains are oriented toward improvement; QWK’s interval includes zero.

Contrast	Estimate	95% interval
Step exact gain vs. Native	7.86	[6.95,8.79]
Step MAE reduction vs. Native	0.2666	[0.2419,0.2920]
Step exact gain vs. PreJudge	2.62	[1.89,3.37]
Step MAE reduction vs. PreJudge	0.0164	[0.0040,0.0293]

Appendix

Table A10: Question-cluster paired step gains. Positive values indicate improvement by TeachGrade.

Task	Native	PreJudge	TeachGrade
Physics S.A.	58.00 / 0.602	62.36 / 0.491	64.91 / 0.491
History S.A.	30.28 / 1.858	31.12 / 1.268	34.77 / 1.275
Physics M.C.	67.03 / 0.560	86.81 / 0.209	82.42 / 0.275
Geography S.A.	44.59 / 0.947	44.85 / 0.858	48.81 / 0.818
Biology F.B.	60.71 / 1.039	72.89 / 0.442	69.58 / 0.466
Chinese F.B.	89.86 / 0.125	94.49 / 0.067	94.78 / 0.061

Appendix

Table A11: All-task step accuracy. Each cell reports exact match / MAE.

Plan component	Stored purpose
Credit semantics	Principles and their application to source material; current-reference credit owners.
Evidence and scope	Teacher behavior, higher/lower contrasts, equivalence, dependencies, duplication, and source IDs.
Skill plan	Workflow, principles, partial credit, deductions, zero credit, high credit, and boundaries.
CaseBank plan	Retrieval goals, owner mapping, behavior/error patterns, dependencies, and relative boundaries.
Consumption contract	Completion-based references interpreted under the current budget; historical absolute points hidden.

Appendix

Table A12: Construction plan extracted from the actual Politics specification.

Layer	Content
Identity	Case/source/step IDs, question summary, complete local answer, reference node.
Teacher observation	2 points; observed budget 3; question maximum 26; incomplete-work label.
Retrieval	Fraction 0.6667; band 7; question, step, behavior, reference, error, and adapted goals.
Boundary status	Same-question, same-owner contrast; construction marker 1.0.
FinalJudge view	67%, credit point, behavior, relative boundary, scope, and current-budget interpretation.
Audit-only fields	Historical absolute points, source budget, confidence, and original construction records.

Appendix

Table A13: Stored and visible layers of the actual case K1_S4 .

Task	K	Cases	Eval. steps	Cases/step	Unique used	Utilization
Physics S.A.	20	32	550	6.00	31	96.88%
Math S.A.	20	101	4234	6.00	97	96.04%
Politics S.A.	20	95	916	6.00	88	92.63%
English F.B.	20	199	2158	6.00	113	56.78%

Appendix

Table A14: Observed retrieval coverage. Case utilization measures breadth of use, not retrieval accuracy.

Automated Grading

Automated grading uses machine learning and language models to score student work against answer keys or rubrics and provide feedback.

See all 88 papers

Grading Needs a Rubric, Not Intelligence
Aug 18, 2026Jhen-Ke LinAutomated Grading Rubric-Based Evaluation
National Yang Ming Chiao Tung University
Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring
Oct 7, 2026Zhifan Sun, Sebastian Gombert, Jannik Lossjew +6Automated Grading Educational Assessment
DIPF | Leibniz Institute for Research and Information in Education · IPN | Leibniz Institute for Science and Mathematics Education · Umeå University +1
EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Grading
Jun 4, 2026Zhihao Wu, Linhai Zhang, Taiyi Wang +4LLM-as-a-Judge Rubric-Based Evaluation
1King’s College London · University of Cambridge · 3AQA +1

Aug 18, 2026J/K move · Enter open · S save

Grading Needs a Rubric, Not Intelligence

Jhen-Ke Lin

National Yang Ming Chiao Tung University

Automated Grading Rubric-Based Evaluation LLM-as-a-Judge LLM Reliability Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

Small language models can grade open-ended examination answers as reliably as substantially more expensive models when they grade against an explicit rubric. We test this claim as the design principle behind any-to-bench: a frontier model reads source documents once, at ingestion, to extract each question and its rubric; lower-cost models then perform all repeated grading work. We evaluate six cost-efficient model configurations from two model families at three reasoning-effort levels. Each configuration answers 24 open-ended examination questions, and each also grades every answer sheet three times, yielding 3,456 per-question grades. Scores depend overwhelmingly on the answer being graded: answer identity explains 95.6% of score variance, whereas judge identity explains only 0.2%. Raising a writer's reasoning effort moves earned scores by as much as 0.143 of full marks, while raising a judge's reasoning effort moves assigned scores by at most 0.006. Six frontier-tier judges, added as a check, reproduce these scores and are no more reliable as a panel. Two ablations then decompose the rubric on the same questions and answers. Removing its criteria and levels while keeping the official answer changes nothing measurable. Removing the official answer as well collapses reliability (ICC 0.888 to 0.628), inflates scores, and makes judge reasoning effort matter again. The rubric is what decouples grading from judge intelligence, and within the rubric the official answer does nearly all the work. We find no evidence of length preference or same-family preference under rubric-anchored grading.

Abstract

Automated Grading

Explore similar work

Grading Needs a Rubric, Not Intelligence

Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring

EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Grading