Autoscoring Anticlimax: A Meta-analytic Understanding of AI's Short-answer Shortcomings and Wording Weaknesses

Michael Hardy

cs.CLMar 5, 2026

Autoscoring Anticlimax: A Meta-analytic Understanding of AI's Short-answer Shortcomings and Wording Weaknesses

Authors: Michael Hardy

Organizations: Stanford University

Abstract

Automated short-answer scoring lags other LLM applications. We meta-analyze 890 culminating results across a systematic review of LLM short-answer scoring studies, modeling the traditional effect size of Quadratic Weighted Kappa (QWK) with mixed effects metaregression. We quantitatively illustrate that that the level of difficulty for human experts to perform the task of scoring written work of children has no observed statistical effect on LLM performance. Particularly, we show that some scoring tasks measured as the easiest by human scorers were the hardest for LLMs. Whether by poor implementation by thoughtful researchers or patterns traceable to autoregressive training, on average decoder-only architectures underperform encoders by 0.37--a substantial difference in agreement with humans. Additionally, we measure the contributions of various aspects of LLM technology on successful scoring such as tokenizer vocabulary size, which exhibits diminishing returns--potentially due to undertrained tokens. Findings argue for systems design which better anticipates known statistical shortcomings of autoregressive models. Finally, we provide additional experiments to illustrate wording and tokenization sensitivity and bias elicitation in high-stakes education contexts, where LLMs demonstrate racial discrimination. Code and data for this study are available.

Figures & tables

Figure 1: LM Benchmarks over time from Kiela et al. (2023) . Short Answer Scoring of K12 student wring on the ASAP-SAS dataset, in purple, are added using the same scaling: human score is set to 0 and baseline performance to -1, and results, X, are scaled as $(X-\text{Human})/|\text{Baseline}-\text{Human}|$ . Human and 2012 Baseline results are from Shermis . Point (A) represents the best LLM model, a transformer ensemble, Ormerod (2022) . Point (B) Best GPT model ensemble, a fine-tuned ensemble, Ormerod and Kwako (2024) . Point (C) [not plotted] is the best implementation using GPT-4 Jiang and Bosch (2024) with prompt engineering and would have a value of -1.52 on this scale. While the plot ends in 2025, to the best of our knowledge this chart still reflects SOTA for this task.

“White” student	“Black” student
ChatGPT’s assigned score: 1 (out of 2)	ChatGPT’s assigned score: 0 (out of 2)
Feedback:	Feedback:
1. Spelling and Grammar: There are several spelling and	1. Spelling and Grammar: There are numerous spelling and grammar
grammar errors throughout the response. For example,	errors throughout the response. For example, “excited” is misspelled
“excited” is misspelled as “exited,” “antiqued” should	as “exited,” “an c tiqued” should be “questioned,” “youre” should be
be “questioned,” “youre” should be “your,” “anoyed”	“your,” “anoyed” should be “annoyed,” “alein’s” should be “alien’s,”

Table 1: Racial Bias Autoscoring Example: Score and feedback for identical student work using ChatGPT

	(1)	(2)	(3)	(4)	(5)	(6)	(6, Bayes)
tok:BPE	0.75*** [0.47, 0.89]	0.78*** [0.52, 0.91]	0.80** [0.41, 0.94]	0.81*** [0.51, 0.93]	0.80*** [0.51, 0.93]	0.72*** [0.63, 0.79]	0.78*** [0.50, 0.92]
tok:Unigram	0.66*** [0.33, 0.85]	0.73*** [0.44, 0.89]	0.76** [0.33, 0.93]	0.77*** [0.44, 0.92]	0.77*** [0.44, 0.91]	0.71*** [0.63, 0.78]	0.76*** [0.44, 0.90]
tok:WordP	0.65*** [0.33, 0.84]	0.74*** [0.45, 0.89]	0.78** [0.37, 0.93]	0.78*** [0.46, 0.92]	0.77*** [0.46, 0.92]	0.72*** [0.64, 0.79]	0.77*** [0.48, 0.91]
$\|$ vocab $\|$	0.35** [0.13, 0.54]	-0.03 [-0.24, 0.18]	0.12 [-0.12, 0.35]	0.04 [-0.15, 0.23]	0.04 [-0.15, 0.22]	0.10 [-0.04, 0.24]	0.10 [-0.04, 0.24]
$\|$ vocab $\|^{2}$	-0.12* [-0.22, -0.01]	0.00 [-0.08, 0.08]	-0.07 [-0.16, 0.03]	-0.03 [-0.10, 0.04]	-0.03 [-0.10, 0.04]	-0.06* [-0.12, -0.01]	-0.06* [-0.11, -0.00]
$\text{QWK}_{\text{hum}}$	0.03 [-0.21, 0.27]	0.03 [-0.22, 0.28]	0.03 [-0.22, 0.28]	0.02 [-0.20, 0.24]	0.03 [-0.19, 0.24]	-0.02 [-0.07, 0.04]	0.00 [-0.27, 0.28]

Table 2: Meta-regression Estimated Parameters

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	Prompt Slots	Experimental Input
(a)	Learning environment	“third grade math class”
(b)	Disability category	“student is ***”
(c)	Current challenges	“struggling in class”
(d)	Current strengths	“good at sports”

Appendix

Table 3: IEP assistant input set-up

Category	Adjustments/Interventions	Black	Hispanic	White	p-val ∗
		N=25	N=23	N=21
Social Development	Group or team class activities	100%	100%	100%
	Structured peer interactions	84%	70%	95%	0.082
	Student mentorship	8%	0%	24%	0.025
	Class leadership opportunities	40%	57%	24%	0.093
Math Content Remediation	Small group or 1-on-1 instruction (teacher)	28%	43%	81%	0.001

Appendix

Table 4: AI Generated IEP Interventions by Student Race/Ethnicity

Figure 2: Khanmigo IEP Assistant Entry Page

	Token Len.	Train	Valid	Test	Assessment Area
Q1	$47.5\pm 22.2$	1,341	331	557	Science
Q2	$59.2\pm 22.6$	1,024	254	426	Science
Q3	$47.9\pm 14.6$	1,445	363	406	Reading (Informational Text)
Q4	$40.3\pm 15.5$	1,308	349	295	Reading (Informational Text)
Q5	$25.1\pm 21.5$	1,459	336	598	Science
Q6	$23.8\pm 22.6$	1,418	379	599	Science

Appendix

Table 5: ASAP-SAS dataset detail by item.

Figure 3: QWK distribution of LLMs on ASAP-SAS dataset for meta-analysis models relative to human performance

Characteristic	Jiang & Bosch	Kim et al.	Ormerod	Ormerod & Kwako	Sivkumar et al.	Surya et al.	Todorov et al.	Zeng et al.
Year	2024	2025	2022	2024	2026	2019	2025	2023
N	60	90	130	80	360	10	40	120
qwk	0.63 (0.52, 0.70)	0.72 (0.65, 0.80)	0.76 (0.70, 0.82)	0.61 (0.34, 0.75)	0.45 (0.30, 0.58)	0.72 (0.66, 0.79)	0.74 (0.68, 0.79)	0.69 (0.61, 0.73)
token
bpe	60 (100%)	70 (78%)	20 (15%)	80 (100%)	300 (83%)	0 (0%)	0 (0%)	0 (0%)
uni	0 (0%)	0 (0%)	30 (23%)	0 (0%)	60 (17%)	0 (0%)	0 (0%)	0 (0%)

Appendix

Table 6: Included Studies and Properties of metadataset

term	npar	AIC	BIC	log( $\mathcal{L}$ )	-2log( $\mathcal{L}$ )	statistic	df	p.value
(1)	12	-359.95	-302.46	191.98	-383.95	NA	NA	NA
(2)	13	-418.17	-355.88	222.09	-444.17	60.22	1	0.00
(3)	14	-711.65	-644.58	369.83	-739.65	295.48	1	0.00
(4)	20	-974.69	-878.87	507.35	-1014.69	275.04	6	0.00
(5)	23	-969.24	-859.04	507.62	-1015.24	0.55	3	0.91
(6)	175	-845.64	-7.17	597.82	-1195.64	180.40	152	0.06

Appendix

Table 7: Anova results for main body models.

model	npar	AIC	BIC	logLik	stat.	df	p.val
m1	11	-495.89	-449.75	258.95
m2	12	-494.00	-443.67	259.00	0.11	1	0.74
m3	12	-493.96	-443.63	258.98	0.00	0
m4	13	-492.07	-437.54	259.04	0.11	1	0.74
m5	15	-488.15	-425.23	259.08	0.08	2	0.96
m6	20	-506.12	-422.23	273.06	27.97	5	0.00

Appendix

Table 8: Anova results for Appendical Nine models fit on selected data.

	m1	m6	m7	m9
(Intercept)	0.69**	0.73***
	[0.25, 0.89]	[0.43, 0.88]
$\operatorname{QWK}_{\text{hum}}$	0.06	0.06	-0.06	-0.06
	[-0.17, 0.29]	[-0.17, 0.29]	[-0.21, 0.08]	[-0.20, 0.09]
read	-0.20*	-0.21**	-0.24***	-0.24***
	[-0.35, -0.05]	[-0.35, -0.05]	[-0.33, -0.15]	[-0.37, -0.11]

Appendix

Table 9: Quick Summary of Four Models from Robustness Checks

	m1	m2	m3	m4	m5	m6	m7	m8	m9
(Intercept)	0.69**	0.69**	0.69**	0.69**		0.73***
	[0.25, 0.89]	[0.25, 0.89]	[0.25, 0.89]	[0.25, 0.89]		[0.43, 0.88]
$\operatorname{QWK}_{\text{hum}}$	0.06	0.06	0.06	0.06	0.06	0.06	-0.06	-0.06	-0.06
	[-0.17, 0.29]	[-0.17, 0.29]	[-0.17, 0.29]	[-0.17, 0.29]	[-0.17, 0.29]	[-0.17, 0.29]	[-0.21, 0.08]	[-0.20, 0.09]	[-0.20, 0.09]
read	-0.20*	-0.20*	-0.21**	-0.21**	-0.21**	-0.21**	-0.24***	-0.24***	-0.24***
	[-0.35, -0.05]	[-0.35, -0.05]	[-0.35, -0.05]	[-0.35, -0.05]	[-0.35, -0.05]	[-0.35, -0.05]	[-0.33, -0.15]	[-0.37, -0.10]	[-0.37, -0.11]

Appendix

Table 10: Full Estimate Table for nine more appendical models under variable permutations.

Explore similar work

Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation
May 8, 2026Abigail Victoria Gurin Schleifer, Moriah Ariely, Beata Beigman Klebanov +2LLM Evaluation AI in Education
Weizmann Institute of Science, Rehovot, Israel · ETS, Princeton, USA
LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias
Mar 31, 2026Filip J. Kucia, Anirban Chakraborty, Anna WróblewskaAutomated Essay Scoring Language Model Bias Evaluation
Faculty of Mathematics and Information Science, Warsaw University of Technology, 00-662 Warsaw, Poland · University of Wolverhampton, UK
Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors
Mar 26, 2026Cole Walsh, Rodica IvanLanguage Model Robustness Educational Assessment
Acuity Insights Inc., Toronto, ON, Canada

May 8, 2026J/K move · Enter open · S save

Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation

Abigail Victoria Gurin Schleifer, Moriah Ariely, Beata Beigman Klebanov, Asaf Salman+1

Weizmann Institute of Science, Rehovot, Israel · ETS, Princeton, USA

LLM Evaluation AI in Education Few-Shot Learning Automated Grading

Open paper AskPDF arXiv

Automated short answer scoring (ASAS) is shifting from discriminative, fine-tuned models to large language models (LLMs) used in few-shot settings. This paradigm leverages LLMs broad world knowledge and ease of deployment, but limited task-specific data may reduce alignment on complex scoring tasks. In particular, its impact on scoring partially correct responses that require nuanced interpretation remains underexplored. We investigate the relationship between the degree of task-specific adaptation of different models and quality-conditioned scoring agreement. We compare three LLMs (GPT-5.2, GPT-4o, Claude Opus 4.5) in few-shot mode, a fine-tuned BERT-based encoder, and a human expert on two open-ended biology items, using several hundred student responses and ground truth scores provided by a biology education expert. The results show that human-human agreement is highest and stable across the full quality spectrum. All AI models perform well on fully correct and fully incorrect responses, but exhibit substantial degradation on mid-range responses. This mid-range degradation is conditioned on task-specific adaptation: It is most severe in few-shot LLMs with few examples and decreases as task-specific data increases, with fine-tuned encoder models performing best. This mid-range degradation may lead to inequitable evaluation of responses produced by students with developing understanding. Our findings highlight the importance of quality-conditioned fairness, with particular attention to mid-range responses.

Abstract

Explore similar work

Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation

LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias

Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors