S-GRADES -- Studying Generalization of Student Response Assessments in Diverse Evaluative Settings

Tasfia Seuti; Sagnik Ray Choudhury

cs.CLMar 10, 2026

S-GRADES -- Studying Generalization of Student Response Assessments in Diverse Evaluative Settings

Authors: Tasfia Seuti, Sagnik Ray Choudhury

Organizations: University of North Texas, Denton, TX, USA

Abstract

Evaluating student responses, from long essays to short factual answers, is a key challenge in educational NLP. Automated Essay Scoring (AES) focuses on holistic writing qualities such as coherence and argumentation, while Automatic Short Answer Grading (ASAG) emphasizes factual correctness and conceptual understanding. Despite their shared goal, these paradigms have progressed in isolation with fragmented datasets, inconsistent metrics, and separate communities. We introduce S-GRADES (Studying Generalization of Student Response Assessments in Diverse Evaluative Settings), a web-based benchmark that consolidates 14 diverse grading datasets under a unified interface with standardized access and reproducible evaluation protocols. The benchmark is fully open-source and designed for extensibility, enabling continuous integration of new datasets and evaluation settings. To demonstrate the utility of S-GRADES, we evaluate three state-of-the-art large language models across the benchmark using multiple reasoning strategies in prompting. We further examine the effects of exemplar selection and cross-dataset exemplar transfer. Our analyses illustrate how benchmark-driven evaluation reveals reliability and generalization gaps across essay and short-answer grading tasks, highlighting the importance of standardized, cross-paradigm assessment.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the S-GRADES benchmark user workflow.

Task Type	Dataset	Domain(s)	Score Range(s)	Public	Test Split Size
AES	ASAP-AES	Persuasive, Narrative, and Source-based Essays	Set 1: 2–12; Set 2: 1–6; Sets 3–4: 0–3; Sets 5–6: 0–4; Set 7: 0–30; Set 8: 0–60	Yes	1,298
	ASAP++	Argumentative and Expository Essays	0–6	Yes	1,069
	ASAP2.0	Extended Essay Scoring (Multi-domain)	0–6	Yes	4,946
	Persuade_2	Persuasive Writing (Argumentative)	1–6	Yes	2,600
	IELTS Writing (General)	General English Writing Tasks	1–9	Yes	144
	IELTS Writing (Task 2)	Academic English Writing Tasks	1–9	Yes	491

Table 1: Summary of datasets included in the S-GRADES benchmark. For datasets that are not currently publicly hosted online, we obtained creator permission for research use and sharing.

Figure 2: Complete benchmark submission interface.

Figure 3: Public leaderboard displaying aggregated results across all datasets and evaluation metrics.

Figure 5: QWK scores for ASAG regression datasets across models and reasoning strategies.

Figure 6: F1 scores across reasoning strategies for 2 classification-based ASAG datasets.

Model	QWK Std
Gemini-2.5-Flash	0.0047
GPT-4o-mini	0.0099
LLaMA-4-Scout	0.0111

Table 2: Exemplar sampling stability on ASAP-AES across three random seeds (42, 123, 456) at temperature = 0. Low standard deviations indicate that evaluation outcomes are largely invariant to the selection of examples.

Model	Reasoning	$\Delta$ QWK%
In-paradigm generalizations
AES: ASAP2.0 $\rightarrow$ ASAP-AES
GPT-4o-mini	Ind+Ded	-5.3
LLaMA-4-Scout	Ind+Ded	-96.2
Gemini-2.5-Flash	Ind+Ded	+9.4
ASAG : ASAP-SAS $\rightarrow$ CSEE

Table 3: Exemplar generalization results. For inductive (5-shot) reasoning setups, exemplars are selected from one dataset and used on another.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Strategy variance (

\sigma_{m,d}

) across 6 reasoning strategies per AES dataset. Higher values indicate greater sensitivity to reasoning strategy selection.

Figure 8: Strategy variance (

\sigma_{m,d}

) across 6 reasoning strategies per ASAG regression dataset. Higher values indicate greater sensitivity to the selection of reasoning strategies.

Strategy	Phase 1	Phase 2
Ind+Ded	Learn patterns from examples (Inductive)	Apply general scoring rules (Deductive)
Ind+Abd	Learn patterns from examples (Inductive)	Infer best explanation (Abductive)
Ded+Abd	Apply general scoring rules (Deductive)	Infer best explanation (Abductive)

Appendix

Table 4: Hybrid strategy prompt composition. Each hybrid combines the system prompt blocks of two solo strategies sequentially.

Figure 9: Pooled mean standard deviation (

\overline{\sigma}

) on numeric datasets across six reasoning strategies and three models. Lower is better.

Figure 10: Pooled mean agreement (%) on categorical datasets across six reasoning strategies and three models. Higher is better.

Figure 11: Per-dataset prediction stability for GPT-4o-mini across six reasoning strategies.

Figure 12: Per-dataset prediction stability for Gemini-2.5-Flash across six reasoning strategies.

Figure 13: Per-dataset prediction stability for LLaMA-4-Scout across six reasoning strategies.

Automated Grading

Automated grading uses machine learning and language models to score student work against answer keys or rubrics and provide feedback.

See all 88 papers

Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory
Apr 30, 2026Longwei Cong, Sonja Hahn, Sebastian Gombert +3LLM Evaluation Automated Grading
DIPF | Leibniz Institute for Research and Information in Education · Faculty of Computer Science, Goethe University Frankfurt · 3Chemnitz University of Technology
TeachGrade: Training-Free Short Answer Scoring with Teacher Evidence
Oct 10, 2026Jinhao Yan, Zhihong Shen, Shengjie Wang +4Automated Grading LLM-as-a-Judge
Fuzhou University · Peking University
Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring
Oct 7, 2026Zhifan Sun, Sebastian Gombert, Jannik Lossjew +6Automated Grading Educational Assessment
DIPF | Leibniz Institute for Research and Information in Education · IPN | Leibniz Institute for Science and Mathematics Education · Umeå University +1

Apr 30, 2026J/K move · Enter open · S save

Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory

Longwei Cong, Sonja Hahn, Sebastian Gombert, Leon Camus+2

DIPF | Leibniz Institute for Research and Information in Education · Faculty of Computer Science, Goethe University Frankfurt · 3Chemnitz University of Technology

LLM Evaluation Automated Grading Item Response Theory

Open paper AskPDF arXiv

Automated short answer grading (ASAG) with large language models (LLMs) is commonly evaluated with aggregate metrics such as macro-F1 and Cohen's kappa. However, these metrics provide limited insight into how grading performance varies across student responses of differing grading difficulty. We introduce an evaluation framework for LLM-based ASAG based on item response theory (IRT), which models grading correctness as a function of latent grader ability and response grading difficulty. This formulation enables response-level analysis of where LLM graders succeed or fail and reveals robustness differences that are not visible from aggregate scores alone. We apply the framework to 17 open-weight LLMs on the SciEntsBank and Beetle benchmarks. The results show that even models with similar overall performance differ substantially in how sharply their grading accuracy declines as response difficulty increases. In addition, confusion patterns show that errors on difficult responses concentrate disproportionately on the partially\_correct\_incomplete label, indicating a tendency toward intermediate-label collapse under ambiguity. To characterize difficult responses, we further analyze semantic and linguistic correlates of estimated difficulty. Across both datasets, higher difficulty is associated with weaker semantic alignment to the reference answer, stronger contradiction signals, and greater semantic isolation in embedding space. Overall, these results show that item response theory offers a useful framework for evaluating LLM-based ASAG beyond aggregate performance measures.

Abstract

Automated Grading

Explore similar work

Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory

TeachGrade: Training-Free Short Answer Scoring with Teacher Evidence

Alice: A Large-Scale German Benchmark for Rubric-Based Multi-Dimensional Automatic Short Answer Scoring