Automated essay scorers (AES) based on large language models (LLMs) promise human-level accuracy. However, training, fine-tuning, and evaluating LLMs for specific assessment tasks is costly. In particular, the question arises whether these AES should be trained for specific domains or whether models based on generic LLMs might be sufficient. Human annotations, by contrast, are often more valid but far more expensive. This study evaluates whether ensembles of AES systems based on general-purpose LLMs can approximate human ratings of medical-education writing tasks by leveraging encoded linguistic and conceptual knowledge. The goal is to assess whether general-purpose models are applicable for assisting or potentially replacing human annotation in this domain. We experiment with 11 AES systems and 8 different LLMs and evaluate 668 student responses to 12 tasks, using \numprint{2076} evaluations by 7 human annotators. These ratings are aggregated to create consistent performance levels that can be used as target classes for classifiers trained on AES outputs. Our results show that learner classes based on human ratings can be accurately reconstructed using a linear SVM trained on a small set of AES features. A single classifier generalized well across tasks and domains, with reasoning and content features being most predictive. This suggests that out-of-the-box LLMs, even without domain-specific fine-tuning, can support automatic essay grading in medical education. While individual AES systems are insufficient alone, their combination approximates human evaluators, pointing to a reduced need for manual scoring or its supplementation with automated methods.
Figures & tables
Model
Essay Scorer Tool
# Output Features
LLM-based prompting
Reference
ES1
engessay-grading-ml
6
transformer-based
[ 46 ]
ES2
ielts-essay-scoring
5
transformer-based
[ 46 ]
ES3
llmaesslow-scorer
4
yes
[ 52 ]
ES4
aes-asap-mts-scorer
4
yes
[ 25 ]
ES5
aag-scorer
1
yes
[ 54 ]
ES6
aes-asap-vanilla-scorer
1
yes
[ 25 ]
Table 1: AES used in this study, along with their output features and whether or not they utilize an LLM.
Tool
Feature
Category
Prompt
Prompt Type
F1
ES1
cohesion
Language
-
-
F2
ES1
vocabulary
Language
-
-
F3
ES1
syntax
Language
-
-
F4
ES1
phraseology
Language
-
-
F5
ES1
grammar
Language
-
-
F6
ES1
conventions
Language
-
-
Table 2: Overview of the essay scorer outputs and utilized prompts that we used as features in our classification.
LLM
Parameters
Reference
LL3
Llama 3.2
3b
[ 19 ]
LQ8
Qwen 3
8b
[ 40 ]
LQ14
Qwen 3
14b
[ 40 ]
LQ32
Qwen 3
32b
[ 40 ]
LG20
GPT-OSS
20b
[ 37 ]
LG120
GPT-OSS
120b
[ 37 ]
Table 3: Overview of the LLMs evaluated in this study.
Figure 1: Analysis pipeline from LLM-based automatic essay scoring to student-performance prediction using a classifier trained on human-graded essays and feature selection via an evolutionary algorithm.
Task
# Criteria
# Answers
# Rater
Kαn
Kαi
Cαl
Cαp
ICC(3,k)
Respiration F
20
59
3
0.54
0.57
0.84
0.84
0.84
Respiration P
19
45
3
0.64
0.69
0.88
0.88
0.88
Respiration T
12
45
3
0.45
0.73
0.91
0.91
0.91
Eye F
19
52
3
0.69
0.75
0.90
0.90
0.90
Eye P
15
50
4
0.59
0.64
0.88
0.88
0.88
Eye T
14
49
4
0.68
0.74
0.92
0.92
0.92
Table 4: Overview of the number of criteria, student responses and raters per CORE assessment task (F: Fundamental; P: Practical; T = Transdisciplinary), and the inter-annotator agreement calculated using mean Krippendorff’s and Cronbach’s alpha and intraclass correlation ( Kαn : Krippendorff’s alpha nominal; Kαi : Krippendorff’s alpha interval; Cαl : Cronbach’s alpha listwise; Cαp : Cronbach’s alpha pairwise).
Figure 2: Results of the classifier performance, optimised using an evolutionary search using the features of all LLMs as input. The y-axis shows the optimised Macro F1 score averaged over the 10 folds as well as the average number of selected features. The x-axis displays the upper number of features that the search algorithm was allowed to select.
Figure 3: Results of the 2-quantile binning variant classification optimised using an evolutionary search for the LLM-family groups. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 4: Results of the 2-uniform binning variant classification optimised using an evolutionary search for the LLM-family groups. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 5: Results of the 3-quantile binning variant classification optimised using an evolutionary search for the LLM-family groups. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 6: Results of the 3-uniform binning variant classification optimised using an evolutionary search for the LLM-family groups. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 7: Results of the 2-quantile binning variant classification optimised using an evolutionary search for the 2 DeepSeek- and the 2 GPT-OSS-LLMs. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 8: Results of the 2-quantile binning variant classification optimised using an evolutionary search for the 3 Qwen-LLMs and Llama. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 9: Barplot showing the mean Macro F1 scores of the classification performance after optimization, comparing the results of the feature selection by single LLM vs. LLM-families.
Figure 10: Scatterplot of LLM parameter size (in billions of parameters) versus mean Macro F1 after evolutionary search. Point size indicates the maximum number of features permitted in the search.
Task
2-quantile
2-uniform
3-quantile
3-uniform
Middle Ear
0.90
0.95
0.80
0.83
Respiration
0.92
0.90
0.82
0.87
Eye
0.90
0.84
0.81
0.85
Circulation
0.95
0.92
0.87
0.93
Table 5: Mean Macro F1 scores of the classifier performance after optimization using evolutionary search with feature inputs from all LLMs, broken down by CORE assessment task (first column).
Figure 11: Volcano plot comparing distributions for all models. The x-axis shows Cliff’s delta (δ∈[−1,1]) , where δ>0 favours class low and δ<0 favours high . The y-axis shows −log10 two-sided Mann–Whitney p -values (higher indicates stronger evidence). The dashed line marks p=0.05 ; the FDR line marks q=0.05 [ 2 ] .
Figure 12: Volcano plot for LLAMA (top left) and Qwen models (same method as Figure 11 ).
Figure 13: Volcano plot for GPT models (same method as Figure 11 ).
Figure 14: Volcano plot for DeepSeek models (same method as Figure 11 ).
Figure 15: Kernel density estimates (KDEs) for two performance classes (low, high).
Figure 16: Global feature importance for low performance class (quantile binning).
Figure 17: Global feature importance for high performance class (quantile binning).
Figure 18: Feature stability and importance across models. The x-axis encodes the number of cross-validation splits (out of 10 total), where the feature is selected by the evolutionary search; the y-axis encodes the magnitude of SHAP importance, the colours indicate the direction (blue - negative, red positive).
Figure 19: Pearson correlation matrix of feature vectors. The colour scale indicates pairwise linear correlations between features (1 = strong positive, -1 = strong negative, 0 = no linear relationship).
Figure 20: Histogram of 4 LLM-detection tools showing the probabilities of proband statements being LLM-generated based on the averaged per-sentence probability.
Grading student responses is costly and time-consuming. Moreover, human annotators are prone to bias, fatigue, and rater drift, which further limits grading reliability. Various methods have been employed to develop automated essay scoring (AES) systems to reduce these costs and address bias-related issues. These methods range from early text-feature-based systems to current machine-learning and transformer-based models. Specifically, large language models (LLMs) offer the potential to streamline the development of AES systems. However, they are subject to well-known limitations, such as hallucinations and prompt sensitivity. This creates a need to explore the potential of a mixed-methods approach that leverages the synergies among individual LLMs through ensemble formation. We evaluate the potential of ensembles of LLMs to match human-rater agreement when grading student responses in economics and medical education across 103 criteria. We use 631 German-language responses and corresponding human ratings by 2 to 5 annotators from a previously conducted assessment. We experiment with 523 rubric-based prompt configurations across 15 open-weight LLMs and assess their consistency with human raters using Krippendorff's alpha for inter-annotator agreement (IAA) and the Alternative Annotator Test (alt-test). Our results show that replacing one human rater with the best-performing LLM--prompt configuration maintains or improves IAA for 91 of the 103 criteria and that LLMs pass the alt-test for 90 criteria. Our findings suggest that current state-of-the-art LLMs can support AES of German responses in economics and medical education, reducing human workload and enabling faster processing, while validation against human ratings remains necessary.
Daniel Baumartz, Alexander Mehler, Patrick Schrottenbacher +2
Goethe University Frankfurt, Germany · DIPF Leibniz Institute for Research and Information in Education, Germany
Large language models have recently been proposed as tools for automated essay scoring, but their agreement with human grading remains unclear. In this work, we evaluate how LLM-generated scores compare with human grades and analyze the grading behavior of several models from the GPT and Llama families in an out-of-the-box setting, without task-specific training. Our results show that agreement between LLM and human scores remains relatively weak and varies with essay characteristics. In particular, compared to human raters, LLMs tend to assign higher scores to short or underdeveloped essays, while assigning lower scores to longer essays that contain minor grammatical or spelling errors. We also find that the scores generated by LLMs are generally consistent with the feedback they generate: essays receiving more praise tend to receive higher scores, while essays receiving more criticism tend to receive lower scores. These results suggest that LLM-generated scores and feedback follow coherent patterns but rely on signals that differ from those used by human raters, resulting in limited alignment with human grading practices. Nevertheless, our work shows that LLMs produce feedback that is consistent with their grading and that they can be reliably used in supporting essay scoring.
Jerin George Mathew, Sumayya Taher, Anindita Kundu +1
University of Alberta, 116 St & 85 Ave, Edmonton, AB, T6G 2R3, Canada
Automated essay scoring (AES) using large language models (LLMs) offers promising use cases for language assessment, yet high-stakes testing requires scores that are both controllable and verifiable, which current LLM-based scorers lack. Given that, we investigate whether LLM-based scorers form distinct internal representations for individual scoring rubrics and whether these representations can be independently controlled. Probing and steering 25 open-weight base LLMs (0.13B-10.31B parameters; 14 families) on human-scored BESTEP English writing tasks, we find that holistic scores are decodable from model activations for the longer tasks (Parts 2 and 3) and only weakly for the shortest (Part 1). Furthermore, scoring rubrics have partly separable directions in representation space for the longer tasks, though not for short responses (Part 1). However, causal steering reveals a critical limitation: applying a matching intervention (e.g., steering "grammar" on a grammar-degraded response) yields only a negligible advantage (about 2% of the effect) over a mismatched intervention (e.g., steering "vocabulary"), and none that is detectable in any of the 10 testable comparisons across reliably measured rubrics. Rather than controlling distinct evaluative dimensions, the rubric directions share a dominant lever whose rubric-specific parts are not selective, which may mirror the halo structure of human ratings. Our contributions are two-fold. First, we show that analytic scoring rubrics designed for human raters do not behave as independent controls in base LLMs, even among the seven models whose scores agree best with human raters (r≥0.6). Second, we establish a methodology that shows that score agreement and decodability do not imply rubric-level control in LLM-based assessment.
Chungche Huang, Po-Ya Angela Wang, Shu-Kai Hsieh
Graduate Institute of Linguistics National Taiwan University · Co-intelligence Humanities AI (CHAI) Future Lab National Taiwan University