Automated essay scorers (AES) based on large language models (LLMs) promise human-level accuracy. However, training, fine-tuning, and evaluating LLMs for specific assessment tasks is costly. In particular, the question arises whether these AES should be trained for specific domains or whether models based on generic LLMs might be sufficient. Human annotations, by contrast, are often more valid but far more expensive. This study evaluates whether ensembles of AES systems based on general-purpose LLMs can approximate human ratings of medical-education writing tasks by leveraging encoded linguistic and conceptual knowledge. The goal is to assess whether general-purpose models are applicable for assisting or potentially replacing human annotation in this domain. We experiment with 11 AES systems and 8 different LLMs and evaluate 668 student responses to 12 tasks, using \numprint{2076} evaluations by 7 human annotators. These ratings are aggregated to create consistent performance levels that can be used as target classes for classifiers trained on AES outputs. Our results show that learner classes based on human ratings can be accurately reconstructed using a linear SVM trained on a small set of AES features. A single classifier generalized well across tasks and domains, with reasoning and content features being most predictive. This suggests that out-of-the-box LLMs, even without domain-specific fine-tuning, can support automatic essay grading in medical education. While individual AES systems are insufficient alone, their combination approximates human evaluators, pointing to a reduced need for manual scoring or its supplementation with automated methods.
Figures & tables
Model
Essay Scorer Tool
# Output Features
LLM-based prompting
Reference
ES1
engessay-grading-ml
6
transformer-based
[ 46 ]
ES2
ielts-essay-scoring
5
transformer-based
[ 46 ]
ES3
llmaesslow-scorer
4
yes
[ 52 ]
ES4
aes-asap-mts-scorer
4
yes
[ 25 ]
ES5
aag-scorer
1
yes
[ 54 ]
ES6
aes-asap-vanilla-scorer
1
yes
[ 25 ]
Table 1: AES used in this study, along with their output features and whether or not they utilize an LLM.
Tool
Feature
Category
Prompt
Prompt Type
F1
ES1
cohesion
Language
-
-
F2
ES1
vocabulary
Language
-
-
F3
ES1
syntax
Language
-
-
F4
ES1
phraseology
Language
-
-
F5
ES1
grammar
Language
-
-
F6
ES1
conventions
Language
-
-
Table 2: Overview of the essay scorer outputs and utilized prompts that we used as features in our classification.
LLM
Parameters
Reference
LL3
Llama 3.2
3b
[ 19 ]
LQ8
Qwen 3
8b
[ 40 ]
LQ14
Qwen 3
14b
[ 40 ]
LQ32
Qwen 3
32b
[ 40 ]
LG20
GPT-OSS
20b
[ 37 ]
LG120
GPT-OSS
120b
[ 37 ]
Table 3: Overview of the LLMs evaluated in this study.
Figure 1: Analysis pipeline from LLM-based automatic essay scoring to student-performance prediction using a classifier trained on human-graded essays and feature selection via an evolutionary algorithm.
Task
# Criteria
# Answers
# Rater
Kαn
Kαi
Cαl
Cαp
ICC(3,k)
Respiration F
20
59
3
0.54
0.57
0.84
0.84
0.84
Respiration P
19
45
3
0.64
0.69
0.88
0.88
0.88
Respiration T
12
45
3
0.45
0.73
0.91
0.91
0.91
Eye F
19
52
3
0.69
0.75
0.90
0.90
0.90
Eye P
15
50
4
0.59
0.64
0.88
0.88
0.88
Eye T
14
49
4
0.68
0.74
0.92
0.92
0.92
Table 4: Overview of the number of criteria, student responses and raters per CORE assessment task (F: Fundamental; P: Practical; T = Transdisciplinary), and the inter-annotator agreement calculated using mean Krippendorff’s and Cronbach’s alpha and intraclass correlation ( Kαn : Krippendorff’s alpha nominal; Kαi : Krippendorff’s alpha interval; Cαl : Cronbach’s alpha listwise; Cαp : Cronbach’s alpha pairwise).
Figure 2: Results of the classifier performance, optimised using an evolutionary search using the features of all LLMs as input. The y-axis shows the optimised Macro F1 score averaged over the 10 folds as well as the average number of selected features. The x-axis displays the upper number of features that the search algorithm was allowed to select.
Figure 3: Results of the 2-quantile binning variant classification optimised using an evolutionary search for the LLM-family groups. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 4: Results of the 2-uniform binning variant classification optimised using an evolutionary search for the LLM-family groups. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 5: Results of the 3-quantile binning variant classification optimised using an evolutionary search for the LLM-family groups. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 6: Results of the 3-uniform binning variant classification optimised using an evolutionary search for the LLM-family groups. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 7: Results of the 2-quantile binning variant classification optimised using an evolutionary search for the 2 DeepSeek- and the 2 GPT-OSS-LLMs. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 8: Results of the 2-quantile binning variant classification optimised using an evolutionary search for the 3 Qwen-LLMs and Llama. The images show the optimised mean Macro F1 score over 10 folds as well as the average number of selected features.
Figure 9: Barplot showing the mean Macro F1 scores of the classification performance after optimization, comparing the results of the feature selection by single LLM vs. LLM-families.
Figure 10: Scatterplot of LLM parameter size (in billions of parameters) versus mean Macro F1 after evolutionary search. Point size indicates the maximum number of features permitted in the search.
Task
2-quantile
2-uniform
3-quantile
3-uniform
Middle Ear
0.90
0.95
0.80
0.83
Respiration
0.92
0.90
0.82
0.87
Eye
0.90
0.84
0.81
0.85
Circulation
0.95
0.92
0.87
0.93
Table 5: Mean Macro F1 scores of the classifier performance after optimization using evolutionary search with feature inputs from all LLMs, broken down by CORE assessment task (first column).
Figure 11: Volcano plot comparing distributions for all models. The x-axis shows Cliff’s delta (δ∈[−1,1]) , where δ>0 favours class low and δ<0 favours high . The y-axis shows −log10 two-sided Mann–Whitney p -values (higher indicates stronger evidence). The dashed line marks p=0.05 ; the FDR line marks q=0.05 [ 2 ] .
Figure 12: Volcano plot for LLAMA (top left) and Qwen models (same method as Figure 11 ).
Figure 13: Volcano plot for GPT models (same method as Figure 11 ).
Figure 14: Volcano plot for DeepSeek models (same method as Figure 11 ).
Figure 15: Kernel density estimates (KDEs) for two performance classes (low, high).
Figure 16: Global feature importance for low performance class (quantile binning).
Figure 17: Global feature importance for high performance class (quantile binning).
Figure 18: Feature stability and importance across models. The x-axis encodes the number of cross-validation splits (out of 10 total), where the feature is selected by the evolutionary search; the y-axis encodes the magnitude of SHAP importance, the colours indicate the direction (blue - negative, red positive).
Figure 19: Pearson correlation matrix of feature vectors. The colour scale indicates pairwise linear correlations between features (1 = strong positive, -1 = strong negative, 0 = no linear relationship).
Figure 20: Histogram of 4 LLM-detection tools showing the probabilities of proband statements being LLM-generated based on the averaged per-sentence probability.