Grading student responses is costly and time-consuming. Moreover, human annotators are prone to bias, fatigue, and rater drift, which further limits grading reliability. Various methods have been employed to develop automated essay scoring (AES) systems to reduce these costs and address bias-related issues. These methods range from early text-feature-based systems to current machine-learning and transformer-based models. Specifically, large language models (LLMs) offer the potential to streamline the development of AES systems. However, they are subject to well-known limitations, such as hallucinations and prompt sensitivity. This creates a need to explore the potential of a mixed-methods approach that leverages the synergies among individual LLMs through ensemble formation. We evaluate the potential of ensembles of LLMs to match human-rater agreement when grading student responses in economics and medical education across 103 criteria. We use 631 German-language responses and corresponding human ratings by 2 to 5 annotators from a previously conducted assessment. We experiment with 523 rubric-based prompt configurations across 15 open-weight LLMs and assess their consistency with human raters using Krippendorff's alpha for inter-annotator agreement (IAA) and the Alternative Annotator Test (alt-test). Our results show that replacing one human rater with the best-performing LLM--prompt configuration maintains or improves IAA for 91 of the 103 criteria and that LLMs pass the alt-test for 90 criteria. Our findings suggest that current state-of-the-art LLMs can support AES of German responses in economics and medical education, reducing human workload and enabling faster processing, while validation against human ratings remains necessary.
Figures & tables
Figure 1: Flowchart of the study design and evaluation process.
Nudging
Pilot strike
Startup
Wind park
#Responses
122
81
81
86
Characters
- Mean
1,125.55
1,089.77
1,181.99
1,063.70
- SD
509.14
507.29
505.85
496.25
- Min
0
0
0
0
- Max
2,364
2,284
2,402
2,367
Table 1: Student response statistics for the economics tasks.
Breathing
Eye
Circulation
Middle ear
#Responses
48
58
55
100
Characters
- Mean
1,109.54
859.69
979.24
1,108.55
- SD
573.16
488.06
497.13
607.73
- Min
0
0
0
0
- Max
2,860
1,941
2,212
3,475
Table 2: Student response statistics for the medicine tasks.
Figure 2: Distribution of student response texts.
NU
PS
SU
WP
BR
EY
CI
ME
#Ratings
2,267
1,294
2,096
1,671
2,208
2,496
3,120
8,240
Avg.
20.80
17.97
27.58
20.89
48
48
60
90.55
– SD
1.15
0.17
1.58
0.80
0
0
0
10.47
– Min
14
17
21
14
48
48
60
60
– Max
21
18
28
21
48
48
60
100
Avg. rater
2.97
3
3.95
2.99
3
4
3
4.53
Table 3: Statistics on the human ratings of the student responses to the eight economics and medicine tasks.
Model
Size
Reference
DeepSeek-R1
70b
Guo et al. [57]
GPT-OSS
20, 120b
OpenAI et al. [105]
Llama 3.3
70b
Grattafiori et al. [53]
MiniCPM-V
8b
Yao et al. [154]
Mixtral
8x7b
Jiang et al. [69]
Nemotron
70b
Wang et al. [139]
Table 5: Overview of the LLMs used in the main study.
Model
Size
Reference
DeepSeek-R1
32, 70b
Guo et al. [57]
DeepSeek v3.1
671b
DeepSeek-AI et al. [36]
Gemma 2
27b
Gemma Team et al. [50]
GPT-OSS
20, 120b
OpenAI et al. [105]
Llama 3.2
3b
Grattafiori et al. [53]
Llama 3.3
70b
Grattafiori et al. [53]
Table 6: Overview of the LLMs used in the Nudging-only study.
Task
Criteria
Range
Group
K α
C α
ICC
UA
NU
filter
Boolean
formal
0.680
0.873
0.945
0.963
NU
coherence
0–3
structure
0.302
0.754
0.849
0.165
NU
argumentation
Boolean
domain
0.173
0.492
0.661
0.468
NU
arguments_count
0–4
structure
0.311
0.782
0.861
0.174
NU
conclusion
Boolean
formal
0.035
0.495
0.586
0.294
NU
reference
Boolean
formal
0.648
0.853
0.925
0.945
Table 7: Inter-annotator agreements in the economics tasks among the two to four raters (see Table 3 ) calculated using Krippendorff’s α (K α ) with interval difference function, consistency using listwise Cronbach’s α (C α ), ICC(3,k), and percentage of unanimous agreement (UA); see subsection 3.2 . Criteria, ranges and groups are further described in subsection 3.1 . UA is not reported for SUM_SCORE .
Task
Criteria
Range
Group
K α
C α
ICC
UA
BR
filter
Boolean
formal
0.481
0.738
0.891
0.935
BR
coherence
0–3
structure
0.437
0.711
0.812
0.478
BR
cr_a1
Boolean
diagnoses
0.614
0.831
0.844
0.717
BR
cr_b1
Boolean
diagnoses
0.418
0.715
0.725
0.587
BR
cr_b2
Boolean
diagnoses
0.851
0.946
0.947
0.957
BR
cr_b3
Boolean
diagnoses
0.839
0.942
0.943
0.935
Table 8: Inter-annotator agreements in the medicine tasks among the three to five raters (see Table 3 ) calculated using Krippendorff’s α (K α ) with interval difference function, consistency using listwise Cronbach’s α (C α ), ICC(3,k), and percentage of unanimous agreement (UA); see subsection 3.2 . Criteria, ranges and groups are further described in subsection 3.1 . UA is not reported for SUM_SCORE .
(a) Medicine tasks
Figure 4: Overview of changes in IAA when replacing a human annotator with an LLM in the main study, per task (panel) and criterion (x-axis). Each box summarises all LLM–prompt configurations for one replaced human rater (distinguished by colour; the letters identify the human raters). Values above the dashed zero line indicate higher agreement of the rater group including the LLM than of the original human group. Panel titles use internal task names: med_atmung ( Breathing ), med_auge ( Eye ), med_kreislauf ( Circulation ), med_mittelohr ( Middle ear ), eco_pilotenstreik ( Pilot strike ), eco_startup ( Startup ), eco_windpark ( Wind park ), eco_nudging ( Nudging ).
Figure 5: Overview of changes in IAA when replacing a human annotator with an LLM in the Nudging-only study, for prompts rating all criteria ( eco_nudging_expNU ) and prompts focusing only on ARGUMENTS_COUNT ( ARG ) or COHERENCE ( COH ). Box elements as in Figure 4 .
Figure 6: Heatmap showing model overlap among the top 10 models per domain and criterion.
Figure 7: Winning rates of the Alternative Annotator Test using a cost-benefit parameter of 0.15 to denote skilled annotators, showing the top three LLM–prompt combinations per task in the economics tasks, ranked by the number of passed criteria. Columns show the criteria and rows the LLM–prompt configurations; empty cells denote criteria not rated by the configuration. Row labels show internal identifiers (task, prompt, and model; truncated) followed by the number of passing criteria; SCHULUNG denotes TRAINING prompts. For Nudging , the rows include the top three configurations each of the main study, the Nudging-only study, and the prompts focusing on ARGUMENTS_COUNT and COHERENCE .
Figure 8: Winning rates of the Alternative Annotator Test using a cost-benefit parameter of 0.15 to denote skilled annotators, showing the top three LLM–prompt combinations per task in the medicine tasks, ranked by the number of passed criteria. Columns show the criteria and rows the LLM–prompt configurations; empty cells denote criteria not rated by the configuration. Row labels show internal identifiers (task, prompt, and model; truncated) followed by the number of passing criteria; SCHULUNG denotes TRAINING prompts.
Figure 9: Inter-annotator agreement when replacing two human raters with all LLM–prompt combinations in the medical tasks.
Figure 10: Inter-annotator agreement when replacing two human raters with all LLM–prompt combinations in the economy tasks.
Figure 11: Results of the IAA-change and alt-test analyses for the 103 criteria considered across all eight tasks.
Figure 12: LLM–prompt configuration overlap when comparing the top N performers according to IAA-change and alt-test.
Figure 13: Beeswarm plots showing feature importance based on the SHAP analysis.
Figure 14: Grouped feature-importance values according to the SHAP analysis.
Figure 15: Comparison of the resulting IAA score after replacing one human rater against the total number of tokens consumed and generated by the LLMs (left) and the number of LLM parameters (right). Log-scaled x-axis.
Figure 16: Consistency of LLM ratings based on inter-annotator agreement across five repetitions.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 24
Figure 18: Distribution of ratings for each economics criterion assigned by human annotators and LLMs.
(a) Medicine task Eye
(a) Medicine task Breathing
(a) Medicine task Circulation
Figure 22: Distribution of ratings for each medical criterion assigned by human annotators and LLMs.
Criteria
IAA
Change
#
LLM
Prompt
filter
0.185
0.038
51
Plutus
CHAIN NOANSWER
coherence
0.658
0.019
1
Nemotron 70b
SIMPLE
arguments
0.437
0.045
1
Qwen 2.5 32b
SIMPLE
balance
0.571
0.195
1
Llama 3.3 70b
TRAINING ECO (full text)
conclusion
0.448
0.126
1
GPT-OSS 20b
CHAIN NOANSWER
problem
0.451
0.203
1
Llama 3.3 70b
TRAINING (full text)
Appendix
Table 9: Overview of the top IAA scores after replacing a human rater with an LLM-based system in the main study task Pilot strike .
Criteria
IAA
Change
#
LLM
Prompt
filter
0.552
-0.270
51
Plutus
CHAIN NOANSWER
coherence
0.485
0.111
1
Llama 3.3 70b
CHAIN NOANSWER
argumentation
0.560
0.200
1
Qwen 2.5 32b
SIMPLE
arguments
0.639
0.060
1
Llama 3.3 70b
TRAINING (full text)
balance
0.364
0.161
1
Qwen 2.5 32b
SIMPLE
conclusion
0.691
0.156
1
Nemotron 70b
CHAIN NOANSWER
Appendix
Table 10: Overview of the top IAA scores after replacing a human rater with an LLM-based system in the main study task Startup .
Criteria
IAA
Change
#
LLM
Prompt
filter
0.396
-0.448
49
GPT-OSS 120b
CHAIN NOANSWER
coherence
0.610
0.124
1
Nemotron 70b
SIMPLE
argumentation
0.606
0.245
1
Llama 3.3 70b
SIMPLE
arguments
0.649
0.046
1
Llama 3.3 70b
TRAINING (full text)
balance
0.766
0.140
1
Llama 3.3 70b
TRAINING (full text)
conclusion
0.413
-0.004
1
Llama 3.3 70b
TRAINING ECO (full text)
Appendix
Table 11: Overview of the top IAA scores after replacing a human rater with an LLM-based system in the main study task Wind park .
Criteria
IAA
Change
#
LLM
Prompt
filter
0.661
-0.019
51
Plutus
CHAIN NOANSWER
coherence
0.538
0.235
1
Nemotron 70b
SIMPLE
reference
0.625
-0.023
2
MedGemma 1.5 4b
FEATURES (JSON)
conclusion
0.471
0.436
1
GPT-OSS 120b
CHAIN NOANSWER
argumentation
0.305
0.132
1
Llama 3.3 70b
TRAINING (full text)
arguments_count
0.357
0.046
1
Qwen 2.5 32b
SIMPLE
Appendix
Table 12: Overview of the top IAA scores after replacing a human rater with an LLM-based system in the main study task Nudging .
Criteria
IAA
Change
#
LLM
Prompt
filter
0.830
0.150
1
Gemma 2 27b
CHAIN
coherence
0.536
0.234
1
DeepSeek-R1 70b
FEATURES (JSON)
reference
0.741
0.093
3
Llama 3.3 70b
FEATURES
ELABORATION (JSON)
conclusion
0.422
0.388
3
DeepSeek-R1 70b
CHAIN
argumentation
0.301
0.128
1
Mixtral 8x7b
CHAIN
Appendix
Table 13: Overview of the top IAA scores after replacing a human rater with an LLM-based system in the Nudging-only study.
Criteria
IAA
Change
#
LLM
Prompt
filter
0.323
-0.158
49
GPT-OSS 120b
CHAIN NOANSWER
coherence
0.531
0.093
1
Llama 3.3 70b
CHAIN NOANSWER
cr_a1
0.912
0.298
4
Llama 3.3 70b
CHAIN NOANSWER
cr_b1
0.557
0.139
1
Nemotron 70b
FEATURES (JSON)
cr_b2
0.930
0.078
1
DeepSeek-R1 70b
OPT
cr_b3
0.939
0.100
1
DeepSeek-R1 70b
OPT
Appendix
Table 14: Overview of the top IAA scores after replacing a human rater with an LLM-based system in the main study task Breathing .
Criteria
IAA
Change
#
LLM
Prompt
filter
0.395
-0.111
49
Plutus
CHAIN NOANSWER
coherence
0.377
0.145
1
Llama 3.3 70b
CHAIN NOANSWER
cr_a1
0.816
0.118
5
GPT-OSS 120b
OPT
cr_b1
0.452
0.130
1
Llama 3.3 70b
CHAIN NOANSWER
cr_c1
0.633
0.118
1
Nemotron 70b
CHAIN NOANSWER
cr_b2
0.477
0.190
1
Llama 3.3 70b
TRAINING (full text)
Appendix
Table 15: Overview of the top IAA scores after replacing a human rater with an LLM-based system in the main study task Eye .
Criteria
IAA
Change
#
LLM
Prompt
filter
0.655
-0.238
49
Plutus
CHAIN NOANSWER
coherence
0.605
0.234
1
Llama 3.3 70b
CHAIN NOANSWER
cr_a1
0.662
0.307
1
Nemotron 70b
FEATURES (JSON)
cr_b1
0.940
0.261
10
Llama 3.3 70b
CHAIN NOANSWER
cr_b2
1.000
0.000
30
Llama 3.3 70b
CHAIN NOANSWER
cr_b3
1.000
0.503
9
Nemotron 70b
CHAIN NOANSWER
Appendix
Table 16: Overview of the top IAA scores after replacing a human rater with an LLM-based system in the main study task Circulation .
Criteria
IAA
Change
#
LLM
Prompt
filter
0.642
-0.115
48
Plutus
CHAIN NOANSWER
coherence
0.596
-0.001
1
GPT-OSS 120b
OPT
cr_a1
0.156
0.037
1
GPT-OSS 20b
OPT
cr_b1
0.520
0.186
1
Nemotron 70b
SIMPLE
cr_c1
0.720
0.055
1
GPT-OSS 120b
OPT
cr_d1
0.497
0.124
1
Llama 3.3 70b
CHAIN NOANSWER
Appendix
Table 17: Overview of the top IAA scores after replacing a human rater with an LLM-based system in the main study task Middle ear .
Figure 23: Jaccard similarity among the top 10 models per domain and criterion.
Figure 24: Sensitivity analysis of the cost-benefit parameter in the Alternative Annotator Test .
Model
Mean
SD
Mean
SD
Reg
Baseline
R2
-0.002
0.001
RMSE
0.307
0.007
Reg
SVR linear
R2
0.434
0.049
RMSE
0.230
0.007
Reg
SVR RBF
R2
0.692
0.036
RMSE
0.170
0.008
Reg
HGBT
R2
0.729
0.044
RMSE
0.159
0.012
Clf
Baseline
F1
0.460
0.006
ROC AUC
0.500
0.000
Clf
SVC linear
F1
0.678
0.009
ROC AUC
0.845
0.010
Appendix
Table 18: Results of the classification (Clf= and regression (Reg) proxy models used in the SHAP analysis.
Automated essay scorers (AES) based on large language models (LLMs) promise human-level accuracy. However, training, fine-tuning, and evaluating LLMs for specific assessment tasks is costly. In particular, the question arises whether these AES should be trained for specific domains or whether models based on generic LLMs might be sufficient. Human annotations, by contrast, are often more valid but far more expensive. This study evaluates whether ensembles of AES systems based on general-purpose LLMs can approximate human ratings of medical-education writing tasks by leveraging encoded linguistic and conceptual knowledge. The goal is to assess whether general-purpose models are applicable for assisting or potentially replacing human annotation in this domain. We experiment with 11 AES systems and 8 different LLMs and evaluate 668 student responses to 12 tasks, using \numprint{2076} evaluations by 7 human annotators. These ratings are aggregated to create consistent performance levels that can be used as target classes for classifiers trained on AES outputs. Our results show that learner classes based on human ratings can be accurately reconstructed using a linear SVM trained on a small set of AES features. A single classifier generalized well across tasks and domains, with reasoning and content features being most predictive. This suggests that out-of-the-box LLMs, even without domain-specific fine-tuning, can support automatic essay grading in medical education. While individual AES systems are insufficient alone, their combination approximates human evaluators, pointing to a reduced need for manual scoring or its supplementation with automated methods.
Daniel Baumartz, Alexander Mehler, Mevlüt Bagci +1
Automated Essay Scoring (AES) has been explored for decades with the goal to support teachers by reducing grading workload and mitigating subjective biases. While early systems relied on handcrafted features and statistical models, recent advances in Large Language Models (LLMs) have made it possible to evaluate student writing with unprecedented flexibility. This paper investigates the application of state-of-the-art open-weight LLMs for the grading of Austrian A-level German texts, with a particular focus on rubric-based evaluation. A dataset of 101 anonymised student exams across three text types was processed and evaluated. Four LLMs, DeepSeek-R1 32b, Qwen3 30b, Mixtral 8x7b and LLama3.3 70b, were evaluated with different contexts and prompting strategies. The LLMs were able to reach a maximum of 40.6% agreement with the human rater in the rubric-provided sub-dimensions, and only 32.8% of final grades matched the ones given by a human expert. The results indicate that even though smaller models are able to use standardised rubrics for German essay grading, they are not accurate enough to be used in a real-world grading environment.
Jonas Kubesch, Lena Huber, Clemens Havas
Salzburg University of Applied Sciences, Salzburg, Austria
Automated essay scoring (AES) using large language models (LLMs) offers promising use cases for language assessment, yet high-stakes testing requires scores that are both controllable and verifiable, which current LLM-based scorers lack. Given that, we investigate whether LLM-based scorers form distinct internal representations for individual scoring rubrics and whether these representations can be independently controlled. Probing and steering 25 open-weight base LLMs (0.13B-10.31B parameters; 14 families) on human-scored BESTEP English writing tasks, we find that holistic scores are decodable from model activations for the longer tasks (Parts 2 and 3) and only weakly for the shortest (Part 1). Furthermore, scoring rubrics have partly separable directions in representation space for the longer tasks, though not for short responses (Part 1). However, causal steering reveals a critical limitation: applying a matching intervention (e.g., steering "grammar" on a grammar-degraded response) yields only a negligible advantage (about 2% of the effect) over a mismatched intervention (e.g., steering "vocabulary"), and none that is detectable in any of the 10 testable comparisons across reliably measured rubrics. Rather than controlling distinct evaluative dimensions, the rubric directions share a dominant lever whose rubric-specific parts are not selective, which may mirror the halo structure of human ratings. Our contributions are two-fold. First, we show that analytic scoring rubrics designed for human raters do not behave as independent controls in base LLMs, even among the seven models whose scores agree best with human raters (r≥0.6). Second, we establish a methodology that shows that score agreement and decodability do not imply rubric-level control in LLM-based assessment.
Chungche Huang, Po-Ya Angela Wang, Shu-Kai Hsieh
Graduate Institute of Linguistics National Taiwan University · Co-intelligence Humanities AI (CHAI) Future Lab National Taiwan University