Large language models have recently been proposed as tools for automated essay scoring, but their agreement with human grading remains unclear. In this work, we evaluate how LLM-generated scores compare with human grades and analyze the grading behavior of several models from the GPT and Llama families in an out-of-the-box setting, without task-specific training. Our results show that agreement between LLM and human scores remains relatively weak and varies with essay characteristics. In particular, compared to human raters, LLMs tend to assign higher scores to short or underdeveloped essays, while assigning lower scores to longer essays that contain minor grammatical or spelling errors. We also find that the scores generated by LLMs are generally consistent with the feedback they generate: essays receiving more praise tend to receive higher scores, while essays receiving more criticism tend to receive lower scores. These results suggest that LLM-generated scores and feedback follow coherent patterns but rely on signals that differ from those used by human raters, resulting in limited alignment with human grading practices. Nevertheless, our work shows that LLMs produce feedback that is consistent with their grading and that they can be reliably used in supporting essay scoring.
Figures & tables
Figure 1: Overview of the proposed analysis framework. Essays are first evaluated by LLMs to produce both predicted scores and textual feedback. Essay-level features and feedback-derived signals are then extracted and analyzed to compare LLM scores with human ratings and to examine how feedback sentiment and essay characteristics relate to the scores assigned by the models.
Figure 2: Distribution of human-assigned essay scores across datasets.
Name
Subset
Num essays
Essay length (words)
Num prompts
Num raters
Score range
Score step
Num traits
NER tags
ASAP
Task 1
1,783
410.30 ± 136.18
1
2
1–6
1
5
Yes
Task 7
1,569
191.67 ± 100.12
1
2
0–12
1
4
Yes
DREsS
New
1,917
360.65 ± 107.01
59
1
3–15
0.5
3
No
Table 1: Summary of the datasets used in the experiments.
Figure 3: Distribution of essay scores assigned by Llama and GPT models on the ASAP Task 7 dataset. Most models produce scores concentrated around the middle of the scale, while GPT-3.5 assigns comparatively lower scores with predictions concentrated toward the lower end.
Figure 4: Quadratic Weighted Kappa (QWK) scores between LLM-generated scores and human ratings across datasets. Lighter cells indicate greater agreement. While human raters exhibit relatively strong agreement with each other, agreement between LLM predictions and human scores remains generally lower across models and datasets.
Figure 5: Pearson correlation between LLM-generated scores and human ratings across datasets. Lighter cells indicate greater agreement. Although some models show moderate correlations with human scores, the overall agreement remains lower than the correlation observed between human raters.
Figure 6: Mean Absolute Error (MAE) between LLM-generated scores and human ratings across datasets. Lighter cells indicate greater agreement (smaller differences between scores). Across datasets and models, LLM predictions consistently exhibit larger errors than the differences observed between human raters.
Figure 7: Differences between Llama-assigned scores and human ratings grouped by human score across datasets. Columns correspond to Llama models and rows to datasets. Positive values indicate higher model scores than human ratings. Llama models assign relatively higher scores to low-scoring essays and lower scores to high-scoring essays, indicating compression of the score range.
Figure 8: Differences between GPT-assigned scores and human ratings grouped by human score across datasets. Columns correspond to GPT models and rows to datasets. Positive values indicate higher model scores than human ratings. Similar to Llama models, GPT models assign higher scores to low-scoring essays and lower scores to high-scoring essays, indicating compression of the score range.
Figure 9: Readability scores (Flesch Reading Ease, top) and essay lengths (number of tokens, bottom) for the ASAP Task 1 dataset grouped by human score. Essays receiving higher scores tend to be substantially longer, while readability varies less across score levels.
Figure 10: Average distribution of LanguageTool-detected error types in essays that received the maximum possible score from human raters. All such essays contain at least one detected error, most commonly spelling mistakes, suggesting that minor language issues are often tolerated by human raters.
Figure 11: Proportion of praise (bottom) and criticism (top) associated with different rubric traits across essays with different LLM-assigned scores on the DREsS dataset. As scores increase, praise becomes more frequent while criticism decreases across most traits, indicating that feedback sentiment generally corresponds to the assigned scores.
Figure 12: Proportion of criticism (top) and praise (bottom) associated with rubric traits across essays with different LLM-assigned scores on ASAP Task 1. Clear score-dependent trends are observed for traits such as content, organization, and awareness of audience, while other traits show more irregular patterns across score levels.
Figure 13: SHAP beeswarm plots showing the contribution of trait-specific praise and criticism mentions to predicted essay scores on ASAP Task 7. Features sorted in decreasing order of importance (mean absolute SHAP value). Points represent individual essays, and horizontal position indicates the impact on the predicted score (SHAP value). Praise-related features generally increase predicted scores, while criticism-related features decrease them.
Figure 14: Distribution of trait-specific praise (left) and criticism (right) across models for essays in ASAP Task 1. Praise is concentrated on a small number of traits, particularly organization and content, while criticism is distributed more broadly across traits.
Figure 15: Distribution of trait-specific praise (left) and criticism (right) across models for essays in ASAP Task 7. Positive mentions are concentrated mainly on ideas , with some models also placing substantial emphasis on style , whereas negative mentions are distributed more broadly across the rubric traits.
Figure 16: Distribution of trait-specific praise (left) and criticism (right) across models for essays in the DREsS dataset. Trait mentions are uneven: praise is most frequently associated with content , while criticism places greater emphasis on language , with organization playing a secondary role.
Figure 17: Average absolute SHAP values for trait-specific praise and criticism features for GPT models on ASAP Task 7. Higher values indicate stronger influence on predicted scores, highlighting differences in trait importance across models.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 24: Distribution of essay scores assigned by Llama and GPT models on the ASAP Task 1 dataset. Llama models tend to assign mid-range scores (around 2-4), while GPT models show stronger variation across versions: GPT-3.5 concentrates on very low scores (1-2), whereas GPT-5 produces mostly mid-range scores.
Figure 25: Distribution of essay scores assigned by Llama and GPT models on the DREsS dataset. Most models concentrate scores in the mid-range (7-10), though the distributions vary across models, with Llama 2 assigning many high scores and GPT-5 showing a smoother, centered distribution..
transition, transitional language, transitional phrases, transitional words, connective, linking word
Appendix
Table 2: Trait-specific lexical cues used to identify praise and criticism mentions in model-generated feedback across datasets.
Figure 26: Agreement between LLM-generated scores and human grades under different prompt configurations on ASAP Task 1, measured using QWK, MAE, and Pearson correlation. Lighter cells indicate higher agreement (and lower error for MAE). Human raters show substantially stronger agreement with each other than with LLM predictions, while prompt variants yield only modest and inconsistent improvements across models.
Figure 27: Agreement between LLM-generated scores and human grades under different prompt configurations on ASAP Task 7, measured using QWK, MAE, and Pearson correlation. Lighter cells indicate higher agreement (and lower error for MAE). Human-human agreement remains substantially stronger than LLM-human agreement, while prompt variants produce only limited and inconsistent improvements.
Figure 28: Agreement between LLM-generated scores and human grades under different prompt configurations on the DREsS dataset, measured using QWK, MAE, and Pearson correlation. Lighter cells indicate higher agreement (and lower error for MAE). Agreement levels vary across models, and prompt variants lead to only modest and inconsistent changes.
Figure 29: Pairwise agreement between LLM-generated scores on the ASAP Task 1 dataset, measured using QWK (top) and Pearson correlation (bottom). Lighter cells indicate higher agreement. LLMs from the same family and prompt configuration tend to show stronger agreement with each other, while agreement between different model families is generally lower.
Figure 30: Pairwise agreement between LLM-generated scores on the ASAP Task 7 dataset, measured using QWK (top) and Pearson correlation (bottom). Lighter cells indicate higher agreement. Agreement is generally stronger between variants of the same model family and prompt configuration than across different models.
Figure 31: Pairwise agreement between LLM-generated scores on the DREsS dataset, measured using QWK (top) and Pearson correlation (bottom). Lighter cells indicate higher agreement. Agreement is generally stronger between variants of the same model family and prompt configuration than across different models.
Figure 35: Readability scores (Flesch Reading Ease, top) and essay lengths (number of tokens, bottom) for the ASAP Task 7 dataset. Each boxplot represents essays within a specific score range. Essay length increases consistently with score, while readability varies less systematically across score levels.
Figure 36: Readability scores (Flesch Reading Ease, top) and essay lengths (number of tokens, bottom) for the DREsS dataset. Each boxplot represents essays within a specific score range. Similar to ASAP Task 7, essay length increases with score, while readability shows weaker variation across score levels.
Figure 37: SHAP beeswarm plots showing the impact of trait-specific praise and criticism mentions on predicted essay scores for GPT models on ASAP Task 7. Features are sorted in decreasing order of importance (mean absolute SHAP value). Points represent individual essays, and horizontal position indicates the contribution to the predicted score. Praise-related features tend to increase predicted scores, while criticism-related features decrease them.
Figure 38: Average absolute SHAP values for trait-specific praise and criticism features for Llama models on ASAP Task 7. Higher values indicate features that contribute more strongly to predicted scores, revealing differences in how models prioritize rubric traits.
Figure 39: SHAP beeswarm plots showing the impact of trait-specific praise and criticism mentions on predicted essay scores for Llama models on DREsS. Features are sorted in decreasing order of importance (mean absolute SHAP value). Points represent individual essays, and horizontal position indicates the contribution to the predicted score. Praise-related features tend to increase predicted scores, while criticism-related features decrease them.
Figure 40: SHAP beeswarm plots showing the impact of trait-specific praise and criticism mentions on predicted essay scores for GPT models on DREsS. Features are sorted in decreasing order of importance (mean absolute SHAP value). Points represent individual essays, and horizontal position indicates the contribution to the predicted score. Praise-related features tend to increase predicted scores, while criticism-related features decrease them.
Figure 41: Average absolute SHAP values for trait-specific praise and criticism features for Llama models on DREsS. Higher values indicate features that contribute more strongly to predicted scores, revealing differences in how models prioritize rubric traits.
Figure 42: Average absolute SHAP values for trait-specific praise and criticism features for GPT models on DREsS. Higher values indicate features that contribute more strongly to predicted scores, revealing differences in how models prioritize rubric traits.
Despite growing interest in using Large Language Models (LLMs) for educational assessment, it remains unclear how closely they align with human scoring. We present a systematic evaluation of instruction-tuned LLMs across three open essay-scoring datasets (ASAP 2.0, ELLIPSE, and DREsS) that cover both holistic and analytic scoring. We analyze agreement with human consensus scores, directional bias, and the stability of bias estimates. Our results show that strong open-weight models achieve moderate to high agreement with humans on holistic scoring (Quadratic Weighted Kappa about 0.6), but this does not transfer uniformly to analytic scoring. In particular, we observe large and stable negative directional bias on Lower-Order Concern (LOC) traits, such as Grammar and Conventions, meaning that models often score these traits more harshly than human raters. We also find that concise keyword-based prompts generally outperform longer rubric-style prompts in multi-trait analytic scoring. To quantify the amount of data needed to detect these systematic deviations, we compute the minimum sample size at which a 95% bootstrap confidence interval for the mean bias excludes zero. This analysis shows that LOC bias is often detectable with very small validation sets, whereas Higher-Order Concern (HOC) traits typically require much larger samples. These findings support a bias-correction-first deployment strategy: instead of relying on raw zero-shot scores, systematic score offsets can be estimated and corrected using small human-labeled bias-estimation sets, without requiring large-scale fine-tuning.
Filip J. Kucia, Anirban Chakraborty, Anna Wróblewska
Faculty of Mathematics and Information Science, Warsaw University of Technology, 00-662 Warsaw, Poland · University of Wolverhampton, UK
Large language models (LLMs) have been widely applied to automated essay scoring (AES) and automated feedback generation (AFG). However, existing studies rely primarily on prompt engineering or supervised fine-tuning, while systematic research on reinforcement learning (RL) post-training and automated evaluation of feedback quality remains limited. We propose RLAES, a unified LLM framework that jointly optimizes essay scoring and feedback generation through RL. To make feedback quality measurable, interpretable, and usable for training, we introduce Rubric-based Feedback Evaluation (RFE), an essay-grounded feedback evaluation framework comprising 166 fine-grained binary rubric items and an LLM-as-judge. Building on RFE, we propose Adaptive Gated Feedback Optimization (AGFO), which activates rubric-based feedback rewards on demand during RL, reducing evaluation overhead while improving feedback quality. We also propose Adjacent Contrastive Reasoning (ACR) to improve ordinal score calibration by explicitly contrasting adjacent score levels. Experimental results show that the RFE framework captures essay-feedback consistency, exhibits strong pairwise discriminative power, and closely aligns with expert preferences. On the ASAP benchmark, RLAES-AGFO achieves the best scoring performance among LLM-based methods (QWK = 0.803), while maintaining feedback quality comparable to GPT-5.5 and avoiding the feedback degradation observed under score-only RL. Code and datasets are publicly available at https://github.com/hellomuyi/RLAES.
Automated Essay Scoring (AES) has been explored for decades with the goal to support teachers by reducing grading workload and mitigating subjective biases. While early systems relied on handcrafted features and statistical models, recent advances in Large Language Models (LLMs) have made it possible to evaluate student writing with unprecedented flexibility. This paper investigates the application of state-of-the-art open-weight LLMs for the grading of Austrian A-level German texts, with a particular focus on rubric-based evaluation. A dataset of 101 anonymised student exams across three text types was processed and evaluated. Four LLMs, DeepSeek-R1 32b, Qwen3 30b, Mixtral 8x7b and LLama3.3 70b, were evaluated with different contexts and prompting strategies. The LLMs were able to reach a maximum of 40.6% agreement with the human rater in the rubric-provided sub-dimensions, and only 32.8% of final grades matched the ones given by a human expert. The results indicate that even though smaller models are able to use standardised rubrics for German essay grading, they are not accurate enough to be used in a real-world grading environment.
Jonas Kubesch, Lena Huber, Clemens Havas
Salzburg University of Applied Sciences, Salzburg, Austria