LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias
Authors: Filip J. Kucia, Anirban Chakraborty, Anna Wróblewska
Organizations: Faculty of Mathematics and Information Science, Warsaw University of Technology, 00-662 Warsaw, Poland · University of Wolverhampton, UK
Despite growing interest in using Large Language Models (LLMs) for educational assessment, it remains unclear how closely they align with human scoring. We present a systematic evaluation of instruction-tuned LLMs across three open essay-scoring datasets (ASAP 2.0, ELLIPSE, and DREsS) that cover both holistic and analytic scoring. We analyze agreement with human consensus scores, directional bias, and the stability of bias estimates. Our results show that strong open-weight models achieve moderate to high agreement with humans on holistic scoring (Quadratic Weighted Kappa about 0.6), but this does not transfer uniformly to analytic scoring. In particular, we observe large and stable negative directional bias on Lower-Order Concern (LOC) traits, such as Grammar and Conventions, meaning that models often score these traits more harshly than human raters. We also find that concise keyword-based prompts generally outperform longer rubric-style prompts in multi-trait analytic scoring. To quantify the amount of data needed to detect these systematic deviations, we compute the minimum sample size at which a 95% bootstrap confidence interval for the mean bias excludes zero. This analysis shows that LOC bias is often detectable with very small validation sets, whereas Higher-Order Concern (HOC) traits typically require much larger samples. These findings support a bias-correction-first deployment strategy: instead of relying on raw zero-shot scores, systematic score offsets can be estimated and corrected using small human-labeled bias-estimation sets, without requiring large-scale fine-tuning.
Figures & tables
Dataset
Scoring regime
# Traits
Assessment scale
# Essays
Avg. Words ± Std.
ASAP 2.0
Holistic
1
1–6 (1.0)
24,728 17,307/7,421
362.9±148.5
ELLIPSE
Analytic
6
1.0–5.0 (0.5)
6,482 3,911/2,571
427.8±191.9
DREsS
Analytic
3
1.0–5.0 (0.5)
8,487 6,508/1,979
329.4±167.6
Table 1: Overview of essay-scoring datasets used in this study. Essay length statistics are reported in word counts. In # Essays, the first value is the number used in this study (after preprocessing); the second line reports the original train/test split.
Keywords
Guidelines
Dataset
Model
QWK ↑
Bias →0
QWK ↑
Bias →0
ASAP 2.0
Llama 3.1 70B
0.533
-0.39
0.601
-0.05
Llama 3.1 405B
0.496
-0.62
0.592
-0.34
Llama 3.1 8B
0.373
-0.66
0.463
-0.44
GPT-OSS 120B
0.117
-1.47
0.299
-0.68
GPT-OSS 20B
0.137
-1.45
0.261
-1.11
Table 2: Performance of all models by prompt strategy, reported as weighted averages across data splits. Bold values indicate the highest QWK for each model and the signed bias closest to zero for each dataset. Note: All bias estimates are statistically different from zero ( p<0.001 ) according to a Wilcoxon Signed-Rank test.
Ref
Keywords Strategy
Guidelines Strategy
Dataset
Trait
σHCS
QWK ↑
Bias →0
MAE ↓
EA ↑
σLLM
QWK ↑
Bias →0
MAE ↓
EA ↑
σLLM
ASAP 2.0
Score
1.01
0.533
-0.39
0.73
17.4
0.72
0.601
-0.05
0.60
47.1
0.91
ELLIPSE
Cohesion (HOC)
0.66
0.566
-0.12
0.43
34.5
0.56
0.412
-0.55
0.63
21.6
0.55
Syntax (LOC)
0.66
0.414
-0.48
0.59
24.6
0.56
0.265
-0.73
0.77
17.0
0.50
Vocabulary (LOC)
0.58
0.278
-0.55
0.64
18.6
0.47
0.236
-0.67
0.71
14.9
0.41
Phraseology (LOC)
0.66
0.247
-0.74
0.79
16.1
0.49
0.132
-1.07
1.07
8.1
0.39
Table 3: Detailed performance of the top-performing model, Llama-3.1-70B , comparing the Keywords and Guidelines prompt strategies.
Dataset
#Combinations
Bias
Not
Median
90th
Max
Detected
Reached
Nmin
Nmin
Nmin
ASAP 2.0
20
20
0
10.0
175.5
1,460.0
ELLIPSE
120
120
0
5.0
5.0
85.0
DREsS
60
58
2
5.0
176.0
650.0
Table 4: Summary of minimum sample size for bias detection across datasets. Note: #Combinations of model–trait–split–strategy.
Dataset
Trait
Guidelines
Keywords
Orig. Test
Orig. Train
Orig. Test
Orig. Train
ASAP 2.0
Score
515
1,460
20
20
ELLIPSE
Cohesion
10
5
70
85
Conventions
5
5
5
5
Grammar
5
5
5
5
Phraseology
5
5
5
5
Table 5: Minimum sample size for bias detection ( Nmin ) for Llama-3.1-70B-Instruct , by dataset, trait, prompt strategy, and official split.
Large language models have recently been proposed as tools for automated essay scoring, but their agreement with human grading remains unclear. In this work, we evaluate how LLM-generated scores compare with human grades and analyze the grading behavior of several models from the GPT and Llama families in an out-of-the-box setting, without task-specific training. Our results show that agreement between LLM and human scores remains relatively weak and varies with essay characteristics. In particular, compared to human raters, LLMs tend to assign higher scores to short or underdeveloped essays, while assigning lower scores to longer essays that contain minor grammatical or spelling errors. We also find that the scores generated by LLMs are generally consistent with the feedback they generate: essays receiving more praise tend to receive higher scores, while essays receiving more criticism tend to receive lower scores. These results suggest that LLM-generated scores and feedback follow coherent patterns but rely on signals that differ from those used by human raters, resulting in limited alignment with human grading practices. Nevertheless, our work shows that LLMs produce feedback that is consistent with their grading and that they can be reliably used in supporting essay scoring.
Jerin George Mathew, Sumayya Taher, Anindita Kundu +1
University of Alberta, 116 St & 85 Ave, Edmonton, AB, T6G 2R3, Canada
This study examines the cross-prompt generalization and first-language (L1) scoring effects of a LoRA-adapted open-weight large language model (Gemma-3-27B-it) applied to automated essay scoring. Using the identical model and inference configuration reported in "AiAWE: An Open-Source LLM Automated Writing Evaluation System Using LoRA-Adapted Instruction-Tuned Models" (Gayed, 2026), which was fine-tuned on 480 argumentative essays from two prompts, we evaluate scoring accuracy on the full TOEFL11 corpus: 12,100 essays written by test-takers from 11 first-language backgrounds across eight prompts, none of which were seen during training. The model's raw scores (0.5-5.0) are mapped to the same three proficiency bands (low, medium, high) used by ETS, enabling direct comparison. The model achieved an overall band agreement of 77.79% and a quadratic weighted kappa of 0.702, with adjacent-band agreement of 99.98%. Accuracy was stable across all eight unseen prompts, with no advantage for prompts thematically related to the training data, indicating robust cross-prompt generalization. However, the model exhibited a systematic, L1-linked scoring offset. Within every proficiency band, essays from European-language backgrounds received consistently higher scores than essays from East-Asian-language backgrounds, a pattern not attributable to the composition of the fine-tuning data. This is the first large-scale L1 fairness analysis of a fine-tuned open-weight LLM for automated essay scoring.
Large language models (LLMs) are increasingly used as essay graders in learning analytics, evaluated almost exclusively with agreement statistics. Educational measurement warns that raters also differ in severity, show halo, and drift as instruments. We treat LLM judges as raters and run a pre-registered rater-effects battery (many-facet Rasch severity, residual halo, generalizability/decision studies, cross-version shifts, differential functioning) on public corpora in two languages (ENEM/Essay-BR; ASAP): 2,377 essays, 12 judges, 4 providers, 5 version contrasts, replicated cells, released as a score tensor. Judge severity spans 219 points on ENEM's 0-1000 scale; on ASAP the panel spread is 15-33% of the score range against a between-trained-human gap near 1%. Judge-human correlations sit in an undiscriminating .47-.56 band. All five version contrasts shift severity beyond a family-wise permutation null (up to 133 points), and one judge was deprecated mid-study, caught by identity canaries. Two pre-registered tests returned honest nulls: severity-adjusted leaderboard reversals did not survive a permutation null, and "silent drift" was refuted: agreement moved with severity in four of five contrasts. Replication yields self-consistency (phi>=.80 at k<=2) but not human-level accuracy, and a same-instrument check overturned our own halo comparison: matched on instrument and calibration, we find no credible evidence that judge halo exceeds the trained-human range.