cs.CLMar 6, 2026

From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring

Authors: Minh Hoang Nguyen, Vu Hoang Pham, Xuan Thanh Huynh, Phuc Hong Mai, Vinh The Nguyen, Quang Nhut Huynh, Huy Tien Nguyen, Tung Le

Organizations: Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) have recently reshaped Automated Essay Scoring (AES), yet prior studies typically examine individual techniques in isolation, limiting understanding of their relative merits for English as a Second Language (L2) writing. To bridge this gap, we presents a comprehensive comparison of major LLM-based AES paradigms on IELTS Writing Task~2. On this unified benchmark, we evaluate four approaches: (i) encoder-based classification fine-tuning, (ii) zero- and few-shot prompting, (iii) instruction tuning and Retrieval-Augmented Generation (RAG), and (iv) Supervised Fine-Tuning combined with Direct Preference Optimization (DPO) and RAG. Our results reveal clear accuracy-cost-robustness trade-offs across methods, the best configuration, integrating k-SFT and RAG, achieves the strongest overall results with F1-Score 93%. This study offers the first unified empirical comparison of modern LLM-based AES strategies for English L2, promising potential in auto-grading writing tasks. Code is public at https://github.com/MinhNguyenDS/LLM_AES-EnL2

Figures & tables

Figure 1: A framework of LLM-based Automated Essay Scoring. Given an IELTS Writing Task 2 prompt (e.g., “Some people think computers should replace teachers”) and a student essay response, the LLM acts as an examiner to produce (i) an overall band score (e.g., 6.5) and (ii) rubric-aligned feedback across Task Response, Coherence and Cohesion, Lexical Resource, and Grammatical Range and Accuracy.
EraRepresentative SystemsStrengthsLimitations
1960s–2010s (Traditional AES)e-rater ( Attali and Burstein, 2006 )Interpretable, low computational cost, effective with small datasetsLimited semantic understanding, heavy feature engineering, weak cross-prompt generalization
2010s–2020s (Neural AES)LSTM-AES ( Taghipour and Ng, 2016 ) ; BERT-based AES ( Yang et al., 2020 ) ; multi-scale models ( Wang et al., 2022 )Better semantic modeling and improved correlation with human scoresRequires large labeled datasets, limited explainability, prone to prompt overfitting
2020s+ (LLM-based, Prompting)GPT-4 prompting ( Mizumoto and Eguchi, 2023 ) ; LCES ( Shibata and Miyamura, 2025 )No training required, strong reasoning ability, flexible rubric adaptationHigh inference cost, sensitivity to prompt design, and calibration issues
2020s+ (LLM-based, Fine-tuning)Fine-tuned RoBERTa ( Qiu et al., 2024 ) ; instruction-tuned LLMs ( Sun and Wang, 2024 )Stable performance and improved alignment with scoring rubricsExpensive annotation and isolated evaluation of single techniques
2020s+ (LLM-based, Hybrid)Hybrid LLM-based AES systems ( Li and Ng, 2024 ; Wang et al., 2025 )Improved robustness, reduced hallucination, and higher feedback qualityFragmented evaluation and lack of unified benchmarks
Table 1: Comparison of existing AES approaches.
Figure 2: Overview of the four LLM-based AES paradigms evaluated in this study: (1) discriminative fine-tuning, (2) prompting-based inference, (3) instruction tuning with RAG, and (4) SFT with DPO and RAG.
Figure 3: Overview of the discriminative fine-tuning approach for IELTS essay scoring.
Figure 4: In-context learning paradigm for IELTS Writing Task 2 scoring, including prompting-based inference and instruction-tuned generative LLMs.
Figure 5: Overview of k -instruction tuning with Retrieval-Augmented Generation (RAG) for IELTS Writing Task 2 scoring. The model is instruction-tuned on criterion-specific subtasks using LoRA adapters, while external rubric descriptions and exemplar essays are retrieved to ground inference and reduce hallucination.
Figure 6: Overview of Supervised Fine-Tuning with Direct Preference Optimization (DPO) and Retrieval-Augmented Generation (RAG). The model is first fine-tuned to generate score–feedback pairs, then aligned using human preference data, and finally grounded with retrieved exemplars at inference time.
Figure 7: Distribution of analytic criterion scores in the dataset: (left) training set, (right) test set.
StatisticTrainTest
# Essays9,833495
Avg. Essay Length (tokens)231228
Score Range0–90–9
Mean Overall Score5.55.5
Std. Dev.1.21.1
Table 2: Dataset statistics for training and test splits.
Figure 8: Distribution of overall band scores in the dataset: (left) training set, (right) test set.
ApproachModel / SettingTraining / Prompting Schemek-shotAccuracyF1RMSEMAE
A1RoBERTa-base (BERT-based)Classifier Only–0.65100.21600.83000.5810
 RoBERTa-base (BERT-based)All Parameters–0.73100.30400.78400.5490
 GPT-2 (LLM-based)Classifier Only–0.70700.27800.75700.5300
 GPT-2 (LLM-based)All Parameters–0.73500.30900.77000.5390
A2GPT-4oPrompting Tuning2-shot0.71000.28101.05000.7350
 GPT-4oPrompting Tuningzero-shot0.72000.29201.13000.7910
Table 3: Main results on IELTS Writing Task 2 Automated Essay Scoring. A1: Discriminative fine-tuning; A2: Prompting-based; A3: Instruction tuning; A4: Supervised fine-tuning with preference optimization. Higher is better for Accuracy and F1-score, lower is better for RMSE and MAE. Bold best results, underline second-best.
Figure 9: Cost–accuracy–robustness trade-offs across LLM-based AES paradigms for IELTS Writing Task 2.
TopicEssay ExcerptGoldPredicted BandLLM Feedback (excerpt)
Technology & Education“Some people think computers should replace teachers…”6.5A3+RAG : 6.5 A4 : 6.5A3+RAG : Task Response : … Addresses both views clearly and remains on-topic , though the evaluation in the conclusion is limited … Coherence : Paragraphing follows a logical structure , with minor lapses in cohesion . Lexical : Uses an adequate range of topic-related vocabulary … Grammar : Mostly accurate sentence forms , despite occasional tense errors … A4 : A clear and consistent position is maintained throughout the essay . Ideas are generally well connected and easy to follow , although smoother transitions would strengthen coherence …
Urbanization“Living in big cities brings more benefits than problems…”7.0A3+RAG : 7.0 A4 : 6.5A3+RAG : Task Response : Presents well-developed and balanced arguments … Coherence : Ideas progress clearly across paragraphs . Lexical : Demonstrates a wide range of topic-specific vocabulary … Grammar : Uses complex sentence structures with good control … A4 : … While the main ideas are relevant, their development is uneven ; some paragraphs lack focus, which results in a slightly lower overall band …
Environmental Protection“Governments should invest more in renewable energy…”6.0A3+RAG : 6.0 A4 : 6.0A3+RAG : Task Response : … Only partially addresses the task , with limited supporting examples. Coherence : … Basic organization and weak use of linking devices … Lexical : Simple but generally understandable vocabulary … Grammar : Frequent minor errors reduce clarity … A4 : The main ideas are understandable and relevant , but they are insufficiently extended . Language control remains limited, with noticeable grammatical errors affecting overall clarity …
Table 4: Case study comparison between A3+RAG and A4. Correct band predictions and criterion-aligned strengths are highlighted in green, while band mismatches and feedback limitations are shown in orange.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

SettingStudy 1 (RoBERTa)Study 2 (GPT-2 Encoder)
Model backboneRoBERTa-baseGPT-2
Training modeClassifier-onlyClassifier-only
Batch size164
Epochs208
OptimizerAdamWAdamW
Initial learning rate2×10−52\times 10^{-5} / 10−210^{-2}5×10−65\times 10^{-6}
Appendix
Table 5: Hyperparameters for discriminative fine-tuning experiments (Approach 1).
SettingGemma-7BLlama-3.1-8BMistral-7BPhi-2
Fine-tuning methodLoRA (SFT)LoRA (SFT)LoRA (SFT)LoRA (SFT)
LoRA rank rr16161616
LoRA α\alpha16161616
Target modulesQ,K,V,O,FFNQ,K,V,O,FFNQ,K,V,O,FFNQ,K,V,O,FFN
Trainable params∼\sim 0.3%∼\sim 0.3%∼\sim 0.3%∼\sim 0.3%
Epochs3333
Appendix
Table 6: Hyperparameters for instruction tuning with LoRA (Approaches 2 and 3).
SettingValue
Backbone modelsLlama-3.1-8B, Mistral-7B
Fine-tuning stagesSFT →\rightarrow DPO
LoRA rank rr16
LoRA α\alpha16 / 32
Batch size2
Gradient accumulation4
Appendix
Table 7: Hyperparameters for supervised fine-tuning with DPO and RAG (Approach 4).
Figure 10: Prompt template used for re-generation of analytic scores in the IELTS Writing Task 2 dataset.
More in this topic

Automated Essay Scoring

Automated essay scoring uses computational models to assess student writing against human ratings, rubrics, or proficiency criteria.

See all 56 papers

Explore similar work

CardsList
Aug 24, 2026

Learning to Grade Efficiently: A Bandit-Driven Prompt-Selection Framework for Low-Cost LLM Essay Scoring

Large Language Models (LLMs) demonstrate strong capabilities in automated essay scoring (AES), but contemporary approaches typically employ fixed prompt selection, failing to address operational cost concerns and evolving optimal configurations. We propose a novel cost-aware approach that treats each prompt type as an arm in a multi-armed bandit (MAB) controller, enabling adaptive selection of optimal prompting strategies during inference. Our experiments on IELTS Writing Task 2 essays show that the MAB framework achieves comparable scoring accuracy to exhaustive grid search while reducing LLM calls by 78.4% to find the best grading approach. We implemented four distinct grading recipes (multi-step vs. single-step assessment, with vs. without calibration examples) and found that the multi-step approach with examples achieves the highest accuracy. By tracking token usage and latency alongside agreement metrics, we produce the first cost-reliability learning curves for essay scoring, providing actionable insights for educational technology platforms that must balance operational costs against assessment validity. This work represents the first application of online control mechanisms to adaptively select prompting strategies in AES, transforming prompt selection from an offline hyperparameter optimization problem into an efficient online learning task.

Olga Manakina, Igor Bogdanov

Carleton University

Automated Essay ScoringCost-Aware InferenceLarge Language Model-Guided Optimization
AskPDF
Jun 18, 2026

From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models

Recent advances in Large Language Models (LLMs) have substantially transformed Automated Essay Scoring (AES), yet the internal mechanisms underlying LLM-based scoring remain poorly understood. In this work, we systematically analyze the hidden representations of eight LLMs across two English essay datasets (ASAP++, CSEE) and one Portuguese dataset (ENEM). Using linear probing, cross-prompt generalization, dimensionality reduction, and neuron-level analyses, we find consistent evidence that essay quality information is encoded in a linearly accessible form within LLM representations. These representations emerge progressively across layers, remain robust across prompting strategies, and partially transfer across essay prompts despite differences in scoring rubrics. In addition, nonlinear probes provide only marginal and inconsistent improvements over linear probes, suggesting that most essay quality information is already linearly decodable. We further identify individual ``essay scoring neurons'' whose activations strongly correlate with essay scores and whose behavior is sensitive to targeted intervention. Moreover, the layer-wise distribution of these neurons systematically shifts with essay length, with longer essays relying more heavily on deeper layers. Overall, our findings provide evidence that LLMs encode structured representations related to essay quality and offer new insights into the interpretability of LLM-based AES systems.

Jiaxu Zuo, Mu You, Kaixin Lan +5

NLP2CT Lab, Department of Computer and Information Science, University of Macau · Institute of International Language Services Studies, Macau Millennium College · School of Data Science and Information Engineering, Guizhou Minzu University

LLM InterpretabilityAutomated Essay ScoringRepresentation Geometry in Language ModelsLanguage Model Probing
AskPDF
Jun 9, 2026

The Order Matters: Sequential Fine-Tuning of LLaMA for Coherent Automated Essay Scoring

Automated Essay Scoring (AES) systems must judge interdependent discourse elements (e.g., lead, claim, evidence, conclusion), yet most approaches treat these in isolation, harming coherence and generalization. We investigate task-aware fine-tuning of LLaMA-3.1-8B for AES using parameter-efficient LoRA with 4-bit quantization and compare three training curricula: (i) Sequential (progressively fine-tuning on lead, then position, then claim, then evidence, then conclusion), (ii) Independent (task-specific models), and (iii) Randomized (shuffled multi-task). Experiments on the PERSUADE~2.0 corpus show that modeling task dependencies matters: Sequential fine-tuning yields the strongest overall results, including F1 scores of 65% (evidence) and 87% (conclusion) and corresponding accuracies of 63% and 85%, surpassing Independent training and outperforming a general-purpose LLaMA-70B baseline on conclusion despite its far larger capacity. Randomized training improves position scoring (57% F1) but is less consistent elsewhere. These findings indicate that (1) curriculum design aligned with discourse structure can materially improve AES, and (2) small, task-optimized models can be competitive with substantially larger Large Language Models (LLM), offering a practical path to scalable, cost-effective assessment. We release templates and implementation details to facilitate reproduction and future work on curriculum design for educational NLP.

Ali Keramati, Mark Warschauer

University of California, Irvine

Fine-TuningLLM Fine-TuningAutomated Essay ScoringCurriculum Learning
AskPDF