ArguLens: An Open-Source System for Automated Essay Scoring and Label-Aware Feedback Generation

Weiran Wang; Hongxiang Shi; Huitao Tang; Wenjuan Qin

cs.CLAug 18, 2026

ArguLens: An Open-Source System for Automated Essay Scoring and Label-Aware Feedback Generation

Authors: Weiran Wang, Hongxiang Shi, Huitao Tang, Wenjuan Qin

Organizations: Fudan University

Abstract

Most automated essay scoring (AES) systems output a single holistic score without interpretable evidence and rely on closed APIs that introduce data privacy and cost barriers. We present ArguLens, an opensource, locally deployable system that decomposes AES into three decoupled components: a discourse-move classifier (Qwen2.5-7B-Instruct fine-tuned with LoRA on PERSUADE 2.0), a grade-independent LightGBM scorer over 31 linguistic and discourse features, and a label-aware feedback generator served through vLLM with a Qwen2.5-14BInstruct backbone. A Gradio web UI exposes pluggable inference backends and supports single-essay and batch scoring with downloadable per-essay breakdowns. On an essaydisjoint PERSUADE 2.0 test split, the logitprobe classifier achieves 82.6% accuracy and 0.727 macro-F1; under prompt-grouped 5-fold cross-validation the scorer reaches a mean QWK of 0.813 under an oracle discoursefeature protocol, and an ablation shows that adding gold discourse annotations yields an increment of +0.055 QWK over the lexical+syntactic configuration (paired t-test, p = 0.010). This is a component-level diagnostic rather than an end-to-end classifier-to-scorer result. The feedback generator ships with a structured evaluation protocol; its human-rater study is left to future work. The system is released under Apache 2.0 at https://github.com/wwrwbs/AI_AWE.

Figures & tables

Module	Path	Responsibility
Discourse-Move Classifier	essay_score/infer	Sentence-level 4-way classification (LoRA on Qwen2.5-7B)
Scorer	essay_score/scoring_pipeline	31-feature LightGBM multiclass (scores 1–6)
Feedback Generator	essay_score/feedback	Label-aware English feedback via Qwen2.5-14B (vLLM)
Frontend	essay_score/app_gradio.py	Gradio UI, batch processing, and ZIP export
Training Code	qwen_move_classifier	LoRA SFT with LLaMA-Factory, data prep

Table 1: Core modules and their responsibilities.

Figure 1: Project workflow for linguistically informed writing assessment. The upper lane summarizes offline training and deployment; the lower lane shows online assessment, scoring, grounded feedback, and the personalized report.

Label	Precision	Recall	F1	Support
claim	0.776	0.835	0.805	4,209
data	0.903	0.825	0.862	7,186
counterclaim	0.501	0.761	0.604	343
rebuttal	0.533	0.794	0.638	262
Overall: accuracy 82.58%, macro-F1 0.727, weighted-F1 0.830

Table 2: Discourse-move classifier results on the PERSUADE 2.0 test set (12,000 sentences from 2,730 essays).

	claim	data	counter.	rebut.
claim	3,515	564	81	49
data	964	5,925	171	126
counterclaim	25	50	261	7
rebuttal	23	23	8	208

Table 3: Confusion matrix on the PERSUADE 2.0 test set (rows: gold label; columns: prediction).

Model	QWK $\uparrow$	Acc. (%) $\uparrow$
Our approach
LightGBM (all 31 features)	0.813 (0.048)	62.73 (2.44)
Ablations
LightGBM (lexical only, 16 feats)	0.750 (0.071)	57.57 (2.16)
LightGBM (lexical + syntactic, 24 feats)	0.759 (0.069)	58.69 (2.32)

Table 4: Scorer performance: QWK and accuracy (mean and standard deviation over prompt-grouped 5-fold cross-validation on the 25,996-essay grade-independent feature table).

Component	Backend	p50 (s)	p95 (s)
Classifier (21-sentence reference essay)	HF (BF16)	1.30	1.32
Scorer (one essay)	CPU (32 threads)	0.006	0.009

Table 5: Inference latency by component and backend. Hardware: 1

\times

RTX 5000 Ada (32 GB); 100 runs per component.

Asset	Location	Reason
Source code, documentation, license	Git repository	Version control
Adapter config + tokenizer	Git repository (1.2 MB)	Small, user IP
LightGBM model + scaler	Git repository (33 MB)	Offline reproducibility
LoRA weights	GitHub Release (309 MB)	Avoids Git LFS bandwidth
Base models (Qwen2.5-7B, 14B)	HuggingFace Hub (Apache 2.0)	License restriction
PERSUADE 2.0 corpus	Upstream non-commercial license, repository	Non-commercial research use

Table 6: Asset distribution policy.

Test module	Covers	Pass rate
Batch outputs	ZIP structure, manifest, CSV	3/3
Classifier prompt	Prompt template, label parsing, shared context contract	4/4
Feedback bands	Band thresholds	4/4
Frozen assets	Adapter config, tokenizer, SHA-256	1/1
Gradio contract	I/O schema (single + batch), branding	4/4
Model configuration	Config precedence, path resolution	2/2

Table 7: Test coverage summary.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Feature	Description
Lexical (16; TAALED and basic counts)
word_count	Total word count
awl	Average word length
basic_nfunction_tokens	Function-word token count
lexical_density_tokens	Lexical density (content tokens / total tokens)
msttr50_*	Moving-average segmental TTR, window 50

Appendix

Table 8: The 31 scorer features.

Automated Essay Scoring

Automated essay scoring uses computational models to assess student writing against human ratings, rubrics, or proficiency criteria.

See all 56 papers

Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards
Jul 21, 2026Xuefeng Jin, Jiashuo Zhang, Teng Cao +1Automated Essay Scoring Rubric-Based RL
Alibaba Cloud Computing, Alibaba Group
Qayyem: A Real-time Platform for Scoring Proficiency of Arabic Essays
Mar 1, 2026Hoor Elbahnasawi, Marwan Sayed, Sohaila Eltanbouly +2Automated Essay Scoring Arabic NLP
Computer Science and Engineering Department, Qatar University
From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models
Jun 18, 2026Jiaxu Zuo, Mu You, Kaixin Lan +5LLM Interpretability Automated Essay Scoring
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Institute of International Language Services Studies, Macau Millennium College · School of Data Science and Information Engineering, Guizhou Minzu University

Jul 21, 2026J/K move · Enter open · S save

Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards

Xuefeng Jin, Jiashuo Zhang, Teng Cao, Bin Yang

Alibaba Cloud Computing, Alibaba Group

Automated Essay Scoring Rubric-Based RL Rubric-Based Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) have been widely applied to automated essay scoring (AES) and automated feedback generation (AFG). However, existing studies rely primarily on prompt engineering or supervised fine-tuning, while systematic research on reinforcement learning (RL) post-training and automated evaluation of feedback quality remains limited. We propose RLAES, a unified LLM framework that jointly optimizes essay scoring and feedback generation through RL. To make feedback quality measurable, interpretable, and usable for training, we introduce Rubric-based Feedback Evaluation (RFE), an essay-grounded feedback evaluation framework comprising 166 fine-grained binary rubric items and an LLM-as-judge. Building on RFE, we propose Adaptive Gated Feedback Optimization (AGFO), which activates rubric-based feedback rewards on demand during RL, reducing evaluation overhead while improving feedback quality. We also propose Adjacent Contrastive Reasoning (ACR) to improve ordinal score calibration by explicitly contrasting adjacent score levels. Experimental results show that the RFE framework captures essay-feedback consistency, exhibits strong pairwise discriminative power, and closely aligns with expert preferences. On the ASAP benchmark, RLAES-AGFO achieves the best scoring performance among LLM-based methods (QWK = 0.803), while maintaining feedback quality comparable to GPT-5.5 and avoiding the feedback degradation observed under score-only RL. Code and datasets are publicly available at https://github.com/hellomuyi/RLAES.

Abstract

Automated Essay Scoring

Explore similar work

Beyond Score Prediction: LLM-Based Essay Scoring and Feedback Generation via Reinforcement Learning with Rubric Rewards

Qayyem: A Real-time Platform for Scoring Proficiency of Arabic Essays

From Texts to Scores: Tracing the Emergence of Essay Quality Representations in Large Language Models