This paper presents a novel prompt engineering framework for trait specific Automatic Essay Scoring (AES) in Arabic, leveraging large language models (LLMs) under zero-shot and few-shot configurations. Addressing the scarcity of scalable, linguistically informed AES tools for Arabic, we introduce a three-tier prompting strategy (standard, hybrid, and rubric-guided) that guides LLMs in evaluating distinct language proficiency traits such as organization, vocabulary, development, and style. The hybrid approach simulates multi-agent evaluation with trait specialist raters, while the rubric-guided method incorporates scored exemplars to enhance model alignment. In zero and few-shot settings, we evaluate eight LLMs on the QAES dataset, the first publicly available Arabic AES resource with trait level annotations. Experimental results using Quadratic Weighted Kappa (QWK) and Confidence Intervals show that Fanar-1-9B-Instruct achieves the highest trait level agreement in both zero and few-shot prompting (QWK = 0.28 and CI = 0.41), with rubric-guided prompting yielding consistent gains across all traits and models. Discourse-level traits such as Development and Style showed the greatest improvements. These findings confirm that structured prompting, not model scale alone, enables effective AES in Arabic. Our study presents the first comprehensive framework for proficiency oriented Arabic AES and sets the foundation for scalable assessment in low resource educational contexts.
Figures & tables
Level
Approach
Key Mechanism
1
Standard Prompting
Direct scoring of all traits via a structured zero-shot prompt
2
Hybrid Trait Prompting
Multi-agent simulation using domain-specialist raters
3
Rubric-Guided Few-Shot Prompting
Trait-specific prompts with rubric guides and scored examples
Word choice, lexical diversity, sophistication, repetition
C
Grammar, Spelling & Mechanics
Punctuation, syntax, spelling, readability
D
Content Development & Reasoning
Argument quality, elaboration, evidence use
E
Style, Tone & Contextual Appropriateness
Voice, stylistic consistency, audience alignment
Table 2: The Five Rater Specializations
Rubric
Assigned Raters
Organization
A, D, C
Vocabulary
B, E, C
Style
B, E, C
Development
D, A, B
Mechanics
C
Structure
A, B, C
Table 3: Rubric-to-Rater Mapping
Essay
Org.
Vocab.
Style
Dev.
Mech.
Struct.
Rel.
Final
1
4
4
4
4
4
4
2
26
2
4
4
4
4
4
4
2
26
3
5
4
4
4
4
4
2
27
4
5
5
4
4
4
4
2
28
Table 4: Sample trait annotations for 10 essays in the QAES dataset Bashendy et al. (2024) .
Model
Prompt Level
Org.
Vocab.
Style
Dev.
Mech.
Struct.
Rel.
Total
Qwen1.5-1.8B-
L1
0.039
0.016
0.005
0.015
0.014
0.018
0.006
0.014
Chat
L2
0.001
0.058
0.035
0.045
-0.017
0.024
0.115
0.045
L3
-0.003
-0.033
0.001
0.002
0.040
-0.020
0.051
0.006
Qwen2.5-7B-
L1
0.015
0.016
0.074
0.015
0.067
0.007
0.00
0.021
Instruct
L2
0.157
-0.028
-0.040
0.004
-0.023
-0.000
-0.026
0.067
L3
0.099
0.101
0.049
-0.010
0.064
0.072
-0.071
0.040
Table 5: Quadratic Weighted Kappa (QWK) scores for each language trait and total score across all models and prompt levels. GPT-4 results for Prompt Level 3 are omitted due to the high computational cost of running extended evaluations.
Figure 1: QWK Scores with 95% Confidence Intervals for All Traits
Metric
Baseline
Our Score
Threshold
QWK
0.26
0.28
0.72
CI Upper
–
0.41
–
Table 6: Comparison of QWK scores and confidence intervals against the threshold and baseline achieved by QAES.
Figure 2: QWK Scores with 95% Confidence Intervals for Prompt Levels 1
Figure 3: QWK Scores with 95% Confidence Intervals for Prompt Level 2
Figure 4: QWK Scores with 95% Confidence Intervals for Prompt Level 3