Exploiting the English Grammar Profile for L2 grammatical analysis with LLMs
Authors: Stefano Bannò, Penny Karanasou, Kate Knill, Mark Gales
Organizations: The authors are with the Institute for Automated Language Teaching and Assessment and the Machine Intelligence Laboratory of the Cambridge University Engineering Department, Trumpington Street, Cambridge CB2 1PZ, UK.
Evaluating the grammatical competence of second language (L2) learners is essential both for providing targeted feedback and for assessing proficiency. To achieve this, we propose a novel framework leveraging the English Grammar Profile (EGP), a taxonomy of grammatical constructs mapped to the proficiency levels of the Common European Framework of Reference (CEFR), to detect learners' attempts at grammatical constructs and classify them as successful or unsuccessful. This detection can then be used to provide fine-grained feedback. Moreover, the grammatical constructs are used as predictors of proficiency assessment by using automatically detected attempts as predictors of holistic CEFR proficiency. For the selection of grammatical constructs derived from the EGP, rule-based and LLM-based classifiers are compared. We show that LLMs outperform rule-based methods on semantically and pragmatically nuanced constructs, while rule-based approaches remain competitive for constructs that rely purely on morphological or syntactic features and do not require semantic interpretation. For proficiency assessment, we evaluate both rule-based and hybrid pipelines and show that a hybrid approach combining a rule-based pre-filter with an LLM consistently yields the strongest performance. Since our framework operates on pairs of original learner sentences and their corrected counterparts, we also evaluate a fully automated pipeline using automatic grammatical error correction. This pipeline closely approaches the performance of semi-automated systems based on manual corrections, particularly for the detection of successful attempts at grammatical constructs. Overall, our framework emphasises learners' successful attempts in addition to unsuccessful ones, enabling positive, formative feedback and providing actionable insights into grammatical development.
Figures & tables
Fig. 1: Diagram of the proposed attempt-based framework.
Text
Can-do statement
Original
Correct
In original
In correct
Attempt
Class
Music used to be my job.
Music used to be my job.
✓
✓
Successful
ω1
Music was use to be my job.
Music used to be my job.
✗
✓
Unsuccessful
ω2
Music is used to evoke emotion.
Music is used to evoke emotion.
✗
✗
No
ω3
I used to play music.
I usually play music.
✓
✗
Other error
TABLE I: Grammatical construct attempt labelling framework adopted in our work. In the example, we used EGP 598 ( Can use ‘used to’ to talk about repeated actions or states in the past that are no longer true ).
Fig. 2: Examples from the EGP website.
Type
EGP
Text
SuperCategory
SubCategory
CEFR
Lexical
37
Can use enough to modify adjectives. (FORM)
adjectives
modifying
B1
266
Can use either … or to connect two words, phrases or clauses. (FORM)
conjunctions
coordinating
B1
295
Can use another to talk about something different. (USE)
determiners
articles
B1
367
Can use would to talk about the future from a point in the past. (USE)
future
future in the past
B1
598
Can use used to to talk about repeated actions or states in the past that are no longer true. (USE)
modality
used to
B1
983
Can use everything as subject, with a singular verb. (FORM)
pronouns
indefinite -thing, -one, -body
A1
TABLE II: Can-do statements considered in the first part of our experiments.
A2
A2+
B1
B1+
B2
B2+
C1
C1+
C2
#Essays
18
78
87
114
100
53
40
13
3
#Sents.
108
435
566
914
1030
634
466
169
30
TABLE III: W&I 2024 statistics.
Fig. 3: Precision–Recall scatterplot for EGP 249 ( Can use a finite subordinate clause, before or after a main clause, with conjunctions to introduce conditions ). Orange points represent the upper-bound precision and recall obtained through MaxPr(r) .
Fig. 4: Lineplot of F1 scores for successful attempts across EGP can-do statements (Manual corrections).
Model
F 1
General
Successful
Unsuccessful
RB
83.52
82.70
68.45
POLKE
78.08
78.26
52.18
Qwen
73.64
72.62
62.15
GPT-4.1
86.36
85.97
68.87
TABLE IV: Average F1 scores for can-do statements implemented in POLKE for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 (manual corrections).
Statement Type
Model
F 1
General
Successful
Unsuccessful
Lexical
RB
76.62
76.26
65.09
Qwen
81.18
80.98
73.63
GPT-4.1
90.46
91.03
81.61
Non-lexical
RB
70.75
68.53
48.05
Qwen
72.77
70.19
54.87
TABLE V: Average F1 scores for attempts, successful, and unsuccessful attempts for all can-do statements for RB, Qwen 2.5 32B, and GPT-4.1 (manual corrections).
Model
Corrections
F 1
General
Successful
Unsuccessful
RB
Manual
73.68
72.39
56.57
GECToR
70.38
71.81
28.80
Qwen
Manual
76.97
75.58
64.25
GECToR
72.96
76.25
47.26
GPT-4.1
Manual
86.16
85.93
73.61
TABLE VI: Average F1 scores for all can-do statements for RB, Qwen 2.5 32B, and GPT-4.1 (manual corrections and GECToR).
Model
On
PCC
SRC
POLKE
General
0.672
0.680
Successful
0.711
0.718
RBFilter (form-based)
General
0.622
0.613
Successful
0.626
0.636
POLKE+Qwen
General attempts
0.719
0.730
Successful
0.754
0.769
TABLE VII: Proficiency assessment performance on the W&I development set of grammatical complexity score-based systems based on 659 form-based can-do statements in the EGP and manual GEC corrections.
GEC
On
PCC
SRC
Manual
General
0.758
0.772
Successful
0.791
0.803
GECToR
General
0.765
0.782
Successful
0.783
0.799
TABLE VIII: Proficiency assessment performance of the RBFilter+Qwen system based on 1,211 can-do statements on the W&I development set with manual and GECTOR GEC corrections.
Fig. 5: Cumulative distribution of detected EGP constructs across CEFR levels. For each essay-level CEFR group (A2–C2), the figure shows the cumulative percentage of general (top) and successful (bottom) attempts at or above each CEFR level. RBFilter+Qwen (Manual corrections).
Fig. 6: eCDFs of grammar-complexity scores derived from general (dotted) and successful (continuous) attempts at grammatical constructs. Curves are shown for each essay-level CEFR group (A2–C2). RBFilter+Qwen (Manual corrections).
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
EGP
#Sents.
Succ.
Unsucc.
No attempt
Lex.
No attempt
Other error
37
300
14.3
2.3
83.3
0.0
266
96
35.4
9.4
55.2
0.0
295
300
54.3
6.0
39.7
0.0
367
363
6.9
2.2
90.9
0.0
598
396
21.5
1.5
76.3
0.7
Appendix
TABLE IX: Classes breakdown - Prevalence (%).
GEC
On
A1
A2
B1
B2
C1
C2
Manual
General
0.50
0.99
0.99
0.99
0.95
0.95
Successful
0.50
0.90
0.99
0.50
0.90
0.80
GECToR
General
0.50
0.95
0.99
0.50
0.60
0.90
Successful
0.50
0.50
0.99
0.50
0.99
0.90
Appendix
TABLE X: Optimal thresholds configuration for the hybrid proficiency assessment systems.
Model
Precision
Recall
F0.5
[ 60 ]
39.3
42.2
39.8
GECToR
49.0
40.3
46.9
Appendix
TABLE XI: GEC performance on W&I development set.
Type
Model
F 1
General
Successful
Unsuccessful
Lexical
RB
71.51
75.32
29.60
Qwen
74.56
80.91
56.41
GPT-4.1
84.38
90.32
56.47
Non-lexical
RB
69.25
68.30
28.01
Qwen
71.36
71.60
38.11
Appendix
TABLE XII: Average F1 scores for all can-do statements for RB, Qwen 2.5 32B, and GPT-4.1 (GECToR corrections).
EGP
Model
General
Successful
Unsuccessful
Lexical
37
RB
92.63
92.68
92.31
POLKE
95.92
96.47
92.31
Qwen
48.13
46.98
44.44
GPT-4.1
90.72
90.48
66.67
266
RB
100.00
98.51
94.74
POLKE
54.24
58.33
18.18
Appendix
TABLE XIII: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for grammatical construct detection and classification in terms of F1 score (manual corrections).
Fig. 7: Precision-Recall scatterplots for EGP 249 ( Can use a finite subordinate clause, before or after a main clause, with conjunctions to introduce conditions ). Orange points represent the upper-bound precision and recall obtained through MaxPr(r) .
Fig. 8: Precision-Recall scatterplots for EGP 598 ( Can use ‘used to’ to talk about repeated actions or states in the past that are no longer true ). Orange points represent the upper-bound precision and recall obtained through MaxPr(r) .
Fig. 9: Precision-Recall curves for EGP 249 ( Can use a finite subordinate clause, before or after a main clause, with conjunctions to introduce conditions ).
Fig. 10: Precision-Recall curves for EGP 598 ( Can use ‘used to’ to talk about repeated actions or states in the past that are no longer true ).
EGP
Model
Precision
Recall
F1
Lexical
37
RB
97.78
88.00
92.63
POLKE
97.92
94.00
95.92
Qwen
32.85
90.00
48.13
GPT-4.1
93.62
88.00
90.72
266
RB
100.00
100.00
100.00
POLKE
100.00
37.21
54.24
Appendix
TABLE XIV: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for general attempts in terms of Precision, Recall, and F1 score (Manual corrections).
EGP
Model
Precision
Recall
F1
Lexical
37
RB
97.44
88.37
92.68
POLKE
97.62
95.35
96.47
Qwen
33.02
81.39
46.98
GPT-4.1
92.68
88.37
90.48
266
RB
100.00
97.06
98.51
POLKE
100.00
41.18
58.33
Appendix
TABLE XV: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for successful attempts in terms of Precision, Recall, and F1 score (Manual corrections).
EGP
Model
Precision
Recall
F1
Lexical
37
RB
100.00
85.71
92.31
POLKE
100.00
85.71
92.31
Qwen
100.00
28.57
44.44
GPT-4.1
62.50
71.43
66.67
266
RB
90.00
100.00
94.74
POLKE
50.00
11.11
18.18
Appendix
TABLE XVI: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for unsuccessful attempts in terms of Precision, Recall, and F1 score (Manual corrections).
EGP
Model
General
Successful
Unsuccessful
Lexical
37
RB
85.39
92.68
0.00
POLKE
88.17
95.35
0.00
Qwen
36.02
47.74
44.44
GPT-4.1
85.71
92.86
18.18
266
RB
88.31
98.51
20.00
POLKE
59.02
61.22
0.00
Appendix
TABLE XVII: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for grammatical construct detection and classification in terms of F1 score (GECToR corrections).
EGP
Model
Precision
Recall
F1
Lexical
37
RB
97.44
76.00
85.39
POLKE
95.35
82.00
88.17
Qwen
23.60
76.00
36.02
GPT-4.1
95.12
78.00
85.71
266
RB
100.00
79.07
88.31
POLKE
100.00
41.86
59.02
Appendix
TABLE XVIII: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for general attempts in terms of Precision, Recall, and F1 score (GECToR corrections).
EGP
Model
Precision
Recall
F1
Lexical
37
RB
97.44
88.37
92.68
POLKE
95.35
95.35
95.35
Qwen
33.04
86.05
47.74
GPT-4.1
95.12
90.70
92.86
266
RB
100.00
97.06
98.51
POLKE
100.00
44.12
61.22
Appendix
TABLE XIX: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for successful attempts in terms of Precision, Recall, and F1 score (GECToR corrections).
EGP
Model
Precision
Recall
F1
Lexical
37
RB
0.00
0.00
0.00
POLKE
0.00
0.00
0.00
Qwen
100.00
28.57
44.44
GPT-4.1
13.33
28.57
18.18
266
RB
100.00
11.11
20.00
POLKE
0.00
0.00
0.00
Appendix
TABLE XX: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for unsuccessful attempts in terms of Precision, Recall, and F1 score (GECToR corrections).
Automated assistants for Grammatical Error Correction are now embedded in educational platforms serving millions of learners, yet three critical gaps remain in this domain: (1) latest-generation Large Language Models (LLMs) lack comprehensive evaluation on grammar correction tasks; (2) whether combining these LLMs improves correction quality is unexplored; and (3) the extent to which reference-based metrics underestimate GEC system performance has not been adequately quantified. In this study, first, we evaluate latest-generation LLMs on edit precision, fluency preservation, and meaning retention, showing fine-tuned GPT-4o achieves state-of-the-art performance across all three dimensions. Second, through grammatical error type analysis we demonstrate that individual LLMs exhibit highly similar error correction patterns (ρ=0.947). Third, we show that reference-based metrics underestimate GEC performance with 73.76% of GPT-4o corrections different from gold standards being equally valid or even superior. These GEC evaluation findings equip educators with guidance for selecting GEC assistants that enhance rather than constrain student linguistic development. We make our data, code, and models publicly available.
Adnan Labib, Qiao Wang, Yixuan Huang +1
King’s College London, London, United Kingdom · Hosei University, Tokyo, Japan · Waseda University, Tokyo, Japan +1
This paper presents new Cantonese ParGram resources and evaluates LLMs for knowledge-driven grammar engineering within a controlled experimental paradigm. Using Cantonese ParGram resources as gold standards, with corresponding English baselines, we investigate whether OpenAI's gpt-oss-120b and GPT-5.4 can generate machine-processable grammars from sentences and target formal structures under systematically varied prompting conditions. GPT-5.4 outperformed gpt-oss-120b, while grammars generated from target formal structures generally outperformed those generated from sentences. Although both models could generate locally plausible phrase-structure rules, lexical entries, and templates, they often struggled to coordinate interacting formal constraints, especially in multi-construction settings. The results characterize both the capabilities and limitations of current LLMs for potential integration into AI-assisted expert workflows: LLMs may support intermediate stages of grammar development, but human linguistic expertise remains central to analysis, validation, and refinement. The study also contributes new Cantonese symbolic grammatical resources.
Corrective feedback is among the best-evidenced drivers of second-language acquisition, yet corrections delivered during lessons rarely accumulate into an actionable view of grammar mastery. Prompted frontier models can provide such a view from learner--tutor lesson transcripts, but they are costly at scale. We close this gap by fine-tuning Qwen3.5 small language models (SLMs) on filtered and rebalanced teacher-generated supervision, then deploying an efficient 0.8B model in an end-to-end grammar mastery tracker for all English learners on our platform. Internalizing the annotation contract into adapter weights enables pairing the 0.8B model with a compact matched prompt rather than verbose instructions. On two human-curated benchmarks, both the deployed 0.8B model and a 4B reference comparator outperform prompted GPT-5.4 and GPT-5.6 Sol in precision and recall under nested matching criteria of increasing strictness: concept, evidence span, and correctness. The deployed 0.8B SLM reduces serving cost by approximately 16×. A feature-level online experiment shows significant gains in learner engagement (+15.8%) and key business metrics, including scheduled hours (+2.1%) and GMV from new lessons (+13.2%).
Marjan Celikik, Ana Peleteiro Ramallo, Javier Morales