Exploiting the English Grammar Profile for L2 grammatical analysis with LLMs

Stefano Bannò; Penny Karanasou; Kate Knill; Mark Gales

cs.CLMar 17, 2026

Exploiting the English Grammar Profile for L2 grammatical analysis with LLMs

Authors: Stefano Bannò, Penny Karanasou, Kate Knill, Mark Gales

Organizations: The authors are with the Institute for Automated Language Teaching and Assessment and the Machine Intelligence Laboratory of the Cambridge University Engineering Department, Trumpington Street, Cambridge CB2 1PZ, UK.

Abstract

Evaluating the grammatical competence of second language (L2) learners is essential both for providing targeted feedback and for assessing proficiency. To achieve this, we propose a novel framework leveraging the English Grammar Profile (EGP), a taxonomy of grammatical constructs mapped to the proficiency levels of the Common European Framework of Reference (CEFR), to detect learners' attempts at grammatical constructs and classify them as successful or unsuccessful. This detection can then be used to provide fine-grained feedback. Moreover, the grammatical constructs are used as predictors of proficiency assessment by using automatically detected attempts as predictors of holistic CEFR proficiency. For the selection of grammatical constructs derived from the EGP, rule-based and LLM-based classifiers are compared. We show that LLMs outperform rule-based methods on semantically and pragmatically nuanced constructs, while rule-based approaches remain competitive for constructs that rely purely on morphological or syntactic features and do not require semantic interpretation. For proficiency assessment, we evaluate both rule-based and hybrid pipelines and show that a hybrid approach combining a rule-based pre-filter with an LLM consistently yields the strongest performance. Since our framework operates on pairs of original learner sentences and their corrected counterparts, we also evaluate a fully automated pipeline using automatic grammatical error correction. This pipeline closely approaches the performance of semi-automated systems based on manual corrections, particularly for the detection of successful attempts at grammatical constructs. Overall, our framework emphasises learners' successful attempts in addition to unsuccessful ones, enabling positive, formative feedback and providing actionable insights into grammatical development.

Figures & tables

Fig. 1: Diagram of the proposed attempt-based framework.

Text		Can-do statement
Original	Correct	In original	In correct	Attempt	Class
Music used to be my job.	Music used to be my job.	✓	✓	Successful	$\omega_{1}$
Music was use to be my job.	Music used to be my job.	✗	✓	Unsuccessful	$\omega_{2}$
Music is used to evoke emotion.	Music is used to evoke emotion.	✗	✗	No	$\omega_{3}$
I used to play music.	I usually play music.	✓	✗	Other error	$\omega_{3}$

TABLE I: Grammatical construct attempt labelling framework adopted in our work. In the example, we used EGP 598 ( Can use ‘used to’ to talk about repeated actions or states in the past that are no longer true ).

Type	EGP	Text	SuperCategory	SubCategory	CEFR
Lexical	37	Can use enough to modify adjectives. (FORM)	adjectives	modifying	B1
	266	Can use either … or to connect two words, phrases or clauses. (FORM)	conjunctions	coordinating	B1
	295	Can use another to talk about something different. (USE)	determiners	articles	B1
	367	Can use would to talk about the future from a point in the past. (USE)	future	future in the past	B1
	598	Can use used to to talk about repeated actions or states in the past that are no longer true. (USE)	modality	used to	B1
	983	Can use everything as subject, with a singular verb. (FORM)	pronouns	indefinite -thing, -one, -body	A1

TABLE II: Can-do statements considered in the first part of our experiments.

	A2	A2+	B1	B1+	B2	B2+	C1	C1+	C2
#Essays	18	78	87	114	100	53	40	13	3
#Sents.	108	435	566	914	1030	634	466	169	30

TABLE III: W&I 2024 statistics.

Fig. 3: Precision–Recall scatterplot for EGP 249 ( Can use a finite subordinate clause, before or after a main clause, with conjunctions to introduce conditions ). Orange points represent the upper-bound precision and recall obtained through $\text{MaxPr}(r)$ .

Fig. 4: Lineplot of F1 scores for successful attempts across EGP can-do statements (Manual corrections).

Model	F 1
Model	General	Successful	Unsuccessful
RB	83.52	82.70	68.45
POLKE	78.08	78.26	52.18
Qwen	73.64	72.62	62.15
GPT-4.1	86.36	85.97	68.87

TABLE IV: Average

F_{1}

scores for can-do statements implemented in POLKE for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 (manual corrections).

Statement Type	Model	F 1
Statement Type	Model	General	Successful	Unsuccessful
Lexical	RB	76.62	76.26	65.09
	Qwen	81.18	80.98	73.63
	GPT-4.1	90.46	91.03	81.61
Non-lexical	RB	70.75	68.53	48.05
	Qwen	72.77	70.19	54.87

TABLE V: Average

F_{1}

scores for attempts, successful, and unsuccessful attempts for all can-do statements for RB, Qwen 2.5 32B, and GPT-4.1 (manual corrections).

Model	Corrections	F 1
Model	Corrections	General	Successful	Unsuccessful
RB	Manual	73.68	72.39	56.57
RB	GECToR	70.38	71.81	28.80
Qwen	Manual	76.97	75.58	64.25
Qwen	GECToR	72.96	76.25	47.26
GPT-4.1	Manual	86.16	85.93	73.61

TABLE VI: Average

F_{1}

scores for all can-do statements for RB, Qwen 2.5 32B, and GPT-4.1 (manual corrections and GECToR).

Model	On	PCC	SRC
POLKE	General	0.672	0.680
POLKE	Successful	0.711	0.718
RBFilter (form-based)	General	0.622	0.613
RBFilter (form-based)	Successful	0.626	0.636
POLKE+Qwen	General attempts	0.719	0.730
POLKE+Qwen	Successful	0.754	0.769

TABLE VII: Proficiency assessment performance on the W&I development set of grammatical complexity score-based systems based on 659 form-based can-do statements in the EGP and manual GEC corrections.

GEC	On	PCC	SRC
Manual	General	0.758	0.772
Manual	Successful	0.791	0.803
GECToR	General	0.765	0.782
GECToR	Successful	0.783	0.799

TABLE VIII: Proficiency assessment performance of the RBFilter+Qwen system based on 1,211 can-do statements on the W&I development set with manual and GECTOR GEC corrections.

Panel of Figure 13 — Fig. 5: Cumulative distribution of detected EGP constructs across CEFR levels. For each essay-level CEFR group (A2–C2), the figure shows the cumulative percentage of general (top) and successful (bottom) attempts at or above each CEFR level. RBFilter+Qwen (Manual corrections).

Fig. 6: eCDFs of grammar-complexity scores derived from general (dotted) and successful (continuous) attempts at grammatical constructs. Curves are shown for each essay-level CEFR group (A2–C2). RBFilter+Qwen (Manual corrections).

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	EGP	#Sents.	Succ.	Unsucc.	No attempt
Lex.					No attempt	Other error
	37	300	14.3	2.3	83.3	0.0
	266	96	35.4	9.4	55.2	0.0
	295	300	54.3	6.0	39.7	0.0
	367	363	6.9	2.2	90.9	0.0
	598	396	21.5	1.5	76.3	0.7

Appendix

TABLE IX: Classes breakdown - Prevalence (%).

GEC	On	A1	A2	B1	B2	C1	C2
Manual	General	0.50	0.99	0.99	0.99	0.95	0.95
Manual	Successful	0.50	0.90	0.99	0.50	0.90	0.80
GECToR	General	0.50	0.95	0.99	0.50	0.60	0.90
GECToR	Successful	0.50	0.50	0.99	0.50	0.99	0.90

Appendix

TABLE X: Optimal thresholds configuration for the hybrid proficiency assessment systems.

Model	Precision	Recall	$F_{0.5}$
[ 60 ]	39.3	42.2	39.8
GECToR	49.0	40.3	46.9

Appendix

TABLE XI: GEC performance on W&I development set.

Type	Model	F 1
Type	Model	General	Successful	Unsuccessful
Lexical	RB	71.51	75.32	29.60
	Qwen	74.56	80.91	56.41
	GPT-4.1	84.38	90.32	56.47
Non-lexical	RB	69.25	68.30	28.01
	Qwen	71.36	71.60	38.11

Appendix

TABLE XII: Average

F_{1}

scores for all can-do statements for RB, Qwen 2.5 32B, and GPT-4.1 (GECToR corrections).

	EGP	Model	General	Successful	Unsuccessful
Lexical	37	RB	92.63	92.68	92.31
		POLKE	95.92	96.47	92.31
		Qwen	48.13	46.98	44.44
		GPT-4.1	90.72	90.48	66.67
	266	RB	100.00	98.51	94.74
		POLKE	54.24	58.33	18.18

Appendix

TABLE XIII: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for grammatical construct detection and classification in terms of

F_{1}

score (manual corrections).

Panel of Figure 20 — Fig. 7: Precision-Recall scatterplots for EGP 249 ( Can use a finite subordinate clause, before or after a main clause, with conjunctions to introduce conditions ). Orange points represent the upper-bound precision and recall obtained through $\text{MaxPr}(r)$ .

Panel of Figure 21 — Fig. 8: Precision-Recall scatterplots for EGP 598 ( Can use ‘used to’ to talk about repeated actions or states in the past that are no longer true ). Orange points represent the upper-bound precision and recall obtained through $\text{MaxPr}(r)$ .

Panel of Figure 22 — Fig. 9: Precision-Recall curves for EGP 249 ( Can use a finite subordinate clause, before or after a main clause, with conjunctions to introduce conditions ).

Panel of Figure 23 — Fig. 10: Precision-Recall curves for EGP 598 ( Can use ‘used to’ to talk about repeated actions or states in the past that are no longer true ).

	EGP	Model	Precision	Recall	$F_{1}$
Lexical	37	RB	97.78	88.00	92.63
		POLKE	97.92	94.00	95.92
		Qwen	32.85	90.00	48.13
		GPT-4.1	93.62	88.00	90.72
	266	RB	100.00	100.00	100.00
		POLKE	100.00	37.21	54.24

Appendix

TABLE XIV: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for general attempts in terms of Precision, Recall, and

F_{1}

score (Manual corrections).

	EGP	Model	Precision	Recall	$F_{1}$
Lexical	37	RB	97.44	88.37	92.68
		POLKE	97.62	95.35	96.47
		Qwen	33.02	81.39	46.98
		GPT-4.1	92.68	88.37	90.48
	266	RB	100.00	97.06	98.51
		POLKE	100.00	41.18	58.33

Appendix

TABLE XV: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for successful attempts in terms of Precision, Recall, and

F_{1}

score (Manual corrections).

	EGP	Model	Precision	Recall	$F_{1}$
Lexical	37	RB	100.00	85.71	92.31
		POLKE	100.00	85.71	92.31
		Qwen	100.00	28.57	44.44
		GPT-4.1	62.50	71.43	66.67
	266	RB	90.00	100.00	94.74
		POLKE	50.00	11.11	18.18

Appendix

TABLE XVI: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for unsuccessful attempts in terms of Precision, Recall, and

F_{1}

score (Manual corrections).

	EGP	Model	General	Successful	Unsuccessful
Lexical	37	RB	85.39	92.68	0.00
		POLKE	88.17	95.35	0.00
		Qwen	36.02	47.74	44.44
		GPT-4.1	85.71	92.86	18.18
	266	RB	88.31	98.51	20.00
		POLKE	59.02	61.22	0.00

Appendix

TABLE XVII: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for grammatical construct detection and classification in terms of

F_{1}

score (GECToR corrections).

	EGP	Model	Precision	Recall	$F_{1}$
Lexical	37	RB	97.44	76.00	85.39
		POLKE	95.35	82.00	88.17
		Qwen	23.60	76.00	36.02
		GPT-4.1	95.12	78.00	85.71
	266	RB	100.00	79.07	88.31
		POLKE	100.00	41.86	59.02

Appendix

TABLE XVIII: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for general attempts in terms of Precision, Recall, and

F_{1}

score (GECToR corrections).

	EGP	Model	Precision	Recall	$F_{1}$
Lexical	37	RB	97.44	88.37	92.68
		POLKE	95.35	95.35	95.35
		Qwen	33.04	86.05	47.74
		GPT-4.1	95.12	90.70	92.86
	266	RB	100.00	97.06	98.51
		POLKE	100.00	44.12	61.22

Appendix

TABLE XIX: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for successful attempts in terms of Precision, Recall, and

F_{1}

score (GECToR corrections).

	EGP	Model	Precision	Recall	$F_{1}$
Lexical	37	RB	0.00	0.00	0.00
		POLKE	0.00	0.00	0.00
		Qwen	100.00	28.57	44.44
		GPT-4.1	13.33	28.57	18.18
	266	RB	100.00	11.11	20.00
		POLKE	0.00	0.00	0.00

Appendix

TABLE XX: Results for RB, POLKE, Qwen 2.5 32B, and GPT-4.1 for unsuccessful attempts in terms of Precision, Recall, and

F_{1}

score (GECToR corrections).

Educational Assessment

Educational assessment measures what learners know and can do, using tests, rubrics, and learning evidence to guide decisions.

See all 151 papers

Multi-Dimensional Evaluation of LLMs for Grammatical Error Correction
May 8, 2026Adnan Labib, Qiao Wang, Yixuan Huang +1LLM Evaluation Grammatical Error Correction
King’s College London, London, United Kingdom · Hosei University, Tokyo, Japan · Waseda University, Tokyo, Japan +1
How Useful are LLMs for Grammar Engineering? Cantonese ParGram Resources and Controlled Experimental Evaluation with English Baselines
Aug 24, 2026Chit-Fung LamLanguage Model Generation Evaluation
University of Oxford · University of Manchester
Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models
Oct 7, 2026Marjan Celikik, Ana Peleteiro Ramallo, Javier MoralesSecond Language Acquisition Small Language Models
Preply

May 8, 2026J/K move · Enter open · S save

Multi-Dimensional Evaluation of LLMs for Grammatical Error Correction

Adnan Labib, Qiao Wang, Yixuan Huang, Zheng Yuan

King’s College London, London, United Kingdom · Hosei University, Tokyo, Japan · Waseda University, Tokyo, Japan+1

LLM Evaluation Grammatical Error Correction Automated Evaluation

Open paper AskPDF arXiv

Automated assistants for Grammatical Error Correction are now embedded in educational platforms serving millions of learners, yet three critical gaps remain in this domain: (1) latest-generation Large Language Models (LLMs) lack comprehensive evaluation on grammar correction tasks; (2) whether combining these LLMs improves correction quality is unexplored; and (3) the extent to which reference-based metrics underestimate GEC system performance has not been adequately quantified. In this study, first, we evaluate latest-generation LLMs on edit precision, fluency preservation, and meaning retention, showing fine-tuned GPT-4o achieves state-of-the-art performance across all three dimensions. Second, through grammatical error type analysis we demonstrate that individual LLMs exhibit highly similar error correction patterns (

ρ=0.947

). Third, we show that reference-based metrics underestimate GEC performance with 73.76% of GPT-4o corrections different from gold standards being equally valid or even superior. These GEC evaluation findings equip educators with guidance for selecting GEC assistants that enhance rather than constrain student linguistic development. We make our data, code, and models publicly available.

Abstract

Educational Assessment

Explore similar work

Multi-Dimensional Evaluation of LLMs for Grammatical Error Correction

How Useful are LLMs for Grammar Engineering? Cantonese ParGram Resources and Controlled Experimental Evaluation with English Baselines

Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models