A Grammatical Error Correction (GEC) system produces a sequence of edits to correct an erroneous sentence. The quality of these edits is typically evaluated against human annotations. However, a sentence may admit multiple valid corrections, and existing evaluation settings do not fully accommodate diverse application scenarios. Recent meta-evaluation approaches rely on human judgments across multiple references, but they are difficult to scale to large datasets. In this paper, we propose a new task, Scoring Edit Impact in GEC, which aims to automatically estimate the importance of edits produced by a GEC system. To address this task, we introduce a scoring framework based on an embedded association graph. The graph captures latent dependencies among edits and syntactically related edits, grouping them into coherent groups. We then perform perplexity-based scoring to estimate each edit's contribution to sentence fluency. Experiments across 4 GEC datasets, 4 languages, and 4 GEC systems demonstrate that our method consistently outperforms a range of baselines. Further analysis shows that the embedded association graph effectively captures cross-linguistic structural dependencies among edits.
Figures & tables
Figure 1: An example GEC instance illustrating the task of identifying edit impact. The source text indicates an erroneous text, and the target text indicates the corrected text generated by a GEC system. We list the corresponding edits associated with the human judgments according to their importance.
Figure 2: Overall architecture of the proposed method. Stage 1 extracts latent associations between edits, while stage 2 ranks the merged edits based on their contribution to sentence fluency reduction.
Dataset
System
# Sents
Avg. Len
Avg. Edits
NLPCC18 (ZH)
Standards
462
25.44
3.72
GPT-4o
479
36.89
4.19
GECToR
147
23.40
3.34
T5
66
27.58
3.35
CoNLL14 (EN)
Standards
624
28.88
4.85
GPT-4o
462
30.65
4.38
Table 1: Statistics of GEC datasets computed over gold-standard annotations and system outputs from GPT-4o, GECToR, and T5.
GEC
Method
NLPCC (ZH)
CoNLL14 (EN)
COWSL2H (ES)
FALKO (DE)
System
Sbound
Srank
Sbound
Srank
Sbound
Srank
Sbound
Srank
T5
Random
72.27
64.62
78.49
73.80
79.28
76.07
77.59
72.70
Vanilla Scorer
77.85
76.49
84.03
83.49
89.50
90.77
84.52
82.54
Greedy Scorer
76.94
75.03
82.11
79.97
87.46
87.87
79.02
73.47
Displacy Scorer
77.95
75.78
84.09
82.31
90.92
91.97
86.10
82.82
Ours
86.87
84.09
88.30
87.71
96.19
97.14
86.84
86.20
Table 2: Main results for scoring edit impact in grammatical error correction.
Figure 3: Experimental analysis and visualization of the proposed method. (a) The results of BERTScore and BARTScore measured by incremental gains with increasing applied edits on CoNLL14 dataset under standard. (b) The results of combined scores of methods under various sentence lengths and edit density on CoNLL14 dataset. (c) Comparison of the constructed association graphs in English and Spanish. (d) Consensus audit results of Qwen-labeled edits across multilingual datasets.
Method
Sbound
Srank
Ours + BERTScore
86.58
84.15
Vanilla + BERTScore
75.94
75.13
Ours + BARTScore
79.55
78.11
Vanilla + BARTScore
75.70
72.04
Table 3: Comparison of methods with alternative fluency scores on CoNLL14 dataset and GPT-4o GEC system.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Environment
τ
Latency (s/sent)
Qwen3-8B
Local
0.6709
4.83
DeepSeek-V3.2
Remote
0.6752
5.91
GPT-4o
Remote
0.5987
7.02
Appendix
Table 4: Consistency and efficiency analysis of LLMs for edit scoring. τ represents the average Kendall’s Rank Correlation Coefficient across different prompts.
Association Rule Mining (Data Preparation)
Min Item Frequency
5 (3 for Spanish)
Min Co-occurrence
2
Min Confidence
0.1
Min Lift
1.1
Min Pair Jaccard
0.01
Similarity Filter (Word Jaccard)
0.6
Appendix
Table 5: Detailed Model Configurations and Training Hyperparameters.
Table 9: Case Study of Cross-Sentence Edit Impact Effectiveness.
You are an experienced and tolerant language expert auditing GEC suggestions from an AI assistant.
Tolerance Principle: As long as the assistant’s judgment “makes sense” or “is reasonable”, please provide your approval.
Audit Guidelines:
1. Language is flexible. If the suggestion makes the expression more idiomatic/neat (even if not a hard error), choose agree .
2. Choose disagree only if the assistant makes a low-level error (e.g., creating a new error or ignoring obvious logical chaos).
Output: Directly output agree or disagree for each task.
Appendix
Table 10: Prompt for cross-model label validation and consistency auditing.
English
You are a helpful assistant assisting in evaluating English sentences. Determine if the following sentences are acceptable in general English contexts. … [List of Sentences] …
Actúa como un asistente útil para evaluar oraciones en español. Tu tarea es determinar si las siguientes oraciones son gramaticalmente aceptables. … [Lista de oraciones] …
Criterios: reasonable (aceptable y comprensible); corrected (errores claros o faltas de ortografía).
Appendix
Table 11: Multilingual prompts for identifying mandatory versus stylistic edits across four languages.
English
You are a professional grammar correction assistant. Please correct all grammatical errors and spelling mistakes in the sentence marked with the <input> tag.
Guidelines: 1. Comprehend the sentence context; 2. Identify and correct all errors (grammar, spelling, expressions); 3. Preserve original structure/style; 4. If error-free, return unchanged; 5. Return ONLY the corrected sentence.
Output Format: Please wrap the corrected sentence with the <output> tag.
Please correct the following sentence:
<input> {} </input> /no_think
Chinese
Appendix
Table 12: Multilingual prompt templates used for LLM-based(GPT-4o) Grammatical Error Correction across four languages.
Grammatical error correction (GEC) evaluation has traditionally relied on reference or edit overlap, which can penalize valid rewrites that differ from gold corrections. Reference-free metrics reduce this dependence, but evaluating whether a fluent output is a valid correction of the source remains challenging. We propose SURE, a source-conditioned reward evaluator trained on within-source preferences spanning minimal-edit and rewrite-oriented corrections. SURE jointly learns an overall reward with criteria-level supervision for grammaticality, faithfulness, and fluency, together with span-level grounding for source-side error resolution. Experiments on SEEDA show that SURE performs competitively against strong baselines, with particular gains on rewrite-style corrections and more disentangled criteria-level diagnostics. Our code is available at https://github.com/hayeonggg/SURE.
Hayeong Ryu, Sunhee Jo, Seunguk Yu +1
Department of Artificial Intelligence, Chung-Ang University · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
Reference-based metrics for Grammatical Error Correction (GEC) such as M2 and ERRANT assume that the reference set enumerates all valid edits, and therefore often penalize corrections that are grammatical and meaning-preserving but phrased differently. We introduce RM-EVAL, a reward model trained on human preference data from SEEDA, as a reference-free meta-evaluator that predicts human-like quality judgments at both full-sequence and partial-sequence levels. Beyond evaluation, we show that the same reward model can be used as a learning signal to improve GEC generation via Reward-Guided Text Generation (RGTG), which keeps a base GEC model frozen and performs online, reward-driven decoding. Across SEEDA, RM-EVAL achieves strong agreement with human rankings, and RGTG yields consistent gains in reward and external validation, demonstrating a unified framework for both assessing and enhancing GEC systems without relying on gold references.
Ruotian Wu, Bill E. Johnson, Gene Saunders +3
University of Waterloo · Vector Institute · Scribendi Inc.
Automated assistants for Grammatical Error Correction are now embedded in educational platforms serving millions of learners, yet three critical gaps remain in this domain: (1) latest-generation Large Language Models (LLMs) lack comprehensive evaluation on grammar correction tasks; (2) whether combining these LLMs improves correction quality is unexplored; and (3) the extent to which reference-based metrics underestimate GEC system performance has not been adequately quantified. In this study, first, we evaluate latest-generation LLMs on edit precision, fluency preservation, and meaning retention, showing fine-tuned GPT-4o achieves state-of-the-art performance across all three dimensions. Second, through grammatical error type analysis we demonstrate that individual LLMs exhibit highly similar error correction patterns (ρ=0.947). Third, we show that reference-based metrics underestimate GEC performance with 73.76% of GPT-4o corrections different from gold standards being equally valid or even superior. These GEC evaluation findings equip educators with guidance for selecting GEC assistants that enhance rather than constrain student linguistic development. We make our data, code, and models publicly available.
Adnan Labib, Qiao Wang, Yixuan Huang +1
King’s College London, London, United Kingdom · Hosei University, Tokyo, Japan · Waseda University, Tokyo, Japan +1