A Grammatical Error Correction (GEC) system produces a sequence of edits to correct an erroneous sentence. The quality of these edits is typically evaluated against human annotations. However, a sentence may admit multiple valid corrections, and existing evaluation settings do not fully accommodate diverse application scenarios. Recent meta-evaluation approaches rely on human judgments across multiple references, but they are difficult to scale to large datasets. In this paper, we propose a new task, Scoring Edit Impact in GEC, which aims to automatically estimate the importance of edits produced by a GEC system. To address this task, we introduce a scoring framework based on an embedded association graph. The graph captures latent dependencies among edits and syntactically related edits, grouping them into coherent groups. We then perform perplexity-based scoring to estimate each edit's contribution to sentence fluency. Experiments across 4 GEC datasets, 4 languages, and 4 GEC systems demonstrate that our method consistently outperforms a range of baselines. Further analysis shows that the embedded association graph effectively captures cross-linguistic structural dependencies among edits.
Figures & tables
Figure 1: An example GEC instance illustrating the task of identifying edit impact. The source text indicates an erroneous text, and the target text indicates the corrected text generated by a GEC system. We list the corresponding edits associated with the human judgments according to their importance.
Figure 2: Overall architecture of the proposed method. Stage 1 extracts latent associations between edits, while stage 2 ranks the merged edits based on their contribution to sentence fluency reduction.
Dataset
System
# Sents
Avg. Len
Avg. Edits
NLPCC18 (ZH)
Standards
462
25.44
3.72
GPT-4o
479
36.89
4.19
GECToR
147
23.40
3.34
T5
66
27.58
3.35
CoNLL14 (EN)
Standards
624
28.88
4.85
GPT-4o
462
30.65
4.38
Table 1: Statistics of GEC datasets computed over gold-standard annotations and system outputs from GPT-4o, GECToR, and T5.
GEC
Method
NLPCC (ZH)
CoNLL14 (EN)
COWSL2H (ES)
FALKO (DE)
System
Sbound
Srank
Sbound
Srank
Sbound
Srank
Sbound
Srank
T5
Random
72.27
64.62
78.49
73.80
79.28
76.07
77.59
72.70
Vanilla Scorer
77.85
76.49
84.03
83.49
89.50
90.77
84.52
82.54
Greedy Scorer
76.94
75.03
82.11
79.97
87.46
87.87
79.02
73.47
Displacy Scorer
77.95
75.78
84.09
82.31
90.92
91.97
86.10
82.82
Ours
86.87
84.09
88.30
87.71
96.19
97.14
86.84
86.20
Table 2: Main results for scoring edit impact in grammatical error correction.
Figure 3: Experimental analysis and visualization of the proposed method. (a) The results of BERTScore and BARTScore measured by incremental gains with increasing applied edits on CoNLL14 dataset under standard. (b) The results of combined scores of methods under various sentence lengths and edit density on CoNLL14 dataset. (c) Comparison of the constructed association graphs in English and Spanish. (d) Consensus audit results of Qwen-labeled edits across multilingual datasets.
Method
Sbound
Srank
Ours + BERTScore
86.58
84.15
Vanilla + BERTScore
75.94
75.13
Ours + BARTScore
79.55
78.11
Vanilla + BARTScore
75.70
72.04
Table 3: Comparison of methods with alternative fluency scores on CoNLL14 dataset and GPT-4o GEC system.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Environment
τ
Latency (s/sent)
Qwen3-8B
Local
0.6709
4.83
DeepSeek-V3.2
Remote
0.6752
5.91
GPT-4o
Remote
0.5987
7.02
Appendix
Table 4: Consistency and efficiency analysis of LLMs for edit scoring. τ represents the average Kendall’s Rank Correlation Coefficient across different prompts.
Association Rule Mining (Data Preparation)
Min Item Frequency
5 (3 for Spanish)
Min Co-occurrence
2
Min Confidence
0.1
Min Lift
1.1
Min Pair Jaccard
0.01
Similarity Filter (Word Jaccard)
0.6
Appendix
Table 5: Detailed Model Configurations and Training Hyperparameters.
Table 9: Case Study of Cross-Sentence Edit Impact Effectiveness.
You are an experienced and tolerant language expert auditing GEC suggestions from an AI assistant.
Tolerance Principle: As long as the assistant’s judgment “makes sense” or “is reasonable”, please provide your approval.
Audit Guidelines:
1. Language is flexible. If the suggestion makes the expression more idiomatic/neat (even if not a hard error), choose agree .
2. Choose disagree only if the assistant makes a low-level error (e.g., creating a new error or ignoring obvious logical chaos).
Output: Directly output agree or disagree for each task.
Appendix
Table 10: Prompt for cross-model label validation and consistency auditing.
English
You are a helpful assistant assisting in evaluating English sentences. Determine if the following sentences are acceptable in general English contexts. … [List of Sentences] …
Actúa como un asistente útil para evaluar oraciones en español. Tu tarea es determinar si las siguientes oraciones son gramaticalmente aceptables. … [Lista de oraciones] …
Criterios: reasonable (aceptable y comprensible); corrected (errores claros o faltas de ortografía).
Appendix
Table 11: Multilingual prompts for identifying mandatory versus stylistic edits across four languages.
English
You are a professional grammar correction assistant. Please correct all grammatical errors and spelling mistakes in the sentence marked with the <input> tag.
Guidelines: 1. Comprehend the sentence context; 2. Identify and correct all errors (grammar, spelling, expressions); 3. Preserve original structure/style; 4. If error-free, return unchanged; 5. Return ONLY the corrected sentence.
Output Format: Please wrap the corrected sentence with the <output> tag.
Please correct the following sentence:
<input> {} </input> /no_think
Chinese
Appendix
Table 12: Multilingual prompt templates used for LLM-based(GPT-4o) Grammatical Error Correction across four languages.