Automatic assessment of textual CEFR levels is an important task in educational natural language processing. However, accurate CEFR level prediction remains challenging due to the naturally imbalanced distribution of CEFR levels and the ambiguous boundaries between adjacent levels. To address these challenges, we propose a unified training objective that combines focal loss with supervised contrastive learning. Focal loss mitigates class imbalance by emphasizing underrepresented and difficult examples, while supervised contrastive learning learns more discriminative representations by encouraging intra-class compactness and inter-class separation. We evaluate the proposed method on sentence- and document-level CEFR assessment in both monolingual and multilingual settings. Experimental results demonstrate that our method consistently outperforms strong baselines across all settings, establishing new state-of-the-art performance on multiple CEFR assessment benchmarks.
Figures & tables
Dataset
Split
Num. of samples
CEFR-SP
train
13,990
dev
1,843
test
1,843
UCEFR-sent / -doc
train
39,473 / 25,537
dev
6,146 / 6,191
test
1,400 / 1,241
Table 1: Number of training, development, and test samples in CEFR-SP and UniversalCEFR.
Dataset
CEFR level
Cross-Entropy
Prototype
SupCon + Focal
( Arase et al., 2022 )
(proposed)
CEFR-SP
A1
0.319±0.257
0.761±0.024
0.779±0.018
A2
0.843±0.012
0.822±0.021
0.844±0.008
B1
0.876±0.013
0.859±0.016
0.871±0.010
B2
0.854±0.027
0.856±0.029
0.869±0.010
C1
0.827±0.033
0.861±0.016
0.856±0.014
Table 2: Per-level F1 and macro-F1 on the CEFR-SP and UniversalCEFR (sentence- and document-level) test sets. For CEFR-SP, we report mean ± std over 12 seeds, discarding the best- and worst-performing seed by macro-F1 (i.e., averaging over the remaining 10 ); for UniversalCEFR, mean ± std over 3 seeds. The best Macro-F1 scores are indicated by bold fonts, while the best per-level F1 scores are indicated by underlines.
Figure 1: Confusion matrices of our model on the CEFR-SP test set against each annotator’s labels. Cells show row-normalised recall, averaged over 12 training seeds discarding the best- and worst-performing seed by macro-F1.
Dataset
Cross-Entropy
Prototype
Proposed
CEFR-SP
−0.033±0.014
−0.014±0.018
0.011±0.005
UCEFR-sent
0.045±0.016
0.116±0.011
0.115±0.014
UCEFR-doc
0.070±0.004
0.168±0.014
0.190±0.014
Table 3: Silhouette scores of the test-set embedding spaces with respect to the gold CEFR levels, averaged over the same seeds as Table 2 . For CEFR-SP, the score is the mean over the two annotators’ label sets.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Split
A1
A2
B1
B2
C1
C2
Total
CEFR-SP (annotator A)
train
513
2,772
6,329
3,603
732
41
13,990
dev
121
442
680
439
125
36
1,843
test
109
440
722
419
119
34
1,843
CEFR-SP (annotator B)
train
77
1,718
5,826
4,727
1,570
72
13,990
dev
45
207
732
588
222
49
1,843
test
43
216
715
595
225
49
1,843
Appendix
Table 4: Number of texts per CEFR level. Every CEFR-SP sentence is labeled by two annotators. For UniversalCEFR the C1 and C2 classes are merged into a single C class, the merged count spans the C1/C2 columns.
Hyperparameter
Proposed
CE
Prototype
τ
0.07
–
–
γ
0.5
–
–
α
0.4
–
0.2
LR
1e−5 / 2e−5
2e−5
1e−5
Appendix
Table 5: Hyperparameters of the proposed method, cross-entropy (CE) and prototype baselines. For the proposed method, the two learning rates (LR) are for the first and second stages, respectively.
Figure 2: t-SNE projections of the test-set embeddings for each method (columns), colored by gold CEFR level. Each panel shows a single training seed: the 6 th-best of 12 seeds by test macro-F1 for CEFR-SP and the 2 nd-best of 3 for UniversalCEFR. Panels are independent projections, so distances are not comparable across panels.
When a text is translated, does the translation retain the complexity of the original? We introduce ComplexityMT, a new challenge for assessing how text complexity and machine translation interact with and influence each other, using the Common European Framework of Reference for Languages (CEFR) levels as the measure of text complexity. Across six languages, including Arabic, Dutch, English, French, Hindi, and Russian, we evaluate three open-weight models, one closed model, and a commercial machine translation system on two tasks: i) correlation of CEFR with translation difficulty, and ii) shifts in CEFR levels of the source texts. Our experiments show that higher CEFR levels make texts more difficult to translate, and that machine translation shifts the CEFR level of the target text compared to the original source, for most languages. These findings provide new insights for researchers and practitioners working on multilingual pedagogical content generation and machine translation difficulty estimation.
Joseph Marvin Imperial, Junhong Liang, Belal Shoer +9
University of Bath · National University Philippines · MBZUAI +7
We present a study on controlled automatic simplification of Russian texts using preference-based optimization algorithms with a focus on adapting texts to the A1, A2 and B1 proficiency levels. We fine-tune YandexGPT using Rejection Sampling and Direct Preference Optimization (DPO) methods to improve control over the complexity of the generated text. We compare several training configurations that differ in dataset composition, text type and the strategy used to construct preference pairs for DPO. For automatic evaluation we train a regression model based on ruRoBERTa-large to predict continuous complexity scores corresponding to CEFR levels. We also consider semantic similarity, lexical coverage and quantitative readability metrics. Human evaluation assesses simplicity, fluency and adequacy. Our best model achieves 87.15% target level accuracy, an MAE of 0.35 CEFR levels and a target interval MAE of 0.11. Manual assessment shows that 75% of the evaluated simplifications match the target morphological and syntactic level, and fluency and adequacy scores are especially high for A2 and B1. Although A1 remains the most challenging level, the model significantly reduces the proportion of vocabulary and grammatical constructions beyond the target level while generally preserving meaning. Our model is freely available on HuggingFace.
Reliable Text Difficulty Assessment is a prerequisite for valid text simplification workflows and personalized learning applications. However, the development of robust assessment models is severely hindered by a critical bottleneck: the scarcity of expert-annotated corpora containing fine-grained difficulty levels (e.g., CEFR), particularly for lower-resource languages. This paper addresses this data scarcity problem in the context of a low-resource European language. We propose a cross-lingual data augmentation strategy that leverages machine translation to transfer labeled resources from high-resource languages to the target low-resource language. We train BERT-based regression models to predict difficulty scores and investigate whether synthetic, translated data can effectively supplement native training sets. Our experiments demonstrate that augmenting scarce native data with machine-translated corpora significantly improves the accuracy of difficulty estimation, offering a viable solution for languages lacking extensive expert annotations.