cs.CLOct 30, 2026

CEFR Level Assessment across Languages and Text Granularities

Authors: Eugene Matsumura, Yuki Arase

Organizations: RWTH Aachen University · Institute of Science Tokyo

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Automatic assessment of textual CEFR levels is an important task in educational natural language processing. However, accurate CEFR level prediction remains challenging due to the naturally imbalanced distribution of CEFR levels and the ambiguous boundaries between adjacent levels. To address these challenges, we propose a unified training objective that combines focal loss with supervised contrastive learning. Focal loss mitigates class imbalance by emphasizing underrepresented and difficult examples, while supervised contrastive learning learns more discriminative representations by encouraging intra-class compactness and inter-class separation. We evaluate the proposed method on sentence- and document-level CEFR assessment in both monolingual and multilingual settings. Experimental results demonstrate that our method consistently outperforms strong baselines across all settings, establishing new state-of-the-art performance on multiple CEFR assessment benchmarks.

Figures & tables

DatasetSplitNum. of samples
CEFR-SPtrain13,990
 dev1,843
 test1,843
UCEFR-sent / -doctrain39,473 / 25,537
 dev6,146 / 6,191
 test1,400 / 1,241
Table 1: Number of training, development, and test samples in CEFR-SP and UniversalCEFR.
DatasetCEFR levelCross-EntropyPrototypeSupCon + Focal
   ( Arase et al., 2022 )(proposed)
CEFR-SPA10.3190.319 ±0.257\pm 0.2570.7610.761 ±0.024\pm 0.0240.7790.779 ±0.018\pm 0.018
A20.8430.843 ±0.012\pm 0.0120.8220.822 ±0.021\pm 0.0210.8440.844 ±0.008\pm 0.008
B10.8760.876 ±0.013\pm 0.0130.8590.859 ±0.016\pm 0.0160.8710.871 ±0.010\pm 0.010
B20.8540.854 ±0.027\pm 0.0270.8560.856 ±0.029\pm 0.0290.8690.869 ±0.010\pm 0.010
C10.8270.827 ±0.033\pm 0.0330.8610.861 ±0.016\pm 0.0160.8560.856 ±0.014\pm 0.014
Table 2: Per-level F1 and macro-F1 on the CEFR-SP and UniversalCEFR (sentence- and document-level) test sets. For CEFR-SP, we report mean ±\pm std over 1212 seeds, discarding the best- and worst-performing seed by macro-F1 (i.e., averaging over the remaining 1010 ); for UniversalCEFR, mean ±\pm std over 33 seeds. The best Macro-F1 scores are indicated by bold fonts, while the best per-level F1 scores are indicated by underlines.
Figure 1: Confusion matrices of our model on the CEFR-SP test set against each annotator’s labels. Cells show row-normalised recall, averaged over 1212 training seeds discarding the best- and worst-performing seed by macro-F1.
DatasetCross-EntropyPrototypeProposed
CEFR-SP−0.033-0.033 ±0.014\pm 0.014−0.014-0.014 ±0.018\pm 0.0180.011\mathbf{0.011} ±0.005\pm 0.005
UCEFR-sent0.0450.045 ±0.016\pm 0.0160.116\mathbf{0.116} ±0.011\pm 0.0110.1150.115 ±0.014\pm 0.014
UCEFR-doc0.0700.070 ±0.004\pm 0.0040.1680.168 ±0.014\pm 0.0140.190\mathbf{0.190} ±0.014\pm 0.014
Table 3: Silhouette scores of the test-set embedding spaces with respect to the gold CEFR levels, averaged over the same seeds as Table 2 . For CEFR-SP, the score is the mean over the two annotators’ label sets.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

DatasetSplitA1A2B1B2C1C2Total
CEFR-SP (annotator A)train5135132,7722{,}7726,3296{,}3293,6033{,}603732732414113,99013{,}990
 dev12112144244268068043943912512536361,8431{,}843
 test10910944044072272241941911911934341,8431{,}843
CEFR-SP (annotator B)train77771,7181{,}7185,8265{,}8264,7274{,}7271,5701{,}570727213,99013{,}990
 dev454520720773273258858822222249491,8431{,}843
 test434321621671571559559522522549491,8431{,}843
Appendix
Table 4: Number of texts per CEFR level. Every CEFR-SP sentence is labeled by two annotators. For UniversalCEFR the C1 and C2 classes are merged into a single C class, the merged count spans the C1/C2 columns.
HyperparameterProposedCEPrototype
τ\tau0.070.07––
γ\gamma0.50.5––
α\alpha0.40.4–0.20.2
LR1e−51e-5 / 2e−52e-52e−52e-51e−51e-5
Appendix
Table 5: Hyperparameters of the proposed method, cross-entropy (CE) and prototype baselines. For the proposed method, the two learning rates (LR) are for the first and second stages, respectively.
Figure 2: t-SNE projections of the test-set embeddings for each method (columns), colored by gold CEFR level. Each panel shows a single training seed: the 66 th-best of 1212 seeds by test macro-F1 for CEFR-SP and the 22 nd-best of 33 for UniversalCEFR. Panels are independent projections, so distances are not comparable across panels.
More in this topic

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 408 papers

Explore similar work

CardsList

  1. ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation

    Jun 3, 2026Joseph Marvin Imperial, Junhong Liang, Belal Shoer +9Machine TranslationMT Evaluation

    University of Bath · National University Philippines · MBZUAI +7

  2. CEFR-oriented Russian Text Simplification Using Reinforcement Learning

    Oct 19, 2026Kristina Mustakova, Alexey SorokinText SimplificationFine-Tuning Language Models

    MSU Center of Artificial Intelligence · Yandex

  3. Translation as Augmentation: Effect of Translated Data on Assessment of Difficulty

    Jul 21, 2026Yiheng Wu, Jue Hou, Roman YangarberSynthetic Data AugmentationAutomated Evaluation

    University of Helsinki, Finland