A CEFR-Inspired Classification Framework with Fuzzy C-Means To Automate Assessment of Programming Skills in Scratch

Ricardo Hidalgo-Aragón; Jesús M. González-Barahona; Gregorio Robles

cs.CYApr 1, 2026

A CEFR-Inspired Classification Framework with Fuzzy C-Means To Automate Assessment of Programming Skills in Scratch

Authors: Ricardo Hidalgo-Aragón, Jesús M. González-Barahona, Gregorio Robles

Organizations: Universidad Rey Juan Carlos, Madrid, Spain

Abstract

Context: Schools, training platforms, and technology firms increasingly need to assess programming proficiency at scale with transparent, reproducible methods that support personalized learning pathways. Objective: This study introduces a pedagogical framework for Scratch project assessment, aligned with the Common European Framework of Reference (CEFR), providing universal competency levels for students and teachers alongside actionable insights for curriculum design. Method: We apply Fuzzy C-Means clustering to 2008246 Scratch projects evaluated via Dr.Scratch, implementing an ordinal criterion to map clusters to CEFR levels (A1-C2), and introducing enhanced classification metrics that identify transitional learners, enable continuous progress tracking, and quantify classification certainty to balance automated feedback with instructor review. Impact: The framework enables diagnosis of systemic curriculum gaps-notably a "B2 bottleneck" where only 13.3% of learners reside due to the cognitive load of integrating Logic Synchronization, and Data Representation--while providing certainty--based triggers for human intervention.

Figures & tables

Dimension	Mean	SD	Q1	Med	Q3
Abstraction	1.47	1.36	1	1	3
Parallelization	1.75	1.60	0	1	4
Logic	1.26	1.60	0	0	3
Synchronization	1.54	1.42	0	1	3
Flow Control	2.07	1.22	1	2	3
User Interactivity	1.55	0.82	1	2	2

Table 1: Descriptive statistics for the complete dataset (N = 2,008,246)

Figure 1: Gap statistic for $k=2$ to $k=12$ clusters. Monotonic increase without a clear peak and negative values for $k\leq 6$ reflect the continuous nature of programming skill distributions.

Figure 2: Train-test performance comparison across four key metrics. Bars represent metric values for training (blue) and test (coral) sets. Difference labels show absolute and percentage changes, demonstrating minimal generalization gap.

CEFR	Abs.	Par.	Log.	Syn.	Flow	User	Data	Math	Mot.	$S_{j}$
A1	0.12	0.18	0.09	0.15	0.52	1.01	0.24	0.08	0.31	2.70
A2	0.89	1.02	0.65	0.88	1.76	1.45	1.34	0.43	1.15	9.57
B1	1.58	1.89	1.35	1.58	2.45	1.68	2.18	0.82	1.92	15.45
B2	2.21	2.64	2.01	2.18	2.85	1.84	2.89	1.21	2.55	20.38
C1	2.78	3.18	2.62	2.71	3.15	1.95	3.45	1.59	3.04	24.47
C2	3.29	3.61	3.19	3.19	3.38	2.03	3.87	1.96	3.46	27.98

Table 2: Ordered cluster centroids with CEFR level assignments (values represent mean CT scores on 0–4 scale)

Figure 3: PCA projection of training set observations colored by FCM cluster assignment. Red X markers indicate CEFR-ordered centroids. Substantial overlap between adjacent levels reflects natural skill progression continuum.

CT Dimension	$\tau$	$p_{\tau}$	$\rho$	$p_{\rho}$
Synchronization	0.966	$<0.001$	1.000	$<0.001$
Flow Control	0.894	$<0.001$	0.986	$<0.001$
Logic	0.873	$<0.001$	0.971	$<0.001$
Abstraction	0.845	$<0.001$	0.957	$<0.001$
Data Representation	0.822	$<0.001$	0.943	$<0.001$
Parallelization	0.810	$<0.001$	0.929	$<0.001$

Table 3: Dimension-specific rank correlations with CEFR level

Figure 4: Enhanced classification system results on test set. (Top left) Distribution by assigned level including transition states (e.g., A1-A2, B2-C1). (Top right) Distribution by primary level ( argmax equivalent). (Bottom left) Pie chart showing classification type proportions. (Bottom right) Histogram of continuous scores (1–6 scale) with CEFR level markers.

Figure 5: Certainty analysis on test set. (Left) Distribution by categorical certainty level (Low/Medium/High). (Center) Histogram of continuous certainty values with threshold markers and mean indicator. (Right) Box plots showing certainty distributions stratified by classification type.

Figure 6: Example A1 (Clear) classification. Complete beginner profile with all CT dimensions at zero. Primary membership = 0.996, certainty = 0.99, continuous score = 1.00/6.00.

Figure 7: Example B2-B1 (Transition) classification. Mixed skill profile with heterogeneous CT scores. Membership balanced between B2 (0.319) and B1 (0.274), certainty = 0.15, continuous score = 3.87/6.00.

Figure 8: Example C2 (Clear) classification. Expert profile with maximal scores (4) across all nine CT dimensions. Primary membership = 0.968, certainty = 0.90, continuous score = 5.95/6.00.

Level	Certainty	Suggested Intervention
A1–A2	High	“How could you make this character repeat its dance?” (Flow Control)
B1	High	“What if you needed to remember the player’s score?” (Data)
B2	Any	Trigger synchronization challenge activity
B2–C1	Low	Flag for instructor review + peer mentoring
C1–C2	High	Can you spot repeated patterns to simplify?

Table 4: Automated feedback strategies by classification

Educational Assessment

Educational assessment measures what learners know and can do, using tests, rubrics, and learning evidence to guide decisions.

See all 151 papers

TeachGrade: Training-Free Short Answer Scoring with Teacher Evidence
Oct 10, 2026Jinhao Yan, Zhihong Shen, Shengjie Wang +4Automated Grading LLM-as-a-Judge
Fuzhou University · Peking University
NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming
Jun 20, 2026Heejin Do, Alexandre Ballenghien, Yang Wu +1Code Generation Code Generation Evaluation
Department of Computer Science, ETH Zurich · ETH AI Center
Using LLMs for Knowledge Component-level Correctness Labeling in Open-ended Coding Problems
Feb 19, 2026Zhangqi Duan, Arnav Kankaria, Dhruv Kartik +1Programming Education User Modeling
University of Massachusetts Amherst

Oct 10, 2026J/K move · Enter open · S save

TeachGrade: Training-Free Short Answer Scoring with Teacher Evidence

Jinhao Yan, Zhihong Shen, Shengjie Wang, Yiting Ren+3

Fuzhou University · Peking University

Automated Grading LLM-as-a-Judge Automated Essay Scoring Education Rubric-Based Evaluation

Open paper AskPDF arXiv

Multi-subject, multi-step scoring requires consistent partial credit across grading conventions. We introduce TeachGrade, a training-free framework using teacher-graded examples. The Task Adapt module builds two frozen memories: Skill captures contextual teacher preferences beyond the rubric; CaseBank supplies local evidence. PreJudge provides retrieval cues; FinalJudge regrades the answer without preliminary scores. On 3,829 SAS-Bench answers across 12 tasks, official macro collaborative consistency score (CCS) rises from Native's 67.99 to 81.32 (+13.33 points), and pooled step exact match from 54.85% to 62.71%. Macro quadratic weighted kappa (QWK) reaches 84.71, exceeding Native, OpenJudge, ProofGrader, and Trojsten by 9.52, 3.50, 11.37, and 17.43 points, respectively, and leading on 8 tasks. Local Qwen3.8-27B gains 8.83 CCS and 4.08 QWK points, improving CCS on all 12 tasks. QWK gains over Native span 6.38--14.65 points on Portuguese automatic essay scoring (AES) and 2.16--2.24 on ASAP-SAS across label views, with lower mean absolute error. These results support reuse across languages, response formats, and backbones without weight updates.

Abstract

Educational Assessment

Explore similar work

TeachGrade: Training-Free Short Answer Scoring with Teacher Evidence

NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming

Using LLMs for Knowledge Component-level Correctness Labeling in Open-ended Coding Problems