Learning analytics often treats unsupervised clusters of intelligent tutoring system (ITS) logs as learner types that should predict learning. We test that assumption on EdNet-KT3. Clustering study-strategy features (resource use, revision, video, problem practice) for 5{,}000 active learners yields a silhouette-selected parent cut (k=5) with 4 contrast poles (reading-focused, video-heavy, revision-heavy, and problem-first) plus a large near-mean residual (∼64.9%). Reclustering that residual adds four finer styles, giving a bootstrap-stable hierarchy of 8 named strategies. We split each learner's timeline by respond count so clusters use only the early half and outcomes only the late half. Early clusters predict later engagement (continuing to practice and finishing late sessions, especially persistence, η2≈0.106; completion η2≈0.021) but not later unassisted accuracy (correctness on late first-attempts without help; padj≈0.093). Volume rises with some styles, yet volume-only clustering barely matches strategy labels (ARI=0.064). A knowledge-tracing model (SAKT) on the seven TOEIC exam sections predicts next correctness only modestly better than a baseline that knows only how hard each section usually is (AUC lift +0.051; CI [+0.045,+0.058]), and that mastery signal is nearly independent of behavior styles (ARI=0.007). Behavioral clustering here describes study styles and engagement, not knowledge gains.
Figures & tables
Category
Feature
Definition
Action-mix frequencies
frac_n_watch_video
Frequency of non-idle actions that are video
frac_n_read_explanation
Frequency that are explanation reads
frac_n_read_passage
Frequency that are passage / stem reads
frac_n_attempt_problem
Frequency that are first attempts
frac_n_revise_answer
Frequency that are revises
Video dwell share
dwell_frac_watch_video
Timed study seconds on video / total timed study seconds. Passage dwell is constant zero on KT3 (excluded); explanation dwell is near-complementary and dropped to avoid collinearity.
Table 1: Locked clustering features (9), grouped by flat study-strategy category. Action-mix values are shares of non-idle actions. A feature is non-finite (and the row is dropped before PCA) when its denominator is zero—almost always dwell_frac_watch_video with no timed study ( 0/0 ); rarely also study_before_answer_ratio , revision_rate , or reengage_after_error_rate when the corresponding count is zero.
Inclusion rule
Users Count ( n )
Role in analysis
Active eligible ( ≥ 50 responds)
45,621
Sampling frame before seeding
Seeded manifest
5,000
Locked analysis cohort
Finite clustering features
4,986
Parent / hierarchy labeling
In typical-majority parent
3,236
Second-stage majority styles
Finite terminal accuracy
2,952
Mastery proxy available
Labeled ∩ scorable
2,939
Mastery and volume–mastery tables
Table 2: Recurring user counts on the locked analysis path.
Figure 1: Internal-validity sweeps for k∈[2,12] . Left: parent-stage K-means; silhouette and Davies–Bouldin selects k=5 (dashed). Right: typical-majority recluster; combined silhouette and Davies–Bouldin rank selects k=4 .
Table 3: Adjusted Rand index (ARI) summary. Bootstrap rows: mean agreement of resampled labels vs. the full-cohort reference (floor = minimum replicate). Comparison rows: agreement between two labelings of the same learners.
Variant
k
Sil.
DB
Structure (role)
locked_baseline
5
0.333
1.106
Locked 9-feature cut (reference)
v1_merged_read
2
0.561
1.053
Coarser residual + video pole
v2_long_sessions
2
0.571
1.104
Coarser; session-length sensitivity
v3_mix_dwell_strategy
2
0.575
1.091
Highest silhouette, still coarse
v4_log_counts
2
0.525
0.687
Better DB; HDBSCAN usable; names change
Table 4: Feature-definition variants at their silhouette-selected k . Silhouette↑ / Davies–Bouldin↓ ( k=5 ).
Figure 3: Descriptive no-quit order cut inside the typical majority (exploratory; not part of the locked hierarchy).
Outcome
Sample # n
η2
padj
Sig.
persistence_late_sessions
4960
0.106
<10−100
yes
late_completion_rate
4960
0.021
<10−20
yes
terminal_unassisted_accuracy
2925
0.003
0.093
no
Table 5: Holm-adjusted one-way ANOVA of late outcomes under strict_early (locked k=5 ).
Figure 4: Late-period outcomes by early-half cluster membership (anti-leakage design). Engagement proxies separate more clearly than terminal unassisted accuracy.
Cluster
Mean responds
Mean actions
Mean sessions
Typical majority
≈ 270
≈ 1,225
≈ 16.7
Reading-focused
≈ 247
≈ 1,211
≈ 19.5
Video-heavy
≈ 314
≈ 1,382
≈ 21.6
Revision-heavy
≈ 506
≈ 1,680
≈ 17.6
Problem-first
≈ 617
≈ 1,835
≈ 19.0
ANOVA η2
0.095
0.025
0.004
Table 6: Mean volume covariates by full-timeline parent cluster (4,986 users). ANOVA η2 uncorrected across the three tests.
Labeling
ARI vs. strategy
Terminal η2
Omnibus range (pp)
Strategy
—
0.003
10.48
Volume-only
0.064
0.001
6.27
Table 7: Activity-only baseline vs. strategy labels ( k=5 ; n= 2,939 scorable labeled users for mastery columns). ARI also appears in Table 3 .
Quantity
Value
Learners with KT sequences
4,994
TOEIC exam sections ( part )
7 ( 1 – 7 )
SAKT validation AUC
0.605
Part-difficulty prior AUC (cohort success rate on section)
0.553
AUC lift over prior (95% CI)
+0.051 ( [+0.045,+0.058] )
KT-cluster vs. behavior ARI
0.007
Table 8: Supervised KT probe summary. Items are labeled by TOEIC part (sections 1 – 7 ). The part-difficulty prior is a naive baseline: for section p , use the cohort’s success rate on p for every learner (no personal history). Lift is AUC(SAKT)−AUC(prior) .
Educational Data Mining studies computational methods for analyzing learning data to understand learners, improve instruction, and support educational decisions.
Sustained effort is essential for realizing the benefits of intelligent tutoring systems (ITS), yet many learners disengage or underuse available practice time. We introduce engagement forecasting as a supervised prediction task based on ITS logs, targeting two outcomes central to effort and learning progress: minutes practiced per week and new skills mastered per week. Using interaction log data from 425 middle-school students over a school year, we benchmark fifteen predictors including regressions, decision trees, and neural networks. We show that these feature-based models reduce mean absolute error (MAE) by 22-33% relative to heuristic baselines, including fixed-percentile rules adapted from prior work in other behavioral domains. We find that percentile heuristics systematically overpredict, whereas feature-based models better track student practice trajectories across weeks. To support explainability, we analyze feature importance and ablations, revealing target-specific patterns: effort forecasting is driven mainly by recent activity features, while progress forecasting depends more on learner-state and content difficulty signals. Finally, in a semi-structured user interview case study with eight college tutors, we examine how tutors reasoned about system-generated predictive features when setting goals with students. We find that tutors reasoned differently about effort versus progress goals in ways that mirror our pattern analysis. Together, these results establish a reproducible benchmark for forecasting weekly effort and learning progress in ITS. By making patterns of sustained effort and progress visible at a weekly timescale, engagement forecasting offers a foundation for supporting tutor-learner goal setting and timely instructional decisions.
Eric S. Qiu, Danielle R. Thomas, Boyuan Guo +2
Cornell University · Ithaca, NY, USA · Carnegie Mellon University +1
Standardized examinations are typically treated as uniform syllabus coverage problems. We argue they are better understood as adversarial systems with stable latent cognitive structures diverging systematically from official syllabi. We introduce LearnOpt, which recovers this structure from historical question papers and generates personalized, time-bounded study plans. Applied to nine years of NEET questions (2016-2024, n=1,496), LearnOpt builds an exam knowledge graph from LLM-tagged questions, extracts a five-category latent skill distribution, and formulates study planning as a knapsack-variant optimization over prerequisite-aware subgraphs with Bayesian Knowledge Tracing. Central finding: NEET's latent skill distribution is stable within a syllabus regime (consecutive-year KL divergence 0.004-0.032 for 2016-2021, non-significant under permutation testing) but shifts significantly with NCERT's 2023 syllabus rationalization: pooling 2016-2021 (n=1,072) vs 2023-2024 (n=392) gives KL=0.040 (p=0.0005), with Elimination/Negation questions rising from ~20-29% to ~31-35%. Latent structure, while not permanently stationary, is piecewise stable, with shifts detectable and attributable to curricular events. Within either regime, subject predicts skill profile more strongly than year. An optimization evaluation, using one real and two synthetic mastery profiles, shows the skill-weighted objective produces a modest but real reordering of recommended topics over a mastery-conditioned frequency baseline. Applying the pipeline to JEE Advanced reveals a profile dominated by Multi-concept Integration (80.9% vs. 33.3% for NEET), with a JEE-vs-NEET divergence (KL=0.505) exceeding NEET's largest cross-subject divergence: exam tier shapes latent cognitive structure more than subject, which shapes it more than time within a regime. Code, knowledge graph, and annotated dataset are released publicly.
Learning management system traces are clustered to identify participation profiles, but unequal observation windows may cause profiles to reflect duration rather than participation alone. Using 16,448 complete learner--course sequences from 4,599 learners across 152 courses at a South African public university, we evaluated eight participation measures. All remained related to observation-window length even when expressed as rates or shares. Course-grouped adjustment reduced these relationships. Five retained measures were evaluated using K-means, Gaussian mixtures, Ward agglomerative clustering and K-medoids, with standardised measures and principal-component summaries. Evidence supported a broad two-profile grouping and a more detailed adjusted four-profile consensus: Gradually declining, Early-concentrated, Cyclical variable and Sustained active. Profile membership accounted for 2.87% of the variation in duration in the primary consensus, versus 38.18% using original measures. Mean assignment support was 0.739; 8.35% of assignments fell below 0.50, with greatest uncertainty in Sustained active. Two week-by-week sequence analyses agreed strongly with each other but only modestly with the four-profile consensus, indicating that the profiles combined multiple aspects of participation rather than distinct sequence patterns. Profiling should therefore assess duration effects, compare clustering methods and participation summaries, and report assignment consistency.
David Otoo-Arthur, Terence L. van Zyl
University of the Witwatersrand, Johannesburg, South Africa · University of Johannesburg, Johannesburg, South Africa