Study-Strategy Clusters from EdNet Logs Track Engagement, Not Mastery

Qingchuan Lyu; Yingxin Li; Albert Yang

cs.LGAug 17, 2026

Study-Strategy Clusters from EdNet Logs Track Engagement, Not Mastery

Authors: Qingchuan Lyu, Yingxin Li, Albert Yang

Organizations: Georgia Institute of Technology, Atlanta, GA, USA · New York University, New York, NY, USA

Abstract

Learning analytics often treats unsupervised clusters of intelligent tutoring system (ITS) logs as learner types that should predict learning. We test that assumption on EdNet-KT3. Clustering study-strategy features (resource use, revision, video, problem practice) for 5{,}000 active learners yields a silhouette-selected parent cut ( $k=5$ ) with 4 contrast poles (reading-focused, video-heavy, revision-heavy, and problem-first) plus a large near-mean residual ( $\sim$ 64.9%). Reclustering that residual adds four finer styles, giving a bootstrap-stable hierarchy of 8 named strategies. We split each learner's timeline by respond count so clusters use only the early half and outcomes only the late half. Early clusters predict later engagement (continuing to practice and finishing late sessions, especially persistence, $η^{2}\approx 0.106$ ; completion $η^{2}\approx 0.021$ ) but not later unassisted accuracy (correctness on late first-attempts without help; $p_{\mathrm{adj}}\approx 0.093$ ). Volume rises with some styles, yet volume-only clustering barely matches strategy labels (ARI $=0.064$ ). A knowledge-tracing model (SAKT) on the seven TOEIC exam sections predicts next correctness only modestly better than a baseline that knows only how hard each section usually is (AUC lift $+0.051$ ; CI $[+0.045,+0.058]$ ), and that mastery signal is nearly independent of behavior styles (ARI $=0.007$ ). Behavioral clustering here describes study styles and engagement, not knowledge gains.

Figures & tables

Category	Feature	Definition
Action-mix frequencies	frac_n_watch_video	Frequency of non-idle actions that are video
	frac_n_read_explanation	Frequency that are explanation reads
	frac_n_read_passage	Frequency that are passage / stem reads
	frac_n_attempt_problem	Frequency that are first attempts
	frac_n_revise_answer	Frequency that are revises
Video dwell share	dwell_frac_watch_video	Timed study seconds on video / total timed study seconds. Passage dwell is constant zero on KT3 (excluded); explanation dwell is near-complementary and dropped to avoid collinearity.

Table 1: Locked clustering features (9), grouped by flat study-strategy category. Action-mix values are shares of non-idle actions. A feature is non-finite (and the row is dropped before PCA) when its denominator is zero—almost always dwell_frac_watch_video with no timed study (

0/0

); rarely also study_before_answer_ratio , revision_rate , or reengage_after_error_rate when the corresponding count is zero.

Inclusion rule	Users Count ( $n$ )	Role in analysis
Active eligible ( $\geq$ 50 responds)	45,621	Sampling frame before seeding
Seeded manifest	5,000	Locked analysis cohort
Finite clustering features	4,986	Parent / hierarchy labeling
In typical-majority parent	3,236	Second-stage majority styles
Finite terminal accuracy	2,952	Mastery proxy available
Labeled $\cap$ scorable	2,939	Mastery and volume–mastery tables

Table 2: Recurring user counts on the locked analysis path.

Panel of Figure 1 — Figure 1: Internal-validity sweeps for $k{\in}[2,12]$ . Left: parent-stage K-means; silhouette and Davies–Bouldin selects $k{=}5$ (dashed). Right: typical-majority recluster; combined silhouette and Davies–Bouldin rank selects $k{=}4$ .

Panel of Figure 2 — Figure 2: Parent styles and hierarchical typology. (a) left: UMAP/PCA views; (b) right: feature contrasts.

Category	Description/comparison	What it asks	ARI
Bootstrap stability (full-timeline typology)
Bootstrap	Parent $k{=}5$	Do parent labels reproduce?	0.984 (floor 0.965)
Bootstrap	Full hierarchy (8 styles)	Do hierarchical labels reproduce?	0.938
Bootstrap	Contrast poles only	Do the 4 poles reproduce?	0.986
Bootstrap	Majority styles only	Do the 4 majority styles reproduce?	0.915 (floor 0.886)
Bootstrap	Quit-in order cut (exploratory)	Stable with idle-return transitions?	0.986

Table 3: Adjusted Rand index (ARI) summary. Bootstrap rows: mean agreement of resampled labels vs. the full-cohort reference (floor

=

minimum replicate). Comparison rows: agreement between two labelings of the same learners.

Variant	$k$	Sil.	DB	Structure (role)
locked_baseline	5	0.333	1.106	Locked 9-feature cut (reference)
v1_merged_read	2	0.561	1.053	Coarser residual + video pole
v2_long_sessions	2	0.571	1.104	Coarser; session-length sensitivity
v3_mix_dwell_strategy	2	0.575	1.091	Highest silhouette, still coarse
v4_log_counts	2	0.525	0.687	Better DB; HDBSCAN usable; names change

Table 4: Feature-definition variants at their silhouette-selected

k

. Silhouette↑ / Davies–Bouldin↓ (

k{=}5

Figure 3: Descriptive no-quit order cut inside the typical majority (exploratory; not part of the locked hierarchy).

Outcome	Sample # $n$	$\eta^{2}$	$p_{\mathrm{adj}}$	Sig.
persistence_late_sessions	4960	0.106	$<10^{-100}$	yes
late_completion_rate	4960	0.021	$<10^{-20}$	yes
terminal_unassisted_accuracy	2925	0.003	0.093	no

Table 5: Holm-adjusted one-way ANOVA of late outcomes under strict_early (locked

k{=}5

Figure 4: Late-period outcomes by early-half cluster membership (anti-leakage design). Engagement proxies separate more clearly than terminal unassisted accuracy.

Cluster	Mean responds	Mean actions	Mean sessions
Typical majority	$\approx$ 270	$\approx$ 1,225	$\approx$ 16.7
Reading-focused	$\approx$ 247	$\approx$ 1,211	$\approx$ 19.5
Video-heavy	$\approx$ 314	$\approx$ 1,382	$\approx$ 21.6
Revision-heavy	$\approx$ 506	$\approx$ 1,680	$\approx$ 17.6
Problem-first	$\approx$ 617	$\approx$ 1,835	$\approx$ 19.0
ANOVA $\eta^{2}$	0.095	0.025	0.004

Table 6: Mean volume covariates by full-timeline parent cluster (4,986 users). ANOVA

\eta^{2}

uncorrected across the three tests.

Labeling	ARI vs. strategy	Terminal $\eta^{2}$	Omnibus range (pp)
Strategy	—	0.003	10.48
Volume-only	0.064	0.001	6.27

Table 7: Activity-only baseline vs. strategy labels (

k{=}5

;

n{=}

2,939 scorable labeled users for mastery columns). ARI also appears in Table 3 .

Quantity	Value
Learners with KT sequences	4,994
TOEIC exam sections ( part )	7 ( $1$ – $7$ )
SAKT validation AUC	0.605
Part-difficulty prior AUC (cohort success rate on section)	0.553
AUC lift over prior (95% CI)	$+0.051$ ( $[+0.045,+0.058]$ )
KT-cluster vs. behavior ARI	0.007

Table 8: Supervised KT probe summary. Items are labeled by TOEIC part (sections

1

–

7

). The part-difficulty prior is a naive baseline: for section

p

, use the cohort’s success rate on

p

for every learner (no personal history). Lift is

\mathrm{AUC}(\mathrm{SAKT})-\mathrm{AUC}(\mathrm{prior})

Figure 5: Nearest-centroid exemplar action timelines for parent styles (first 200 actions).

Educational Data Mining

Educational Data Mining studies computational methods for analyzing learning data to understand learners, improve instruction, and support educational decisions.

See all 72 papers

From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning
May 12, 2026Eric S. Qiu, Danielle R. Thomas, Boyuan Guo +2Intelligent Tutoring Systems Time Series Forecasting
Cornell University · Ithaca, NY, USA · Carnegie Mellon University +1
LearnOpt: Recovering the Latent Cognitive Structure of Standardized Examinations via Knowledge Graphs and Constrained Optimization
Jun 13, 2026Joy Bose, Om ThomasCognitive Modeling Educational Assessment
Independent Researchers Bengaluru, India
Longitudinal Learner Participation Profiling in Online Learning: Heterogeneous Consensus Clustering under Unequal Observation Windows
Oct 12, 2026David Otoo-Arthur, Terence L. van ZylEducational Data Mining Clustering
University of the Witwatersrand, Johannesburg, South Africa · University of Johannesburg, Johannesburg, South Africa

May 12, 2026J/K move · Enter open · S save

From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning

Eric S. Qiu, Danielle R. Thomas, Boyuan Guo, Vincent Aleven+1

Cornell University · Ithaca, NY, USA · Carnegie Mellon University+1

Intelligent Tutoring Systems Time Series Forecasting Forecasting Benchmarks Educational Data Mining Human Behavior Prediction Adaptive Learning

Open paper AskPDF arXiv

Sustained effort is essential for realizing the benefits of intelligent tutoring systems (ITS), yet many learners disengage or underuse available practice time. We introduce engagement forecasting as a supervised prediction task based on ITS logs, targeting two outcomes central to effort and learning progress: minutes practiced per week and new skills mastered per week. Using interaction log data from 425 middle-school students over a school year, we benchmark fifteen predictors including regressions, decision trees, and neural networks. We show that these feature-based models reduce mean absolute error (MAE) by 22-33% relative to heuristic baselines, including fixed-percentile rules adapted from prior work in other behavioral domains. We find that percentile heuristics systematically overpredict, whereas feature-based models better track student practice trajectories across weeks. To support explainability, we analyze feature importance and ablations, revealing target-specific patterns: effort forecasting is driven mainly by recent activity features, while progress forecasting depends more on learner-state and content difficulty signals. Finally, in a semi-structured user interview case study with eight college tutors, we examine how tutors reasoned about system-generated predictive features when setting goals with students. We find that tutors reasoned differently about effort versus progress goals in ways that mirror our pattern analysis. Together, these results establish a reproducible benchmark for forecasting weekly effort and learning progress in ITS. By making patterns of sustained effort and progress visible at a weekly timescale, engagement forecasting offers a foundation for supporting tutor-learner goal setting and timely instructional decisions.

Abstract

Educational Data Mining

Explore similar work

From Heuristics to Analytics: Forecasting Effort and Progress in Online Learning

LearnOpt: Recovering the Latent Cognitive Structure of Standardized Examinations via Knowledge Graphs and Constrained Optimization

Longitudinal Learner Participation Profiling in Online Learning: Heterogeneous Consensus Clustering under Unequal Observation Windows