Limited Structural Reliability in Public Educational Prediction Benchmarks: A Four-Dimension Audit of Seven Datasets

Yan Ma; Lizhuo Zhang

cs.LGAug 28, 2026

Limited Structural Reliability in Public Educational Prediction Benchmarks: A Four-Dimension Audit of Seven Datasets

Authors: Yan Ma, Lizhuo Zhang

Organizations: School of Foreign Studies, Changsha University of Science and Technology, Changsha 410114, China · School of Education, Hunan Agricultural University, Changsha 410128, China · School of Information and Intelligence, Hunan Agricultural University, Changsha 410128, China

Abstract

Across seven public educational prediction datasets, three passed all four pre-modeling reliability checks; the remaining four either failed group-aware generalization tests or lacked the provenance metadata needed to run them. One dataset was initially classified as failing but corrected after excluding group-identifier features from the holdout matrix, demonstrating that the audit can distinguish genuine cross-group confounding from feature-encoding artifacts. Each dataset was audited before model optimization using four checks: baseline gap, split instability, null separation, and metadata adequacy under group-aware holdout. The dominant failure mode was not weak iid performance alone but cross-group fragility: in the clearest case, UCI Student declined from iid R-squared 0.242 to group-holdout R-squared -0.097, while Higher Ed collapsed from 0.041 to -8.79. Increasing model complexity did not remove this pattern: ensemble models improved structurally sound datasets but amplified instability or failed under group holdout on fragile ones. An exploratory cross-dataset comparison further showed that stronger profiles clustered in larger, richer-grouped, performance-proximal datasets, while random-split performance severely overstated deployable signal in fragile datasets. Classification-metric sensitivity analyses reached the same substantive conclusions. The results show that benchmark reliability in educational AI is constrained less by algorithm choice than by data structure, group heterogeneity, and evaluation design. A reusable pre-modeling audit offers a minimum quality gate before public educational datasets support strong benchmark or deployment claims.

Figures & tables

Figure 1: Four-dimension pre-modeling audit protocol. Seven public educational datasets are evaluated by 100 random 80/20 splits across four model families (linear, ridge, RF, GBT). Each dataset is then scored on baseline gap, split instability (

I=\mathrm{SD}/|\Delta_{\mathrm{MAE}}|

), null separation (permutation

p<0.05

across

\geq 80\%

of splits), and metadata adequacy (group-holdout retention

\geq 50\%

). The number of passed dimensions defines the readiness profile (Fragile / Partial / Mostly Passing / Strong).

Dataset	$N$	Features	Target	Type	Grouping variable	Groups
MM-TBA	186	13	Lecture quality (0–7.5)	Continuous	None available	—
Higher Ed	145	31	Course grade (0–7)	Continuous	Course ID	9
xAPI-Edu	480	72	Engagement (L/M/H $\to$ 0/1/2)	Ordinal 3-class	Topic	12
Entrance Exam	666	49	Performance (4-class $\to$ 0–3)	Ordinal 4-class	Education board	3
UCI Student	649	56	Final grade $G3$ (0–20)	Continuous	School	2
Dropout	3,630	36	Graduate/Dropout (1/0)	Binary	Course	17

Table 1: Overview of the seven audited educational datasets.

Dataset	$N$	$I_{\mathrm{lin}}$	$I_{\mathrm{gbt}}$	Beat	Perm%	iid $R^{2}$	Group $R^{2}$	Profile
OULAD	32,593	0.010	0.008	1.00	1.00	0.471	0.410	Strong (4/4)
Dropout	3,630	0.021	0.019	1.00	1.00	0.651	0.527	Strong (4/4)
xAPI-Edu	480	0.124	0.102	1.00	1.00	0.614	0.484	Strong (4/4)
Entrance Exam	666	0.117	0.115	1.00	1.00	0.438	$-0.060^{\dagger}$	Mostly Passing (3/4)
UCI Student	649	0.363	0.406	1.00	1.00	0.242	$-0.097^{\dagger}$	Mostly Passing (3/4)
Higher Ed	145	0.895	0.226	0.83	0.73	0.041	$-8.79^{\dagger}$	Partial (1/4)

Table 2: Seven-dataset audit comparison.

I_{\mathrm{lin}}

and

I_{\mathrm{gbt}}

are instability ratios for linear and Gradient Boosting models. Beat rate is the fraction of 100 splits where the linear model outperforms the mean baseline. Perm% is the fraction of tested splits with permutation

p<0.05

. Group

R^{2}

is the mean leave-one-group-out

R^{2}

for the linear model; iid

R^{2}

is the mean

R^{2}

under random splitting.

Figure 2: Audit summary heatmap: four-dimension normalised scores for the seven datasets. Cells show the score (0–1) and the corresponding PASS/FAIL judgement under the operational thresholds (beat rate

\geq 0.90

;

1-I/(1+I)\geq 0.50

, i.e.

I<1

; permutation significance

\geq 0.80

; group-holdout retention

\geq 0.50

using the linear model). The right margin lists the assigned readiness profile. Four of seven datasets fail metadata adequacy (including MM-TBA, which lacks grouping metadata); OULAD, Dropout, and xAPI-Edu pass all four dimensions.

Figure 3: Comparison of iid

R^{2}

(left panel) and group-holdout

R^{2}

(right panel) across six datasets and four models. Datasets are ordered by audit profile. The contrast between iid and group-holdout performance is largest for Higher Ed, where all models collapse under group holdout. MM-TBA is omitted because it has no grouping metadata.

Dataset	Linear	Ridge	RF	GBT
OULAD	0.410	0.410	0.482	0.533
Dropout	0.527	0.527	0.531	0.564
xAPI-Edu	0.484	0.532	0.594	0.573
Entrance Exam	$-0.060$	0.047	0.322	0.351
UCI Student	$-0.097$	$-0.094$	0.018	$-0.057$
Higher Ed	$-8.79$	$-8.58$	$-13.0$	$-10.6$

Table 3: Group-holdout

R^{2}

by model across datasets where group-aware holdout is available. Regularized models (ridge) and ensemble methods often survive group holdout where unregularized linear regression collapses, indicating that the group-level confounding is partly a function of model capacity matching on high-dimensional features.

Figure 4: Empirical permutation null vs. observed linear

R^{2}

for three representative datasets. The red density is the pooled empirical null distribution from 500 permutation draws on each of 30 random 80/20 splits (15,000 values total per dataset); the coloured density is the observed-label

R^{2}

across the same 30 splits, with vertical ticks marking individual splits and dashed/dotted lines showing the means. OULAD shows clean separation (Perm%

=100\%

), Higher Ed is borderline but below the pass threshold (Perm%

=73\%

), and MM-TBA shows substantial overlap (Perm%

=23\%

, median

p=0.117

Figure 5: Exploratory cross-dataset structural pattern and the practical cost of skipping group-aware audit. Panel A places the seven datasets by sample size and available grouping structure; marker size denotes construct proximity (1 = distal background/context variables, 2 = mixed features, 3 = performance-proximal behavioral or administrative traces), and colour denotes readiness profile. MM-TBA is displayed at 0.35 groups because no grouping metadata are available. Panel B reports the iid-to-group collapse gap,

R^{2}_{iid}-R^{2}_{group}

, for datasets with grouping metadata. Strong datasets show small collapse gaps (all

\leq 0.13

), whereas Higher Ed (group

R^{2}=-8.79

), Entrance Exam, and UCI Student show larger gaps.

Figure 6: Instability ratio (

I

) for all seven datasets and four models (log scale). The dashed red line marks

I=1

, the failure threshold where between-split variance equals the mean advantage over baseline. MM-TBA’s Gradient Boosting (

I=4.6

) and Random Forest (

I=4.0

) are extreme outliers; structurally sound datasets (OULAD, Dropout) cluster near

I=0.01

Dataset	Classifier	iid Acc (mean $\pm$ SD)	iid AUC	$I_{\mathrm{acc}}$	Group Acc (mean)	Group AUC
OULAD	Logistic	$0.837\pm 0.004$	$0.910$	$0.013$	$0.817$	$0.920$
(binary)	RF	$0.849\pm 0.004$	$0.922$	$0.012$	$0.825$	$0.926$
	GBT	$0.861\pm 0.004$	$0.938$	$0.012$	$0.842$	$0.939$
Dropout	Logistic	$0.912\pm 0.009$	$0.953$	$0.031$	$0.897$	$0.943$
(binary)	RF	$0.907\pm 0.010$	$0.952$	$0.035$	$0.885$	$0.928$
	GBT	$0.909\pm 0.010$	$0.954$	$0.033$	$0.886$	$0.938$

Table 4: Classification-metric sensitivity check on the four datasets with binary or ordinal targets. Accuracy and AUC-ROC are reported for each classifier under iid 100-split resampling and group-aware holdout.

I_{\mathrm{acc}}

is the accuracy-based instability ratio. Group-holdout results for xAPI-Edu and Entrance Exam use the corrected feature matrix with group-identifier columns excluded (see Section 2.4 ).

Audit dimension	Pass criterion	Fail indicator	Recommended action on failure
Baseline gap	$\Delta_{\mathrm{MAE}}>0$ with beat rate $\geq 0.90$ across $\geq 50$ splits	Beat rate $<0.80$ or $\Delta_{\mathrm{MAE}}$ smaller than cross-split SD	Do not escalate to complex models; investigate target quality and feature relevance
Split instability	$I<1.0$ (model variability smaller than gain)	$I\geq 1.0$	Increase sample size, reduce feature dimensionality, or improve split design
Null separation	Permutation $p<0.05$ consistent across $\geq 80\%$ of splits	Inconsistent or non-significant separation	Treat as insufficient evidence; do not publish benchmark claims
Metadata adequacy	Group holdout retains $\geq 50\%$ of iid $R^{2}$ under the linear reference model	$R^{2}$ collapse under group holdout	Add grouping metadata; run group-aware validation before deployment

Table 5: Operational decision criteria derived from the seven-dataset audit. Thresholds are empirically motivated and intended as practical starting points.

Dataset	Model	MAE (mean $\pm$ SD)	$R^{2}$ (mean)	$r$ (mean)	$I$	Beat rate
MM-TBA	Linear	$0.795\pm 0.089$	$-0.067$	$0.178$	$2.21$	$0.80$
	Ridge	$0.793\pm 0.089$	$-0.061$	$0.183$	$2.09$	$0.81$
	Random Forest	$0.815\pm 0.081$	$-0.063$	$0.196$	$3.96$	$0.65$
	Gradient Boosting	$0.856\pm 0.095$	$-0.227$	$0.129$	$4.60$	$0.45$
Higher Ed	Linear	$1.695\pm 0.199$	$0.041$	$0.469$	$0.90$	$0.83$
	Ridge	$1.686\pm 0.195$	$0.054$	$0.470$	$0.84$	$0.83$

Supplementary Table S1: Model-complexity ablation: instability ratio (

I

) and beat rate across linear and nonlinear models on 100 repeated 80/20 splits.

Dataset	$N$	Feat.	Target	Dist.	$n_{\mathrm{grp}}$	Grp sz (min/mean/max)
OULAD	32,593	44	Pass=1, Fail/W=0	0.47:0.53	22	365/1482/2498
Dropout	3,630	36	Grad=1, Dropout=0	0.61:0.39	17	9/214/666
Entrance Exam	666	49	4-class ordinal	24%:32%:30%:15%	3	21/222/396
UCI Student	649	56	Grade $G3$ [0,20]	$\mu=11.9$ , $\sigma=3.2$	2	226/324/423
xAPI-Edu	480	72	3-class ordinal	26%:44%:30%	12	19/40/95
MM-TBA	186	13	Lecture quality [0.5,7.5]	$\mu=6.0$ , $\sigma=1.0$	0	—

Supplementary Table S2: Exploratory data analysis summary for the seven audited datasets. All releases contain zero missing values. The datasets that fail the audit (MM-TBA, Entrance Exam, Higher Ed, UCI Student) each fail at least one dimension, yet none show distinctive EDA red flags—such as extreme imbalance or redundant features—that would pre-emptively disqualify them; the fragility is only revealed through the four-dimension audit.

Dataset	Model	Train $R^{2}$ mean	Test $R^{2}$ mean	Gap mean	Gap SD
UCI Student	Linear	0.347	0.242	0.105	0.091
	Ridge	0.347	0.242	0.105	0.091
	RF	0.895	0.265	0.630	0.084
	GBT	0.603	0.259	0.345	0.090
Higher Ed	Linear	0.460	$-0.092$	0.552	0.335
	Ridge	0.459	$-0.079$	0.539	0.328

Supplementary Table S3: Train-test

R^{2}

gap across 100 repeated 80/20 splits. Within each split, Z-score parameters are fitted exclusively on the training partition and applied to the test partition. The gap (train

R^{2}

– test

R^{2}

) quantifies how much of the apparent iid signal is lost on held-out samples from the same distribution. On UCI Student, the linear-model train-test gap (0.105) is small relative to the iid-to-group-holdout collapse (0.242 to

-0.097

), confirming that the group-holdout failure is not attributable to ordinary overfitting. On OULAD, both gaps remain small.

Dataset	Model	Groups tested	Mean $R^{2}$	SD $R^{2}$	Min / Max $R^{2}$
UCI Student	Linear	2	$-0.097$	0.109	$-0.206$ / 0.012
	Ridge	2	$-0.094$	0.106	$-0.201$ / 0.012
	RF	2	0.018	0.109	$-0.091$ / 0.127
	GBT	2	$-0.057$	0.136	$-0.193$ / 0.079
Higher Ed	Linear	4	$-8.79$	6.39	$-18.65$ / $-0.975$
	Ridge	4	$-8.58$	6.27	$-18.35$ / $-0.969$

Supplementary Table S4: Across-group variation in group-holdout

R^{2}

. The SD quantifies how consistent the group-holdout performance is across held-out groups. For UCI Student (2 schools), the linear-model SD of 0.109 with a narrow range (

-0.206

0.012

) confirms uniform collapse rather than group-specific failure. For OULAD (22 cohorts), the across-cohort SD of 0.162 and full range (

0.041

0.654

) reflect positive—though variable—cross-group generalization.

Dataset	Feature	Mean coef	SD coef
UCI Student	f5	$-0.862$	0.063
	f4	0.393	0.057
	f49	0.270	0.032
	f48	$-0.270$	0.032
	f11	$-0.253$	0.051
	f30	0.229	0.061

Supplementary Table S5: Top-10 linear-model standardized coefficient stability across 100 repeated 80/20 splits for the three illustrative datasets. The sign-flip rate is the proportion of splits where the coefficient sign differs from the sign of the mean coefficient. On OULAD, all top-10 features show zero sign flips with low SD, indicating stable directional relationships. On UCI Student, sign-flip rates are near zero for most top features but f9 shows a 3% rate and f2 a 4% rate, indicating occasional directional instability; SDs are proportionally larger than on OULAD. On Higher Ed, sign-flip rates are near zero for most top features (f9: 2%) with SDs reflecting greater uncertainty about effect magnitude.

Dataset	Feature	Mean importance	SD importance	CV
UCI Student	f5	0.193	0.015	0.076
	f12	0.068	0.007	0.100
	f8	0.047	0.006	0.125
	f0	0.044	0.005	0.107
	f7	0.039	0.005	0.121
	f4	0.038	0.006	0.145

Supplementary Table S6: Top-10 Random Forest feature importance stability across 100 repeated 80/20 splits. The coefficient of variation (CV = SD / mean) quantifies relative uncertainty. On OULAD, all top-10 CVs are below 0.12, indicating stable importance rankings. On UCI Student, CVs range from 0.08 to 0.20; on Higher Ed, from 0.18 to 0.41, reflecting greater split sensitivity in smaller datasets.

Dataset	Code	Feature name
UCI Student	f0	age
	f2	Fedu
	f4	studytime
	f5	failures
	f6	famrel
	f7	freetime

Supplementary Table S7: Mapping from feature codes used in Supplementary Tables S5–S6 to human-readable feature names for the three datasets included in the feature-attribution stability analysis. Feature codes correspond to zero-indexed column positions in the preprocessed feature matrix after one-hot encoding. Group-identifier columns (school for UCI Student, Course ID for Higher Ed) are excluded from the feature matrix, so their corresponding features do not appear in the tables.

Dataset	$p$	$\rho_{\mathrm{all}}$		$\rho_{\mathrm{top10}}$
		Mean	SD	Mean	SD
UCI Student	54	0.496	0.093	0.516	0.242
Higher Ed	30	0.285	0.126	0.498	0.245
OULAD	44	0.752	0.048	0.817	0.065

Supplementary Table S8: Cross-model Spearman rank correlation between linear-model absolute standardized coefficients and Random Forest impurity-based importances across 100 repeated 80/20 splits.

\rho_{\mathrm{all}}

uses all features;

\rho_{\mathrm{top10}}

uses the top-10 features by linear absolute coefficient in each split. On OULAD, cross-model agreement is high and stable across splits (

\rho_{\mathrm{top10}}=0.817

, SD 0.065). On UCI Student and Higher Ed, agreement is weaker and more variable, consistent with their lower-readiness profiles.

Supplementary Figure S1: Baseline gap: linear-model mean

R^{2}

(with cross-split SD error bars) over 100 random 80/20 splits for the seven datasets, ordered by readiness profile. The grey reference bar denotes the trivial mean baseline (

R^{2}=0

). The figure complements Table 2 by showing the dispersion of split-level

R^{2}

around the reported mean; MM-TBA and Higher Ed have means near or below zero with relatively wide spread, consistent with their Fragile / Partial classifications.

Supplementary Figure S2: Split instability: violin and box plots of the linear-model

R^{2}

across 100 random 80/20 splits for each dataset, ordered by readiness profile. The instability ratio

I

is annotated above each violin. Strong datasets (OULAD, Dropout) have tight distributions (

I<0.03

); MM-TBA’s distribution is broad and centred on negative

R^{2}

(

I=2.21

), reproducing Table 2 visually.

Supplementary Figure S3: Threshold-sensitivity sweeps for the two operational cutoffs used in the audit. Panel (a) varies the instability cutoff over

I\in[0.1,3.0]

and reports the number of datasets that would be flagged as failing split instability; the paper’s choice of

I=1.0

flags one dataset (MM-TBA, with linear

I=2.21

), while more stringent cutoffs below about

0.9

additionally flag Higher Ed. Panel (b) varies the beat-rate cutoff over

[0.5,1.0]

; the paper’s choice of

0.90

flags two datasets, whereas more lenient cutoffs below about

0.84

reduce the flagged count. The main qualitative pattern remains unchanged: MM-TBA is always the weakest case, and Higher Ed is the only borderline dataset near the operational thresholds.

Benchmark Auditing

Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.

See all 97 papers

The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection
Jun 2, 2026Wojciech Zarzecki, Jan Dubiński, Sebastian CygertBenchmark Auditing LLM Auditing
NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland · Gdańsk University of Technology, Gdańsk, Poland
AI Evaluation Should Require Standardized Item-Level Data Releases
Feb 27, 2026Han Jiang, Susu Zhang, Dongyao Zhu +6Benchmark Validity
Johns Hopkins University · University of Illinois Urbana-Champaign · North Carolina State University +3
ELBench: A Multi-Dimensional Benchmark for Education-Facing Large Language Models
Aug 10, 2026Yilin Jiang, Xiaorong Zhu, Fei Tan +9LLM Safety Benchmarks LLM Evaluation
1East China Normal University · 2The Hong Kong University of Science and Technology (Guangzhou) · 3Shanghai Jiao Tong University +1

Jun 2, 2026J/K move · Enter open · S save

The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection

Wojciech Zarzecki, Jan Dubiński, Sebastian Cygert

NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland · Gdańsk University of Technology, Gdańsk, Poland

Benchmark Auditing LLM Auditing Benchmark Contamination Benchmark Validity

Open paper AskPDF arXiv

Benchmark contamination, where evaluation examples appear in a model's training data, threatens the validity of LLM assessment. Statistical tools for detecting training-data membership exist, but have been validated almost exclusively in controlled academic regimes: large, homogeneous pre-training corpora and transparent, single-stage training pipelines. Whether these methods remain reliable in realistic auditing scenarios remains unclear. We identify two under-studied failure modes: distribution shift, which arises when suspect and validation sets violate the IID assumption, and scale constraints, which arise because benchmarks are orders of magnitude smaller than pre-training corpora. We systematically evaluate three leading paradigms, LLM Dataset Inference, Post-Hoc Dataset Inference, and CoDeC, across 25 models from multiple families (including Pythia, OLMo 2, and specialised cultural and medical LLMs) and scales (up to 27B). We then further extend our analysis to frontier industry models. Across 335 evaluations, only 201 yield correct outcomes. LLM Dataset Inference results in false positives under distribution shift, Post-Hoc Dataset Inference is underpowered at benchmark scale, and CoDeC provides only coarse provenance signals that are insufficient to verify individual benchmark splits. Our results reveal a systematic reliability gap between controlled validation and practical benchmark auditing, and show that statistical detection cannot yet replace transparent data provenance. We open-source our benchmark for further research.

Abstract

Benchmark Auditing

Explore similar work

The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection

AI Evaluation Should Require Standardized Item-Level Data Releases

ELBench: A Multi-Dimensional Benchmark for Education-Facing Large Language Models