EduEVAL-DB: A Role-Based Dataset for Pedagogical Risk Evaluation in Educational Explanations
Authors: Javier Irigoyen, Roberto Daza, Aythami Morales, Julian Fierrez, Francisco Jurado, Alvaro Ortigosa, Ruben Tolosana
Organizations: BiometricsAI, Universidad Autónoma de Madrid, Campus de Cantoblanco, Madrid, 28049, Spain · Group for Advanced Interactive Tools (GHIA), Universidad Autónoma de Madrid, Campus de Cantoblanco, Madrid, 28049, Spain · Department of Mathematics, Universidad de Las Palmas de Gran Canaria, 35017, Spain
This work introduces EduEVAL-DB, a dataset based on teacher roles designed to support the evaluation and training of automatic pedagogical evaluators and AI tutors for instructional explanations. The dataset comprises 854 explanations corresponding to 139 questions from a curated subset of the ScienceQA benchmark, spanning science, language, and social science across K-12 grade levels. For each question, one human-teacher explanation is provided and six are generated by LLM-simulated teacher roles. These roles are inspired by instructional styles and shortcomings observed in real educational practice and are instantiated via prompt engineering. We further propose a pedagogical risk rubric aligned with established educational standards, operationalizing five complementary risk dimensions: factual correctness, explanatory depth and completeness, focus and relevance, student-level appropriateness, and ideological bias. All explanations are annotated with binary risk labels through a semi-automatic process with expert teacher review. Finally, we present preliminary validation experiments to assess the suitability of EduEVAL-DB for evaluation. We benchmark a state-of-the-art education-oriented model (Gemini 2.5 Pro) against a lightweight local Llama 3.1 8B model and examine whether supervised fine-tuning on EduEVAL-DB supports pedagogical risk detection using models deployable on consumer hardware.
Figures & tables
Figure 1: Overview of the framework used in this work for the construction of EduEVAL-DB and pedagogical risk evaluation. The diagram illustrates the main stages of the proposed pipeline: i) the definition of a pedagogical risk rubric; ii) the construction and annotation of EduEVAL-DB, where each K–12 question is paired with multiple explanations generated by LLM-simulated teacher roles and a human teacher, and annotated with binary risk labels; and iii) the use of EduEVAL-DB to benchmark and fine-tune LLM-based pedagogical evaluators.
Figure 2: Example explanations generated by the six teacher-inspired roles. Each role generates an instructional explanation for the same question, conditioned on the student’s grade level and the multiple-choice context. To ensure responsible presentation, the Sarcastic Teacher output is blurred in the figure, although the complete explanation is included in the released dataset.
Criteria
Gemini (Baseline)
Llama (Baseline)
Llama (Fine-tuned)
Factual Accuracy
0.012
0.164
0.048
Focus & Relevance
0.216
0.227
0.069
Depth & Completeness
0.320
0.288
0.048
Student-Level Appropriateness
0.054
0.220
0.003
Ideological Bias
0.008
0.049
0.006
Table 1: Mean Absolute Error (MAE), normalized to the [0,1] range, reported for each pedagogical risk dimension defined in the proposed rubric, for the baseline evaluators (Gemini 2.5 Pro and Llama 3.1 8B) and the fine-tuned Llama 3.1 8B evaluator on EduEVAL-DB. Lower values indicate better agreement with the ground-truth annotations. Best results per dimension are highlighted in bold.
Figure 3: Confusion matrices for the Llama 3.1 8B pedagogical evaluator on EduEVAL-DB in zero-shot baseline (top) and fine-tuned (bottom) settings across the five pedagogical risk dimensions defined in the proposed rubric. Each matrix reports the distribution of predicted versus ground-truth binary labels, where 0 denotes the absence of pedagogical risk and 1 denotes its presence (detection). Cell color intensity reflects the proportion of samples in each category, with warmer colors indicating higher frequencies.
BiometricsAI, Universidad Aut´onoma de Madrid (UAM), Spain · GHIA, Universidad Aut´onoma de Madrid (UAM), Spain · Universidad de Las Palmas de Gran Canaria (ULPGC), Spain
BiometricsAI, Universidad Autónoma de Madrid (UAM), Spain · GHIA, Universidad Autónoma de Madrid (UAM), Spain · Universidad de Las Palmas de Gran Canaria (ULPGC), Spain
Department of Computer Science and Engineering, University at Buffalo, USA · Department of Teaching and Learning, Washington State University, USA · College of Chemistry, Beijing Normal University, China +1