Automatically Inferring Teachers' Geometric Content Knowledge: A Skills Based Approach

Ziv Fenigstein; Kobi Gal; Avi Segal; Osama Swidan; Inbal Israel; Hassan Ayoob

cs.CYApr 15, 2026

Automatically Inferring Teachers' Geometric Content Knowledge: A Skills Based Approach

Authors: Ziv Fenigstein, Kobi Gal, Avi Segal, Osama Swidan, Inbal Israel, Hassan Ayoob

Organizations: Ben-Gurion University, Israel · University of Edinburgh, U.K.

Abstract

Assessing teachers' geometric content knowledge is essential for geometry instructional quality and student learning, but difficult to scale. The Van Hiele model characterizes geometric reasoning through five hierarchical levels. Traditional Van Hiele assessment relies on manual expert analysis of open-ended responses. This process is time-consuming, costly, and prevents large-scale evaluation. This study develops an automated approach for diagnosing teachers' Van Hiele reasoning levels using large language models grounded in educational theory. Our central hypothesis is that integrating explicit skills information significantly improves Van Hiele classification. In collaboration with mathematics education researchers, we built a structured skills dictionary decomposing the Van Hiele levels into 33 fine-grained reasoning skills. Through a custom web platform, 31 pre-service teachers solved geometry problems, yielding 226 responses. Expert researchers then annotated each response with its Van Hiele level and demonstrated skills from the dictionary. Using this annotated dataset, we implemented two classification approaches: (1) retrieval-augmented generation (RAG) and (2) multi-task learning (MTL). Each approach compared a skills-aware variant incorporating the skills dictionary against a baseline without skills information. Results showed that for both methods, skills-aware variants significantly outperformed baselines across multiple evaluation metrics. This work provides the first automated approach for Van Hiele level classification from open-ended responses. It offers a scalable, theory-grounded method for assessing teachers' geometric reasoning that can enable large-scale evaluation and support adaptive, personalized teacher learning systems.

Figures & tables

Figure 1: Overview of the proposed skills-based Van Hiele classification framework.

Figure 2: Retrieval-augmented generation architecture for Method I.

Figure 3: Multi-task learning architecture for Method II.

Figure 4: Average results across 5-fold cross-validation comparing baseline and skills-aware variants for RAG (Method I) and MTL (Method II). Numbers above bars show mean scores; lines within the bars indicate standard deviation. For MAE measure, lower is better.

Variant	F1-macro	F1-weighted	QWK	MAE
Baseline (No Skills)	$0.624\pm 0.093$	$0.678\pm 0.095$	$0.625\pm 0.035$	$0.47\pm 0.09$
Skills-Aware	$\mathbf{0.695\pm 0.09}$	$\mathbf{0.736\pm 0.08}$	$\mathbf{0.721\pm 0.054}$	$\mathbf{0.376\pm 0.06}$
Noisy Skills	$0.61\pm 0.122$	$0.635\pm 0.117$	$0.673\pm 0.093$	$0.495\pm 0.156$

Table 1: Skills sensitivity analysis for Method I. Values are reported as mean

\pm

standard deviation across five cross-validation folds.

Variant	F1-macro	F1-weighted	QWK	MAE
Baseline	$0.646\pm 0.082$	$0.657\pm 0.092$	$0.586\pm 0.157$	$0.523\pm 0.119$
Attention-Guided	$0.676\pm 0.058$	$0.679\pm 0.066$	$0.632\pm 0.129$	$0.487\pm 0.086$
Skills-Supervised	$0.652\pm 0.063$	$0.653\pm 0.088$	$0.581\pm 0.089$	$0.558\pm 0.119$
Full Model (Method II)	$\mathbf{0.725\pm 0.043}$	$\mathbf{0.725\pm 0.053}$	$\mathbf{0.717\pm 0.103}$	$\mathbf{0.403\pm 0.102}$

Table 2: Skills component analysis for Method II. Values are reported as mean

\pm

standard deviation across five cross-validation folds.

Educational Assessment

Educational assessment measures what learners know and can do, using tests, rubrics, and learning evidence to guide decisions.

See all 151 papers

Apr 3, 2026

Toward an Artificial General Teacher: Procedural Geometry Data Generation and Visual Grounding with Vision-Language Models

We study visual explanation in geometry education as a Referring Image Segmentation (RIS) problem: given a diagram and a natural language description, the task is to produce a pixel-level mask for the referred geometric element. However, existing RIS models trained on natural image benchmarks such as RefCOCO fail catastrophically on geometric diagrams due to the fundamental domain shift between photographic scenes and abstract, textureless schematics. To address the absence of suitable training data, we present a fully automated procedural data engine that generates over 200,000 synthetic geometry diagrams with pixel-perfect segmentation masks and linguistically diverse referring expressions, requiring zero manual annotation. We further propose domain-specific fine-tuning of vision-language models (VLMs), demonstrating that a fine-tuned Florence-2 achieves 49% IoU and 85% Buffered IoU (BIoU), compared to <1% IoU in zero-shot settings. We introduce Buffered IoU, a geometry-aware evaluation metric that accounts for thin-structure localization, and show that it better reflects true segmentation quality than standard IoU. Our results establish a foundation for building Artificial General Teachers (AGTs) capable of providing visually grounded, step-by-step explanations of geometry problems.

Hai Nguyen-Truong, Alper Balbay, Tunga Bayrak

Freya

Referring Image Segmentation Visual Reasoning Educational Technology VLM Adaptation Synthetic Data Generation

Ask PDF

Feb 25, 2026

GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams

Evaluating the symbolic reasoning of large language models (LLMs) calls for geometry benchmarks that require multi-step proofs grounded in both text and diagrams. However, existing benchmarks are often limited in scale and rarely provide visually grounded multiple-choice questions, limiting reliable evaluation of complex reasoning. We introduce GeoChallenge, a dataset of 90K automatically generated multiple-choice geometry proof problems, each requiring multi-step reasoning over aligned textual descriptions and diagrams. GeoChallenge provides fine-grained complexity ratings and formal language annotations to enable controlled evaluation. Experiments on multiple advanced LLMs show a clear performance gap between models and humans (the best-performing model, GPT-5-nano, achieves 75.89 exact match vs. 94.74 for humans). Further analysis also reveals three common failure patterns of LLMs: (1) exact match failures under the multiple-choice setting; (2) weak visual reliance; and (3) overextended reasoning without convergence.

Yushun Zhang, Weiping Fu, Zesheng Yang +6

School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, Xi’an, China

Mathematical Reasoning Benchmarks LLM Mathematical Reasoning Visually Grounded Reasoning

Ask PDF

Apr 3, 2026

Measuring Representation Robustness in Large Language Models for Geometry

Large language models (LLMs) are increasingly evaluated on mathematical reasoning, yet their robustness to equivalent problem representations remains poorly understood. In geometry, identical problems can be expressed in Euclidean, coordinate, or vector forms, but existing benchmarks report accuracy on fixed formats, implicitly assuming representation invariance and masking failures caused by representational changes alone. We propose GeoRepEval, a representation-aware evaluation framework that measures correctness, invariance, and consistency at the problem level across parallel formulations, combining strict answer matching, bootstrap confidence intervals, paired McNemar tests, representation-flip analyses, and regression controls for surface complexity. We prove that our Invariance@3 metric decomposes accuracy into robust and fragile components and is bounded by the weakest representation. Evaluating eleven LLMs on 158 curated high-school geometry problems (474 instances), we find accuracy gaps of up to 14 percentage points induced solely by representation choice. Vector formulations emerge as a consistent failure point, with Invariance@3 as low as 0.044 even after controlling for length and symbolic complexity. A convert-then-solve prompting intervention improves vector accuracy by up to 52 percentage points for high-capacity models, suggesting that failures reflect representation sensitivity rather than inability; however, low-capacity models show no gains, indicating deeper limitations. These results suggest that current models rely on representation-specific heuristics rather than abstract geometric reasoning. All datasets, prompts, and scripts are released at https://github.com/vedjaw/GeoRepEval.

Vedant Jawandhia, Yash Sinha, Murari Mandal +2

Department of Computer Science and Information Systems, BITS Pilani · School of Computer Science, KIIT University · Department of Mathematics, BITS Pilani

LLM Evaluation Representation Geometry in Language Models Invariant Representation Learning

Ask PDF

Abstract

Educational Assessment

Explore similar work

Toward an Artificial General Teacher: Procedural Geometry Data Generation and Visual Grounding with Vision-Language Models

GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams

Measuring Representation Robustness in Large Language Models for Geometry