Toward an Artificial General Teacher: Procedural Geometry Data Generation and Visual Grounding with Vision-Language Models

Hai Nguyen-Truong; Alper Balbay; Tunga Bayrak

cs.CVApr 3, 2026

Toward an Artificial General Teacher: Procedural Geometry Data Generation and Visual Grounding with Vision-Language Models

Authors: Hai Nguyen-Truong, Alper Balbay, Tunga Bayrak

Organizations: Freya

Abstract

We study visual explanation in geometry education as a Referring Image Segmentation (RIS) problem: given a diagram and a natural language description, the task is to produce a pixel-level mask for the referred geometric element. However, existing RIS models trained on natural image benchmarks such as RefCOCO fail catastrophically on geometric diagrams due to the fundamental domain shift between photographic scenes and abstract, textureless schematics. To address the absence of suitable training data, we present a fully automated procedural data engine that generates over 200,000 synthetic geometry diagrams with pixel-perfect segmentation masks and linguistically diverse referring expressions, requiring zero manual annotation. We further propose domain-specific fine-tuning of vision-language models (VLMs), demonstrating that a fine-tuned Florence-2 achieves 49% IoU and 85% Buffered IoU (BIoU), compared to <1% IoU in zero-shot settings. We introduce Buffered IoU, a geometry-aware evaluation metric that accounts for thin-structure localization, and show that it better reflects true segmentation quality than standard IoU. Our results establish a foundation for building Artificial General Teachers (AGTs) capable of providing visually grounded, step-by-step explanations of geometry problems.

Figures & tables

Figure 1: Overview. We formalize the “pointing” mechanism of human geometry teaching as a Referring Image Segmentation (RIS) task. Given a diagram and a textual description (e.g., “triangle ABC”), the model produces a pixel-level mask highlighting the referred element.

Figure 2: Data generation pipeline. Stage 1: Constraint-based coordinate generation via analytical solvers. Stage 2: Vector rendering through L a T e X /TikZ templates. Stage 3: Render-based ground-truth mask extraction using channel arithmetic.

Shape	Defining Constraint
Parallelogram	$\vec{AB}\parallel\vec{DC}$ , $\vec{AD}\parallel\vec{BC}$
Rectangle	Parallelogram $\wedge$ all angles $=90^{\circ}$
Trapezoid	Exactly one pair of parallel sides
Isosceles Trapezoid	Trapezoid $\wedge$ equal leg lengths
Rhombus	Parallelogram $\wedge$ all sides equal
Square	Rectangle $\wedge$ all sides equal

Table 1: Supported shape types and their defining constraints. Each type is sampled with equal probability (12.5%).

Figure 3: Hierarchical language template system. Referring expressions are generated at multiple levels of linguistic complexity to encourage robust visual grounding.

Aspect	Natural Images	Geometry Diagrams
Visual features	Color, texture, shape	Black lines on white
Semantics	Object categories	Topological relations
Scene structure	Dense, cluttered	Sparse, schematic
Target elements	Large objects	Thin structures (1–3 px)

Table 2: Domain comparison between natural images and geometry diagrams.

Model	Type	Text Enc.	Focus	Geometry Hypothesis
LAVT	Vision	BERT	Visual attr.	Insufficient for text-heavy geometry
Florence-2	Text-aware	Transf.	Grounding	Better text understanding
Qwen-VL	LLM	LLM	Reasoning	Deep geometric reasoning

Table 3: Architecture comparison. We select text-centric models that prioritize language understanding over visual appearance matching.

Figure 4: VLM fine-tuning architecture with polygon-based RIS. The vision-language model employs LoRA adapters (shown in red) for efficient fine-tuning while keeping the vision encoder, cross-attention fusion and Token Decoder weights frozen (shown in purple). The model autoregressively generates coordinate tokens that are rasterized into binary masks.

Model	IoU (%)	BIoU (%)
LAVT (RefCOCO, zero-shot)	$<$ 1	3
Florence-2 (zero-shot)	$<$ 1	3
Qwen-VL (zero-shot)	3	5
Qwen-VL (fine-tuned)	10	42
Florence-2 (fine-tuned)	49	85

Table 4: Main results. Comparison of zero-shot and fine-tuned models on our geometry RIS benchmark.

Training Strategy	Low-DPI	High-DPI	Average
High-DPI only ( $\geq$ 250)	51	87	69
Mixed (ours)	75	93	82

Table 5: Resolution robustness (BIoU, %). Mixed-resolution training substantially improves generalization to low-DPI inputs.

Description Type	Length	BIoU (%)
Short	2–3 words	83
Medium	4–6 words	86
Long	8+ words	80

Table 6: Description complexity analysis (Florence-2 fine-tuned, BIoU %).

Figure 5: Qualitative results. The fine-tuned Florence-2 model accurately segments lines, triangles, and circles across diverse geometric queries and diagrammatic complexities.

Model	Params	Time	Hardware
Florence-2 (LoRA)	230M	6 h	4 $\times$ A100
Qwen-VL (LoRA)	7B	12 h	8 $\times$ A100

Table 7: Fine-tuning efficiency.

Figure 6: Artificial General Teacher (AGT) pipeline. The system processes a geometry problem through four stages: understanding, solving, explaining, and visually grounding (our contribution).

Referring Image Segmentation

Referring image segmentation maps a natural-language description of an object or region to its pixels in an image.

See all 45 papers

VeriGeo: Controllable Geometry Question Generation with Numerical and Analytical Verification
Jun 12, 2026Xiaoxian Duan, Zequn Liu, Yingce XiaSynthetic Data Generation Multimodal Reasoning
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Zhongguancun Academy, Beijing, China
GeoDial: A Multimodal Conversational Tutoring Dataset for Geometry Problem-Solving with Visual Tutor Turns
May 8, 2026Sankalan Pal Chowdhury, Junling Wang, Donya Rooein +2Multimodal Grounding Intelligent Tutoring Systems
ETH Zurich · ETH Zurich, ETH AI Center · Bocconi University
GeoSym127K: Scalable Symbolically-verifiable Synthesis for Multimodal Geometric Reasoning
May 10, 2026Jinhao Jing, Zheng Ma, Jinwei Liang +9Mathematical Reasoning Benchmarks Synthetic Data Generation
SenseTime Research · The Chinese University of Hong Kong, Shenzhen · University of California, Los Angeles +3

Jun 12, 2026J/K move · Enter open · S save

VeriGeo: Controllable Geometry Question Generation with Numerical and Analytical Verification

Xiaoxian Duan, Zequn Liu, Yingce Xia

Institute of Automation, Chinese Academy of Sciences, Beijing, China · Zhongguancun Academy, Beijing, China

Synthetic Data Generation Multimodal Reasoning Visually Grounded Reasoning Reasoning Verification Mathematical Problem Generation

Open paper AskPDF arXiv

Geometry problem generation is useful for AI-assisted education and multimodal mathematical reasoning, but reliable synthesis remains difficult because the problem statement, diagram, constraints, and solution should be mutually consistent. Existing methods often trade off controllability and reliability: seed-based rewriting is flexible but weakly verifiable, whereas diagram-first construction improves validity but is less suited to arbitrary user-specified constraints. We introduce VeriGeo, a controllable geometry generation framework grounded in executable reasoning traces. Given user constraints such as target concepts and difficulty, an Author agent generates a problem and diagram, and a Solver agent produces a proof-aligned solution. Both agents use a shared action sequence that connects natural language, diagrams, geometric constraints, and proof steps into a verifiable representation. A three-stage pipeline checks numerical consistency, analytical realizability, and global consistency, using verification-guided reflection to repair recoverable failures and reject unrecoverable ones. Across five LLM backbones, raw generations frequently fail these checks, while VeriGeo repairs a substantial fraction of the invalid attempts. Supervised fine-tuning on 8.7k examples generated by VeriGeo achieves the best reported GeoQA performance among end-to-end multimodal LLM-based solvers, and obtains strong results on PGPS9K and MathVista-GPS, demonstrating the effectiveness of verified synthetic data for improving multimodal geometry reasoning.

Abstract

Referring Image Segmentation

Explore similar work

VeriGeo: Controllable Geometry Question Generation with Numerical and Analytical Verification

GeoDial: A Multimodal Conversational Tutoring Dataset for Geometry Problem-Solving with Visual Tutor Turns

GeoSym127K: Scalable Symbolically-verifiable Synthesis for Multimodal Geometric Reasoning