Organizations: University of Pennsylvania · University of Michigan · The Ohio State University · USTC · Independent · City University of Hong Kong · University of Wisconsin · UCSB · University of Hong Kong · Peking University · Carnegie Mellon University
Modern generative models have demonstrated the ability to solve challenging mathematical problems. In many real-world settings, however, mathematical solutions must be expressed visually through diagrams, plots, geometric constructions, and structured symbolic layouts, where correctness depends on precise visual composition. This naturally raises the question of whether generative models can still do so when the answer must be rendered visually rather than written in text? To study this problem, we introduce MathGen, a rigorous benchmark of 420 problems spanning seven core domains, including 350 Clean-Scene problems and 70 paired Open-Scene problems. Each problem is evaluated under a Script-as-a-Judge protocol with problem-specific verification criteria implemented through reusable executable scripts for deterministic and reproducible evaluation. Experiments on representative open-source and proprietary text-to-image models show that mathematical fidelity remains a major bottleneck: even the best closed-source model reaches only 53.7% overall accuracy, while open-source models achieve just 1--11%, often near 0% on structured tasks, particularly those requiring precise geometric and functional rendering. Overall, current T2I models remain far from reliable at even elementary mathematical visual generation.
Figures & tables
Figure 1: Task taxonomy of MathGen . MathGen covers seven fundamental mathematical domains. Example prompts and reference illustrations are shown to provide an overview of the mathematical concepts evaluated in each domain. They highlight the diverse forms of numerical, geometric, and structural constraints that generative models are required to express, and illustrate the types of visual outcomes expected under correct mathematical interpretation.
Figure 2: Performance comparison. The chart shows the accuracy of representative open-source and closed-source text-to-image models on each MathGen domain.
Benchmark
Theme
Deterministic Eval
Geometry3K [ 16 ]
Geometry question answering
✗
ChartQA [ 17 ]
Chart reasoning from visual plots
✗
MathVista [ 1 ]
Visual mathematical reasoning tasks
✗
T2I-CompBench [ 9 ]
Compositional text-to-image generation
✗
GenExam [ 11 ]
Multidisciplinary generation tasks
✗
Math2Visual [ 12 ]
Puzzle-solving
✗
Table 1: Comparison with existing benchmarks. Prior datasets mainly evaluate visual mathematical understanding, while MathGen focuses on mathematical image generation with script-based verification.
Figure 3: Overview of the MathGen benchmark and evaluation pipeline. MathGen evaluates text-to-image models on seven mathematical domains using structured prompts and automatic verification. Generated images are validated against domain-specific structural, geometric, and logical constraints. Each criterion Ci is checked independently, and the final correctness is determined through logical aggregation.
Set
SD-3 Medium
SD-3.5 Medium
SD-3.5 Large
FLUX-2
FLUX-2 Pro
Clean-Scene
2.9
2.9
4.3
12.9
30.0
Open-Scene
10.0
8.6
5.7
7.1
12.9
FLUX Kontext-Pro
PixArt Σ
PixArt XL-2
HiDream I1
Qwen Image
Clean-Scene
10.0
4.3
4.3
2.9
12.9
Open-Scene
10.0
2.9
4.3
1.4
10.0
Infinity 8B
GoT-R1 7B
BAGEL
show-o2 1.5B
show-o2 7B
Table 2: Paired Clean-Scene/Open-Scene comparison on 70 matched mathematical instances ( 10 per domain). Each pair preserves the same numerical values, mathematical relations, required answer, and evaluation criteria; only the scene realization and necessary lexical grounding differ. Both rows report overall accuracy (%), computed as the number of correct generations divided by the corresponding 70 prompts.
Model
counting
angle
fraction
function
plane
set
solid
Overall
Diffusion Models
SD-3-Medium
0.0
0.0
0.0
0.0
6.0
0.0
6.0
1.7
SD-3.5-Medium
4.0
0.0
0.0
0.0
12.0
2.0
6.0
3.4
SD-3.5-Large
12.0
0.0
2.0
0.0
12.0
2.0
4.0
4.6
FLUX-2
8.0
2.0
8.0
2.0
42.0
8.0
8.0
11.1
PixArt- Σ
10.0
0.0
2.0
0.0
12.0
2.0
8.0
4.9
Table 3: Main results on the 350 -problem Clean-Scene set of MathGen . We report accuracy across seven mathematical domains, with 50 problems per domain. Closed-source models generally achieve the strongest overall performance, although performance remains far from perfect in several domains, particularly set and solid-geometry tasks. Best results in each column are highlighted in red and the second-best results in blue.
Figure 4: Qualitative comparison of T2I models on representative MathGen tasks.
Figure 5: Typical success and failure examples from the Clean-Scene and Open-Scene settings.
Figure 6: Error ratio of qualitative and quantitative failures across different mathematical topics for FLUX-2-Pro and Nano Banana Pro.
Figure 7: Representative error examples illustrating common qualitative and quantitative failures in preserving numerical and geometric constraints.
The generation of mathematically precise diagrams from tex- tual prompts has emerged as a critical yet underexplored capability of Large Language Models (LLMs). This has been of interest to researchers in the areas of curriculum preparation, automated ranking of problem sets, and scientific publishing. For LLMs to achieve this, it requires per- fect coordination between Spatial Reasoning, Mathematical Reasoning, and Rendering systems. While existing benchmarks such as MathVision, MathVista are built for Math Reasoning or DiagramGenBenchmark, Mer- maidSeqBench on general purpose diagram generation, no prior work provides a standardized set of prompt, image pairs that can be used to evaluate the LLMs specifically on math diagram generation. This includes fields that span both both text-to-code and text-to-image paradigms. We introduce Math-Vision Diagrams, the first benchmark specifically designed to evaluate LLMs on mathematical diagram generation, and the first to assess text-to-code and text-to-image generation paradigms together in a single unified setting, agnostic of the underlying coding lan- guage or model type. Building on the Math-Vision benchmark, we select a subset of 2920 images out of 3040 from high-quality competition problems with essential visual context. A novel pipeline combining an ensemble of LLMs with Subject Matter Expert (SME) curation is presented, together with a suite of evaluation metrics. Testing several leading models against this benchmark, we demonstrate that LLMs struggle with math diagram generation. All code, data, curation pipeline, and evaluation scripts will be fully open-sourced.
This paper introduces MathVis, a benchmark designed to evaluate LLMs' ability to visualize frontier mathematical problems and perform diagram-assisted reasoning. The benchmark consists of 512 tasks, each comprising a mathematical statement, a reference diagram, executable TikZ or SVG code, a Gold Intermediate Representation (IR), atomic rubrics, and error codes. Through rigorous evaluation, GPT-6-Astra, Claude Fable 5, and Gemini 3.1 Pro Preview achieve pass@1 scores of 31.25%, 26.56%, and 22.85%, respectively. We found that although the tasks require visualization, LLMs produce outputs dominated by formal languages, symbols, and graphics programs, rather than explicit diagram-mediated reasoning. Furthermore, while the solutions generated by LLMs may contain all the necessary elements, they are often visually confusing to humans. Ultimately, this benchmark aims to visualize mathematical problems and promote research on enabling LLMs to simplify their reasoning via visual diagrams.
Jingjing Wei, Hao Zhao, Xinke Li +3
Zhihu (zhihu.com) · Tsinghua University (tsinghua.edu.cn) · City University of Hong Kong (cityu.edu.hk) +1
AI systems are increasingly used to support educational content creation, yet it remains unclear whether they can generate outputs that faithfully represent the pedagogical concepts they are intended to teach. Thus, we introduce equation-to-visual generation, a task that, in contrast to conventional image generation, requires producing pedagogically meaningful visuals from arithmetic equations while precisely preserving their numerical and relational structure. Informed by interviews with teachers and an analysis of educational materials, we construct E2V-Bench, a benchmark spanning four pedagogically grounded visual types, along with automatic metrics for evaluating visual correctness. Our evaluation reveals that recent text-to-image (T2I) models frequently fail on this task, with errors dominated by incorrect object counts and broken relational structure. Building on this, we explore benchmark-guided enhancement strategies. These strategies improve representative models, while the remaining gap calls for stronger numerical and relational grounding in future T2I models.
Junling Wang, Boqi Chen, Heejin Do +3
Department of Computer Science, ETH Zurich · ETH AI Center