MathVis: Can AI Reason through Drawing? A Benchmark for Mathematical Figure Generation

Jingjing Wei; Hao Zhao; Xinke Li; Linxi Yu; Xiaosong Gao; Shanshan Xiao

cs.AIOct 9, 2026

MathVis: Can AI Reason through Drawing? A Benchmark for Mathematical Figure Generation

Authors: Jingjing Wei, Hao Zhao, Xinke Li, Linxi Yu, Xiaosong Gao, Shanshan Xiao

Organizations: Zhihu (zhihu.com) · Tsinghua University (tsinghua.edu.cn) · City University of Hong Kong (cityu.edu.hk) · CIS, Cornell University (cornell.edu)

Abstract

This paper introduces MathVis, a benchmark designed to evaluate LLMs' ability to visualize frontier mathematical problems and perform diagram-assisted reasoning. The benchmark consists of 512 tasks, each comprising a mathematical statement, a reference diagram, executable TikZ or SVG code, a Gold Intermediate Representation (IR), atomic rubrics, and error codes. Through rigorous evaluation, GPT-6-Astra, Claude Fable 5, and Gemini 3.1 Pro Preview achieve pass@1 scores of 31.25%, 26.56%, and 22.85%, respectively. We found that although the tasks require visualization, LLMs produce outputs dominated by formal languages, symbols, and graphics programs, rather than explicit diagram-mediated reasoning. Furthermore, while the solutions generated by LLMs may contain all the necessary elements, they are often visually confusing to humans. Ultimately, this benchmark aims to visualize mathematical problems and promote research on enabling LLMs to simplify their reasoning via visual diagrams.

Figures & tables

Figure 1: Visualizing the Jordan Curve Theorem. While the formal topological proof of this theorem is complex, a precise diagram delineating the inner and outer regions can make the reasoning process more intuitive.

Figure 2: The four core capabilities required for mathematical figure generation.

Figure 3: Representative benchmarks positioned by first public-release date and qualitative research scope.

Figure 4: Curation pipeline and verification architecture.

Model	pass@1	$D$ -PSR	$T$	$S$ -PSR	pass@3	pass^3
GPT-6-Astra	31.25	30.73	94.40	31.20	38.28	24.22
Claude Fable 5	26.56	25.13	84.11	26.07	29.30	19.14
Gemini 3.1 Pro Preview	22.85	20.70	78.13	20.70	27.34	15.23

Top three configurations extracted from Table 6 .

Figure 6: Observed strict $D$ -PSR rates for the 16 evaluated model configurations, ordered from highest to lowest. Only models listed in the accompanying evaluation workbook are shown.

Question	Mathematical bottleneck	Direct baseline evidence
DG-FIG-01	Coordinate image arrows, transformation directions, and inverse mapping labels	Archived outputs reverse or omit required mappings; displayed artifacts fail the strict delivery gate
DG-FIG-02	Four open semi-circular domains, quadrant intersections, and radical signs	Archived outputs include rendering failures and incomplete domain constructions
CV2-FIG-13-01	Disk interior angles, symmetry axes, and the $2\alpha$ relation	Archived outputs include missing angle relations and rendering failures

Table 1: Agent challenge examples. Direct baselines and independently evaluated agent runs are recorded separately under the same tasks, prompts, and budget definitions.

Figure 7: Expert reference figure for the Rouché’s Theorem task: the images of $f(z)=z^{3}$ and $f(z)+g(z)=z^{3}+z+1$ on $C=\{z:|z|=2\}$ wind three times around the origin, so both have three zeros inside $C$ .

ID	Type	Atomic decision question	Evidence and allowed equivalent forms	Weight / code
CA-H1	Hard	Is $C$ correctly specified as the circle centered at $0$ with radius $2$ ?	A labeled equation or domain-plane circle, allowing global translation and uniform scaling when geometry is shown	DOMAIN
CA-H2	Hard	Are the images of $C$ under $f(z)=z^{3}$ and $f(z)+g(z)=z^{3}+z+1$ correctly drawn and labeled?	Curve geometry and function–curve binding, allowing equivalent color schemes and line styles	FUNCTION
CA-H3	Hard	Does the figure or its associated formula give the correct boundary comparison, $\|z+1\|\leq 3<8=\|z^{3}\|$ for $z\in C$ ?	Boundary evaluation on $\|z\|=2$ , accepting equivalent algebraic presentations of the same inequality chain	BOUNDARY
CA-H4	Hard	Does the figure indicate that $f(z)+g(z)$ has exactly three zeros inside $C$ , either by marking the three zeros or by showing a winding number of three?	Count and region membership of zero markers, or the winding number of the image curve about the origin; exact root coordinates are not required	ZEROCOUNT
CA-H5	Hard	Regardless of the graphical perspective, does the conclusion state that $f$ and $f+g$ have the same number of zeros inside $C$ , both being three counting multiplicities?	Text or formula extraction, accepting equivalent phrasing	CONCLUSION
CA-S1	Soft	Are the chosen contours, regions, winding cues, and any zero markers easy to distinguish?	Scored as $0$ , $0.5$ , or $1$ by experts based on public anchors	0.30

Table 2: Atomic constraint card for the Rouché’s Theorem task. Hard constraints gate delivery; soft-constraint scores are credited only once every hard constraint passes.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Professional work	Main risks	Archived records
Proposition screening	Recover visualizable propositions from traceable sources and assign to corresponding math sub-disciplines	Forcing illustrations for theorems lacking stable 2D semantics	Source page, local context, rights status, sub-discipline, and visualizability rationale
Instance specification	Fix regions, functions, parameters, manifolds, knots, directions, and conventions	Multiple legal instances having different visual meanings	Specific instance $\eta_{i}$ , public prompt $x_{i}$ , allowed equivalent forms, and prohibited conditions
Reference figure generation	Generate executable candidate figures using numerical, parametric, off-the-shelf libraries, or manual vector paths	Renderer style becoming a shortcut to identify questions	Source program, renderer version, path label, rendering results, and editing records
Automatic pre-check	Check schema, compilation, non-empty rendering, numerical invariants, and mechanically verifiable IR relations	Mistaking successful rendering as evidence of mathematical correctness	Machine logs, measured quantities, failed check items, and evidence confidence
Expert verification	Verify source consistency, instance validity, Gold IR coverage, constraint types, reference figure correctness, and ambiguous outputs	Figure appearance reasonably masking subtle semantic errors	Predicate-level decisions, evidence, reviewer sub-discipline, return reasons, and arbitration records

Appendix

Professional work, risks, and archived records at each stage.

Predicate family	Typical hard judgment	Typical soft judgment	Priority evidence
Regions and boundaries	Do regions, holes, boundary branches, inclusion relations, and interior/exterior markers have correct topology?	Are different branches easy to distinguish?	Vector paths, point-in-region tests, and expert review
Directions and traversals	Are contour directions mathematically correct and consistent with arrow or parameter directions?	Are direction hints clear and not overly crowded?	Path ordering, arrow extraction, and local tangent checks
Functions and images	Are pre-image domains, image domains, branch selections, and boundary correspondences correct?	Is the visual correspondence between pre-image and image clear?	Gold IR edge matching, function sample values, and label recognition
Zeros, poles, and critical points	Are multiplicities, positions relative to contours, and point type symbols correct?	Are different point types easy to distinguish?	Numerical roots or pole checks and region membership judgments
Quantitative invariants	Are modulus, argument, winding number, residue relations, or inequalities on specified sets correct?	Do invariants establish clear visual connections with relevant objects?	Symbol or numerical checks and formula parsing
Branches and branch cuts	Are branch cuts, Riemann surfaces, discontinuities, and analytic continuation paths mathematically compatible?	Are different layers or sides of branch cuts easy to distinguish?	Topological adjacency, sampled arguments, and expert review

Appendix

Table 3: Complex analysis predicate family.

Field	Admission decision question	Required evidence
Source consistency	Does the instantiated question maintain the source proposition, notation, assumptions, and directions?	Source page or original figure, local context, and constructor’s verification notes
Instance validity	Are regions, functions, parameters, objects, and conventions clear and mutually consistent?	Symbol table, parameter check, and executable or numerical witness
Prompt sufficiency	Can a qualified solver reconstruct the required artifact without knowing the evaluation-side information?	Independent reconstruction record based solely on the prompt
Gold IR coverage	Does the IR contain all objects, relations, formulas, invariants, equivalent forms, and prohibited conditions required by the instance?	Item-by-item correspondence from source proposition to IR fields
Constraint validity	Is each atomic constraint decidable, non-redundant, linked to IR evidence, and correctly classified as hard or soft?	Constraint card with type rationale, weight rationale, and failure codes
Reference artifact correctness	Does the executable reference artifact satisfy all hard constraints and expected soft specifications?	Compilation logs, rendering results, automatic measurements, and expert sign-off

Appendix

Table 4: Task admission rubric.

Model	pass@1	$D$ -PSR	$T$	$S$ -PSR	pass@3	pass^3
GPT-6-Astra	31.25	30.73	94.40	31.20	38.28	24.22
Claude Fable 5	26.56	25.13	84.11	26.07	29.30	19.14
Gemini 3.1 Pro Preview	22.85	20.70	78.13	20.70	27.34	15.23
GPT-5.6-sol	21.48	19.99	92.32	20.05	24.22	16.60
Claude Opus 4.8	20.90	19.92	78.91	20.12	24.80	15.04
Doubao Seed 2.1 Pro	20.90	19.14	81.05	20.03	24.22	14.84

Appendix

Table 6: Scorecard for the 16 evaluated model configurations on the 512 official questions. All rates are observed percentages. pass@1, pass@3, and pass^3 each use 512 question-level outcomes;

D

-PSR,

T

, and

S

-PSR use

512\times 3=1536

rollout-level outcomes. pass@3 measures whether at least one of three outputs strictly passes, whereas pass^3 measures whether all three strictly pass.

Field	Definition	Current status
Questions	Question ID, source, prompt, reference materials, and Gold IR	512 official questions
Output contract	Complete independent TikZ or SVG, output code only	Executed
Technical gate	Parsing / compilation / rendering / non-empty output	Audited
Semantic gate	Rubrics	Diagnostic report
Delivery gate	Technical gate, rubrics, readability	Main strict results
Repeated runs	Outputs per question under the same prompt and Harness	All 16 configurations: 3 reported rollouts; question-level logs retained for 3 configurations

Appendix

Table 2: Reproducibility checklist: fields recorded for evaluation.

Explore similar work

Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities
Aug 9, 2026Harish Kashyap, Kiran Byadarhaly, Sriram Chakaravarthy +2LLM Evaluation T2I Generation Evaluation
Pandita AI Inc., 2540 N First St, San Jose CA 95131, USA
MathGen: Revealing the Illusion of Mathematical Competence through Text-to-Image Generation
Mar 30, 2026Ruiyao Liu, Hui Shen, Ping Zhang +16T2I Generation T2I Generation Evaluation
University of Pennsylvania · University of Michigan · The Ohio State University +8
VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark
Jun 2, 2026Amirhossein Dabiriaghdam, Shayan Vassef, Mohammadreza Bakhtiari +5Mathematical Reasoning Benchmarks Mathematical Reasoning
University of British Columbia · University of Illinois Chicago · Stony Brook University

Aug 9, 2026J/K move · Enter open · S save

Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities

Harish Kashyap, Kiran Byadarhaly, Sriram Chakaravarthy, Sanyukta Tuti+1

Pandita AI Inc., 2540 N First St, San Jose CA 95131, USA

LLM Evaluation T2I Generation Evaluation

Open paper AskPDF arXiv

The generation of mathematically precise diagrams from tex- tual prompts has emerged as a critical yet underexplored capability of Large Language Models (LLMs). This has been of interest to researchers in the areas of curriculum preparation, automated ranking of problem sets, and scientific publishing. For LLMs to achieve this, it requires per- fect coordination between Spatial Reasoning, Mathematical Reasoning, and Rendering systems. While existing benchmarks such as MathVision, MathVista are built for Math Reasoning or DiagramGenBenchmark, Mer- maidSeqBench on general purpose diagram generation, no prior work provides a standardized set of prompt, image pairs that can be used to evaluate the LLMs specifically on math diagram generation. This includes fields that span both both text-to-code and text-to-image paradigms. We introduce Math-Vision Diagrams, the first benchmark specifically designed to evaluate LLMs on mathematical diagram generation, and the first to assess text-to-code and text-to-image generation paradigms together in a single unified setting, agnostic of the underlying coding lan- guage or model type. Building on the Math-Vision benchmark, we select a subset of 2920 images out of 3040 from high-quality competition problems with essential visual context. A novel pipeline combining an ensemble of LLMs with Subject Matter Expert (SME) curation is presented, together with a suite of evaluation metrics. Testing several leading models against this benchmark, we demonstrate that LLMs struggle with math diagram generation. All code, data, curation pipeline, and evaluation scripts will be fully open-sourced.

Abstract

Explore similar work

Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities

MathGen: Revealing the Illusion of Mathematical Competence through Text-to-Image Generation

VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark