GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams
Authors: Yushun Zhang, Weiping Fu, Zesheng Yang, Bo Zhao, Lingling Zhang, Jian Zhang, Yumeng Fu, Jiaxing Huang, Jun Liu
Organizations: School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, Xi’an, China
Evaluating the symbolic reasoning of large language models (LLMs) calls for geometry benchmarks that require multi-step proofs grounded in both text and diagrams. However, existing benchmarks are often limited in scale and rarely provide visually grounded multiple-choice questions, limiting reliable evaluation of complex reasoning. We introduce GeoChallenge, a dataset of 90K automatically generated multiple-choice geometry proof problems, each requiring multi-step reasoning over aligned textual descriptions and diagrams. GeoChallenge provides fine-grained complexity ratings and formal language annotations to enable controlled evaluation. Experiments on multiple advanced LLMs show a clear performance gap between models and humans (the best-performing model, GPT-5-nano, achieves 75.89 exact match vs. 94.74 for humans). Further analysis also reveals three common failure patterns of LLMs: (1) exact match failures under the multiple-choice setting; (2) weak visual reliance; and (3) overextended reasoning without convergence.
Figures & tables
Figure 1: Examples in GeoChallenge-90K dataset.
Figure 2: Pipeline of dataset generation
Dataset
Size
AG
IF
CR
Avg PL
Avg DL
FA
LT
QT
GeoS Seo et al. (2015)
186
✗
T,I
✗
NA
23.34
✗
EN
SA
MATH Hendrycks et al. (2021)
12,500
✗
T only
✔
5.48
202.66
✗
EN
OE
Geometry3K Lu et al. (2021)
3,002
✗
T,I
✗
NA
12.19
✔
EN
SA
GeoQA Chen et al. (2021)
5,010
✗
T,I
✗
2.77
52.46
✗
ZH
SA
UniGeo Chen et al. (2022)
9,543
✗
T,I
✗
2.57
16.23
✔
EN
SA, OE
GPSM8K Cobbe et al. (2021)
8,790
✗
T only
✗
4.59
45.27
✗
EN
OE
Table 1: Comparison between GeoChallenge-90K benchmark and existing geometry problem solving benchmarks. AG: Automatic Generation. IF: Input Format, T for text and I for image. CR: Complexity Rating. Avg PL: Average Proof Length. Avg DL: Average Description Length. FA: Formal Annotation. LT: Language Type, EN for English and ZH for Chinese. QT: Question Type, SA for single-answer, MA for multiple-answer and OE for open-ended question
Model
MATH
Geome- try3K
GeoCha- llenge
Gemini 1.5 Pro
82.19
73.88
24.45
Claude 3.5 Sonnet
58.05
71.65
21.81
GPT-4o
57.72
64.92
17.51
Table 2: Performance comparison of different benchmarks
Model
EMA ↑
EME ↑
EMM ↑
EMH ↑
P ↑
R ↑
F1 ↑
HA ↑
Avg #Sel
Baselines
Random
6.25
6.25
6.25
6.25
50.00
50.00
50.00
46.88
2.13
General-Purpose Models
GPT-4o
17.51
29.09
13.81
9.24
52.87
68.83
59.81
58.62
2.18
Claude 3.5 Sonnet
21.81
27.27
22.94
10.87
54.88
71.73
62.19
63.22
2.20
Gemini 1.5 Pro
24.45
38.18
22.27
9.24
62.31
76.79
68.79
69.74
2.01
Table 3: Overall performance on GeoChallenge with both text and images provided. EMA/EME/EMM/EMH report Exact Match (EM) on the All/Easy/Medium/Hard splits, respectively.
Figure 3: Geometry elements in GeoChallenge-90K
Model
EMA ↑
EME ↑
EMM ↑
EMH ↑
P ↑
R ↑
F1 ↑
HA ↑
Avg #Sel
Baselines
Random
6.25
6.25
6.25
6.25
50.00
50.00
50.00
46.88
2.13
General-Purpose Models
GPT-4o
13.99
32.00
8.69
0.0
43.75
55.89
49.08
48.62
2.14
Claude 3.5 Sonnet
17.62
32.73
15.37
0.54
51.43
63.48
56.83
60.21
2.02
Gemini 1.5 Pro
20.59
44.00
14.70
0.0
60.32
74.14
66.52
66.77
1.97
Table 4: Overall performance on GeoChallenge with text-only provided. EMA/EME/EMM/EMH report Exact Match (EM) on the All/Easy/Medium/Hard splits, respectively.
Figure 4: Error type across different models
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: the diagram corresponding to the premise above
Model
EMA ↑
EME ↑
EMM ↑
EMH ↑
P ↑
R ↑
F1 ↑
HA ↑
Avg #Sel
Both text and images provided
GPT-4o
11.89
19.64
9.13
7.07
40.27
54.66
46.38
45.84
2.25
Claude 3.5 Sonnet
17.62
30.18
13.59
8.70
52.37
64.86
57.95
60.85
2.05
Gemini 1.5 Pro
24.23
38.55
21.16
10.33
62.85
82.98
71.52
71.31
2.12
llava-1.5-7B
12.11
25.09
8.24
2.17
37.00
23.76
28.94
50.61
1.05
Qwen2-VL-7B
24.01
58.91
12.47
0.0
57.16
48.97
52.75
63.13
1.41
Appendix
Table 5: General-purpose models’ performance on GeoChallenge, where the same problems are presented with Chinese prompts rather than English.
Statistics
Total Problems
90,279
Coverage (Img / Sol / EN / ZH)
100% / 100% / 100% / 100%
Difficulty
90,279
* Easy
27,083 (30.0%)
* Medium
45,140 (50.0%)
* Hard
18,056 (20.0%)
Appendix
Table 6: Detailed statistics of the GeoChallenge dataset. EN and ZH denote English and Chinese statements, respectively.
name
meaning
X = angle_bisector(A,B,C)
Construct point X on the angle bisector of ∠ ABC
X = angle_mirror(A,B,C)
Construct point X such that BC is the bisector of ∠ ABX
X,Y,Z,I = centroid(A,B,C)
Construct point I as the centroid of △ ABC with midpoints X, Y, Z
X = circle(A,B,C)
Construct point X as the circumcenter of △ ABC
A,B,C,D = eq_quadrangle()
Construct quadrilateral ABCD with AD = BC
A,B,C,D = eq_trapezoid()
Construct trapezoid ABCD with AD = BC
Appendix
Table 7: Predefined Clause Templates for Generating Geometric Problems (Adapted and modified from Work AlphaGeometry)
premise(s)
conclusion
perp A B C D, perp C D E F, ncoll A B E
para A B E F
cong O A O B, cong O B O C, cong O C O D
cyclic A B C D
eqangle A B P Q C D P Q
para A B C D
cyclic A B P Q
eqangle P A P B Q A Q B
eqangle6 P A P B Q A Q B, ncoll P Q A B
cyclic A B P Q
cyclic A B C P Q R, eqangle C A C B R P R Q
cong A B P Q
Appendix
Table 8: Rules Used by the Symbolic Reasoning Engine for Deriving New Conclusions
Figure 6: English version of prompt to LLMs
Figure 7: Chinese version of prompt to LLMs
Figure 8: Example of easy level problem in GeoChallenge
Figure 9: Example of medium level problem in GeoChallenge
Figure 10: Example of hard level problem in GeoChallenge