Measuring Representation Robustness in Large Language Models for Geometry

Vedant Jawandhia; Yash Sinha; Murari Mandal; Ankan Pal; Dhruv Kumar

cs.CLApr 3, 2026

Measuring Representation Robustness in Large Language Models for Geometry

Authors: Vedant Jawandhia, Yash Sinha, Murari Mandal, Ankan Pal, Dhruv Kumar

Organizations: Department of Computer Science and Information Systems, BITS Pilani · School of Computer Science, KIIT University · Department of Mathematics, BITS Pilani

Abstract

Large language models (LLMs) are increasingly evaluated on mathematical reasoning, yet their robustness to equivalent problem representations remains poorly understood. In geometry, identical problems can be expressed in Euclidean, coordinate, or vector forms, but existing benchmarks report accuracy on fixed formats, implicitly assuming representation invariance and masking failures caused by representational changes alone. We propose GeoRepEval, a representation-aware evaluation framework that measures correctness, invariance, and consistency at the problem level across parallel formulations, combining strict answer matching, bootstrap confidence intervals, paired McNemar tests, representation-flip analyses, and regression controls for surface complexity. We prove that our Invariance@3 metric decomposes accuracy into robust and fragile components and is bounded by the weakest representation. Evaluating eleven LLMs on 158 curated high-school geometry problems (474 instances), we find accuracy gaps of up to 14 percentage points induced solely by representation choice. Vector formulations emerge as a consistent failure point, with Invariance@3 as low as 0.044 even after controlling for length and symbolic complexity. A convert-then-solve prompting intervention improves vector accuracy by up to 52 percentage points for high-capacity models, suggesting that failures reflect representation sensitivity rather than inability; however, low-capacity models show no gains, indicating deeper limitations. These results suggest that current models rely on representation-specific heuristics rather than abstract geometric reasoning. All datasets, prompts, and scripts are released at https://github.com/vedjaw/GeoRepEval.

Figures & tables

Panel of Figure 1 — Figure 1: GeoRepEval pipeline overview. The framework natively constructs and tracks mathematically equivalent variants (Euclidean, Coordinate, and Vector) through parallel LLM inference to isolate true reasoning capacity from representation sensitivity.

Source	#Prob.	Categories	Repr.
NCERT (IX–XII)	62	Len., Area, Ratio, Angle	Euc., Coord., Vec.
RD Sharma (X–XII)	53	Len., Area, Ratio, Angle	Euc., Coord., Vec.
RS Aggarwal	43	Len., Area, Ratio, Angle	Euc., Coord., Vec.
Total	158

Table 1: Dataset summary. 158 core problems drawn from three textbook sources, each instantiated in three representations (474 total instances). All problems are at the high-school level.

Model	Architecture	Parameters	Provider	Access
Claude Haiku 4.5	Not Disclosed	Not Disclosed	Anthropic	Closed
DeepSeek Chat	Decoder-only Transformer	Not Disclosed	DeepSeek	Open
Gemini 2.5 Flash	Transformer-based architecture	Not Disclosed	Google	Closed
Gemma 2 9B IT $\dagger$	Decoder-only Transformer	9B	Google	Open
Gemma 3 12B IT	Decoder-only Transformer	12B	Google	Open
LLaMA 3.1 8B Instruct	Decoder-only Transformer	8B	Meta	Open

Table 2: Large language models evaluated in this study. All models were accessed via the OpenRouter API ( https://openrouter.ai ) in a zero-shot setting without any fine-tuning.

\dagger

Gemma-2-9B produced valid responses for 110 of 158 problems due to output formatting failures; only valid responses are included in its evaluation.

Metric	Measures	Higher = Better?
Accuracy (Euclid / Coord / Vec)	Per-representation accuracy	Yes
Accuracy Gap	Best $-$ worst representation accuracy	No
Invariance@3	Correct across all three representations	Yes
Consistency@3	Identical output across all three	Yes
Representation-Flip Categories	Problem-level patterns (CCC, CCW, etc.)	–
Pairwise Transfer / Coherence	Cross-representation success rates	Yes

Table 3: Evaluation metrics. See § 3.2 and § 3.3 for formal definitions and properties.

Figure 2: Accuracy by geometry representation across models. Each group shows performance under Euclidean, Coordinate, and Vector formulations of the same problems.

Model	Coord	Euclid	Vector	Best Rep.	Acc. Gap	Inv@3	Cons@3
Claude Haiku 4.5	0.53	0.60	0.46	Euclid	0.14	0.401	0.631
DeepSeek Chat	0.44	0.45	0.35	Euclid	0.10	0.266	0.424
Gemini 2.5 Flash	0.56	0.58	0.45	Euclid	0.13	0.392	0.494
Gemma 2 9B IT	0.27	0.29	0.28	Euclid	0.02	0.145	0.245
Gemma 3 12B IT	0.31	0.37	0.32	Euclid	0.06	0.178	0.268
LLaMA 3.1 8B Inst.	0.14	0.21	0.13	Euclid	0.08	0.044	0.101

Table 4: Representation-wise accuracy, invariance, and consistency metrics across evaluated language models. Accuracy Gap = best

-

worst representation accuracy. Inv@3 = Invariance@3 (correct across all three). Cons@3 = Consistency@3 (identical output across all three). Inconsistency =

1-\text{Cons@3}

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Representation-flip patterns. Stacked bars show problem-level correctness patterns (E, C, V) for each model.

Model	#Prob	Fail-E	Fail-C	Fail-V	EC $\|$ V	CV $\|$ E	EV $\|$ C	EC-Co	CV-Co
Claude-Haiku-4.5	157	2	6	11	0.13	0.03	0.08	0.82	0.84
DeepSeek-Chat	158	5	5	16	0.16	0.06	0.06	0.85	0.80
Gemini-2.5-Flash	158	2	4	15	0.17	0.03	0.06	0.84	0.80
Gemma-2-9B	110	5	7	5	0.06	0.07	0.09	0.77	0.78
Gemma-3-12B	157	6	9	8	0.08	0.06	0.08	0.77	0.80
LLaMA-3.1-8B	158	3	4	8	0.06	0.02	0.03	0.84	0.85

Appendix

Table 5: Pairwise transfer and coherence analysis. Transfer rates (EC

|

V, CV

|

E, EV

|

C) and pairwise coherence (EC-Co, CV-Co).

Figure 4: Pairwise transfer rates. Heatmap of how often two representations succeed when the third fails.

Model	#Prob	CCC	CCW	CWC	WCC	CWW	WCW	WWC	WWW
Claude-Haiku-4.5	157	63	11	6	2	14	6	2	53
DeepSeek-Chat	158	42	16	5	5	8	6	4	72
Gemini-2.5-Flash	158	62	15	4	2	10	9	3	53
Gemma-2-9B	110	16	5	7	5	6	7	5	59
Gemma-3-12B	157	28	8	9	6	14	7	8	77
LLaMA-3.1-8B	158	7	8	4	3	14	4	7	111

Appendix

Table 6: Problem-level representation-flip pattern counts. Each triplet (E,C,V) records correctness (C/W) across Euclidean, Coordinate, and Vector formulations.

Figure 5: Geometry sensitivity across models. Representation-wise accuracy variation for each evaluated model under Euclidean, Coordinate, and Vector formulations, illustrating differential robustness to problem representation.

Model	Coord	Euclid	Vector	Acc. Gap	Inv@3	Cons@3
Claude-Haiku-4.5	0.96	0.94	0.96	0.02	0.918	0.741
DeepSeek-Chat	0.90	0.91	0.88	0.03	0.791	0.646
Gemini-2.5-Flash	0.96	0.95	0.97	0.02	0.948	0.812
LLaMA-3.1-8B	0.13	0.23	0.16	0.10	0.026	0.082
GPT-OSS-20B	0.89	0.87	0.86	0.03	0.772	0.519
Qwen-2.5-7B	0.83	0.77	0.78	0.06	0.596	0.443

Appendix

Table 7: Convert-then-solve (CTS) accuracy. Models first rewrite coordinate/vector problems into Euclidean form, then solve. Compared to direct evaluation (Table 4 ), CTS substantially narrows representation gaps for high-capacity models while having minimal effect on low-capacity ones.

Figure 6: Convert-then-solve accuracy by representation. Accuracy under the CTS intervention for each model across Euclidean, Coordinate, and Vector formulations. Compared to direct evaluation (Figure 2 ), representation gaps are substantially reduced for high-capacity models.

Figure 7: Representation-flip patterns under convert-then-solve. Stacked bars show problem-level correctness patterns (E, C, V) for each model after CTS prompting. Compared to direct evaluation (Figure 3 ), high-capacity models show a marked shift from partial-correctness patterns (CCW, CWC) toward fully invariant solutions (CCC).

Model	$p$ (C–E)	$\chi^{2}$	$p$ (C–V)	$\chi^{2}$	$p$ (E–V)	$\chi^{2}$
Claude-Haiku-4.5	0.036	4.32	0.108	2.56	0.0001	13.79
DeepSeek-Chat	0.839	0.04	0.029	4.65	0.014	5.94
Gemini-2.5-Flash	0.690	0.16	0.003	8.26	0.0003	12.03
Gemma-2-9B	1.000	0.00	1.000	0.00	1.000	0.00
Gemma-3-12B	0.132	2.25	0.860	0.03	0.243	1.36
LLaMA-3.1-8B	0.043	4.00	1.000	0.00	0.050	3.78

Appendix

Table 8: McNemar paired significance tests across geometry representations. Bold

p

-values indicate statistically significant differences (

p<0.05

Model	$b_{CE}$	$c_{CE}$	$b_{CV}$	$c_{CV}$	$b_{EV}$	$c_{EV}$
Claude-Haiku-4.5	8	20	17	8	25	4
DeepSeek-Chat	11	13	22	9	24	9
Gemini-2.5-Flash	11	14	24	7	25	5
Gemma-2-9B	12	13	12	12	11	10
Gemma-3-12B	13	23	15	17	22	14
LLaMA-3.1-8B	7	18	12	11	22	10

Appendix

Table 9: Paired disagreement counts used in McNemar tests. Asymmetric counts indicate representation-specific error patterns.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,402 papers

GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams
Feb 25, 2026Yushun Zhang, Weiping Fu, Zesheng Yang +6Mathematical Reasoning Benchmarks LLM Mathematical Reasoning
School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, Xi’an, China
Diagnosing Sampled LLM Reasoning in Formal Geometry: Coverage, Realization, and Validity Evidence
Sep 26, 2026Xiao Yue, Guangzhi QuLLM Evaluation Reasoning Consistency in Language Models
Oakland University, Rochester, United States
From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models
Jun 3, 2026Chen Chu, Bita Azarijoo, Li Xiong +2Spatial Reasoning Benchmarks Multimodal Large Language Models
University of Southern California Los Angeles, CA, USA · Emory University Atlanta, GA, USA · Novateur Research Solutions Ashburn, Virginia, USA

Feb 25, 2026J/K move · Enter open · S save

GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams

Yushun Zhang, Weiping Fu, Zesheng Yang, Bo Zhao+5

School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, Xi’an, China

Mathematical Reasoning Benchmarks LLM Mathematical Reasoning Visually Grounded Reasoning

Open paper AskPDF arXiv

Evaluating the symbolic reasoning of large language models (LLMs) calls for geometry benchmarks that require multi-step proofs grounded in both text and diagrams. However, existing benchmarks are often limited in scale and rarely provide visually grounded multiple-choice questions, limiting reliable evaluation of complex reasoning. We introduce GeoChallenge, a dataset of 90K automatically generated multiple-choice geometry proof problems, each requiring multi-step reasoning over aligned textual descriptions and diagrams. GeoChallenge provides fine-grained complexity ratings and formal language annotations to enable controlled evaluation. Experiments on multiple advanced LLMs show a clear performance gap between models and humans (the best-performing model, GPT-5-nano, achieves 75.89 exact match vs. 94.74 for humans). Further analysis also reveals three common failure patterns of LLMs: (1) exact match failures under the multiple-choice setting; (2) weak visual reliance; and (3) overextended reasoning without convergence.

Abstract

LLM Evaluation

Explore similar work

GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams

Diagnosing Sampled LLM Reasoning in Formal Geometry: Coverage, Realization, and Validity Evidence

From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models