Measuring Representation Robustness in Large Language Models for Geometry

Vedant Jawandhia; Yash Sinha; Murari Mandal; Ankan Pal; Dhruv Kumar

cs.CLApr 3, 2026

Measuring Representation Robustness in Large Language Models for Geometry

Authors: Vedant Jawandhia, Yash Sinha, Murari Mandal, Ankan Pal, Dhruv Kumar

Organizations: Department of Computer Science and Information Systems, BITS Pilani · School of Computer Science, KIIT University · Department of Mathematics, BITS Pilani

Abstract

Large language models (LLMs) are increasingly evaluated on mathematical reasoning, yet their robustness to equivalent problem representations remains poorly understood. In geometry, identical problems can be expressed in Euclidean, coordinate, or vector forms, but existing benchmarks report accuracy on fixed formats, implicitly assuming representation invariance and masking failures caused by representational changes alone. We propose GeoRepEval, a representation-aware evaluation framework that measures correctness, invariance, and consistency at the problem level across parallel formulations, combining strict answer matching, bootstrap confidence intervals, paired McNemar tests, representation-flip analyses, and regression controls for surface complexity. We prove that our Invariance@3 metric decomposes accuracy into robust and fragile components and is bounded by the weakest representation. Evaluating eleven LLMs on 158 curated high-school geometry problems (474 instances), we find accuracy gaps of up to 14 percentage points induced solely by representation choice. Vector formulations emerge as a consistent failure point, with Invariance@3 as low as 0.044 even after controlling for length and symbolic complexity. A convert-then-solve prompting intervention improves vector accuracy by up to 52 percentage points for high-capacity models, suggesting that failures reflect representation sensitivity rather than inability; however, low-capacity models show no gains, indicating deeper limitations. These results suggest that current models rely on representation-specific heuristics rather than abstract geometric reasoning. All datasets, prompts, and scripts are released at https://github.com/vedjaw/GeoRepEval.

Figures & tables

Panel of Figure 1 — Figure 1: GeoRepEval pipeline overview. The framework natively constructs and tracks mathematically equivalent variants (Euclidean, Coordinate, and Vector) through parallel LLM inference to isolate true reasoning capacity from representation sensitivity.

Source	#Prob.	Categories	Repr.
NCERT (IX–XII)	62	Len., Area, Ratio, Angle	Euc., Coord., Vec.
RD Sharma (X–XII)	53	Len., Area, Ratio, Angle	Euc., Coord., Vec.
RS Aggarwal	43	Len., Area, Ratio, Angle	Euc., Coord., Vec.
Total	158

Table 1: Dataset summary. 158 core problems drawn from three textbook sources, each instantiated in three representations (474 total instances). All problems are at the high-school level.

Model	Architecture	Parameters	Provider	Access
Claude Haiku 4.5	Not Disclosed	Not Disclosed	Anthropic	Closed
DeepSeek Chat	Decoder-only Transformer	Not Disclosed	DeepSeek	Open
Gemini 2.5 Flash	Transformer-based architecture	Not Disclosed	Google	Closed
Gemma 2 9B IT $\dagger$	Decoder-only Transformer	9B	Google	Open
Gemma 3 12B IT	Decoder-only Transformer	12B	Google	Open
LLaMA 3.1 8B Instruct	Decoder-only Transformer	8B	Meta	Open

Table 2: Large language models evaluated in this study. All models were accessed via the OpenRouter API ( https://openrouter.ai ) in a zero-shot setting without any fine-tuning.

\dagger

Gemma-2-9B produced valid responses for 110 of 158 problems due to output formatting failures; only valid responses are included in its evaluation.

Metric	Measures	Higher = Better?
Accuracy (Euclid / Coord / Vec)	Per-representation accuracy	Yes
Accuracy Gap	Best $-$ worst representation accuracy	No
Invariance@3	Correct across all three representations	Yes
Consistency@3	Identical output across all three	Yes
Representation-Flip Categories	Problem-level patterns (CCC, CCW, etc.)	–
Pairwise Transfer / Coherence	Cross-representation success rates	Yes

Table 3: Evaluation metrics. See § 3.2 and § 3.3 for formal definitions and properties.

Figure 2: Accuracy by geometry representation across models. Each group shows performance under Euclidean, Coordinate, and Vector formulations of the same problems.

Model	Coord	Euclid	Vector	Best Rep.	Acc. Gap	Inv@3	Cons@3
Claude Haiku 4.5	0.53	0.60	0.46	Euclid	0.14	0.401	0.631
DeepSeek Chat	0.44	0.45	0.35	Euclid	0.10	0.266	0.424
Gemini 2.5 Flash	0.56	0.58	0.45	Euclid	0.13	0.392	0.494
Gemma 2 9B IT	0.27	0.29	0.28	Euclid	0.02	0.145	0.245
Gemma 3 12B IT	0.31	0.37	0.32	Euclid	0.06	0.178	0.268
LLaMA 3.1 8B Inst.	0.14	0.21	0.13	Euclid	0.08	0.044	0.101

Table 4: Representation-wise accuracy, invariance, and consistency metrics across evaluated language models. Accuracy Gap = best

-

worst representation accuracy. Inv@3 = Invariance@3 (correct across all three). Cons@3 = Consistency@3 (identical output across all three). Inconsistency =

1-\text{Cons@3}

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Representation-flip patterns. Stacked bars show problem-level correctness patterns (E, C, V) for each model.

Model	#Prob	Fail-E	Fail-C	Fail-V	EC $\|$ V	CV $\|$ E	EV $\|$ C	EC-Co	CV-Co
Claude-Haiku-4.5	157	2	6	11	0.13	0.03	0.08	0.82	0.84
DeepSeek-Chat	158	5	5	16	0.16	0.06	0.06	0.85	0.80
Gemini-2.5-Flash	158	2	4	15	0.17	0.03	0.06	0.84	0.80
Gemma-2-9B	110	5	7	5	0.06	0.07	0.09	0.77	0.78
Gemma-3-12B	157	6	9	8	0.08	0.06	0.08	0.77	0.80
LLaMA-3.1-8B	158	3	4	8	0.06	0.02	0.03	0.84	0.85

Appendix

Table 5: Pairwise transfer and coherence analysis. Transfer rates (EC

|

V, CV

|

E, EV

|

C) and pairwise coherence (EC-Co, CV-Co).

Figure 4: Pairwise transfer rates. Heatmap of how often two representations succeed when the third fails.

Model	#Prob	CCC	CCW	CWC	WCC	CWW	WCW	WWC	WWW
Claude-Haiku-4.5	157	63	11	6	2	14	6	2	53
DeepSeek-Chat	158	42	16	5	5	8	6	4	72
Gemini-2.5-Flash	158	62	15	4	2	10	9	3	53
Gemma-2-9B	110	16	5	7	5	6	7	5	59
Gemma-3-12B	157	28	8	9	6	14	7	8	77
LLaMA-3.1-8B	158	7	8	4	3	14	4	7	111

Appendix

Table 6: Problem-level representation-flip pattern counts. Each triplet (E,C,V) records correctness (C/W) across Euclidean, Coordinate, and Vector formulations.

Figure 5: Geometry sensitivity across models. Representation-wise accuracy variation for each evaluated model under Euclidean, Coordinate, and Vector formulations, illustrating differential robustness to problem representation.

Model	Coord	Euclid	Vector	Acc. Gap	Inv@3	Cons@3
Claude-Haiku-4.5	0.96	0.94	0.96	0.02	0.918	0.741
DeepSeek-Chat	0.90	0.91	0.88	0.03	0.791	0.646
Gemini-2.5-Flash	0.96	0.95	0.97	0.02	0.948	0.812
LLaMA-3.1-8B	0.13	0.23	0.16	0.10	0.026	0.082
GPT-OSS-20B	0.89	0.87	0.86	0.03	0.772	0.519
Qwen-2.5-7B	0.83	0.77	0.78	0.06	0.596	0.443

Appendix

Table 7: Convert-then-solve (CTS) accuracy. Models first rewrite coordinate/vector problems into Euclidean form, then solve. Compared to direct evaluation (Table 4 ), CTS substantially narrows representation gaps for high-capacity models while having minimal effect on low-capacity ones.

Figure 6: Convert-then-solve accuracy by representation. Accuracy under the CTS intervention for each model across Euclidean, Coordinate, and Vector formulations. Compared to direct evaluation (Figure 2 ), representation gaps are substantially reduced for high-capacity models.

Figure 7: Representation-flip patterns under convert-then-solve. Stacked bars show problem-level correctness patterns (E, C, V) for each model after CTS prompting. Compared to direct evaluation (Figure 3 ), high-capacity models show a marked shift from partial-correctness patterns (CCW, CWC) toward fully invariant solutions (CCC).

Model	$p$ (C–E)	$\chi^{2}$	$p$ (C–V)	$\chi^{2}$	$p$ (E–V)	$\chi^{2}$
Claude-Haiku-4.5	0.036	4.32	0.108	2.56	0.0001	13.79
DeepSeek-Chat	0.839	0.04	0.029	4.65	0.014	5.94
Gemini-2.5-Flash	0.690	0.16	0.003	8.26	0.0003	12.03
Gemma-2-9B	1.000	0.00	1.000	0.00	1.000	0.00
Gemma-3-12B	0.132	2.25	0.860	0.03	0.243	1.36
LLaMA-3.1-8B	0.043	4.00	1.000	0.00	0.050	3.78

Appendix

Table 8: McNemar paired significance tests across geometry representations. Bold

p

-values indicate statistically significant differences (

p<0.05

Model	$b_{CE}$	$c_{CE}$	$b_{CV}$	$c_{CV}$	$b_{EV}$	$c_{EV}$
Claude-Haiku-4.5	8	20	17	8	25	4
DeepSeek-Chat	11	13	22	9	24	9
Gemini-2.5-Flash	11	14	24	7	25	5
Gemma-2-9B	12	13	12	12	11	10
Gemma-3-12B	13	23	15	17	22	14
LLaMA-3.1-8B	7	18	12	11	22	10

Appendix

Table 9: Paired disagreement counts used in McNemar tests. Asymmetric counts indicate representation-specific error patterns.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,402 papers

Feb 25, 2026

GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams

Evaluating the symbolic reasoning of large language models (LLMs) calls for geometry benchmarks that require multi-step proofs grounded in both text and diagrams. However, existing benchmarks are often limited in scale and rarely provide visually grounded multiple-choice questions, limiting reliable evaluation of complex reasoning. We introduce GeoChallenge, a dataset of 90K automatically generated multiple-choice geometry proof problems, each requiring multi-step reasoning over aligned textual descriptions and diagrams. GeoChallenge provides fine-grained complexity ratings and formal language annotations to enable controlled evaluation. Experiments on multiple advanced LLMs show a clear performance gap between models and humans (the best-performing model, GPT-5-nano, achieves 75.89 exact match vs. 94.74 for humans). Further analysis also reveals three common failure patterns of LLMs: (1) exact match failures under the multiple-choice setting; (2) weak visual reliance; and (3) overextended reasoning without convergence.

Yushun Zhang, Weiping Fu, Zesheng Yang +6

School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, Xi’an, China

Mathematical Reasoning Benchmarks LLM Mathematical Reasoning Visually Grounded Reasoning

Ask PDF

Sep 26, 2026

Diagnosing Sampled LLM Reasoning in Formal Geometry: Coverage, Realization, and Validity Evidence

Repeated sampling can reveal a correct numerical answer without yielding either a reliable system output or a supported derivation. We present Coverage, Realization, and Validity Evidence (CRV), an evaluation protocol for sampled large language model (LLM) reasoning over formal geometry states. Coverage is answer availability, realization is readout accuracy on the frozen candidate pool, and validity evidence is a label-blinded critic judgment of derivational support rather than a proof certificate. CRV freezes each candidate pool before comparing readouts and analyzes covered failures by correct-answer multiplicity and within-problem discrimination. On HardShift441, a 441-problem set for which a reference solver leaves 406 problems unsolved, a LoRA-adapted Qwen2.5-7B generator obtains 24.2% average single-sample accuracy and 68.9% pass@16, whereas verifier-weighted self-consistency (WSC) reaches 38.0%. Readout accuracy is particularly low when the correct answer occurs only once or twice in the pool. In a separate constructed audit of 195 covered problems, the critic labels 12 correct-answer representatives as supported, 181 as refuted, and two as uncertain. These results show that coverage, realization, and validity evidence from the critic are distinct quantities and should be reported separately.

Xiao Yue, Guangzhi Qu

Oakland University, Rochester, United States

LLM Evaluation Reasoning Consistency in Language Models

Ask PDF

Jun 3, 2026

From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models

Recent large language models (LLMs) often appear to exhibit spatial reasoning ability; however, this capability is largely symbolic, arising from pattern matching over spatial language rather than true geometric reasoning over space. Because LLMs operate on discrete tokens, they lack native support for continuous spatial representations, explicit geometric computation, and structured spatial operators. To address this limitation, we introduce the Spatial Language Model (SLM), the first multimodal LLM that treats location information as a first-class modality and enables geometric spatial reasoning within the model's inference process. SLM directly operates on learned spatial representations rather than textual descriptions of spatial relations. To support effective training, we construct a Spatial Instruction Dataset that aligns spatial representations, atomic geometric operations, and natural language instructions. We further propose a new benchmark named SpatialEval, which is designed to evaluate spatial reasoning across attributes, distance, topology, and relative-position tasks. Extensive experiments show that SLM significantly outperforms existing LLM-based approaches that rely on symbolic reasoning via prompt engineering or textual abstraction, demonstrating the benefits of integrating geometric spatial representations for robust spatial reasoning. Our instruction dataset, evaluation benchmark, model training codes, and models' checkpoints can be found at: \hyperlink{https://github.com/chuchen2017/SLM}{https://github.com/chuchen2017/SLM}.

Chen Chu, Bita Azarijoo, Li Xiong +2

University of Southern California Los Angeles, CA, USA · Emory University Atlanta, GA, USA · Novateur Research Solutions Ashburn, Virginia, USA

Spatial Reasoning Benchmarks Multimodal Large Language Models Geometric Representation Learning Spatial Reasoning Spatial Reasoning in Language Models

Ask PDF

Abstract

LLM Evaluation

Explore similar work

GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams

Diagnosing Sampled LLM Reasoning in Formal Geometry: Coverage, Realization, and Validity Evidence

From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models