Diagnosing Sampled LLM Reasoning in Formal Geometry: Coverage, Realization, and Validity Evidence

Xiao Yue; Guangzhi Qu

cs.AISep 26, 2026

Diagnosing Sampled LLM Reasoning in Formal Geometry: Coverage, Realization, and Validity Evidence

Authors: Xiao Yue, Guangzhi Qu

Organizations: Oakland University, Rochester, United States

Abstract

Repeated sampling can reveal a correct numerical answer without yielding either a reliable system output or a supported derivation. We present Coverage, Realization, and Validity Evidence (CRV), an evaluation protocol for sampled large language model (LLM) reasoning over formal geometry states. Coverage is answer availability, realization is readout accuracy on the frozen candidate pool, and validity evidence is a label-blinded critic judgment of derivational support rather than a proof certificate. CRV freezes each candidate pool before comparing readouts and analyzes covered failures by correct-answer multiplicity and within-problem discrimination. On HardShift441, a 441-problem set for which a reference solver leaves 406 problems unsolved, a LoRA-adapted Qwen2.5-7B generator obtains 24.2% average single-sample accuracy and 68.9% pass@16, whereas verifier-weighted self-consistency (WSC) reaches 38.0%. Readout accuracy is particularly low when the correct answer occurs only once or twice in the pool. In a separate constructed audit of 195 covered problems, the critic labels 12 correct-answer representatives as supported, 181 as refuted, and two as uncertain. These results show that coverage, realization, and validity evidence from the critic are distinct quantities and should be reported separately.

Figures & tables

Training set	$N$	E/M/H	Examples
Generator training	2,556	479/847/1,230	4,901
Verifier training	3,338	607/1,049/1,682	31,701

Table 1: Training-set composition

Evaluation set	$N$	E/M/H	Unsolved (%)	Steps	Nodes
Legacy E/M	386	127/259/0	0.0	1.97	12.16
Legacy hard	184	0/0/184	0.0	5.34	14.19
HardShift441	441	0/0/441	92.1	5.84	18.70

Table 2: Evaluation-set composition

Setting	pass@16	Majority	Best-of- $K$	WSC	$\Delta_{\mathrm{WSC}}$	$\eta_{\mathrm{WSC}}$	$\mathbf{b}_{\mathrm{WSC}}$
Legacy hard	79.35	42.93	52.17	53.44	25.91	28.86	7.69/38.89/87.58
HardShift441	68.93	33.56	36.51	37.99	30.94	12.54	4.87/13.53/75.63

Table 3: Coverage and realization components of CRV at

K=16

. Rates are percentages;

\mathbf{b}_{\mathrm{WSC}}

lists accuracy on

c_{1}/c_{2}/c_{3+}

Figure 1: True-prefix budget curves with fixed calibration and pointwise 95% problem-identifier bootstrap intervals. The HardShift441 panel uses the Qwen2.5-7B diversity-trained generator, whereas the legacy-hard panel uses an earlier Qwen2.5-7B targeted-distillation generator trained with the same number of anchor examples that the base model answered correctly. Because the generators differ, the panels show within-setting trends rather than a direct domain comparison. In both settings, coverage continues to rise after WSC nearly plateaus.

Figure 2: HardShift441 readout accuracy by correct-answer multiplicity among 16 frozen samples. Values average nine generator-seed

\times

verifier-seed cells; error bars are pointwise 95% problem-identifier bootstrap intervals and labels give the per-cell count range. Absent bars denote zero accuracy;

c_{0}

is zero by construction because the correct answer is absent.

Figure 3: First identified errors among the 181 correct-answer and 195 incorrect-answer representatives labeled as refuted by the blinded process critic. Percentages are normalized separately for each group. Because the paired representatives were selected for audit, these distributions are conditional critic outputs rather than estimates of valid-trace prevalence.

Group	Intervention	Dataset / split	Metric(s)	Main result
Generation	Generator–budget trade-off	Legacy hard + E/M (reused)	WSC accuracy	Qwen3-8B $K=8$ vs. Qwen2.5-7B diversity-trained $K=16$ : $+7.07$ pp (hard); $+2.94$ pp (E/M)
	Increased sampling	Legacy hard + E/M (reused)	WSC accuracy	Qwen3-8B $K=16$ vs. $K=8$ : $+1.93$ pp (hard); $+1.76$ pp (E/M)
	Generator mixture	HardShift441 development (81 problems, reused)	pass@16; best evaluated readout accuracy	12 Qwen3-8B + 4 Qwen2.5-7B vs. 16 Qwen3-8B: pass@16 $80.25\%\!\to\!81.01\%$ ; accuracy $59.26\%\!\to\!57.88\%$
Selection	Trained pointwise verifier	Legacy hard	WSC accuracy	Majority 42.93%; verifier-based WSC 53.44% ( $+10.51$ pp)
	Qwen3-32B verifier	Legacy hard	WSC accuracy	Qwen2.5-7B verifier 53.44%; Qwen3-32B verifier 57.61% ( $+4.17$ pp)
	Groupwise rank training	HardShift441 development (81 problems) + legacy hard	WSC accuracy	HardShift441 development: $+4.93$ pp; legacy hard: $-9.24$ pp

Table 4: Exploratory intervention-screening results. Rates are percentages and differences are percentage points (pp). Each row reports an independent comparison on the indicated dataset and split; results should not be compared across rows.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Measuring Representation Robustness in Large Language Models for Geometry
Apr 3, 2026Vedant Jawandhia, Yash Sinha, Murari Mandal +2LLM Evaluation Representation Geometry in Language Models
Department of Computer Science and Information Systems, BITS Pilani · School of Computer Science, KIIT University · Department of Mathematics, BITS Pilani
GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams
Feb 25, 2026Yushun Zhang, Weiping Fu, Zesheng Yang +6Mathematical Reasoning Benchmarks LLM Mathematical Reasoning
School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, China · Shaanxi Province Key Laboratory of Big Data Knowledge Engineering, Xi’an, China
X-RAY: Mapping LLM Reasoning Capability via Formalized and Calibrated Probes
Mar 5, 2026Tianxi Gao, Yufan Cai, Yusi Yuan +1Structured Reasoning in Language Models Reasoning Evaluation
National University of Singapore, Singapore

Apr 3, 2026J/K move · Enter open · S save

Measuring Representation Robustness in Large Language Models for Geometry

Vedant Jawandhia, Yash Sinha, Murari Mandal, Ankan Pal+1

Department of Computer Science and Information Systems, BITS Pilani · School of Computer Science, KIIT University · Department of Mathematics, BITS Pilani

LLM Evaluation Representation Geometry in Language Models Invariant Representation Learning

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly evaluated on mathematical reasoning, yet their robustness to equivalent problem representations remains poorly understood. In geometry, identical problems can be expressed in Euclidean, coordinate, or vector forms, but existing benchmarks report accuracy on fixed formats, implicitly assuming representation invariance and masking failures caused by representational changes alone. We propose GeoRepEval, a representation-aware evaluation framework that measures correctness, invariance, and consistency at the problem level across parallel formulations, combining strict answer matching, bootstrap confidence intervals, paired McNemar tests, representation-flip analyses, and regression controls for surface complexity. We prove that our Invariance@3 metric decomposes accuracy into robust and fragile components and is bounded by the weakest representation. Evaluating eleven LLMs on 158 curated high-school geometry problems (474 instances), we find accuracy gaps of up to 14 percentage points induced solely by representation choice. Vector formulations emerge as a consistent failure point, with Invariance@3 as low as 0.044 even after controlling for length and symbolic complexity. A convert-then-solve prompting intervention improves vector accuracy by up to 52 percentage points for high-capacity models, suggesting that failures reflect representation sensitivity rather than inability; however, low-capacity models show no gains, indicating deeper limitations. These results suggest that current models rely on representation-specific heuristics rather than abstract geometric reasoning. All datasets, prompts, and scripts are released at https://github.com/vedjaw/GeoRepEval.

Abstract

LLM Evaluation

Explore similar work

Measuring Representation Robustness in Large Language Models for Geometry

GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagrams

X-RAY: Mapping LLM Reasoning Capability via Formalized and Calibrated Probes