Proof-step verification rewrites a solver's answer into typed steps and uses LLM judges to certify or decline. We evaluate this pipeline on open-weight models across GPQA, AIME-2025, and a MATH-500 subset. Certification raises answer-key-match precision by 31 percentage points on the strong GPQA configuration and 46 on AIME, while answering only 35% and 21% of attempts. On near-ceiling MATH-500, gains are at most 5 points at a loss of 13-27 points of coverage. An output-token-matched self-consistency selector obtains 82.9% precision at 35% coverage, compared with the verifier's pooled 87.6% (paired difference: +4.8 points, 95% interval [-8.2, 18.5]). A diagnostic sample reveals a distinct failure mode: 52 of 88 completed problems end in invalid proof construction without ever reaching a judge. Finally, external-LLM acceptance of rendered certified chains is 61.0-67.6%, compared with 87.6% answer-key match. These audits lack human or formal-kernel ground truth. Open-weight auditor pairs show near-zero chance-corrected agreement, which measures consistency rather than accuracy against proof truth. These results motivate separate reporting of execution failures, selected-answer precision, and independently assessed proof validity. We provide per-configuration results and evaluation details in the supplement.
Figures & tables
Figure 1: (a) Precision gain versus initial-answer accuracy for Table 1 . (b) GPQA verifier runs at 33%, 35%, and 37% coverage; self-consistency (SC) k=30 with all draws in the agreement denominator (gray, low-coverage region); output-token-matched k=8 with a 95% Wilson interval (blue); and k=30 with answering draws in the denominator (green). Lines connect measured thresholds as visual guides.
Benchmark
Configuration
No verif.
Coverage
Precision [95% CI]
Gain
GPQA
strong
56.3%
35.0%
87.6% [77.7, 95.8]
+31.3
GPQA
budget
60.5%
30.5%
75.4% [64.1, 85.5]
+14.9
GPQA
32B-FP8
58.7%
45.3%
72.8% [63.3, 81.6]
+14.1
GPQA
Phi-4
44.3%
8.0%
75.0% [52.6, 91.7]
+30.7
AIME-2025
strong
48.9%
21.1%
94.7% [77.8, 100.0]
+45.8
AIME-2025
budget
51.1%
26.7%
79.2% [56.2, 95.8]
+28.1
Table 1: Selected-answer precision and coverage. “No verif.” is initial-answer accuracy over all problems; gain subtracts that baseline from certified precision. CIs use problem-cluster bootstrap. Small AIME samples and broad intervals preclude reliable configuration rankings. The GPQA budget baseline uses two graded runs (200 attempts), whereas verification pools four runs (400 attempts).
Judge pair
Overlap
Cohen’s κ
Proprietary (Claude-family + GPT-family)
57
0.573
gpt-oss-20b + Qwen3-14B
105
0.038
Qwen3-32B + gpt-oss-20b
105
−0.005
Table 2: Inter-auditor agreement on certified GPQA chains. Overlap sizes differ. These are consistency statistics, not accuracy against independently adjudicated proof validity.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Arm
Coverage [95% CI]
Precision [95% CI]
GPQA strong
35.0% [27.0, 43.3]
87.6% [77.7, 95.8]
GPQA budget
30.5% [24.5, 36.5]
75.4% [64.1, 85.5]
GPQA fswap
30.0% [23.3, 37.0]
90.0% [81.9, 96.2]
GPQA 32B
45.3% [38.7, 52.0]
72.8% [63.3, 81.6]
GPQA Phi-4
8.0% [4.3, 12.0]
75.0% [52.6, 91.7]
GPQA effort=high
9.0% [4.0, 15.0]
88.9% [62.5, 100.0]
Appendix
Table S1: Per-arm coverage and certified precision, with 95% cluster-bootstrap intervals over problems.
Arm
k
P
C
Per-run precision
Spread
GPQA strong
3
100
105
84.8 / 88.6 / 89.2
4.4
GPQA budget
4
100
122
71.0 / 82.8 / 67.9 / 79.4
14.9
GPQA fswap
3
100
90
84.8 / 93.1 / 92.9
8.3
GPQA 32B
3
100
136
82.0 / 65.9 / 68.9
16.1
GPQA Phi-4
3
100
24
63.6 / 83.3 / 85.7
22.1
GPQA effort=high
1
100
9
88.9
–
Appendix
Table S2: Sample counts and variation across runs. P counts problems and C counts certified problem–run rows. Per-run precision is in percent; spread is the reported range in percentage points. The effort=high arm has one run.
Benchmark
Arm
k
N
Coverage
Precision
GPQA
ctl
4
400
14.5%
81.0%
GPQA
fswap+budget
3
300
36.0%
86.1%
GPQA
crossjudge
3
300
19.3%
74.1%
GPQA
minus-completeness
3
300
35.7%
72.0%
GPQA
effort=low
3
300
11.0%
81.8%
MATH-500
strong
3
300
86.7%
98.5%
Appendix
Table S3: Additional arms reported without intervals. N is the number of problem–run attempts.
Acceptance type
Key matches / certified
Coverage
Precision [95% CI]
Unconditional
52 / 54
18.0%
96.3% [90.6, 100.0]
Under recorded conventions
40 / 51
17.0%
78.4% [62.5, 92.7]
Pooled
92 / 105
35.0%
87.6% [77.7, 95.8]
Appendix
Table S4: GPQA strong certification types. Coverage uses all 300 attempts; precision uses the certified rows of the indicated type.
Arm
Attempts
Answered
Declined
Execution error
GPQA strong
300
105
83
112 (37%)
GPQA budget
400
122
182
96 (24%)
GPQA fswap
300
90
111
99 (33%)
GPQA 32B
300
136
61
103 (34%)
GPQA Phi-4
300
24
241
35 (12%)
GPQA effort=high
100
9
10
81 (81%)
Appendix
Table S5: Coverage decomposed by recorded termination label. Execution-error percentages use all attempts as the denominator.
k
Vote denominator
Most selective
At 35%
Maximum coverage
8
Answering draws
57.0 / 82.5
–
99.0 / 63.6
8
All draws
35.0 / 82.9
82.9 (29/35)
99.0 / 63.6
19
Answering draws
47.0 / 85.1
–
100.0 / 65.0
19
All draws
28.0 / 82.1
–
100.0 / 65.0
30
Answering draws
41.0 / 92.7
–
100.0 / 63.0
30
All draws
21.0 / 90.5
82.9 (29/35)
100.0 / 63.0
Appendix
Table S6: Observed threshold operating points. Pairs are coverage / precision, in percent. The fourth column gives precision at exactly 35% coverage; “–” means no reported threshold reaches that coverage.
k
Total draws
Parsed
Unparsed
Of these, empty
8
800
693
107
98
19
1,900
1,635
265
229
30
3,000
2,576
424
358
Appendix
Table S7: Draw parsing. Empty completions account for most unparsed draws.
Error class
Issues
Share
Hidden premise
2020
44.4%
Misapplied theorem
1117
24.6%
Fabricated citation
678
14.9%
Arithmetic error
510
11.2%
Other
163
3.6%
Circular reasoning
30
0.7%
Appendix
Table S8: Assigned issue labels across 1,878 rejections and 4,547 issues from 67 run files. Multiple issues can be associated with one rejection.
Rule
Accepted
Rate
Definition
Codex only
71 / 105
67.6%
Use the auditor with full coverage.
Claude-else-Codex
68 / 105
64.8%
Use Claude where available, otherwise Codex.
Neither auditor rejects
64 / 105
61.0%
Retain rows unless an available auditor rejects them.
Both auditors VALID
32 / 57
56.1%
Require both auditors to accept, on the overlap only.
Appendix
Table S9: Auditor acceptance by aggregation rule. The denominator is explicit because the final row uses a different cohort.
Benchmark
Cases
Accuracy
False positives
False negatives
GPQA
200
100.0%
0
0
AIME-2025
159
98.7%
2
0
MATH-500
114
99.1%
1
0
Appendix
Table S10: Answer-grader checks. False positives accept incorrect or empty answers; false negatives reject correct answers. No calls failed.
Step
Type
Assertion added or final state
Recorded justification, paraphrased
1
C
H : only finitely many primes exist.
Open a proof by contradiction.
2
C
2 is prime; shorten H ’s label.
Definition of a prime.
3
C
p1,…,pn list all primes, n≥1 .
A finite nonempty set can be enumerated.
4
C
Each pi≥2 .
Every listed number is prime.
5
M
N:=p1⋯pn+1 .
Introduce a fresh integer by definition.
6
M
N>1 .
The nonempty product is at least 2 .
Appendix
Table S11: All sixteen transitions of the recorded Euclid example. C denotes citation and M computation; these types select LLM judge roles.