When Is Proof-Step Verification Worth Its Cost? Operating Requirements of Certify-or-Decline Verification for Open-Weight Mathematical Reasoning
Organizations: Independent Researcher
Abstract
Proof-step verification rewrites a solver's answer into typed steps and uses LLM judges to certify or decline. We evaluate this pipeline on open-weight models across GPQA, AIME-2025, and a MATH-500 subset. Certification raises answer-key-match precision by 31 percentage points on the strong GPQA configuration and 46 on AIME, while answering only 35% and 21% of attempts. On near-ceiling MATH-500, gains are at most 5 points at a loss of 13-27 points of coverage. An output-token-matched self-consistency selector obtains 82.9% precision at 35% coverage, compared with the verifier's pooled 87.6% (paired difference: +4.8 points, 95% interval [-8.2, 18.5]). A diagnostic sample reveals a distinct failure mode: 52 of 88 completed problems end in invalid proof construction without ever reaching a judge. Finally, external-LLM acceptance of rendered certified chains is 61.0-67.6%, compared with 87.6% answer-key match. These audits lack human or formal-kernel ground truth. Open-weight auditor pairs show near-zero chance-corrected agreement, which measures consistency rather than accuracy against proof truth. These results motivate separate reporting of execution failures, selected-answer precision, and independently assessed proof validity. We provide per-configuration results and evaluation details in the supplement.
Figures & tables
| Benchmark | Configuration | No verif. | Coverage | Precision [95% CI] | Gain |
|---|---|---|---|---|---|
| GPQA | strong | 56.3% | 35.0% | 87.6% [77.7, 95.8] | +31.3 |
| GPQA | budget | 60.5% | 30.5% | 75.4% [64.1, 85.5] | +14.9 |
| GPQA | 32B-FP8 | 58.7% | 45.3% | 72.8% [63.3, 81.6] | +14.1 |
| GPQA | Phi-4 | 44.3% | 8.0% | 75.0% [52.6, 91.7] | +30.7 |
| AIME-2025 | strong | 48.9% | 21.1% | 94.7% [77.8, 100.0] | +45.8 |
| AIME-2025 | budget | 51.1% | 26.7% | 79.2% [56.2, 95.8] | +28.1 |
| Judge pair | Overlap | Cohen’s |
|---|---|---|
| Proprietary (Claude-family + GPT-family) | 57 | 0.573 |
| gpt-oss-20b + Qwen3-14B | 105 | 0.038 |
| Qwen3-32B + gpt-oss-20b | 105 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Arm | Coverage [95% CI] | Precision [95% CI] |
|---|---|---|
| GPQA strong | 35.0% [27.0, 43.3] | 87.6% [77.7, 95.8] |
| GPQA budget | 30.5% [24.5, 36.5] | 75.4% [64.1, 85.5] |
| GPQA fswap | 30.0% [23.3, 37.0] | 90.0% [81.9, 96.2] |
| GPQA 32B | 45.3% [38.7, 52.0] | 72.8% [63.3, 81.6] |
| GPQA Phi-4 | 8.0% [4.3, 12.0] | 75.0% [52.6, 91.7] |
| GPQA effort=high | 9.0% [4.0, 15.0] | 88.9% [62.5, 100.0] |
| Arm | Per-run precision | Spread | |||
|---|---|---|---|---|---|
| GPQA strong | 3 | 100 | 105 | 84.8 / 88.6 / 89.2 | 4.4 |
| GPQA budget | 4 | 100 | 122 | 71.0 / 82.8 / 67.9 / 79.4 | 14.9 |
| GPQA fswap | 3 | 100 | 90 | 84.8 / 93.1 / 92.9 | 8.3 |
| GPQA 32B | 3 | 100 | 136 | 82.0 / 65.9 / 68.9 | 16.1 |
| GPQA Phi-4 | 3 | 100 | 24 | 63.6 / 83.3 / 85.7 | 22.1 |
| GPQA effort=high | 1 | 100 | 9 | 88.9 | – |
| Benchmark | Arm | Coverage | Precision | ||
|---|---|---|---|---|---|
| GPQA | ctl | 4 | 400 | 14.5% | 81.0% |
| GPQA | fswap+budget | 3 | 300 | 36.0% | 86.1% |
| GPQA | crossjudge | 3 | 300 | 19.3% | 74.1% |
| GPQA | minus-completeness | 3 | 300 | 35.7% | 72.0% |
| GPQA | effort=low | 3 | 300 | 11.0% | 81.8% |
| MATH-500 | strong | 3 | 300 | 86.7% | 98.5% |
| Acceptance type | Key matches / certified | Coverage | Precision [95% CI] |
|---|---|---|---|
| Unconditional | 52 / 54 | 18.0% | 96.3% [90.6, 100.0] |
| Under recorded conventions | 40 / 51 | 17.0% | 78.4% [62.5, 92.7] |
| Pooled | 92 / 105 | 35.0% | 87.6% [77.7, 95.8] |
| Arm | Attempts | Answered | Declined | Execution error |
|---|---|---|---|---|
| GPQA strong | 300 | 105 | 83 | 112 (37%) |
| GPQA budget | 400 | 122 | 182 | 96 (24%) |
| GPQA fswap | 300 | 90 | 111 | 99 (33%) |
| GPQA 32B | 300 | 136 | 61 | 103 (34%) |
| GPQA Phi-4 | 300 | 24 | 241 | 35 (12%) |
| GPQA effort=high | 100 | 9 | 10 | 81 (81%) |
| Vote denominator | Most selective | At 35% | Maximum coverage | |
|---|---|---|---|---|
| 8 | Answering draws | 57.0 / 82.5 | – | 99.0 / 63.6 |
| 8 | All draws | 35.0 / 82.9 | 82.9 (29/35) | 99.0 / 63.6 |
| 19 | Answering draws | 47.0 / 85.1 | – | 100.0 / 65.0 |
| 19 | All draws | 28.0 / 82.1 | – | 100.0 / 65.0 |
| 30 | Answering draws | 41.0 / 92.7 | – | 100.0 / 63.0 |
| 30 | All draws | 21.0 / 90.5 | 82.9 (29/35) | 100.0 / 63.0 |
| Total draws | Parsed | Unparsed | Of these, empty | |
|---|---|---|---|---|
| 8 | 800 | 693 | 107 | 98 |
| 19 | 1,900 | 1,635 | 265 | 229 |
| 30 | 3,000 | 2,576 | 424 | 358 |
| Error class | Issues | Share |
|---|---|---|
| Hidden premise | 2020 | 44.4% |
| Misapplied theorem | 1117 | 24.6% |
| Fabricated citation | 678 | 14.9% |
| Arithmetic error | 510 | 11.2% |
| Other | 163 | 3.6% |
| Circular reasoning | 30 | 0.7% |
| Rule | Accepted | Rate | Definition |
|---|---|---|---|
| Codex only | 71 / 105 | 67.6% | Use the auditor with full coverage. |
| Claude-else-Codex | 68 / 105 | 64.8% | Use Claude where available, otherwise Codex. |
| Neither auditor rejects | 64 / 105 | 61.0% | Retain rows unless an available auditor rejects them. |
| Both auditors VALID | 32 / 57 | 56.1% | Require both auditors to accept, on the overlap only. |
| Benchmark | Cases | Accuracy | False positives | False negatives |
|---|---|---|---|---|
| GPQA | 200 | 100.0% | 0 | 0 |
| AIME-2025 | 159 | 98.7% | 2 | 0 |
| MATH-500 | 114 | 99.1% | 1 | 0 |
| Step | Type | Assertion added or final state | Recorded justification, paraphrased |
|---|---|---|---|
| 1 | C | : only finitely many primes exist. | Open a proof by contradiction. |
| 2 | C | is prime; shorten ’s label. | Definition of a prime. |
| 3 | C | list all primes, . | A finite nonempty set can be enumerated. |
| 4 | C | Each . | Every listed number is prime. |
| 5 | M | . | Introduce a fresh integer by definition. |
| 6 | M | . | The nonempty product is at least . |
Reasoning Verification
Reasoning verification checks whether intermediate steps and final answers are logically valid and supported by evidence.
See all 62 papers