LeanPolish: Verified Supervision for Lean Proof Compression
Organizations: Imperial College London
Abstract
Verified proof edits offer a natural source of supervision for improving language-model-generated Lean proofs. Yet verification establishes that an edit is correct, not that its training signal is free of search artifacts. We introduce LeanPolish, a symbolic Lean 4 pipeline that releases 33,402 accepted local edits and 65,596 same-state failed attempts, and use it to study what models learn from this supervision. First-success search admits a goal-independent rule with perfect ranking accuracy; teacher-selected evaluation sites also reward trivial deletions. Continuing menu evaluation beyond the first success removes the ordering shortcut: a trained ranker selects the best candidate on 70.1% of evaluated held-out states, versus 36.9% for the strongest frozen baseline. For compression, iterating the symbolic pass raises miniF2F savings from 19.7% to 27.5%, exceeding the neural hybrids we test there. Verified neural editing helps on other proof sources, but matched frozen-model controls show that its gains need not come from training. The supervision does improve whole-proof rewriting: fine-tuning raises verified token reduction from 2.8% to 5.5% on 19 PutnamBench proofs. Together, the released edits, complete candidate pools, and controlled evaluations separate learning to imitate a search policy from improving on that search. They provide a reproducible basis for studying proof improvement while keeping correctness, compression, and edit policy distinct.
Figures & tables
| Source (generator) | Inputs | Files | Accepted | Failed siblings | Tok. red. (%) |
|---|---|---|---|---|---|
| Mathlib v4.21.0 subset (human) | 5,789 | 2,233 | 6,695 | 26,912 | 0.27 |
| Goedel-Workbook (Goedel-Prover-V2) | 29,750 | 10,052 | 20,822 | 28,525 | 5.48 |
| PutnamBench sample (Goedel-Prover-V2) | 437 | 352 | 4,354 | 5,930 | 8.14 |
| miniF2F verified (Goedel-Prover-V2) | 351 | 308 | 1,184 | 3,753 | 19.72 |
| PutnamBench verified (Goedel-Prover-V2) | 19 | 16 | 80 | 254 | 6.14 |
| Putnam 2025 (AxiomProver) | 12 | 10 | 142 / 125 | 147 / 75 | 1.31 |
| miniF2F | PutnamBench-verified | Putnam 2025 (AxiomProver) | |||||||||
| Method | All | Tac. | Tok. | All | Tac. | Tok. | All | Tac. | Tok. | ||
| Reference (pipeline) | 94.6 | 91.6 | 27,622 | 88.8 | 79.1 | 1,359 | 84.5 | 61.4 | 1,746 | ||
| Delete rule (no model) | 36.8 | 0.0 | 8,122 | 46.3 | 0.0 | 975 | 59.9 | 0.0 | 1,500 | ||
| DeepSeek, frozen | 0.1 | 0.1 | 28 | 0.0 | 0.0 | 0 | 0.0 | 0.0 | 0 | ||
| Qwen, frozen | 0.5 | 0.8 | 189 | 0.0 | 0.0 | 0 | 1.4 | 0.0 | 32 | ||
| DeepSeek, 4-shot | 36.9 | 4.7 | 7,539 | 48.8 | 4.7 | 979 | 53.5 | 0.0 | 1,350 | ||
| Method | Top-1 |
| Last in menu order ∗ | 0.0 |
| First in menu order | 11.6 |
| Random | 8.2 |
| Shortest string | 5.8 |
| Per-tactic prior | 22.6 |
| Frozen Qwen-32B log-prob | 28.3 |
| Method | AxiomProver (12) | PutnamBench (19) | miniF2F-99 |
|---|---|---|---|
| Symbolic only | |||
| LeanPolish , release run | 1.31 | 6.14 | 20.02 |
| LeanPolish , clean rerun | 1.41 | 11.91 | 20.52 |
| LeanPolish (clean) iterated to a fixed point | 1.76 | 16.07 | 29.76 |
| Neural only | |||
| LLM whole-proof rewrite, frozen | 1.04 a | 2.76 | 12.52 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Input proofs (files shortened / total) | LeanPolish | unusedTactic linter | ProofOptimizer (ref.) |
|---|---|---|---|
| miniF2F, Goedel-Prover-V2 (308/351) | 19.7 | 0.002 | 87.9 |
| PutnamBench, Goedel-Prover-V2 (16/19) | 6.1 | 0.000 | 57.2 |
| Goedel-Workbook | 5.48 | 0.010 | — |
| Mathlib v4.21.0 subset | 0.27 | 0.000 | — |
| Putnam 2025, AxiomProver | 1.31 | 0.000 | — |
| Source | Density | Tok/edit |
|---|---|---|
| Mathlib v4.21.0 subset | 0.46 | 5.9 |
| Goedel-Workbook | 4.72 | 11.6 |
| PutnamBench sample | 9.10 | 8.9 |
| miniF2F verified | 8.45 | 23.3 |
| PutnamBench verified | 3.62 | 17.0 |
| Putnam 2025 (AxiomProver) | 1.06 | 12.3 |
| Goedel-Workbook | Mathlib v4.21.0 | |
|---|---|---|
| Files sampled | 1,500 | 250 |
| Candidate groups discovered | 86 | 49 |
| Reached the dependency filter | 16 | 1 |
| Rejected by dependency filter | 6 | 0 |
| Applied | 10 | 1 |
| LeanPolish | linter.unusedTactic | |||||
|---|---|---|---|---|---|---|
| Corpus | B | T | L | B | T | L |
| miniF2F (Goedel-V2 verified) | 19.83 | 19.72 | 21.59 | — | 0.002 | — |
| PutnamBench (Goedel-V2 verified) | 6.22 | 6.14 | 7.29 | — | 0.000 | — |
| Mathlib v4.21.0 | 0.24 | 0.27 | 0.18 | 0.000 | 0.000 | 0.000 |
| Goedel-Workbook | 10.66 | 5.48 | 12.05 | 0.018 | 0.010 | 0.000 |
| PutnamBench (Goedel sample) | 8.82 | 8.14 | 11.39 | 0.013 | 0.007 | 0.000 |
| Configuration | Shorter | B | T | L | Gen | Filt |
| (a) random slice | ||||||
| Full pipeline | 167 | 11.75 | 6.58 | 9.90 | 2 | 3 |
| – tactic replacement | 108 | 1.94 | 3.86 | 6.31 | 2 | 3 |
| – generalization | 166 | 11.74 | 6.61 | 9.88 | 0 | 0 |
| – unused-fact removal | 151 | 11.31 | 5.40 | 8.66 | 2 | 3 |
| – cleanup | 131 | 11.06 | 5.34 | 8.09 | 2 | 3 |
| Problem | Seed-Prover 1.5 | AxiomProver |
|---|---|---|
| A1 | 371 / 9,105 (4.1%) | 118 / 7,067 (1.7%) |
| A2 | 0 / 6,115 (0.0%) | 72 / 5,385 (1.3%) |
| B1 | 0 / 12,039 (0.0%) | 115 / 15,590 (0.7%) |
| B2 | 1,702 / 27,478 (6.2%) | 64 / 5,502 (1.2%) |
| B3 | 0 / 6,063 (0.0%) | 76 / 3,552 (2.1%) |
| B4 | 230 / 8,503 (2.7%) | 393 / 13,360 (2.9%) |
| Train held-out | Train | Held-out | Jaccard | |
|---|---|---|---|---|
| Goedel miniF2F | 5,193 | 696 | 9 | 0.0015 |
| Goedel PutnamBench-verified | 5,193 | 38 | 1 | 0.0002 |
| Goedel Putnam 2025 | 5,193 | 55 | 1 | 0.0002 |
| Mathlib miniF2F | 6,490 | 696 | 2 | 0.0003 |
| Mathlib PutnamBench-verified | 6,490 | 38 | 1 | 0.0002 |
| Mathlib Putnam 2025 | 6,490 | 55 | 2 | 0.0003 |
| Candidate | Outcome | Note |
|---|---|---|
| rfl , ring , abel , norm_num , norm_cast , positivity , decide | kernel failure | tried and failed: the only negatives a first-success release stores |
| linarith | valid, not chosen | first-success winner (8 characters) |
| omega | chosen | shortest valid candidate (5 characters) |
| field_simp , contradiction , ext , simp? | kernel failure | never tried by first-success search |
| gcongr , tauto | valid, not chosen | never tried by first-success search |