VeruSAGE-Bench: A Benchmark Suite for Rust System Verification
Organizations: University of Illinois Urbana-Champaign · Carnegie Mellon University · Microsoft Research · University of Chicago
Abstract
Large language models (LLMs) have shown impressive capability to understand and develop code. However, their capability to rigorously reason about and formally prove the correctness of software systems remains in question. Current benchmarks for formal proof generation focus on isolated tasks, typically algorithmic-style problems utilizing basic data structures. We curate a new benchmark suite for system-level proof generation, VeruSAGE-Bench, which consists of 849 proof tasks extracted from eight open-source Verus-verified Rust systems, covering operating systems, storage systems, memory allocators, and more. These tasks are much more complicated than those in previous benchmarks, containing on average more than 20x the formal specification (in Lines of Code) and requiring a much broader array of verification techniques. We evaluate six LLMs on VeruSAGE-Bench (o4-mini, GPT-5, Sonnet 4, Sonnet 4.5, Opus 4.5, Qwen3.5), and demonstrate that different models and agentic setups lead to vastly different success rates and costs. We also show that the best models possess impressive capability in writing correctness proofs and hence should inspire new ways of developing trustworthy software. Meanwhile, some difficult tasks in VeruSAGE-Bench stress even the best model, forcing long reasoning time (over an hour on the hardest tasks, 15.3 min per task on average), high token cost (USD 13.65 per task on average), and proof failures, which we hope will guide future improvements in models and agents. This is particularly true for a harder no-lemma version of VeruSAGE-Bench, where the helper lemmas written by human experts are removed: there, the best model proves only 41% of the tasks from the hardest project, at USD 96 per task.
Figures & tables
| Proof | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| System | Abbr. | Description | #Tasks | Spec | Lemma | Target | Other | Total | License |
| Anvil Library | AL | Temporal-logic library | 104 | 29 | 10 | 9 | 36 | 84 | MIT |
| Anvil Controller | AC | A Kubernetes controller | 63 | 2037 | 33 | 75 | 1871 | 4016 | MIT |
| IronKV | IR | Sharded key-value store | 118 | 141 | 8 | 40 | 225 | 414 | MIT |
| Memory Allocator | MA | Mimalloc in Rust | 89 | 51 | 4 | 49 | 34 | 138 | MIT |
| Node Replication | NO | Replication library | 29 | 41 | 2 | 25 | 32 | 99 | MIT |
| Avg. LoC per task | Avg. proof LoC | Avg. count per task | ||||||
|---|---|---|---|---|---|---|---|---|
| Benchmark | #Tasks | Spec | Proof | Loop-inv | Non-loop | Loops | Helper lemmas | Total LoC |
| VerusBench [ 48 ] | 150 | 8 | 11 | 7 | 4 | 1.6 | 0.07 | 52 |
| HumanEval-Verus [ 34 ] | 74 | 25 | 29 | 7 | 22 | 1.3 | 0.96 | 166 |
| AlphaVerus translations [ 1 ] | 247 | 8 | 4 | 3 | 1 | 0.9 | 0.04 | 38 |
| VeruSAGE-Bench | 849 | 498 | 54 | 1 | 53 | 0.08 | 2.6 | 954 |
| Normal tasks (849) | No-lemma tasks (460) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| o4-mini | GPT-5 | Sonnet 4 | Sonnet 4.5 | Opus 4.5 | Sonnet 4.5 | Opus 4.5 | |||||
| Off | On | Off | On | Off | On | Off | On | Off | Off | Off | |
| Success rate | 17% | 41% | 50% | 56% | 64% | 58% | 81% | 67% | 94% | 57% | 87% |
| 95% CI ( pp) | 2.5 | 3.3 | 3.4 | 3.3 | 3.2 | 3.3 | 2.6 | 3.2 | 1.6 | 4.5 | 3.1 |
| Avg. time (min) | 3.3 | 12.8 | 3.7 | 12.7 | 9.2 | 7.1 | 7.2 | 6.9 | 15.3 | 10.9 | 30.8 |
| Avg. cost ($) | 0.8 | 0.7 | 1.3 | 0.5 | 6.7 | 1.7 | 5.6 | 1.5 | 13.7 | 10.8 | 52.0 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Success rate | Time (min) | Cost ($) | Verus runs | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Project | o4-mini | GPT-5 | S4 | S4.5 | o4-mini | GPT-5 | S4 | S4.5 | o4-mini | GPT-5 | S4 | S4.5 | o4-mini | GPT-5 | S4 | S4.5 |
| AL | 25% | 77% | 86% | 100% | 3.8 | 3.3 | 3.9 | 3.1 | 1.00 | 0.81 | 2.57 | 1.86 | 10.4 | 6.6 | 9.0 | 7.0 |
| AC | 11% | 16% | 24% | 37% | 2.1 | 2.7 | 14.5 | 12.5 | 0.57 | 1.96 | 15.80 | 12.31 | 2.6 | 4.8 | 17.0 | 13.6 |
| IR | 25% | 62% | 69% | 84% | 4.3 | 3.9 | 7.4 | 6.7 | 0.91 | 1.09 | 5.10 | 4.56 | 10.7 | 6.3 | 10.3 | 9.5 |
| MA | 15% | 60% | 75% | 90% | 4.9 | 3.2 | 8.3 | 5.2 | 0.85 | 1.14 | 6.00 | 3.81 | 9.8 | 8.2 | 11.6 | 9.6 |
| NO | 48% | 93% | 86% | 100% | 3.9 | 2.6 | 4.5 | 2.0 | 1.17 | 0.46 | 2.18 | 0.85 | 11.3 | 4.5 | 7.6 | 4.9 |
| Success rate | Time (min) | Cost ($) | Verus runs | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Project | o4-mini | GPT-5 | S4 | S4.5 | o4-mini | GPT-5 | S4 | S4.5 | o4-mini | GPT-5 | S4 | S4.5 | o4-mini | GPT-5 | S4 | S4.5 |
| AL | 48% | 79% | 69% | 83% | 12.4 | 8.7 | 6.7 | 4.6 | 0.45 | 0.31 | 0.60 | 0.51 | 32.9 | 10.0 | 25.4 | 20.3 |
| AC | 19% | 32% | 21% | 37% | 16.9 | 16.5 | 14.6 | 13.1 | 1.35 | 0.86 | 6.09 | 5.20 | 36.8 | 13.3 | 31.6 | 25.5 |
| IR | 35% | 44% | 53% | 67% | 13.8 | 14.1 | 7.5 | 7.1 | 0.54 | 0.48 | 1.03 | 0.92 | 45.7 | 17.0 | 35.3 | 28.5 |
| MA | 62% | 72% | 75% | 84% | 9.9 | 10.2 | 4.4 | 3.9 | 0.37 | 0.35 | 0.53 | 0.40 | 30.9 | 13.2 | 26.5 | 20.3 |
| NO | 72% | 83% | 100% | 97% | 9.2 | 7.6 | 1.2 | 1.6 | 0.29 | 0.25 | 0.19 | 0.18 | 31.0 | 9.4 | 25.1 | 19.4 |
| Project | #Tasks | Success rate | Time (min) | Cost ($) |
|---|---|---|---|---|
| AL | 104 | 100% | 1.3 | 1.86 |
| AC | 63 | 83% | 22.8 | 32.87 |
| IR | 118 | 96% | 7.6 | 9.78 |
| MA | 89 | 94% | 3.4 | 5.53 |
| NO | 29 | 100% | 1.3 | 2.21 |
| NR | 204 | 91% | 32.0 | 22.30 |
| AL | AC | IR | MA | NO | NR | OS | ST | VE | All | |
| Success rate | 30% | 6% | 24% | 32% | 34% | 15% | 13% | 19% | 23% | 20% |
| Success rate | Time (min) | Cost ($) | |||||
|---|---|---|---|---|---|---|---|
| Project | #Tasks | S4.5 | O4.5 | S4.5 | O4.5 | S4.5 | O4.5 |
| AL | 83 | 95% | 100% | 2.6 | 1.8 | 2.7 | 1.5 |
| AC | 58 | 19% | 41% | 10.7 | 45.9 | 11.5 | 96.3 |
| IR | 30 | 50% | 97% | 11.7 | 10.1 | 13.2 | 21.5 |
| MA | 47 | 72% | 100% | 9.9 | 25.2 | 11.8 | 24.4 |
| NO | 5 | 100% | 100% | 5.7 | 3.2 | 6.3 | 7.3 |
| AL | AC | IR | MA | NO | NR | OS | ST | VE | All | |
|---|---|---|---|---|---|---|---|---|---|---|
| #Tasks | 104 | 63 | 118 | 89 | 29 | 204 | 157 | 63 | 22 | 849 |
| Hands-Off | 40% | 11% | 34% | 31% | 38% | 20% | 19% | 8% | 32% | 25% |
| Hands-On | 59% | 19% | 46% | 67% | 83% | 38% | 41% | 27% | 59% | 45% |
| A | B | (A only) | (B only) | exact |
|---|---|---|---|---|
| o4-mini Off | GPT-5 Off | 23 | 304 | |
| GPT-5 Off | Sonnet 4 Off | 27 | 146 | |
| Sonnet 4 Off | Sonnet 4.5 Off | 6 | 149 | |
| Sonnet 4.5 Off | Opus 4.5 Off | 10 | 120 | |
| o4-mini Off | o4-mini On | 45 | 250 | |
| GPT-5 Off | GPT-5 On | 83 | 137 |
| AL | AC | IR | MA | NO | NR | OS | ST | VE | All | |
| Normal tasks no configuration proves | 0 | 9 | 3 | 2 | 0 | 14 | 8 | 1 | 0 | 37 |
| with helper-lemma stubs | 0 | 9 | 3 | 1 | 0 | 13 | 3 | 1 | 0 | 30 |
| without helper lemmas | 0 | 0 | 0 | 1 | 0 | 1 | 5 | 0 | 0 | 7 |
| No-lemma AC tasks | – | 58 | – | – | – | – | – | – | – | 58 |
| VeruSAGE-Bench-Hard | 0 | 67 | 3 | 2 | 0 | 14 | 8 | 1 | 0 | 95 |
| Project (task code) | GitHub repository | License |
|---|---|---|
| Anvil (AL, AC) | anvil-verifier/anvil | MIT |
| IronKV (IR) | verus-lang/verified-ironkv | MIT |
| Memory Allocator (MA) | verus-lang/verified-memory-allocator | MIT |
| Node Replication (NO) | verus-lang/verified-node-replication | MIT |
| NRKernel (NR) | matthias-brun/verified-nrkernel | MIT or Apache-2.0 |
| Atmosphere (OS) | mars-research/atmosphere | MIT |
Formal Verification
Formal verification uses mathematical models and machine-checked proofs to establish whether software and systems satisfy precise specifications.
See all 381 papers