SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification
Authors: Álvaro F. Silva, Martim Afonso, Nuno Saavedra, Emily First, Alexandra Mendes, João F. Ferreira
Organizations: Faculty of Engineering, University of Porto, Porto, Portugal · INESC-ID & IST, University of Lisbon, Lisbon, Portugal · Rutgers University, New Jersey, USA · INESC-ID and Faculty of Engineering, University of Porto, Porto, Portugal
Coding agents achieve high success rates on existing benchmarks for Verus, a verifier for Rust programs. These benchmarks typically provide the auxiliary lemmas needed to complete a proof. In practice, however, such lemmas must be discovered and proved. To capture this challenge, we present SEVERUS, a context-aware technique to generate Verus benchmarks that organizes verification tasks into quests, each divided into levelsin a gamified manner. Using the original proof dependencies in verified projects, SEVERUS constructs one quest per verification target and progressively removes proof context to create individual levels. These levels range from the easiest, requiring completing only a single missing lemma, to the hardest, requiring proving the target without any supporting lemmas. An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. A noisy binary search estimates the highest level that an agent can solve reliably. Applied to six open-source Verus projects, SEVERUS generates 819 quests comprising 10,095 levels. Experiments with Codex using GPT-6 Sol and GPT-6 Luna, and with Claude Code using Sonnet 4.6, show that success decreases and cost increases as proof context is withdrawn. The strongest agent solves the final level of 52 of 60 sampled quests, yet can only reliably solve 35% of their levels, with failures concentrated in longer quests. Successful GPT-6 Sol proofs rarely reproduce the withheld lemmas, instead, using alternative proof decompositions. By systematically varying proof context, SEVERUS provides a foundation for constructing verification benchmarks that can continue to challenge increasingly capable coding agents.
Figures & tables
Fig. 1 : Simplified proof structure of is_sorted . The executable root R uses L1 and L2 , which both depend, directly or indirectly, on the shared lemma L3 . Ellipses mark omitted code, and contracts are shown schematically ( complete file ).
Fig. 2 : The quest of is_sorted : six of its 24 levels (left) and the four steps of expand() that generate it (right).
Full benchmark
Sample
Project
kLOC
Proofs
Quests
Levels
Length
Levels
Attempts
Vest [ 21 ]
10.9
221
147
544
3 / 11
68
50–96
IronKV [ 22 , 23 ]
10.9
91
96
356
3 / 13
82
54–105
PMemLog [ 23 ]
3.4
45
16
167
8.5 / 35
157
72–129
NRKernel [ 23 ]
26.6
289
82
3,953
5 / 214
968
100–180
TLA-RS [ 24 ]
94.6
1,447
427
3,935
3 / 237
528
86–159
TABLE I : The SeVerusBench-Strix benchmark. Full benchmark : Rust code size (kLOC), proof functions, quests (one per target), levels (one task each), and median/maximum quest length. Sample : total levels across the ten RQ2 quests per project, and minimum/maximum agent attempts required by noisy binary search.
Fig. 3 : Outcome of each attempt by relative level (0 = Level 1, 1 = root). Filled markers indicate success. Right-hand labels report successful/total attempts and Mann–Kendall p -values (n/v: no variance, as all attempts succeeded).
Highest solved
τb (level, cost)
Cost ($)
Project ( n )
Sonnet
Sol
Luna
Sonnet
Sol
Luna
Sonnet
Sol
Luna
Vest (11)
9 ∗
11
11
.72
.75
.49
36
6.3
0.3
IronKV (13)
11 ∗
13
13
.89
.79
.67
29
8.2
0.8
PMemLog (35)
29 ∗
35
35
.69
.62
.70
115
20
2.7
NRKernel (214)
7 ∗
64 ∗
57 ∗
–
.71
.82
50
74
16
TLA-RS (237)
22 ∗
218 ∗
223 ∗
1.00
.82
.83
54
67
18
TABLE II : RQ1 per quest of n levels. Bold: furthest agent; ∗ : significant success decrease; τb on solved attempts (–: under three). Pooled : stratified Z for success decrease and cost increase.
Root
Mean
RLS
Attempts
Cost ($)
Project
Sol
Luna
Sol
Luna
Sol
Luna
Sol
Luna
Sol
Luna
Vest
10
10
1.00
1.00
1.00
1.00
50
50
16
1.7
IronKV
10
10
1.00
1.00
1.00
1.00
55
55
32
6.4
PMemLog
10
10
1.00
1.00
1.00
1.00
72
72
26
4.9
NRKernel
5
4
.58
.55
.22
.22
113
116
372
91
TLA-RS
8
6
.89
.87
.67
.65
92
102
181
38
TABLE III : RQ2 results over the ten quests of each project. Root : quests solved up to the root. Mean : per-quest mean of ℓ∗/n . Attempts : agent attempts the search made.
Vest
Anvil
IronKV
TLA-RS
NRKernel
PMemLog
All
Tasks
23
19
13
56
34
55
200
Recall
0.73
0.11
0.10
0.02
0.01
0.00
0.11
Jaccard
0.56
0.06
0.05
0.01
0.00
0.00
0.08
TABLE IV : Reference deleted-lemma coverage on each project’s Hidden tasks. Re-stated recall measures one-sided coverage (ignoring extra agent helpers), while Hidden -only statement Jaccard measures symmetric overlap. GPT-6 Sol, RQ1+RQ2 pooled, one row per task.
Fig. 4 : Lemma reuse across projects.
Fig. 5 : Variation of proof similarity with the quest depth.
Figure 10
How it ends
S
L
Why the agent stops
S
L
Failed attempts
107
126
mid-plan, no stated reason
5
44
$5 budget
88
0
task impossible as stated
3
17
4-hour limit
0
25
work too hard, out of ideas
4
25
agent reports it incomplete
19
99
believes time or tokens run out
7
6
agent claims success
0
2
broke workspace, believes done
0
9
TABLE V : How failed attempts end (left) and, for those the agent ends itself, why it stops (right), from a reading of every one of these stops. S: GPT-6 Sol, L: GPT-6 Luna.
VeriContest is a benchmark of 1007 competitive-programming problems in Rust, each with a Verus specification, a judge-accepted solution, and a Verus proof. Its authors report that proof generation is the bottleneck for frontier models: given the specification and the code, the best model produces an accepted Verus proof for 13.95% of the problems on the first attempt. We report on solving the same proof-generation task with Rust-Prover, a verifier for Rust backed by Lean 4. The Verus specification and the Rust code are restated and translated into Lean, each specification becomes a theorem, and agents prove the theorems with Lean's kernel as the final check. All 1325 theorems of all 1007 problems were proved. 1259 of them were proved in one run of under 32 hours on Claude Opus 5.5, at a median of 3.2 minutes and $1.17 per proof, and 70% of them on the first iteration. The restated specifications were checked against the benchmark's test suites, and reviewed where no suite applies. None was wrong or weakened. The translated Lean programs were run on 21,413 of the benchmark's test cases and produced the same output as the Rust programs on every one. Across four Claude and four GPT models at five reasoning-effort settings, every current frontier model proves nearly all of a ten-theorem sample at every setting, and more effort raises the cost without raising the number of proofs. The cheapest Claude setting, Sonnet 5.5 at low effort, proves all of the 50 hardest theorems. We also rerun the benchmark's own Verus protocol with Claude Opus 5.5 on the 50 problems with the longest reference proofs. Opus 5.5 alone fails to prove one of them.
Large language models can generate useful code from natural language, but their outputs come without correctness guarantees. Verifiable code generation offers a path beyond testing by requiring models to produce not only executable code, but also formal specifications and machine-checkable proofs. Progress in this direction, however, is difficult to measure: existing benchmarks are often small, cover only part of the pipeline, lack ground-truth proofs or rigorous specification validation, or target verification settings far from mainstream software development. We present VeriContest, a benchmark of 1,006 competitive-programming problems from LeetCode and Codeforces for verifiable code generation in Rust with Verus. Each problem includes a natural language description, expert-validated formal specifications, judge-accepted Rust code, Verus-checked proofs, and positive and negative test suites. VeriContest is constructed through a three-phase pipeline that scales from manually verified seed problems to semi-automated expansion with human-in-the-loop review. To strengthen benchmark quality, we also use testing to validate postcondition completeness. VeriContest supports both isolated and compositional evaluation of specification, code, and proof generation, and end-to-end verified program synthesis. Evaluating ten state-of-the-art models reveals a sharp gap between ordinary coding and verifiable code generation: the strongest model reaches 97.51% on natural-language-to-code generation, but only 57.16% on specification generation, 13.82% on proof generation, and 6.76% end-to-end. These results identify proof and specification generation as the central bottlenecks for current models and establish VeriContest as a rigorous platform for evaluating and training future systems that generate code with machine-checkable correctness.
Zichen Xie, Mrigank Pawagi, Yuxin Liu +5
University of Virginia · Indian Institute of Science · Rice University +1
Concurrent software underpins critical infrastructure, including operating systems and databases, yet thread interactions can have bugs that bypass conventional testing. Verus extends Rust with formal verification capabilities, enabling developers to specify, implement and prove the correctness of concurrent code in the same language. Although coding agents have been explored for automated proof construction, existing Verus benchmarks offer limited coverage of concurrency-specific examples that connect atomic operations to abstract protocols. To address this gap, we introduce VerusWeave, the first benchmark dedicated to concurrent proof repair, derived from four open-source projects. It comprises 75 focused tasks and 17 full-module variants that exercise features used to verify low-level concurrent code. Tasks remove selected proof annotations and require agents to fill gaps, while preserving executable code and specifications. The evaluation harness validates repairs using Verus and a change checker, recording solutions, execution traces, and resource usage. An initial evaluation of four general-purpose coding-agent harnesses achieved 75-85% success on focused tasks and 29-47% on full-module variants, highlighting the challenge of autonomous module-wide repairs. VerusWeave provides a reusable foundation for evaluating automated proof construction for concurrent software.
Pedro Ribeiro, Martim Afonso, André Vasconcelos +1
INESC-ID and Instituto Superior Técnico, Universidade de Lisboa, Portugal · INESC-ID and Faculty of Engineering, University of Porto, Portugal