SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification
Authors: Álvaro F. Silva, Martim Afonso, Nuno Saavedra, Emily First, Alexandra Mendes, João F. Ferreira
Organizations: Faculty of Engineering, University of Porto, Porto, Portugal · INESC-ID & IST, University of Lisbon, Lisbon, Portugal · Rutgers University, New Jersey, USA · INESC-ID and Faculty of Engineering, University of Porto, Porto, Portugal
Coding agents achieve high success rates on existing benchmarks for Verus, a verifier for Rust programs. These benchmarks typically provide the auxiliary lemmas needed to complete a proof. In practice, however, such lemmas must be discovered and proved. To capture this challenge, we present SEVERUS, a context-aware technique to generate Verus benchmarks that organizes verification tasks into quests, each divided into levelsin a gamified manner. Using the original proof dependencies in verified projects, SEVERUS constructs one quest per verification target and progressively removes proof context to create individual levels. These levels range from the easiest, requiring completing only a single missing lemma, to the hardest, requiring proving the target without any supporting lemmas. An independent checker rejects changes to executable code, weakened specifications, and unsound proof constructs. A noisy binary search estimates the highest level that an agent can solve reliably. Applied to six open-source Verus projects, SEVERUS generates 819 quests comprising 10,095 levels. Experiments with Codex using GPT-6 Sol and GPT-6 Luna, and with Claude Code using Sonnet 4.6, show that success decreases and cost increases as proof context is withdrawn. The strongest agent solves the final level of 52 of 60 sampled quests, yet can only reliably solve 35% of their levels, with failures concentrated in longer quests. Successful GPT-6 Sol proofs rarely reproduce the withheld lemmas, instead, using alternative proof decompositions. By systematically varying proof context, SEVERUS provides a foundation for constructing verification benchmarks that can continue to challenge increasingly capable coding agents.
Figures & tables
Fig. 1 : Simplified proof structure of is_sorted . The executable root R uses L1 and L2 , which both depend, directly or indirectly, on the shared lemma L3 . Ellipses mark omitted code, and contracts are shown schematically ( complete file ).
Fig. 2 : The quest of is_sorted : six of its 24 levels (left) and the four steps of expand() that generate it (right).
Full benchmark
Sample
Project
kLOC
Proofs
Quests
Levels
Length
Levels
Attempts
Vest [ 21 ]
10.9
221
147
544
3 / 11
68
50–96
IronKV [ 22 , 23 ]
10.9
91
96
356
3 / 13
82
54–105
PMemLog [ 23 ]
3.4
45
16
167
8.5 / 35
157
72–129
NRKernel [ 23 ]
26.6
289
82
3,953
5 / 214
968
100–180
TLA-RS [ 24 ]
94.6
1,447
427
3,935
3 / 237
528
86–159
TABLE I : The SeVerusBench-Strix benchmark. Full benchmark : Rust code size (kLOC), proof functions, quests (one per target), levels (one task each), and median/maximum quest length. Sample : total levels across the ten RQ2 quests per project, and minimum/maximum agent attempts required by noisy binary search.
Fig. 3 : Outcome of each attempt by relative level (0 = Level 1, 1 = root). Filled markers indicate success. Right-hand labels report successful/total attempts and Mann–Kendall p -values (n/v: no variance, as all attempts succeeded).
Highest solved
τb (level, cost)
Cost ($)
Project ( n )
Sonnet
Sol
Luna
Sonnet
Sol
Luna
Sonnet
Sol
Luna
Vest (11)
9 ∗
11
11
.72
.75
.49
36
6.3
0.3
IronKV (13)
11 ∗
13
13
.89
.79
.67
29
8.2
0.8
PMemLog (35)
29 ∗
35
35
.69
.62
.70
115
20
2.7
NRKernel (214)
7 ∗
64 ∗
57 ∗
–
.71
.82
50
74
16
TLA-RS (237)
22 ∗
218 ∗
223 ∗
1.00
.82
.83
54
67
18
TABLE II : RQ1 per quest of n levels. Bold: furthest agent; ∗ : significant success decrease; τb on solved attempts (–: under three). Pooled : stratified Z for success decrease and cost increase.
Root
Mean
RLS
Attempts
Cost ($)
Project
Sol
Luna
Sol
Luna
Sol
Luna
Sol
Luna
Sol
Luna
Vest
10
10
1.00
1.00
1.00
1.00
50
50
16
1.7
IronKV
10
10
1.00
1.00
1.00
1.00
55
55
32
6.4
PMemLog
10
10
1.00
1.00
1.00
1.00
72
72
26
4.9
NRKernel
5
4
.58
.55
.22
.22
113
116
372
91
TLA-RS
8
6
.89
.87
.67
.65
92
102
181
38
TABLE III : RQ2 results over the ten quests of each project. Root : quests solved up to the root. Mean : per-quest mean of ℓ∗/n . Attempts : agent attempts the search made.
Vest
Anvil
IronKV
TLA-RS
NRKernel
PMemLog
All
Tasks
23
19
13
56
34
55
200
Recall
0.73
0.11
0.10
0.02
0.01
0.00
0.11
Jaccard
0.56
0.06
0.05
0.01
0.00
0.00
0.08
TABLE IV : Reference deleted-lemma coverage on each project’s Hidden tasks. Re-stated recall measures one-sided coverage (ignoring extra agent helpers), while Hidden -only statement Jaccard measures symmetric overlap. GPT-6 Sol, RQ1+RQ2 pooled, one row per task.
Fig. 4 : Lemma reuse across projects.
Fig. 5 : Variation of proof similarity with the quest depth.
Figure 10
How it ends
S
L
Why the agent stops
S
L
Failed attempts
107
126
mid-plan, no stated reason
5
44
$5 budget
88
0
task impossible as stated
3
17
4-hour limit
0
25
work too hard, out of ideas
4
25
agent reports it incomplete
19
99
believes time or tokens run out
7
6
agent claims success
0
2
broke workspace, believes done
0
9
TABLE V : How failed attempts end (left) and, for those the agent ends itself, why it stops (right), from a reading of every one of these stops. S: GPT-6 Sol, L: GPT-6 Luna.