cs.OSDec 20, 2025

VeruSAGE-Bench: A Benchmark Suite for Rust System Verification

Authors: Chenyuan Yang, Natalie Neamtu, Chris Hawblitzel, Jacob R. Lorch, Shan Lu

Organizations: University of Illinois Urbana-Champaign · Carnegie Mellon University · Microsoft Research · University of Chicago

Abstract

Large language models (LLMs) have shown impressive capability to understand and develop code. However, their capability to rigorously reason about and formally prove the correctness of software systems remains in question. Current benchmarks for formal proof generation focus on isolated tasks, typically algorithmic-style problems utilizing basic data structures. We curate a new benchmark suite for system-level proof generation, VeruSAGE-Bench, which consists of 849 proof tasks extracted from eight open-source Verus-verified Rust systems, covering operating systems, storage systems, memory allocators, and more. These tasks are much more complicated than those in previous benchmarks, containing on average more than 20x the formal specification (in Lines of Code) and requiring a much broader array of verification techniques. We evaluate six LLMs on VeruSAGE-Bench (o4-mini, GPT-5, Sonnet 4, Sonnet 4.5, Opus 4.5, Qwen3.5), and demonstrate that different models and agentic setups lead to vastly different success rates and costs. We also show that the best models possess impressive capability in writing correctness proofs and hence should inspire new ways of developing trustworthy software. Meanwhile, some difficult tasks in VeruSAGE-Bench stress even the best model, forcing long reasoning time (over an hour on the hardest tasks, 15.3 min per task on average), high token cost (USD 13.65 per task on average), and proof failures, which we hope will guide future improvements in models and agents. This is particularly true for a harder no-lemma version of VeruSAGE-Bench, where the helper lemmas written by human experts are removed: there, the best model proves only 41% of the tasks from the hardest project, at USD 96 per task.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SeVerus: Context-Aware, Gamified Benchmarking for Rust Verification

    Nov 4, 2026Álvaro F. Silva, Martim Afonso, Nuno Saavedra +3Coding Agent BenchmarksFormal Verification

  2. ExVerus: Verus Proof Repair via Counterexample Reasoning

    Mar 26, 2026Jun Yang, Yuechun Sun, Yi Wu +5Formal Verification