CyberBinGym: Can AI Agents Turn Binary Patches into Real Exploits?
Organizations: National University of Singapore · UC Riverside · UC Berkeley · UC Santa Barbara
Abstract
Security fixes usually reach users as compiled updates, but deployment delays leave unpatched systems exposed. During this window, comparing patched and unpatched binaries can reveal how to exploit a vulnerability, turning a defensive update into actionable information for attackers. Existing benchmarks do not measure this capability: they expose source code, rely on human-crafted challenges, or evaluate binary comprehension without requiring exploitation. We introduce CyberBinGym, a benchmark containing 100 tasks derived from real vulnerabilities in 53 widely used projects, spanning six vulnerability families and three levels of analysis complexity. Agents receive stripped vulnerable and patched binaries without source code or vulnerability information, and must combine static and dynamic analysis to recover and exploit the vulnerability. Evaluation separately measures vulnerability recovery through differential crash testing and end-to-end exploitation through flag retrieval, with trajectory audits to reject reward hacking. The strongest evaluated agent recovers the target vulnerability in 88% of tasks and exploits it in 20%. Providing the patched binary raises its exploit rate from 10% to 25% on a matched 20-task subset; with standard binary hardening enabled, it still exploits two of its 20 previously successful tasks. These results demonstrate that current agents can turn released binary patches into working exploits in a controlled setting, although exploitation remains unreliable. These capabilities lower the barriers to targeting unpatched software and compress defenders' effective response window, increasing the urgency of timely patch deployment.
Figures & tables
| Agent | Model | Recover Vuln. | Exploit | Cost ($) | Time (h) | LLM calls | Tool calls |
| Codex | GPT-5.6 | 88/100 (88.0%) | 20/100 (20.0%) | 28.11 | 2.13 | 286.2 | 283.4 |
| Claude Code | Claude Opus 5 | 85/100 (85.0%) | 6/100 (6.0%) | 14.90 | 1.26 | 115.8 | 131.2 |
| Kimi Code | Kimi K3 | 64/100 (64.0%) | 0/100 (0.0%) | 38.34 | 5.80 | 486.5 | 503.3 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Dimension | Score | Interpretation |
| Reachability | 0–2 | Uses sanitizer stack depth when available: shallow paths score low, moderate parser or control-flow paths score medium, and deep or indirect paths score high. Static call-graph evidence provides a lower-confidence fallback. |
| PoC construction | 0–1 | Scores direct malformed inputs as low and generated or coordinated artifacts as high. The current deterministic release caps this dimension at one point because heap-grooming, timing, and multi-stage exploitation are not separated into a harder track. |
| Observability | 0–1 | Scores tasks lower when the ordinary vulnerable binary shows a clear, reproducible crash. Tasks score higher when the failure requires a sanitizer, assertion, special diagnostic path, or oracle-only validation. |
| Binary complexity | 0–2 | Ranks candidates by static-analysis burden, using binary size, function counts, imports, indirect calls, call-graph structure, recursion, switches, and C++ indicators. The normalized release assigns low, medium, and high tiers relative to the candidate pool. |
| Patch complexity | 0–2 | Scores small local guards as low, multi-hunk or semantic fixes as medium, and broad fixes as high when they span several files, many hunks, many changed lines, or combined memory-management and type/API changes. |
| Metric | Median | Range |
| Vulnerable binary size (MiB) | 1.60 | 0.11–38.04 |
| Recovered functions | 2,794 | 151–17,829 |
| Recovered call-graph edges | 8,116 | 205–69,303 |
| Indirect call sites | 1,402 | 45–35,755 |
| Estimated call depth | 18 | 7–62 |
| Patch files changed | 30.5 | 1–12,162 |
| Vulnerability family | Construction pool | Target release | CVE reference |
| Spatial memory error | 55.3% | 40% | 68.0% |
| Temporal memory error | 10.1% | 20% | 15.2% |
| Invalid pointer access | 15.7% | 15% | 12.3% |
| Invalid deallocation | 2.3% | 10% | 1.7% |
| Initialization error | 15.6% | 9% | 1.0% |
| Type-safety error | 1.0% | 6% | 1.7% |
| Group | Agent | Recover Vuln. | Exploit | Cost ($) | Time (s) |
| Spatial memory | Codex | 37/40 (92.5%) | 4/40 (10.0%) | 23.05 | 6,308.9 |
| Claude | 36/40 (90.0%) | 3/40 (7.5%) | 13.51 | 4,025.8 | |
| Kimi | 25/40 (62.5%) | 0/40 (0.0%) | 31.70 | 19,258.6 | |
| Temporal memory | Codex | 18/20 (90.0%) | 6/20 (30.0%) | 29.53 | 8,507.4 |
| Claude | 17/20 (85.0%) | 1/20 (5.0%) | 15.50 | 4,917.3 | |
| Kimi | 15/20 (75.0%) | 0/20 (0.0%) | 30.87 | 20,452.5 |
| Group | Agent | Recover Vuln. | Exploit | Cost ($) | Time (s) |
| Low | Codex | 19/20 (95.0%) | 3/20 (15.0%) | 26.52 | 6,795.2 |
| Claude | 17/20 (85.0%) | 0/20 (0.0%) | 17.06 | 5,301.2 | |
| Kimi | 14/20 (70.0%) | 0/20 (0.0%) | 38.23 | 19,732.1 | |
| Medium | Codex | 35/40 (87.5%) | 6/40 (15.0%) | 26.67 | 6,849.0 |
| Claude | 34/40 (85.0%) | 2/40 (5.0%) | 13.90 | 4,198.3 | |
| Kimi | 25/40 (62.5%) | 0/40 (0.0%) | 42.35 | 22,473.9 |
| Agent | Binary analysis | Script/payload | Target exec. | File/artifact | Orchestration |
| Codex | 11,174 | 8,299 | 1,848 | 4,925 | 1,140 |
| Kimi | 21,904 | 12,102 | 5,532 | 7,487 | 2,872 |
| Claude | 4,364 | 3,641 | 1,539 | 2,977 | 293 |
| Category | Rule and representative command/tool triggers |
| Binary analysis | Calls containing analysis terms such as readelf , objdump , nm , r2 / radare2 , radiff2 , rabin2 , Ghidra, gdb , valgrind , strace , ltrace , addr2line , diff / cmp , or disassembly/decompilation/ELF keywords. |
| Script/payload | Commands beginning with an interpreter or build/scripting tool (e.g., python , bash , node , apply_patch ), or later matching script extensions and terms such as payload , exploit , PoC , fuzz , generator , or ROP . This includes both payload generators and programmatic analysis helpers. |
| Target execution | Remote interaction with the victim (e.g., nc / netcat addressing victim , victim:33333 , or the flag helper), or a timeout-guarded execution of an /out-vul or /out-fix target that is not an analysis command. |
| File/artifact | Remaining file and artifact operations, including cat , grep , head , tail , sed , awk , ls , find , cp , mv , mkdir , sha256sum , tar , base64 , file , xxd , and hexdump ; also Read and Grep tool calls. |
| Orchestration | Non-shell coordination tools, including update_plan , TodoList / TaskList , task/subagent controls, GetGoal , Skill , resource listings, crash validation, and wait . |
| General | Calls that match no earlier rule, including environment inventory commands such as command -v loops. This category is omitted from Table 7 . |
| Agent | Setting | Recover Vuln. | Exploit | Cost ($) | Time (s) |
| Codex | No description | 45/50 (90.0%) | 12/50 (24.0%) | 32.93 | 9,133.0 |
| Codex | With description | 43/50 (86.0%) | 10/50 (20.0%) | 34.33 | 7,932.2 |
| Kimi | No description | 32/50 (64.0%) | 0/50 (0.0%) | 30.90 | 17,854.9 |
| Kimi | With description | 35/50 (70.0%) | 1/50 (2.0%) | 32.49 | 17,646.0 |
| Agent | Setting | Recover Vuln. | Exploit | Cost ($) | Time (s) |
| Codex | No reasoning | 6/20 (30.0%) | 0/20 (0.0%) | 2.33 | 1,347.5 |
| Codex | Maximum reasoning | 18/20 (90.0%) | 5/20 (25.0%) | 31.42 | 8,705.4 |
| Agent | Setting | Recover Vuln. | Exploit | Cost ($) | Time (s) |
| Codex | Vulnerable only | 11/20 (55.0%) | 2/20 (10.0%) | 36.59 | 9,775.4 |
| Codex | Vulnerable + patched | 18/20 (90.0%) | 5/20 (25.0%) | 31.42 | 8,705.4 |
| Recover Vuln. | Exploit | Cost ($) | Time (s) | LLM calls | Tool calls | |
| 1 | 68.3% | 11.7% | 35.53 | 9,761.7 | 354.1 | 350.6 |
| 2 | 83.3% | 20.0% | 71.06 | 19,523.5 | 708.3 | 701.1 |
| 3 | 90.0% | 25.0% | 106.59 | 29,285.2 | 1,062.4 | 1,051.7 |
| Setting | Recover Vuln. | Exploit | Cost ($) | Time (h) | LLM calls |
| Non-hardened | 18/20 (90.0%) | 20/20 (100.0%) | 41.69 | 3.05 | 403.9 |
| Hardened | 15/20 (75.0%) | 2/20 (10.0%) | 34.50 | 2.49 | 334.9 |
| Agent | Direct | Indirect | Weak | Total |
| Codex | 13 | 7 | 0 | 20 |
| Claude | 4 | 0 | 2 | 6 |
| Total | 17 | 7 | 2 | 26 |