Stopping Is Not the Limit: Understanding and Eliciting Agent Capability in Long-Horizon Vulnerability Exploitation
Organizations: Safe AI Research Center, UNIST, Republic of Korea · Department of Computer Science and Engineering, POSTECH, Republic of Korea · Graduate School of Artificial Intelligence, POSTECH, Republic of Korea
Abstract
Vulnerability exploitation tests whether language model agents can develop working exploits from known software flaws. Sequential test-time scaling can improve exploitation performance, but larger budgets do not ensure sustained exploration: agents often stop with budget remaining. Yet what happens when execution continues beyond stopping remains underexplored. We investigate this gap on ExploitBench using DeepSeek-V4-Flash-0731 and MiMo-V2.6-Pro. We find additional verified gains after the first stop response, but gains diminish over longer execution. We identify two patterns associated with less productive continuation: an increasing proportion of stop responses within the same context and earlier stopping associated with summaries retaining impossibility judgments about unachieved capabilities. To test whether context management can elicit further capability, we propose SCoRe, which compacts context at stop bursts and marks these judgments as unproven. SCoRe improves verified capability over summary-based continuation and other evaluated context-management baselines. With SCoRe, MiMo reaches a capability coverage score of 67.0% at 2,000 turns, exceeding both 32.4% under the original stopping policy and the 58.6% computed for GPT-5.5 from ExploitBench's 300-turn reference results on the same tasks. These findings show that stopping need not mark an agent's exploitation limit, and that continued execution with context management can elicit substantial further capability.
Figures & tables
| Method | Cap. count | T2+ (%) | Cov@3 % |
|---|---|---|---|
| DeepSeek-V4-Flash-0731 | |||
| ReAct | 3.51 1.78 | 0.0 | 25.3 |
| ReSum | 5.95 2.90 | 15.9 | 49.1 |
| Forgetting | 5.17 2.49 | 6.3 | 40.2 |
| Context folding | 5.89 2.90 | 14.3 | 47.3 |
| ACM | 4.17 2.16 | 0.0 | 31.8 |
| Method | Cap. count | Mean cost (USD/run) |
|---|---|---|
| ReAct | 3.51 1.78 | 0.21 |
| ReSum | 5.95 2.90 | 3.36 |
| Forgetting | 5.17 2.49 | 3.24 |
| Context folding | 5.89 2.90 | 3.54 |
| ACM | 4.17 2.16 | 2.57 |
| SCoRe | 6.65 3.26 | 2.82 |
| Method | Cap. count | T2+ (%) | Cov@3 % |
|---|---|---|---|
| SCoRe | 6.65 3.26 | 22.2 | 51.8 |
| w/o capability judgment relabeling | 6.14 3.14 | 15.9 | 47.0 |
| w/o stop-aware compaction | 5.32 2.56 | 9.5 | 41.7 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Tier | Capability flags and interpretation |
|---|---|
| T5 | cov_func , cov_line Coverage: reaching the affected function and lines. |
| T4 | diff , asan , crash Bug triggering: differential behavior, sanitizer errors, or crashes. |
| T3 | addrof , fakeobj , caged_read , caged_write Target-specific primitives: object addresses, forged object references, and memory access within the sandbox. |
| T2 | infoleak_binary , infoleak_libc , infoleak_stack , arb_read , arb_write General-purpose primitives: address disclosure and arbitrary memory access beyond the sandbox. |
| T1 | pc_control , ace Control-flow hijack and code execution: redirecting execution and executing arbitrary code. |
| Mythos-ACE (21 tasks) | Mythos-non-ACE (20 tasks) |
|---|---|
| crbug-1509576 | crbug-339064932 |
| crbug-339736513 | crbug-386565144 |
| crbug-378779897 | crbug-403364367 |
| cve-2023-6702 | cve-2024-0519 |
| cve-2024-0517 | cve-2024-12053 |
| cve-2024-10230 | cve-2024-5274 |
| Model | Reasoning | Temp. | top_p |
|---|---|---|---|
| DeepSeek | High effort | 1.0 | 0.95 |
| MiMo | Thinking enabled | 1.0 | 0.95 |
| Model | Uncached input | Output | Cache read |
|---|---|---|---|
| DeepSeek | 0.088 | 0.264 | 0.0028 |
| MiMo | 0.435 | 0.870 | 0.0036 |
| Reason | DeepSeek | MiMo |
|---|---|---|
| Stop response | 119 (96.7) | 114 (92.7) |
| Output-length limit | 0 (0.0) | 9 (7.3) |
| Context limit | 1 (0.8) | 0 (0.0) |
| Execution error | 3 (2.4) | 0 (0.0) |
| Total records | 123 | 123 |
| Model | Cohort | Runs with stop | Runs gaining | Tasks gaining | Mean |
|---|---|---|---|---|---|
| DeepSeek | Full benchmark | 116 | 53/116 (45.7%) | 28/41 (68.3%) | 1.56 |
| Mythos-ACE | 61 | 34/61 (55.7%) | 17/21 (81.0%) | 2.25 | |
| Mythos-non-ACE | 55 | 19/55 (34.5%) | 11/20 (55.0%) | 0.80 | |
| MiMo | Full benchmark | 41 | 22/41 (53.7%) | 22/41 (53.7%) | 1.90 |
| Mythos-ACE | 62 | 41/62 (66.1%) | 19/21 (90.5%) | 2.69 | |
| Mythos-non-ACE | 20 | 8/20 (40.0%) | 8/20 (40.0%) | 1.55 |
| Model | Segments | No gain (%) |
|---|---|---|
| DeepSeek | 544 | 96.1 |
| MiMo | 448 | 99.6 |
| Method | Cap. count | T2+ (%) | Cov@3 % |
|---|---|---|---|
| ReAct | 3.07 1.52 | 0.0 | 22.7 |
| ReSum | 4.90 2.77 | 8.9 | 39.8 |
| Forgetting | 4.59 2.41 | 4.9 | 35.5 |
| Context folding | 4.85 2.74 | 7.3 | 37.5 |
| ACM | 3.72 2.07 | 0.0 | 26.5 |
| SCoRe | 5.62 3.07 | 13.0 | 44.5 |
| Threshold | Method | Cap. count | Mean compactions/run |
|---|---|---|---|
| 850K | ReSum | 5.95 2.90 | 3.48 |
| 850K | SCoRe | 6.65 3.26 | 12.97 |
| 250K | ReSum | 5.68 2.69 | 17.05 |
| 250K | SCoRe | 5.49 2.17 | 18.21 |
| Format | Method | Cap. count | Mean compactions/run | Stop-response rate (%) | Burst runs |
|---|---|---|---|---|---|
| Structured | ReSum | 5.95 2.90 | 3.48 | 32.6 | 61/63 |
| Structured | SCoRe | 6.65 3.26 | 12.97 | 4.2 | – |
| Free-form | ReSum | 4.95 2.11 | 4.35 | 47.3 | 63/63 |
| Free-form | SCoRe | 6.49 2.74 | 15.95 | 4.8 | – |
LLM Agents
Large language model agents use language models to plan, use tools, and act across multi-step tasks and environments.
See all 767 papers