A.S.E 2.0: Understanding How Repository Context Impacts the Security of LLM-Generated Code
Organizations: Nanyang Technological University Singapore, Singapore · Tencent China · Zhejiang University Hangzhou, China · IIE, CAS Beijing, China
Abstract
Large language model (LLM) coding assistants and agents construct a repository context before generating code. For security-sensitive code, this context decides whether the model sees the project's safeguards (e.g., authorization checks) or insecure code to imitate. However, context construction is designed almost exclusively for functional correctness, and secure-code benchmarks treat the context as fixed, so how repository context affects the security of generated code remains unclear. We construct CASE-Bench, a benchmark of 145 repository-level tasks from real CVEs in 51 projects. Each task runs in Docker, has functional tests and a proof-of-concept exploit validated in both directions, and is annotated with the repository code that enforces or undermines its security. Using CASE-Bench, we compare 13 context construction methods and six controls with six LLMs, edit single context elements, and analyze coding-agent trajectories. We find that (1) functionality-oriented context raises the functional pass rate by up to 8.2 percentage points (pp) but the secure pass rate by at most 3.7 pp, (2) withholding a safeguard lowers the secure pass rate by up to 8.1 pp, while one inserted insecure exemplar raises correct-but-exploitable code by 6.3 pp, and (3) coding agents observe the relevant safeguard in only 27-36% of trajectories. These findings inform security-aware context construction and secure-code evaluation.
Figures & tables
| Check | Criterion | Result |
| V1 Vulnerable version | builds, tests pass, PoC succeeds | 145/145 |
| V2 Patched version | builds, tests pass, PoC fails | 145/145 |
| V3 Alternative secure implementation | builds, tests pass, PoC fails | 143/145 |
| V4 Deny-all implementation | tests fail | 145/145 a |
| V5 Determinism | identical verdicts | 145/145 b |
| V6 Manual audit | agreement / Cohen’s ( Cohen, 1960 ) | 95.7% / 0.90 |
| Family | Method | Context policy (as reproduced) | Gen.- dep. | Tokens | Fix-API |
| Reference/ control | Local-only | masked target file only | - | 0 | .12 |
| BM25+README ( Lian et al., 2026 ) | BM25-ranked files + README | - | 7.6K | .29 | |
| Repo-128K | serialized repository, truncated to 128K tokens | - | 118K | .63 | |
| Security reminder | Local-only + generic security instruction | - | 0 | .12 | |
| Placebo | Local-only + security code of another project | - | 3.9K | .12 | |
| Reference context | annotated safeguards + source-sink path | - | 2.7K | 1.00 |
| Method / condition | FuncPass @1 | SecPass @1 [95% CI] | CbE @1 | FuncPass | SecPass |
| Reference / control | |||||
| Local-only | 56.9 | 12.6 [9.3, 16.2] | 44.3 | - | - |
| BM25+README | 59.7 | 12.4 [9.1, 16.0] | 47.3 | +2.8 | -0.2 |
| Security reminder | 55.1 | 13.9 [10.5, 17.6] | 41.2 | -1.8 | +1.3 |
| Placebo | 56.2 | 12.9 [9.6, 16.5] | 43.3 | -0.7 | +0.3 |
| Repo-128K | 60.9 | 12.1 [8.9, 15.6] | 48.8 | +4.0 | -0.5 |
| Family | Tasks | Local-only | Best func. | Best-of | CodeGuarder | Source-sink | Reference |
| Memory safety | 65 | 16.2 | 21.4 | 25.6 | 21.8 | 19.9 | 26.2 |
| Injection | 42 | 13.4 | 16.8 | 19.8 | 17.7 | 21.2 | 28.0 |
| Access control | 25 | 2.0 | 2.9 | 4.4 | 3.1 | 9.3 | 17.6 |
| Other | 13 | 12.0 | 15.0 | 17.9 | 15.4 | 15.8 | 20.9 |
| All | 145 | 12.6 | 16.3 | 19.6 | 16.8 | 18.1 | 24.8 |
| Share of CbE | Per 100 gen. | |||||
| Pattern | Definition | All | Func. | Sec. | Func. | Sec. |
| Missing | no acceptable safeguard set (L2/L3) | 44% | 46% | 29% | 20.6 | 12.3 |
| Knowledge | L1 task; generic defense omitted | 13% | 13% | 12% | 5.8 | 5.1 |
| Unused | present, neither called nor replicated | 16% | 15% | 28% | 6.7 | 11.9 |
| Misused | called, yet exploited | 10% | 9% | 22% | 4.0 | 9.4 |
| Imitated | similar to an exemplar in context | 19% | 20% | 9% | 9.0 | 3.8 |
| Agent | Backbone | FuncPass @1 | SecPass @1 | CbE @1 | SG obs. | IE obs. | Tokens | Cost |
| SWE-agent | Sonnet 4.5 | 69.7 | 16.6 | 53.1 | 29% | 46% | 141K | $1.08 |
| SWE-agent | GPT-5.4 | 71.0 | 16.1 | 54.9 | 27% | 43% | 127K | $0.91 |
| Claude Code | Sonnet 4.5 | 74.0 | 18.4 | 55.6 | 36% | 51% | 204K | $1.41 |
| ClaudeCode-Ret (best) | Sonnet 4.5 | 66.9 | 17.9 | 49.0 | 22% | 36% | 7.2K | $0.09 |
| ReACC-Hybrid (best) | GPT-5.4 | 68.3 | 17.0 | 51.3 | 19% | 41% | 7.8K | $0.05 |
| PoC fails | PoC succeeds | Summary | ||
| StaticSecure | 5,138 | 8,778 | Agreement / | 63.2% / 0.25 |
| StaticAlarm | 1,901 | 13,167 | Precision / recall † | .87 / .60 |
| Resists the base PoC, exploited by a variant (40 tasks) | Share | 7.3% | ||
| † Static alarms are evaluated relative to the base PoC verdict; PoC-resistant alarms are not necessarily false positives. | ||||
| Benchmark | Lang. | Tasks | Real CVEs | Func. | Exploit | Context | Agents | Annot. |
| A.S.E ( Lian et al., 2026 ) | 5 | 120 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| SecRepoBench ( Shen et al., 2026 ) | 2 | 318 | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ |
| RealSec-bench ( Wang et al., 2026b ) | 1 | 105 | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| SecureVibeBench ( Chen et al., 2026a ) | 2 | 105 | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ |
| SusVibes ( Zhao et al., 2026 ) | 1 | 200 | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ |
| CASE-Bench (ours) | 7 | 145 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
LLM Code Generation
Large language model code generation studies how models translate instructions and context into executable programs and how to make them reliable.
See all 333 papers