cs.CROct 19, 2026
SaveSecure Agentic Coding through Counterexample-Grounded Feedback
Organizations: Quantum AI & Cyber Security Institute (QACI), FPT Corporation · Mila - Quebec Artificial Intelligence Institute · Université de Montréal · Google
Abstract
As coding models and agent harnesses become more capable, reliable security verification is becoming a central bottleneck. Improving secure code generation requires a signal whose reported progress remains aligned with actual program behavior under optimization; otherwise, an optimizer can improve the signal's verdict without removing the underlying vulnerability. Existing model-based approaches require models to either infer program behavior from source code or specify correct outcomes for adversarial tests, placing substantial reasoning demands on the security signal. We introduce \falcon{}, a counterexample-grounded verification framework that separates attack exploration from behavioral verification and connects them through execution traces of the program under test. In test-time repair, \falcon{} yields substantially stronger held-out functional-security gains than alternative security signals across secure code generation benchmarks, while preserving or improving functionality. As an RL reward, \falcon{} also outperforms static-analysis and learned rewards on held-out functional-security, even when all policies successfully optimize their respective training rewards. Combining \falcon{} with either proxy reward further improves held-out performance, suggesting that execution-grounded feedback can mitigate reward overoptimization while preserving useful guidance from existing security signals.
Figures & tables
Figure 1: Comparison with prior security assessment approaches. (a) Source-code judges must infer program behavior and assess security, while test generators must propose inputs and define correct expected outcomes. Both place substantial reasoning demands on the model. (b) FALCON reduces the reasoning demand of security judgment by separating attack exploration from verification and connecting them through an execution process that produces concrete traces for the verifier.
Figure 2: Overview of FALCON . The attacker generates deterministic PoC probes targeting code produced by the coding agent. The faithfulness check rejects probes that shadow the target or fabricate the apparent violation. Admitted probes execute in a sandbox, and a three-stage verifier evaluates the resulting traces using a crash oracle, deterministic CWE detectors, and an LLM trace judge for unresolved cases.
Figure 3: Repair loop on CWEval with four security feedback signals and gpt-5.4-mini as the coding model. ( left ) Held-out Func-Sec@1, graded by CWEval’s ground-truth tests, which no signal observes. ( middle ) Proxy secure rate, the improvement each signal reports on its own verdicts, relative to round one. ( right ) Reported versus held-out improvement from round one to five. FALCON produces the largest held-out improvement. CodeQL transfers a smaller gain, while the LLM-as-a-judge and attacker-only signals almost do not transfer.
Figure 4: Test-time repair with FALCON feedback on SecCodeBench-V2 (left) and CWEval (right) . Starred models are repaired with FALCON for up to five rounds. The other frontier models are shown as single-pass reference points. FALCON improves Func-Sec@1 for every repaired model on both benchmarks.
| Model | FuncPass | SecPass | Func-TestRate | Sec-TestRate |
| gpt-5.4-mini | 44.09 | 12.36 | 74.01 | 69.76 |
| gpt-5.4-mini + FALCON | 46.59 (+2.50) | 17.20 (+4.84) | 76.24 (+2.23) | 72.55 (+2.79) |
| gpt-5.4 | 68.82 | 20.25 | 81.65 | 77.55 |
| gpt-5.4 + FALCON | 70.97 (+2.15) | 25.27 (+5.02) | 83.12 (+1.47) | 79.35 (+1.80) |
| gpt-5.6-luna | 62.54 | 19.00 | 80.99 | 77.78 |
| gpt-5.6-luna + FALCON | 63.62 (+1.08) | 22.22 (+3.22) | 81.47 (+0.48) | 79.36 (+1.58) |
Table 1: Repair with FALCON inside coding agents on SusVibes ( Zhao et al., 2025 ) . Results are averaged over three harnesses: SWE-agent, OpenCode, and Claude Code. FuncPass and SecPass follow the benchmark’s task-level acceptance criteria, which permit failures up to task-specific baseline allowances rather than requiring all tests to pass. Func-TestRate and Sec-TestRate report the fraction of functional and security test cases passed, respectively, averaged across tasks.
Figure 5: FALCON provides a robust RL signal and strengthens existing security rewards. Every policy steadily increases its own training reward ( left ), yet held-out Func-Sec@1 separates sharply ( middle ). When optimized alone, REAL ( Yao et al., 2025 ) and SecCodePRM ( Yu et al., 2026 ) transfer much less of their reward gains to held-out Func-Sec@1, most clearly at 3B, where Func-Sec@1 eventually declines while training reward keeps increasing, consistent with reward overoptimization. In contrast, FALCON maintains stronger held-out Func-Sec@1, and combining it with either reward further improves transfer, whereas combining REAL and SecCodePRM does not. Configurations that include FALCON also keep KL lower ( right ).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
| Hardware | A100 40 GB |
| Learning rate | |
| Train batch size | 64 |
| PPO mini-batch size | 32 |
| Micro-batch size per GPU | 2 |
| Rollouts per prompt | 8 |
Table 3: GRPO training hyperparameters. The six reward settings share every value here and differ only in how the security term is computed.
| Benchmark | Time (yymm) | #Tasks | Context | Multi-file Edit | #Edited Lines | #CWEs |
| CWEval | 2501 | 119 | File | ✗ | 10.0 | 31 |
| SusVibes | 2512 | 186 | Repository | ✓ | 172.1 | 77 |
| SecCodeBench-V2 | 2602 | 98 | Project | ✗ | N/A | 22 |
Table 4: Secure coding benchmarks used for test-time repair, ordered by release date. Context is the supplied code context, not the edit scope. Edited lines is the average reference-patch size, which SecCodeBench-V2 does not report.
| # | Detector | Runtime evidence required | CWE |
|---|---|---|---|
| Probe payload reaches an execution or query sink | |||
| 1 | Command injection | Payload reaches the captured arguments of a process-execution sink unneutralised | 77, 78, 88 |
| 2 | SQL injection | Payload appears inline in the raw query string reaching the driver, with no bound parameter carrying it | 89 |
| 3 | Code and eval injection | Injected expression executes: its marker fires as a side effect, or its evaluated result is returned | 94, 95 |
| 4 | Template injection | Target output contains the evaluated result of the probe’s template expression | 77 |
| 5 | Unsafe deserialisation | Payload reaches a deserialisation sink and its marker executes or its object is reconstructed | 502 |
Table 7: The 21 deterministic detectors of Stage 2, in application order. marks detectors that match language-specific exception or library strings.
| Run | Tasks | Generated probes | rejected | rejected | Admitted probes |
| SWE-agent / DeepSeek-V4-Pro | 186 | 2737 | 117 (4.27%) | 299 (10.92%) | 2321 (84.80%) |
| SWE-agent / GPT-5.4 | 186 | 5209 | 67 (1.29%) | 377 (7.24%) | 4765 (91.48%) |
| OpenCode / GPT-5.4 | 186 | 6988 | 108 (1.55%) | 545 (7.80%) | 6335 (90.66%) |
| OpenCode / GPT-5.6-luna | 186 | 3449 | 75 (2.17%) | 503 (14.58%) | 2871 (83.24%) |
| Claude Code / GPT-5.6-luna | 186 | 3275 | 63 (1.92%) | 237 (7.24%) | 2975 (90.84%) |
| Claude Code / DeepSeek-V4-Pro | 186 | 2857 | 72 (2.52%) | 330 (11.55%) | 2455 (85.93%) |
Table 8: Faithfulness check outcomes on SusVibes. is evaluated only for probes that pass , so the rejection categories are disjoint. All percentages use the total number of generated probes as the denominator. The Average row reports mean counts across configurations and percentages pooled over all probes. Admitted probes pass both checks, regardless of their subsequent security verdict.
| Requested Tests | Attacker | Attack Success Rate (%) | Avg. Tests Generated | CWE Types Targeted | Avg. CWE / Task |
| gpt-oss-20b | 14.29 | 0.84 | 38 | 0.69 | |
| gpt-oss-120b | 14.29 | 1.02 | 34 | 0.88 | |
| gpt-5.2 | 15.97 | 1.00 | 35 | 0.99 | |
| gpt-5.4 | 20.17 | 1.00 | 37 | 0.80 | |
| – | gpt-oss-20b | 14.29 | 1.88 | 49 | 1.16 |
| gpt-oss-120b | 22.69 | 3.14 | 55 | 2.55 |
Table 9: Attacker effectiveness on CWEval. Target code is generated by gpt-5.4 and judged by the full verifier. Requested Tests is the min–max number of PoC probes asked for in the prompt. Attack Success Rate is the fraction of tasks with at least one verifier-flagged PoC probe. Avg. Tests Generated is the mean number of PoC probes the attacker actually produces per task. CWE Types Targeted counts distinct CWE categories assigned by the attacker to its probes, which may extend beyond CWEval’s 31 annotated categories; it measures proposed attack diversity, not verified vulnerability coverage. Avg. CWE / Task is the mean number of these categories per task. The lower block hints the attacker with the true (CWE-hint) or a random wrong CWE (Wrong-CWE-hint).
Figure 6: Per-round repair with FALCON feedback on SecCodeBench-V2 (top) and CWEval (bottom), reporting functional correctness (Func), security (Sec), and their joint rate (Func-Sec). Across all models, security and Func-Sec rise sharply in the first two rounds and then plateau, while functional correctness is maintained or improved.
| Model | C | C++ | Go | JavaScript | Python |
| claude-opus-4.8 | 80.65 | 66.67 | 26.32 | 78.26 | 84.00 |
| claude-opus-4.7 | 74.19 | 71.43 | 52.63 | 73.91 | 72.00 |
| claude-sonnet-4.6 | 77.42 | 52.38 | 10.53 | 65.22 | 68.00 |
| deepseek-v4-pro-04-24 | 41.94 | 42.86 | 47.37 | 69.57 | 72.00 |
| glm-5p1 | 51.61 | 42.86 | 36.84 | 56.52 | 52.00 |
| kimi-k2p5 | 32.26 | 28.57 | 15.79 | 56.52 | 56.00 |
Table 10: Per-language Func-Sec@1 (%) on CWEval. The top block reports frontier models without FALCON ; the lower blocks pair each GPT-5.4 backbone with its FALCON counterpart, where bold marks the better value in the pair. Adding FALCON improves every backbone in all five languages, and gpt-5.4 + FALCON exceeds the best frontier baseline in each language.
| Model | C++ | Go | Java | JavaScript | Python |
| claude-opus-4.8 | 45.00 | 21.15 | 76.89 | 87.50 | 75.00 |
| claude-opus-4.7 | 50.00 | 23.08 | 77.83 | 68.75 | 65.38 |
| claude-sonnet-4.6 | 41.67 | 15.38 | 80.66 | 56.25 | 59.62 |
| deepseek-v4-pro-04-24 | 46.67 | 15.38 | 66.98 | 81.25 | 57.69 |
| glm-5p1 | 45.00 | 13.46 | 69.34 | 81.25 | 59.62 |
| kimi-k2p5 | 46.67 | 11.54 | 66.51 | 75.00 | 63.46 |
Table 11: Per-language Func-Sec@1 (%) on SecCodeBench-V2. The top block reports frontier models without FALCON ; the lower blocks pair each GPT-5.4 backbone with its FALCON counterpart, where bold marks the better value in the pair. Adding FALCON improves every backbone in all five languages, and gpt-5.4 + FALCON exceeds the best frontier baseline in each language.
| Feedback signal | Round | C | C++ | Go | JavaScript | Python |
| FALCON | 1 | 64.52 | 66.67 | 47.37 | 60.87 | 64.00 |
| 2 | 83.87 | 85.71 | 73.68 | 69.57 | 92.00 | |
| 3 | 87.10 | 90.48 | 73.68 | 73.91 | 92.00 | |
| 4 | 90.32 | 90.48 | 73.68 | 82.61 | 92.00 | |
| 5 | 90.32 | 85.71 | 73.68 | 86.96 | 92.00 | |
| Attacker-only | 1 | 64.52 | 61.90 | 47.37 | 56.52 | 72.00 |
Table 12: Per-language Func-Sec@1 (%) over five repair rounds on CWEval with gpt-5.4-mini as the coding model, for the four feedback signals compared in Section 4.2.1 . From round 2 onward, a cell is shaded green when its score is above the round-1 score, red when below, and gray when unchanged. FALCON is the only signal that stays above its round-1 score in every language at every round. The static analyzer improves most languages but by smaller margins, while attacker-only and LLM-as-a-Judge feedback fall below round 1 in at least two languages by round 5.
| Harness | Model | FuncPass | SecPass | Func-TestRate | Sec-TestRate |
| swe- agent | gpt-5.4-mini | 25.27 | 8.60 | 68.01 | 64.68 |
| gpt-5.4-mini + FALCON | 29.57 (+4.30) | 12.90 (+4.30) | 72.62 (+4.61) | 69.77 (+5.09) | |
| gpt-5.4 | 54.30 | 17.74 | 79.45 | 75.45 | |
| gpt-5.4 + FALCON | 55.91 (+1.61) | 22.58 (+4.84) | 82.19 (+2.74) | 78.30 (+2.85) | |
| gpt-5.6-luna | 38.17 | 16.13 | 74.42 | 71.72 | |
| gpt-5.6-luna + FALCON | 39.78 (+1.61) | 20.43 (+4.30) | 74.98 (+0.56) | 73.07 (+1.35) |
Table 13: Repair with FALCON inside coding agents on SusVibes [ Zhao et al., 2025 ] . FuncPass and SecPass follow the benchmark’s task-level acceptance criteria, which permit failures up to task-specific baseline allowances rather than requiring all tests to pass. Func-TestRate and Sec-TestRate report the fraction of functional and security test cases passed, respectively, averaged across tasks.
More in this topic
Secure Code Generation
Secure code generation studies how to produce code that meets functional requirements while avoiding security vulnerabilities.
See all 48 papers