Secure Agentic Coding through Counterexample-Grounded Feedback
Authors: Tue Le, Anh Nguyen Hoang, Gabriel Sasseville, Doseok Jang, Pierre-Luc Bacon, Nghi D. Q. Bui
Organizations: Quantum AI & Cyber Security Institute (QACI), FPT Corporation · Mila - Quebec Artificial Intelligence Institute · Université de Montréal · Google
As coding models and agent harnesses become more capable, reliable security verification is becoming a central bottleneck. Improving secure code generation requires a signal whose reported progress remains aligned with actual program behavior under optimization; otherwise, an optimizer can improve the signal's verdict without removing the underlying vulnerability. Existing model-based approaches require models to either infer program behavior from source code or specify correct outcomes for adversarial tests, placing substantial reasoning demands on the security signal. We introduce \falcon{}, a counterexample-grounded verification framework that separates attack exploration from behavioral verification and connects them through execution traces of the program under test. In test-time repair, \falcon{} yields substantially stronger held-out functional-security gains than alternative security signals across secure code generation benchmarks, while preserving or improving functionality. As an RL reward, \falcon{} also outperforms static-analysis and learned rewards on held-out functional-security, even when all policies successfully optimize their respective training rewards. Combining \falcon{} with either proxy reward further improves held-out performance, suggesting that execution-grounded feedback can mitigate reward overoptimization while preserving useful guidance from existing security signals.
Figures & tables
Figure 1: Comparison with prior security assessment approaches. (a) Source-code judges must infer program behavior and assess security, while test generators must propose inputs and define correct expected outcomes. Both place substantial reasoning demands on the model. (b) FALCON reduces the reasoning demand of security judgment by separating attack exploration from verification and connecting them through an execution process that produces concrete traces for the verifier.
Figure 2: Overview of FALCON . The attacker generates deterministic PoC probes targeting code produced by the coding agent. The faithfulness check rejects probes that shadow the target or fabricate the apparent violation. Admitted probes execute in a sandbox, and a three-stage verifier evaluates the resulting traces using a crash oracle, deterministic CWE detectors, and an LLM trace judge for unresolved cases.
Figure 3: Repair loop on CWEval with four security feedback signals and gpt-5.4-mini as the coding model. ( left ) Held-out Func-Sec@1, graded by CWEval’s ground-truth tests, which no signal observes. ( middle ) Proxy secure rate, the improvement each signal reports on its own verdicts, relative to round one. ( right ) Reported versus held-out improvement from round one to five. FALCON produces the largest held-out improvement. CodeQL transfers a smaller gain, while the LLM-as-a-judge and attacker-only signals almost do not transfer.
Figure 4: Test-time repair with FALCON feedback on SecCodeBench-V2 (left) and CWEval (right) . Starred models are repaired with FALCON for up to five rounds. The other frontier models are shown as single-pass reference points. FALCON improves Func-Sec@1 for every repaired model on both benchmarks.
Model
FuncPass
SecPass
Func-TestRate
Sec-TestRate
gpt-5.4-mini
44.09
12.36
74.01
69.76
gpt-5.4-mini + FALCON
46.59 (+2.50)
17.20 (+4.84)
76.24 (+2.23)
72.55 (+2.79)
gpt-5.4
68.82
20.25
81.65
77.55
gpt-5.4 + FALCON
70.97 (+2.15)
25.27 (+5.02)
83.12 (+1.47)
79.35 (+1.80)
gpt-5.6-luna
62.54
19.00
80.99
77.78
gpt-5.6-luna + FALCON
63.62 (+1.08)
22.22 (+3.22)
81.47 (+0.48)
79.36 (+1.58)
Table 1: Repair with FALCON inside coding agents on SusVibes ( Zhao et al., 2025 ) . Results are averaged over three harnesses: SWE-agent, OpenCode, and Claude Code. FuncPass and SecPass follow the benchmark’s task-level acceptance criteria, which permit failures up to task-specific baseline allowances rather than requiring all tests to pass. Func-TestRate and Sec-TestRate report the fraction of functional and security test cases passed, respectively, averaged across tasks.
Figure 5: FALCON provides a robust RL signal and strengthens existing security rewards. Every policy steadily increases its own training reward ( left ), yet held-out Func-Sec@1 separates sharply ( middle ). When optimized alone, REAL ( Yao et al., 2025 ) and SecCodePRM ( Yu et al., 2026 ) transfer much less of their reward gains to held-out Func-Sec@1, most clearly at 3B, where Func-Sec@1 eventually declines while training reward keeps increasing, consistent with reward overoptimization. In contrast, FALCON maintains stronger held-out Func-Sec@1, and combining it with either reward further improves transfer, whereas combining REAL and SecCodePRM does not. Configurations that include FALCON also keep KL lower ( right ).
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Hardware
8× A100 40 GB
Learning rate
1×10−6
Train batch size
64
PPO mini-batch size
32
Micro-batch size per GPU
2
Rollouts per prompt
8
Appendix
Table 3: GRPO training hyperparameters. The six reward settings share every value here and differ only in how the security term is computed.
Benchmark
Time (yymm)
#Tasks
Context
Multi-file Edit
#Edited Lines
#CWEs
CWEval
2501
119
File
✗
10.0
31
SusVibes
2512
186
Repository
✓
172.1
77
SecCodeBench-V2
2602
98
Project
✗
N/A
22
Appendix
Table 4: Secure coding benchmarks used for test-time repair, ordered by release date. Context is the supplied code context, not the edit scope. Edited lines is the average reference-patch size, which SecCodeBench-V2 does not report.
#
Detector
Runtime evidence required
CWE
Probe payload reaches an execution or query sink
1
Command injection
Payload reaches the captured arguments of a process-execution sink unneutralised
77, 78, 88
2
SQL injection
Payload appears inline in the raw query string reaching the driver, with no bound parameter carrying it
89
3
Code and eval injection
Injected expression executes: its marker fires as a side effect, or its evaluated result is returned
94, 95
4
Template injection
Target output contains the evaluated result of the probe’s template expression
77
5
Unsafe deserialisation
Payload reaches a deserialisation sink and its marker executes or its object is reconstructed
502
Appendix
Table 7: The 21 deterministic detectors of Stage 2, in application order. † marks detectors that match language-specific exception or library strings.
Run
Tasks
Generated probes
Fstat rejected
Fatt rejected
Admitted probes
SWE-agent / DeepSeek-V4-Pro
186
2737
117 (4.27%)
299 (10.92%)
2321 (84.80%)
SWE-agent / GPT-5.4
186
5209
67 (1.29%)
377 (7.24%)
4765 (91.48%)
OpenCode / GPT-5.4
186
6988
108 (1.55%)
545 (7.80%)
6335 (90.66%)
OpenCode / GPT-5.6-luna
186
3449
75 (2.17%)
503 (14.58%)
2871 (83.24%)
Claude Code / GPT-5.6-luna
186
3275
63 (1.92%)
237 (7.24%)
2975 (90.84%)
Claude Code / DeepSeek-V4-Pro
186
2857
72 (2.52%)
330 (11.55%)
2455 (85.93%)
Appendix
Table 8: Faithfulness check outcomes on SusVibes. Fatt is evaluated only for probes that pass Fstat , so the rejection categories are disjoint. All percentages use the total number of generated probes as the denominator. The Average row reports mean counts across configurations and percentages pooled over all probes. Admitted probes pass both checks, regardless of their subsequent security verdict.
Requested Tests
Attacker
Attack Success Rate (%)
Avg. Tests Generated
CWE Types Targeted
Avg. CWE / Task
1
gpt-oss-20b
14.29
0.84
38
0.69
gpt-oss-120b
14.29
1.02
34
0.88
gpt-5.2
15.97
1.00
35
0.99
gpt-5.4
20.17
1.00
37
0.80
2 – 5
gpt-oss-20b
14.29
1.88
49
1.16
gpt-oss-120b
22.69
3.14
55
2.55
Appendix
Table 9: Attacker effectiveness on CWEval. Target code is generated by gpt-5.4 and judged by the full verifier. Requested Tests is the min–max number of PoC probes asked for in the prompt. Attack Success Rate is the fraction of tasks with at least one verifier-flagged PoC probe. Avg. Tests Generated is the mean number of PoC probes the attacker actually produces per task. CWE Types Targeted counts distinct CWE categories assigned by the attacker to its probes, which may extend beyond CWEval’s 31 annotated categories; it measures proposed attack diversity, not verified vulnerability coverage. Avg. CWE / Task is the mean number of these categories per task. The lower block hints the attacker with the true (CWE-hint) or a random wrong CWE (Wrong-CWE-hint).
Figure 6: Per-round repair with FALCON feedback on SecCodeBench-V2 (top) and CWEval (bottom), reporting functional correctness (Func), security (Sec), and their joint rate (Func-Sec). Across all models, security and Func-Sec rise sharply in the first two rounds and then plateau, while functional correctness is maintained or improved.
Model
C
C++
Go
JavaScript
Python
claude-opus-4.8
80.65
66.67
26.32
78.26
84.00
claude-opus-4.7
74.19
71.43
52.63
73.91
72.00
claude-sonnet-4.6
77.42
52.38
10.53
65.22
68.00
deepseek-v4-pro-04-24
41.94
42.86
47.37
69.57
72.00
glm-5p1
51.61
42.86
36.84
56.52
52.00
kimi-k2p5
32.26
28.57
15.79
56.52
56.00
Appendix
Table 10: Per-language Func-Sec@1 (%) on CWEval. The top block reports frontier models without FALCON ; the lower blocks pair each GPT-5.4 backbone with its FALCON counterpart, where bold marks the better value in the pair. Adding FALCON improves every backbone in all five languages, and gpt-5.4 + FALCON exceeds the best frontier baseline in each language.
Model
C++
Go
Java
JavaScript
Python
claude-opus-4.8
45.00
21.15
76.89
87.50
75.00
claude-opus-4.7
50.00
23.08
77.83
68.75
65.38
claude-sonnet-4.6
41.67
15.38
80.66
56.25
59.62
deepseek-v4-pro-04-24
46.67
15.38
66.98
81.25
57.69
glm-5p1
45.00
13.46
69.34
81.25
59.62
kimi-k2p5
46.67
11.54
66.51
75.00
63.46
Appendix
Table 11: Per-language Func-Sec@1 (%) on SecCodeBench-V2. The top block reports frontier models without FALCON ; the lower blocks pair each GPT-5.4 backbone with its FALCON counterpart, where bold marks the better value in the pair. Adding FALCON improves every backbone in all five languages, and gpt-5.4 + FALCON exceeds the best frontier baseline in each language.
Feedback signal
Round
C
C++
Go
JavaScript
Python
FALCON
1
64.52
66.67
47.37
60.87
64.00
2
83.87
85.71
73.68
69.57
92.00
3
87.10
90.48
73.68
73.91
92.00
4
90.32
90.48
73.68
82.61
92.00
5
90.32
85.71
73.68
86.96
92.00
Attacker-only
1
64.52
61.90
47.37
56.52
72.00
Appendix
Table 12: Per-language Func-Sec@1 (%) over five repair rounds on CWEval with gpt-5.4-mini as the coding model, for the four feedback signals compared in Section 4.2.1 . From round 2 onward, a cell is shaded green when its score is above the round-1 score, red when below, and gray when unchanged. FALCON is the only signal that stays above its round-1 score in every language at every round. The static analyzer improves most languages but by smaller margins, while attacker-only and LLM-as-a-Judge feedback fall below round 1 in at least two languages by round 5.
Harness
Model
FuncPass
SecPass
Func-TestRate
Sec-TestRate
swe- agent
gpt-5.4-mini
25.27
8.60
68.01
64.68
gpt-5.4-mini + FALCON
29.57 (+4.30)
12.90 (+4.30)
72.62 (+4.61)
69.77 (+5.09)
gpt-5.4
54.30
17.74
79.45
75.45
gpt-5.4 + FALCON
55.91 (+1.61)
22.58 (+4.84)
82.19 (+2.74)
78.30 (+2.85)
gpt-5.6-luna
38.17
16.13
74.42
71.72
gpt-5.6-luna + FALCON
39.78 (+1.61)
20.43 (+4.30)
74.98 (+0.56)
73.07 (+1.35)
Appendix
Table 13: Repair with FALCON inside coding agents on SusVibes [ Zhao et al., 2025 ] . FuncPass and SecPass follow the benchmark’s task-level acceptance criteria, which permit failures up to task-specific baseline allowances rather than requiring all tests to pass. Func-TestRate and Sec-TestRate report the fraction of functional and security test cases passed, respectively, averaged across tasks.
Coding agents are now proficient enough to generate complex software applications from a single prompt. As their capabilities have grown, human oversight has increasingly shifted from line-by-line code review toward hands-off evaluation of outcomes. However, recent studies have shown that such a transition exposes a critical risk: functional correctness alone does not guarantee a secure implementation. Despite growing attention to code security, training safer coding agents remains challenging because reliable security supervision is difficult to obtain at scale from real-world repositories. We introduce AuraForge to synthesize and validate executable security tests for training secure coding agents. Our approach combines attack-oriented test synthesis, language-extensible task construction, and safeguards against reward hacking. Using AuraForge, we construct AuraGym, a multi-language and multi-CWE executable training gym: 679 executable feature-implementation tasks from 344 real-world repositories across Python, JavaScript, and TypeScript, covering 177 CWE categories. On the subset with human-written security tests, AuraForge produces about 3 times as many test cases on average and reduces the false-positive rate by 83.23%, allowing alternative secure implementations to receive correct supervision. Training Qwen3.5-4B with synthesized security tests gains larger improvements than human-written security tests (average 19.7 FuncPass and 6.2 SecPass vs. 14.9 FuncPass and 4.4 SecPass) on three languages. These results demonstrate that AuraForge provides more diverse and reliable security supervision to train secure coding agents.
Danqing Wang, Songwen Zhao, Harsh Sharma +4
Carnegie Mellon University · University of California, Los Angeles · ScOp Venture Capital.
Large Language Models (LLMs) can generate functionally correct code that remains insecure. Security-focused fine-tuning can be expensive and is not always available, motivating inference-time approaches. Security-oriented prompting and retrieval can reduce vulnerabilities, but may also remove required behavior or apply defenses incompatible with the task. The challenge is to identify relevant security knowledge and use it to improve security without sacrificing functionality. We present SecGent, an inference-time framework that retrieves weakness context from the Common Weakness Enumeration (CWE) database and applies it to secure code generation. By connecting task specifications to relevant textual CWE records, SecGent provides task-specific security guidance without relying on retrieved vulnerable or patched code examples. Complementary functional and security tests check the resulting implementation, and their execution feedback guides repairs toward satisfying both objectives. We evaluate SecGent with four open-source LLMs and Claude Sonnet 5.5 across five programming languages on the CWEval benchmark. It achieves the highest joint functional--security success against four inference-time baselines for all five models. Across the open-weight models, mean functional-security@1 (FS@1) improves by 15.5% over RAG, with gains in both functionality and security, and by 19.8% over the strongest prompt optimizer compiled for each model. On Sonnet, FS@1 reaches 92.1%, outperforming the strongest baseline AutoGen by 4.8%. In a separate comparison, SecGent outperforms Claude Code with security-focused prompting by over 6% in FS@1. Feedback-guided repair further improves FS@1 by up to 6.7% over generation without feedback, primarily by recovering functionality while checking security.
Md Afif Al Mamun, Gias Uddin
University of Calgary, Calgary, Canada · York University, Toronto, Canada
LLM coding agents now generate code at an unprecedented scale, yet LLM-generated code introduces cybersecurity vulnerabilities into codebases without human involvement. Even when frontier models are explicitly asked to write secure production code with relevant weaknesses to avoid in context, we find that they still produce verifiable vulnerabilities on average 23% of the time across a corpus of 250 benign coding prompts. We introduce SecureForge, an automated pipeline that both audits security risks of frontier models and produces auditing-informed secure system prompts that reduce output security vulnerabilities while maintaining unit test performance. SecureForge first identifies benign prompts that produce statically detectable vulnerabilities, and then amplifies them into a large synthetic prompt corpus of diverse scenarios using a Markovian sampling technique to jointly maintain error rates and prompt diversity. This corpus is then used to iteratively optimize the system prompts to reduce output security vulnerabilities. On frontier models, SecureForge yields a statistically significant Pareto improvement in both unit test success and output security, with output vulnerabilities reduced by up to 48%. The resulting system prompts transfer zero-shot to in-the-wild coding agent prompts, without any exposure to real user prompt distributions during optimization.