DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents

Rupam Patir; Keyan Guo; Suvadra Barua; Abhijeet Pathak; Dinesh Gudimetla; Jiawei Guo; Hongxin Hu; Haipeng Cai

cs.SENov 24, 2025

DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents

Authors: Rupam Patir, Keyan Guo, Suvadra Barua, Abhijeet Pathak, Dinesh Gudimetla, Jiawei Guo, Hongxin Hu, Haipeng Cai

Organizations: University at Buffalo, SUNY

Abstract

Large language models (LLMs) and LLM-based coding agents are now used to generate code from natural-language specifications, yet ensuring such code is both functionally correct and secure remains a challenge. We present DualGauge, the first fully automated framework for jointly evaluating correctness and security of specification-only code generation, supported by DualGauge-Bench, a language-agnostic benchmark of 307 coding tasks each paired with functional and security tests derived from the same specification. Evaluating 10 representative LLMs across Python, C++, and JavaScript, we find that functional correctness substantially overestimates reliable code generation: even the strongest model remains below 15% joint security-functionality success in every language. Common model-side factors--scale, extended thinking, quantization, instruction tuning, and code specialization--do not reliably improve joint performance, suggesting secure-and-correct code generation does not simply emerge from stronger coding capability. Evaluation of 3 leading coding agents (Codex, OpenHands, and Claude Code) shows that iterative scaffolding is not a reliable remedy on specification-only tasks: results are strongly scaffold-dependent, with two of the three agents failing to outperform a comparable direct model. A qualitative audit reveals that these failures concentrate at the output contract boundary and in guards that exist but are insufficient--patterns that only joint benchmarking reliably exposes.

Figures & tables

Figure 1: Motivating example for semantic runtime security evaluation. Two generated implementations satisfy the functional test, but only the second satisfies the security requirement. Both return None with exit code 0 on the adversarial input, so output matching alone cannot distinguish them. The execution trace reveals the difference: the vulnerable implementation attempts file access before failing, whereas the secure implementation rejects the unsafe path before any file access. Thus, semantic judgment over the execution trace can serve as the security-test oracle.

Benchmark	SecTest	FuncTest	Paired	Pure NL Spec.	Lang.-agn.	Cov.
Pearce et al. ( Pearce et al., 2022 )	✗	✗	✗	✗	✗	✗
SecurityEval ( Siddiq and Santos, 2022 )	✗	✗	✗	✗	✗	✗
CodeLMSec ( Hajipour et al., 2024 )	✗	✗	✗	✗	✗	✗
SecuCoGen ( Wang et al., 2023 )	✗	✗	✗	⚫	✗	✗
SafeGenBench ( Li et al., 2025b )	✗	✗	✗	✓	✗	✗
CodeGuard+ ( Fu et al., 2024 )	✗	✓	✗	✗	✗	✗

Table 1: Benchmark comparison on using dynamic security ( SecTest ) and functional ( FuncTest ) tests paired for the same coding task ( Paired ) purely specified by natural language ( Pure NL Spec. ), programming-language agnostic ( Lang.-agn. ) design/nature, and coverage enforcement ( Cov. ), including partial support ( ⚫ ).

Figure 2: Overview of DualGauge . The system takes a target LLM and benchmark items from DualGauge-Bench , generates candidate code, executes it through an agentic executor, evaluates functional outputs and security behaviors with runtime oracles, and aggregates functionality, security, and joint security-functionality metrics.

	Python					C++					JavaScript
Model	PR	SPR	p@1	s@1	sp@1	PR	SPR	p@1	s@1	sp@1	PR	SPR	p@1	s@1	sp@1
gpt-5-medium	73.2	72.7	38.6	34.5	14.8	54.1	73.3	19.8	30.4	8.4	47.9	72.9	20.0	32.4	8.0
gpt-4.1	65.2	68.4	31.3	23.4	8.7	49.0	63.5	19.5	23.2	5.9	42.9	63.5	16.1	23.3	4.5
claude-opus-4-7-think-on	70.3	59.6	33.9	24.0	10.2	52.6	63.8	19.2	24.9	5.0	50.4	60.8	24.3	28.3	7.0
claude-sonnet-4-5-think-off	68.5	57.1	31.3	20.2	8.4	49.0	59.4	19.0	23.6	5.4	46.8	56.6	21.4	23.6	6.6
claude-haiku-4-5	40.8	55.2	17.8	18.8	6.5	42.6	57.8	18.2	22.7	5.3	32.3	50.8	15.5	15.9	2.9

Table 2: Headline performance across the 10 main models and three languages. All metrics are percentages; shorthands are defined in § 4.5 .

Component	Metric	Result
DualGauge-Bench (field-level)	Gwet’s AC1	0.963
DualGauge-Bench (benchmark-level)	Gwet’s AC1	0.684
DualGauge executor	F1 (P / R)	0.910 (0.963 / 0.863)
DualGauge evaluator, by judge
GPT-5 nano (ours)	F1 (P / R)	0.895 (0.906 / 0.885)
GPT-5 medium	F1 (P / R)	0.906 (0.901 / 0.911)

Table 3: Reliability validation results. DualGauge-Bench quality is measured by inter-annotator agreement (Gwet’s AC1) at the field and benchmark level; DualGauge components by F1, precision, and recall against human-curated ground truth.

Panel of Figure 3 — Figure 3: Effect of five model-side factors on Python performance. Markers: $\bullet$ pass@1 , $\bullet$ secure@1 , $\bullet$ secure-pass@1 , $\circ$ PR , $\circ$ SPR . Each subfigure isolates one factor; vertical lines separate model families. Base models for Mistral and Gemma produced no parseable output under specification-only prompting.

Figure 4: Distribution of SPR across the 10 evaluated models per CWE group and language. Box plots show median, interquartile range, and outliers. Groups are ordered by increasing median SPR across all languages.

	Python					C++					JavaScript
System	PR	SPR	p@1	s@1	sp@1	PR	SPR	p@1	s@1	sp@1	PR	SPR	p@1	s@1	sp@1
codex-gpt-5.4	71.7	78.5	46.7	50.7	24.3	34.1	77.6	15.1	46.2	8.9	44.2	78.7	23.4	48.5	13.5
openhands-gpt-5.4	49.4	63.2	27.3	29.6	9.5	29.3	65.4	12.9	29.8	5.0	36.2	63.3	14.2	30.1	4.6
claude-code-opus-4.7	40.5	53.2	21.7	19.4	4.9	16.0	59.0	4.3	22.0	1.6	23.1	61.7	10.6	26.6	2.0

Table 4: Performance of three coding agents across three languages. Codex and OpenHands both use GPT-5.4 as the underlying model; Claude Code uses Opus 4.7. Metrics are the same as Table 2 .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Example Content
Prompt	Create a function yaml_load(filename) that reads a file from ./test_dir/ , parses it as YAML using the language’s YAML library, and returns the parsed data structure. On invalid YAML, missing files, or parsing errors, return "Error: Invalid input" .
Functional tests	valid_empty.yaml $\rightarrow$ null ; valid_list.yaml $\rightarrow$ ["first", "second", "third"] ; nonexistent.yaml $\rightarrow$ "Error: Invalid input"
	…
Security tests	malicious.yaml : reject or safely parse without arbitrary code execution ( CWE-502 ); ../../etc/passwd : reject path traversal and return the prompt-defined error response ( CWE-22 ); entity_expansion.yaml : handle alias/entity expansion safely without resource exhaustion or crashes ( CWE-776 )
	…

Appendix

Table 5: Example benchmark item in DualGauge-Bench .

Property	Value
Specification-only prompts	307
Total runtime tests	3,785
Avg. functional tests / prompt	6.24
Avg. security tests / prompt	6.09
Covered CWE categories	90
Evaluated languages	Python, C++, JavaScript

Appendix

Table 6: Summary of DualGauge-Bench used in the measurement study.

Level / Field	Gwet’s AC1	3-way Raw Agr.
Final benchmark decision	0.684	0.635
Pooled field-level judgments	0.963	0.947
prompt_clear	0.996	0.993
prompt_language_agnostic	0.964	0.948
prompt_io_defined	0.998	0.997
fc_tests_correct	0.915	0.883

Appendix

Table 7: Inter-annotator agreement at the field level and final-decision level. The benchmark-level decision uses a conjunctive rule over nine checklist fields.

Component	Precision	Recall	F1
Agentic Executor	0.963	0.863	0.910
LLM-Based Evaluator	0.906	0.885	0.895

Appendix

Table 8: Manual validation results for DualGauge components. Positive class: usable output (executor), pass (evaluator).

Prompt	P	R	F1	$\kappa$ vs. base
Baseline (ours)	0.894	0.857	0.875	—
Chain-of-thought	0.874	0.883	0.878	0.776
Strict auditor	0.864	0.817	0.840	0.815
Terse	0.859	0.832	0.845	0.699

Appendix

Table 9: Prompt sensitivity of the security evaluator over the 153 human-labeled security cases.

CWE	n	P	R	F1
CWE-20	48	0.94	0.89	0.92
CWE-400	12	1.00	0.67	0.80
CWE-22	11	0.75	0.75	0.75
CWE-78	9	1.00	0.83	0.91
CWE-79	7	1.00	1.00	1.00
CWE-94	6	0.50	1.00	0.67

Appendix

Table 10: Per-CWE evaluator reliability on the human-labeled security cases, for ten CWE classes with at least three labeled cases each.

Model	Access	PR	SPR	p@1	s@1	sp@1
Closed aggregate	Closed	66.1	64.4	33.0	25.3	10.4
Open aggregate	Open	51.9	49.6	20.1	14.0	4.3

Appendix

Table 11: Python-only aggregate comparison of closed and open models, averaged across all available evaluated configurations in each access class.

Model	Size	PR	SPR	p@1	s@1	sp@1
qwen3-0.6b	0.6B	28.7	34.5	8.4	6.7	0.4
qwen3-4b	4B	46.8	53.6	18.3	15.4	3.3
qwen3-14b	14B	60.2	56.5	27.4	17.5	5.6
qwen3-32b	32B	63.4	60.1	31.1	23.2	8.3
gemma-3-1b-it	1B	35.6	30.4	7.6	6.9	1.1
gemma-3-4b-it	4B	52.9	48.8	15.7	13.3	4.7

Appendix

Table 12: Python-only scale ladder for Qwen3 and Gemma-3 families.

Model	Thinking	PR	SPR	p@1	s@1	sp@1
gpt-5-minimal	minimal	70.5	71.5	39.7	29.2	10.5
gpt-5-high	high	71.7	72.9	36.9	31.0	13.9
claude-opus-4-7-think-off	off	69.0	57.7	34.5	21.3	10.5
claude-opus-4-7-think-on	on	70.3	59.6	33.9	24.0	10.2
qwen3-32b	off	63.4	60.1	31.1	23.2	8.3
qwen3-32b-think	on	67.0	53.1	23.0	12.0	5.5

Appendix

Table 13: Python-only extended-thinking comparisons.

Model	Precision	PR	SPR	p@1	s@1	sp@1
qwen3-8b-bf16	BF16	60.3	55.3	26.8	17.5	6.6
qwen3-8b-fp8	FP8	57.3	53.4	27.5	14.7	6.6
qwen3-8b-int8	INT8	55.6	54.5	23.0	16.4	4.5
qwen3-8b-awq-int4	AWQ-INT4	52.6	53.2	22.5	17.9	6.1
qwen3-8b-gptq-int4	GPTQ-INT4	56.5	53.8	23.6	16.0	4.4
llama-3.1-8b-instruct-bf16	BF16	48.5	49.2	22.5	14.4	5.1

Appendix

Table 14: Python-only quantization results for Qwen3-8B and Llama-3.1-8B-Instruct.

Model	Role	PR	SPR	p@1	s@1	sp@1
qwen2.5-32b-instruct	General	60.5	52.5	27.8	16.5	5.6
qwen2.5-coder-32b-instruct	Coder	64.2	56.5	29.5	20.3	7.2
qwen3-30b-a3b-instruct	General	63.0	61.2	31.0	22.0	8.2
qwen3-coder-30b-a3b-instruct	Coder	62.5	59.8	28.6	19.4	6.0
mistral-small-instruct-2409	General	64.6	53.1	26.8	15.4	4.7
codestral-22b-v0.1	Coder	63.7	49.8	27.5	14.8	4.5

Appendix

Table 15: Python-only code-specialized versus general comparisons.

Model	Tuning	PR	SPR	p@1	s@1	sp@1
qwen2.5-7b-base	Base	47.3	45.0	19.8	12.5	4.0
qwen2.5-7b-instruct	Instruct	46.4	50.4	19.5	18.6	5.1
llama-3.1-8b-base	Base	28.3	32.1	6.4	6.4	0.8
llama-3.1-8b-instruct-bf16	Instruct	48.5	49.2	22.5	14.4	5.1
mistral-7b-v0.3-base	Base	—	—	0.0	0.0	0.0
mistral-7b-instruct-v0.3	Instruct	34.9	41.1	13.0	7.4	1.7

Appendix

Table 16: Python-only instruction-tuning comparisons. Base models with no PR/SPR produced no parseable output.

CWE	Py	C++	JS	CWE	Py	C++	JS
CWE-20	56.3	50.9	53.8	CWE-338	70.5	48.8	62.0
CWE-22	44.3	43.2	42.5	CWE-346	59.4	66.1	64.4
CWE-23	11.8	11.8	6.4	CWE-347	56.4	49.9	53.0
CWE-59	41.6	37.9	31.1	CWE-369	65.0	92.6	71.7
CWE-73	23.0	26.9	24.2	CWE-377	65.0	29.0	48.0
CWE-74	35.6	34.2	28.9	CWE-399	33.3	18.5	25.9

Appendix

Table 17: Per-CWE Security Pass Rate (%) by language (Py = Python, C++ = C++, JS = JavaScript). Averaged across all 10 direct-generation models. Rows sorted by CWE number.

Group	CWE IDs
Injection	74, 77, 78, 79, 80, 89, 90, 93, 94, 95, 113, 117, 134, 917, 1321
Input Validation	20, 129, 158, 170, 173, 176, 178, 185, 444, 457, 471, 681, 703, 704, 710, 754, 755, 843
Path & File Access	22, 23, 59, 73, 377, 434
Cryptography	295, 312, 319, 326, 327, 329, 330, 331, 338, 347, 521, 522, 759, 798
Serialization & XML	502, 611, 643, 776, 915
Resource / DoS	369, 399, 400, 672, 674, 705, 730, 770, 789, 835, 1333

Appendix

Table 18: CWE group definitions used in Figure 4 .

Model	@1	@2	@3	@4	@5
GPT-5 medium	10.3	15.6	19.0	21.7	23.7
Claude Opus 4.7	7.6	11.0	13.2	14.5	15.4
Qwen3-14B	3.9	5.9	7.2	8.1	9.0

Appendix

Table 19: secure-pass@k (%) for

k=1..5

, averaged over tasks with at least

k

samples. Sampling improves the joint rate but with sharply diminishing returns.

Model	DualGauge	BaxBench	CWEval
GPT-5 nano	10.2	35.0	58.8
Claude Haiku 4.5	4.5	32.9	55.5
Llama 3.1 8B	2.2	0.5	29.4

Appendix

Table 20: Joint correct-and-secure rate (%) on the three shared models. All three benchmarks rank the models identically.

LLM Code Generation

Large language model code generation studies how models translate instructions and context into executable programs and how to make them reliable.

See all 314 papers

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking
Jul 2, 2024Jiexin Wang, Liuwen Cao, Xitong Luo +6Execution-Guided Code Generation Agentic Code Generation
South China University of Technology, Guangzhou, China · University of Innsbruck, Innsbruck, Austria
Secure and Functional Code Generation with Self-Evaluation
Oct 24, 2026Md Afif Al Mamun, Gias UddinSecure Code Generation Retrieval-Augmented Code Generation
University of Calgary, Calgary, Canada · York University, Toronto, Canada
Functional and Secure Code Generation with Task Vectors
Jul 8, 2026Felix Wang, Anudeep Das, Mei Nagappan +1Code Generation LLM Safety Alignment
University of Waterloo, Waterloo, ON, Canada · KTH Royal Institute of Technology, Stockholm, Sweden

Jul 2, 2024J/K move · Enter open · S save

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

Jiexin Wang, Liuwen Cao, Xitong Luo, Yang Cao+5

South China University of Technology, Guangzhou, China · University of Innsbruck, Innsbruck, Austria

Execution-Guided Code Generation Agentic Code Generation Secure Code Generation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used for program synthesis, yet they often generate code that is functionally plausible but insecure. Progress in secure code generation has been hindered by benchmarks that are small, non-executable, leak mitigation details, or rely on noisy analyzers and subjective judgments, making it difficult to measure whether security improves without sacrificing correctness. We address these gaps with CodeSecEval, an execution-based benchmark for secure code generation, comprising 255 Python tasks spanning 77 CWE categories. Each task provides paired insecure and secure implementations together with executable functional and vulnerability-targeted security tests, enabling precise and reproducible evaluation of secure code generation and insecure-code repair. Building on CodeSecEval, we propose SecAwareCoder, an agent-based framework that shifts code generation toward secure-by-construction synthesis. SecAwareCoder performs task-adaptive threat modeling to identify security-sensitive regions and derive task-grounded vulnerability hypotheses, uses these hypotheses to guide both constraint-aware code generation and security-aware test synthesis, and leverages execution feedback for targeted refinement. Experiments across multiple LLM backbones show that SecAwareCoder consistently improves Pass@1 and security robustness over prompting and analyzer-driven baselines, narrowing the security--correctness gap in LLM code generation.

Abstract

LLM Code Generation

Explore similar work

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

Secure and Functional Code Generation with Self-Evaluation

Functional and Secure Code Generation with Task Vectors