LogicEval: A Systematic Framework for Evaluating Automated Repair Techniques for Logical Vulnerabilities in Real-World Software

Syed Md Mukit Rashid; Abdullah Al Ishtiaq; Kai Tu; Yilu Dong; Tianwei Wu; Ali Ranjbar; Tianchang Yang; Najrin Sultana

cs.CRApr 14, 2026

LogicEval: A Systematic Framework for Evaluating Automated Repair Techniques for Logical Vulnerabilities in Real-World Software

Authors: Syed Md Mukit Rashid, Abdullah Al Ishtiaq, Kai Tu, Yilu Dong, Tianwei Wu, Ali Ranjbar, Tianchang Yang, Najrin Sultana,

Organizations: The Pennsylvania State University

Abstract

Logical vulnerabilities in software stem from flaws in program logic rather than memory safety, which can lead to critical security failures. Although existing automated program repair techniques primarily focus on repairing memory corruption vulnerabilities, they struggle with logical vulnerabilities because of their limited semantic understanding of the vulnerable code and its expected behavior. On the other hand, recent successes of large language models (LLMs) in understanding and repairing code are promising. However, no framework currently exists to analyze the capabilities and limitations of such techniques for logical vulnerabilities. We aim to systematically evaluate both traditional and LLM based repair approaches for addressing real world logical vulnerabilities. To facilitate our assessment, we created the first ever dataset, LogicDS, comprising 122 logical vulnerabilities that reflect tangible security impact. We also developed a systematic framework, LogicEval, to evaluate patches for logical vulnerabilities. Evaluations suggest that compilation and testing failures are primarily driven by prompt sensitivity, loss of code context, and difficulty in patch localization.

Figures & tables

Dataset	Tool	C	T	CSL	CSQ	JL	JQ
Real	VRPilot	0.82	0.08	0.65	0.66	0.02	0.07
Synthetic	SimFix	0.01	0.00	0.62	0.64	0.01	0.00
	KNOD	0.35	0.00	0.64	0.65	0.02	0.00
	VRPilot	0.56	0.09	0.65	0.65	0.15	0.03

Table 1: Baseline tool results. “C” and “T” represent the % of generated patches successfully compiled and passed all tests, respectively. “CS” represents average cosine similarity among suggested patches and ground-truth explanations, “J” represents the % of patches a judging LLM determined similar to ground-truth fix. Suffixes “L”, “Q” after each “CS” and “J” represent judging LLM, Llama, and Qwen, respectively. Color heat is relative to all results for the same metric across all our experiments.

	Llama-3.1						Qwen-2.5						OpenAI-o3-mini
P	C	T	CSQ	CSO	JQ	JO	C	T	CSL	CSO	JL	JO	C	T	CSL	CSQ	JL	JQ
P1	0.50	0.06	0.81	0.76	0.10	0.35	0.66	0.04	0.81	0.73	0.60	0.53	0.58	0.07	0.77	0.77	0.79	0.32
P2	0.48	0.07	0.81	0.76	0.13	0.33	0.70	0.04	0.81	0.74	0.54	0.54	–	–	–	–	–	–
P3	0.54	0.06	0.81	0.77	0.13	0.36	0.64	0.04	0.80	0.74	0.54	0.52	–	–	–	–	–	–
P4	0.50	0.06	0.81	0.75	0.11	0.49	0.66	0.04	0.80	0.70	0.49	0.51	0.66	0.07	0.79	0.79	0.64	0.41
P5	0.53	0.03	0.82	0.77	0.13	0.44	0.57	0.03	0.80	0.74	0.51	0.46	0.68	0.08	0.77	0.77	0.66	0.28

Table 2: Compilation, testing, and reasoning score results for different LLM configurations in real-world vulnerabilities. Column meanings are the same as in Table 1 . “P” represents the prompt IDs. Suffixes “L”, “Q”, and “O” after each “CS” and “J” represent judging LLM Llama, Qwen, and OpenAI-o3-mini, respectively.

	Llama-3.1						Qwen-2.5						OpenAI-o3-mini
P	C	T	CSQ	CSO	JQ	JO	C	T	CSL	CSO	JL	JO	C	T	CSL	CSQ	JL	JQ
P9	0.53	0.07	0.82	0.76	0.08	0.41	0.60	0.07	0.81	0.74	0.54	0.49	0.64	0.14	0.78	0.78	0.70	0.31
P10	0.55	0.07	0.78	0.76	0.02	0.34	0.65	0.08	0.80	0.74	0.54	0.58	0.71	0.16	0.75	0.75	0.61	0.36
P11	0.49	0.06	0.81	0.76	0.10	0.43	0.62	0.11	0.82	0.75	0.49	0.49	0.68	0.14	0.77	0.77	0.72	0.36

Table 3: Compilation, testing, and reasoning score results for varying portions of the source code provided in real-world vulnerabilities. Column meanings are the same as in Table 1 and 2 .

	Llama-3.1						Qwen-2.5						OpenAI-o3-mini
P	C	T	CSQ	CSO	JQ	JO	C	T	CSL	CSO	JL	JO	C	T	CSL	CSQ	JL	JQ
P12	0.66	0.04	0.67	0.63	0.02	0.10	0.75	0.05	0.62	0.61	0.11	0.13	0.60	0.07	0.64	0.64	0.20	0.07
P13	0.55	0.03	0.81	0.76	0.11	0.41	0.60	0.04	0.80	0.73	0.59	0.48	0.64	0.14	0.77	0.77	0.70	0.38
P14	0.49	0.00	0.81	0.78	0.18	0.51	0.55	0.00	0.80	0.74	0.69	0.64	0.51	0.09	0.78	0.77	0.72	0.39
P15	0.62	0.07	0.81	0.77	0.46	0.72	0.70	0.08	0.81	0.75	0.79	0.77	0.55	0.13	0.76	0.76	0.84	0.57
P16	0.63	0.00	0.77	0.74	0.33	0.70	0.63	0.00	0.74	0.72	0.67	0.67	0.51	0.08	0.72	0.71	0.72	0.44

Table 4: Compilation, testing, and reasoning scores for varying auxiliary information provided in prompts for real-world vulnerabilities. Column meanings are the same as in Table 1 and 2 .

	Llama-3.1						Qwen-2.5						OpenAI-o3-mini
P	C	T	CSQ	CSO	JQ	JO	C	T	CSL	CSO	JL	JO	C	T	CSL	CSQ	JL	JQ
P17	0.51	0.03	0.56	0.52	0.02	0.07	0.38	0.00	0.60	0.58	0.02	0.08	0.30	0.03	0.56	0.55	0.02	0.00
P18	0.43	0.00	0.63	0.59	0.00	0.07	0.38	0.00	0.60	0.57	0.05	0.07	0.23	0.05	0.61	0.61	0.05	0.02
P19	0.43	0.00	0.62	0.59	0.03	0.08	0.38	0.00	0.61	0.60	0.05	0.10	0.17	0.00	0.61	0.60	0.10	0.02
P20	0.42	0.00	0.70	0.66	0.07	0.16	0.38	0.00	0.64	0.63	0.28	0.20	0.30	0.00	0.69	0.70	0.34	0.05
P21	0.36	0.00	0.80	0.75	0.10	0.48	0.38	0.00	0.74	0.69	0.67	0.64	0.21	0.00	0.75	0.75	0.75	0.23

Table 5: Compilation, testing, and reasoning score results using templates provided in Pearce et al. (2023) in real-world vulnerabilities. Column meanings are the same as in Table 1 and 2 .

Metric	Precision	Recall	F1 Score	Accuracy
Compilation	0.424	0.6	0.497	0.571
J-Qwen	0.773	0.667	0.716	0.8
J-OpenAI	0.5	0.721	0.59	0.701
J-LLaMA	0.547	0.839	0.662	0.677
CodeEmbed 70th (0.944)	0.47	0.508	0.488	0.622
CodeEmbed 90th (0.981)	0.615	0.262	0.368	0.68

Table 6: Precision, recall, F1 score, and accuracy across metrics.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Optimal Scenario (When it Works)	Adaptable for Logical Vulnerability Repair	Limitations When Applying for Logical Vulnerabilities
Template guided	When the vulnerability fix follows a template/pattern	No	Logical vulnerabilities does not follow any general fix pattern.
Constraint based	When the vulnerability can be fixed through enforcing constraints easily extractable from source code	Yes, but difficult	Constraint extraction for logical vulnerabilities struggle with path explosion in large codebases and often lacks the necessary specifications or testcases to derive required constraints.
Search based	When the vulnerability can be fixed by reusing or slightly adapting a similar code segment elsewhere in the source project	Yes, but achieves poor performance	Usually logical vulnerabilities cannot be fixed by reusing existing code, as each one has distinct expected behavior and requires its own custom logic.
Deep learning based	When the vulnerability fix follows a pattern learnable by deep learning models	Yes, but achieves poor performance	Usually logical vulnerabilities do not follow such recognizable pattern

Appendix

Table 7: Analysis of existing AVR techniques and limitations towards repairing logical vulnerabilities

Input	Mandatory	Example
Vulnerable Source Code	✔	Commit (2019a) which is the parent commit of the fix in (2019c)
Fixed Source Code	✔	Commit (2019c) available in national vulnerability database (2019f)
Vulnerability Description	✔	CVE description for CVE-2019-1543 found in (2019f)
Behavioral Specification	✘	Extracted from RFC 7539 [ 72 ] Section 2.8
Context Lines	✘	Lines 21-31 and 140-147 in $crypto/x509/x509\_vfy.c$ in (2019a)
Repair Description	✘	A text description generated by observing the fixed source code (2019c) .

Appendix

Table 8: Example input and vulnerability localization (CVE-2019-1543 (2019f) ) for LogicEval .

ID	Description
P1-P3	Temperature set to 0.2, 0.5, and 0.9, respectively
P4	Task-oriented prompt
P5	Zero-shot prompt (same as P1 )
P6	Few-shot prompt including an example real-world vulnerability description, and vulnerable and fixed blocks
P7	Chain-of-thought prompting, first prompt obtains a repair suggestion $R_{suggestion}$ , which is placed in the second prompt that asks for patch
P8	Same as P7 , but $R_{suggestion}$ is not added to the second prompt

Appendix

Table 9: Prompt Descriptions for Different Prompts Tested With Off-the-shelf LLMs. For prompts P1 to P16 , by default, the vulnerable block

V_{b}

is provided as source code,

	Llama-3.1						Qwen-2.5						OpenAI-o3-mini
P	C	T	CSQ	CSO	JQ	JO	C	T	CSL	CSO	JL	JO	C	T	CSL	CSQ	JL	JQ
P1	0.27	0.05	0.81	0.76	0.12	0.42	0.17	0.07	0.80	0.76	0.48	0.50	0.32	0.13	0.78	0.76	0.68	0.34
P2	0.25	0.05	0.80	0.75	0.12	0.44	0.19	0.08	0.79	0.74	0.47	0.51	–	–	–	–	–	–
P3	0.17	0.03	0.79	0.75	0.08	0.38	0.15	0.08	0.78	0.74	0.47	0.49	–	–	–	–	–	–
P4	0.23	0.07	0.81	0.74	0.08	0.42	0.25	0.10	0.77	0.71	0.49	0.54	0.32	0.13	0.80	0.78	0.67	0.30
P5	0.27	0.05	0.82	0.75	0.08	0.45	0.17	0.10	0.80	0.73	0.57	0.48	0.32	0.12	0.77	0.76	0.62	0.28

Appendix

Table 10: Compilation, testing, and reasoning score results for different LLM configurations for synthetic vulnerabilities in LogicDS . Column meanings are the same as in Table 1 . “P” represents the prompt IDs. Suffixes “L”, “Q”, and “O” after each “CS” and “J” represent judging LLM Llama, Qwen, and OpenAI-o3-mini, respectively.

	Llama-3.1						Qwen-2.5						OpenAI-o3-mini
P	C	T	CSQ	CSO	JQ	JO	C	T	CSL	CSO	JL	JO	C	T	CSL	CSQ	JL	JQ
P9	0.23	0.05	0.81	0.75	0.10	0.38	0.20	0.07	0.81	0.75	0.50	0.55	0.30	0.12	0.78	0.76	0.63	0.27
P10	0.40	0.07	0.76	0.75	0.08	0.35	0.40	0.17	0.78	0.73	0.45	0.52	0.48	0.22	0.76	0.74	0.72	0.30
P11	0.23	0.08	0.80	0.76	0.07	0.42	0.30	0.13	0.80	0.75	0.48	0.53	0.43	0.18	0.76	0.75	0.72	0.30

Appendix

Table 11: Compilation, testing, and reasoning score results for varying portions of the source code provided for synthetic vulnerabilities in LogicDS . Column meanings are the same as in Table 1 and 2 .

	Llama-3.1						Qwen-2.5						OpenAI-o3-mini
P	C	T	CSQ	CSO	JQ	JO	C	T	CSL	CSO	JL	JO	C	T	CSL	CSQ	JL	JQ
P12	0.48	0.02	0.65	0.62	0.00	0.03	0.47	0.00	0.59	0.57	0.05	0.03	0.42	0.05	0.61	0.60	0.13	0.00
P13	0.22	0.07	0.81	0.75	0.08	0.37	0.18	0.05	0.81	0.75	0.48	0.48	0.32	0.12	0.77	0.75	0.63	0.25
P14	0.12	0.05	0.77	0.78	0.17	0.52	0.13	0.05	0.80	0.76	0.68	0.65	0.17	0.07	0.78	0.76	0.75	0.30
P15	0.43	0.33	0.79	0.82	0.72	0.97	0.58	0.53	0.80	0.78	0.97	0.98	0.38	0.32	0.77	0.75	0.95	0.73
P16	0.42	0.28	0.77	0.82	0.79	0.93	0.53	0.49	0.77	0.79	0.93	1.00	0.33	0.26	0.75	0.73	0.95	0.63

Appendix

Table 12: Compilation, testing, and reasoning scores for varying auxiliary information provided in prompts for synthetic vulnerabilities in LogicDS . Column meanings are the same as in Table 1 and 2 .

ID	Claim	C (McNemar)	CS (Wilcoxon p, Cliff $\delta$ , Mean Diff., CI)	J (McNemar)
1	P1 and P2 show similar results	86/219 vs 85/219 ( $p$ =1.0000, OR=1.14)	0.7801 vs 0.7740; $p$ =0.1516 ( ns ); $\delta$ =+0.080; $\Delta$ =+0.0061 [-0.0018, +0.0140]	169/438 vs 175/438 ( $p$ =0.5446, OR=0.84)
2	P1 and P4 show similar results	136/332 vs 141/332 ( $p$ =0.5114, OR=0.76)	0.7753 vs 0.7696; $p$ =0.1634 ( ns ); $\delta$ =+0.062; $\Delta$ =+0.0057 [-0.0028, +0.0141]	289/664 vs 283/664 ( $p$ =0.6137, OR=1.13)
3	P5 shows better C , CS , J than P7	139/339 vs 114/339 ( $p$ =0.0015, OR=2.47)	0.7757 vs 0.7420; $p\leq 0.001$ ( sig ); $\delta$ =+0.537; $\Delta$ =+0.0337 [+0.0260, +0.0413]	300/726 vs 285/726 ( $p$ =0.2513, OR=1.22)
4	P8 shows more C but less CS , J than P7	196/339 vs 114/339 ( $p\leq 0.001$ , OR=5.56)	0.6344 vs 0.7420; $p\leq 0.001$ ( sig ); $\delta$ =-0.884; $\Delta$ =-0.1076 [-0.1218, -0.0939]	54/726 vs 285/726 ( $p\leq 0.001$ , OR=0.08)
5a	P12 has higher C but lower CS , J than P13	189/339 vs 138/339 ( $p\leq 0.001$ , OR=3.12)	0.6206 vs 0.7731; $p\leq 0.001$ ( sig ); $\delta$ =-0.967; $\Delta$ =-0.1525 [-0.1662, -0.1389]	53/726 vs 301/726 ( $p\leq 0.001$ , OR=0.04)
5b	P12 has higher C but lower CS , J than P14	189/339 vs 107/339 ( $p\leq 0.001$ , OR=6.47)	0.6206 vs 0.7769; $p\leq 0.001$ ( sig ); $\delta$ =-0.950; $\Delta$ =-0.1564 [-0.1715, -0.1411]	53/726 vs 375/726 ( $p\leq 0.001$ , OR=0.04)

Appendix

Table 13: Paired significance test results for the primary claims, aggregated across all evaluated LLMs for each prompt type.

	Llama-3.1						Qwen-2.5						OpenAI-o3-mini
P	C	T	CSQ	CSO	JQ	JO	C	T	CSL	CSO	JL	JO	C	T	CSL	CSQ	JL	JQ
P17	0.18	0.00	0.58	0.55	0.00	0.02	0.00	0.00	0.59	0.57	0.05	0.07	0.18	0.02	0.58	0.57	0.02	0.00
P18	0.10	0.00	0.62	0.61	0.00	0.05	0.00	0.00	0.60	0.57	0.03	0.05	0.08	0.00	0.59	0.58	0.07	0.03
P19	0.18	0.00	0.60	0.61	0.00	0.03	0.00	0.00	0.62	0.61	0.07	0.08	0.10	0.02	0.60	0.59	0.08	0.02
P20	0.10	0.00	0.69	0.66	0.07	0.25	0.02	0.00	0.65	0.62	0.18	0.25	0.08	0.02	0.68	0.68	0.38	0.13
P21	0.07	0.02	0.77	0.74	0.18	0.42	0.00	0.00	0.74	0.70	0.60	0.57	0.07	0.00	0.76	0.75	0.73	0.23

Appendix

Table 14: Compilation, testing, and reasoning score results using templates provided in Pearce et al. (2023) for synthetic vulnerabilities in LogicDS . Column meanings are the same as in Table 1 and 2 .

Table 15: Llama Fix for CVE-2015-1793 (2015)

Automated Program Repair

Automated program repair develops methods that locate software faults and produce validated fixes with limited human effort.

See all 159 papers

Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation
Mar 10, 2026Amir Al-MaamariAutomated Program Repair Software Security
University of Passau, Germany
Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Sep 22, 2026Om Nepal, Sushant Aryal, Oluseyi Olukola +1Automated Program Repair LLMs for Cybersecurity
School of Computing Sciences and Computer Engineering, The University of Southern Mississippi, Hattiesburg, MS 39406, USA
RAVEN: Agentic RAG for Automated Vulnerability Repair
Jun 21, 2026Varun Gadey, Zijie Liu, Alexandra DmitrienkoRetrieval-Augmented Generation Automated Program Repair
University of Duisburg-Essen, Germany

Mar 10, 2026J/K move · Enter open · S save

Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation

Amir Al-Maamari

University of Passau, Germany

Automated Program Repair Software Security LLM Reliability

Open paper AskPDF arXiv

Large Language Models (LLMs) show promise for Automated Program Repair (APR), yet their effectiveness on security vulnerabilities remains poorly characterized. This study analyzes 319 LLM-generated security patchesacross 64 Java vulnerabilities from the Vul4J benchmark. Using tri-axis evaluation (compilation, security via PoV tests, functionality via test suites), the analysis reveals that only 24.8% of patches achieve full correctness, while 51.4% fail both security and functionality. The dominant failure mode is semantic misunderstanding: LLMs produce syntactically valid code but apply incorrect repair strategies. The proposed Security Repair Score (SRS) quantifies this gap, showing LLMs preserve functionality (mean 0.832) but struggle with security (mean 0.251). Vulnerability type strongly predicts difficulty, with fix rates ranging from 0% (input validation) to 45% (infinite loop). These findings demonstrate that LLM security patches require rigorous validation before deployment.

Abstract

Automated Program Repair

Explore similar work

Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

RAVEN: Agentic RAG for Automated Vulnerability Repair