Repairing Broken Access Control in Web Applications with LLMs: An Empirical Benchmark Study

Mingrui Liu; Liang He; Xinchu Wang; Yeting Li; Dongxu Han; Yunpeng Li; Yuling Liu

cs.CROct 25, 2026

Repairing Broken Access Control in Web Applications with LLMs: An Empirical Benchmark Study

Authors: Mingrui Liu, Liang He, Xinchu Wang, Yeting Li, Dongxu Han, Yunpeng Li, Yuling Liu

Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China

Abstract

Broken access control (BAC) ranks among the most severe web-application vulnerability classes, yet its automated repair remains largely unexplored. Unlike memory-safety vulnerabilities, BAC is a logic vulnerability whose repair demands a deep understanding of an application's access-control model and business semantics, which traditional automated vulnerability repair techniques handle poorly. Large language models (LLMs) open a promising avenue for this problem, but no standard benchmark exists for studying how well they repair BAC vulnerabilities. We construct BAC-Bench, the first benchmark for BAC repair, consisting of 46 real-world BAC vulnerabilities drawn from six widely used web applications spanning four programming languages. Each vulnerability is equipped with a sandboxed runtime environment, a PoC validation program and a functional test program. We further propose a BAC repair-strategy framework that characterizes any BAC patch along four dimensions, Position, Coverage, Mechanism, and Content. Evaluating eight base LLMs and two commercial LLM agents on BAC-Bench, we find that single-turn base LLMs repair 37.68% of the vulnerabilities on average, while two minimal scaffolds, Apply-Feedback and Navigate-and-Locate, raise this average to 66.03%, with the best base LLM reaching 75.36%, close to the 78.99% of the best agent. Notably, some LLM-generated patches surpass the official patches, and one incidentally closed a zero-day vulnerability that had lain dormant for three years. Our findings delineate the current capability boundary of LLM-based BAC repair and offer guidance toward its practical adoption.

Figures & tables

Figure 1. Overview of our study design for evaluating the effectiveness of LLMs in repairing BAC vulnerabilities in web applications.

Application	Language	Stars	Domain	Access-control model
OpenEMR	PHP	5.2k	Healthcare Management	ACL + RBAC hybrid (phpGACL)
Usememos	Go	52.8k	Note-taking Platform	RBAC
Apache InLong	Java	1.5k	Data Integration Framework	RBAC + groups
BookWyrm	Python	2.6k	Social Reading Platform	RBAC
XWiki	Java	1.2k	Knowledge Management	ACL + groups
Nextcloud Server	PHP	34k	Cloud Storage Platform	DAC + ACL + RBAC hybrid

Table 1. Web applications in BAC-Bench .

Category	Model	Developer	Release	Access
LLM Agents	Claude Code (2.1.160)	Anthropic	2026-06-02	Closed
LLM Agents	Codex (0.139.0)	OpenAI	2026-06-09	Closed
Base LLMs	Claude Opus 4.8	Anthropic	2026-05-28	Closed
	GPT-5.5	OpenAI	2026-04-23	Closed
	Qwen3.7-Max	Alibaba	2026-05-20	Closed
	Gemini 3.5 Flash	Google	2026-05-19	Closed

Table 2. LLM-based systems evaluated on our BAC-Bench . The DeepSeek entry refers to the preview version used in our experiments. Release gives the release or publication date of the listed version.

Dimension	Fine-grained categories and criterion	Binary mapping
Position	Presentation, control, business, data, and cross-cutting authorization layers. A repair point is accurate only when it lies on the vulnerable execution path and is placed at a time that can constrain the sensitive operation.	Success: All required repair points are accurate. Failure: At least one repair point is inaccurate.
Coverage	The ratio $k/n$ , where $n$ is the number of vulnerability units frozen before inspecting candidate patches and $k$ is the number of units receiving a structural security intervention.	Success: $k=n$ , coverage complete. Failure: $k<n$ , coverage incomplete.
Mechanism	①Standard mechanism reuse. ② Reliable project-native mechanism. ③ Fragile substitute mechanism. ④ Mechanism implementation failure.	Success: ① or ②. Failure: ③ or ④.
Content	Correct, partially correct, and incorrect.	Success: Correct only. Failure: Partially correct or incorrect .

Table 3. Operationalization of the four-dimensional BAC repair-strategy framework.

Model	Condition	Run 1	Run 2	Run 3	Mean
Claude Code(Opus 4.8)	LLM Agent	84.78%	71.74%	80.43%	78.99%
Codex(GPT-5.5)	LLM Agent	78.26%	76.09%	76.09%	76.81%
Macro-average	LLM Agent	81.52%	73.92%	78.26%	77.90%
DeepSeek-V4 Pro	Base LLM	36.96%	43.48%	28.26%	36.23%
Qwen3.7-Max	Base LLM	36.96%	41.30%	36.96%	38.41%
Claude Opus 4.8	Base LLM	30.43%	34.78%	26.09%	30.43%

Table 4. Repair Effectiveness of LLM Agents and Single-Turn Base LLMs on BAC-BENCH. Each percentage denotes the proportion of these 46 vulnerability instances for which the generated patch is effective after execution in the corresponding sandboxed environment. The Mean column reports the arithmetic average over the three independent runs, while each macro-average row reports the unweighted arithmetic mean across models within the corresponding category for that run. The Mean value of a macro-average row is then computed from its three run-level macro-averages. A patch is counted as finally effective only if it is successfully applied, successfully compiled when compilation is applicable, resistant to the PoC exploit, and compatible with the functional tests.

Model	Condition	Applicability	PoC	Functional	Effective
DeepSeek-V4 Pro	Base LLM	60.14%	39.86%	49.28%	36.23%
Qwen3.7-Max	Base LLM	62.32%	42.75%	47.83%	38.41%
Claude Opus 4.8	Base LLM	42.03%	32.61%	38.41%	30.43%
GPT-5.5	Base LLM	86.96%	68.12%	73.19%	60.87%
Gemini 3.5 Flash	Base LLM	69.57%	39.13%	44.93%	30.43%
GLM-5.2	Base LLM	64.49%	43.48%	55.07%	41.30%

Table 5. Stage-wise success rates of base LLMs in BAC repair.

Figure 2. Case A: a patch generated by Qwen3.7-Max under Base condition for OpenEMR’s CVE-2022-2730. The patch declares incorrect old and new hunk line counts, causing a hunk-header line-count mismatch.

Figure 3. Case B: a patch generated by Opus4.8 under Base condition for Usememos’s CVE-2022-4684. The patch uses a hunk header without the required old and new line ranges, resulting in malformed hunk-header syntax.

Figure 4. Repair performance under Base and Apply-Feedback conditions.

Figure 5. Case E: a patch generated by Kimi K2.7-Code under the Base LLM condition for XWiki’s CVE-2021-32620. The patch checks the nonexistent disabled property instead of XWiki’s active property, causing the account-state authorization check to fail.

Model	Applicability	PoC	Functional	Effective
DeepSeek-V4 Pro	91.30% ( -5.07 )	65.22% ( +3.62 )	79.71% (+0.00)	62.32% ( +6.52 )
Qwen3.7-Max	100.00% ( +0.72 )	76.09% ( +17.39 )	78.26% ( -1.45 )	71.01% ( +15.94 )
Claude Opus 4.8	100.00% ( +3.62 )	77.54% ( +19.57 )	91.30% ( +5.07 )	75.36% ( +18.84 )
GPT-5.5	86.23% ( -13.77 )	81.16% ( +12.32 )	76.81% ( -6.52 )	73.19% ( +14.49 )
Gemini 3.5 Flash	87.68% ( -7.25 )	73.91% ( +26.09 )	73.19% ( +9.42 )	64.49% ( +23.91 )
GLM-5.2	97.83% (+0.00)	69.57% ( +26.09 )	81.88% ( -5.80 )	63.77% ( +23.19 )

Table 6. Incremental effect of the Navigate-and-Locate scaffold combined with Apply-Feedback . Each cell reports the mean rate under the combined condition, followed in parentheses by the change in percentage points relative to the Apply-Feedback -only condition.

Model	Condition	$\boldsymbol{N}$	Position	Coverage	Mechanism	Content
Claude Code	LLM Agent	133	3.01%	9.02%	8.27%	14.29%
Codex	LLM Agent	132	2.27%	3.03%	5.30%	11.36%
DeepSeek-V4 Pro	Base LLM	72	5.56%	16.67%	13.89%	26.39%
Qwen3.7-Max	Base LLM	73	5.48%	10.96%	20.55%	24.66%
Claude Opus 4.8	Base LLM	55	1.82%	14.55%	9.09%	21.82%
GPT-5.5	Base LLM	114	7.02%	8.77%	19.30%	14.91%

Table 7. Failure prevalence across the four dimensions of the BAC repair-strategy framework among all LLM-generated patches that pass application and compilation when applicable. Only the

810

dynamically failing patches are analyzed under the framework, whereas the

1{,}925

dynamically successful patches are treated as successful in all four dimensions by convention for aggregate prevalence estimation. Percentages use the corresponding

N

as the denominator and are pooled across the three runs. Because a patch may fail in multiple dimensions, the four prevalence values are not mutually exclusive and need not sum to

100\%

. AF denotes Apply-Feedback , and NL denotes Navigate-and-Locate .

Dynamic-failure type	$N$	Position	Coverage	Mechanism	Content
PoC ✗, functional ✓	588	23.98%	45.92%	61.05%	81.12%
PoC ✓, functional ✗	169	8.88%	9.47%	55.62%	72.19%
PoC ✗, functional ✗	53	15.09%	7.55%	84.91%	79.25%

Table 8. Conditional failure prevalence across the four dimensions of the BAC repair-strategy framework within each dynamic-failure category. ✓ and ✗ denote passing and failing a test, respectively. Percentages in each row use the corresponding category size

N

as the denominator, and a patch may fail in multiple dimensions.

Dimension	Agreement	Cohen’s $\boldsymbol{\kappa}$	Corr.
Position	98.1%	0.955	2
Coverage	100.0%	1.000	0
Mechanism	94.4%	0.874	6
Content	92.6%	0.781	8
Overall	96.3%	0.926	16

Table 9. Reliability of the LLM-assisted static audit. The table reports, for each dimension, the agreement between the labels assigned by the LLM and the majority-vote consensus of three blinded cybersecurity PhD annotators on a stratified sample of 108 patches. Each dimension contains 108 patch-dimension cells, yielding 432 cells overall. Corr. denotes the number of LLM-assigned labels that disagree with the majority consensus.

Vulnerability	Application	Official patch
Vulnerability	Application	PoC test	Functional test
CVE-2022-4505	OpenEMR	✓	✗
CVE-2023-2946	OpenEMR	✓	✗
CVE-2022-4684	Usememos	✗	✓
CVE-2022-4802	Usememos	✗	✗
CVE-2022-4803	Usememos	✗	✓

Table 10. Official patches with defects. ✓ and ✗ denote passing and failing a test, respectively, whereas

\blacktriangle

indicates that a better-performing patch exists

Figure 6. Excerpt illustrating the core logic of OpenEMR’s official patch for CVE-2022-4505 (commit 235b1910 ), released on August 20, 2022 ( OpenEMR Project, 2022 ) . The patch checks whether the requested appointment belongs to the authenticated patient’s appointment set, but incorrectly interprets the index returned by array_search() as a Boolean value.

Figure 7. Excerpt illustrating OpenEMR’s follow-up repair for CVE-2022-4505 (commit 210a63b4 ), released on August 8, 2024 ( OpenEMR Project, 2024 ) . The strict comparison with false distinguishes a valid match at index $0$ from the absence of a match.

Figure 8. Excerpt illustrating an effective patch generated by Gemini 3.5 Flash for CVE-2022-4505, which validates appointment ownership directly through an SQL query.

Figure 9. Excerpt from the original official patch for CVE-2023-2946 (commit 81832ac ), released on March 14, 2023 ( OpenEMR Project, 2023a ) . The patch incorrectly invokes the central ACL facility with the ACO name prescriptions rather than its configured value rx .

Figure 10. Excerpt illustrating the follow-up official patch for CVE-2023-2946 (commit a8ed9cd ), released on May 12, 2023 ( OpenEMR Project, 2023b ) . The correction replaces the invalid ACO name prescriptions with the configured value rx .

Figure 11. Excerpt illustrating an effective Kimi K2.7-Code-generated patch for CVE-2023-2946. The patch guards the prescription controller using the correct (’patients’, ’rx’) ACL coordinate.

Figure 12. Excerpt from an effective patch generated by DeepSeek-V4 Pro for CVE-2023-31066, showing its authorization check for the sibling forceDelete path.

Figure 13. Excerpt illustrating the official patch for the sibling forceDelete path associated with CVE-2023-31066, released on June 22, 2026 ( Apache InLong Project, 2026 ) .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 14. Case C: a patch generated by Kimi K2.7-Code under Base condition for BookWyrm’s BW-2 vulnerability. The patch includes an extra blank line that is absent from the target source, resulting in a hunk-context mismatch.

Figure 15. Case D: a patch generated by Qwen3.7-Max under Base condition for Apache InLong’s CVE-2023-31454. The patch omits the a/ and b/ path prefixes, resulting in a patch-path resolution failures under the git apply -p1 rule.

Figure 16. Case F: a patch generated by DeepSeek-V4 Pro under the Apply-Feedback condition for OpenEMR’s CVE-2022-1177. The patch adds the patients/pat_rep entry check but omits the ACL configuration updates that remove this permission from the Front Office and Accounting groups, resulting in an incomplete ACL-policy update.

Figure 17. Case G: a patch generated by DeepSeek-V4 Pro under the Apply-Feedback condition for Usememos’ CVE-2022-4684. The patch adds MemoPatch.Validate , but the vulnerable PATCH /memo/:memoId handler never invokes it, leaving the authorization check outside the effective request path.

Application	Vulnerability ( $n$ )
OpenEMR (PHP)	CVE-2022-2730 ( $n{=}1$ ); CVE-2022-2824 ( $n{=}4$ ); CVE-2022-4505 ( $n{=}1$ ); CVE-2022-4567 ( $n{=}2$ ); CVE-2023-2942 ( $n{=}1$ ); CVE-2023-2944 ( $n{=}3$ ); CVE-2023-2945 ( $n{=}1$ ); CVE-2023-2946 ( $n{=}3$ ); (OE-1) ( $n{=}1$ ); CVE-2022-1177 ( $n{=}1$ )
BookWyrm (Python)	BW-1 ( $n{=}1$ ); BW-2 ( $n{=}1$ ); BW-3 ( $n{=}1$ ); BW-4 ( $n{=}1$ ); BW-5 ( $n{=}1$ )
Apache InLong (Java)	CVE-2023-31066 ( $n{=}3$ ); CVE-2023-31064 ( $n{=}1$ ); CVE-2023-31453 ( $n{=}1$ ); CVE-2023-31454 ( $n{=}2$ )
XWiki (Java)	CVE-2022-23621 ( $n{=}1$ ); CVE-2022-23615 ( $n{=}2$ ); CVE-2021-32620 ( $n{=}1$ ); CVE-2022-41929 ( $n{=}2$ ); CVE-2021-32731 ( $n{=}1$ )
Usememos (Go)	CVE-2022-4684 ( $n{=}1$ ); CVE-2022-4689 ( $n{=}1$ ); CVE-2022-4796 ( $n{=}1$ ); CVE-2022-4797 ( $n{=}1$ ); CVE-2022-4798 ( $n{=}1$ ); CVE-2022-4799 ( $n{=}1$ ); CVE-2022-4801 ( $n{=}1$ ); CVE-2022-4802 ( $n{=}3$ ); CVE-2022-4803 ( $n{=}2$ ); CVE-2022-4804 ( $n{=}1$ ); CVE-2022-4805 ( $n{=}3$ ); CVE-2022-4806 ( $n{=}1$ ); CVE-2022-4807 ( $n{=}2$ ); CVE-2022-4808 ( $n{=}1$ ); CVE-2022-4809 ( $n{=}1$ ); CVE-2022-4810 ( $n{=}1$ ); CVE-2022-4811 ( $n{=}2$ ); CVE-2022-4812 ( $n{=}1$ ); CVE-2022-4813 ( $n{=}1$ ); CVE-2022-4814 ( $n{=}1$ )
Nextcloud (PHP)	CVE-2021-32688 ( $n{=}2$ ); CVE-2024-52514 ( $n{=}1$ )

Appendix

Table 11. The value of

n

(the number of frozen vulnerability units) for each vulnerability in BAC-Bench .

Access Control

Access control specifies and enforces who or what may access data and perform actions, under which conditions.

See all 72 papers

Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation
Mar 10, 2026Amir Al-MaamariAutomated Program Repair Software Security
University of Passau, Germany
Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen
Sep 22, 2026Om Nepal, Sushant Aryal, Oluseyi Olukola +1Automated Program Repair LLMs for Cybersecurity
School of Computing Sciences and Computer Engineering, The University of Southern Mississippi, Hattiesburg, MS 39406, USA
SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
May 26, 2026Hwiwon Lee, Jiawei Liu, Dongjun Kim +3Software Vulnerability Detection Software Security
University of Illinois Urbana-Champaign

Mar 10, 2026J/K move · Enter open · S save

Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation

Amir Al-Maamari

University of Passau, Germany

Automated Program Repair Software Security LLM Reliability

Open paper AskPDF arXiv

Large Language Models (LLMs) show promise for Automated Program Repair (APR), yet their effectiveness on security vulnerabilities remains poorly characterized. This study analyzes 319 LLM-generated security patchesacross 64 Java vulnerabilities from the Vul4J benchmark. Using tri-axis evaluation (compilation, security via PoV tests, functionality via test suites), the analysis reveals that only 24.8% of patches achieve full correctness, while 51.4% fail both security and functionality. The dominant failure mode is semantic misunderstanding: LLMs produce syntactically valid code but apply incorrect repair strategies. The proposed Security Repair Score (SRS) quantifies this gap, showing LLMs preserve functionality (mean 0.832) but struggle with security (mean 0.251). Vulnerability type strongly predicts difficulty, with fix rates ranging from 0% (input validation) to 45% (infinite loop). These findings demonstrate that LLM security patches require rigorous validation before deployment.

Abstract

Access Control

Explore similar work

Why LLMs Fail: A Failure Analysis and Partial Success Measurement for Automated Security Patch Generation

Metrics Failure in LLM-Based Code Vulnerability Repair: An Empirical Study and a Change-Aware Screen

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?