Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes

Utku Boran Torun; Veli Karakaya; Eray Tüzün

cs.SEOct 8, 2026

Can LLMs Fix It Without Code? Toward Automated Verification of No-Code Bug Fixes

Authors: Utku Boran Torun, Veli Karakaya, Eray Tüzün

Organizations: Bilkent University, Ankara, Turkey

Abstract

A no-code fix resolves an invalid bug report by directing the user to change a setting, update to a version where the problem is already fixed, or adjust their workflow. Manually verifying whether a proposed no-code fix resolves the reported bug takes considerable developer time. This study proposes an automated, execution-based pipeline for evaluating the capability of large language models (LLMs) to generate no-code fixes in a real browser environment. We evaluate 322 no-code fixes generated by the 12 configurations released with the benchmark of a previous study, covering bug reports categorized as Faulty Configuration, Wrong Version, or External System & Dependency. An executor agent applies each fix by following its natural-language instructions, and an issue-specific checker determines whether the reported bug persists. We repeat the pipeline with three executors: two Computer-Use Agents, OpenCUA-72B and Claude Sonnet 5, and one multimodal agentic LLM, Meta's Muse Glimmer. Only 17.6% of the candidate issues could be set up and passed both sanity gates. Across the 322 fixes, 14.6% to 49.7% resolved the bug depending on the executor, and the strongest configuration, Claude Opus 4.6 in the Vanilla pipeline, resolved up to 74.1% of its fixes under Claude Sonnet 5. Changing only the executor shifted a configuration's resolution rate by 38.8% on average, and the three executors reached the same verdict on only 46.9% of the fixes. Compared with human execution, the executors matched the human consensus for 66.1% to 88.1% of the sampled fixes. Even under the best executor, fewer than half of the LLM-generated no-code fixes resolve the reported bug, so such fixes need verification before they reach users. Execution-based verification can provide this, but the measured capability depends strongly on the executor, which evaluations must report and control.

Figures & tables

	Backbone LLM
Pipeline	Gemini 3.1 Pro Preview	GPT-5.4	Claude Opus 4.6	GLM-5	MiniMax M2.7	Kimi K2.5
Vanilla	V-Gemini	V-GPT	V-Claude	V-GLM	V-MiniMax	V-Kimi
Ablation (no subclass prior)	Abl-Gemini	–	–	–	–	Abl-Kimi
RAG	RAG-Gemini	–	–	–	–	RAG-Kimi
Agentic web search	Agentic-Gemini	–	–	–	–	Agentic-Kimi

Table 1 . The 12 LLM-based fix-generation configurations reused from Gön et al. ( Gon et al., 2026 ) , with the short identifiers used throughout this paper. Rows are pipelines and columns are backbone LLMs; a dash marks a combination that was not run.

Figure 1 . Overview of the evaluation pipeline. Flow diagram: a bug report, the maintainer's fix and 12 LLM fixes enter a Docker environment; two sanity checks bracket a rollback from fixed to buggy state; each LLM fix is applied by an executor and checked.

Executor	Resolution (%)	Applicability (%)	Resolution among applicable (%)
Claude Sonnet 5	49.7 (160)	66.1 (213)	69.5 (148)
OpenCUA-72B	45.7 (147)	99.1 (319)	45.8 (146)
Muse Glimmer	14.6 (47)	70.5 (227)	18.1 (41)

Table 2 . Resolution and applicability per executor over the 322 fixes (fix counts in parentheses). The last column restricts the resolution rate to the fixes the executor found applicable.

		Vanilla						Ablation		RAG		Agentic
		Gem.	GPT	Cla.	GLM	MiniM.	Kimi	Gem.	Kimi	Gem.	Kimi	Gem.	Kimi
Res.	S5	44.4	40.7	74.1	44.4	25.9	44.4	59.3	51.9	48.1	44.4	63.0	56.0
	OC	51.9	48.1	51.9	37.0	44.4	33.3	44.4	40.7	48.1	55.6	55.6	36.0
	MG	11.1	11.1	25.9	18.5	11.1	0.0	14.8	11.1	7.4	11.1	25.9	28.0
App.	S5	55.6	55.6	88.9	51.9	44.4	66.7	70.4	74.1	63.0	70.4	81.5	72.0
	OC	100.0	96.3	100.0	100.0	100.0	100.0	100.0	96.3	96.3	100.0	100.0	100.0

Table 3 . Resolution (Res.) and applicability (App.) rate (%) per configuration and executor. Each configuration has 27 fixes (Agentic-Kimi: 25). Backbones: Gem. = Gemini 3.1 Pro, Cla. = Claude Opus 4.6, MiniM. = MiniMax M2.7. Executors: S5 = Sonnet 5, OC = OpenCUA-72B, MG = Muse Glimmer.

			Sonnet 5		OpenCUA-72B		Muse Glimmer
Subclass	Iss.	Fixes	Res.	App.	Res.	App.	Res.	App.
Faulty Configuration	5	178	43.8	66.3	65.7	98.9	13.5	69.7
Wrong Version	1	36	50.0	44.4	8.3	100.0	0.0	83.3
External System	3	108	59.3	73.1	25.0	99.1	21.3	67.6

Table 4 . Resolution and applicability rate (%) per subclass.

Executor	Run 1	Run 2	Run 3	Any run	All runs agree
Claude Sonnet 5	136 (42.2%)	125 (38.8%)	130 (40.4%)	160 (49.7%)	82.3%
OpenCUA-72B	100 (31.1%)	109 (33.9%)	100 (31.1%)	147 (45.7%)	73.9%
Muse Glimmer	34 (10.6%)	35 (10.9%)	32 (9.9%)	47 (14.6%)	92.2%

Table 5 . Resolved fixes per individual executor run, fixes resolved in at least one run, and the share of fixes on which all three runs agreed (

n=322

	Faulty Configuration					Wrong Version	External System
Issue	20537	22987	25057	37372	45912	25379	22252	24747	31108	Total
Fixes	36	36	35	36	35	36	36	36	36	322
Sonnet 5	9	13	10	20	26	18	16	19	29	160
OpenCUA-72B	19	20	19	36	23	3	12	0	15	147
Muse Glimmer	3	9	0	12	0	0	10	0	13	47

Table 6 . Resolved fixes per issue and executor. Column headers are Brave GitHub issue numbers, grouped by subclass.

Executor	Human 1	Human 2	Consensus	$\kappa$ (consensus)
Claude Sonnet 5	81.3 [71.1, 88.5]	78.7 [68.1, 86.4]	88.1 [77.5, 94.1]	0.74
OpenCUA-72B	68.0 [56.8, 77.5]	57.3 [46.1, 67.9]	66.1 [53.4, 76.9]	0.28
Muse Glimmer	78.7 [68.1, 86.4]	68.0 [56.8, 77.5]	79.7 [67.7, 88.0]	0.48
Human 1 vs. Human 2	78.7 [68.1, 86.4]			0.54

Table 7 . Agreement (%) between each executor and human execution on the 75 randomly sampled fixes, with 95% Wilson confidence intervals. Consensus: the 59 fixes on which both humans reached the same outcome.

Explore similar work

Sep 9, 2026cs.SE

If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs

Large language models (LLMs) have become ubiquitous in software development, with LLM-based automated program repair tools increasingly used during code review. In this report, we explore the iterative blind use of LLMs as bug-fixers. Across multiple models and repair environments, we find that LLMs consistently claim to detect bugs in entirely bug-free programs while the rate of repair of buggy programs is less than that of the damage to correct programs. We also explore the long-term dynamics of this iterative process, and find that this frequently reaches a pseudo-bug-fixing cycle where the same changes are added and removed again ad infinitum. Lastly, via mechanistic probing, we unveil the existence of a steering vector which controls the editing propensity, suggesting that LLMs have an internal representation of ``buggy code", and that this representation is what is falsely activated to induce pseudo-bug fixing. These results provide insight towards the dynamics of fully autonomous bug-fixing systems, as well as stopping conditions under ambiguous goals.

Xietao Wang-Lin, Anton Isopoussu, Louis Mahon

University of Warwick · UnlikelyAI

LLM Self-Correction Automated Program Repair LLM Reliability

Ask PDF

May 17, 2026cs.SE

Automated Root-Cause Subclassification and No-Code Fix Generation for Invalid Bug Reports

Issues faced when using software are reported in the form of bug reports. However, many bug reports are invalid, meaning they do not require code changes, and are resolved with a no-code fix. Manually determining the root cause of the invalid bug reports and providing actionable resolutions by the customer support causes a serious waste of resources. Our goal is to introduce a standardized taxonomy for root-cause oriented invalid bug report subclassification, and perform experiments to test the accuracy of various approaches on invalid subclassification and no-code fix generation. We study how different configurations perform on a gold-standard benchmark we have created. Using a manually curated benchmark for higher quality analysis, we experimented with vanilla LLMs, Retrieval Augmented Generation, and agentic web search to identify invalid subclasses and generate no-code fixes. We evaluated the results against manually labeled ground truth data that includes the invalid subclass and no-code fixes from the original bug reports. We measured subclass detection performance with weighted F1-Score, and assessed no-code fix suggestions using BERTScore and Judge LLM success rates. For subclassification, retrieval augmented generation achieves the highest overall performance with 0.66 weighted F1, slightly outperforming vanilla LLMs at 0.65 and agentic web search at 0.64. At the subclass level, performance peaks at 0.85 F1 for Non-reproducibility and 0.79 for Feature Request and Question, while Wrong Version remains the most challenging with scores between 0.00 and 0.29. For no-code fix generation, agentic web search achieves the highest overall Judge LLM success rate at 68.9%, compared to 64.4% for RAG applications and 64.9% for vanilla LLMs, with subclass-level peaks of 87.4% for Working as Designed and 72.2% for Question.

Mahmut Furkan Gon, Emre Dinc, Tevfik Emre Sungur +1

Department of Computer Engineering, Bilkent University, Ankara, T̈urkiye.

Customer Support Automation Root Cause Analysis

Ask PDF

Sep 24, 2026cs.CL

Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?

Recent studies have shown that Large Language Models can effectively solve problems and fix bugs in diverse programming environments, including competitive programming. Existing approaches primarily evaluate LLM performance in problem solving or bug fixing independently, but do not explore the relationship between these two capabilities. This work focuses on determining how much the LLM deviates from a buggy solution to fix the bug compared to a human-written patch, and if there is a bias towards generating entirely new solutions. We construct a dataset with all the submissions (

\sim

3000) from a couple of users from Codeforces, and we match each buggy submission with its corresponding human fix. By using the similarity between the buggy solution and the human fix as a baseline, we evaluate the quality of LLM-generated bug fixes on 3 OpenAI GPT models (gpt-5-nano, gpt-5-mini, gpt-5.1). We check if the generated solutions solve the problem by using the Codeforces-R1 dataset, an openly available dataset that has tests generated with the DeepSeek-R1 model. Our findings suggest that LLMs tend to modify more lines than necessary compared to human fixes and, in some cases, generate entirely new solutions. We also observe that LLMs solve more problems correctly when allowed to generate solutions from scratch rather than patch buggy submissions, even when those submissions are close to the human patch. This has important implications for the design of AI-assisted programming tools, particularly in supporting user debugging processes and promoting incremental problem-solving strategies rather than solution replacement.

Alexandru Stefan Stoica, Traian Rebedea, Marian Cristian Mihaescu

University Politehnica of Bucharest Bucharest, Romania · University of Craiova Craiova, Romania

LLM Evaluation Automated Program Repair LLM Code Generation

Ask PDF