As Large Language Models shift the programming toward human-guided ''vibe coding'', agentic coding tools increasingly rely on models to self-diagnose and repair their own subtle faults -- a capability central to autonomous software engineering yet never systematically evaluated. We present \name{}, the first empirical decomposition that jointly evaluates two coupled tasks: Fault-Triggering Test Generation (FT-Test) constructing a discriminative witness that exposes a latent bug, and Fault-targeted Program Repair (FPR), repairing it under varying diagnostic conditions. \name{} pairs competitive programming problems with LLM-generated solutions that pass partial test suites but fail on semantic edge cases, enabling controlled identification of where the diagnostic chain breaks down. Evaluating 12 frontier LLMs, we find that fault-targeted reasoning does not scale with general coding ability. Models produce syntactically valid test inputs at near-ceiling rates yet collapse on discriminative generation, with fault hypothesis generation -- not output validation -- as the dominant bottleneck. Test-guided repair reveals a complementary insight: when self-generated tests successfully witness a fault, the resulting repair matches or outperforms repair guided by externally provided tests, but tests that fail to witness the fault actively degrade repair below unguided baselines. Together, these results reframe the challenge of autonomous debugging: the binding bottleneck is not code synthesis or test validity but fault-target reasoning, a capability that remains deficient across all frontier models. As Large Language Models shift the programming toward human-guided ''vibe coding'', agentic coding tools increasingly rely on models to self-diagnose and repair their own subtle faults -- a capability central to autonomous software engineering yet never systematically evaluated.
Figures & tables
Figure 1: VibePass evaluates LLM performance across roles requiring fault-targeted reasoning, as typical in practical coding agents. Given a problem description and a buggy solution, the LLM ( Judge ) first determines whether a bug exists. If a bug is detected, the LLM ( Tester ) generates a fault-triggering (FT) test, consisting of an input and expected output (FT-Test Bug Discovery) . An FT-test is correct if it satisfies three conditions: the input is valid, the buggy solution fails the test, and a silver solution passes it. The FT-test is then used by the LLM ( Debugger ) to produce a revised solution (Fault-Targeted Program Repair) , which must pass the official test suite to be considered a valid fix.
Model
CG
Bug-Aware FT-Test
Bug-Discovery FT-Test
P@1
VI
VIO
DI
DIO
J
J+ VI
J+ VIO
J+ DI
J+ DIO
Gemini-3.1 Flash-Lite
25.0
89.6
69.4
31.8
26.0
39.9
33.0
24.3
20.8
15.6
Gemini-3 Flash
78.9
95.4
92.5
63.6
63.0
75.1
74.0
69.4
60.1
59.0
Gemini-3 Pro
81.6
94.8
89.6
72.3
69.4
83.2
80.4
71.1
64.2
59.5
Gemini-3.1 Pro
92.1
89.0
83.2
69.9
68.8
80.4
77.5
59.5
59.5
48.0
GPT-5 (nano)
48.7
92.5
82.1
57.8
52.0
54.9
52.0
28.9
35.3
17.9
Table 1: Bug-Aware and Bug-Discovery FT-Test Generation Performance. In the Bug-Aware, the model knows the code is buggy, and performance measures FT-Test generation quality directly: VI / VIO for input/output validity, and DI / DIO for discriminative effectiveness. In the Bug-Discovery, J denotes judgment accuracy (buggy or not), and joint metrics (J+*) require both correct judgment and the corresponding quality criterion. Pass@1 is also reported to compare FT-Test with code generation (CG) reasoning ability.
Model
FT-Test discovery
FT-Test aware
Agreement
J
J+DIO/J
DIO/J
DIO/∼J
J+DIO∩DIO
Gemini-3.1 Flash-Lite
39.9
39.1
34.8
20.2
87.0
Gemini-3 Flash
75.1
78.5
73.8
30.2
87.7
Gemini-3 Pro
83.2
71.5
75.0
41.4
84.0
Gemini-3.1 Pro
80.3
59.7
73.4
50.0
73.4
GPT-5 (nano)
54.9
32.6
66.3
34.6
57.9
Table 2: Model performance on FT-Test Generation under Bug-Discovery and Bug-Aware settings. J denotes judgment accuracy; J+DIO/J measures joint success in discovery. In Bug-Aware, DIO is split by correct ( DIO/J ) and incorrect ( DIO/∼J ) verdicts. Agreement between discriminative test outcomes across settings is also reported.
Model
CG
No
Int
Ext
NoTest
IntTest
ExtTest
P@1
SR
VIO
DIO
SR
VIO
DIO
SR
Gemini-3.1 Flash-Lite
25.0
45.7
23.4
48.1
29.0
70.5
31.5
15.5
67.5
22.5
29.5
Gemini-3 Flash
78.9
86.1
66.8
82.5
70.5
79.5
62.0
56.0
85.0
55.5
71.0
Gemini-3 Pro
81.6
80.4
64.5
78.6
68.5
86.5
70.1
53.5
90.0
65.5
67.0
GPT-5 (nano)
48.7
38.3
35.6
39.2
24.0
65.0
47.0
23.5
81.0
49.0
28.5
GPT-5 (mini)
43.4
45.2
52.9
51.6
35.0
60.5
47.0
37.5
78.5
58.0
39.0
Table 3: NoTest , IntTest , and ExtTest Fault-Targeted Program Repair Performance. The first four columns ( left ) report P@1 under different settings. The remaining columns ( right ) show Success Rate (SR), VIO , and DIO (input/output validity and discriminative effectiveness) for internally generated or externally provided tests. Pass@1 for code generation (CG) is also included for comparison. ”No”, ”Int”, and ”Ext” correspond to NoTest, IntTest, and ExtTest FPR settings, respectively.
Figure 2: Controlled comparison of test feedback mechanisms on valid corner-case intersection . Debugging performance is shown for samples where both external (Task-1) and self-generated (Task-3) tests are valid. Left : Success rates under NoTest (gray), ExtTest (blue), and IntTest (red) across 11 models. Right : Performance difference ( IntTest − ExtTest ) per model, with red bars favoring self-generated tests and blue bars favoring external tests.
Figure 3: Pipeline Stage Correlations and Model Family Performance Across Cumulative Requirements. Using VibePass instances, we evaluate 12 frontier LLMs across a progression of coding-reasoning tasks: input generation ( Valid Input ), output prediction ( Valid-IO ), fault-triggering discrimination ( FT-Input , FT-IO ), and program Repair , with Judge metrics assessing final correctness. [Left] Pearson correlations reveal that fault-triggering metrics ( FT-Input – FT-IO , r=0.988 ) and their relationship to Judge performance ( r≥0.86 ) are the strongest predictors of success, while Valid Input alone weakly predicts downstream results ( r=0.311 ). This suggests that the ability to generate fault-revealing tests—rather than mere syntactic validity—is more closely aligned with solving complex bugs. [Right] Cumulative success rates show the largest performance drops at the Valid IO → FT-Input (14.7 pp) and FT-IO → Repair (21.2 pp) transitions, identifying these as the primary reasoning bottlenecks. Family-level trends highlight that while OpenAI models maintain the highest stability (54.3%), Google models struggle with fault-triggering tasks and open-source models underperform significantly in repair (12.1%). VibePass maps the full spectrum from basic test generation to advanced debugging, exposing critical gaps in current model capabilities.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Example prompt and corresponding model output for input-validity generation.
Statistic
Value
Total instances (problem, gold, buggy tuples)
173
Unique problems
76
Avg. instances per problem
2.3
Problem Difficulty (platform labels)
Easy
1%
Medium
25%
Appendix
Table 4: Dataset statistics for the VibePass benchmark.
Large language models (LLMs) have become ubiquitous in software development, with LLM-based automated program repair tools increasingly used during code review. In this report, we explore the iterative blind use of LLMs as bug-fixers. Across multiple models and repair environments, we find that LLMs consistently claim to detect bugs in entirely bug-free programs while the rate of repair of buggy programs is less than that of the damage to correct programs. We also explore the long-term dynamics of this iterative process, and find that this frequently reaches a pseudo-bug-fixing cycle where the same changes are added and removed again ad infinitum. Lastly, via mechanistic probing, we unveil the existence of a steering vector which controls the editing propensity, suggesting that LLMs have an internal representation of ``buggy code", and that this representation is what is falsely activated to induce pseudo-bug fixing. These results provide insight towards the dynamics of fully autonomous bug-fixing systems, as well as stopping conditions under ambiguous goals.
Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but over-edited solutions during debugging. To evaluate how far LLMs are from precise debugging, we introduce the Precise Debugging Benchmark (PDB) framework, which automatically converts any coding dataset into a debugging benchmark with precision-aware evaluation. PDB generates buggy programs by synthesizing verified atomic bugs and composing them into multi-bug programs. We define two novel metrics, edit-level precision and bug-level recall, which measure how many necessary edits are made and how many bugs are resolved. We release two evaluation benchmarks: PDB-Single-Hard on single-line bugs, and PDB-Multi on multi-line bugs. Experiments show that frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%, even when explicitly instructed to perform minimal debugging. Finally, we show that iterative and agentic debugging strategies do not substantially improve precision or recall, highlighting the need to rethink post-training pipelines for coding models.
Miaosen Chai, Wang Bill Zhu, Shangshang Wang +5
University of Southern California · Microsoft · University of Wisconsin–Madison +1
Code language models are now trusted collaborators in production workflows for debugging, refactoring, and iterative repair, and every benchmark that evaluates them assumes the instructions they act on are correct. We study what happens when that assumption breaks. We evaluate code language models across four experiments designed to assess whether models resist or obey incorrect instructions in single-pass and iterative repair settings, using the RunBugRun dataset of algorithmic Python problems with deterministic test cases. Our findings reveal a striking behavioral pattern: models correctly identify an incorrect instruction as wrong, then follow it anyway. This compliance unknowingly introduces errors beyond the original bug, and the corrupted code state cannot be recovered through subsequent self-guided iterative repair, which fails to converge across passes. We term this Blind Obedience, characterize the Ghost (Unknown) Errors it introduces, quantify the proportion of cases where semantic corruption proves irrecoverable, and show that extended reasoning cannot reverse it. These findings surface behavioral properties invisible to pass-rate evaluation, with direct consequences for code language models deployed in production settings.
Raj Jaiswal, Anany Singh Divy, Savar Bhasin +3
IIIT Delhi · Microsoft Research India · IIT Kanpur