VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?

Srijan Bansal; Jiao Fangkai; Yilun Zhou; Austin Xu; Shafiq Joty; Semih Yavuz

cs.SEMar 16, 2026

VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?

Authors: Srijan Bansal, Jiao Fangkai, Yilun Zhou, Austin Xu, Shafiq Joty, Semih Yavuz

Organizations: Salesforce AI Research · Nanyang Technological University · A*STAR

Abstract

As Large Language Models shift the programming toward human-guided ''vibe coding'', agentic coding tools increasingly rely on models to self-diagnose and repair their own subtle faults -- a capability central to autonomous software engineering yet never systematically evaluated. We present \name{}, the first empirical decomposition that jointly evaluates two coupled tasks: Fault-Triggering Test Generation (FT-Test) constructing a discriminative witness that exposes a latent bug, and Fault-targeted Program Repair (FPR), repairing it under varying diagnostic conditions. \name{} pairs competitive programming problems with LLM-generated solutions that pass partial test suites but fail on semantic edge cases, enabling controlled identification of where the diagnostic chain breaks down. Evaluating 12 frontier LLMs, we find that fault-targeted reasoning does not scale with general coding ability. Models produce syntactically valid test inputs at near-ceiling rates yet collapse on discriminative generation, with fault hypothesis generation -- not output validation -- as the dominant bottleneck. Test-guided repair reveals a complementary insight: when self-generated tests successfully witness a fault, the resulting repair matches or outperforms repair guided by externally provided tests, but tests that fail to witness the fault actively degrade repair below unguided baselines. Together, these results reframe the challenge of autonomous debugging: the binding bottleneck is not code synthesis or test validity but fault-target reasoning, a capability that remains deficient across all frontier models. As Large Language Models shift the programming toward human-guided ''vibe coding'', agentic coding tools increasingly rely on models to self-diagnose and repair their own subtle faults -- a capability central to autonomous software engineering yet never systematically evaluated.

Figures & tables

Figure 1: VibePass evaluates LLM performance across roles requiring fault-targeted reasoning, as typical in practical coding agents. Given a problem description and a buggy solution, the LLM ( Judge ) first determines whether a bug exists. If a bug is detected, the LLM ( Tester ) generates a fault-triggering (FT) test, consisting of an input and expected output (FT-Test Bug Discovery) . An FT-test is correct if it satisfies three conditions: the input is valid, the buggy solution fails the test, and a silver solution passes it. The FT-test is then used by the LLM ( Debugger ) to produce a revised solution (Fault-Targeted Program Repair) , which must pass the official test suite to be considered a valid fix.

Model	CG	Bug-Aware FT-Test				Bug-Discovery FT-Test
	$P_{@1}$	$V_{I}$	$V_{IO}$	$D_{I}$	$D_{IO}$	$J$	J+ $V_{I}$	J+ $V_{IO}$	J+ $D_{I}$	J+ $D_{IO}$
Gemini-3.1 Flash-Lite	25.0	89.6	69.4	31.8	26.0	39.9	33.0	24.3	20.8	15.6
Gemini-3 Flash	78.9	95.4	92.5	63.6	63.0	75.1	74.0	69.4	60.1	59.0
Gemini-3 Pro	81.6	94.8	89.6	72.3	69.4	83.2	80.4	71.1	64.2	59.5
Gemini-3.1 Pro	92.1	89.0	83.2	69.9	68.8	80.4	77.5	59.5	59.5	48.0
GPT-5 (nano)	48.7	92.5	82.1	57.8	52.0	54.9	52.0	28.9	35.3	17.9

Table 1: Bug-Aware and Bug-Discovery FT-Test Generation Performance. In the Bug-Aware, the model knows the code is buggy, and performance measures FT-Test generation quality directly:

V_{I}

V_{IO}

for input/output validity, and

D_{I}

D_{IO}

for discriminative effectiveness. In the Bug-Discovery, J denotes judgment accuracy (buggy or not), and joint metrics (J+*) require both correct judgment and the corresponding quality criterion. Pass@1 is also reported to compare FT-Test with code generation (CG) reasoning ability.

Model	FT-Test discovery		FT-Test aware		Agreement
	$J$	$J+D_{IO}/J$	${D_{IO}}/J$	${D_{IO}}/\sim J$	$J+D_{IO}\cap D_{IO}$
Gemini-3.1 Flash-Lite	39.9	39.1	34.8	20.2	87.0
Gemini-3 Flash	75.1	78.5	73.8	30.2	87.7
Gemini-3 Pro	83.2	71.5	75.0	41.4	84.0
Gemini-3.1 Pro	80.3	59.7	73.4	50.0	73.4
GPT-5 (nano)	54.9	32.6	66.3	34.6	57.9

Table 2: Model performance on FT-Test Generation under Bug-Discovery and Bug-Aware settings.

J

denotes judgment accuracy;

J+D_{IO}/J

measures joint success in discovery. In Bug-Aware,

D_{IO}

is split by correct (

D_{IO}/J

) and incorrect (

D_{IO}/\sim J

) verdicts. Agreement between discriminative test outcomes across settings is also reported.

Model	CG	No	Int	Ext	NoTest	IntTest			ExtTest
	$P_{@1}$				$SR$	$V_{IO}$	$D_{IO}$	$SR$	$V_{IO}$	$D_{IO}$	$SR$
Gemini-3.1 Flash-Lite	25.0	45.7	23.4	48.1	29.0	70.5	31.5	15.5	67.5	22.5	29.5
Gemini-3 Flash	78.9	86.1	66.8	82.5	70.5	79.5	62.0	56.0	85.0	55.5	71.0
Gemini-3 Pro	81.6	80.4	64.5	78.6	68.5	86.5	70.1	53.5	90.0	65.5	67.0
GPT-5 (nano)	48.7	38.3	35.6	39.2	24.0	65.0	47.0	23.5	81.0	49.0	28.5
GPT-5 (mini)	43.4	45.2	52.9	51.6	35.0	60.5	47.0	37.5	78.5	58.0	39.0

Table 3: NoTest , IntTest , and ExtTest Fault-Targeted Program Repair Performance. The first four columns ( left ) report P@1 under different settings. The remaining columns ( right ) show Success Rate (SR),

V_{IO}

, and

D_{IO}

(input/output validity and discriminative effectiveness) for internally generated or externally provided tests. Pass@1 for code generation (CG) is also included for comparison. ”No”, ”Int”, and ”Ext” correspond to NoTest, IntTest, and ExtTest FPR settings, respectively.

Figure 2: Controlled comparison of test feedback mechanisms on valid corner-case intersection . Debugging performance is shown for samples where both external (Task-1) and self-generated (Task-3) tests are valid. Left : Success rates under NoTest (gray), ExtTest (blue), and IntTest (red) across 11 models. Right : Performance difference ( IntTest − ExtTest ) per model, with red bars favoring self-generated tests and blue bars favoring external tests.

Figure 3: Pipeline Stage Correlations and Model Family Performance Across Cumulative Requirements. Using VibePass instances, we evaluate 12 frontier LLMs across a progression of coding-reasoning tasks: input generation ( Valid Input ), output prediction ( Valid-IO ), fault-triggering discrimination ( FT-Input , FT-IO ), and program Repair , with Judge metrics assessing final correctness. [Left] Pearson correlations reveal that fault-triggering metrics ( FT-Input – FT-IO , $r=0.988$ ) and their relationship to Judge performance ( $r\geq 0.86$ ) are the strongest predictors of success, while Valid Input alone weakly predicts downstream results ( $r=0.311$ ). This suggests that the ability to generate fault-revealing tests—rather than mere syntactic validity—is more closely aligned with solving complex bugs. [Right] Cumulative success rates show the largest performance drops at the Valid IO $\to$ FT-Input (14.7 pp) and FT-IO $\to$ Repair (21.2 pp) transitions, identifying these as the primary reasoning bottlenecks. Family-level trends highlight that while OpenAI models maintain the highest stability (54.3%), Google models struggle with fault-triggering tasks and open-source models underperform significantly in repair (12.1%). VibePass maps the full spectrum from basic test generation to advanced debugging, exposing critical gaps in current model capabilities.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Example prompt and corresponding model output for input-validity generation.

Statistic	Value
Total instances (problem, gold, buggy tuples)	173
Unique problems	76
Avg. instances per problem	2.3
Problem Difficulty (platform labels)
Easy	1%
Medium	25%

Appendix

Table 4: Dataset statistics for the VibePass benchmark.

LLM Self-Correction

Large language model self-correction studies how language models detect and repair errors in their own answers, reasoning, and actions.

See all 159 papers

If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs
Sep 9, 2026Xietao Wang-Lin, Anton Isopoussu, Louis MahonLLM Self-Correction Automated Program Repair
University of Warwick · UnlikelyAI
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?
Apr 19, 2026Miaosen Chai, Wang Bill Zhu, Shangshang Wang +5LLM Evaluation Automated Program Repair
University of Southern California · Microsoft · University of Wisconsin–Madison +1
Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse
Jul 5, 2026Raj Jaiswal, Anany Singh Divy, Savar Bhasin +3LLM Self-Correction Automated Program Repair
IIIT Delhi · Microsoft Research India · IIT Kanpur

Sep 9, 2026J/K move · Enter open · S save

If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs

Xietao Wang-Lin, Anton Isopoussu, Louis Mahon

University of Warwick · UnlikelyAI

LLM Self-Correction Automated Program Repair LLM Reliability

Open paper AskPDF arXiv

Large language models (LLMs) have become ubiquitous in software development, with LLM-based automated program repair tools increasingly used during code review. In this report, we explore the iterative blind use of LLMs as bug-fixers. Across multiple models and repair environments, we find that LLMs consistently claim to detect bugs in entirely bug-free programs while the rate of repair of buggy programs is less than that of the damage to correct programs. We also explore the long-term dynamics of this iterative process, and find that this frequently reaches a pseudo-bug-fixing cycle where the same changes are added and removed again ad infinitum. Lastly, via mechanistic probing, we unveil the existence of a steering vector which controls the editing propensity, suggesting that LLMs have an internal representation of ``buggy code", and that this representation is what is falsely activated to induce pseudo-bug fixing. These results provide insight towards the dynamics of fully autonomous bug-fixing systems, as well as stopping conditions under ambiguous goals.

Abstract

LLM Self-Correction

Explore similar work

If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse