Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair

Veljko Skarich

cs.SEOct 15, 2026

Poison Valleys: Diagnosing Path-Dependent Failure in LLM Code Repair

Authors: Veljko Skarich

Abstract

When code-repair agents fail, final tests reveal that a patch is wrong but not which earlier commitment caused it or which software obligation failed. Poison Valleys traces this path in 1,844 model requests on two synthetic Python cases: PersistQueue adds atomic batch claiming to a disk-backed queue, while Dijkstra adds source-sensitive filtering to shortest-path search. We define plausible but incorrect software mechanisms, place each in a model's hypothesis, implementation plan, or both, and evaluate the generated code with an Oracle and targeted diagnostic coordinates. Moving the misconception from hypothesis to plan increased failure by 32.2% in PersistQueue and 28.9% in Dijkstra after matching the experimental mix. Different misconceptions produced distinct, model-dependent patterns of violated obligations, although several could converge on the same behavior. This creates a measured connection from upstream misconception to code mechanism and failure form. Code, data, and demo: https://poisonvalleys.org.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Family	False hypothesis and plan commitment	Defeating behavioral distinction
A01: outer rollback	The outer batch transaction is assumed to be the only uncommitted unit. The plan explicitly rolls it back after a database error while leaving the nested status helpers unchanged.	Nested helpers can already have committed earlier status writes. Rolling back the wrapper therefore leaves a claimed prefix visible after the call fails.
A02: compensating reset	Returning every staged row from unacknowledged to ready is treated as equivalent to rollback because the final status values match. The plan performs reverse status writes after an error.	Compensation is itself committed behavior. An external observer can see ready–unacknowledged–ready transitions, and concurrent actors can interact with the transiently claimed rows.
A03: ownerless transaction	Existing instance locks are assumed to imply that every helper called while a queue-level transaction field is populated belongs to the active batch. The plan routes selection and writes through that field without binding its owner.	An unrelated same-instance thread can borrow the active transaction, coupling its write to another caller’s commit or rollback; reopen then exposes the cross-thread state error.
A04: read visibility	Only writes are considered relevant to transaction ownership. The plan routes decorated writes into the batch transaction but leaves selection on the stale reader connection because reads are assumed harmless.	Selection does not see the transaction’s own updates. A successful loop can repeat or lose record identities, and concurrent claimers cease to be serializable even though the writes share a transaction.
A05: preflight validation	The batch is diagnosed as entering mutation before every row-level status statement is known to be admissible. The plan preflights selected statements with SQL EXPLAIN before running the existing per-item loop.	Static preflight cannot predict a later execution or serialization failure. Once a real later update fails, earlier per-item commits can remain durable. This shares a qualified committed-prefix witness signature with A01 and A07.
A07: lock-only serialization	Inconsistent lock lifetime is blamed for partial claims. The plan holds the instance transaction lock across the entire existing claim loop but preserves the selection and status-helper implementation.	Serialization prevents one class of interleaving but does not make individually committed writes atomic. A single-threaded later failure still leaves a prefix, and a second queue instance is outside the instance-local lock.

Appendix

Table 1

Family	False hypothesis and plan commitment	Defeating behavioral distinction
B01: post-filter	The core search is assumed correct and only its public projection is blamed. The plan runs ordinary multi-source Dijkstra, reconstructs winners, and removes paths that fail source_filter afterward.	The node-keyed search may already have pruned a more expensive but permitted source alternative. Deleting the invalid winner cannot reconstruct the valid route, in either target or all-target output.
B02: current-node origin	The callback’s first argument is treated as the node whose adjacency is being expanded rather than the depot that began the path. The plan passes the current node as both policy source and edge source.	The policy then changes meaning at every hop. A single-source reference route and a shared-junction route distinguish fixed path origin from current search location.
B03: first-edge only	The source policy is treated as an admission rule for leaving a depot. The plan checks only the first edge and runs ordinary Dijkstra for the remainder of the path.	A later forbidden edge becomes traversable after an allowed first hop. The source-switching trap exposes this first-hop origin erasure and its effects on target and all-target routing.
B04: partial source state	Only tentative collisions are blamed. The plan keys provisional distances and paths by $(origin,node)$ but still finalizes and prunes all alternatives by node.	Finalized dominance remains invalid because origin still changes future edge permission. A later legal continuation can require the source-labeled arrival discarded at the shared node.
B05: paths-only state	Distances are assumed correct while reconstructed paths alone are said to lose the depot. The plan stores source-indexed paths but retains one node-level distance, finalization decision, and heap entry.	Path identity and distance identity become incoherent: the source used for filtering or publication need not own the distance that won the node-level competition. Branching and source-order probes expose the mismatch.
B06: super-source	The search is modeled as a zero-cost synthetic super-source whose branch label matters only on the first real edge. The plan erases that label afterward and permits a later edge if any depot would permit it.	Future permission still depends on the original branch. Its qualification witness signature is grouped with B03 under first-hop origin erasure, but it reaches the defect by unioning permissions after a synthetic-source construction.

Appendix

Table 2

Prompt Clause	Tests
NetworkX recently added a source_filter option to multi_source_dijkstra , … _path , and _path_length …	single_source_reference_route ; two_junction_all_target_projection ; graph_and_multigraph_projection ; source_identity_or_graph_mutation ; medium_branching_projection ; callback_exception_propagation
origin remains that fixed depot for the complete path …	shared_junction_combined_distances ; shared_junction_combined_paths ; shared_junction_source_order_invariance ; shared_junction_target_projection ; source_switching_trap ; two_junction_all_target_projection ; two_junction_target_projection ; false_success_all_target_projection ; false_success_target_projection ; late_all_origins_denied
A truthy result permits the adjacency …and a falsey result hides it.	source_switching_trap ; false_success_all_target_projection ; false_success_target_projection ; two_junction_all_target_projection ; two_junction_target_projection ; shared_junction_combined_distances ; shared_junction_combined_paths ; single_source_reference_route ; late_all_origins_denied
On multigraphs, edge_data has the same adjacency-data shape supplied to a callable weight.	graph_and_multigraph_projection , specifically _graph_classes and _parallel_edges
The filter must compose with string or callable weights, including hidden edges …	weight_and_cutoff_projection : missing/default and callable weights, hidden edges, zero weights, and cycles; filtered negative-weight probe
…cutoff …	_cutoff_boundaries within weight_and_cutoff_projection

Appendix

Table 3

Prompt Clause	Tests
SQLiteAckQueue.claim_batch(max_items, raw=False) should claim up to the requested number of ready records as one public operation.	_batch_claim_contract_case ; _success_case ; _bulk_identity_case
If the call raises before returning a result, no record may have been consumed or left claimed.	_replacement_compensation_case ; _retry_after_failure_case ; _instance_lock_failure_case ; _cross_instance_failure_case ; _concurrency_case
A database error during one batch caused the call to raise without returning records.	The same injected-database-failure families: _replacement_compensation_case ; _retry_after_failure_case ; _instance_lock_failure_case ; _cross_instance_failure_case ; _concurrency_case
Afterwards, qsize() and the persisted ready count disagreed.	_success_case ; _concurrent_success_case ; _cross_instance_failure_case ; _concurrency_case , especially their persisted-row and public-count assertions
Reopening with automatic resumption disabled showed records that were ready before the call but were unavailable to a later batch claim.	_replacement_compensation_case ; _retry_after_failure_case ; _success_case ; _bulk_identity_case ; _cross_instance_failure_case ; _concurrency_case
FIFO and FILO order	Every per-queue function; _filo_sqlite_order_case ; order portions of _success_case and _bulk_identity_case

Appendix

Table 4

Task	Primary category	Specific subcategory	Count
Dijkstra	NO_CANDIDATE_ARTIFACT	MISSING_CANDIDATE_SOURCE	64
Dijkstra	ORACLE_ASSERTION_FAILURE	false_success_target_projection	1
Dijkstra	ORACLE_ASSERTION_FAILURE	medium_branching_projection	1
Dijkstra	ORACLE_ASSERTION_FAILURE	networkx_3_6_error_regression	166
Dijkstra	ORACLE_ASSERTION_FAILURE	shared_junction_combined_distances	264
Dijkstra	ORACLE_ASSERTION_FAILURE	shared_junction_combined_paths	1

Appendix

Table 5

Agent Failure Analysis

Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.

See all 152 papers

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair
Oct 11, 2026Elad Sharony, Ido Greenberg, Jialin Song +3Automated Program Repair
Technion · NVIDIA
Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse
Jul 5, 2026Raj Jaiswal, Anany Singh Divy, Savar Bhasin +3LLM Self-Correction Automated Program Repair
IIIT Delhi · Microsoft Research India · IIT Kanpur
Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code
Mar 25, 2026Myeongsoo Kim, Dingmin Wang, Siwei Cui +6Agent Failure Analysis AI Coding Agents
AWS AI Labs · Monash University

Oct 11, 2026J/K move · Enter open · S save

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair

Elad Sharony, Ido Greenberg, Jialin Song, Shie Mannor+2

Technion · NVIDIA

Automated Program Repair

Open paper AskPDF arXiv

Large language model (LLM) code-repair systems fix faulty programs by repeatedly generating and testing revisions. Each revision is costly, making the choice of which repair to pursue important for efficiency. Existing search policies rely on test pass rates and search statistics, ignoring the details of test failures and proposed edits. We introduce CodeTriage, a learned policy that prioritizes edits before their revised programs are generated and tested. Its 0.6B-parameter model reads the test output, failure diagnosis, and proposed edit, then predicts a distribution over the number of repair steps needed to reach a fix through that edit. These predictions prioritize all pending edits across programs before generating and testing the revised code. On held-out repair tasks, CodeTriage finds fixes with 21% fewer program revisions and 12% fewer output tokens than the strongest baseline on each metric. Without retraining, it achieves the highest average success across search budgets for each of 11 unseen repair LLMs. Ablations show that both the failure context and edit text, as well as comparisons across programs, improve search. We also release a corpus of 12.2 million program versions from four LLMs, organized into repair trees with proposed edits and test results.

Abstract

Agent Failure Analysis

Explore similar work

Make Every Expansion Count: Learning to Search Efficiently in LLM Code Repair

Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse

Coherence Collapse: Diagnosing Why Code Agents Fail After Reaching the Right Code