Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations
Authors: Xiaobo Chen, Genliang Zhu, Ziyuan Wang, Chu Wang
Organizations: University of Massachusetts Amherst, Amherst, Massachusetts, USA · Accentrust, Vancouver, Canada · Georgia Institute of Technology, School of Computer Science, Atlanta, Georgia, USA · Duke University, Durham, North Carolina, USA · University of Illinois Urbana-Champaign, Department of Computer Science, Urbana, Illinois, USA
After a runtime or API migration, an LLM-repaired procedure can run cleanly yet return wrong results, and a release check on typical test inputs ships it. We report a preregistered study of 73 tasks covering 60 documented changes in CPython, Node.js and SQLite. Every candidate is executed on the real runtimes against withheld contracts. Over the complete repair space, the all-assertions gate on typical inputs released 0.410 of wrong repairs. One edge input written from the release note cut this to 0.065, and to 0.061 on 15 unseen clusters. Three inputs from a blind GPT-5.6-luna writer that never saw the program reached 0.070. Two further frozen stages enlarged the menu to 1491 fragments and then added 233 hard distractors that match the legacy output on typical inputs. The edge input (0.058) and the frontier-model writer (0.058) passed their preregistered tests. Yet 0.168 and 0.153 of the hard wrong programs still passed. Two local-model writers failed their preregistered tests. Across four model families, the edge input cut wrong releases per attempt in all three open-weight families on all three menus, for example from 0.233 to 0.057, without lowering correct completion. Self-tests helped less. A blind AI audit flagged 34 contracts as stricter than their requirements. Dropping the disputed checks changed 0 verdicts, but class-label disputes remain open. No model-written code was executed. Artifacts are available from the authors on request.
Figures & tables
Visible inputs
Added
FA
FR
Stale
typical only
0.00
0.410
0.010
34/52
+ blind writer, k=1
0.96
0.130
0.010
49/52
+ blind writer, k=2
1.85
0.075
0.010
50/52
+ blind writer, k=3
2.46
0.070
0.010
50/52
+ guarded writer, k=1
0.73
0.206
0.010
42/52
+ guarded writer, k=2
1.40
0.161
0.010
42/52
Table 1. Model-free release-gate adequacy over the complete finite repair space of the 52 affected test tasks (49 clusters), all-consumer-assertions gate. Added: mean visible inputs added to the typical ones. FA: share of hidden-wrong procedures released; FR: share of hidden-correct procedures blocked (task means). Stale: tasks in which the unrepaired procedure fails visibly.
Figure 1. Headline result. (a) Share of wrong procedures released over the complete finite repair space as blind-writer inputs are added ( k=0 is typical inputs only; in parentheses, mean inputs actually added per task). The dashed line marks the author edge input. Unguarded writer inputs (W k ) reach the author level at no cost in false rejection; the preregistered guard (G k ) does not. (b) The same gates applied offline to each model’s own hidden-wrong first-step repairs (arm T, test partition; n wrong candidates). Writer cost: one call per task.
Subset
FA T
FA E
FA W 3
E − T
W 3− T
class S (23 tasks)
0.571
0.080
0.091
− 0.503 (<0.001)
− 0.491 (<0.001)
class E (25 tasks)
0.297
0.053
0.055
− 0.235 (<0.001)
− 0.233 (<0.001)
class B (4 tasks)
0.192
0.050
0.050
− 0.142 (0.500)
− 0.142 (0.500)
without CPython (16 tasks)
–
–
–
− 0.328 (0.002)
− 0.368 (0.002)
without Node.js (42 tasks)
–
–
–
− 0.321 (<0.001)
− 0.313 (<0.001)
without SQLite (46 tasks)
–
–
–
− 0.367 (<0.001)
− 0.346 (<0.001)
Table 2. Post-hoc robustness of the model-free effects (affected test tasks; FA task means; cluster-level differences with unadjusted sign-flip p ).
Model
T
S
E
ES
no gate
GPT-5.6-luna
398/22/0
403/9/0
408/7/0
409/0/0
45
Qwen3-8B
268/98/0
271/80/0
270/24/0
268/20/0
182
gpt-oss-20b
320/71/0
327/47/1
353/8/0
351/8/0
111
qwen3.6-35b-a3b
354/49/0
363/30/0
368/7/0
379/8/0
97
Table 3. Model repairs on the test partition under the all-consumer-assertions gate: correct completions / false acceptances / false rejections per arm (8 cells per task and arm; 456 records per arm). T typical inputs; S + the model’s own self-tests; E + the author edge input; ES both. Last column: false acceptances with no gate (arm T).
Model
Status
H1
H2
H3
H4
GPT-5.6-luna
descriptive
− 0.038 (0.499)
− 0.033 (0.749)
− 0.018 (1.000)
0.013 (1.000)
Qwen3-8B
confirmatory
− 0.176 (<0.001)
− 0.046 (0.749)
− 0.010 (1.000)
0.010 (0.998)
gpt-oss-20b
confirmatory
− 0.161 (<0.001)
− 0.064 (0.088)
0.000 (–)
0.023 (0.385)
qwen3.6-35b-a3b
descriptive
− 0.102 (0.004)
− 0.048 (0.256)
0.003 (1.000)
0.023 (0.256)
Table 4. Family A4 per model (test clusters; cluster means of per-attempt rates): difference and Holm-adjusted p . H1 FA(E) − FA(T); H2 FA(S) − FA(T); H3 FA(ES) − FA(E); H4 CC(S) − CC(T).
Model
Tasks
Tests
Binding
=hidden
Sim.
FA (S)
FA (raw)
FR (raw)
GPT-5.6-luna
71/73
187
153
5
0.69
0.153
0.054
0.031
Qwen3-8B
30/73
49
27
2
0.71
0.355
0.240
0.031
gpt-oss-20b
57/73
128
93
1
0.69
0.235
0.149
0.031
qwen3.6-35b-a3b
67/73
168
122
3
0.68
0.215
0.100
0.031
Table 5. Self-tests written by each model from the release note and the visible contract: tasks with at least one valid test, valid tests, tests declared legacy-binding, tests identical to a hidden input, mean highest similarity to a hidden input, and the model-free FA of the gate built from the guarded (S) and unguarded (raw) self-tests, with the unguarded gate’s FR.
Model
wrong
W1
W3
G3
Sraw
S
FR Sraw
GPT-5.6-luna
45
0.178
0.044
0.178
0.044
0.178
0.023
Qwen3-8B
176
0.261
0.170
0.261
0.330
0.455
0.038
gpt-oss-20b
109
0.156
0.083
0.266
0.339
0.376
0.030
qwen3.6-35b-a3b
92
0.098
0.054
0.337
0.326
0.337
0.026
Table 6. Offline gates applied to the hidden-wrong first-step candidates of arm T (test partition): share released by the blind-writer gate with 1 and 3 inputs (W1, W3), the guarded writer gate (G3), and the model’s own unguarded (Sraw) and guarded (S) self-tests; last column: share of hidden-correct candidates blocked by Sraw. All other gates blocked no correct candidate.
Figure 2. Wrong releases per attempt under all assertions with typical inputs (filled) and with the author edge input (open), against each family’s capability (test partition, cluster means).
Writer ↓ / repairer →
Tasks
GPT-5.6-luna
Qwen3-8B
gpt-oss-20b
qwen3.6-35b-a3b
model-free
GPT-5.6-luna (writer-001)
50
0.044
0.170
0.083
0.054
0.069
GPT-5.6-luna
52
0.044
0.176
0.128
0.065
0.075
Qwen3-8B
23
0.356
0.330
0.385
0.337
0.109
gpt-oss-20b
42
0.178
0.330
0.339
0.272
0.088
qwen3.6-35b-a3b
48
0.311
0.210
0.202
0.326
0.078
Table 7. Post-hoc writer × repairer matrix (zero calls). Each family’s self-test step used exactly the blind-writer prompt and validation, so its unguarded self-tests are that family’s blind-writer inputs. Cells: share of the repairer’s hidden-wrong first-step candidates (arm T, test partition) released by the all-assertions gate on typical inputs plus the writer’s inputs. Tasks: affected test tasks with ≥ 1 valid input. Last column: model-free FA over the complete repair space on the 21 tasks where every writer has inputs (typical only: 0.510).
Writer \ repairer
Tasks
GPT-5.6-luna
Qwen3-8B
gpt-oss-20b
qwen3.6-35b-a3b
GPT-5.6-luna (writer-001)
50
0.000
0.029
0.000
0.000
GPT-5.6-luna
52
0.000
0.157
0.125
0.021
Qwen3-8B
23
0.400
0.143
0.179
0.106
gpt-oss-20b
42
0.200
0.257
0.268
0.021
qwen3.6-35b-a3b
48
0.400
0.157
0.214
0.234
Table 8. Post-hoc writer × repairer matrix restricted to the 21 affected test tasks where every writer has at least one valid input (removes the coverage difference; 30–70 wrong first-step repairs per repairer). Cells: share of the repairer’s hidden-wrong first-step candidates (arm T) released by typical inputs plus the writer’s inputs. Tasks: tasks with ≥ 1 valid input overall. Descriptive.
Visible inputs
FA
FR
typical only
0.278
0.014
+ author edge input
0.045
0.014
+ blind writer, k=1
0.089
0.036
+ blind writer, k=2
0.054
0.043
+ blind writer, k=3
0.053
0.043
+ guarded writer, k=1
0.135
0.036
Table 9. Large-menu stage, model-free: share of hidden-wrong (FA) and hidden-correct blocked (FR) procedures over the complete large-menu space of the affected test tasks (1986 procedures in all tasks), all-consumer-assertions gate, task means.
Model
Status
T
E
LH1
LH2
LH3
GPT-5.6-luna
descriptive
391/29
401/12
0.074 → 0.031 (0.378)
0.018 (0.378)
− 0.018
Qwen3-8B
confirmatory
241/108
244/27
0.253 → 0.062 (<0.001)
0.019 (0.634)
− 0.047
gpt-oss-20b
confirmatory
310/85
334/15
0.212 → 0.038 (<0.001)
0.031 (0.335)
− 0.026
qwen3.6-35b-a3b
descriptive
327/72
344/9
0.177 → 0.023 (<0.001)
0.057 (0.047)
− 0.069
Table 10. Large-menu stage, model repairs (test partition, 456 attempts per arm): correct completions / wrong releases under all consumer assertions for arms T and E; LH1 wrong releases per attempt T → E (strict Holm p ); LH2 change in wrong releases per attempt with typical inputs relative to the v4 menu (strict Holm p ); LH3 change in correct completion relative to the v4 menu.
Programs
T
E
W 3
Menu
median
wrong
tasks
wrong
tasks
wrong
tasks
earlier menu
5
113 (2.17)
41
14 (0.27)
11
15 (0.29)
12
large menu
30
445 (8.56)
42
46 (0.88)
17
62 (1.19)
15
Table 11. Post-hoc, menu-independent view (52 affected test tasks). Top: wrong programs that pass the all-assertions gate on the visible inputs, i.e. that the visible inputs cannot tell apart from a correct one (total, mean per task in parentheses), and tasks with at least one such program. Bottom: tasks in which a model released at least one wrong repair in any of its 8 cells under all assertions, on the earlier and the large menu.
Visible inputs
FA
FR
FA hard
typical only
0.337
0.010
0.990
+ blind writer (GPT-5.6-luna), k=3
0.058
0.042
0.153
+ guarded writer, k=3
0.135
0.042
0.492
+ author edge input
0.058
0.010
0.168
+ blind writer v2 (qwen3.6), k=3
0.094
0.083
–
+ blind writer v2 (Qwen3-8B), k=3
0.182
0.049
–
Table 12. Hard-distractor stage, model-free over the complete hard-menu space of the affected test tasks (2310 procedures in all tasks): FA and FR task means under all consumer assertions, and FA restricted to hard-origin programs (wrong options that match the legacy output on the typical inputs).
Model
T
E
HA1
HA2
luna
qwen3.6
Qwen3-8B
own
GPT-5.6-luna
385/42
394/12
0.107 → 0.031 (0.063)
0.033 (0.094)
0.117
0.433
0.300
0.117
Qwen3-8B c
245/105
244/22
0.250 → 0.050 (<0.001)
− 0.003 (1.000)
0.173
0.303
0.378
0.395
gpt-oss-20b c
306/88
339/17
0.218 → 0.043 (<0.001)
0.006 (0.757)
0.104
0.392
0.384
0.416
qwen3.6-35b-a3b
317/80
347/11
0.198 → 0.028 (<0.001)
0.020 (0.225)
0.048
0.373
0.349
0.317
Table 13. Hard-distractor stage, model repairs (test partition, 456 attempts per arm): correct completions / wrong releases under all consumer assertions in arms T and E; HA1 wrong releases per attempt T → E (strict Holm p ); HA2 change of typical-input wrong releases relative to the large menu. Last four columns (descriptive, offline): share of each model’s hidden-wrong first-step repairs released by the typical inputs plus three inputs of the GPT-5.6-luna writer, the qwen3.6 and Qwen3-8B writers (prompt v2), and the model’s own self-tests. c confirmatory family.
Model
CPython
Node.js
SQLite 3.53
SQLite 3.54
GPT-5.6-luna
37/45 (37)
10/13 (10)
3/4 (2)
3/4 (3)
Qwen3-8B
19/45 (24)
6/13 (9)
1/4 (0)
1/4 (0)
gpt-oss-20b
29/45 (28)
7/13 (8)
2/4 (1)
3/4 (3)
qwen3.6-35b-a3b
32/45 (34)
10/13 (6)
2/4 (3)
2/4 (2)
Table 14. Contamination probe: hidden-correct first-step repairs without any release-note text (in parentheses: with the excerpts, typical inputs, full representation, no diff view, listed order). Uniform random choice from each finite domain would be expected to be hidden-correct in 9.9/45, 3.8/13, 0.8/4 and 1.0/4 tasks.
Condition
Joint
affected
controls
silent
abstain
probes
doc. bytes
edits
stale procedure (B0)
24/100
–
–
48
0
0.00
0.00
0.00
scope-neutral reconstruction (B1R)
96/100
72
24
0
4
1.04
119.83
1.00
contract-guided local repair (B4)
96/100
72
24
0
4
2.60
11.16
1.00
Table 15. Prior evidence: sealed deterministic study, held-out test split (20 lineages, 100 task units). Scope-neutral reconstruction B1R and local repair B4 share the finite candidate base and the 24-probe ceiling; B1R also reads the complete synthetic new-version document. Resource columns are the only resources this layer measured (means per unit); tokens, latency and graph-construction effort were not measured. The injected B1/B1C rows are in Appendix C .
Task
predicates
partial score
candidate operations
verdicts
decidable missing obligation
w02
4
0.75
1 call(s); auth header reused; edits 0
visible: yes; upstream: fail
re-authentication via /v3/oauth/token requested but not performed (stale bearer accepted by the mock)
w08
3
0.67
1 call(s); auth header reused; edits 0
visible: yes; upstream: fail
re-authentication via /v3/oauth/token requested but not performed (stale bearer accepted by the mock)
w14
2
0.50
1 call(s); auth header reused; edits 0
visible: yes; upstream: fail
re-authentication via /v3/oauth/token requested but not performed (stale bearer accepted by the mock); requested second record; answer is the first record
Table 16. The three public tasks that met the frozen visible-completion rule (“the main request completes without an error status and every declared output field can be derived”) but failed the upstream validator. Columns are read mechanically from retained records; the validator’s per-predicate reasons were stored only as hashes, so which predicates failed is unknown and left blank. Exploratory, post hoc.
Threat
What we did
What remains
Author-written edge and hidden inputs
blind writer that never sees program or hidden inputs; 3 of 176 inputs equal a hidden input
hidden inputs are still author-written
Effect seen before it was tested
fresh replication on 15 new clusters; freeze copies of the design text
M4c computable from the build before F0 (disclosed)
Contracts stricter than requirements
two blind AI audits per contract; post-hoc removal of p0 and d0 changes 0 verdicts
no human review; R2 class-label objections unresolved
One model in three roles
writer × repairer matrix (post hoc); two further preregistered writers (qwen3.6, Qwen3-8B)
only GPT-5.6-luna’s writer passed; auditor and writer are the same model
distractors remain author-generated by documented operators
Strong-evidence criterion
reported against its frozen definition
not met: M4a not positive; audit flags > 10%
Table 17. Threats to validity and how they are addressed.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Task
Runtime
Origin
Class
Split
Correct
typical
edge
py-binascii-hqx
cpython
v2
B
test
0/6
yes
yes
py-distutils-looseversion
cpython
v2
B
test
0/18
yes
yes
py-shuffle-random-param
cpython
v2
B
test
0/6
yes
yes
py-control-csv
cpython
v2
C
development
1/4
no
no
py-control-json-canonical
cpython
v2
C
development
1/4
no
no
py-control-sha256
cpython
v2
C
test
2/4
no
no
Appendix
Table 18. The 73 tasks. Class: S silent change, E explicit error, B no-safe-repair task, C unchanged control. Correct: hidden-correct procedures in the finite domain. Last two columns: whether the stale procedure fails a visible all-assertions check with typical inputs and with the author edge input added (by the edge-input QA rule always yes for affected tasks).
Row
Joint
silent
discordant with B4
B1 (sealed)
88/100
12
8
B1C (post-audit)
88/100
–
8
Appendix
Table 19. Evaluator qualification (not a method comparison): the sealed B1 row and the post-audit B1C row contain an injected, scope-marker-triggered preservation perturbation. Their rejections show that the hidden preservation evaluator detects it.
Budget
Repr.
Feedback
Joint
silent
abstain
mean edits
2
full
behavioral
45/48
0
3
1.24
2
full
none
45/48
0
3
1.24
2
patch
behavioral
46/48
0
2
1.28
2
patch
none
46/48
0
2
1.28
∞
full
behavioral
48/48
0
0
1.40
∞
full
none
48/48
0
0
1.40
Appendix
Table 20. Study B (appendix, descriptive): stated edit budget × representation on the 12-source minimal-reproduction bank, Codex. Joint: hidden acceptance, preservation and forbidden-effect contracts all hold.
Large language model (LLM) code-repair systems fix faulty programs by repeatedly generating and testing revisions. Each revision is costly, making the choice of which repair to pursue important for efficiency. Existing search policies rely on test pass rates and search statistics, ignoring the details of test failures and proposed edits. We introduce CodeTriage, a learned policy that prioritizes edits before their revised programs are generated and tested. Its 0.6B-parameter model reads the test output, failure diagnosis, and proposed edit, then predicts a distribution over the number of repair steps needed to reach a fix through that edit. These predictions prioritize all pending edits across programs before generating and testing the revised code. On held-out repair tasks, CodeTriage finds fixes with 21% fewer program revisions and 12% fewer output tokens than the strongest baseline on each metric. Without retraining, it achieves the highest average success across search budgets for each of 11 unseen repair LLMs. Ablations show that both the failure context and edit text, as well as comparisons across programs, improve search. We also release a corpus of 12.2 million program versions from four LLMs, organized into repair trees with proposed edits and test results.
Code language models are now trusted collaborators in production workflows for debugging, refactoring, and iterative repair, and every benchmark that evaluates them assumes the instructions they act on are correct. We study what happens when that assumption breaks. We evaluate code language models across four experiments designed to assess whether models resist or obey incorrect instructions in single-pass and iterative repair settings, using the RunBugRun dataset of algorithmic Python problems with deterministic test cases. Our findings reveal a striking behavioral pattern: models correctly identify an incorrect instruction as wrong, then follow it anyway. This compliance unknowingly introduces errors beyond the original bug, and the corrupted code state cannot be recovered through subsequent self-guided iterative repair, which fails to converge across passes. We term this Blind Obedience, characterize the Ghost (Unknown) Errors it introduces, quantify the proportion of cases where semantic corruption proves irrecoverable, and show that extended reasoning cannot reverse it. These findings surface behavioral properties invisible to pass-rate evaluation, with direct consequences for code language models deployed in production settings.
Raj Jaiswal, Anany Singh Divy, Savar Bhasin +3
IIIT Delhi · Microsoft Research India · IIT Kanpur
When code-repair agents fail, final tests reveal that a patch is wrong but not which earlier commitment caused it or which software obligation failed. Poison Valleys traces this path in 1,844 model requests on two synthetic Python cases: PersistQueue adds atomic batch claiming to a disk-backed queue, while Dijkstra adds source-sensitive filtering to shortest-path search. We define plausible but incorrect software mechanisms, place each in a model's hypothesis, implementation plan, or both, and evaluate the generated code with an Oracle and targeted diagnostic coordinates. Moving the misconception from hypothesis to plan increased failure by 32.2% in PersistQueue and 28.9% in Dijkstra after matching the experimental mix. Different misconceptions produced distinct, model-dependent patterns of violated obligations, although several could converge on the same behavior. This creates a measured connection from upstream misconception to code mechanism and failure form. Code, data, and demo: https://poisonvalleys.org.