cs.SENov 5, 2026
SaveAccepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations
Organizations: University of Massachusetts Amherst, Amherst, Massachusetts, USA · Accentrust, Vancouver, Canada · Georgia Institute of Technology, School of Computer Science, Atlanta, Georgia, USA · Duke University, Durham, North Carolina, USA ·
Abstract
After a runtime or API migration, an LLM-repaired procedure can run cleanly yet return wrong results, and a release check on typical test inputs ships it. We report a preregistered study of 73 tasks covering 60 documented changes in CPython, Node.js and SQLite. Every candidate is executed on the real runtimes against withheld contracts. Over the complete repair space, the all-assertions gate on typical inputs released 0.410 of wrong repairs. One edge input written from the release note cut this to 0.065, and to 0.061 on 15 unseen clusters. Three inputs from a blind GPT-5.6-luna writer that never saw the program reached 0.070. Two further frozen stages enlarged the menu to 1491 fragments and then added 233 hard distractors that match the legacy output on typical inputs. The edge input (0.058) and the frontier-model writer (0.058) passed their preregistered tests. Yet 0.168 and 0.153 of the hard wrong programs still passed. Two local-model writers failed their preregistered tests. Across four model families, the edge input cut wrong releases per attempt in all three open-weight families on all three menus, for example from 0.233 to 0.057, without lowering correct completion. Self-tests helped less. A blind AI audit flagged 34 contracts as stricter than their requirements. Dropping the disputed checks changed 0 verdicts, but class-label disputes remain open. No model-written code was executed. Artifacts are available from the authors on request.
Figures & tables
| Visible inputs | Added | FA | FR | Stale |
|---|---|---|---|---|
| typical only | 0.00 | 0.410 | 0.010 | 34/52 |
| + blind writer, k=1 | 0.96 | 0.130 | 0.010 | 49/52 |
| + blind writer, k=2 | 1.85 | 0.075 | 0.010 | 50/52 |
| + blind writer, k=3 | 2.46 | 0.070 | 0.010 | 50/52 |
| + guarded writer, k=1 | 0.73 | 0.206 | 0.010 | 42/52 |
| + guarded writer, k=2 | 1.40 | 0.161 | 0.010 | 42/52 |
Table 1. Model-free release-gate adequacy over the complete finite repair space of the 52 affected test tasks (49 clusters), all-consumer-assertions gate. Added: mean visible inputs added to the typical ones. FA: share of hidden-wrong procedures released; FR: share of hidden-correct procedures blocked (task means). Stale: tasks in which the unrepaired procedure fails visibly.
Figure 1. Headline result. (a) Share of wrong procedures released over the complete finite repair space as blind-writer inputs are added ( is typical inputs only; in parentheses, mean inputs actually added per task). The dashed line marks the author edge input. Unguarded writer inputs (W k ) reach the author level at no cost in false rejection; the preregistered guard (G k ) does not. (b) The same gates applied offline to each model’s own hidden-wrong first-step repairs (arm T, test partition; wrong candidates). Writer cost: one call per task.
| Subset | FA T | FA E | FA W 3 | E T | W T |
|---|---|---|---|---|---|
| class S (23 tasks) | 0.571 | 0.080 | 0.091 | 0.503 (<0.001) | 0.491 (<0.001) |
| class E (25 tasks) | 0.297 | 0.053 | 0.055 | 0.235 (<0.001) | 0.233 (<0.001) |
| class B (4 tasks) | 0.192 | 0.050 | 0.050 | 0.142 (0.500) | 0.142 (0.500) |
| without CPython (16 tasks) | – | – | – | 0.328 (0.002) | 0.368 (0.002) |
| without Node.js (42 tasks) | – | – | – | 0.321 (<0.001) | 0.313 (<0.001) |
| without SQLite (46 tasks) | – | – | – | 0.367 (<0.001) | 0.346 (<0.001) |
Table 2. Post-hoc robustness of the model-free effects (affected test tasks; FA task means; cluster-level differences with unadjusted sign-flip ).
| Model | T | S | E | ES | no gate |
|---|---|---|---|---|---|
| GPT-5.6-luna | 398/22/0 | 403/9/0 | 408/7/0 | 409/0/0 | 45 |
| Qwen3-8B | 268/98/0 | 271/80/0 | 270/24/0 | 268/20/0 | 182 |
| gpt-oss-20b | 320/71/0 | 327/47/1 | 353/8/0 | 351/8/0 | 111 |
| qwen3.6-35b-a3b | 354/49/0 | 363/30/0 | 368/7/0 | 379/8/0 | 97 |
Table 3. Model repairs on the test partition under the all-consumer-assertions gate: correct completions / false acceptances / false rejections per arm (8 cells per task and arm; 456 records per arm). T typical inputs; S + the model’s own self-tests; E + the author edge input; ES both. Last column: false acceptances with no gate (arm T).
| Model | Status | H1 | H2 | H3 | H4 |
|---|---|---|---|---|---|
| GPT-5.6-luna | descriptive | 0.038 (0.499) | 0.033 (0.749) | 0.018 (1.000) | 0.013 (1.000) |
| Qwen3-8B | confirmatory | 0.176 (<0.001) | 0.046 (0.749) | 0.010 (1.000) | 0.010 (0.998) |
| gpt-oss-20b | confirmatory | 0.161 (<0.001) | 0.064 (0.088) | 0.000 (–) | 0.023 (0.385) |
| qwen3.6-35b-a3b | descriptive | 0.102 (0.004) | 0.048 (0.256) | 0.003 (1.000) | 0.023 (0.256) |
Table 4. Family A4 per model (test clusters; cluster means of per-attempt rates): difference and Holm-adjusted . H1 FA(E) FA(T); H2 FA(S) FA(T); H3 FA(ES) FA(E); H4 CC(S) CC(T).
| Model | Tasks | Tests | Binding | =hidden | Sim. | FA (S) | FA (raw) | FR (raw) |
|---|---|---|---|---|---|---|---|---|
| GPT-5.6-luna | 71/73 | 187 | 153 | 5 | 0.69 | 0.153 | 0.054 | 0.031 |
| Qwen3-8B | 30/73 | 49 | 27 | 2 | 0.71 | 0.355 | 0.240 | 0.031 |
| gpt-oss-20b | 57/73 | 128 | 93 | 1 | 0.69 | 0.235 | 0.149 | 0.031 |
| qwen3.6-35b-a3b | 67/73 | 168 | 122 | 3 | 0.68 | 0.215 | 0.100 | 0.031 |
Table 5. Self-tests written by each model from the release note and the visible contract: tasks with at least one valid test, valid tests, tests declared legacy-binding, tests identical to a hidden input, mean highest similarity to a hidden input, and the model-free FA of the gate built from the guarded (S) and unguarded (raw) self-tests, with the unguarded gate’s FR.
| Model | wrong | W1 | W3 | G3 | Sraw | S | FR Sraw |
|---|---|---|---|---|---|---|---|
| GPT-5.6-luna | 45 | 0.178 | 0.044 | 0.178 | 0.044 | 0.178 | 0.023 |
| Qwen3-8B | 176 | 0.261 | 0.170 | 0.261 | 0.330 | 0.455 | 0.038 |
| gpt-oss-20b | 109 | 0.156 | 0.083 | 0.266 | 0.339 | 0.376 | 0.030 |
| qwen3.6-35b-a3b | 92 | 0.098 | 0.054 | 0.337 | 0.326 | 0.337 | 0.026 |
Table 6. Offline gates applied to the hidden-wrong first-step candidates of arm T (test partition): share released by the blind-writer gate with 1 and 3 inputs (W1, W3), the guarded writer gate (G3), and the model’s own unguarded (Sraw) and guarded (S) self-tests; last column: share of hidden-correct candidates blocked by Sraw. All other gates blocked no correct candidate.
Figure 2. Wrong releases per attempt under all assertions with typical inputs (filled) and with the author edge input (open), against each family’s capability (test partition, cluster means).
| Writer / repairer | Tasks | GPT-5.6-luna | Qwen3-8B | gpt-oss-20b | qwen3.6-35b-a3b | model-free |
|---|---|---|---|---|---|---|
| GPT-5.6-luna (writer-001) | 50 | 0.044 | 0.170 | 0.083 | 0.054 | 0.069 |
| GPT-5.6-luna | 52 | 0.044 | 0.176 | 0.128 | 0.065 | 0.075 |
| Qwen3-8B | 23 | 0.356 | 0.330 | 0.385 | 0.337 | 0.109 |
| gpt-oss-20b | 42 | 0.178 | 0.330 | 0.339 | 0.272 | 0.088 |
| qwen3.6-35b-a3b | 48 | 0.311 | 0.210 | 0.202 | 0.326 | 0.078 |
Table 7. Post-hoc writer repairer matrix (zero calls). Each family’s self-test step used exactly the blind-writer prompt and validation, so its unguarded self-tests are that family’s blind-writer inputs. Cells: share of the repairer’s hidden-wrong first-step candidates (arm T, test partition) released by the all-assertions gate on typical inputs plus the writer’s inputs. Tasks: affected test tasks with 1 valid input. Last column: model-free FA over the complete repair space on the 21 tasks where every writer has inputs (typical only: 0.510).
| Writer \ repairer | Tasks | GPT-5.6-luna | Qwen3-8B | gpt-oss-20b | qwen3.6-35b-a3b |
|---|---|---|---|---|---|
| GPT-5.6-luna (writer-001) | 50 | 0.000 | 0.029 | 0.000 | 0.000 |
| GPT-5.6-luna | 52 | 0.000 | 0.157 | 0.125 | 0.021 |
| Qwen3-8B | 23 | 0.400 | 0.143 | 0.179 | 0.106 |
| gpt-oss-20b | 42 | 0.200 | 0.257 | 0.268 | 0.021 |
| qwen3.6-35b-a3b | 48 | 0.400 | 0.157 | 0.214 | 0.234 |
Table 8. Post-hoc writer repairer matrix restricted to the 21 affected test tasks where every writer has at least one valid input (removes the coverage difference; 30–70 wrong first-step repairs per repairer). Cells: share of the repairer’s hidden-wrong first-step candidates (arm T) released by typical inputs plus the writer’s inputs. Tasks: tasks with 1 valid input overall. Descriptive.
| Visible inputs | FA | FR |
|---|---|---|
| typical only | 0.278 | 0.014 |
| + author edge input | 0.045 | 0.014 |
| + blind writer, k=1 | 0.089 | 0.036 |
| + blind writer, k=2 | 0.054 | 0.043 |
| + blind writer, k=3 | 0.053 | 0.043 |
| + guarded writer, k=1 | 0.135 | 0.036 |
Table 9. Large-menu stage, model-free: share of hidden-wrong (FA) and hidden-correct blocked (FR) procedures over the complete large-menu space of the affected test tasks (1986 procedures in all tasks), all-consumer-assertions gate, task means.
| Model | Status | T | E | LH1 | LH2 | LH3 |
|---|---|---|---|---|---|---|
| GPT-5.6-luna | descriptive | 391/29 | 401/12 | 0.074 0.031 (0.378) | 0.018 (0.378) | 0.018 |
| Qwen3-8B | confirmatory | 241/108 | 244/27 | 0.253 0.062 (<0.001) | 0.019 (0.634) | 0.047 |
| gpt-oss-20b | confirmatory | 310/85 | 334/15 | 0.212 0.038 (<0.001) | 0.031 (0.335) | 0.026 |
| qwen3.6-35b-a3b | descriptive | 327/72 | 344/9 | 0.177 0.023 (<0.001) | 0.057 (0.047) | 0.069 |
Table 10. Large-menu stage, model repairs (test partition, 456 attempts per arm): correct completions / wrong releases under all consumer assertions for arms T and E; LH1 wrong releases per attempt T E (strict Holm ); LH2 change in wrong releases per attempt with typical inputs relative to the v4 menu (strict Holm ); LH3 change in correct completion relative to the v4 menu.
| Programs | T | E | W 3 | ||||
|---|---|---|---|---|---|---|---|
| Menu | median | wrong | tasks | wrong | tasks | wrong | tasks |
| earlier menu | 5 | 113 (2.17) | 41 | 14 (0.27) | 11 | 15 (0.29) | 12 |
| large menu | 30 | 445 (8.56) | 42 | 46 (0.88) | 17 | 62 (1.19) | 15 |
Table 11. Post-hoc, menu-independent view (52 affected test tasks). Top: wrong programs that pass the all-assertions gate on the visible inputs, i.e. that the visible inputs cannot tell apart from a correct one (total, mean per task in parentheses), and tasks with at least one such program. Bottom: tasks in which a model released at least one wrong repair in any of its 8 cells under all assertions, on the earlier and the large menu.
| Visible inputs | FA | FR | FA hard |
|---|---|---|---|
| typical only | 0.337 | 0.010 | 0.990 |
| + blind writer (GPT-5.6-luna), k=3 | 0.058 | 0.042 | 0.153 |
| + guarded writer, k=3 | 0.135 | 0.042 | 0.492 |
| + author edge input | 0.058 | 0.010 | 0.168 |
| + blind writer v2 (qwen3.6), k=3 | 0.094 | 0.083 | – |
| + blind writer v2 (Qwen3-8B), k=3 | 0.182 | 0.049 | – |
Table 12. Hard-distractor stage, model-free over the complete hard-menu space of the affected test tasks (2310 procedures in all tasks): FA and FR task means under all consumer assertions, and FA restricted to hard-origin programs (wrong options that match the legacy output on the typical inputs).
| Model | T | E | HA1 | HA2 | luna | qwen3.6 | Qwen3-8B | own |
|---|---|---|---|---|---|---|---|---|
| GPT-5.6-luna | 385/42 | 394/12 | 0.107 0.031 (0.063) | 0.033 (0.094) | 0.117 | 0.433 | 0.300 | 0.117 |
| Qwen3-8B c | 245/105 | 244/22 | 0.250 0.050 (<0.001) | 0.003 (1.000) | 0.173 | 0.303 | 0.378 | 0.395 |
| gpt-oss-20b c | 306/88 | 339/17 | 0.218 0.043 (<0.001) | 0.006 (0.757) | 0.104 | 0.392 | 0.384 | 0.416 |
| qwen3.6-35b-a3b | 317/80 | 347/11 | 0.198 0.028 (<0.001) | 0.020 (0.225) | 0.048 | 0.373 | 0.349 | 0.317 |
Table 13. Hard-distractor stage, model repairs (test partition, 456 attempts per arm): correct completions / wrong releases under all consumer assertions in arms T and E; HA1 wrong releases per attempt T E (strict Holm ); HA2 change of typical-input wrong releases relative to the large menu. Last four columns (descriptive, offline): share of each model’s hidden-wrong first-step repairs released by the typical inputs plus three inputs of the GPT-5.6-luna writer, the qwen3.6 and Qwen3-8B writers (prompt v2), and the model’s own self-tests. c confirmatory family.
| Model | CPython | Node.js | SQLite 3.53 | SQLite 3.54 |
|---|---|---|---|---|
| GPT-5.6-luna | 37/45 (37) | 10/13 (10) | 3/4 (2) | 3/4 (3) |
| Qwen3-8B | 19/45 (24) | 6/13 (9) | 1/4 (0) | 1/4 (0) |
| gpt-oss-20b | 29/45 (28) | 7/13 (8) | 2/4 (1) | 3/4 (3) |
| qwen3.6-35b-a3b | 32/45 (34) | 10/13 (6) | 2/4 (3) | 2/4 (2) |
Table 14. Contamination probe: hidden-correct first-step repairs without any release-note text (in parentheses: with the excerpts, typical inputs, full representation, no diff view, listed order). Uniform random choice from each finite domain would be expected to be hidden-correct in 9.9/45, 3.8/13, 0.8/4 and 1.0/4 tasks.
| Condition | Joint | affected | controls | silent | abstain | probes | doc. bytes | edits |
|---|---|---|---|---|---|---|---|---|
| stale procedure (B0) | 24/100 | – | – | 48 | 0 | 0.00 | 0.00 | 0.00 |
| scope-neutral reconstruction (B1R) | 96/100 | 72 | 24 | 0 | 4 | 1.04 | 119.83 | 1.00 |
| contract-guided local repair (B4) | 96/100 | 72 | 24 | 0 | 4 | 2.60 | 11.16 | 1.00 |
Table 15. Prior evidence: sealed deterministic study, held-out test split (20 lineages, 100 task units). Scope-neutral reconstruction B1R and local repair B4 share the finite candidate base and the 24-probe ceiling; B1R also reads the complete synthetic new-version document. Resource columns are the only resources this layer measured (means per unit); tokens, latency and graph-construction effort were not measured. The injected B1/B1C rows are in Appendix C .
| Task | predicates | partial score | candidate operations | verdicts | decidable missing obligation |
|---|---|---|---|---|---|
| w02 | 4 | 0.75 | 1 call(s); auth header reused; edits 0 | visible: yes; upstream: fail | re-authentication via /v3/oauth/token requested but not performed (stale bearer accepted by the mock) |
| w08 | 3 | 0.67 | 1 call(s); auth header reused; edits 0 | visible: yes; upstream: fail | re-authentication via /v3/oauth/token requested but not performed (stale bearer accepted by the mock) |
| w14 | 2 | 0.50 | 1 call(s); auth header reused; edits 0 | visible: yes; upstream: fail | re-authentication via /v3/oauth/token requested but not performed (stale bearer accepted by the mock); requested second record; answer is the first record |
Table 16. The three public tasks that met the frozen visible-completion rule (“the main request completes without an error status and every declared output field can be derived”) but failed the upstream validator. Columns are read mechanically from retained records; the validator’s per-predicate reasons were stored only as hashes, so which predicates failed is unknown and left blank. Exploratory, post hoc.
| Threat | What we did | What remains |
|---|---|---|
| Author-written edge and hidden inputs | blind writer that never sees program or hidden inputs; 3 of 176 inputs equal a hidden input | hidden inputs are still author-written |
| Effect seen before it was tested | fresh replication on 15 new clusters; freeze copies of the design text | M4c computable from the build before F0 (disclosed) |
| Contracts stricter than requirements | two blind AI audits per contract; post-hoc removal of p0 and d0 changes 0 verdicts | no human review; R2 class-label objections unresolved |
| One model in three roles | writer repairer matrix (post hoc); two further preregistered writers (qwen3.6, Qwen3-8B) | only GPT-5.6-luna’s writer passed; auditor and writer are the same model |
| Menu composition | hard-distractor stage (233 legacy-matching distractors); menu-independent counts | distractors remain author-generated by documented operators |
| Strong-evidence criterion | reported against its frozen definition | not met: M4a not positive; audit flags 10% |
Table 17. Threats to validity and how they are addressed.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Runtime | Origin | Class | Split | Correct | typical | edge |
|---|---|---|---|---|---|---|---|
| py-binascii-hqx | cpython | v2 | B | test | 0/6 | yes | yes |
| py-distutils-looseversion | cpython | v2 | B | test | 0/18 | yes | yes |
| py-shuffle-random-param | cpython | v2 | B | test | 0/6 | yes | yes |
| py-control-csv | cpython | v2 | C | development | 1/4 | no | no |
| py-control-json-canonical | cpython | v2 | C | development | 1/4 | no | no |
| py-control-sha256 | cpython | v2 | C | test | 2/4 | no | no |
Table 18. The 73 tasks. Class: S silent change, E explicit error, B no-safe-repair task, C unchanged control. Correct: hidden-correct procedures in the finite domain. Last two columns: whether the stale procedure fails a visible all-assertions check with typical inputs and with the author edge input added (by the edge-input QA rule always yes for affected tasks).
| Row | Joint | silent | discordant with B4 |
|---|---|---|---|
| B1 (sealed) | 88/100 | 12 | 8 |
| B1C (post-audit) | 88/100 | – | 8 |
Table 19. Evaluator qualification (not a method comparison): the sealed B1 row and the post-audit B1C row contain an injected, scope-marker-triggered preservation perturbation. Their rejections show that the hidden preservation evaluator detects it.
| Budget | Repr. | Feedback | Joint | silent | abstain | mean edits |
|---|---|---|---|---|---|---|
| 2 | full | behavioral | 45/48 | 0 | 3 | 1.24 |
| 2 | full | none | 45/48 | 0 | 3 | 1.24 |
| 2 | patch | behavioral | 46/48 | 0 | 2 | 1.28 |
| 2 | patch | none | 46/48 | 0 | 2 | 1.28 |
| full | behavioral | 48/48 | 0 | 0 | 1.40 | |
| full | none | 48/48 | 0 | 0 | 1.40 |
Table 20. Study B (appendix, descriptive): stated edit budget representation on the 12-source minimal-reproduction bank, Codex. Joint: hidden acceptance, preservation and forbidden-effect contracts all hold.
More in this topic
Automated Program Repair
Automated program repair develops methods that locate software faults and produce validated fixes with limited human effort.
See all 164 papers