When Passing Checks Do Not Establish Completion: Evidence Interpretation in Coding Agents
Organizations: Alibaba Cloud Computing
Abstract
Verification procedures in coding agents must preserve execution facts, interpret their scope and discharge checking obligations. We study these responsibilities through qualified repository checks, controlled evidence consumers and executable workflows. A prospective thirteen-task factorial study finds that correct reported facts need not yield the requested finite verdict. Conversely, all 122 delivered labels in a held-out receipt-supply study are correct, while sixteen responses misreport facts and eleven others exceed authorized scope. Production repair supplies successful controls and finite-supported preservation failures. Across 118 repair episodes on 31 tasks, source interventions separate startup reachability from repair and witness production from the claimed operation. Lifetime and resource controls delimit what passing validation establishes. Generated-note comparisons yield local repair differences but can state false explanations; strong ordinary baselines, new tasks and Skill-transfer studies supply no consistent overall advantage. These results characterize evidence interpretation, applicability and fulfilled actions while limiting claims of general diagnostic superiority, Skill deterioration or deployed folding effects.
Figures & tables
| Cohort | Tasks | Episodes | Conditions and Qwen/Devstral |
|---|---|---|---|
| A | 4 | 16 | ordinary 1/2; composed 1/2 |
| B | 6 | 36 | ordinary 0/2; checklist 0/1; |
| composed 0/1 | |||
| C | 5 | 20 | ordinary 2/1; dependency 2/2 |
| D | 3 | 6 | ordinary 2/3 |
| E | 4 | 8 | ordinary 0/0 |
| Source | Requested check | Three controls |
|---|---|---|
| Original | FAIL | PASS |
| Official fixed | PASS | PASS |
| Actual candidate | UNKNOWN | UNKNOWN |
| Return restored | FAIL | PASS |
| Mode restored only | UNKNOWN | UNKNOWN |
| Source | Official | Explicit | Explicit | Write-only |
|---|---|---|---|---|
| check | events | behavior | output | |
| Original | FAIL | 0 | FAIL | PASS |
| Official fixed | PASS | 1 | PASS | PASS |
| Qwen, both | PASS | 0 | FAIL | FAIL |
| Devstral, both | PASS | 1 | PASS | PASS |
| Fixture context | Fixed/Dev | Qwen |
|---|---|---|
| Canonical | P:2 flush | P:2 write |
| Omit explicit flush | P:1 flush | P:2 write |
| Direct writes, explicit flush | P:2 flush | F:0 |
| Direct writes, no explicit flush | P:1 flush | F:0 |
| Same, retain wrapper | F:0 | F:0 |
| Canonical, retain wrapper | P:1 flush | P:2 write |
| Source | Snapshot | Thread | Notifications |
|---|---|---|---|
| Original / unchanged Qwen | FAIL | FAIL | 0 |
| Official fixed | PASS | PASS | 2 |
| Actual Devstral | PASS | FAIL | 0 |
| +thread name | PASS | PASS | 0 |
| +registration | PASS | FAIL | 2 |
| +registration +name | PASS | PASS | 2 |
| Source | PK | Relation | Direct ID |
|---|---|---|---|
| Original | FAIL | PASS | PASS |
| Official | PASS | PASS | PASS |
| Actual Qwen | PASS | FAIL | PASS |
| Restore return | FAIL | PASS | PASS |
| Remove identities | PASS | FAIL | PASS |
| Return + selective guard | PASS | PASS | PASS |
| Source | Cold empty | Warm empty | Cold nonempty |
|---|---|---|---|
| Original / sham / rollback | 4 | 0 | 0 |
| Full / minimal fix | 0 | 0 | 0 |
| Source | Four checks | Settled | Delivery |
|---|---|---|---|
| Base | F/F/P/P | 1/2 | 1/2 |
| Proposed reference | P/P/P/P | 2/2 | 2/2 |
| Old broadcast only | P/P/P/P | 2/2 | 1/2 |
| Old shutdown only | F/P/P/P | 1/2 | 2/2 |
| Comment sham | P/P/P/P | 2/2 | 2/2 |
| Case | Ordinary evidence | Separating intervention | Supported conclusion and boundary |
|---|---|---|---|
| Pytest startup | Runner errors before target entry | Restore deleted return: startup recovers, target FAIL, controls PASS | Return deletion obstructs this startup path; recovery alone is not repair |
| Flush | Named boolean assertion PASS | Reset before explicit flush: FAIL; trace locates write-origin events | Writes supply the positive witness; no explicit live forwarding in that context |
| Flush lifecycle | Named boolean assertion PASS | Bypass writes, retain wrapper: FAIL; explicit flush retained: PASS | Cleanup is a separate witness channel; not proof of the exact finalizer chain |
| Polling | Thread-name check and notification check disagree | Name-only: 0 notifications; registration-only: 2 while name check still fails | Registration controls progress on the constructed schedule; no deployment-frequency claim |
| Warning | Broader unchanged extraction test FAIL | Inspect intact warning, then substitute exact-category lookup: PASS | Capture succeeds; the modified generic consumer rejects the category |
| Relation ordering | Broader unchanged ordering test FAIL | Remove new branches: FAIL; restore return: ordering PASS, PK defect reopens | The return expression mediates this tradeoff; no incremental defect-detection claim |
| Transferred content | Original workflow | Source replay |
|---|---|---|
| Compatible-repair section | 4/4 | 4/4 |
| Full generated note | 1/4 | 2/4 |
| No note | 0/4 | 1/4 |
LLM Agent Verification
LLM agent verification checks whether agents’ actions, evidence, policies, and completion claims satisfy explicit requirements.
See all 170 papers