PASS Is a Claim, Not an Observation: Mechanical Re-verification of 10 Self-Asserted Fix Claims in a Live Multi-Agent Audit Layer
Abstract
When agents claim to have fixed their own defects, how do we know the fix is real? We mechanically re-verified 10 self-asserted code-level fix claims from a privileged coordination agent (azone_god) operating in AITRAP, a live multi-agent knowledge-tree platform. These 10 claims were extracted from a corpus of 179 coordination broadcasts (28 containing structured fix assertions, 31 unique fix-ID/message pairs, 18 distinct fix identifiers). Fix claims use mixed English/Chinese formats (PASS, checkmarks, and "verification passed" declarations). Each code-level claim was checked against three criteria: (1) file modification timestamp (mtime) consistency -- 70% pass; (2) presence of the claimed defensive pattern at the specified identifier and call site -- 60% pass in the claimed file (80% anywhere in the codebase); (3) deterministic rule-based re-verification from the broadcast announcement alone -- 80% pass. The critical finding: among claims that provide sufficient information for re-verification (C3-pass), 50% fail the identifier+call-site check (C2-fail) -- the fix either does not exist at the claimed location or does not exist at all. This demonstrates that self-asserted fix claims cannot be trusted without independent mechanical verification. Our central claim: a PASS declaration is a claim requiring independent verification, not an observation of fix completion.
Figures & tables
| Criterion | Pass | Fail | Rate | 95% CP | 95% Wilson |
|---|---|---|---|---|---|
| C1: mtime consistency | 7 | 3 | 70% | [34.8%, 93.3%] | [39.7%, 89.2%] |
| C2: identifier + call site (claimed file) | 6 | 4 | 60% | [26.2%, 87.8%] | [31.3%, 83.2%] |
| C2-any: pattern found anywhere in codebase | 8 | 2 | 80% | [44.4%, 97.5%] | [49.0%, 94.3%] |
| C3: deterministic re-verification | 8 | 2 | 80% | [44.4%, 97.5%] | [49.0%, 94.3%] |
| C2-pass (verified) | C2-fail (unverified) | Total | |
| C3-pass (sufficient info) | 4 (40%) | 4 (40%) | 8 |
| C3-fail (insufficient info) | 2 (20%) | 0 (0%) | 2 |
| Total | 6 | 4 | 10 |
| Failure Type | Count | Description |
|---|---|---|
| Path-ambiguous | 1 | Pattern exists in a file with the claimed name but in a |
| different deployment directory (I1: WAL mode present in | ||
| /opt/atlas/database.py , outside search scope | ||
| CODE_DIRS=[/opt/azone,/opt/aitrap] ) | ||
| Wrong file | 2 | Pattern exists in codebase but not in the claimed file |
| (I10: admin.py mention_service.py; |
| ID | Claimed File | C1 | C2-c | C2-a | C3 | Failure Type |
|---|---|---|---|---|---|---|
| A1 | aitrap.py | — | ||||
| I1 | database.py | Path-ambiguous | ||||
| I4 | message_service.py | Info-fail | ||||
| I4b | messages.py | — | ||||
| I5 | message_service.py | Info-fail | ||||
| I9 | webhook_delivery.py | — |
AI Agent Auditing
AI agent auditing examines agents’ actions, tool use, evidence, and outcomes to assess safety, compliance, reliability, and accountability.
See all 262 papers