Discriminating Fixture Coverage in Agent-Infrastructure Verification Suites
Organizations: Carnegie Mellon University Eragon
Abstract
Invariant suites and runtime monitors increasingly gate agent deployment decisions, and the evidence offered for any particular suite is almost always a single observation: it passes an implementation believed correct and fails one believed broken. We measure what that observation is worth. Applying mutation analysis to an invariant suite for a multi-session agent state-projection layer, we first find that this standard validation certifies a suite in which a first-order mutant removing event-identity deduplication survives every check. We then freeze the repaired twelve-check suite, record its hash, and run it once against ten mutants specified by an adversarial reader who designed none of its fixtures: it kills five. Instrumenting the five survivors against the reference shows they fail in two distinct ways, not one. Three are never activated, because no fixture supplies an input on which the mutated code behaves differently at all. The other two corrupt internal state that no oracle in the suite can observe. The two modes need different repairs, and neither is visible from a pass/fail report. Treating the missing inputs as a coverage question, we enumerate seven discriminating dimensions of the input space, register in advance which are uncovered and which survivors they should explain, and add one fixture per uncovered dimension while reusing the existing oracles verbatim. All five survivors then die, each to the check written for its predicted dimension. We report this as a repair result on the same challenge set rather than a second held-out estimate, and give the artifact, including the frozen hash, the registered predictions, all mutants and the run logs, so the distinction is checkable.
Figures & tables
| Injected fault | A | B | Killed by (registered dim.) | |
|---|---|---|---|---|
| H1 | deduplication keyed on messageId | ✓ | ✓ | C-08 (convergence) |
| H2 | deduplication only for message.* | ✗ | ✓ | C-13 (D3) |
| H3 | deduplication cache bounded to one entry | ✗ | ✓ | C-14, also C-13 (D4) |
| H4 | snapshot resets the cursor unconditionally | ✓ | ✓ | C-09 |
| H5 | over-aggressive merge on role + text | ✓ | ✓ | C-03 |
| H6 | session identifier compared case-insensitively | ✗ | ✓ | C-15 (D5) |
| New check | New input dimension | Oracle predicate, reused from |
|---|---|---|
| C-13 | run.* event family (D3) | digestOf + runStatus , as C-01 and C-10 |
| C-14 | redelivery after an intervening event (D4) | digestOf before/after, as C-01 |
| C-15 | session identifier differing only in case (D5) | digestOf before/after, as C-04 |
| C-16 | equal version rather than strictly lower (D6) | runStatus , as C-10 |
| C-17 | acknowledgment after the WebSocket event (D7) | digestOf + visibleMessages , as C-07 |
Automated Software Testing
Automated software testing uses tools and algorithms to generate, run, and evaluate tests that find defects and prevent regressions.
See all 136 papers