Generating an evaluation suite from an agent's specification is diversity-driven search: a generator fills a descriptor space (requirements × personas × scenario kinds), and the suite it emits is an archive, then scored like a benchmark by its mean pass rate, which throws away the search's product. We inject 7 controlled defects into a τ2-bench agent across two domains. Coverage yields a ceiling: if a defect's traced footprint holds b scenarios that pass at baseline, it can itself contribute at most b/N to a suite of N. For 4 of 5 traceable defects that ceiling lies at or below the detection threshold θ=0.10, so no within-footprint effect can move the suite past it; in both domains the tool-removal defect destroys every covered scenario it can (exactly its ceiling) while the suite-level pass rate moves the other way. Defects split into two regimes: trace-aligned ones, where per-cell scoring recovers the tool-removal defect in full and reveals a second as a significant improvement, and trace-orthogonal ones, whose footprint is behavioural: forced escalation is detected in retail at a traced coverage of exactly zero. Diversity along the generator's own axes helps only where it aligns: at fixed suite size a happy-path-only suite detects nothing, while a kind-diverse one detects the orchestration defect in 5 of 20 draws. Evaluation archives should be scored per cell and should publish coverage as a first-class property.
Figures & tables
∣Cm∣/N
ceil
whole-suite Δ [95% CI]
scoped Δ [95% CI]
airline
R1
S1
13/50
0.04
-0.06 [-0.22, +0.10]
-0.31 [-0.54, -0.08]
R1
S3
0 0/90
0.00
+0.04 [-0.03, +0.13]
n/a
T1
S1
0 7/50
0.06
-0.04 [-0.18, +0.10]
+0.43 [+0.14, +0.86] †
T1
S3
12/90
0.09
+0.08 [ + 0.00, +0.17]
+0.67 [+0.42, +0.92] †
O1
S3
0 9/90
0.10
+0.14 [+0.04, +0.26] †
+0.11 [ + 0.00, +0.33]
Table 1: Coverage, ceiling, and what each score sees. ∣Cm∣/N is the traced footprint over suite size, ceil is b/N , positive Δ a drop in pass rate, † a detection, “n/a” a coverage hole; remaining rows in Appendices C and D .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Id
Class
Operator
Field grounding
R1
Requirement
Delete one policy clause (24-hour cancellation)
specification violation [ 3 ]
R2
Requirement
Invert one policy condition (“cannot” → “can”)
incorrect functionality [ 19 ]
T1
Tool
Remove one tool ( cancel_reservation )
tool-contract breakage [ 1 ]
T2
Tool
Rename one parameter in a tool schema
API/serialization mismatch [ 19 ]
O1
Orchestration
Invert the escalation rule (always transfer to a human)
orchestration and execution control [ 1 ]
C1
Context
Truncate the policy document to its first 40%
context loss [ 1 , 3 ]
Appendix
Table 2: Mutation operators, one instance each; field grounding names the failure class each models.
Mutation
S1
S2
S3
R1
-0.06 [-0.22, +0.10]
+0.05 [-0.05, +0.16]
+0.04 [-0.03, +0.13]
R2
+0.06 [-0.10, +0.22]
+ 0.00 [-0.10, +0.10]
+ 0.00 [-0.08, +0.08]
T1
-0.04 [-0.18, +0.10]
+0.08 [-0.02, +0.19]
+0.08 [ + 0.00, +0.17]
T2
+0.12 [-0.02, +0.26]
+0.10 [-0.01, +0.22]
+0.10 [+0.01, +0.19]
O1
-0.02 [-0.16, +0.12]
+0.15 [+0.02, +0.28]
+0.14 [+0.04, +0.26] †
C1
+0.14 [ + 0.00, +0.28]
+0.02 [-0.10, +0.13]
+0.02 [-0.07, +0.10]
Appendix
Table 3: Whole-suite score as specified before running experiments, airline, all seven mutations: Δm with paired-bootstrap 95% interval. S2 values are means over the 20 size-matched draws. Detection requires Δm>θ=0.10 and an interval excluding zero; the only detection is marked † . S2 is detected only if a majority of draws detect, so its mean interval can exclude zero without a detection. T2 on S3 equals θ exactly (9 of 90 scenarios) and so does not exceed it. Baseline pass rates 0.50 (S1), 0.88 (S2 mean), 0.88 (S3). The single detection is threshold-fragile: at θ = 0.05 T2 is also detected on S3, and at θ = 0.15 nothing is detected by any suite.
Mutation
Suite
n
Δ [95% CI]
detected
O1
S1
48
+0.17 [+0.04, +0.31]
yes
O1
S2
48
+0.10 [-0.04, +0.24]
no
O1
S3
50
+0.10 [-0.04, +0.24]
no
O1
S3, programmatic only
33
+ 0.00 [-0.12, +0.12]
no
O1
S3, judge only
50
+0.20 [+0.06, +0.34]
yes
T1
S1
48
-0.02 [-0.21, +0.17]
no
Appendix
Table 4: Retail replication, airline protocol unchanged: Δm with paired-bootstrap 95% interval, positive is a drop in pass rate; combined grading unless a view is named; S2 is the mean over 20 draws. Detection as in Table 3 ; scoped rows are post hoc, n is the number of scenarios exercising the mutated element, flips are pass → fail / fail → pass.
airline
retail
Scenario kind (size-matched)
size 30
size 16
happy path only
0 / 7
0 / 2
all three kinds
6 / 140 (O1: 5 of 20 draws)
0 / 40
Personas
(not size-matched)
P=1
0 2 / 140 (size 11)
2 / 40 (size 0 6)
P=2
0 6 / 140 (size 23)
4 / 40 (size 12)
Appendix
Table 5: Diversity ablations on the generated suite, counted as detections over (draw × mutation) trials at θ=0.10 . Airline instantiates 7 mutations, retail 2. Scenario kind is size-matched: the happy-only arm is the single deterministic sub-suite of all happy scenarios, the all-kinds arm is 20 draws of the same size from all three kinds. Personas is not size-matched (Appendix E ). Retail’s kind ablation is degenerate and reported for completeness.
airline detections
retail detections
ceiling-bound
θ
S1
S2
S3
S1
S2, S3
(airline, of 5)
0.05
none
none
O1, T2
O1
none
1 (R1)
0.10
none
none
O1
O1
none
4 (R1, R2, T1, O1)
0.15
none
none
none
O1
none
4 (R1, R2, T1, O1)
Appendix
Table 6: Specified-score detections and ceiling-bound airline defects at the three thresholds fixed before the runs. “none” means no mutation is detected on that suite.
Invariant suites and runtime monitors increasingly gate agent deployment decisions, and the evidence offered for any particular suite is almost always a single observation: it passes an implementation believed correct and fails one believed broken. We measure what that observation is worth. Applying mutation analysis to an invariant suite for a multi-session agent state-projection layer, we first find that this standard validation certifies a suite in which a first-order mutant removing event-identity deduplication survives every check. We then freeze the repaired twelve-check suite, record its hash, and run it once against ten mutants specified by an adversarial reader who designed none of its fixtures: it kills five. Instrumenting the five survivors against the reference shows they fail in two distinct ways, not one. Three are never activated, because no fixture supplies an input on which the mutated code behaves differently at all. The other two corrupt internal state that no oracle in the suite can observe. The two modes need different repairs, and neither is visible from a pass/fail report. Treating the missing inputs as a coverage question, we enumerate seven discriminating dimensions of the input space, register in advance which are uncovered and which survivors they should explain, and add one fixture per uncovered dimension while reusing the existing oracles verbatim. All five survivors then die, each to the check written for its predicted dimension. We report this as a repair result on the same challenge set rather than a second held-out estimate, and give the artifact, including the frozen hash, the registered predictions, all mutants and the run logs, so the distinction is checkable.
Coding agents can return plausible patches that omit required behavior. These failures are hard to review because long traces and confident summaries often hide what was missed. We ask when a nominally weaker reviewer can reliably decide whether a patch solves its issue. We study 411 execution-labeled traces from three agents and 101 controlled cases. On 154 GPT-5.4 traces, structured but unchecked evidence raises both defect catch and over-rejection. We then provide official execution evidence as an upper-bound diagnostic. After choosing and freezing one of two formats per reviewer, five of six reviewers improve both rates on 122 held-out traces; two classify every trace correctly. Reviewer size is not a consistent predictor of quality. Because official tests are unavailable in deployment, we also evaluate a frozen cascade with patch-caused static errors and generated tests that first fail on the unpatched repository. On 121 scored held-out GPT-5.4 traces and 59 Gemini traces, its coverage is 0.89 and 0.86, risk is 0.33 and 0.26, catch is 0.76 and 0.80, and over-rejection is 0.66 and 0.67. Most false rejections occur when unresolved cases reach the reviewer. Official execution evidence shows the potential of weak review when decisive checks are available. Producing equally reliable checks without official tests remains the main bottleneck.
Junyu Guo, Shangding Gu, Ming Jin +1
University of California, Berkeley · Virginia Tech
Enterprise agents need retrieval evaluation sets that expose domain-specific failures and guide developers towards efficient deployment tradeoffs. Neither expert-human-authored sets nor current synthetic generation methods provide a methodology for achieving the necessary coverage, reliability, efficiency, and discriminability required for such decision-making. Simultaneously, the field does not have an assessment framework for determining when an evaluation set has satisfied these properties. We introduce an evaluation set assessment framework for these criteria and CoverSynth -- a generator specifically designed to satisfy these criteria without incurring the high cost of human expert labeling. Starting from an unannotated corpus, CoverSynth targets coverage across retrieval demands and system capabilities. Deterministic checks locate evidence quotes in their sources, while model-based checks assess semantic validity to guarantee reliable ground truth. Across five public enterprise-style corpora, our method provides informative variation in difficulty, revealing unseen failure modes such as an inability of agents to handle underspecified or ambiguous requests. On the RepLiQA corpus, 91% of human-generated queries can be answered by all members on a panel of 12 retrieval systems, compared to only 12% of CoverSynth-generated queries. Finally, in a repeated TechQA/RepLiQA study, 12 of 12 within-harness model contrasts have positive nominal 95% intervals on generated sets, compared with 4 of 12 on human sets. CoverSynth makes coverage and evidence quality explicit construction objectives, producing corpus-grounded evaluations that reveal retrieval headroom and differences between agent configurations.