Generated Evaluation Suites as Diversity-Driven Search for Real Agent Regressions
Organizations: Independent Researcher
Abstract
Generating an evaluation suite from an agent's specification is diversity-driven search: a generator fills a descriptor space (requirements personas scenario kinds), and the suite it emits is an archive, then scored like a benchmark by its mean pass rate, which throws away the search's product. We inject 7 controlled defects into a -bench agent across two domains. Coverage yields a ceiling: if a defect's traced footprint holds scenarios that pass at baseline, it can itself contribute at most to a suite of . For 4 of 5 traceable defects that ceiling lies at or below the detection threshold , so no within-footprint effect can move the suite past it; in both domains the tool-removal defect destroys every covered scenario it can (exactly its ceiling) while the suite-level pass rate moves the other way. Defects split into two regimes: trace-aligned ones, where per-cell scoring recovers the tool-removal defect in full and reveals a second as a significant improvement, and trace-orthogonal ones, whose footprint is behavioural: forced escalation is detected in retail at a traced coverage of exactly zero. Diversity along the generator's own axes helps only where it aligns: at fixed suite size a happy-path-only suite detects nothing, while a kind-diverse one detects the orchestration defect in 5 of 20 draws. Evaluation archives should be scored per cell and should publish coverage as a first-class property.
Figures & tables
| ceil | whole-suite [95% CI] | scoped [95% CI] | |||
| airline | |||||
| R1 | S1 | 13/50 | 0.04 | -0.06 [-0.22, +0.10] | -0.31 [-0.54, -0.08] |
| R1 | S3 | 0 0/90 | 0.00 | +0.04 [-0.03, +0.13] | n/a |
| T1 | S1 | 0 7/50 | 0.06 | -0.04 [-0.18, +0.10] | +0.43 [+0.14, +0.86] † |
| T1 | S3 | 12/90 | 0.09 | +0.08 [ + 0.00, +0.17] | +0.67 [+0.42, +0.92] † |
| O1 | S3 | 0 9/90 | 0.10 | +0.14 [+0.04, +0.26] † | +0.11 [ + 0.00, +0.33] |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Id | Class | Operator | Field grounding |
|---|---|---|---|
| R1 | Requirement | Delete one policy clause (24-hour cancellation) | specification violation [ 3 ] |
| R2 | Requirement | Invert one policy condition (“cannot” “can”) | incorrect functionality [ 19 ] |
| T1 | Tool | Remove one tool ( cancel_reservation ) | tool-contract breakage [ 1 ] |
| T2 | Tool | Rename one parameter in a tool schema | API/serialization mismatch [ 19 ] |
| O1 | Orchestration | Invert the escalation rule (always transfer to a human) | orchestration and execution control [ 1 ] |
| C1 | Context | Truncate the policy document to its first 40% | context loss [ 1 , 3 ] |
| Mutation | S1 | S2 | S3 |
|---|---|---|---|
| R1 | -0.06 [-0.22, +0.10] | +0.05 [-0.05, +0.16] | +0.04 [-0.03, +0.13] |
| R2 | +0.06 [-0.10, +0.22] | + 0.00 [-0.10, +0.10] | + 0.00 [-0.08, +0.08] |
| T1 | -0.04 [-0.18, +0.10] | +0.08 [-0.02, +0.19] | +0.08 [ + 0.00, +0.17] |
| T2 | +0.12 [-0.02, +0.26] | +0.10 [-0.01, +0.22] | +0.10 [+0.01, +0.19] |
| O1 | -0.02 [-0.16, +0.12] | +0.15 [+0.02, +0.28] | +0.14 [+0.04, +0.26] † |
| C1 | +0.14 [ + 0.00, +0.28] | +0.02 [-0.10, +0.13] | +0.02 [-0.07, +0.10] |
| Mutation | Suite | [95% CI] | detected | |
|---|---|---|---|---|
| O1 | S1 | 48 | +0.17 [+0.04, +0.31] | yes |
| O1 | S2 | 48 | +0.10 [-0.04, +0.24] | no |
| O1 | S3 | 50 | +0.10 [-0.04, +0.24] | no |
| O1 | S3, programmatic only | 33 | + 0.00 [-0.12, +0.12] | no |
| O1 | S3, judge only | 50 | +0.20 [+0.06, +0.34] | yes |
| T1 | S1 | 48 | -0.02 [-0.21, +0.17] | no |
| airline | retail | ||
| Scenario kind (size-matched) | size 30 | size 16 | |
| happy path only | 0 / 7 | 0 / 2 | |
| all three kinds | 6 / 140 (O1: 5 of 20 draws) | 0 / 40 | |
| Personas | (not size-matched) | ||
| 0 2 / 140 (size 11) | 2 / 40 (size 0 6) | ||
| 0 6 / 140 (size 23) | 4 / 40 (size 12) | ||
| airline detections | retail detections | ceiling-bound | ||||
|---|---|---|---|---|---|---|
| S1 | S2 | S3 | S1 | S2, S3 | (airline, of 5) | |
| 0.05 | none | none | O1, T2 | O1 | none | 1 (R1) |
| 0.10 | none | none | O1 | O1 | none | 4 (R1, R2, T1, O1) |
| 0.15 | none | none | none | O1 | none | 4 (R1, R2, T1, O1) |
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 563 papers