Offline tests can fail to reveal whether an agent update will remain safe after deployment. We operationalize this post-audit mirage using matched safe and harmful lifecycles that expose identical verifier-visible evidence while an isolated scorer records opposite deployment truth. Our theorem applies standard observational-equivalence and indistinguishability ideas to this executable audit boundary and states the resulting error-abstention tradeoff. The Post-Audit Mirage benchmark makes that boundary executable across authorization, scheduling, and shared-capacity environments. IdentifiedRangeMonitor uses fresh group-labelled harm outcomes to return deploy, hold, cannot determine, or unsupported. Experiments show that monitoring can recover the missing distinction in favorable settings but may remain undecided for rare groups, small effects, or missing observations. Verifier-agent evaluation must therefore test both calculation validity and whether the available evidence supports the deployment decision.
Figures & tables
Figure 1: Matched worlds expose the same complete offline evidence, forcing the same verifier decision, while their post-update outcomes require opposite actions. The Safe and Harmful labels shown for World A and World B are protected scorer-only truth and are never available to the offline verifier.
Family
Agent update
Public audit
Scorer-only deployment effect
AuthGate
Change route access by group
Immediate access, audit harm, task quality
Delayed route harm, correlated incidents, live group occupancy
Co-arrival correlation, contention, backlog, delayed group harm
Table 1: Exact benchmark families and their evidence boundary. Public audit fields are method-facing; deployment effects remain scorer-only.
Figure 2: The fail-closed monitor uses fresh group-labelled harm outcomes after the update is fixed. It deploys only when every required group’s upper bound is safe, holds when any lower bound is harmful, and returns Need More Evidence , meaning cannot_determine , while the plausible harm-rate range still crosses the threshold; biased or shifted monitoring is unsupported .
Figure 3: Fresh monitoring needed for reliable separation. Each cell gives the first attempted-observation count at which safe and harmful streams meet the stated correctness and harmful-approval requirements. “Not reached” means no inspected look qualifies by 20,000 attempted observations. Larger safety margins, more common affected groups, and more complete observations generally require less data.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Safe world (%)
Harmful world (%)
Look
Deploy
Hold
Cannot
Deploy
Hold
Cannot
5
0.0
0.0
100.0
0.0
0.0
100.0
10
0.0
0.0
100.0
0.0
0.0
100.0
20
0.0
0.0
100.0
0.0
0.4
99.6
50
0.0
0.0
100.0
0.0
36.8
63.2
100
0.0
0.0
100.0
0.0
36.8
63.2
Appendix
Table 2: Fixed-panel Hoeffding monitor status rates at every evaluated look. These 272 heterogeneous synthetic lifecycles per world form a fixed descriptive panel, not independent draws supporting a population error-rate estimate.
Safe world (%)
Harmful world (%)
Look
Deploy
Hold
Cannot
Deploy
Hold
Cannot
5
0.0
0.0
100.0
0.0
0.0
100.0
10
0.0
0.0
100.0
0.0
0.0
100.0
20
0.0
0.0
100.0
0.0
0.0
100.0
50
0.0
0.0
100.0
0.0
0.0
100.0
100
0.0
0.0
100.0
0.0
0.0
100.0
Appendix
Table 3: Controlled-grid Hoeffding monitor status rates at every evaluated look.
Safe world (%)
Harmful world (%)
First decision
Deploy
Hold
Deploy
Hold
5
0.0
0.0
0.0
0.0
10
0.0
0.0
0.0
0.0
20
0.0
0.0
0.0
0.4
50
0.0
0.0
0.0
36.4
100
0.0
0.0
0.0
0.0
Appendix
Table 4: Fixed-panel first-decision stopping-time distribution, including right-censoring after 20,000 observations.
Safe world (%)
Harmful world (%)
First decision
Deploy
Hold
Deploy
Hold
5
0.0
0.0
0.0
0.0
10
0.0
0.0
0.0
0.0
20
0.0
0.0
0.0
0.0
50
0.0
0.0
0.0
0.0
100
0.0
0.0
0.0
0.0
Appendix
Table 5: Controlled-grid first-decision stopping-time distribution, including right-censoring after 20,000 observations.
Safe world (%)
Harmful world (%)
Method
Deploy
Hold
Cannot
Deploy
Hold
Cannot
Hoeffding CS
100.0
0.0
0.0
0.0
100.0
0.0
Wilson pointwise
100.0
0.0
0.0
0.0
100.0
0.0
Beta-binomial CS
100.0
0.0
0.0
0.0
100.0
0.0
Appendix
Table 6: Fixed-panel final-look status comparison at 20,000 attempted observations. Wilson is pointwise only; both confidence-sequence rows are stopping-valid under the declared model. Unsupported was 0.0% throughout.
Table 7: Controlled-grid decision-onset comparison. Counts in parentheses are numbers of the 18 prespecified cells whose first qualifying inspected look occurred at that value. Wilson onsets are descriptive because Wilson is pointwise.