cs.AIOct 29, 2026
SaveEnvOS: Auditing Computer-Use Agents When the World Moves On : Phenomenon-Based Task Construction, Certification, and Evidence-Based Evaluation
Organizations: QpiAI, Bangalore, India
Abstract
Computer-use agents built on large language models (LLMs) work in software that can change while they act: an appointment moves, a policy is revised, a record is updated. Testing agents under such changes is harder than it looks. When an agent fails, the cause may be the agent, but it may also be a change that did not fire as intended, an update the agent never received, or a grader that scored the episode wrongly. We introduce EnvOS, a framework for building and certifying environments that change mid-task. Before any agent runs, EnvOS checks that each change fires correctly, that the task remains solvable, and that the grader separates correct from incorrect behavior; after each run, separate code re-checks the verdict from archived evidence. We apply it to stale information, where a value the agent has already seen changes before the agent acts on it. In clinic booking, Claude Opus 5 succeeds on 0/16 tasks when nothing extra is shown, 2/16 when reminded to re-check the source, and 15/16 when the updated value is shown; the gain over the reminder holds after correcting for all tests (p = 0.00293). In insurance-claim approval, Opus 5 re-checks on its own and succeeds on 10/10, whereas Claude Sonnet 5 rises from 1/10 to 9/10 when the current policy is shown again, a gain significant in its pre-registered test but not after correcting for all tests. Whether an agent caught a change thus varied with the model and the interface. We also stress-test the grading itself: on corrupted episodes designed blind from the rules, the EnvOS pipeline accepted none of 26 as clean, and on an externally generated set it caught 25 of 39, against 4 for simple log checks. Because every EnvOS environment is generated from a seed and graded by an audited automatic verifier, the same environments are natural candidates for training agents with reinforcement learning from verifiable rewards.
Figures & tables
Figure 1: Authoritative state and model observations during a stale-information intervention. The application serves before a one-time exposure-conditioned mutation produces . The review surface may disclose nothing, a link to the source, or the current value; these are observation conditions, not changes to the underlying mutation. Observation delivery is logged separately, and the evaluator scores committed application state rather than the disclosure review display.
Figure 2: EnvOS evaluation lifecycle. Task and phenomenon specifications define a candidate executable intervention. Pre-run checks assess the intervention, observation channel, feasibility, behavioral discrimination, and scorer robustness, determining whether the candidate is admitted to evaluation. Admitted instances are executed under fixed rules; their archived trajectories support validity classification, state-based task scoring, independently implemented reconstruction, and evaluator challenges. Admission applies to the specified checks and does not guarantee the validity of every subsequent episode.
| Assurance question | Candidate-instance checks | Run-level evidence and limits |
|---|---|---|
| World validity | Authorized mutation, bounded scope, one-shot execution, anchored timing | State and access logs establish when the authoritative event actually fired |
| Observation consistency | Rendering comes from updated source state; available channels are consistent with intended condition | Served pages and delivered model frames are archived separately; neither establishes comprehension |
| Recovery validity | Reference completion route and declared deficient-policy discrimination, including frozen-instance feasibility | Actual model actions, rereads and commits are observed, not inferred from the scripted policy |
| Checker robustness | Scorer localization checks, negative mutations and positive controls | Validity-first inclusion, independently coded verdict reconstruction and external archive challenges test residual mistakes |
Table 1: One EnvOS evaluation pipeline. The four assurance questions organize pre-run certification; the six audit stages track the evidence before, during, and after a particular model episode. Passing one part does not imply that another part was checked.
| Clinic booking | Insurance claim approval | |
|---|---|---|
| Phenomenon | A booked session’s previously served location is superseded | A policy’s previously served reimbursement limit is superseded |
| Authoritative object | Target clinic session row | Target insurance policy row |
| Prior serve | Sessions page displays previous location | Policy page displays previous limit |
| Perturbation trigger | First eligible booking-wizard request after serve | First eligible form-entry request after serve |
| Mutation | Update the relevant stored location | Update the relevant stored limit |
| Disclosure | None, source link, or current location | None or re-exposed current policy record |
Table 2: The phenomenon–perturbation mapping in the two prospective workflows. Entries specify the experimental roles, not the literal values of any generated instance.
| Task | Code | Information available at the review step |
|---|---|---|
| Clinic | OFF-A | No additional information at booking confirmation; the agent may revisit the original session record. |
| Clinic | CUE-A | A reminder identifies the session and provides a link for verification, without revealing the updated location. |
| Clinic | LIVE-A | The same reminder and link are accompanied by the updated session location. |
| Insurance | OFF-A | No additional policy information before approval; the original policy record remains accessible. |
| Insurance | ON-A | Current policy record is shown again, including the limit but not the approval amount. |
| Controls | -I, L0 | Irrelevant-record change at the trigger, or no change, respectively. |
Table 3: Experimental information conditions. In all principal comparisons, the application undergoes the same task-relevant state change and the original source remains accessible. Only the information supplied to the agent before its final decision differs.
| Check family | What is tested | Example invalid case | What passing does not prove |
|---|---|---|---|
| World / trigger | old-value serve; anchored one-time update; condition-equivalent neutral operations | mutation fires before prior serve or fires twice | agent noticed the old value |
| Condition discrimination | reference and deficient scripted-policy trajectories and expected signatures | a value-only script passes when the value was hidden | arbitrary models will find the reference route |
| Instance feasibility | scripted completion and discrimination on each frozen generated instance | generated slot or decision is impossible | all future cases are feasible |
| Scoring instrumentation | event localization and registered target/commit invariants | scorer associates change with wrong request | every outcome field is reconstructable |
| Mutation / positive controls | deliberately corrupted and valid archives | damaged event enters clean; good archive is rejected | unknown corruption classes are covered |
Table 4: Certification checks are admission evidence with specific failure targets. All checks are scoped to the tested scripted routes and generated instances.
| Stage | Evidence | Fault exposed | Boundary |
|---|---|---|---|
| Certify | scripted policies, trigger logs, per-instance feasibility and mutation controls | update fires early/twice; reference path impossible; scorer misses event | scripted paths only |
| Isolate | sealed OS user, allowed origins, canary access probes, observation-delivery records | policy leaks evaluator data or misses promised channel | unarchived provider-side returns remain |
| Freeze | preregistration, SHA-256 manifests, fixed templates | post-outcome changes to analysis or software | dated amendments are still possible and disclosed |
| Count | validity table, provider/harness log, reserves and stop rules | provider outage treated as model failure; invalid rerun cherry-picked | validity classification can itself err |
| Re-derive | separately implemented outcome scorer and standard-library validity checker | incorrect success invariant or eligibility verdict | same team; shared archive; unresolved terms |
| Challenge | corrupted archive copies, sealed expectations, baselines and external generator | detector false accepts and valid-control false rejects | construction access, disputes and coverage vary |
Table 5: Audit stages, evidence and scope. Every successful check establishes a limited property rather than a general guarantee of benchmark correctness.
| arm successes | paired | for/ag./ | local | indep. | |||
|---|---|---|---|---|---|---|---|
| model (block) | OFF | ON/CUE | LIVE | contrast | tie | (block Holm) | endpoint |
| insurance-claim approval | |||||||
| Opus 5 (CLM-B) | 10/10 | 10/10 | — | ON OFF | 0/0/10 | 1.00 | 24/24 |
| Sonnet 5 (CLM-BS) | 1/10 | 9/10 | — | ON OFF | 8/0/2 | 0.00781 | 24/24 |
| Gemini (CLM-BG) | 0/10 | 5/10 | — | ON OFF | 5/0/5 | 0.0625 | 24/24 |
| clinic booking | |||||||
Table 6: Prospective model results. Success requires the full evaluator endpoint; in clinic rows its calendar-compatibility term rests on a stated design condition, not independent re-derivation. Paired counts: treatment-only/control-only/ties; : exact two-sided sign test (BC: pre-registered block-Holm value in parentheses). ‘indep. endpoint’: counted rollouts whose independent reconstruction (clinic: without calendar compatibility) agrees with the evaluator. Blocks without results print their state; nets, intervals: Table 31 . Gemini 3.1 Pro Preview received each frame inline in a stateless REST request with no tools; the Claude models were instructed to read each frame file with their file-read tool. Provider, model, invoker and image-delivery channel differ together; rows describe model–invoker configurations, not vendor effects.
Figure 3: Matched experimental outcomes by instance. Insurance-claim approval compares no additional policy information (OFF-A) against re-exposure to the current policy (ON-A); clinic booking compares no additional information (OFF-A), a verification reminder and source link (CUE-A), and direct disclosure of the updated location (LIVE-A). Each panel corresponds to a distinct registered model–workflow configuration; outcomes are not pooled across models.
| checker | controls | corruptions (in scope) | would enter the results unflagged |
| clinic: the sealed challenge’s 10 controls and 18 corruptions, its recorded outputs re-tabulated | |||
| protocol, implemented reading | 10/10 | 15/18 (11/12) | notice delivery ( C01 ); delivery of review-time information ( C02 ); commit before change ( C08 ) |
| protocol, registered reading | 10/10 | 16/18 (11/12) | commit before change ( C08 ); truncated trajectory ( C15 ) |
| simple baseline (143 lines) | 9/10 | 11/18 (11/12) | 7, of which 6 outside its declared scope |
| claims: 10 controls and 19 corruptions we built from counted archives | |||
| operational pipeline | 10/10 | 16/19 (10/12) | duplicate commit in the access log only ( C09a ); truncated trajectory ( C15 ); stale pages after the change ( C17p ) |
Table 7: The protocol against a simple raw-log baseline (post hoc, author-built, not blind; separate from Table 12 ). Same raw evidence and scored properties. Controls accepted; corruptions rejected (not counted, or counted with the corrupted invariant flagged), in brackets those in the baseline’s declared scope. Clinic protocol rows are derived from the sealed tools’ recorded outputs; the claims pipeline voids C09b as invalid rather than failing its single-commit term. Every case: Appendix D .
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
| rule | blocks | registered | implemented | retrospective correction | archived values re-derived |
|---|---|---|---|---|---|
| Trigger anchor | A (superseded) | PRECOMMIT:19 | anchor ’second authenticated GET /’ (code replaced; archived in superseded log rows) | ADD 3 (PRECOMMIT:203): exposure-conditioned anchor for new blocks; scorer after_login fix (PRECOMMIT:213) | none persisted: archived C4 of superseded S-L2/S-C s0-s2 still reads False; 3 untriggered episodes stay uncounted; block superseded |
| Trigger anchor | A2, A2S | PRECOMMIT:206 | superseded_session.py:442 | none | none |
| Trigger anchor | B, BS, BF | PRECOMMIT:274 | superseded_session.py:449 | ADD 5 kept the late fire and redefined the reference policy (PRECOMMIT:280) – before any rollout | none (prospective) |
| Validity before outcome (ladder order) | all | PRECOMMIT:41 | telemetry.py:32 (doc) vs telemetry.py:232 precedes telemetry.py:239 | ADD 11 discloses success-first order (PRECOMMIT:454); ADD 16 repeats it (PRECOMMIT:653) | no verdict re-derived; this script lists SUCCESS rows that validity-first would void |
| Silent-arm perception branch | B (row 1); BS, BF prospective | PRECOMMIT:42-54 (the §2 table has no silent-arm row) | telemetry.py:253-254 | ADD 10 (PRECOMMIT:401) | changed_FRX-OFF-A_test_id_s20_claude-opus-5: delivery_ambiguous model_failure; pre-fix classifier would have voided 25 silent-arm model failures |
| Notice delivery (C3) | A2, A2S (announced arms) | PRECOMMIT:50 | preimage[2026-09-22T0004]/provenance.py:632 (pre-image; frame + ack on classified route) provenance.py:663; verdict telemetry.py:266 | ADD 11 (ack on unclassified route, PRECOMMIT:438); ADD 16 (text or image, PRECOMMIT:619) | ADD 11: 5 A2 rows re-derived (changed_S-L2_s9 opus-5, changed_S-L2_s11 opus-5, cosmetic_S-C_s13 opus-5, cosmetic_S-C_s14 opus-5, cosmetic_S-C_s15 opus-5): C3 false true on 4, 4 signature changes; ADD 16: C3 false true on 2 rows (cosmetic_S-C_s12 opus-5, cosmetic_S-C_s15 opus-5); 0 verdict changes |
| rule | common | claims | clinic | historical |
|---|---|---|---|---|
| trigger | fires on the first eligible request after the old value was served (exposure-conditioned) | first form entry after the policy records page | first booking-wizard request after the sessions page | exposure-conditioned from Block A2 on (ADDENDUM 3) |
| untriggered episode | registered per workflow | counted as a failure, reported separately | not counted; replaced | not counted; replaced |
| counting | validity first; only uncounted validity failures replaced, under a reserve (2 per cell, 6 per block) | the claims validity-first table; the generic classifier verdict is never read | the ADDENDUM 18 validity table | success first as implemented; the validity-first re-analysis is Table 10 |
| frame delivery | admitted per channel | file reads | file reads (an inline channel is defined for GPT-6 blocks) | read rule as implemented, one reading post hoc (above); ADDENDUM 5’s review-screenshot rule not enforced |
| endpoint | invariants on world state and the agent’s report | four (no step after the commit) | five | five, evaluator-scored; the nearest-slot term cannot be independently verified |
| independent audits | outcome scorer and standard-library validity re-derivation, claimed only once the artifact exists | both present | claimed when produced | archive re-analyses (Appendix C.1 ) |
Table 9: The protocol as operated. Common: every prospective block. Claims: CLM-B and CLM-BS (PRECOMMIT_CLAIMS). Clinic: BC and BCS (ADDENDA 18–19). Historical: Blocks A2, A2S, B, BS and BF (predecessor versions; Appendix T ).
| block | analysis | control | treatment | ON-I / S-L0 | for/against/tied; | unobserved rows |
| B | reported (implemented rules) | 0/10 | 10/10 | 3/3 | 10/0/0; 0.00195 | none |
| ADD. 5 screenshot rule as registered | 0/10 | 10/10 | 3/3 | 10/0/0; 0.00195 | none | |
| ADD. 5, literal (also voids review information never rendered) | 0/10 | 10/10 | 3/3 | 10/0/0; 0.00195 | none | |
| BS | reported (implemented rules) | 0/10 | 6/10 | 2/3 | 6/0/4; 0.0312 | none |
| ADD. 5 screenshot rule as registered | 0/10 | 6/7 | 0/1 | 6/0/1; 0.0312 | FRX-ON-A s22, FRX-ON-A s27, FRX-ON-A s28, FRX-ON-I s21, FRX-ON-I s22 | |
| ADD. 5, literal (also voids review information never rendered) | 0/10 | 6/6 | 0/0 | 6/0/0; 0.0312 | FRX-ON-A s22, FRX-ON-A s23, FRX-ON-A s27, FRX-ON-A s28, FRX-ON-I s20, FRX-ON-I s21, FRX-ON-I s22 |
Table 10: Reported results and results under the original rules. “Validity first” applies the delivery rule before success, with the notice checkpoint as first archived, after ADDENDUM 11, and after ADDENDUM 16. for/against/tied are paired discordances; unobserved rows are those the rule voids.
| held-out case | kind | expected to fail | observed to fail | met |
|---|---|---|---|---|
| valid_reference_passes | noop | none | none | yes |
| stale_location_via_history_actor | sql | I1 | I1, I5 | no |
| second_active_row_same_label | sql | I2 | I2 | yes |
| collateral_slot_shift_on_bystander | sql | I3 | I3 | yes |
| cancelled_then_reconfirmed_row | sql | I1, I2, I4, I5 | I1, I2, I4, I5 | yes |
| report_case_and_whitespace_variant_is_still_correct | report | none | none | yes |
Table 11: The held-out mutation challenge, run once. Expected: the invariants the frozen expectation says must fail (I1 target state correct, I2 exactly one commit, I3 no collateral change, I4 workflow continued from the commit, I5 final report correct); “none” marks a valid case the checker must accept.
| layer | sealed challenge | what it establishes | known gap |
|---|---|---|---|
| independent outcome scorer | 28/28 | every frozen expectation met on this suite | a term it cannot resolve is reported unresolved; clinic calendar compatibility is assumed; no historical success is verified in full |
| validity re-derivation (standard library) | 24/28 | accepted 10/10 valid controls and rejected 14/18 corruptions on this suite | blind spots: missing observation delivery (2); commit before change (1); truncated trajectory (1) |
| evaluator, archived verdict path | 10/28 | met 10/10 valid controls but rejected 0/18 corruptions because archived validity/invariant claims were retained | re-derives the delivery checkpoints from raw logs but keeps the archived run-validity and invariant claims |
Table 12: Auditing layers against the sealed archive challenge (first and only run; expectations sealed by digest before it). Counts are frozen expectations met; the per-case table is Table 13 .
| case class | cases | outcome scorer | validity re-derivation | evaluator (replay) |
|---|---|---|---|---|
| valid controls | 10 | 10 of 10 | 10 of 10 | 10 of 10 |
| C01 success-before-validity: notice | 1 | met | no : validity | no : validity |
| C02 missing review-time information delivery | 1 | met | no : validity | no : validity |
| C03 change before old value served | 1 | met | met | no : validity |
| C04 change fired twice | 1 | met | met | no : validity |
| C05 change in nonagent request | 1 | met | met | no : validity |
Table 13: The sealed archive challenge, first and only run, per tool and case class. “met”: every field the frozen expectation assesses for that tool was met; “no” names the fields that were not (for the validity re-derivation, validity is its verdict under the rules as implemented and validity (registered) under the rules as registered). The outcome scorer assesses I1–I5 only; the validity re-derivation assesses validity only; the evaluator’s invariants are its archived claims. From CHALLENGE_RESULT.json , pinned by sha256 in the ledger.
| verdict established | |||||||
| block | model | counted | validity (all att.) | successes | failures | contra. | unresolved or assumed |
| historical blocks (exploratory; predecessor protocol versions; archive-derived evidence) | |||||||
| A2 | Opus 5 | 18 | 19/19 | 0/17 | 1/1 | 0 | nearest slot (I1) on every success |
| A2S | Sonnet 5 | 9 | 9/9 | 0/7 | 2/2 | 0 | nearest slot (I1) on every success |
| B | Opus 5 | 23 | 23/23 | 0/13 | 10/10 | 0 | nearest slot (I1) on every success |
| BS | Sonnet 5 | 23 | 26/26 | 0/8 | 15/15 | 0 | nearest slot (I1) on every success |
Table 14: What the independent audits established, per block. Counted rollouts; validity verdicts re-derived by the standard-library implementation, agreeing / all attempts (exclusions included; not run where the block’s re-derivation artifact does not exist); verdicts established by the independent scorer (for a failure, one term resolved false suffices; for a success, every term must resolve true); invariant contradictions; and what is unresolved or rests on a design assumption rather than on archived evidence. Prospective cells count only audits that have run.
| counted successes | counted failures | ||||||
|---|---|---|---|---|---|---|---|
| block | attempts | clinic term | slot evidence | all five | all five | ||
| A2 | 19 | 17 | 17 | 0 | 0 | 1 | 1 |
| A2S | 9 | 7 | 7 | 0 | 0 | 2 | 2 |
| B | 23 | 13 | 13 | 0 | 0 | 10 | 10 |
| of which FRX-ON-A (Opus’s rescues) | 10 | 10 | 0 | 0 | |||
| BS | 26 | 8 | 8 | 0 | 0 | 15 | 15 |
Table 15: What is independently verified, by outcome. Counted rollouts per block: the booked-clinic term resolved and agreeing; the same-episode slot evidence complete; all five invariants resolved and agreeing. From SLOT_EVIDENCE.json and the r19 model audit, pinned by sha256 in the ledger.
| block | att. | counted, archive | counted, re-derived | agree | counted, registered | registered rule differs on |
|---|---|---|---|---|---|---|
| A2 | 19 | 18 | 18 | 19 | 18 | – |
| A2S | 9 | 9 | 9 | 9 | 9 | – |
| B | 23 | 23 | 23 | 23 | 23 | – |
| BS | 26 | 23 | 23 | 26 | 16 | FRX-ON-A s22, FRX-ON-A s23, FRX-ON-A s27, FRX-ON-A s28, FRX-ON-I s20, FRX-ON-I s21, FRX-ON-I s22 |
| BF | 5 | 5 | 5 | 5 | 5 | – |
| all | 82 | 78 | 78 | 82 | 71 |
Table 16: Validity verdicts, re-derived independently. Per block: attempts; counted in the archive; counted by the re-derivation under the implemented rules; verdict classes agreeing with the archive; counted under the rules as registered, and the attempts on which the registered rule differs. From VALIDITY_INDEPENDENT.json , pinned by sha256 in the ledger.
| episodes | I1 | I2 | I3 | I4 | I5 | |
|---|---|---|---|---|---|---|
| Block A2, counted | 18 | 1/0/17 | 18/0/0 | 18/0/0 | 18/0/0 | 18/0/0 |
| Block A2S, counted | 9 | 2/0/7 | 9/0/0 | 9/0/0 | 9/0/0 | 9/0/0 |
| Block B, counted | 23 | 10/0/13 | 23/0/0 | 23/0/0 | 23/0/0 | 23/0/0 |
| Block BS, counted | 23 | 15/0/8 | 23/0/0 | 23/0/0 | 23/0/0 | 23/0/0 |
| Block BF, counted | 5 | 4/0/1 | 5/0/0 | 5/0/0 | 5/0/0 | 5/0/0 |
| all counted model episodes | 78 | 32/0/46 | 78/0/0 | 78/0/0 | 78/0/0 | 78/0/0 |
Table 17: Independent-scorer coverage. Resolved-and-agreeing / contradiction / unresolved, archived versus recomputed invariant (I1–I5 as in Table 11 ), over counted model episodes per block, and over the certification episodes as re-scored now and as audited when cited.
| case | class tested | expected | C8 | scorer | eval./ classifier | baseline | pipeline |
|---|---|---|---|---|---|---|---|
| V01 | clean control (unarmed L0) | counted | met | met | met | met | met |
| V02 | inapplicable change control (ON-I) | counted | met | met | met | met | met |
| V03 | OFF-A fired, correct approval | counted | met | met | met | met | met |
| V04 | ON-A information-delivery control (Opus) | counted | met | met | met | met | met |
| V05 | ON-A information-delivery control (Sonnet) | counted | met | met | met | met | met |
| V06 | valid behavioural failure (stale approval) | counted; I1 fail | met | met | met | met | met |
Table 18: The post-hoc claims-path challenge, per case (author-constructed, not blind; one run). Expected: the class sealed before the run (counted cases: the invariants that must fail). Controls: “met” or no . Corruptions: caught , rejected or failed as sealed; wrong class , kept from a clean score but not with the sealed class; missed , scored as a clean result; scope , counted as expected but the tool does not assess the corrupted term. Pipeline: the classifier followed by the registered void gate. Separate from the sealed challenge (Table 13 ). From VALIDATION_RESULT.json , pinned by sha256 in the ledger.
| checker | validity: controls | validity: corruptions | outcome: controls | outcome: corruptions | end to end: controls | end to end: corruptions | FA / FR |
|---|---|---|---|---|---|---|---|
| clinic: the sealed challenge’s cases (10 controls, 18 corruptions) | |||||||
| validity re-derivation, implemented | 10/10 | 9/12 | n/a | n/a | n/a | n/a | n/a |
| validity re-derivation, registered | 10/10 | 10/12 | n/a | n/a | n/a | n/a | n/a |
| outcome scorer | n/a | n/a | 10/10 | 6/6 | n/a | n/a | n/a |
| evaluator (archived verdicts) | 10/10 | 0/12 | 10/10 | 0/6 | 10/10 | 0/18 | 18 / 0 |
| protocol, implemented (derived) | 10/10 | 9/12 | 10/10 | 6/6 | 10/10 | 15/18 | 3 / 0 |
Table 19: Matched comparison, every decision (post hoc, author-built, not blind). Validity: controls accepted, validity-property corruptions rejected; outcome: controls scored correctly, outcome-property corruptions with a target invariant flagged; end to end: an episode is rejected if it is not counted, or is counted with a target invariant flagged. n/a: the checker has no such output; FA/FR: false acceptances/rejections. The clinic rows other than the baseline re-tabulate the sealed challenge’s recorded first-run outputs (not re-run; the sealed result, Table 13 , is unchanged); the protocol rows are derived from them, not a tool that was run. Claims rows: the post-hoc run of Table 18 . From matched_table.json , pinned by sha256.
| case | operational pipeline (frozen) | C8 | scorer | evaluator/ classifier | r28 baseline | repaired checker | reconstruction |
|---|---|---|---|---|---|---|---|
| V01 clean control (unarmed L0, Sonnet) | met | met | met | met | met | met | met |
| V02 inapplicable change control (ON-I X_down, Opus) | met | met | met | met | met | met | met |
| V03 inapplicable change control (ON-I X_up, Sonnet) | met | met | met | met | met | met | met |
| V04 OFF-A fired, correct approval (Opus, X_down) | met | met | met | met | met | met | met |
| V05 OFF-A fired, correct approval (Sonnet, B_down) | met | met | met | met | met | met | met |
| V06 valid behavioural failure (stale approval, Sonnet) | met | met | met | met | met | met | met |
Table 20: Fresh sealed cases (post hoc, author-built, not blind; one run). Expectations and every tool digest sealed ( SEAL.txt , 10:03:30Z) before any tool ran on any case; run at 10:03:34Z. No case reuses a base archive of Table 18 . Cells as in Table 18 . The pipeline is the frozen operational path, unchanged; the repaired checker is the r28 baseline plus step-list completeness, post-change page consistency (policy pages and review display) and an explicit single commit; the reconstruction contains both kinds of check. From FRESH_RESULT.json , pinned.
| tool | controls | FR | caught | FA | scope | errors | exact |
| Claims | |||||||
| combined pipeline (classifier, void gate) | 7/7 | 0 | 18/20 | 0 | 0 | 2 | 24/27 |
| evaluator with classifier | 7/7 | 0 | 18/20 | 0 | 0 | 2 | 24/27 |
| claims validity re-derivation (C8) | 7/7 | 0 | 6/20 | 9 | 4 | 1 | 16/27 |
| independent scorer | 7/7 | 0 | 10/20 | 9 | 0 | 1 | 12/27 |
| simple raw-log baseline (r28) | 7/7 | 0 | 3/20 | 14 | 2 | 1 | 12/27 |
Table 21: Rule-only case-design challenge, per tool (cases and expectations specified by GPT-6 from the registered rules and archive schema only; implemented by the authors, fidelity-reviewed by a model, run once). Controls accepted; FR: controls rejected; corruptions rejected, or counted with the expected failing invariant flagged; FA: corruptions counted as clean; scope: counted, with the attacked invariant outside the tool’s outputs; errors: no verdict; exact: countedness and class (or invariants) equal GPT-6’s expectation. Exact is stricter than clean-versus-corrupt detection: a corruption rejected under a different registered class is non-clean but not exact.
| case | class | expected | Pipe. | Eval. | C8 | Scorer | Base. | Rep. |
|---|---|---|---|---|---|---|---|---|
| KC01 | missing trajectory | harness_error | ERR | ERR | ERR | ERR | ERR | ERR |
| KC02 | provider limit stop | infra_rejected | R | R | R | FA | FA | FA |
| KC03 | recorded infrastructure error | infra_failure | R | R | R | FA | FA | FA |
| KC04 | harness stop | harness_error | R | R | R* | FA | FA | FA |
| KC05 | unauthorized file read | harness_error | R | R | R | FA | FA | FA |
| KC06 | wrong resolved model | harness_error | R | R | FA | FA | FA | FA |
Table 22: Rule-only case-design challenge, every admitted case (class names are GPT-6’s). A: accepted; R: rejected; C: counted with the expected failing invariant flagged; FA: counted as clean; ERR: no verdict; scope: the attacked invariant is outside the tool’s outputs; R*: rejected with another class (named under each part). Bold : GPT-6’s exact expectation not met. Tools: Pipe. the combined pipeline, Eval. the evaluator with classifier, C8 and Valid. the claims and Block BC validity re-derivations, Scorer the independent scorer, Base. the simple raw-log baseline, Rep. the repaired checker. Controls are pooled into one row per workflow ( : alike).
| tool | controls | FR | caught | FA | scope | errors | exact |
| Claims | |||||||
| combined pipeline (classifier, void gate) | 6/8 | 2 | 14/20 | 4 | 0 | 2 | 14/28 |
| evaluator with classifier | 8/8 | 0 | 14/20 | 4 | 0 | 2 | 15/28 |
| claims validity re-derivation (C8) | 8/8 | 0 | 3/20 | 12 | 4 | 1 | 15/28 |
| independent scorer | 6/8 | 2 | 8/20 | 11 | 0 | 1 | 5/28 |
| simple raw-log baseline (r28) | 8/8 | 0 | 1/20 | 15 | 2 | 2 | 6/28 |
Table 23: Externally designed and instantiated challenge, per tool (cases, expectations and generator by a Gemini CLI agent from the rules, schema and archive copies only; generator run unread, cases sealed, each tool run once; the agent’s expectations, disputed ones included). Columns as in Table 21 : controls accepted; FR: controls not counted; corruptions caught (rejected, or counted with the expected failing invariant flagged); FA: corruptions counted as clean; scope: the attacked invariant is outside the tool’s outputs; errors: no verdict; exact: the agent’s full expectation met.
| CLM-B (Opus 5) | CLM-BS (Sonnet 5) | CLM-BG (Gemini 3.1 Pro Preview) | |
|---|---|---|---|
| OFF-A success | 10/10 | 1/10 | 0/10 |
| ON-A success | 10/10 | 9/10 | 5/10 |
| rescues/harms/ties | 0/0/10 | 8/0/2 | 5/0/5 |
| exact paired | 1.00 | 0.00781 | 0.0625 |
| reading | R2 | R1 | R3 |
| ON-I; L0 | 3/3; 1/1 | 3/3; 1/1 | 3/3; 1/1 |
Table 24: The three claims configurations side by side (descriptive; no pooling, no between-model test, no interval on a between-model difference). Gemini 3.1 Pro Preview received each frame inline in a stateless REST request with no tools; the Claude models were instructed to read each frame file with their file-read tool. Provider, model, invoker and image-delivery channel differ together; rows describe model–invoker configurations, not vendor effects.
| pair | case | OFF-A | ON-A | ON-A stop |
|---|---|---|---|---|
| 1 | B_up | stale approval | no commit | step cap |
| 2 | X_up | stale approval | no commit | step cap |
| 3 | B_down | stale approval | pass | model done |
| 4 | X_up | stale approval | no commit | step cap |
| 5 | X_down | stale approval | pass | model done |
| 6 | X_down | stale approval | pass (attempt a1) | model done |
Table 25: CLM-BG per pair (top) and per arm (bottom). Evaluator labels of the counted rollouts; “stop” is the ON-A rollout’s stop reason; Wilson intervals with ; “re-read”: policy record requested after the change; “back”: returned from the review step. Registered diagnostics, descriptive only.
Figure 4: Block CLM-BG in Figure 3 ’s encoding. Counted rollouts per paired instance, OFF-A and ON-A; row totals at the right. Gemini 3.1 Pro Preview received each frame inline in a stateless REST request with no tools; the Claude models were instructed to read each frame file with their file-read tool. Provider, model, invoker and image-delivery channel differ together; rows describe model–invoker configurations, not vendor effects.
| seed | case | first | claimed | deduct. | old limit | new limit | correct | stale | copy-limit OFF-A | copy-limit ON-A | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | B_up | ON-A | 4006.22 | 75.00 | 3931.22 | 1400.00 | 3575.00 | 3575.00 | 1400.00 | 1400.00 fail | 3575.00 pass |
| 1 | X_up | OFF-A | 2769.51 | 75.00 | 2694.51 | 1550.00 | 3050.00 | 2694.51 | 1550.00 | 1550.00 fail | 3050.00 fail |
| 2 | B_down | OFF-A | 1732.65 | 125.00 | 1607.65 | 1325.00 | 1175.00 | 1175.00 | 1325.00 | 1325.00 fail | 1175.00 pass |
| 3 | X_up | ON-A | 1763.86 | 175.00 | 1588.86 | 725.00 | 2150.00 | 1588.86 | 725.00 | 725.00 fail | 2150.00 fail |
| 4 | X_down | OFF-A | 4206.18 | 350.00 | 3856.18 | 6725.00 | 1700.00 | 1700.00 | 3856.18 | 6725.00 fail | 1700.00 pass |
| 5 | X_down | ON-A | 2590.59 | 350.00 | 2240.59 | 3575.00 | 1400.00 | 1400.00 | 2240.59 | 3575.00 fail | 1400.00 pass |
Table 26: The claims instances (seeds of the held-out test split; first: the arm run first in the pair). From claims_instances.json , pinned by sha256.
| block | order | pairs | OFF-A pass | ON-A pass | for ON-A | for OFF-A | ties |
|---|---|---|---|---|---|---|---|
| CLM-B (Opus 5) | OFF-A first | 5 | 5 | 5 | 0 | 0 | 5 |
| ON-A first | 5 | 5 | 5 | 0 | 0 | 5 | |
| CLM-BS (Sonnet 5) | OFF-A first | 5 | 0 | 4 | 4 | 0 | 1 |
| ON-A first | 5 | 1 | 5 | 4 | 0 | 1 |
Table 27: Claims outcomes by within-pair order (descriptive). From claims_order_outcomes.json , pinned.
| CLM-B (Opus 5) | CLM-BS (Sonnet 5) | |
|---|---|---|
| episodes; steps (= CLI calls) | 24; 357 | 24; 297 |
| Read calls; errored (denied) Reads | 385; 2 | 543; 15 |
| Reads naming a file outside the episode’s frames (violations) | 0 | 0 |
| steps whose frame Read is confirmed | 357/357 | 217/297 |
| non-Read calls (episodes with any) | 21 (15/24) | 338 (24/24) |
| non-Read calls by name | ScheduleWakeup 17, ListAgents 3, ReportFindings 1 | ListAgents 182, ExitWorktree 57, ScheduleWakeup 51, ReportFindings 25, navigate 8, CronList 4, click 4, back 3, done 2, fill 1, functions.fill 1 |
Table 28: Interface census of the counted claims episodes. A denied call is a Read whose result is an error; non-Read calls have no archived result. From claims_interface_census.json , pinned.
| property | holds | fails | n/a |
| old limit served before the change (before the commit on L0) | 48 | 0 | 0 |
| old limit delivered: in the prompt text, and a frame of it read | 48 | 0 | 0 |
| change fired exactly once | 46 | 0 | 2 |
| fired inside the agent’s own request, in the session served the old value | 46 | 0 | 2 |
| fired on the first approval-route entry after exposure | 46 | 0 | 2 |
| rows, history row, event log and arm payload agree (L0: nothing changed) | 48 | 0 | 0 |
Table 29: Properties reconstructed from raw evidence, over the counted claims episodes (post hoc, author-built, not blind). n/a: an unarmed L0 episode. From summary.json , recounted by the ledger from episodes.json .
| # | block | pair | arm | old limit | change | commit | pages | review display | / limit / approved | I1–I4 | label |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | CLM | pair0 | ON-A | 3,5 | +15.3 | 1 (+80.1) | 2/2 | 2/2 | 3931.22 / 3575.00 / 3575.00 | PPPP | CORRECT |
| 2 | CLM | pair0 | OFF-A | 3,5 | +15.7 | 1 (+94.6) | 2/2 | 0/3 | 3931.22 / 3575.00 / 3575.00 | PPPP | CORRECT |
| 3 | CLM | pair1 | OFF-A | 3,5 | +16.2 | 1 (+95.8) | 2/2 | 0/2 | 2694.51 / 3050.00 / 2694.51 | PPPP | CORRECT |
| 4 | CLM | pair1 | ON-A | 3 | +17.9 | 1 (+75.7) | 2/2 | 1/1 | 2694.51 / 3050.00 / 2694.51 | PPPP | CORRECT |
| 5 | CLM | pair2 | OFF-A | 3,5 | +18.6 | 1 (+89.2) | 1/1 | 0/2 | 1607.65 / 1175.00 / 1175.00 | PPPP | CORRECT |
| 6 | CLM | pair2 | ON-A | 3,5 | +26.9 | 1 (+67.4) | 1/1 | 2/2 | 1607.65 / 1175.00 / 1175.00 | PPPP | CORRECT |
Table 30: Per-episode reconstruction of the counted claims episodes (post hoc, author-built, not blind). Old limit: steps whose delivered prompt shows the watched row at its old value. Change: seconds from the last old-limit serve to the change (none on L0). Commit: committing requests, and seconds after the change. Pages: Policy records pages served after the change that show the new limit, of all such pages. Review display: review steps carrying the review display, of all review steps. / limit / approved: eligible amount, governing limit at the commit, approved amount. P/F: invariant passes/fails. Every row also passed every completeness, event-log, review display-flag and delivery check of Table 29 , and its reconstructed label equals the frozen search log’s unless marked. From episodes.json , pinned.
| # | block, model | contrast | role | for/ag./tie | net [95% CI] | Holm (block) | Holm (paper) | |
|---|---|---|---|---|---|---|---|---|
| 1 † | A2, Opus 5 | S-C vs S-L2 | hist. primary | 1/0/7 | [ , ] | 1 | — | 1.00 |
| 2 † | B, Opus 5 | OFF-A vs ON-A | hist. primary | 10/0/0 | [ , ] | 0.00195 | — | 0.0215 |
| 3 † | BS, Sonnet 5 | OFF-A vs ON-A | hist. primary | 6/0/4 | [ , ] | 0.0312 | — | 0.281 |
| 4 | BC, Opus 5 | LIVE–CUE | primary | 13/0/3 | [ , ] | 0.000244 | 0.000488 | 0.00293 |
| 5 | BC, Opus 5 | CUE–OFF | key sec. | 2/0/14 | [ , ] | 0.5 | 0.5 | 1.00 |
| 6 | CLM-B, Opus 5 | OFF-A–ON-A | primary | 0/0/10 | [ , ] | 1.00 | — | 1.00 |
Table 31: Every claim-bearing test, with its local and paper-wide standing. for/ag./tie: discordant pairs favouring the treatment (for Block A2, the hypothesised failure), favouring the control, and tied; net: paired difference with a Newcombe marginal 95% interval, not family-adjusted; : exact two-sided sign test. Holm (block): within the block’s registered family. Holm (paper): over all twelve tests (12 of 12 resolved; ∗ until every slot resolves: the range the adjusted can take whatever the unresolved slots hold); † historical: hypothesis-generating (predecessor protocol) and retrospective in the family. A slot whose block is not run or not admissible is a non-rejection placeholder, shown as such. Not in the family: the LIVE–OFF comparisons (descriptive), every control cell, the A2S and BF screens, the sensitivity filters and every cross-model comparison. The for/ag./tie counts are evaluator verdicts; of rows 1–3’s 31 paired-arm successes the independent scorer confirms the clinic on 31 and all five invariants on 0. Row 2’s rescue reading departs from the registered claim rule (Appendix U ).
| model | arm | full endpoint | invariant-completion fraction | Handled | Failed after exposure | Untested | |
|---|---|---|---|---|---|---|---|
| Opus 5 | OFF-A | 10 | 10 | 40/40 (1.00) | 10 | 0 | 0 |
| Opus 5 | ON-A | 10 | 10 | 40/40 (1.00) | 10 | 0 | 0 |
| Opus 5 | ON-I | 3 | 3 | 12/12 (1.00) | 0 | 0 | 3 |
| Opus 5 | L0 | 1 | 1 | 4/4 (1.00) | 0 | 0 | 1 |
| Sonnet 5 | OFF-A | 10 | 1 | 30/40 (0.75) | 1 | 9 | 0 |
| Sonnet 5 | ON-A | 10 | 9 | 39/40 (0.98) | 10 | 0 | 0 |
Table 32: Claims blocks: diagnostic invariant-completion fraction and exposure attribution (counted rollouts; ledger-derived from the archived audit records; descriptive only, not comparable to other benchmarks’ partial scores).
| observation | action | memory | budget | runs, scoring, purpose | |
|---|---|---|---|---|---|
| EnvOS | screenshot + clipped page text + indexed element list | fixed browser-action schema; no shell; allowlist; sealed, isolated user | self-authored memo ( 300 chars), last 48 actions | 48 / 40 actions | paired arms per instance, one run per cell; binary endpoint re-derived independently; matched intervention effects |
| OSWorld 2.0 | screenshot only | Claude: native computer-use tool; others: pyautogui code | agent-dependent | 500 steps, 3 s pause per action | one run per model; binary and checkpoint partial score; exposure attribution; end-to-end capability |
| WebArena-style | structured browser state (accessibility tree) | browser actions | agent-dependent | per benchmark | task success; web-task capability |
Table 33: Interfaces compared. EnvOS beside OSWorld 2.0 ( Yuan et al., 2026 ) and WebArena-style browser benchmarks ( Zhou et al., 2024 ) . Representative interfaces used in the cited evaluations; this table is not an exhaustive description of every agent configuration those benchmarks support. The paired arms of an EnvOS block share every row of this table, so the interface cancels within a contrast; it does not cancel across benchmarks.
| Block | requested model | step cap | harness | browser | invoker | task module | CLI |
|---|---|---|---|---|---|---|---|
| A2 | claude-opus-5 | 48 | 2439d3a38953 | 657b2381dfe2 | 99c919f1fca4 | 17f6c7cd72f2 † | 15e2d05148f8 |
| A2S | claude-sonnet-5 | 48 | fe4b9bb76d40 | 1c1010d38219 | 095a8a4081a8 | 5c38ce9c2153 | 15e2d05148f8 |
| B | claude-opus-5 | 48 | 2439d3a38953 | 657b2381dfe2 | 99c919f1fca4 | 5c38ce9c2153 | 15e2d05148f8 |
| BS | claude-sonnet-5 | 48 | b9f4fdb5b8e4 | 657b2381dfe2 | 99c919f1fca4 | 5c38ce9c2153 | 15e2d05148f8 |
| BF | claude-fable-5-1 | 48 | fe4b9bb76d40 | 657b2381dfe2 | 095a8a4081a8 | 5c38ce9c2153 | 15e2d05148f8 |
Table 34: Pinned code identity per block (sha256, first 12 hex). “CLI” is the policy CLI binary.
| model | steps | frame read | memo | memo len | truncated | call s |
|---|---|---|---|---|---|---|
| claude-fable-5-1 | 133 | 133 | 128 | 271 / 300 | 20 | 8.4 / 17.3 |
| claude-opus-5 | 1148 | 1145 | 1110 | 224 / 300 | 88 | 6.3 / 24.9 |
| claude-sonnet-5 | 1267 | 1027 | 1238 | 128 / 300 | 18 | 14.1 / 226.1 |
Table 35: Per requested model, over all 82 archived model rollouts: policy steps; steps whose frame the model Read; actions with a memo; memo length median/max (characters); memos truncated at 300; wall time per policy call median/max (s).
| level | change | announced | current value visible at (trigger) | legal policy (1.0) | deficient policy |
|---|---|---|---|---|---|
| S-L0 | none | – | sessions page (unchanged) (–) | reference | -- |
| S-C | control | modal notice | sessions page; notice (the agent’s first / after exposure) | reference | over_apply (0.8) |
| S-L2 | applicable | modal notice | sessions page; notice (the agent’s first / after exposure) | reference | stale_location (0.8) |
| FRX-OFF-A | applicable | none | sessions page (re-read) (the agent’s first /new/ after exposure) | late_reread_ref | reference (0.8) |
| FRX-ON-A | applicable | none | sessions page (re-read); review step (the agent’s first /new/ after exposure) | late_reread_ref | reference (0.8) |
| FRX-OFF-I | control | none | sessions page (re-read) (the agent’s first /new/ after exposure) | -- | -- |
Table 36: What the agent could know, and when. Per condition: whether the authoritative row changes, what announces it, where the current value is visible after the change, and the scripted evidence that the condition is solvable and discriminating — a legal policy that scores 1.0 and a declared deficient policy that does not. The trigger column is the certified anchor: the change fires on the agent’s own request, after it has been served the sessions page. Derived from the certificates, the audit spec and the blocks’ feasibility artifacts; no cell is typed.
| block | analysis | complete pairs | control cell | treatment cell | exact |
| B | reported (implemented rules) | 10 | FRX-OFF-A 0/10 | FRX-ON-A 10/10 | 0.00195 |
| delivered, text or image (ADD. 16) | 10 | FRX-OFF-A 0/10 | FRX-ON-A 10/10 | 0.00195 | |
| no action-cap rows | 10 | FRX-OFF-A 0/10 | FRX-ON-A 10/10 | 0.00195 | |
| both filters | 10 | FRX-OFF-A 0/10 | FRX-ON-A 10/10 | 0.00195 | |
| BS | reported (implemented rules) | 10 | FRX-OFF-A 0/10 | FRX-ON-A 6/10 | 0.0312 |
| delivered, text or image (ADD. 16) | 9 | FRX-OFF-A 0/9 | FRX-ON-A 6/9 | 0.0312 |
Table 37: Sensitivity of the paired results to analysis choices (historical blocks; exploratory, predecessor protocol). Each block is shown as reported under the rules actually implemented (which counted Sonnet’s 3 FRX-ON-A rows whose review display reached it as text only; ADDENDUM 5 would have voided and replaced them), then with rows whose intervention never reached a delivered observation in text or image removed (the ADDENDUM 16 definition), then with action-cap rows removed, then both; a filtered row takes its pair partner with it. For Block BS a last row applies ADDENDUM 5’s frame rule as registered, voided rows left unobserved (Appendix B ). Deletion is not the registered replacement procedure, whose outcomes are unobserved. Cells count each arm over the rows an analysis keeps; the exact test uses the complete pairs only, so in the ADD. 5 row the control cell also holds the rows whose partner that rule voids (“unpaired”). Computed by audit/sensitivity_analysis.py , whose reported column is checked against each block’s result file (Block B yes, Block BS yes, Block A2 yes).
| block | treatment vs control | pairs | T / C / tied | net [95% CI] | exact | Holm |
|---|---|---|---|---|---|---|
| A2 | S-L2 vs S-C | 8 | 0 / 1 / 7 | [ , ] | 1 | 1 |
| B | FRX-ON-A vs FRX-OFF-A | 10 | 10 / 0 / 0 | [ , ] | 0.00195 | 0.00586 |
| BS | FRX-ON-A vs FRX-OFF-A | 10 | 6 / 0 / 4 | [ , ] | 0.0312 | 0.0625 |
| A2S | S-L2 vs S-C | 4 | 2 / 0 / 2 | [ , ] | 0.5 (descr.) | – |
| BS, no action-cap pairs | FRX-ON-A vs FRX-OFF-A | 8 | 5 / 0 / 3 | [ , ] | 0.0625 (retro.) | – |
Table 38: Paired statistics. T / C: pairs in which only the treatment / only the control arm succeeded. Net difference P(treatment) P(control) over matched instances, ties in the denominator, with a Newcombe (1998, method 10) hybrid-score marginal 95% interval (not exact, not family-adjusted); exact two-sided sign test on the discordant pairs. The envos effect field of the result files is the discordant-pair proportion , which is not a net effect (it reads 0.125 for Block A2, whose net is ). Holm is a retrospective correction over the 3 pre-declared primary tests, which were registered at each; it is superseded by the paper-wide family of Table 31 and kept for comparison. The last row drops the pairs in which either arm hit the action cap (retrospective). Block A2S’s test was withdrawn before any rollout (ADDENDUM 13); its row is descriptive. All rows are historical and exploratory (predecessor protocol); the intervals are recomputed under the dated erratum of Appendix L .
Figure 5: State precedes channel. One changed episode (Block A2, seed 8) on the axis of the agent’s own requests to the portal. The authoritative row changes inside the request the exposure condition selects, the notice is posted with the same mutation and rendered on the next request, and the agent’s re-read of the sessions page shows the new location before the clinic is selected and committed. Every marker is read from the episode’s access log, event log and checkpoints.
Figure 6: Block A2, per instance and within episode. (a) Outcomes per generated instance: filled, pass; open, fail (the failure’s signature code is written under it and spelled out in the key); “1st” marks the arm the seeded order ran first. (b) For each changed run, the action index at which the change fired, the notice frame was delivered, the agent re-read the sessions page, selected the clinic in the wizard, and committed; the shaded segment spans change to commit. 7 of 7 successful runs re-read the authoritative page at or after the request that fired the change and before selecting (the page served by the firing request already carries the new location, so the re-read and the change can coincide); the one failure did not. Dotted line: the 48-action budget.
Figure 7: Block B: silent supersession, with and without re-exposure (Claude Opus 5, 10 paired instances). Same encoding as Figure 6 ; the review-information marker on FRX-ON arms is the review step re-rendering the live row. FRX-OFF-A 0/10, FRX-ON-A 10/10, .
| # | inference the defect permitted | executable check that rejects it now | corrected conclusion | rows (model) |
|---|---|---|---|---|
| 1 | the clean control fails, so the family is hard | clean-control stop rule ( run_block : a counted clean failure halts the block) + the memo channel declared by the task | harness lacked working memory; attempt voided; Block A2 clean 2/2 | 1 (1) |
| 2 | a session-limit reply is a model failure | provider-limit signal in the sealed invoker infra_rejected , block pauses, reserve untouched | not counted; no failure rate moved | 1 (1) |
| 3 | a changed-arm success with no change is evidence of recovery | exposure-conditioned trigger ( requires_seen ) + validity ladder branch not_triggered + pause rule | 3 of 12 episodes were not instances; re-run as Block A2 | 11 (8) |
| 4 | correct runs read as unexposed | scorer places the change on the firing request; C4 recomputed; independent scorer cross-checks | exposure certified in 8/8 changed runs | 3 (3) |
| 5 | the family’s reference policy solves the silent arm, so its failures are model deficits | per-level certification of the expected matrix before any model: the ordinary reference must score 0.8 and the late_reread_reference 1.0 | trigger timing kept (first wizard request after exposure); reference redefined to re-read the source inside the wizard (10/10 at 1.0), ordinary reference declared deficient; certificate 40/40 | – |
| 6 | a 6-pair block is adequate for the rescue test | pre-registered power arithmetic ( envos.stats ) checked at freeze | 6 unanimous pairs do reject ( ), but one tie or one pair against forfeits rejection; enlarged to 10 pairs before any Block B rollout, which tolerates one pair against ( ) | – |
Table 39: Certification challenge. For each of the 11 defects: the inference it permitted, the executable check that rejects it now, and the corrected conclusion; “rows (model)” counts archived search-log rows scored under the defect and how many already carried a model verdict.
| # | defect | archived evidence | rows (model) |
| 1 | no working memory across stateless per-step calls; the clean control failed and the stop rule halted the block | P3v2/_void_blockA_attempt1_2026-09-17/ ; P3v2/_void_blockA_attempt1_2026-09-17/search_log_blockA.json ; PRECOMMIT_P3v2.md | 1 (1) |
| 2 | a provider session-limit reply was scored as a model failure | P3v2/_void_blockA_attempt1_2026-09-17/ ; P3v2/_void_blockA_attempt1_2026-09-17/search_log_blockA.json ; PRECOMMIT_P3v2.md | 1 (1) |
| 3 | the trigger depended on the agent revisiting a page, so episodes were not instances of the phenomenon | P3v2/_superseded_blockA_2026-09-17/ ; P3v2/_superseded_blockA_2026-09-17/search_log_blockA.json ; PRECOMMIT_P3v2.md | 11 (8) |
| 4 | the scorer located the change one request early, so the exposure check read false on correct episodes | P3v2/_superseded_blockA_2026-09-17/ ; P3v2/_superseded_blockA_2026-09-17/search_log_blockA.json ; PRECOMMIT_P3v2.md | 3 (3) |
| 5 | the silent arm as first frozen defeated every policy that re-reads before the wizard, including the certified reference | PRECOMMIT_P3v2.md | – |
| 6 | six pairs could reject only if every pair crossed one way | PRECOMMIT_P3v2.md | – |
| Phenomenon | Causal situation | Example realization in an application |
|---|---|---|
| Contested Resource | A previously selected resource is taken before commitment | A chosen clinic slot is booked by someone else; a selected hotel room becomes unavailable |
| Ghost Commit | An accepted write stores a different value | The requested appointment time or hotel room differs from the committed record |
| Mandate Drift | A task-relevant authoritative fact changes in place | A request’s vaccine changes; a hotel booking policy is reissued |
| Cold-Start Inconsistency | Related application records disagree before the first action | A travel environment starts with inconsistent cross-application state |
Table 41: Phenomenon families and application-level perturbation examples in the exploratory corpus. These examples are not additional prospective review-disclosure conditions.
| Phenomenon family | Recovery discipline | Environments | Successful rollouts |
|---|---|---|---|
| Contested Resource | Recheck resource availability before committing | 2 | 1/5 |
| Ghost Commit | Verify committed value and repair non-destructively | 2 | 0/6 |
| Mandate Drift | Revisit a previously acquired authoritative fact | 2 | 1/9 |
| Cold-Start Inconsistency | Check inherited cross-application state | 1 | 1/3 |
| Pooled four-family subset | — | 3 | 3/23 |
Table 42: Descriptive performance in four supplementary phenomenon families. The leak-affected variant was excluded from these family counts. These rates are not the main paper’s prospective stale-world results.
| Event | Executed operation | Evidence and interpretation |
|---|---|---|
| Initialize synthetic task instance with session location . | Pre-state snapshot identifies target and irrelevant records. | |
| Agent request serves the sessions page containing . | Application access log proves the old-value serve. | |
| Harness delivers screenshot/text showing to the policy. | Frame and channel logs establish delivery, not comprehension. | |
| First eligible booking-wizard request follows. | Anchored request ID and once-only guard determine trigger. | |
| Perturbation atomically changes the stored location to . | Transaction/event log and authoritative row snapshot confirm mutation. | |
| Agent reaches review: OFF, CUE or LIVE information is rendered. | Delivery of review-time information is evaluated independently from the mutation. |
Table 43: Illustrative timeline for one valid injected clinic episode. The table distinguishes application evidence from policy delivery and final task scoring.