EnvOS: Auditing Computer-Use Agents When the World Moves On : Phenomenon-Based Task Construction, Certification, and Evidence-Based Evaluation

Ankush Kadu; Athira Gopal; Niharika Singh; Arpit Jain; Aswanth Krishnan

cs.AIOct 29, 2026

EnvOS: Auditing Computer-Use Agents When the World Moves On : Phenomenon-Based Task Construction, Certification, and Evidence-Based Evaluation

Authors: Ankush Kadu, Athira Gopal, Niharika Singh, Arpit Jain, Aswanth Krishnan

Organizations: QpiAI, Bangalore, India

Abstract

Computer-use agents built on large language models (LLMs) work in software that can change while they act: an appointment moves, a policy is revised, a record is updated. Testing agents under such changes is harder than it looks. When an agent fails, the cause may be the agent, but it may also be a change that did not fire as intended, an update the agent never received, or a grader that scored the episode wrongly. We introduce EnvOS, a framework for building and certifying environments that change mid-task. Before any agent runs, EnvOS checks that each change fires correctly, that the task remains solvable, and that the grader separates correct from incorrect behavior; after each run, separate code re-checks the verdict from archived evidence. We apply it to stale information, where a value the agent has already seen changes before the agent acts on it. In clinic booking, Claude Opus 5 succeeds on 0/16 tasks when nothing extra is shown, 2/16 when reminded to re-check the source, and 15/16 when the updated value is shown; the gain over the reminder holds after correcting for all tests (p = 0.00293). In insurance-claim approval, Opus 5 re-checks on its own and succeeds on 10/10, whereas Claude Sonnet 5 rises from 1/10 to 9/10 when the current policy is shown again, a gain significant in its pre-registered test but not after correcting for all tests. Whether an agent caught a change thus varied with the model and the interface. We also stress-test the grading itself: on corrupted episodes designed blind from the rules, the EnvOS pipeline accepted none of 26 as clean, and on an externally generated set it caught 25 of 39, against 4 for simple log checks. Because every EnvOS environment is generated from a seed and graded by an audited automatic verifier, the same environments are natural candidates for training agents with reinforcement learning from verifiable rewards.

Figures & tables

Figure 1: Authoritative state and model observations during a stale-information intervention. The application serves $v_{0}$ before a one-time exposure-conditioned mutation produces $v_{1}$ . The review surface may disclose nothing, a link to the source, or the current value; these are observation conditions, not changes to the underlying mutation. Observation delivery is logged separately, and the evaluator scores committed application state rather than the disclosure review display.

Figure 2: EnvOS evaluation lifecycle. Task and phenomenon specifications define a candidate executable intervention. Pre-run checks assess the intervention, observation channel, feasibility, behavioral discrimination, and scorer robustness, determining whether the candidate is admitted to evaluation. Admitted instances are executed under fixed rules; their archived trajectories support validity classification, state-based task scoring, independently implemented reconstruction, and evaluator challenges. Admission applies to the specified checks and does not guarantee the validity of every subsequent episode.

Assurance question	Candidate-instance checks	Run-level evidence and limits
World validity	Authorized mutation, bounded scope, one-shot execution, anchored timing	State and access logs establish when the authoritative event actually fired
Observation consistency	Rendering comes from updated source state; available channels are consistent with intended condition	Served pages and delivered model frames are archived separately; neither establishes comprehension
Recovery validity	Reference completion route and declared deficient-policy discrimination, including frozen-instance feasibility	Actual model actions, rereads and commits are observed, not inferred from the scripted policy
Checker robustness	Scorer localization checks, negative mutations and positive controls	Validity-first inclusion, independently coded verdict reconstruction and external archive challenges test residual mistakes

Table 1: One EnvOS evaluation pipeline. The four assurance questions organize pre-run certification; the six audit stages track the evidence before, during, and after a particular model episode. Passing one part does not imply that another part was checked.

	Clinic booking	Insurance claim approval
Phenomenon	A booked session’s previously served location is superseded	A policy’s previously served reimbursement limit is superseded
Authoritative object	Target clinic session row	Target insurance policy row
Prior serve	Sessions page displays previous location	Policy page displays previous limit
Perturbation trigger	First eligible booking-wizard request after serve	First eligible form-entry request after serve
Mutation	Update the relevant stored location	Update the relevant stored limit
Disclosure	None, source link, or current location	None or re-exposed current policy record

Table 2: The phenomenon–perturbation mapping in the two prospective workflows. Entries specify the experimental roles, not the literal values of any generated instance.

Task	Code	Information available at the review step
Clinic	OFF-A	No additional information at booking confirmation; the agent may revisit the original session record.
Clinic	CUE-A	A reminder identifies the session and provides a link for verification, without revealing the updated location.
Clinic	LIVE-A	The same reminder and link are accompanied by the updated session location.
Insurance	OFF-A	No additional policy information before approval; the original policy record remains accessible.
Insurance	ON-A	Current policy record is shown again, including the limit but not the approval amount.
Controls	-I, L0	Irrelevant-record change at the trigger, or no change, respectively.

Table 3: Experimental information conditions. In all principal comparisons, the application undergoes the same task-relevant state change and the original source remains accessible. Only the information supplied to the agent before its final decision differs.

Check family	What is tested	Example invalid case	What passing does not prove
World / trigger	old-value serve; anchored one-time update; condition-equivalent neutral operations	mutation fires before prior serve or fires twice	agent noticed the old value
Condition discrimination	reference and deficient scripted-policy trajectories and expected signatures	a value-only script passes when the value was hidden	arbitrary models will find the reference route
Instance feasibility	scripted completion and discrimination on each frozen generated instance	generated slot or decision is impossible	all future cases are feasible
Scoring instrumentation	event localization and registered target/commit invariants	scorer associates change with wrong request	every outcome field is reconstructable
Mutation / positive controls	deliberately corrupted and valid archives	damaged event enters clean; good archive is rejected	unknown corruption classes are covered

Table 4: Certification checks are admission evidence with specific failure targets. All checks are scoped to the tested scripted routes and generated instances.

Stage	Evidence	Fault exposed	Boundary
Certify	scripted policies, trigger logs, per-instance feasibility and mutation controls	update fires early/twice; reference path impossible; scorer misses event	scripted paths only
Isolate	sealed OS user, allowed origins, canary access probes, observation-delivery records	policy leaks evaluator data or misses promised channel	unarchived provider-side returns remain
Freeze	preregistration, SHA-256 manifests, fixed templates	post-outcome changes to analysis or software	dated amendments are still possible and disclosed
Count	validity table, provider/harness log, reserves and stop rules	provider outage treated as model failure; invalid rerun cherry-picked	validity classification can itself err
Re-derive	separately implemented outcome scorer and standard-library validity checker	incorrect success invariant or eligibility verdict	same team; shared archive; unresolved terms
Challenge	corrupted archive copies, sealed expectations, baselines and external generator	detector false accepts and valid-control false rejects	construction access, disputes and coverage vary

Table 5: Audit stages, evidence and scope. Every successful check establishes a limited property rather than a general guarantee of benchmark correctness.

insurance-claim approval
	arm successes			paired	for/ag./	local $p$	indep.
model (block)	OFF	ON/CUE	LIVE	contrast	tie	(block Holm)	endpoint
Opus 5 (CLM-B)	10/10	10/10	—	ON $-$ OFF	0/0/10	1.00	24/24
Sonnet 5 (CLM-BS)	1/10	9/10	—	ON $-$ OFF	8/0/2	0.00781	24/24
Gemini (CLM-BG)	0/10	5/10	—	ON $-$ OFF	5/0/5	0.0625	24/24
clinic booking

Table 6: Prospective model results. Success requires the full evaluator endpoint; in clinic rows its calendar-compatibility term rests on a stated design condition, not independent re-derivation. Paired counts: treatment-only/control-only/ties;

p

: exact two-sided sign test (BC: pre-registered block-Holm value in parentheses). ‘indep. endpoint’: counted rollouts whose independent reconstruction (clinic: without calendar compatibility) agrees with the evaluator. Blocks without results print their state; nets, intervals: Table 31 . Gemini 3.1 Pro Preview received each frame inline in a stateless REST request with no tools; the Claude models were instructed to read each frame file with their file-read tool. Provider, model, invoker and image-delivery channel differ together; rows describe model–invoker configurations, not vendor effects.

Figure 3: Matched experimental outcomes by instance. Insurance-claim approval compares no additional policy information (OFF-A) against re-exposure to the current policy (ON-A); clinic booking compares no additional information (OFF-A), a verification reminder and source link (CUE-A), and direct disclosure of the updated location (LIVE-A). Each panel corresponds to a distinct registered model–workflow configuration; outcomes are not pooled across models.

checker	controls	corruptions (in scope)	would enter the results unflagged
clinic: the sealed challenge’s 10 controls and 18 corruptions, its recorded outputs re-tabulated
protocol, implemented reading	10/10	15/18 (11/12)	notice delivery ( C01 ); delivery of review-time information ( C02 ); commit before change ( C08 )
protocol, registered reading	10/10	16/18 (11/12)	commit before change ( C08 ); truncated trajectory ( C15 )
simple baseline (143 lines)	9/10	11/18 (11/12)	7, of which 6 outside its declared scope
claims: 10 controls and 19 corruptions we built from counted archives
operational pipeline	10/10	16/19 (10/12)	duplicate commit in the access log only ( C09a ); truncated trajectory ( C15 ); stale pages after the change ( C17p )

Table 7: The protocol against a simple raw-log baseline (post hoc, author-built, not blind; separate from Table 12 ). Same raw evidence and scored properties. Controls accepted; corruptions rejected (not counted, or counted with the corrupted invariant flagged), in brackets those in the baseline’s declared scope. Clinic protocol rows are derived from the sealed tools’ recorded outputs; the claims pipeline voids C09b as invalid rather than failing its single-commit term. Every case: Appendix D .

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

rule	blocks	registered	implemented	retrospective correction	archived values re-derived
Trigger anchor	A (superseded)	PRECOMMIT:19	anchor ’second authenticated GET /’ (code replaced; archived in superseded log rows)	ADD 3 (PRECOMMIT:203): exposure-conditioned anchor for new blocks; scorer after_login fix (PRECOMMIT:213)	none persisted: archived C4 of superseded S-L2/S-C s0-s2 still reads False; 3 untriggered episodes stay uncounted; block superseded
Trigger anchor	A2, A2S	PRECOMMIT:206	superseded_session.py:442	none	none
Trigger anchor	B, BS, BF	PRECOMMIT:274	superseded_session.py:449	ADD 5 kept the late fire and redefined the reference policy (PRECOMMIT:280) – before any rollout	none (prospective)
Validity before outcome (ladder order)	all	PRECOMMIT:41	telemetry.py:32 (doc) vs telemetry.py:232 precedes telemetry.py:239	ADD 11 discloses success-first order (PRECOMMIT:454); ADD 16 repeats it (PRECOMMIT:653)	no verdict re-derived; this script lists SUCCESS rows that validity-first would void
Silent-arm perception branch	B (row 1); BS, BF prospective	PRECOMMIT:42-54 (the §2 table has no silent-arm row)	telemetry.py:253-254	ADD 10 (PRECOMMIT:401)	changed_FRX-OFF-A_test_id_s20_claude-opus-5: delivery_ambiguous $\rightarrow$ model_failure; pre-fix classifier would have voided 25 silent-arm model failures
Notice delivery (C3)	A2, A2S (announced arms)	PRECOMMIT:50	preimage[2026-09-22T0004]/provenance.py:632 (pre-image; frame + ack on classified route) $\rightarrow$ provenance.py:663; verdict telemetry.py:266	ADD 11 (ack on unclassified route, PRECOMMIT:438); ADD 16 (text or image, PRECOMMIT:619)	ADD 11: 5 A2 rows re-derived (changed_S-L2_s9 opus-5, changed_S-L2_s11 opus-5, cosmetic_S-C_s13 opus-5, cosmetic_S-C_s14 opus-5, cosmetic_S-C_s15 opus-5): C3 false $\rightarrow$ true on 4, 4 signature changes; ADD 16: C3 false $\rightarrow$ true on 2 rows (cosmetic_S-C_s12 opus-5, cosmetic_S-C_s15 opus-5); 0 verdict changes

Appendix

Table 11

rule	common	claims	clinic	historical
trigger	fires on the first eligible request after the old value was served (exposure-conditioned)	first form entry after the policy records page	first booking-wizard request after the sessions page	exposure-conditioned from Block A2 on (ADDENDUM 3)
untriggered episode	registered per workflow	counted as a failure, reported separately	not counted; replaced	not counted; replaced
counting	validity first; only uncounted validity failures replaced, under a reserve (2 per cell, 6 per block)	the claims validity-first table; the generic classifier verdict is never read	the ADDENDUM 18 validity table	success first as implemented; the validity-first re-analysis is Table 10
frame delivery	admitted per channel	file reads	file reads (an inline channel is defined for GPT-6 blocks)	read rule as implemented, one reading post hoc (above); ADDENDUM 5’s review-screenshot rule not enforced
endpoint	invariants on world state and the agent’s report	four (no step after the commit)	five	five, evaluator-scored; the nearest-slot term cannot be independently verified
independent audits	outcome scorer and standard-library validity re-derivation, claimed only once the artifact exists	both present	claimed when produced	archive re-analyses (Appendix C.1 )

Appendix

Table 9: The protocol as operated. Common: every prospective block. Claims: CLM-B and CLM-BS (PRECOMMIT_CLAIMS). Clinic: BC and BCS (ADDENDA 18–19). Historical: Blocks A2, A2S, B, BS and BF (predecessor versions; Appendix T ).

block	analysis	control	treatment	ON-I / S-L0	for/against/tied; $p$	unobserved rows
B	reported (implemented rules)	0/10	10/10	3/3	10/0/0; 0.00195	none
	ADD. 5 screenshot rule as registered	0/10	10/10	3/3	10/0/0; 0.00195	none
	ADD. 5, literal (also voids review information never rendered)	0/10	10/10	3/3	10/0/0; 0.00195	none
BS	reported (implemented rules)	0/10	6/10	2/3	6/0/4; 0.0312	none
	ADD. 5 screenshot rule as registered	0/10	6/7	0/1	6/0/1; 0.0312	FRX-ON-A s22, FRX-ON-A s27, FRX-ON-A s28, FRX-ON-I s21, FRX-ON-I s22
	ADD. 5, literal (also voids review information never rendered)	0/10	6/6	0/0	6/0/0; 0.0312	FRX-ON-A s22, FRX-ON-A s23, FRX-ON-A s27, FRX-ON-A s28, FRX-ON-I s20, FRX-ON-I s21, FRX-ON-I s22

Appendix

Table 10: Reported results and results under the original rules. “Validity first” applies the delivery rule before success, with the notice checkpoint as first archived, after ADDENDUM 11, and after ADDENDUM 16. for/against/tied are paired discordances; unobserved rows are those the rule voids.

held-out case	kind	expected to fail	observed to fail	met
valid_reference_passes	noop	none	none	yes
stale_location_via_history_actor	sql	I1	I1, I5	no
second_active_row_same_label	sql	I2	I2	yes
collateral_slot_shift_on_bystander	sql	I3	I3	yes
cancelled_then_reconfirmed_row	sql	I1, I2, I4, I5	I1, I2, I4, I5	yes
report_case_and_whitespace_variant_is_still_correct	report	none	none	yes

Appendix

Table 11: The held-out mutation challenge, run once. Expected: the invariants the frozen expectation says must fail (I1 target state correct, I2 exactly one commit, I3 no collateral change, I4 workflow continued from the commit, I5 final report correct); “none” marks a valid case the checker must accept.

layer	sealed challenge	what it establishes	known gap
independent outcome scorer	28/28	every frozen expectation met on this suite	a term it cannot resolve is reported unresolved; clinic calendar compatibility is assumed; no historical success is verified in full
validity re-derivation (standard library)	24/28	accepted 10/10 valid controls and rejected 14/18 corruptions on this suite	blind spots: missing observation delivery (2); commit before change (1); truncated trajectory (1)
evaluator, archived verdict path	10/28	met 10/10 valid controls but rejected 0/18 corruptions because archived validity/invariant claims were retained	re-derives the delivery checkpoints from raw logs but keeps the archived run-validity and invariant claims

Appendix

Table 12: Auditing layers against the sealed archive challenge (first and only run; expectations sealed by digest before it). Counts are frozen expectations met; the per-case table is Table 13 .

case class	cases	outcome scorer	validity re-derivation	evaluator (replay)
valid controls	10	10 of 10	10 of 10	10 of 10
C01 success-before-validity: notice	1	met	no : validity	no : validity
C02 missing review-time information delivery	1	met	no : validity	no : validity
C03 change before old value served	1	met	met	no : validity
C04 change fired twice	1	met	met	no : validity
C05 change in nonagent request	1	met	met	no : validity

Appendix

Table 13: The sealed archive challenge, first and only run, per tool and case class. “met”: every field the frozen expectation assesses for that tool was met; “no” names the fields that were not (for the validity re-derivation, validity is its verdict under the rules as implemented and validity (registered) under the rules as registered). The outcome scorer assesses I1–I5 only; the validity re-derivation assesses validity only; the evaluator’s invariants are its archived claims. From CHALLENGE_RESULT.json , pinned by sha256 in the ledger.

				verdict established
block	model	counted	validity (all att.)	successes	failures	contra.	unresolved or assumed
historical blocks (exploratory; predecessor protocol versions; archive-derived evidence)
A2	Opus 5	18	19/19	0/17	1/1	0	nearest slot (I1) on every success
A2S	Sonnet 5	9	9/9	0/7	2/2	0	nearest slot (I1) on every success
B	Opus 5	23	23/23	0/13	10/10	0	nearest slot (I1) on every success
BS	Sonnet 5	23	26/26	0/8	15/15	0	nearest slot (I1) on every success

Appendix

Table 14: What the independent audits established, per block. Counted rollouts; validity verdicts re-derived by the standard-library implementation, agreeing / all attempts (exclusions included; not run where the block’s re-derivation artifact does not exist); verdicts established by the independent scorer (for a failure, one term resolved false suffices; for a success, every term must resolve true); invariant contradictions; and what is unresolved or rests on a design assumption rather than on archived evidence. Prospective cells count only audits that have run.

		counted successes				counted failures
block	attempts	$n$	clinic term	slot evidence	all five	$n$	all five
A2	19	17	17	0	0	1	1
A2S	9	7	7	0	0	2	2
B	23	13	13	0	0	10	10
of which FRX-ON-A (Opus’s rescues)		10	10	0	0
BS	26	8	8	0	0	15	15

Appendix

Table 15: What is independently verified, by outcome. Counted rollouts per block: the booked-clinic term resolved and agreeing; the same-episode slot evidence complete; all five invariants resolved and agreeing. From SLOT_EVIDENCE.json and the r19 model audit, pinned by sha256 in the ledger.

block	att.	counted, archive	counted, re-derived	agree	counted, registered	registered rule differs on
A2	19	18	18	19	18	–
A2S	9	9	9	9	9	–
B	23	23	23	23	23	–
BS	26	23	23	26	16	FRX-ON-A s22, FRX-ON-A s23, FRX-ON-A s27, FRX-ON-A s28, FRX-ON-I s20, FRX-ON-I s21, FRX-ON-I s22
BF	5	5	5	5	5	–
all	82	78	78	82	71

Appendix

Table 16: Validity verdicts, re-derived independently. Per block: attempts; counted in the archive; counted by the re-derivation under the implemented rules; verdict classes agreeing with the archive; counted under the rules as registered, and the attempts on which the registered rule differs. From VALIDITY_INDEPENDENT.json , pinned by sha256 in the ledger.

episodes	$n$	I1	I2	I3	I4	I5
Block A2, counted	18	1/0/17	18/0/0	18/0/0	18/0/0	18/0/0
Block A2S, counted	9	2/0/7	9/0/0	9/0/0	9/0/0	9/0/0
Block B, counted	23	10/0/13	23/0/0	23/0/0	23/0/0	23/0/0
Block BS, counted	23	15/0/8	23/0/0	23/0/0	23/0/0	23/0/0
Block BF, counted	5	4/0/1	5/0/0	5/0/0	5/0/0	5/0/0
all counted model episodes	78	32/0/46	78/0/0	78/0/0	78/0/0	78/0/0

Appendix

Table 17: Independent-scorer coverage. Resolved-and-agreeing / contradiction / unresolved, archived versus recomputed invariant (I1–I5 as in Table 11 ), over counted model episodes per block, and over the certification episodes as re-scored now and as audited when cited.

case	class tested	expected	C8	scorer	eval./ classifier	baseline	pipeline
V01	clean control (unarmed L0)	counted	met	met	met	met	met
V02	inapplicable change control (ON-I)	counted	met	met	met	met	met
V03	OFF-A fired, correct approval	counted	met	met	met	met	met
V04	ON-A information-delivery control (Opus)	counted	met	met	met	met	met
V05	ON-A information-delivery control (Sonnet)	counted	met	met	met	met	met
V06	valid behavioural failure (stale approval)	counted; I1 fail	met	met	met	met	met

Appendix

Table 18: The post-hoc claims-path challenge, per case (author-constructed, not blind; one run). Expected: the class sealed before the run (counted cases: the invariants that must fail). Controls: “met” or no . Corruptions: caught , rejected or failed as sealed; wrong class , kept from a clean score but not with the sealed class; missed , scored as a clean result; scope , counted as expected but the tool does not assess the corrupted term. Pipeline: the classifier followed by the registered void gate. Separate from the sealed challenge (Table 13 ). From VALIDATION_RESULT.json , pinned by sha256 in the ledger.

checker	validity: controls	validity: corruptions	outcome: controls	outcome: corruptions	end to end: controls	end to end: corruptions	FA / FR
clinic: the sealed challenge’s cases (10 controls, 18 corruptions)
validity re-derivation, implemented	10/10	9/12	n/a	n/a	n/a	n/a	n/a
validity re-derivation, registered	10/10	10/12	n/a	n/a	n/a	n/a	n/a
outcome scorer	n/a	n/a	10/10	6/6	n/a	n/a	n/a
evaluator (archived verdicts)	10/10	0/12	10/10	0/6	10/10	0/18	18 / 0
protocol, implemented (derived)	10/10	9/12	10/10	6/6	10/10	15/18	3 / 0

Appendix

Table 19: Matched comparison, every decision (post hoc, author-built, not blind). Validity: controls accepted, validity-property corruptions rejected; outcome: controls scored correctly, outcome-property corruptions with a target invariant flagged; end to end: an episode is rejected if it is not counted, or is counted with a target invariant flagged. n/a: the checker has no such output; FA/FR: false acceptances/rejections. The clinic rows other than the baseline re-tabulate the sealed challenge’s recorded first-run outputs (not re-run; the sealed result, Table 13 , is unchanged); the protocol rows are derived from them, not a tool that was run. Claims rows: the post-hoc run of Table 18 . From matched_table.json , pinned by sha256.

case	operational pipeline (frozen)	C8	scorer	evaluator/ classifier	r28 baseline	repaired checker	reconstruction
V01 clean control (unarmed L0, Sonnet)	met	met	met	met	met	met	met
V02 inapplicable change control (ON-I X_down, Opus)	met	met	met	met	met	met	met
V03 inapplicable change control (ON-I X_up, Sonnet)	met	met	met	met	met	met	met
V04 OFF-A fired, correct approval (Opus, X_down)	met	met	met	met	met	met	met
V05 OFF-A fired, correct approval (Sonnet, B_down)	met	met	met	met	met	met	met
V06 valid behavioural failure (stale approval, Sonnet)	met	met	met	met	met	met	met

Appendix

Table 20: Fresh sealed cases (post hoc, author-built, not blind; one run). Expectations and every tool digest sealed ( SEAL.txt , 10:03:30Z) before any tool ran on any case; run at 10:03:34Z. No case reuses a base archive of Table 18 . Cells as in Table 18 . The pipeline is the frozen operational path, unchanged; the repaired checker is the r28 baseline plus step-list completeness, post-change page consistency (policy pages and review display) and an explicit single commit; the reconstruction contains both kinds of check. From FRESH_RESULT.json , pinned.

tool	controls	FR	caught	FA	scope	errors	exact
Claims
combined pipeline (classifier, void gate)	7/7	0	18/20	0	0	2	24/27
evaluator with classifier	7/7	0	18/20	0	0	2	24/27
claims validity re-derivation (C8)	7/7	0	6/20	9	4	1	16/27
independent scorer	7/7	0	10/20	9	0	1	12/27
simple raw-log baseline (r28)	7/7	0	3/20	14	2	1	12/27

Appendix

Table 21: Rule-only case-design challenge, per tool (cases and expectations specified by GPT-6 from the registered rules and archive schema only; implemented by the authors, fidelity-reviewed by a model, run once). Controls accepted; FR: controls rejected; corruptions rejected, or counted with the expected failing invariant flagged; FA: corruptions counted as clean; scope: counted, with the attacked invariant outside the tool’s outputs; errors: no verdict; exact: countedness and class (or invariants) equal GPT-6’s expectation. Exact is stricter than clean-versus-corrupt detection: a corruption rejected under a different registered class is non-clean but not exact.

case	class	expected	Pipe.	Eval.	C8	Scorer	Base.	Rep.
KC01	missing trajectory	harness_error	ERR	ERR	ERR	ERR	ERR	ERR
KC02	provider limit stop	infra_rejected	R	R	R	FA	FA	FA
KC03	recorded infrastructure error	infra_failure	R	R	R	FA	FA	FA
KC04	harness stop	harness_error	R	R	R*	FA	FA	FA
KC05	unauthorized file read	harness_error	R	R	R	FA	FA	FA
KC06	wrong resolved model	harness_error	R	R	FA	FA	FA	FA

Appendix

Table 22: Rule-only case-design challenge, every admitted case (class names are GPT-6’s). A: accepted; R: rejected; C: counted with the expected failing invariant flagged; FA: counted as clean; ERR: no verdict; scope: the attacked invariant is outside the tool’s outputs; R*: rejected with another class (named under each part). Bold : GPT-6’s exact expectation not met. Tools: Pipe. the combined pipeline, Eval. the evaluator with classifier, C8 and Valid. the claims and Block BC validity re-derivations, Scorer the independent scorer, Base. the simple raw-log baseline, Rep. the repaired checker. Controls are pooled into one row per workflow (

\times k

k

alike).

tool	controls	FR	caught	FA	scope	errors	exact
Claims
combined pipeline (classifier, void gate)	6/8	2	14/20	4	0	2	14/28
evaluator with classifier	8/8	0	14/20	4	0	2	15/28
claims validity re-derivation (C8)	8/8	0	3/20	12	4	1	15/28
independent scorer	6/8	2	8/20	11	0	1	5/28
simple raw-log baseline (r28)	8/8	0	1/20	15	2	2	6/28

Appendix

Table 23: Externally designed and instantiated challenge, per tool (cases, expectations and generator by a Gemini CLI agent from the rules, schema and archive copies only; generator run unread, cases sealed, each tool run once; the agent’s expectations, disputed ones included). Columns as in Table 21 : controls accepted; FR: controls not counted; corruptions caught (rejected, or counted with the expected failing invariant flagged); FA: corruptions counted as clean; scope: the attacked invariant is outside the tool’s outputs; errors: no verdict; exact: the agent’s full expectation met.

	CLM-B (Opus 5)	CLM-BS (Sonnet 5)	CLM-BG (Gemini 3.1 Pro Preview)
OFF-A success	10/10	1/10	0/10
ON-A success	10/10	9/10	5/10
rescues/harms/ties	0/0/10	8/0/2	5/0/5
exact paired $p$	1.00	0.00781	0.0625
reading	R2	R1	R3
ON-I; L0	3/3; 1/1	3/3; 1/1	3/3; 1/1

Appendix

Table 24: The three claims configurations side by side (descriptive; no pooling, no between-model test, no interval on a between-model difference). Gemini 3.1 Pro Preview received each frame inline in a stateless REST request with no tools; the Claude models were instructed to read each frame file with their file-read tool. Provider, model, invoker and image-delivery channel differ together; rows describe model–invoker configurations, not vendor effects.

pair	case	OFF-A	ON-A	ON-A stop
1	B_up	stale approval	no commit	step cap
2	X_up	stale approval	no commit	step cap
3	B_down	stale approval	pass	model done
4	X_up	stale approval	no commit	step cap
5	X_down	stale approval	pass	model done
6	X_down	stale approval	pass (attempt a1)	model done

Appendix

Table 25: CLM-BG per pair (top) and per arm (bottom). Evaluator labels of the counted rollouts; “stop” is the ON-A rollout’s stop reason; Wilson intervals with

z=1.96

; “re-read”: policy record requested after the change; “back”: returned from the review step. Registered diagnostics, descriptive only.

Figure 4: Block CLM-BG in Figure 3 ’s encoding. Counted rollouts per paired instance, OFF-A and ON-A; row totals at the right. Gemini 3.1 Pro Preview received each frame inline in a stateless REST request with no tools; the Claude models were instructed to read each frame file with their file-read tool. Provider, model, invoker and image-delivery channel differ together; rows describe model–invoker configurations, not vendor effects.

seed	case	first	claimed	deduct.	$E$	old limit	new limit	correct	stale	copy-limit OFF-A	copy-limit ON-A
0	B_up	ON-A	4006.22	75.00	3931.22	1400.00	3575.00	3575.00	1400.00	1400.00 fail	3575.00 pass
1	X_up	OFF-A	2769.51	75.00	2694.51	1550.00	3050.00	2694.51	1550.00	1550.00 fail	3050.00 fail
2	B_down	OFF-A	1732.65	125.00	1607.65	1325.00	1175.00	1175.00	1325.00	1325.00 fail	1175.00 pass
3	X_up	ON-A	1763.86	175.00	1588.86	725.00	2150.00	1588.86	725.00	725.00 fail	2150.00 fail
4	X_down	OFF-A	4206.18	350.00	3856.18	6725.00	1700.00	1700.00	3856.18	6725.00 fail	1700.00 pass
5	X_down	ON-A	2590.59	350.00	2240.59	3575.00	1400.00	1400.00	2240.59	3575.00 fail	1400.00 pass

Appendix

Table 26: The claims instances (seeds of the held-out test split; first: the arm run first in the pair). From claims_instances.json , pinned by sha256.

block	order	pairs	OFF-A pass	ON-A pass	for ON-A	ties
CLM-B (Opus 5)	OFF-A first	5	5	5	0	5
	ON-A first	5	5	5	0	5
CLM-BS (Sonnet 5)	OFF-A first	5	0	4	4	1
	ON-A first	5	1	5	4	1

Appendix

Table 27: Claims outcomes by within-pair order (descriptive). From claims_order_outcomes.json , pinned.

	CLM-B (Opus 5)	CLM-BS (Sonnet 5)
episodes; steps (= CLI calls)	24; 357	24; 297
Read calls; errored (denied) Reads	385; 2	543; 15
Reads naming a file outside the episode’s frames (violations)	0	0
steps whose frame Read is confirmed	357/357	217/297
non-Read calls (episodes with any)	21 (15/24)	338 (24/24)
non-Read calls by name	ScheduleWakeup 17, ListAgents 3, ReportFindings 1	ListAgents 182, ExitWorktree 57, ScheduleWakeup 51, ReportFindings 25, navigate 8, CronList 4, click 4, back 3, done 2, fill 1, functions.fill 1

Appendix

Table 28: Interface census of the counted claims episodes. A denied call is a Read whose result is an error; non-Read calls have no archived result. From claims_interface_census.json , pinned.

property	holds	fails	n/a
old limit served before the change (before the commit on L0)	48	0	0
old limit delivered: in the prompt text, and a frame of it read	48	0	0
change fired exactly once	46	0	2
fired inside the agent’s own request, in the session served the old value	46	0	2
fired on the first approval-route entry after exposure	46	0	2
rows, history row, event log and arm payload agree (L0: nothing changed)	48	0	0

Appendix

Table 29: Properties reconstructed from raw evidence, over the counted claims episodes (post hoc, author-built, not blind). n/a: an unarmed L0 episode. From summary.json , recounted by the ledger from episodes.json .

#	block	pair	arm	old limit	change	commit	pages	review display	$E$ / limit / approved	I1–I4	label
1	CLM	pair0	ON-A	3,5	+15.3	1 (+80.1)	2/2	2/2	3931.22 / 3575.00 / 3575.00	PPPP	CORRECT
2	CLM	pair0	OFF-A	3,5	+15.7	1 (+94.6)	2/2	0/3	3931.22 / 3575.00 / 3575.00	PPPP	CORRECT
3	CLM	pair1	OFF-A	3,5	+16.2	1 (+95.8)	2/2	0/2	2694.51 / 3050.00 / 2694.51	PPPP	CORRECT
4	CLM	pair1	ON-A	3	+17.9	1 (+75.7)	2/2	1/1	2694.51 / 3050.00 / 2694.51	PPPP	CORRECT
5	CLM	pair2	OFF-A	3,5	+18.6	1 (+89.2)	1/1	0/2	1607.65 / 1175.00 / 1175.00	PPPP	CORRECT
6	CLM	pair2	ON-A	3,5	+26.9	1 (+67.4)	1/1	2/2	1607.65 / 1175.00 / 1175.00	PPPP	CORRECT

Appendix

Table 30: Per-episode reconstruction of the counted claims episodes (post hoc, author-built, not blind). Old limit: steps whose delivered prompt shows the watched row at its old value. Change: seconds from the last old-limit serve to the change (none on L0). Commit: committing requests, and seconds after the change. Pages: Policy records pages served after the change that show the new limit, of all such pages. Review display: review steps carrying the review display, of all review steps.

E

/ limit / approved: eligible amount, governing limit at the commit, approved amount. P/F: invariant passes/fails. Every row also passed every completeness, event-log, review display-flag and delivery check of Table 29 , and its reconstructed label equals the frozen search log’s unless marked. From episodes.json , pinned.

#	block, model	contrast	role	for/ag./tie	net [95% CI]	$p$	Holm (block)	Holm (paper)
1 †	A2, Opus 5	S-C vs S-L2	hist. primary	1/0/7	$-0.13$ [ $-0.47$ , $0.22$ ]	1	—	1.00
2 †	B, Opus 5	OFF-A vs ON-A	hist. primary	10/0/0	$+1.00$ [ $0.61$ , $1.00$ ]	0.00195	—	0.0215
3 †	BS, Sonnet 5	OFF-A vs ON-A	hist. primary	6/0/4	$+0.60$ [ $0.20$ , $0.83$ ]	0.0312	—	0.281
4	BC, Opus 5	LIVE–CUE	primary	13/0/3	$+0.81$ [ $0.49$ , $0.92$ ]	0.000244	0.000488	0.00293
5	BC, Opus 5	CUE–OFF	key sec.	2/0/14	$+0.13$ [ $-0.09$ , $0.36$ ]	0.5	0.5	1.00
6	CLM-B, Opus 5	OFF-A–ON-A	primary	0/0/10	$0.00$ [ $-0.28$ , $0.28$ ]	1.00	—	1.00

Appendix

Table 31: Every claim-bearing test, with its local and paper-wide standing. for/ag./tie: discordant pairs favouring the treatment (for Block A2, the hypothesised failure), favouring the control, and tied; net: paired difference with a Newcombe marginal 95% interval, not family-adjusted;

p

: exact two-sided sign test. Holm (block): within the block’s registered family. Holm (paper): over all twelve tests (12 of 12 resolved; ∗ until every slot resolves: the range the adjusted

p

can take whatever the unresolved slots hold); † historical: hypothesis-generating (predecessor protocol) and retrospective in the family. A slot whose block is not run or not admissible is a non-rejection placeholder, shown as such. Not in the family: the LIVE–OFF comparisons (descriptive), every control cell, the A2S and BF screens, the sensitivity filters and every cross-model comparison. The for/ag./tie counts are evaluator verdicts; of rows 1–3’s 31 paired-arm successes the independent scorer confirms the clinic on 31 and all five invariants on 0. Row 2’s rescue reading departs from the registered claim rule (Appendix U ).

model	arm	$n$	full endpoint	invariant-completion fraction	Handled	Failed after exposure	Untested
Opus 5	OFF-A	10	10	40/40 (1.00)	10	0	0
Opus 5	ON-A	10	10	40/40 (1.00)	10	0	0
Opus 5	ON-I	3	3	12/12 (1.00)	0	0	3
Opus 5	L0	1	1	4/4 (1.00)	0	0	1
Sonnet 5	OFF-A	10	1	30/40 (0.75)	1	9	0
Sonnet 5	ON-A	10	9	39/40 (0.98)	10	0	0

Appendix

Table 32: Claims blocks: diagnostic invariant-completion fraction and exposure attribution (counted rollouts; ledger-derived from the archived audit records; descriptive only, not comparable to other benchmarks’ partial scores).

	observation	action	memory	budget	runs, scoring, purpose
EnvOS	screenshot + clipped page text + indexed element list	fixed browser-action schema; no shell; allowlist; sealed, isolated user	self-authored memo ( $\leq$ 300 chars), last 48 actions	48 / 40 actions	paired arms per instance, one run per cell; binary endpoint re-derived independently; matched intervention effects
OSWorld 2.0	screenshot only	Claude: native computer-use tool; others: pyautogui code	agent-dependent	500 steps, 3 s pause per action	one run per model; binary and checkpoint partial score; exposure attribution; end-to-end capability
WebArena-style	structured browser state (accessibility tree)	browser actions	agent-dependent	per benchmark	task success; web-task capability

Appendix

Table 33: Interfaces compared. EnvOS beside OSWorld 2.0 ( Yuan et al., 2026 ) and WebArena-style browser benchmarks ( Zhou et al., 2024 ) . Representative interfaces used in the cited evaluations; this table is not an exhaustive description of every agent configuration those benchmarks support. The paired arms of an EnvOS block share every row of this table, so the interface cancels within a contrast; it does not cancel across benchmarks.

Block	requested model	step cap	harness	browser	invoker	task module	CLI
A2	claude-opus-5	48	2439d3a38953	657b2381dfe2	99c919f1fca4	17f6c7cd72f2 †	15e2d05148f8
A2S	claude-sonnet-5	48	fe4b9bb76d40	1c1010d38219	095a8a4081a8	5c38ce9c2153	15e2d05148f8
B	claude-opus-5	48	2439d3a38953	657b2381dfe2	99c919f1fca4	5c38ce9c2153	15e2d05148f8
BS	claude-sonnet-5	48	b9f4fdb5b8e4	657b2381dfe2	99c919f1fca4	5c38ce9c2153	15e2d05148f8
BF	claude-fable-5-1	48	fe4b9bb76d40	657b2381dfe2	095a8a4081a8	5c38ce9c2153	15e2d05148f8

Appendix

Table 34: Pinned code identity per block (sha256, first 12 hex). “CLI” is the policy CLI binary.

model	steps	frame read	memo	memo len	truncated	call s
claude-fable-5-1	133	133	128	271 / 300	20	8.4 / 17.3
claude-opus-5	1148	1145	1110	224 / 300	88	6.3 / 24.9
claude-sonnet-5	1267	1027	1238	128 / 300	18	14.1 / 226.1

Appendix

Table 35: Per requested model, over all 82 archived model rollouts: policy steps; steps whose frame the model Read; actions with a memo; memo length median/max (characters); memos truncated at 300; wall time per policy call median/max (s).

level	change	announced	current value visible at (trigger)	legal policy (1.0)	deficient policy
S-L0	none	–	sessions page (unchanged) (–)	reference	--
S-C	control	modal notice	sessions page; notice (the agent’s first / after exposure)	reference	over_apply (0.8)
S-L2	applicable	modal notice	sessions page; notice (the agent’s first / after exposure)	reference	stale_location (0.8)
FRX-OFF-A	applicable	none	sessions page (re-read) (the agent’s first /new/ after exposure)	late_reread_ref	reference (0.8)
FRX-ON-A	applicable	none	sessions page (re-read); review step (the agent’s first /new/ after exposure)	late_reread_ref	reference (0.8)
FRX-OFF-I	control	none	sessions page (re-read) (the agent’s first /new/ after exposure)	--	--

Appendix

Table 36: What the agent could know, and when. Per condition: whether the authoritative row changes, what announces it, where the current value is visible after the change, and the scripted evidence that the condition is solvable and discriminating — a legal policy that scores 1.0 and a declared deficient policy that does not. The trigger column is the certified anchor: the change fires on the agent’s own request, after it has been served the sessions page. Derived from the certificates, the audit spec and the blocks’ feasibility artifacts; no cell is typed.

block	analysis	complete pairs	control cell	treatment cell	exact $p$
B	reported (implemented rules)	10	FRX-OFF-A 0/10	FRX-ON-A 10/10	0.00195
	delivered, text or image (ADD. 16)	10	FRX-OFF-A 0/10	FRX-ON-A 10/10	0.00195
	no action-cap rows	10	FRX-OFF-A 0/10	FRX-ON-A 10/10	0.00195
	both filters	10	FRX-OFF-A 0/10	FRX-ON-A 10/10	0.00195
BS	reported (implemented rules)	10	FRX-OFF-A 0/10	FRX-ON-A 6/10	0.0312
	delivered, text or image (ADD. 16)	9	FRX-OFF-A 0/9	FRX-ON-A 6/9	0.0312

Appendix

Table 37: Sensitivity of the paired results to analysis choices (historical blocks; exploratory, predecessor protocol). Each block is shown as reported under the rules actually implemented (which counted Sonnet’s 3 FRX-ON-A rows whose review display reached it as text only; ADDENDUM 5 would have voided and replaced them), then with rows whose intervention never reached a delivered observation in text or image removed (the ADDENDUM 16 definition), then with action-cap rows removed, then both; a filtered row takes its pair partner with it. For Block BS a last row applies ADDENDUM 5’s frame rule as registered, voided rows left unobserved (Appendix B ). Deletion is not the registered replacement procedure, whose outcomes are unobserved. Cells count each arm over the rows an analysis keeps; the exact test uses the complete pairs only, so in the ADD. 5 row the control cell also holds the rows whose partner that rule voids (“unpaired”). Computed by audit/sensitivity_analysis.py , whose reported column is checked against each block’s result file (Block B yes, Block BS yes, Block A2 yes).

block	treatment vs control	pairs	T / C / tied	net [95% CI]	exact $p$	Holm
A2	S-L2 vs S-C	8	0 / 1 / 7	$-0.13$ [ $-0.47$ , $0.22$ ]	1	1
B	FRX-ON-A vs FRX-OFF-A	10	10 / 0 / 0	$+1.00$ [ $0.61$ , $1.00$ ]	0.00195	0.00586
BS	FRX-ON-A vs FRX-OFF-A	10	6 / 0 / 4	$+0.60$ [ $0.20$ , $0.83$ ]	0.0312	0.0625
A2S	S-L2 vs S-C	4	2 / 0 / 2	$+0.50$ [ $-0.10$ , $0.85$ ]	0.5 (descr.)	–
BS, no action-cap pairs	FRX-ON-A vs FRX-OFF-A	8	5 / 0 / 3	$+0.63$ [ $0.17$ , $0.86$ ]	0.0625 (retro.)	–

Appendix

Table 38: Paired statistics. T / C: pairs in which only the treatment / only the control arm succeeded. Net difference P(treatment)

-

P(control) over matched instances, ties in the denominator, with a Newcombe (1998, method 10) hybrid-score marginal 95% interval (not exact, not family-adjusted); exact two-sided sign test on the discordant pairs. The envos effect field of the result files is the discordant-pair proportion , which is not a net effect (it reads 0.125 for Block A2, whose net is

-0.13

). Holm is a retrospective correction over the 3 pre-declared primary tests, which were registered at

\alpha=0.05

each; it is superseded by the paper-wide family of Table 31 and kept for comparison. The last row drops the pairs in which either arm hit the action cap (retrospective). Block A2S’s test was withdrawn before any rollout (ADDENDUM 13); its row is descriptive. All rows are historical and exploratory (predecessor protocol); the intervals are recomputed under the dated erratum of Appendix L .

Figure 5: State precedes channel. One changed episode (Block A2, seed 8) on the axis of the agent’s own requests to the portal. The authoritative row changes inside the request the exposure condition selects, the notice is posted with the same mutation and rendered on the next request, and the agent’s re-read of the sessions page shows the new location before the clinic is selected and committed. Every marker is read from the episode’s access log, event log and checkpoints.

Figure 6: Block A2, per instance and within episode. (a) Outcomes per generated instance: filled, pass; open, fail (the failure’s signature code is written under it and spelled out in the key); “1st” marks the arm the seeded order ran first. (b) For each changed run, the action index at which the change fired, the notice frame was delivered, the agent re-read the sessions page, selected the clinic in the wizard, and committed; the shaded segment spans change to commit. 7 of 7 successful runs re-read the authoritative page at or after the request that fired the change and before selecting (the page served by the firing request already carries the new location, so the re-read and the change can coincide); the one failure did not. Dotted line: the 48-action budget.

Figure 7: Block B: silent supersession, with and without re-exposure (Claude Opus 5, 10 paired instances). Same encoding as Figure 6 ; the review-information marker on FRX-ON arms is the review step re-rendering the live row. FRX-OFF-A 0/10, FRX-ON-A 10/10,

p=0.002

#	inference the defect permitted	executable check that rejects it now	corrected conclusion	rows (model)
1	the clean control fails, so the family is hard	clean-control stop rule ( run_block : a counted clean failure halts the block) + the memo channel declared by the task	harness lacked working memory; attempt voided; Block A2 clean 2/2	1 (1)
2	a session-limit reply is a model failure	provider-limit signal in the sealed invoker $\to$ infra_rejected , block pauses, reserve untouched	not counted; no failure rate moved	1 (1)
3	a changed-arm success with no change is evidence of recovery	exposure-conditioned trigger ( requires_seen ) + validity ladder branch not_triggered + pause rule	3 of 12 episodes were not instances; re-run as Block A2	11 (8)
4	correct runs read as unexposed	scorer places the change on the firing request; C4 recomputed; independent scorer cross-checks	exposure certified in 8/8 changed runs	3 (3)
5	the family’s reference policy solves the silent arm, so its failures are model deficits	per-level certification of the expected matrix before any model: the ordinary reference must score 0.8 and the late_reread_reference 1.0	trigger timing kept (first wizard request after exposure); reference redefined to re-read the source inside the wizard (10/10 at 1.0), ordinary reference declared deficient; certificate 40/40	–
6	a 6-pair block is adequate for the rescue test	pre-registered power arithmetic ( envos.stats ) checked at freeze	6 unanimous pairs do reject ( $p=0.0312$ ), but one tie or one pair against forfeits rejection; enlarged to 10 pairs before any Block B rollout, which tolerates one pair against ( $p=0.021$ )	–

Appendix

Table 39: Certification challenge. For each of the 11 defects: the inference it permitted, the executable check that rejects it now, and the corrected conclusion; “rows (model)” counts archived search-log rows scored under the defect and how many already carried a model verdict.

#	defect	archived evidence	rows (model)
1	no working memory across stateless per-step calls; the clean control failed and the stop rule halted the block	P3v2/_void_blockA_attempt1_2026-09-17/ ; P3v2/_void_blockA_attempt1_2026-09-17/search_log_blockA.json ; PRECOMMIT_P3v2.md	1 (1)
2	a provider session-limit reply was scored as a model failure	P3v2/_void_blockA_attempt1_2026-09-17/ ; P3v2/_void_blockA_attempt1_2026-09-17/search_log_blockA.json ; PRECOMMIT_P3v2.md	1 (1)
3	the trigger depended on the agent revisiting a page, so episodes were not instances of the phenomenon	P3v2/_superseded_blockA_2026-09-17/ ; P3v2/_superseded_blockA_2026-09-17/search_log_blockA.json ; PRECOMMIT_P3v2.md	11 (8)
4	the scorer located the change one request early, so the exposure check read false on correct episodes	P3v2/_superseded_blockA_2026-09-17/ ; P3v2/_superseded_blockA_2026-09-17/search_log_blockA.json ; PRECOMMIT_P3v2.md	3 (3)
5	the silent arm as first frozen defeated every policy that re-reads before the wizard, including the certified reference	PRECOMMIT_P3v2.md	–
6	six pairs could reject only if every pair crossed one way	PRECOMMIT_P3v2.md	–

Appendix

Table 47

Phenomenon	Causal situation	Example realization in an application
Contested Resource	A previously selected resource is taken before commitment	A chosen clinic slot is booked by someone else; a selected hotel room becomes unavailable
Ghost Commit	An accepted write stores a different value	The requested appointment time or hotel room differs from the committed record
Mandate Drift	A task-relevant authoritative fact changes in place	A request’s vaccine changes; a hotel booking policy is reissued
Cold-Start Inconsistency	Related application records disagree before the first action	A travel environment starts with inconsistent cross-application state

Appendix

Table 41: Phenomenon families and application-level perturbation examples in the exploratory corpus. These examples are not additional prospective review-disclosure conditions.

Phenomenon family	Recovery discipline	Environments	Successful rollouts
Contested Resource	Recheck resource availability before committing	2	1/5
Ghost Commit	Verify committed value and repair non-destructively	2	0/6
Mandate Drift	Revisit a previously acquired authoritative fact	2	1/9
Cold-Start Inconsistency	Check inherited cross-application state	1	1/3
Pooled four-family subset	—	3	3/23

Appendix

Table 42: Descriptive performance in four supplementary phenomenon families. The leak-affected variant was excluded from these family counts. These rates are not the main paper’s prospective stale-world results.

Event	Executed operation	Evidence and interpretation
$e_{0}$	Initialize synthetic task instance with session location $v_{0}$ .	Pre-state snapshot identifies target and irrelevant records.
$e_{1}$	Agent request serves the sessions page containing $v_{0}$ .	Application access log proves the old-value serve.
$e_{2}$	Harness delivers screenshot/text showing $v_{0}$ to the policy.	Frame and channel logs establish delivery, not comprehension.
$e_{3}$	First eligible booking-wizard request follows.	Anchored request ID and once-only guard determine trigger.
$e_{4}$	Perturbation atomically changes the stored location to $v_{1}$ .	Transaction/event log and authoritative row snapshot confirm mutation.
$e_{5}$	Agent reaches review: OFF, CUE or LIVE information is rendered.	Delivery of review-time information is evaluated independently from the mutation.

Appendix

Table 43: Illustrative timeline for one valid injected clinic episode. The table distinguishes application evidence from policy delivery and final task scoring.

Computer-Use Agent Evaluation

See all 38 papers

May 9, 2026

When Agents Overtrust Environmental Evidence: An Extensible Agentic Framework for Benchmarking Evidence-Grounding Defects in LLM Agents

Large language model agents increasingly operate through environment-facing scaffolds that expose files, web pages, APIs, and logs. These observations influence tool use, state tracking, and action sequencing, yet their reliability and authority are often uncertain. Environmental grounding is therefore a systems-level problem involving context admission, evidence provenance, freshness checking, verification policy, action gating, and model reasoning. Existing agent benchmarks mainly evaluate task capability or specific attacks such as prompt injection and memory poisoning, but they under-specify a fundamental reliability question: whether agents remain grounded in the true environment state when observations are stale, incorrect, or malicious. We introduce EnvTrustBench, an agentic framework for benchmarking this failure mode. We define an evidence-grounding defect (EGD) as a behavioral failure in which an agent treats an environment-facing claim as sufficient evidence for action without resolving it against available current evidence, leading to a task-incorrect false path under the true environment state. Given a task scenario, EnvTrustBench generates the workspace, environment, agent-facing objective, and validation oracle, executes the evaluated agent, records its action-observation trajectory and final state, and applies the oracle to produce a verdict. Using 6 LLM backbones and 5 widely used scaffolds, we evaluate 55 generated cases across 11 task scenarios, with each scenario expanded through five feedback-guided generation iterations. Results show that EGDs consistently emerge across operational workflows, highlighting environmental grounding as a core agent reliability problem with important security implications.

Strick Sheng, Ziyue Wang, Liyi Zhou

The University of Sydney · Nanjing University

AI Agent Reliability LLM Agent Evaluation AI Agent Benchmarks LLM Agents LLM Agent Reliability

Ask PDF

Sep 26, 2026

"You're Right, Let Me Fix It": How LLM Agents Damage Correct Work When Falsely Accused

LLM agents increasingly keep working after a task succeeds as they resume after compaction or take over handoffs. Their finished work keeps receiving follow-up input that sometimes falsely accuses it for later failures. We call an agent's acceptance of such a false accusation gaslight sycophancy, and destructive over-correction when acting on it damages previously correct work. We introduce CAVE-Bench, a benchmark of 365 agentic tasks across six domains built around opaque tasks. Every scored run first reaches a verified correct state, whose supporting rationale and history stay in the workspace while the facts that would settle the accusation lie in external or runtime state beyond the agent's reach. The agent cannot confirm or refute the claim with a local check, so the right response should keep the work and ask for the missing evidence. Each task either hands the agent correct work with saved evidence or let it build and verify that work first, and five risk factors set how the accusation enters the workflow. We score accusation acceptance and evidence use from the trajectory and measure harm by deterministic replay of downstream events. Across 14 of the latest models in Claude Code, false accusations damage correct work in up to 60.06% of runs, and stronger models often do so after recovering the supporting evidence. The same model behaves differently across OpenCode, Codex, and Hermes, and a harness gate driven by the benchmark's live signals cuts replayed harm by 74%. These results show that preserving already-correct work under unsupported accusation is a distinct safety challenge for long-lived agents. Our project is in https://henrymao2004.github.io/agent-over-correction/.

Xutao Mao, Rui Qian, Longxiang Wang +6

City University of Hong Kong · Fudan University · Southeast University +2

LLM Agents LLM Agent Safety LLM Agent Reliability LLM Sycophancy AI Agent Safety Benchmarks

Ask PDF

Oct 8, 2026

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation

Verifier scores now serve as both benchmark metrics and training rewards for large language model (LLM) agents, and a change in score is routinely read as a change in capability. It may instead reflect a change in the evaluation. We introduce TRACE, a protocol that turns a score change from a verdict into a testable diagnosis: it applies a targeted change to one part of an evaluation, compares paired runs, checks whether the agent's behavior changed, and rescores unchanged trajectories to test whether the scoring rule is responsible. In a controlled suite of 25 synthetic tasks, renaming tools lowers a scripted agent's score by 0.250 even though it performs exactly the same operations; restoring the original names at scoring time closes the entire gap, while the same mutation exposes a genuine behavioral failure in a second agent. On public

τ^2

-bench tasks with four LLM agents, an initial 30-task study finds mixed reward changes whose one clear effect does not replicate. In a larger follow-up on 88 new tasks with repeated runs per condition, renaming tools or reformatting tool outputs leaves reward unchanged to within

\pm

0.10 for seven of eight agent-change pairs, whereas tool names that deliberately mislead lower every agent's reward by 0.20-0.44, showing that the setup can detect real effects. Identical reruns flip 15-36% of task outcomes, so single-run comparisons cannot separate presentation effects from run-to-run variation. Two frontier LLM judges give consistent verdicts when a fixed trajectory is presented differently, yet disagree with each other on 57% of the same records, largely because one grades procedure rather than outcome. TRACE thus separates what a score change says about the agent from what it says about the measurement.

Radhika Gaonkar

Prime Intellect

LLM Agent Evaluation AI Agent Evaluation AI Agent Benchmarks

Ask PDF

Abstract

Computer-Use Agent Evaluation

Explore similar work

When Agents Overtrust Environmental Evidence: An Extensible Agentic Framework for Benchmarking Evidence-Grounding Defects in LLM Agents

"You're Right, Let Me Fix It": How LLM Agents Damage Correct Work When Falsely Accused

TRACE: Diagnosing Verifier Brittleness in Agentic Evaluation