Agent evaluations commonly score the state observed when a run stops and count the run as one trial. Interpreting that score as a final result from a separate trial requires outcome finality and cross-unit separation. Outcome finality requires that later events cannot change the claimed result, while cross-unit separation requires that earlier runs cannot change the relevant conditions of later ones. The endpoint establishes neither condition by itself, and the two can hold independently. Waiting for a delayed outcome may settle the label even though its state remains available to another run. Isolation may prevent carryover even though the scored outcome remains unresolved. We develop a completion argument that identifies the evidence needed for each decision. A final success or failure label is justified only when every relevant effect is resolved or bounded tightly enough to fix the outcome. Any remaining uncertainty must be reported. First, in a controlled replay with fixed agent actions, we find that endpoint and terminal labels differ for every nonzero-delay operation and that a delayed write changes the next run's score under shared state but has no such effect after namespacing or verified reset. Second, in a review of ten public protocols, we find that reset or deliberate retention is documented explicitly more often than unfinished operations or evidence for separate scoring. Finally, we propose an open-effects record for operations and resources that may remain relevant after the endpoint, their status, and their possible effects on the scored outcome or another run.
Figures & tables
Decision
Evidence needed
Decision supported
Response when evidence is missing
Outcome finality
Every relevant operation or event that could change the outcome has been resolved or bounded tightly enough to fix the label.
Assigning a final success or failure label.
Wait for completion, cancel and confirm, or report the case as unresolved.
Cross-unit separation
No relevant operation or shared state lets one run change another’s relevant conditions or outcome.
Counting runs as separate analysis units.
Isolate and verify, or stop counting connected runs separately by modeling the connection or grouping the runs.
Table 1: Outcome finality and cross-unit separation require different evidence and support different decisions.
Test
Policy or condition
Exact outcome
What changes
Finality
Snapshot
0/150 successes at nonzero delays
The outcome is unresolved when scored.
Finality
Reconciliation
150/150 successes at nonzero delays
The write’s success is observed.
Finality
Verified cancellation
0/150 successes at nonzero delays
Each pending operation reaches a confirmed cancelled state and is scored as failure.
Finality
Synchronous control
0/600 disagreements
Endpoint and terminal labels agree.
Separation
Shared state
150/200 exposures
B’s mean score is 0.75 lower than under namespacing.
Separation
Namespacing, reset, and no-write controls
0/200 in each condition
The route between A and B is absent or blocked.
Table 2: Exact outcomes across the fixed schedule. The counts describe constructed conditions and do not estimate a benchmark population.
Publicly visible field
Explicit
Partial
NR
NA
Reset or deliberate retention
8
2
0
0
Open or descendant work
0
2
6
2
Evidence for separate scoring
3
7
0
0
Table 3: Documentation of boundary-related fields in the ten reviewed protocols. NR means not reported and NA means not applicable.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Protocol
Open work
Reset or retention
Separate evidence
WebArena
NR
E
P
WorkArena
NR
E
P
OSWorld
NR
E
P
SWE-bench
P
E
E
τ -bench
NA
E
E
ToolSandbox
NA
E
P
Appendix
Table A.1: Protocol-level documentation codes. E means explicit, P means partial, NR means not reported, and NA means not applicable to the declared task and claim.
Protocol
Field
Code
Source location
Basis
WebArena
Open work
NR
Paper Sec. 3.2 and App. A.9
Stop actions and state checkers are specified, but handling of background work before scoring is not [ 37 ] .
Reset or retention
E
Paper App. A.2 and ABC T.4
The self-hosted sites can be restored, and the ABC review reports that site state is cleared between runs [ 37 , 38 ] .
Separate evidence
P
Paper App. A.2 and ABC T.4
Site restoration is described, but the reviewed sources do not document whether state outside those sites is reset [ 37 , 38 ] .
WorkArena
Open work
NR
Paper Secs. 4.2 and 5.2
The task lifecycle is specified, but the sources do not give a general rule for descendant or queued work [ 10 ] .
Reset or retention
E
Paper Sec. 4.2
Setup and teardown hooks cover resources created for the task [ 10 ] .
Separate evidence
P
Paper Secs. 4.2 and 5.2
Cleanup is described, but the sources do not report a check that every covered resource was restored [ 10 ] .
Appendix
Table A.2: Cell-level source ledger for the three coded fields.
Outcome finality test
Fixed trace schedules a delayed write
⟶ Interaction ends after the request is acknowledged
⟶ Score at the endpoint, after completion, or after verified cancellation
Compare the endpoint label with the terminal result
Cross-unit separation test
Run A schedules a service write
⟶ Run A ends and run B begins
⟶ Use shared state, a namespace, verified reset, or grouped analysis
Compare B’s exposure, score, and analysis unit
Appendix
Figure B.1: Controlled replay. The tool-call trace remains fixed. The protocol changes when the delayed operation is scored and whether service state is shared, isolated, reset, or represented by a grouped unit.
Field
Required content
Use at the boundary
Run and task
Run ID, task ID, and any larger stream or group ID.
Connects the effect to the proposed analysis unit.
Initiating event
Trace position, action, timestamp, and arguments needed to identify the effect.
Records where the effect began.
Resource and controller
Process, transaction, account, URL, file, queue, job, or service and the system that controls it.
Identifies where the effect must be observed or closed.
Observation handle
Stable process ID, transaction ID, job ID, namespace, or equivalent query handle.
Permits a system query without relying on the agent’s description.
Prevents an acknowledgement from being treated as completion.
Possible transition
States still reachable and the period over which they remain relevant to the claimed outcome.
Shows whether the label may still change.
Appendix
Table C.1: Fields in an open-effects record. Free text may supplement the fields. The record should include a stable handle and system evidence whenever they are available.