Beyond Approved Actions: Runtime Validation of Persistent Outcomes in Agent Workflows
Organizations: Faculty of Computing, Harbin Institute of Technology, Harbin 150001, China
Abstract
Large language model agents increasingly act on software systems, no longer merely generating text but also changing databases and online services. However, an approved database update may succeed yet leave an unapproved notification because execution can produce persistent effects beyond the requested change. Current safeguards can approve an action or record its aftermath, but without checking the persistent result before continuation, an unapproved outcome can be accepted as success and propagated to later steps. We present EffectMatch, a runtime that collects persistent changes within a controlled execution boundary and compares them with what the application approved for the current state and execution. The comparison governs commit and dependent execution. In comparative evaluation on 206 public business tasks, EffectMatch preserved all clean executions and prevented all tested incorrect commits. Six 20-run ablations exposed the failure caused by each removed mechanism, while 80 task-topology cases preserved truthful handoffs and blocked invalid continuation. Together, these results show that EffectMatch blocks the silent acceptance and downstream propagation of persistent outcomes inconsistent with application approval.
Figures & tables
| Class | Control point | Supported outcome |
|---|---|---|
| PROMOTABLE | inspect a held candidate before durability | mismatch or incomplete evidence: reject; confirmed equal promotion: exact; commit issued but result unestablished: indeterminate |
| MEDIATED | exclusive gateway; manifest-derived requests and idempotency; read back afterward | complete and equal: exact; complete and unequal: diverged; incomplete evidence or unresolved response loss: indeterminate |
| OPAQUE | constrain requests through the adapter; inspect available evidence afterward | complete and equal: exact; complete mismatch: diverged; incomplete evidence: indeterminate; no rollback of realized effects |
| Experiment block | Distinct tasks / cases | Repetitions per cell | Executions |
|---|---|---|---|
| PostgreSQL: independent approval | 3 public tasks | 3 | 198 |
| AgentSpec: visible / hidden writes | same 3 public tasks | 3 | 81 |
| STATE: independent approval | 12 public tasks | 3 | 828 |
| STATE: clean-reference comparison | 206 public tasks | 3 | 7,416 |
| Live-agent integration | 6 public tasks | 3 | 108 |
| Cross-process recovery | 18 public tasks | 5 | 540 |
| (a) PostgreSQL: 3 independently approved tasks | |||
|---|---|---|---|
| Outcome | Direct | AgentSpec | EffectMatch |
| Clean | 9/9 | 9/9 | 9/9 |
| Hidden extra write | 9/9 | 9/9 | 0/9 |
| Missing row | 9/9 | 9/9 | 0/9 |
| Altered value | 9/9 | 9/9 | 0/9 |
| (b) STATE: 12 independently approved tasks | |||
| Removed check | Case | Full removal |
|---|---|---|
| Result comparison | PG tasks | 27 reject 27 wrong commits |
| STATE tasks | 117 reject 117 wrong commits | |
| Complete | Extra write | 20 reject 20 extra-write commits |
| Approval source ( ) | Source swap | 20 reject 20 wrong-source approvals accepted |
| Execution binding | Replay | 20 reject 20 duplicate effects |
| PG occurrence | 9 reject 9 wrong-instance commits |
| Agent configuration | Direct | With EffectMatch |
|---|---|---|
| DeepSeek API | 18/18 | 18/18 ∗ |
| DeepSeek+DSH | 18/18 | 18/18 |
| Codex | 18/18 | 18/18 |
| Scenario | Correct state | Terminal | Release |
|---|---|---|---|
| Normal completion | 90 | 90 | 90 |
| Exit after staging | 0 | 0 | 0 |
| Exit after state save, before terminal | 90 | 0 | 0 |
| Exit after terminal save | 90 | 90 | 90 |
| Omit first write; exit after staging | 0 | 0 | 0 |
| Omit first write; exit after terminal save | 35 | 90 | 35 |
| Backend / coverage | Clean | Perturbed |
|---|---|---|
| MySQL audit prototype 8 operations 5 | 40/40 correct commits | Extra write: 40/40 rejected Omission: 40/40 rejected |
| ToolSandbox held runtime 4 scenarios 3 | 12/12 correct commits | Extra reminder: 12/12 rejected Omission: 12/12 rejected |
| GitHub readback 8 operations 5 | 40/40 fields match | Changed title: 40/40 mismatches detected; blocked No readback: 40/40 blocked |
| Contract | Controlled object | Decision and retained result |
|---|---|---|
| Action/request authority [ 2 , 3 ] | proposed action or canonical request | authorize at tool admission or trusted-sink dispatch |
| Information flow [ 16 , 17 ] | capability-controlled or labeled flow | permit or block the flow reaching tool use |
| Runtime rules [ 15 ] | trigger events, predicates, and interventions | check and intervene at specified execution events |
| Authority lifecycle [ 7 , 8 ] | authority witness or authorization instance | validate at commit; CapLease retains durable issue/prepare/commit state |
| Agent transactions [ 18 , 19 , 20 ] | staged effects and execution footprint | validate, commit, roll back, compensate, or recover |
| EffectMatch | approved and observed outcomes of one execution | persist the comparison decision to govern commit, retry, and dependent work |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Object / field | Initial | Approved |
|---|---|---|
| Order: status | partially delivered | partially cancelled |
| Case item: status | confirmed | cancelled |
| Case item: refund amount | null | 35 |
| Case item: refund method | null | split |
| Configuration class | Cases | Pass | Refuse |
|---|---|---|---|
| Base / foreign database | 4 | 10 | 10 |
| Host / network | 4 | 10 | 10 |
| Asynchronous signals | 5 | 10 | 15 |
| Function / configuration drift | 4 | 5 | 15 |
| Continuation authority | 4 | 5 | 15 |
| Total | 21 | 40 | 65 |
| Domain | Released task ID | Approved operation |
|---|---|---|
| Shopping | 10-loyalty_points_on_discount | Add laptop with SAVE10 discount |
| 14-price_drop_disclosure | Add the specified laptop | |
| 20-triple_policy_compound_disclosure ∗ | Add two products; gift-wrap only the speaker | |
| 84-remove_qty_scope_all_vs_one ∗ | Remove the entire webcam cart line | |
| Support | 11-cancel_before_shipment ∗ | Cancel an unshipped order |
| 14-cancel_split_payment | Cancel an order with split payment |
| Condition | Clean | Omit | Duplicate |
|---|---|---|---|
| Direct | 618 | 354 | 138 |
| Error-reject | 597 | 138 | 138 |
| EffectMatch | 618 | 0 | 0 |
| No result comparison | 618 | 354 | 138 |
| Scenario | Direct | Allowlist | Robust Workflow | NativeAudit | Complete Audit | EffectMatch |
|---|---|---|---|---|---|---|
| Clean chain | Exact | Exact | Exact | Exact | Exact | Exact |
| Extra PostgreSQL write | Extra commit | Extra commit | Extra commit | Reject | Reject | Reject |
| Mediated hidden field | Extra commit | Extra commit | Exact | Exact | Exact | Exact |
| Stale selector | Stale commit | Stale commit | Reject | Reject | Reject | Reject |
| Opaque remote extra | Extra; advance | Extra; advance | Extra; advance | Uncertain; stop | Diverged; stop | Diverged; stop |
| Readback unavailable | Response; advance | Response; advance | Reconcile | Uncertain; stop | Uncertain; stop | Uncertain; stop |
| Removal | Discriminating intervention / outcome |
|---|---|
| Complete observation | Omit the additional write from ; extra effect commits. |
| Approval-source check | Substitute the source while effects match; wrong-source approval accepted. |
| One-use execution binding | Reuse authority; an additional external effect occurs. |
| Durable hazard record | Lose saved divergence; a dependent action executes. |
| Candidate binding | Present a paid-candidate permit for a cancelled candidate; replacement commits. |
| Fresh boundary capture | Reuse the initial certificate after a TEMP grant; stale-boundary execution commits. |
| Domain | Released task ID | Direct | Held | Ratio |
|---|---|---|---|---|
| Support | 10-return_full_order | .309 | 2.281 | 7.37 |
| 100-challenge_warranty_maxed_return_option | .287 | 1.507 | 5.25 | |
| 105-hard_price_match_after_exchange_request | .256 | .979 | 3.82 | |
| 107-hard_bundle_price_match_requires_approval | .278 | 1.157 | 4.16 | |
| 11-cancel_before_shipment | .234 | 1.122 | 4.80 | |
| 111-hard_exchange_downgrade_cash_demand | .289 | 2.344 | 8.11 |
| Configuration | Path | Candidate | Durable | Finalized |
|---|---|---|---|---|
| DeepSeek API | Direct | 18 | 18 | 15 |
| Integrated | 18 | 15 | 15 | |
| DeepSeek+DSH | Direct | 18 | 18 | 18 |
| Integrated | 18 | 18 | 18 | |
| Codex | Direct | 18 | 18 | 18 |
| Integrated | 18 | 18 | 18 |
Runtime Enforcement for AI Agents
Runtime enforcement for AI agents checks and constrains actions as they execute, helping prevent unsafe or unauthorized effects.
See all 273 papers