cs.AIOct 12, 2026
Saveagent-risk: Risk Assessment for Changes to AI Agents
Organizations: University of Southern California · London Business School · Arizona State University
Abstract
Before changing a deployed AI agent (a stricter safety gate, a new model), organizations must approve the change, starting from existing logs. Extending classical risk analysis to partial records and partial changes to running systems, agent-risk replays logged episodes, assuming nothing else changes and the agent does not anticipate the change. The log settles episodes acting as before, a declared stop value settles stopped ones, and the rest need logged support, a saved-state rerun, or a bound. We prove what logs must record for exact replay and which saved states serve a gate family. For independent episodes, we bound the reruns a decision needs and tolerable drift from live behavior. In a bash sandbox, a 7,200-episode log alone decided neither of two retry gates, which let the agent act after refusals the log never shows. Rerunning only each gate's blocked episodes from saved states reached both live decisions at half the agent calls of a live test sized post hoc for them, given the log. Without per-step harm, a stopping gate's certified harm ceiling rose up to 2.5-fold, since the log hides whether harm preceded the stop. Approval thus needs evidence designed in advance: log fields, saved states, targeted reruns.
Figures & tables
Figure 1: Replaying a logged episode under a proposed change. With sufficient records, the log gives the agreeing case and a declared stop value the stopped case; a snapshot run, logged support, or bounds the continuing case.
| Characteristic (where it enters) | What the logs can establish |
| 1 Approval gates (execution) | Stricter stopping gates: replay plus a declared stop value (§ 3.1 ). Retry gates: similar logged cases, reruns, or bounds (§ 3.4 ) |
| 2 Partial records (records) | Replay is exact only if the log keeps each episode’s contribution, e.g., harm timing relative to a stop (§ 3.2 ) |
| 3 Long horizons (execution) | Stopping changes replay at any length; for continuing ones, the share that agrees can stay the same or fall (§ 3.3 ) |
| 4 Model releases (admissible changes) | A new model rerun on logged inputs and seeds settles episodes where it acts as logged; the rest need similar cases, reruns, or bounds (§ 3.4 ) |
| 5 Shared state (execution) | With the shared state’s law known, single-agent tests bound joint risk; reruns from saved shared state settle it (§ 3.5 ) |
| 6 Oversight cues (execution) | Results carry over only for cues seen in testing; an agent told of the gate can act differently, breaking (A2) (§ 3.6 ) |
Table 1: Six organizing agent characteristics and what logs can establish under each. Characteristic enters the part of the risk specification (Section 2.3 ) in parentheses; its result is in Section 3 . and demonstration in Section 4 . .
Figure 2: Missing harm timing leaves stop risk unresolved. Identical records hide whether executed harm (coral) came before (A) or after (B) the stop at step 2; the gate keeps A’s harm and removes B’s.
Table 2: The log alone leaves retry gates open (?); rerunning only what it leaves open matches live runs ( ✓ ); simpler readings misestimate harm, and clean starts and an announced gate miss live runs ( ✗ ). : a logging requirement. P(harm) unless labeled. Match: Newcombe 90% interval inside the registered margin. Certified limits are exact one-sided 95%; log intervals are plug-in bounds. In / cells, is gate 1; blank: not used.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Symbol | Meaning | Exact definition |
| Risk specification: execution mechanism, consequence map, observation mechanism, admissible replacements | Appendix A.1 | |
| ; | Finite horizon; step index, shared by execution and records | Appendix A.1 |
| ; | Tool calls proposed at step ; the monitor’s score for them | Section 2.2 |
| ; ; | A replacement (a change); the stopping gate; its score threshold | Appendix A.1 |
| ; | Full-episode loss vector (failure, harm, cost); its finite set of values | Appendix A.1 |
| ; | Evaluator records up to ; a value of with positive baseline probability | Appendix A.1 |
Table A.1: Symbols of Sections 2 and 3 .
| Classical tool | Role, essential condition, and status in specification |
|---|---|
| Scenario triplets ( Kaplan and Garrick, 1981 ) | Kept with named kernels of and loss ; requires declared consequence definitions under changed systems. |
| Partial observation ( Bäuerle and Rieder, 2017 ) | Kept in observation record and Proposition 3.1 ; deployer-designed records differ from agent context. |
| Mechanism intervention ( Pearl, 1995 ) | Kept in intervention family under (A1) and (A2); replaces agent decision rules with declared timing and inputs. |
| g-formula and censoring at deviation ( Robins, 1986 ; Cain et al., 2010 ) | Inherited in Proposition A.4.1 ; agreement uses (R), (A1), and (A2), and stopping uses declared abort valuations (A3). |
| Logged replay ( Li et al., 2011 ) | Inherited in Proposition A.4.1 and deviation time ; requires rerunnability (R) via recorded seeds and invariant serving. |
| Worst-case bounds ( Manski, 2003 ) | Inherited in Proposition E.1.1 and Proposition A.5.1 ; bounded losses give prefix-feasible bounds, sharp under (S) with full records and outer otherwise; shared-law restrictions may narrow them. |
Table B.1: Classical tools, standard assumptions, agent strains, and roles in the specification.
| Run | Gate | Episodes | Serving | Role |
| Exploratory log | Pass-all | 300 | Default kernels | Exploratory; chosen here |
| Exploratory live gates | Stopping gate at , , | 300 each (286 scored at 0.99) | Default kernels | Exploratory live comparison with replay from the exploratory log |
| Held-out log | Pass-all | 300 | Batch-invariant | Primary held-out log; source of the retry bounds and the step-0 continuations |
| Live retry gate | Retry gate, , retry cap 2 | 300 | Batch-invariant | Live check of the retry bounds, after the fact |
| Targeted-execution run | Pass-all with a snapshot at the first block; continuations under the retry gate; live retry gate | 1,200 units (2,885 episodes) | Batch-invariant, two hosts | Registered; prediction frozen before the live arm was read |
| First announcement test | Stopping gate, announced | 300 | Batch-invariant | Registered null against the held-out log (no margin; analysis fixed after launch); followed up by the paired announcement test |
Table D.1: Runs behind Section 4 . Batch-invariant serving returns the same output for the same input whatever else shares the batch; default kernels do not. ARTIFACT.md maps each run to the code that its logs and commands use.
| Change | Status | Harm (interval type) | Rests on | To settle |
| 1 Stopping gate, | Identified; implementation check | P(harm): replay 0.039, live 0.030, equiv. within 0.03 (Newcombe 90%). | (R), (A1), (A2); (A3) declared, harm untested | Test (A3) for harm |
| 1 Stopping gate, replications | Identified; implementation check | Equiv. within 0.03 (Newcombe paired 90%): stronger agent ; airline . | (R), (A1), (A2); outputs vary across runs; (A3) declared | Test (A3) for harm; margin vs effect |
| 1 Two retry gates, one log | Identified by targeted execution | P(harm): log and ; log + 1,032 and 2,929 snapshot continuations: 0.279 and 0.158; live 0.288 and 0.165, equiv. within 0.015 (Newcombe paired 90% and ); pass-all and retry-as-stop outside the margin. Cost: 0.08 and 0.23 of live agent calls, given the log (partial). | (R), (A1), (A2); snapshot (T): 3,961 of 3,961 continuations valid | Check (T) beyond file tree |
| 1 Retry gate, bash agent | Identified by targeted execution | P(harm): log (plug-in outer bounds); log + 485 snapshot continuations: 0.029 (exact 95% ); live 0.030, equiv. within 0.02 (Newcombe paired 90% ). Cost: 0.30 of live agent calls, given the log (partial). | (R), (A1), (A2); snapshot (T): 485 of 485 restoration checks passed | Check (T) beyond file tree |
| 2 Log without per-step harm | Set; identified with per-step harm | Certified P(harm) without per-step harm: held-out 0.159, replication 0.111; with it: 0.064, 0.051 (exact one-sided 95%) | As stopping gate | Log harm timing |
| 3 Both gates, 40-step cap | Stop identified; retry bounded | P(harm): stopping 0.043 for horizons (unknowns harmful: 0.067); retry (plug-in outer bounds), agreement 0.77 ( ) to 0.57 ( ) | As both gates; 17 unknown outcomes (disk incident) | Execute retry continuations |
Table D.2: Table 2 in full: what the deployer can state before each change ships, and what would settle the rest. Honest agents; in the bash setting, failure is near saturation and was chosen on an exploratory log; the stronger-agent and airline thresholds were chosen on pilots. Mint: identified, by the first two terms of display ( 2 ) or by targeted execution under (T). Gray: needs support or bounds, or replay does not apply. Coral: evidence that would settle the rest (last column). Newcombe: difference-of-proportions score interval; exact: Clopper-Pearson; plug-in: without sampling uncertainty; equiv.: equivalent. Failure, task success, and cost: Appendix D .
Figure D.1: Targeted execution and snapshot starts match within registered margins; the unannounced gate stays compatible. A thick 90% Newcombe bar inside the gray band is a match (with registered cluster checks). Otherwise a thin 95% whisker decides: excluding zero is a failure, and containing zero is compatible (unknown outcomes: midpoints and envelopes).
Figure D.2: The log alone leaves each gate’s P(harm) on both sides of the ceiling; executed continuations settle it and match live runs (bars: exact 95%). Gray: point estimates from retry as stop to the log-only upper bound; diamond: post hoc skip-blocked reading.
Figure D.3: Implementation checks: stopping-gate replay matches live harm in all three settings. The two “Stopping gate” rows use unpaired Newcombe intervals. Intervals, midpoint dots, and verdicts follow Figure D.1 ; the airline check also requires its task-cluster interval to fit the margin.
| Field | Needed for | Droppable when | Evidence |
| Monitor scores | Deviation | Monitor reruns deterministically | Held-out: 3,589 of 3,589 recomputed |
| Per-step harm | Stop valuation | Calls and a deterministic checker fix timing | Certified ceiling with vs. without it: held-out 0.064 vs. 0.159, replication 0.051 vs. 0.111; airline: 1,200 of 1,200 recomputed |
| Per-step cost | Pre-stop cost | Another field fixes it | Mean tokens, exploratory gates (companion) |
| Inputs, seeds, serving mode | Agent rerun | Change reads logged fields only | 1,650 of 1,650 decisions; other kernel mode: 2 of 99 episodes |
| Gate decisions | Nothing | Always (Remark E.5.1 ) | Unused |
Table D.3: What a stopping-gate replay needs from the log here: per-step harm (shaded) is the one field whose removal raised the certified harm ceiling. An instance of Proposition 3.1 ; no general field list. Certification: exact one-sided 95% bounds on replayed P(harm).
More in this topic
AI Risk Management
AI risk management studies how to identify, assess, reduce, and monitor harms from AI across development, deployment, and wider social systems.
See all 108 papers