Organizations: Carnegie Mellon University Pittsburgh, PA 15213, USA · Feinity LLC Sacramento, CA 95816, USA · Computer Science Department Dartmouth College Hanover, NH 03755, USA · Independent Researcher Los Angeles, CA 90095, USA
Tool-using language agents increasingly modify external systems, yet most runtime safeguards allocate scrutiny by generic uncertainty scores or coarse mutation heuristics. We introduce CASEL (Consequence-Aware Selective Execution Layer), a model-agnostic runtime allocation policy that combines calibrated failure probability, state-dependent consequence, reversibility, ambiguity, and evidence freshness to choose, for each proposed tool call, among direct execution, verification, sandboxing, clarification, approval, and abstention by minimizing a per-mode cost. We formalize direct execution as selective prediction and state and test the validity conditions of a finite-sample threshold and a trajectory risk ledger. In a controlled simulation over 20 seeds with 120,000 test actions each, in which every method has the same intervention menu and the same compute cost, CASEL reduces harmful-side-effect trajectories from 14.75 percent for an uncertainty-ranked gate to 7.02 percent and mean normalized harm from 0.184 to 0.123. A gate ranked by the product of failure probability and consequence reaches similar harm (5.82 percent, 0.120) but sends 45 percent more actions to human approval, and removing reversibility-aware routing raises the rate to 15.53 percent. A trajectory-level risk budget, in contrast, increases harm at equal cost (11.82 percent). Intervention effectiveness is assumed, not measured: the results rank allocation rules under the simulator's assumptions and do not establish deployment effectiveness.
Figures & tables
Figure 1: CASEL routes every proposed tool call by consequence-weighted risk, not by uncertainty alone. (a) Structured risk signals feed a controller that minimizes the per-mode cost J(m) over the modes feasible for the action’s reversibility class; dashed elements are not part of the main comparison; clarify is tested in Sec. VI-E , abstain is not exercised. (b) Worked example with the nominal parameters (inset): three calls with the same p^t but different c^t go to direct execution, sandboxing, and approval (lowest J in bold); an uncertainty-only gate cannot tell them apart.
Approach
Problem focus
Granularity
Risk signal
Intervention space
Relation to CASEL
SAGE-Agent [ 9 ]
Underspecified intent
Question
Spec. vs. model unc.
Clarify
Implements clarify ; triggered by gt
REFLEX [ 22 ]
Compute-efficient decisions
Decision
Model confidence
Call strong LLM
Orthogonal: chooses the decision maker
CA2I [ 23 ]
Pre-execution outcomes
Action sequence
Predicted return, risk
Select a plan
Orthogonal: admits whole plans
CASEL
Scrutiny before commit
Step
p^tc^t,ρt,gt,dt
Six modes, cost-aware
—
Table 1: Qualitative comparison with alternative approaches. Step : one decision per proposed tool call.
Figure 2: Isotonic calibration turns the raw failure score into a usable probability. (a) ECE and (b) Brier score of the raw score and of the calibrated p^ on test actions. Bars: means over 20 seeds with 95% CIs; lines: paired per-seed values.
Method (ranking)
Menu
Task Success ↑
HSR ↓
MNH ↓
Direct Cov.
Cost
Approvals ↓
Base
—
29.48±0.49
32.76±0.70
0.428
100.0%
0.000
0.0%
Always-Verify
verify
58.81±0.71
15.47±0.53
0.180
0.0%
1.000
0.0%
Mutation-Gated
verify
43.78±0.62
15.47±0.53
0.196
58.0%
0.420
0.0%
Uncertainty gate ( p^ )
verify
47.78±0.56
19.25±0.57
0.233
51.4%
0.486
0.0%
Consequence gate ( c^ )
verify
45.10±0.63
15.47±0.53
0.195
51.4%
0.486
0.0%
Uncertainty-Menu ( p^ )
full
48.79±0.63
14.75±0.41
0.184
66.9%
0.486
6.9%
Table 2: Controlled evaluation over 20 seeds (task success and HSR: mean ± one standard deviation). All methods below Mutation-Gated have compute cost 0.486 . Menu: interventions a method may use (full = verify, sandbox, approve).
Figure 3: Ranking actions by p^c^ halves the selective-risk AUC of uncertainty-only ranking. (a) Selective risk (mean consequence-weighted loss of directly executed actions) versus coverage when actions are admitted in increasing order of p^ , c^ , or p^c^ ; the arrow marks the largest gap. (b) Per-seed paired difference from the p^ ranking. (c) Area under each curve (average selective risk over coverage 0.1 – 1.0 ); dots are seeds. Bands and whiskers are 95% CIs over 20 seeds.
Figure 4: With equal menus, consequence-aware allocation has far lower harm than uncertainty ranking at every cost; CASEL matches a risk-ranked gate with fewer approvals, and the trajectory budget does not help. (a) HSR versus compute cost; the band marks the common cost 0.486 . (b) HSR versus the share of actions sent to human approval. (c) Paired differences at cost 0.486 , CASEL minus comparator, in HSR (red diamonds) and task success (blue dots), in points; ∗∗∗ : p<0.001 , two-sided paired t -test over the 20 seeds, Holm-corrected over the ten tests. Bands and bars are 95% CIs.
Runtime guardrails act before irreversible tool calls, but their guarantees depend on what policy state is representable, what a judge observes, and whether intervention changes future behavior. We separate three questions. First, relative to fixed oracle predicates, a deterministic gate enforces exactly the nonempty safety policies whose good prefixes its register model recognizes; policy nontriviality is undecidable with two decrementable counters but in PSPACE for a separable monotone fragment. Second, under a fixed exogenous law, Neyman-Pearson gives the exact false-block/miss frontier and conformal calibration gives a finite-sample marginal certificate, possibly via block-all. Third, once blocking changes future proposals, static scores and ungated trajectories need not identify the closed-loop frontier; a specified finite controlled model instead yields an occupancy program. Bounded representation attacks add a robustness margin, so benign calibration alone does not transfer. Experiments target these distinctions through static diagnostics, controlled-model enumeration, representation rewrites, and paired closed-loop reruns.
A valid tool call is not necessarily a valid intervention. Tool-using language agents are guarded by schema validators, policy filters, provenance checks, state predictors, and self-verification, yet such safeguards do not certify that a state-changing action has an identifiable causal effect. In confounded workflows, the action that looks optimal in observational logs can reduce utility when executed. We introduce CIVeX, a causal intervention verifier that maps proposed actions to structural causal queries over a committed action-state graph, checks identifiability, and returns one of four auditable verdicts: EXECUTE, REJECT, EXPERIMENT, or ABSTAIN. Execution requires an assumption-scoped causal certificate carrying graph commitments, an identification argument, a one-sided lower confidence bound (LCB), provenance, and risk limits. On Causal-ToolBench (1,890 instances, 7 seeds), CIVeX yields zero observed false executions across moderate and adversarial confounding. Under adversarial confounding it reaches 84.9% accuracy and 81.1% of oracle utility (+2.23 vs +2.76) and is the only non-oracle method whose constrained utility under a zero-false-execution constraint exceeds the AlwaysAbstain floor. On IHDP and ZOZO Open Bandit (real production logs with uniform-random ground truth), CIVeX matches Oracle correct-execution within 0.1pp and cuts per-execute false-execution by >=50x over naive baselines. A chain-of-thought LLM verifier (Claude Opus, Sonnet) cuts false-execution by an order of magnitude over a terse baseline, yet under adversarial confounding Opus's utility falls to 74% of CIVeX's. Intervention identifiability, not action validity, is the missing primitive for reliable tool use.
Modern AI agents execute real-world side effects through tool calls such as file operations, shell commands, HTTP requests, and database queries. A single unsafe action, including accidental deletion, credential exposure, or data exfiltration, can cause irreversible harm. Existing defenses are incomplete: post-hoc benchmarks measure behavior after execution, static guardrails miss obfuscation and multi-step context, and infrastructure sandboxes constrain where code runs without understanding what an action means. We present AgentTrust, a runtime safety layer that intercepts agent tool calls before execution and returns a structured verdict: allow, warn, block, or review. AgentTrust combines a shell deobfuscation normalizer, SafeFix suggestions for safer alternatives, RiskChain detection for multi-step attack chains, and a cache-aware LLM-as-Judge for ambiguous inputs. We release a 300-scenario benchmark across six risk categories and an additional 630 independently constructed real-world adversarial scenarios. On the internal benchmark, the production-only ruleset achieves 95.0% verdict accuracy and 73.7% risk-level accuracy at low-millisecond end-to-end latency. On the 630-scenario benchmark, evaluated under a patched ruleset and not claimed as zero-shot, AgentTrust achieves 96.7% verdict accuracy, including about 93% on shell-obfuscated payloads. AgentTrust is released under the AGPL-3.0 license and provides a Model Context Protocol server for MCP-compatible agents.