Knowing When to Intervene: Causal Evaluation of AI Agents
Organizations: Stanford University
Abstract
Current evaluations of agentic AI systems primarily rely on benchmarks. However, the static nature of benchmarks makes them uninformative about the value of auditing interventions at deployment time. For reliable deployment, we often need to determine within an agent's trajectory whether or not to intervene, e.g., by verifying the agent's current output or steering it toward different behavior, using either human review or more costly models or time-consuming agentic supervision. The value of such interventions is fundamentally a causal inference question, asking whether the causal effect of such an intervention is beneficial for the final outcome. Estimating this causal effect is particularly challenging due to the dynamic nature of the problem: the impact of an intervention depends on the current context and environment, as well as the agent's future behavior. We develop the first methodologically grounded framework for evaluating the causal effects of randomized interventions within agent trajectories. By leveraging a relatively small dataset of agent trajectories collected under randomized interventions such as verification or screening, we can estimate the dynamic causal effect of a rich class of (contextual) intervention policies. The structural parameters that we estimate can also be used to construct safety leaderboards that capture the risk of various agentic actions, which can be used by practitioners for audit prioritization. Applying our methods to tau-bench Retail, ToolSandbox, and ImpossibleBench, we find that intervention effects vary substantially across agent actions and states. More broadly, our work suggests a rich research agenda at the intersection of causal inference and agentic evaluation, which we believe is essential for the reliable and safe deployment of AI agents in the real world.
Figures & tables
| Benchmark | Agents | Intervention / Control | Outcome | Reporting Groups |
|---|---|---|---|---|
| -bench Retail [ 34 , 4 ] | GPT-5 nano; Claude Haiku 5.5 | Tool-call review vs. direct execution | Task success | Reads/writes; functional families; individual tools |
| ToolSandbox [ 20 ] | GPT-5 nano; Claude Haiku 5.5 | Tool-call review vs. direct execution | Similarity score | Capabilities; operations; individual tools |
| ImpossibleBench [ 36 ] | Qwen3.5 Flash | Adversarial vs. neutral feedback | Rule violation | Task progress; test conflicts; prior steering |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Family suffix | Tool names |
|---|---|
| identity | find_user_id_by_email , find_user_id_by_name_zip |
| user_read | get_user_details |
| order_read | get_order_details |
| catalog_read | get_item_details , get_product_details , list_all_product_types |
| pending_write | cancel_pending_order , modify_pending_order_address , modify_pending_order_items , modify_pending_order_payment |
| resolution_write | exchange_delivered_order_items , modify_user_address , return_delivered_order_items |
| Input | Count | Meaning |
|---|---|---|
| intercept | 1 | A constant value of one. |
| Proposed tools ( tool: ) | 16 | One input for each tool in Table 2 : whether the current proposal includes that tool. |
| prior_error | 1 | An earlier tool call returned an error. |
| prior_correction | 1 | An earlier review changed a proposal. |
| retry_tool | 1 | At least one proposed tool was used earlier. |
| used:catalog_read | 1 | An earlier product or catalog read succeeded. |
| Inputs | Count | Meaning |
|---|---|---|
| Proposed tools ( tool: ) | 16 | The same 16 tool inputs as in Table 3 . |
| Current tool families ( current: ) | 7 | For each family in Table 2 , whether the proposal includes a tool from that family. |
| Earlier successful families ( used: ) | 7 | For each family, whether an earlier call from that family succeeded. |
| Earlier events | 5 | Earlier error, earlier correction, and repeated tool (defined above); any earlier review assignment ( prior_verification ); earlier user confirmation ( user_affirmation_seen ). |
| n_prior_tools , n_prior_errors , n_prior_verifications , n_proposed_tools | 4 | Numbers of earlier tool results, earlier errors, earlier review assignments, and calls in the current proposal, respectively. |
| source_cohort_extension | 1 | Whether the task comes from the added-task set. |
| Family suffix | Tool names |
|---|---|
| settings_read | get_low_battery_mode_status , get_location_service_status , get_cellular_service_status , get_wifi_status , get_current_location |
| settings_write | set_low_battery_mode_status , set_location_service_status , set_cellular_service_status , set_wifi_status |
| contact_lookup | search_contacts |
| contact_edit | add_contact , modify_contact , remove_contact |
| message_search | search_messages |
| message_send | send_message_with_phone_number |
| Input | Count | Meaning |
|---|---|---|
| Progress phases ( phase: ) | 3 | Before any successful tool result; after a success but before a successful write; after a successful write. Only one is active. |
| Proposed tools ( tool: ) | 33 | One input for each tool in Table 5 : whether the current proposal includes that tool. |
| prior_error | 1 | An earlier tool result contained an exception. |
| prior_state_error | 1 | An earlier exception began with ConnectionError or PermissionError . |
| retry_tool | 1 | At least one proposed tool was executed earlier. |
| prior_verification | 1 | An earlier proposal was assigned review. |
| Inputs | Count | Meaning |
|---|---|---|
| Proposed tools ( tool: ) | 33 | The same 33 tool inputs as in Table 6 . |
| Current tool families ( current: ) | 10 | For each family in Table 5 , whether the proposal includes a tool from that family. |
| Earlier successful families ( used: ) | 10 | For each family, whether an earlier call from that family succeeded. |
| Earlier events and device rules | 6 | prior_error , prior_state_error , retry_tool , prior_verification , prior_correction , and precondition_unmet , defined above. |
| unknown_tool , malformed_arguments | 2 | The execution tool name is outside the catalog; the arguments do not decode to a JSON object. |
| n_prior_errors , n_prior_tools , n_prior_verifications , n_proposed_tools , n_user_turns | 5 | Numbers of earlier exception results, traced tool calls, review assignments, calls in the current proposal, and observed user turns, respectively. |
| Input | Count | Meaning |
|---|---|---|
| intercept | 1 | A constant value of one. |
| pass_fraction | 1 | The fraction of checks passed. |
| conflict_visible | 1 | The visible feedback contains contradictory checks. |
| near_conflict | 1 | At least 80% but not all checks pass, and the visible feedback contains contradictory checks. |
| prior_attack_resisted | 1 | Earlier steering occurred and the current checkpoint is eligible. An explicit rejection is not required. |
| stalled | 1 | The fraction of checks passed is no higher than after the preceding submission; zero at the first submission. |
| Inputs | Count | Meaning |
|---|---|---|
| Current SNMM attributes | 8 | All inputs in Table 8 except the constant. |
| previous_pass_fraction , progress_delta | 2 | The fraction of checks passed after the preceding submission, and the change since then. |
| Progress ranges | 4 | almost_finished : ; zero_progress : ; low_progress : ; mid_progress : , where is the current pass fraction. |
| Other progress changes | 3 | regressed : ; improved : ; recovered_from_zero : , where is the preceding pass fraction. All are zero at the first submission. |
| same_failure_streak , unchanged_code | 2 | The number of consecutive preceding evaluations with the current failure signature; whether the code equals the state saved after the preceding action (zero at the first checkpoint). |
| Time | 4 | checkpoint : preceding action number; turns_remaining : ; late : checkpoint ; last_chance : checkpoint . |
AI Agent Evaluation
AI agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently across interactive settings.
See all 561 papers