Agentic Path Maps: Exposing an Agent's Traversal Through the Garden of Forking Paths
Organizations: University of Washington Seattle, Washington, USA · Allen Institute for AI Seattle, Washington, USA
Abstract
When an AI agent executes an underspecified task, it embeds choices that may contradict unarticulated user context and preferences. However, interrupting the user for clarification on all such decisions is untenable. How might agents communicate impactful, but unexplored, paths to the user after execution? To investigate, we first distill a design space for communicating alternative paths, supporting the generation of new systems and structuring dimensions of future study. We then measure the impact of one vital design space dimension on user comprehension: the number of alternatives displayed. Through a user study (n=48) in the data analysis domain, we find that participants rarely consider alternative paths of their own volition, but AI overreliance reduces when alternative paths are visualized for them. By revealing paths the agent did not take, we demonstrate how users can better interpret results and incorporate personal context.
Figures & tables
| Dimension | Alternatives | |||||||
|---|---|---|---|---|---|---|---|---|
| Goals | Objectives | Building understanding Integrating user perspective | Improving task performance | Fostering creativity | ||||
| Structure | Outcome set | Independent decisions Hierarchical decisions | ||||||
| Initiative | Actor | Generating multiples A MI U | Selecting multiples A MI U | Requesting more A MI U | Determining granularity A MI U I | Determining quantity A MI U I | ||
| Timing | Once | Iterative sequence | Linear sequence | On demand | ||||
| Content | Quantity | All Samples | Current step | No outcomes | ||||
| Explanations | Preferred choice Path details Highlight differences Decision sensitivity Performance metrics Trend aggregations | |||||||
| Hypo. | Description | Effect | Effect CI | Estimate | Estimate CI | Post. Prob |
|---|---|---|---|---|---|---|
| H1 | Lower WoA | -0.63 | [-0.99, -0.24] | -0.28 | [-0.41, -0.15] | 0.9992 |
| H2 | Closer to median | -0.38 | [-0.76, 0.00] | -0.95 | [-1.78, -0.10] | 0.9645 |
| H3 | Larger interval | 0.09 | [-0.23, 0.39] | 0.02 | [-0.02, 0.05] | 0.7385 |
| H4 | Longer duration | 0.96 | [0.68, 1.25] | 137.94 | [103.30, 172.65] | 1.0000 |
| H5 | Higher ICL | 0.40 | [0.10, 0.70] | 0.67 | [0.24, 1.09] | 0.9935 |
| H6 | Higher ECL | 0.31 | [-0.06, 0.67] | 0.41 | [0.01, 0.80] | 0.9522 |
| Trial 1 Interface | Trial 1 Task | Trial 2 Interface | Trial 2 Task |
|---|---|---|---|
| Baseline (Agree) | Affairs | Treatment (Three) | Crofoot |
| Baseline (Agree) | Affairs | Treatment (Nine) | Crofoot |
| Baseline (Agree) | Crofoot | Treatment (Three) | Affairs |
| Baseline (Agree) | Crofoot | Treatment (Nine) | Affairs |
| Baseline (Disgree) | Affairs | Tretment (Three) | Crofoot |
| Baseline (Disgree) | Affairs | Treatment (Nine) | Crofoot |
| Expl. | Effect | Effect CI | Estimate | Estimate CI |
|---|---|---|---|---|
| Treatment: Displaying 3 vs 9 samples | ||||
| T-H1 | -0.20 | [-0.74, 0.34] | -0.08 | [-0.25, 0.10] |
| T-H2 | -0.24 | [-0.80, 0.34] | -0.45 | [-1.66, 0.77] |
| T-H3 | 0.03 | [-0.40, 0.44] | 0.01 | [-0.08, 0.09] |
| T-H4 | 0.18 | [-0.42, 0.74] | 25.65 | [-64.49, 115.45] |
| T-H5 | 0.22 | [-0.36, 0.78] | 0.35 | [-0.56, 1.24] |
Explainable Artificial Intelligence
Explainable artificial intelligence develops ways to make AI decisions understandable, testable, and useful to people.
See all 343 papers