Agentic Path Maps: Exposing an Agent's Traversal Through the Garden of Forking Paths
Authors: Madeleine Grunde-McLaughlin, Hafsah Usman, Daniel Mitroshkov, Emily Reif, Nino Migineishvili, Lakshya Chauhan. Kevin Yu, Jeffrey Heer, Daniel S. Weld
Organizations: University of Washington Seattle, Washington, USA · Allen Institute for AI Seattle, Washington, USA
When an AI agent executes an underspecified task, it embeds choices that may contradict unarticulated user context and preferences. However, interrupting the user for clarification on all such decisions is untenable. How might agents communicate impactful, but unexplored, paths to the user after execution? To investigate, we first distill a design space for communicating alternative paths, supporting the generation of new systems and structuring dimensions of future study. We then measure the impact of one vital design space dimension on user comprehension: the number of alternatives displayed. Through a user study (n=48) in the data analysis domain, we find that participants rarely consider alternative paths of their own volition, but AI overreliance reduces when alternative paths are visualized for them. By revealing paths the agent did not take, we demonstrate how users can better interpret results and incorporate personal context.
Figures & tables
Figure 1. Agentic workflows make assumptions about user’s goals to complete underspecified tasks. Although the agent’s found output may be valid under the task constraints, it may not be valid given the users’ context and preferences. In this paper, we contribute Agentic Path Maps to communicate untaken alternative paths to users. We first distill a space of designs for generating and studying Agentic Path Maps. We then manipulate one dimension, the quantity of paths shown, in an empirical user study. We find that participants overrelied less and were more cognitively engaged when shown alternative paths. Overview diagram linking an agent's decision tree, the Path Map design space, and the user study. On the left is a ticket-purchasing task which branches through decisions about state residency, day of the week, and youth versus adult tickets, ending in prices. A highlighted path marks the agent's assumptions and leads to a task-valid output. The other branches mark choices that contradict user preferences and overlooked outputs valid to the user. In the center are designer goals, showing objectives and structure underneath. Arrows then point to a Path Map interface paired with a branching diagram and analysis results. On the right are two rows that pair study interfaces with illustrations of users' beliefs. The standard interface shows one pathway and a thought bubble containing one output. The Path Map interface shows three or nine pathways and a thought bubble containing multiple outputs.
Dimension
Alternatives
Goals
Objectives
Building understanding Integrating user perspective
Table 1. Design space dimensions. Our design space for Agentic Path Maps has eight dimensions in four categories: goals, initiative, content, and display. The space draws from ten related domains, so grey highlighting indicates options specifically relevant for Agentic Path Maps. Black outlines situate our empirical study’s stimulus within the design space, with the dotted outline around the quantity dimension we manipulate. In the actor row, A is agent, MI is mixed-initiative, U is user, and I is inherent. A design space table with category labels at the left, dimensions in the middle, and alternatives across each row. Goals groups Objectives and Structure, Initiative groups Actor and Timing, Content groups Quantity and Explanations, and Display groups Visualizations and Interactions. Solid outlines select Building understanding, Hierarchical decisions, Once, Path details, Decision sensitivity, specification curve, and Elicit details. In the Actor row, outlines select agent for Generating multiples and inherent for Determining granularity and Determining quantity. A shared dotted outline encloses All and Samples in the Quantity row. Gray backgrounds mark relevant alternatives. In the Actor row, though, these backgrounds apply to individual actor labels.
Figure 2. Belief elicitation figure. Participants reviewed the analysis question [A] and input what they thought was the “most likely relationship” by moving the yellow bar [B]. They then used a slider [C] to adjust a distribution of grey bars indicating the “set of plausible alternatives.” The right panel contains the exact values and an interpretive sentence [D]. A belief-elicitation screen with a wide plot panel on the left and a narrower summary panel on the right. The horizontal axis shows average marginal effect ranging from negative to positive two with a dotted reference line at 0. The selected estimate is 0.139, emphasized with yellow, with possible values distributed on both sides of the origin. The summary reports a 95 percent confidence interval ranging from negative 0.135 to (positive) 0.412.
Figure 3. Baseline interface displaying one analysis. The interface shows the [A] analysis question. [B] found output, [C] what analysis decisions were made, [D] specifications and rationales, as well as [E] code. A single-analysis interface with a result card above Jupyter notebook-style code cells. The question at the top, marked A, asks whether having children decreases engagement in extramarital affairs. The result, marked B, is an average marginal effect of 1.189 with a 95 percent confidence interval from 0.71 to 1.67. Below it, chips marked with the label C identify Children binary as the independent variable, Affairs as the dependent variable, covariates, and a Poisson regression as the method. Beneath the method chip a label marked D describes the model and gives an italicized rationale. The notebook below, marked with the label E, shows Python code and the resulting Poisson regression summary.
Figure 4. Treatment interface displaying multiple analyses. This Agentic Path Maps display of multiple analysis alternatives uses a specification curve dependency visualization . Every row indicates a defensible analysis choice, and every column an individual analysis with associated code and an output. The boxes indicate which combination of alternative analysis choices was used for that column’s analysis. The display shows the [A] analysis question, [B] found outputs, [C] what analysis decisions were made, [D] specifications and rationales, and [E] code. A multiple-analysis interface with a plot and choice matrix on the left and explanation cards on the right. The question at the top right is marked A. A specification curve marked B orders nine average marginal effects from negative 0.163 to 1.189, with vertical confidence intervals. Three rows below give estimates and interval bounds. In the matrix marked C, columns align with the plotted analyses and rows list choices for variables, covariates, and statistical methods. Squares indicate the choices used in each analysis. The rightmost column is highlighted, linking its choices to cards marked D that provide specifications and italicized rationales. A notebook cell marked E sits below and displays the selected analysis's Python code.
Figure 5. Posterior draws. We report the posterior draws from our Bayesian linear mixed models for each of the preregistered hypotheses and sub-condition analyses. Showing nine analyses compared to three resulted in changes in the same direction as showing multiple analyses compared to one, with the exception of Germane Cognitive Load (H7). 21 posterior density plots are arranged in three rows and seven columns. The first row is labeled Preregistered with the subheader One vs. Multiple, the second with Multiple analyses with a Three vs. Nine subheader, and the third with One analysis with an Agree vs Disagree subheader, from top to bottom respectively. Columns correspond to hypotheses H1 through H7. Rows show the main posterior distributions, treatment estimates, and baseline estimates. Each plot shows the distribution of posterior draws along an estimate axis and includes a dashed vertical marker at zero. Axis ranges differ across hypotheses. In the Preregistered row, H1 and H2 are centered below zero, H3 is centered near zero, and H4 through H7 are centered above zero; H4 has a much larger numerical scale than the other hypotheses.
Hypo.
Description
Effect
Effect CI
Estimate
Estimate CI
Post. Prob
H1
Lower WoA
-0.63
[-0.99, -0.24]
-0.28
[-0.41, -0.15]
0.9992
H2
Closer to median
-0.38
[-0.76, 0.00]
-0.95
[-1.78, -0.10]
0.9645
H3
Larger interval
0.09
[-0.23, 0.39]
0.02
[-0.02, 0.05]
0.7385
H4
Longer duration
0.96
[0.68, 1.25]
137.94
[103.30, 172.65]
1.0000
H5
Higher ICL
0.40
[0.10, 0.70]
0.67
[0.24, 1.09]
0.9935
H6
Higher ECL
0.31
[-0.06, 0.67]
0.41
[0.01, 0.80]
0.9522
Table 2. Results for preregistered hypotheses. We find a large or very large effect with a high posterior probability (Post. Prob) that participants place a lower Weight of Advice on extreme values [H1] and end with beliefs closer to the multiverse median [H2] when shown alternative analysis pathways, but report higher Intrinsic [H5] and Extraneous Cognitive Load [H6]. Participants interacted for longer [H4] and reported higher beneficial Germane Cognitive Load with a small effect [H7]. Seven hypothesis rows with seven columns: hypothesis (abbreviated as "hypo."), Description, standardized effect (abbreviated as Effect), the Effect Confidence interval, estimate in the outcome's original units, its confidence interval, and posterior probability of the hypothesized direction. Standardized effect intervals exclude 0 for H1, H4, and H5. H2's upper bound rounds to zero. Estimate intervals exclude zero for H1, H2, H4, H5, and H6. H3 and H7 both have intervals spanning zero.
Trial 1 Interface
Trial 1 Task
Trial 2 Interface
Trial 2 Task
Baseline (Agree)
Affairs
Treatment (Three)
Crofoot
Baseline (Agree)
Affairs
Treatment (Nine)
Crofoot
Baseline (Agree)
Crofoot
Treatment (Three)
Affairs
Baseline (Agree)
Crofoot
Treatment (Nine)
Affairs
Baseline (Disgree)
Affairs
Tretment (Three)
Crofoot
Baseline (Disgree)
Affairs
Treatment (Nine)
Crofoot
Table 3. Participant assignments. We recruit 48 participants such that each of the rows is assigned to three participants. Baseline (Agree) and Baseline (Disagree) are sub-conditions displaying an analysis agreeing or disagreeing with the user’s prior. Treatment (Three) and Treatment (Nine) are sub-conditions displaying three or nine analyses. Sixteen assignment rows with four columns, grouped into two pairs. Each pair gives the interface and task for one trial. Every row pairs one baseline interface with one treatment interface and assigns Affairs in one trial and Crofoot in the other. The rows cover all combinations of baseline agreement or disagreement, three or nine treatment analyses, interface order, and task order. The first eight rows begin with baseline, and the remaining eight begin with treatment.
Expl.
Effect
Effect CI
Estimate
Estimate CI
Treatment: Displaying 3 vs 9 samples
T-H1
-0.20
[-0.74, 0.34]
-0.08
[-0.25, 0.10]
T-H2
-0.24
[-0.80, 0.34]
-0.45
[-1.66, 0.77]
T-H3
0.03
[-0.40, 0.44]
0.01
[-0.08, 0.09]
T-H4
0.18
[-0.42, 0.74]
25.65
[-64.49, 115.45]
T-H5
0.22
[-0.36, 0.78]
0.35
[-0.56, 1.24]
Table 4. Sub-condition results. We test the differences between showing three and nine analyses (T-) and showing one outcome that agrees or disagrees with their prior (B-). For the treatment sub-conditions, most effects are in the same direction as the main hypotheses but of a smaller magnitude, except for Germane Cognitive Load (T-H7). Five columns show the hypothesis identifier, standardized effect, its interval, the estimate in the outcome's original units, and its interval. Fourteen rows form two blocks: seven treatment comparisons labeled T-H1 through T-H7, followed by seven baseline comparisons labeled B-H1 through B-H7. Every reported effect interval and estimate interval spans zero. Treatment point estimates are negative for H1, H2, and H7 and positive for H3 through H6. Baseline point estimates are negative except for H5, which is positive, and H6, whose estimate rounds to zero.
Figure 6. Participant belief shifts. Participants’ posterior beliefs (orange) shift from their prior beliefs (blue) differently based on which displayed value(s) they review (Outputs lines). When shown one output, the posterior shifts closer to the displayed extreme value. When shown multiple analysis paths, the posteriors place a lower Weight of Advice on the extremes and lie closer to the median value. Eight panels compare distributions of participant belief estimates before and after reviewing analysis results. Rows represent the Affairs and Crofoot tasks. Columns represent the positive-output baseline, negative-output baseline, three-analysis treatment, and nine-analysis treatment. Each panel contains an upper box plot of initial estimates, a lower box plot of final estimates, as well as reference tick marks beneath them indicating the displayed analysis results. The shared horizontal axis shows belief estimates as z-scores, with labeled ticks from minus 4 to 12 and a vertical line at zero. Boxes span the middle half of estimates, with an internal median line. Whiskers extend to the most extreme observations within one and a half interquartile ranges of the box. Isolated points show the observations past the whiskers. Final estimates are more tightly clustered in most panels, especially the Crofoot positive-output baseline and three-analysis treatment.
Figure 7. Cognitive Load questionnaire ( Krieglstein et al., 2023 ) . After viewing alternative paths, participants reported higher Intrinsic and Extraneous Cognitive Load. They also reported higher Germane Cognitive Load, a beneficial cognitive engagement associated with learning. Stacked bar charts of cognitive load questionnaire responses. Three columns show Intrinsic, Extraneous, and Germane Cognitive Load, with five items per column. Each item has an upper bar for One Path and a lower bar for Multiple Paths. Their segment widths show response counts on a common horizontal scale. Ratings run from 1, not at all applicable, to 9, fully applicable, ordered from left to right. These ratings are shaded progressively darker. Lower ratings are labeled better for intrinsic and extraneous load; higher ratings are better for germane load.
Figure 8. Participant preference for the Path Maps display. Participants rated each interface by preference such that the ratings added to 100. Some participants highly preferred each display, with the majority preferring the Path Maps display. A histogram of participants' preference for the Path Maps interface. The horizontal axis gives the percentage of each participant's preference allocated to the Path Maps interface, with the remaining percentage being allocated to the single-analysis interface. An allocation of 50 percent indicates equal preference. The vertical axis gives the number of participants, with label ticks ranging from 0 to 16. The graph comprises ten adjacent bins. The tallest bar, near 70 percent, contains sixteen participants. The neighboring bar to its right contains nine. A smaller concentration lies between about 20 and 40 percent, and few participants are near either endpoint.
AI systems have long been expected to interact with users, answering questions, generating content, and continuing (social) conversations. Agentic AI, however, breaks from this expectation, as its primary objective is workflow execution on behalf of the users. If a system becomes more agentic, do users need less interaction with the system? Our answer is: less routine back-and-forth, but more communication for oversight and explanation, as agentic AI proactively acts, not just responds. Grounded in a communication perspective, we discuss how users perceive the communicative roles of AI systems (whether as the source of actions or merely a channel), and how this can shape trust. Because agentic AI can play multiple communicative roles, it can complicate this source perception and introduce potential risks. To address this, we propose three types of explanations that agentic AI needs to incorporate (action-process, uncertainty, and coordination), and suggest that customization affordances that allow users to decide when and which explanations they see may be key to preserving human agency as AI autonomy increases.
Eunchae Jang, S. Shyam Sundar
Media Effects Research Laboratory, Pennsylvania State University, Pennsylvania, USA · Center for Socially Responsible AI, Pennsylvania State University, Pennsylvania, USA
To enable human oversight, agentic AI systems often provide a trace of reasoning and action steps. Designing traces to have an informative, but not overwhelming, level of detail remains a critical challenge. In three user studies on a Computer User Agent, we investigate the utility of basic action traces for verification, explore three alternatives via design probes, and test a novel interface's impact on error finding in question-answering tasks. As expected, we find that current practices are cumbersome, limiting their efficacy. Conversely, our proposed design reduced the time participants spent finding errors. However, although participants reported higher levels of confidence in their decisions, their final accuracy was not meaningfully improved. To this end, our study surfaces challenges for human verification of agentic systems, including managing built-in assumptions, users' subjective and changing correctness criteria, and the shortcomings, yet importance, of communicating the agent's process.
Madeleine Grunde-McLaughlin, Hussein Mozannar, Maya Murad +3
University of Washington, United States · Microsoft Research, United States
When automating plan generation for a real-world sequential decision problem, the goal is often not to replace the human planner, but to facilitate an iterative reasoning and elicitation process, where the human's role is to guide the AI planner according to their preferences and expertise. In this context, explanations that respond to users' questions are crucial to improve their understanding of potential solutions and increase their trust in the system. To enable natural interaction with such a system, we present a multi-agent Large Language Model (LLM) architecture that is agnostic to the explanation framework and enables user- and context-dependent interactive explanations. We also describe an instantiation of this framework for goal-conflict explanations, which we use to conduct a user study comparing the LLM-powered interaction with a baseline template-based explanation interface.
Guilhem Fouilhé, Rebecca Eifler, Antonin Poché +2
IRIT, Toulouse, France · LAAS-CNRS, Toulouse, France · IRT Saint Exupery, Toulouse, France +2