Overseeing Agents Without Constant Oversight: Challenges and Opportunities

Madeleine Grunde-McLaughlin; Hussein Mozannar; Maya Murad; Jingya Chen; Saleema Amershi; Adam Fourney

cs.HCFeb 18, 2026

Overseeing Agents Without Constant Oversight: Challenges and Opportunities

Authors: Madeleine Grunde-McLaughlin, Hussein Mozannar, Maya Murad, Jingya Chen, Saleema Amershi, Adam Fourney

Organizations: University of Washington, United States · Microsoft Research, United States

Abstract

To enable human oversight, agentic AI systems often provide a trace of reasoning and action steps. Designing traces to have an informative, but not overwhelming, level of detail remains a critical challenge. In three user studies on a Computer User Agent, we investigate the utility of basic action traces for verification, explore three alternatives via design probes, and test a novel interface's impact on error finding in question-answering tasks. As expected, we find that current practices are cumbersome, limiting their efficacy. Conversely, our proposed design reduced the time participants spent finding errors. However, although participants reported higher levels of confidence in their decisions, their final accuracy was not meaningfully improved. To this end, our study surfaces challenges for human verification of agentic systems, including managing built-in assumptions, users' subjective and changing correctness criteria, and the shortcomings, yet importance, of communicating the agent's process.

Figures & tables

Figure 1. Through three user studies, we investigate how the presentation of an agentic workflow’s actions affects human verification. We first find that participants miss small but impactful errors with the current verbose display listing agent steps with interleaved summaries. With three design probes, we next find that focusing on the steps provides useful context but can justify overreliance. Finally, with our novel interface that explicitly defines the task’s requirements and the agent’s assumptions, participants found errors faster but also reported increased confidence with no meaningful effect on accuracy. There are two rows of items. One is titled "Human oversight aims to improve accuracy and reduce cost". Under that heading, there are two boxes, one of which says (HIL) Agent Workflow) that has a Task'' input and an Output'' output. An arrow connects this box to another titled Review Mechanism''. This arrow says actions reported to user''. There is an arrow going from the Review Mechanism'' box to the (HIL) Agent Workflow'' box that says user takeaways inform''. The second row of items is titled

We inspect to improve the review mechanism.'' This section is split into three columns, each of which represent a user study and have an n=12 indication. First is titled

Step summaries'' and has the takeaway

Existing methods are verbose and hide errors.'' It shows a diagram of small action steps that are split into groups of actions that are summarized in text. The second column says

Design probes'' with the takeaway A reasonable process justifies overreliance.'' It has three items along an arrow going from Process-Oriented'' to Outcome'' with three methods Flowchart'', Citation'' and Spec'' along that arrow. The third column is titled Control'' with the takeaway Find errors faster but more false confidence.'' It has two parts, the Baseline: Step summaries approach with a final summary'' and Treatment: Spec design with improved linking and detail acess''.

Figure 2. Magentic-UI interface. Our formative study illuminates verification practices with the HIL CUA Magentic-UI ( Mozannar et al., 2025 ) . Users first input a task [A], then use various features to co-plan, co-task, and adapt to errors as the system runs. To verify, they can first review the plan summary that decomposes the task into proposed plan steps [B]. Each plan step is then executed, and each action is described in the trace for transparency [C]. Collapsing these plan steps [D] shows summaries of each group of actions. While the task runs, users may watch the live view as the agent interacts with the browser [E]. This visual trace is stored in a series of screenshots as the agent progresses [F]. Clicking the icon next to a particular action [C] links to the associated screenshot [F] to improve navigation. Users can open tabs between different tasks to multitask [G] and interact with chat [H]. A “Final Answer” summary appears upon completion (not visible). An interface display of Magentic UI. The top left contains a small icon that, when clicked, opens different tabs next to the Magentic-UI logo. The left half of the screen mostly contains first the task, then a given plan of steps in distinct boxes, then a series of bullet points that show certain actions, such as

I typed <url> into the browser address bar''. Below is a chat input box. On the right side, there are two tabs: Screenshots and Live View. The screenshots tab is selected and shows one screenshot. There is an indicator that it is showing 3/14 screenshots, with arrow buttons to the left and right. The bottom of the interface stats

Magentic-UI can make mistakes. Please monitor its work and intervene if necessary.''

Figure 3. Three design probes. Our design probes focus on the process (Flowchart), outcome (Specification), or both (Citation). This figure illustrates each design for the task “How much would I save by getting annual passes for my family (2 adults, 1 kid age 5, 1 kid age 2) for the Seattle Aquarium, compared to buying daily tickets, if we visit 4 times in a year?” Additional examples are in Supplementary Materials. Three columns, each of which has a method. The Flowchart column has three rows of nodes. The first node says ``Go to Seattle Aquarium site'' which connects to two nodes in the same row ``Find daily prices'' and ``Find membership prices''. Each of those connects to a node in the final row of ``Calculate the difference''. The second Citation column says the following, with [brackets] indicating underlined portions ``Based on the official Seattle Aquarium ticket page (https://www.seattleaquarium.org/), the family plan membership costs [\$249] total. The daily ticket price is [\$47.95 per adult] and estimated [\$32 for children 4-12 and free for children under 4], for [\$127.90] total per visit. If visiting 4 times in a year, the membership would save you [\$262.60].'' The final column is the specification, which includes the requirements 1) ``Daily prices for two adults, one 5yo and one 2yo found'', 2) ``Membership plan for two adults, one 5yo and one 2yo found'', and 3) ``Difference between membership and daily prices calculated'' and the assumptions ``Estimated youth price of \$32 is a typical rate (not shown on site)'', ``Your family is not WA residents'', ``Peak price used (changes daily)'', ``No guests needed on membership pass'', and ``Aquarium website is a reliable source''. — Figure 3. Three design probes. Our design probes focus on the process (Flowchart), outcome (Specification), or both (Citation). This figure illustrates each design for the task “How much would I save by getting annual passes for my family (2 adults, 1 kid age 5, 1 kid age 2) for the Seattle Aquarium, compared to buying daily tickets, if we visit 4 times in a year?” Additional examples are in Supplementary Materials. Three columns, each of which has a method. The Flowchart column has three rows of nodes. The first node says `Go to Seattle Aquarium site'' which connects to two nodes in the same row` Find daily prices'' and `Find membership prices''. Each of those connects to a node in the final row of` Calculate the difference''. The second Citation column says the following, with [brackets] indicating underlined portions `Based on the official Seattle Aquarium ticket page (https://www.seattleaquarium.org/), the family plan membership costs [\$249] total. The daily ticket price is [\$47.95 per adult] and estimated [\$32 for children 4-12 and free for children under 4], for [\$127.90] total per visit. If visiting 4 times in a year, the membership would save you [\$262.60].'' The final column is the specification, which includes the requirements 1)` Daily prices for two adults, one 5yo and one 2yo found'', 2) `Membership plan for two adults, one 5yo and one 2yo found'', and 3)` Difference between membership and daily prices calculated'' and the assumptions `Estimated youth price of \$32 is a typical rate (not shown on site)'',` Your family is not WA residents'', `Peak price used (changes daily)'',` No guests needed on membership pass'', and ``Aquarium website is a reliable source''.

Figure 4. Design probe study interface. The interface displays the given task [A], task output [B], and one of the three design probes [C]. Clicking on the design probe surfaces the relevant screenshot(s) [D] that have highlighted annotations of the relevant information [E]. The original Magentic-UI trace is available for further review [F]. Participants then answer the questions “Is this result correct?” and “Why or why not?” below [G]. This figure shows an interface display. The top row says ``Flowchart Page - aquarium'' with the task ``How much will I save by getting annual passes for my family (2 adults, 1 kid age 5, 1 kid age 2) for the Seattle Aquarium, compared to buying daily tickets, if we visit 4 times in a year?'' The left side shows the outcome ``By purchasing the annual pass instead of daily tickets for 4 visits, your family would save \$511.60 − \$249 = \$262.60 annually'' as well as the flowchart from the design probes figure. The right side shows a screenshot with a yellow highlighted annotation over the membership price used. There is an indication that this is ``Image 1 of 2'' as well as text saying ``Session: 70 Image: 11'' — Figure 4. Design probe study interface. The interface displays the given task [A], task output [B], and one of the three design probes [C]. Clicking on the design probe surfaces the relevant screenshot(s) [D] that have highlighted annotations of the relevant information [E]. The original Magentic-UI trace is available for further review [F]. Participants then answer the questions “Is this result correct?” and “Why or why not?” below [G]. This figure shows an interface display. The top row says `Flowchart Page - aquarium'' with the task` How much will I save by getting annual passes for my family (2 adults, 1 kid age 5, 1 kid age 2) for the Seattle Aquarium, compared to buying daily tickets, if we visit 4 times in a year?'' The left side shows the outcome `By purchasing the annual pass instead of daily tickets for 4 visits, your family would save \$511.60 − \$249 = \$262.60 annually'' as well as the flowchart from the design probes figure. The right side shows a screenshot with a yellow highlighted annotation over the membership price used. There is an indication that this is` Image 1 of 2'' as well as text saying ``Session: 70 Image: 11''

GT Ans.	Method	Accuracy (%)	Duration (s)
No	Flowchart	39.13	427.38
	Citation	52.17	293.97
	Spec	65.22	542.15
Yes	Flowchart	91.67	319.85
	Citation	83.33	573.87
	Spec	100.00	491.65

Table 1. Design probe accuracy and duration performance. The Spec method shows the highest accuracy but also the highest duration. GT Ans. indicates if the output is correct, with “All” including all trials. This exploratory investigation had two incorrect questions and one correct question per method.

Method	Preference	Validation
Flowchart	2.75	3.25
Citation	2.17	2.50
Spec	2.08	2.25
Magentic-UI	3.00	2.00

Table 2. Mean rank preference for design probes and Magentic-UI. Participants ranked methods by Preference (P1-P12) and by their perception of which best supported Validation (P5-P12). For both conditions, Spec was the best and Flowchart the worst ranked, although there was much variation between participants (lower is better). The full Magentic-UI interface was less preferred but considered more likely to support validation.

Figure 5. Controlled study interface. The interface provides the task [A] and the output to verify [B]. It integrates the Magentic-UI trace directly into the interface, adding highlights to both the text [C] and screenshots [D]. The screenshots can be paged through, with the ability to magnify for detail and to open a new tab with the associated website for further review [E]. For additional path context, there is a brief summary [F] provided in the Citation format. For guiding review, there are requirements and assumptions [G], with additional explanatory context and icons indicating if requirements are completed, unknown, or contradicted. Each requirement and assumption refers to annotation(s) in the text and image trace with underlying icons [H] and tooltips for easier scanning [I]. For the study, there is a 5-minute timer [J] and a series of questions [K]. This figure shows an interface display. The top bar has a timer countdown and a question ``Is this result correct'' with a dropdown selection and a submit button. There is a pink box that contains the task text at the top left, then a white box inset that contains the trace of actions, then a final answer, then a small paragraph stating ``I first navigated to the [Seattle Aquarium website]. I then checked for [annual membership rates] and [daily rates]. Finally, I [calculated the difference]. Below this pink box, the interface is split into two columns titled ``Requirements'' and ``Assumptions. Requirements has two icons at the top, one is green with a checkmark, and the number 2, and the other is yellow with a question mark and the number 1. Next to assumptions is an orange number 5. Below each are boxes containing a requirement or assumption. Each box has the text (e.g., ``Estimated youth price of \$32 is a typical rate (not shown on site)'') and several icons below, formatted either as an image icon or a document icon. Hovered icons show a black tooltip, in this case ``2yo''. Selected icons turn a magenta color. Below the icons is a brief explanatory description (e.g., ``Youth ages 4-12 receive reduced pricing, but the exact numbers are not available. \$32 is an estimate based on similar museums.'' The right side has a header titled ``Screenshot Review'' indicating image 14 of 14. This header has a magnification icon, a browser icon, and arrow keys. The image itself has yellow highlighting. The text in the trace also has yellow highlighting. — Figure 5. Controlled study interface. The interface provides the task [A] and the output to verify [B]. It integrates the Magentic-UI trace directly into the interface, adding highlights to both the text [C] and screenshots [D]. The screenshots can be paged through, with the ability to magnify for detail and to open a new tab with the associated website for further review [E]. For additional path context, there is a brief summary [F] provided in the Citation format. For guiding review, there are requirements and assumptions [G], with additional explanatory context and icons indicating if requirements are completed, unknown, or contradicted. Each requirement and assumption refers to annotation(s) in the text and image trace with underlying icons [H] and tooltips for easier scanning [I]. For the study, there is a 5-minute timer [J] and a series of questions [K]. This figure shows an interface display. The top bar has a timer countdown and a question `Is this result correct'' with a dropdown selection and a submit button. There is a pink box that contains the task text at the top left, then a white box inset that contains the trace of actions, then a final answer, then a small paragraph stating` I first navigated to the [Seattle Aquarium website]. I then checked for [annual membership rates] and [daily rates]. Finally, I [calculated the difference]. Below this pink box, the interface is split into two columns titled `Requirements'' and` Assumptions. Requirements has two icons at the top, one is green with a checkmark, and the number 2, and the other is yellow with a question mark and the number 1. Next to assumptions is an orange number 5. Below each are boxes containing a requirement or assumption. Each box has the text (e.g., `Estimated youth price of \$32 is a typical rate (not shown on site)'') and several icons below, formatted either as an image icon or a document icon. Hovered icons show a black tooltip, in this case` 2yo''. Selected icons turn a magenta color. Below the icons is a brief explanatory description (e.g., `Youth ages 4-12 receive reduced pricing, but the exact numbers are not available. \$32 is an estimate based on similar museums.'' The right side has a header titled` Screenshot Review'' indicating image 14 of 14. This header has a magnification icon, a browser icon, and arrow keys. The image itself has yellow highlighting. The text in the trace also has yellow highlighting.

Figure 6. Control study results. The chart shows the Hedges’

g

effect size and the 95% confidence interval. The Duration and Confidence charts show subsets such that “No-Yes” displays when the ground truth answer is “No” and the user answers “Yes”. Overall, we find no effect on accuracy (0.18), a small effect decreasing duration (-0.29), and a medium effect increasing confidence (0.56). The effect on duration is medium-sized when the user correctly finds an error (“No-No”: -0.65). The largest confidence effect occurs when the answer is incorrect, but the user thinks it is correct (“No-Yes”: 0.85). Exact values are in Table 3 . Three dotted vertical lines show the cutoffs for small, medium, and large effect sizes. Three graphs. Values are from the main results table. Each has Hedges' g on the x-axis. First graph titled All'' has y-axis values Accuracy'' Duration'' and Confidence''. Second graph titled Duration'' and third graph titled Confidence'' have y axis values Yes-Yes'', No-No'', and ``No-Yes''. Each graph has three points indicating the effect size with horizontal bars spanning the 95% confidence interval. There are light grey bars for small, medium, and large effect sizes at 0.2, 0.5, and 0.8.

				Accuracy (%)					Duration (s)					Confidence (1-7)
GT	User	Cohen’s	Cond.	Mean	STD	Eff.	CI L	CI H	Mean	STD	Eff.	CI L	CI H	Mean	STD	Eff.	CI L	CI H
No	No	$d_{ind}$	Base.	100.00	0.00	–	–	–	270.79	62.93	-0.65	-1.54	0.12	3.54	1.85	0.37	-0.41	1.23
No	No	$d_{ind}$	Treat.	100.00	0.00	–	–	–	229.63	60.49	-0.65	-1.54	0.12	4.23	1.74	0.37	-0.41	1.23
No	Yes	$d_{ind}$	Base.	0.00	0.00	–	–	–	251.85	93.01	-0.20	-1.12	0.67	3.64	1.63	0.85	0.01	1.88
No	Yes	$d_{ind}$	Treat.	0.00	0.00	–	–	–	234.48	69.99	-0.20	-1.12	0.67	4.82	0.98	0.85	0.01	1.88
Yes	No	$d_{ind}$	Base.	0.00	0.00	–	–	–	151.31	98.17	–	–	–	1.50	0.71	–	–	–

Table 3. Quantitative results between the Baseline and Treatment conditions. GT and User refer to the ground truth and user answer to the question “Is this output correct?” respectively. “All” includes both “Yes” and “No” responses. Effects compare the Baseline (Base.) and Treatment (Treat.) conditions. We report the type of Cohen’s

d

calculated (Cohen’s) and all effect sizes (Eff.) have an unbiasing correction applied to become Hedges’

g

. We include the lower (CI L) and upper (CI U) confidence intervals. Dashes occur where the effect cannot be calculated. Confidence is measured on a 7-point Likert scale.

	Task 1	Task 2	Task 3
Correct	8	1	2
Acceptable	N/A	4	N/A
Incorrect	1	5	10
Incompete – timing	2	4	0
Incomplete – technical	3	1	1

Table 4. Outcome instances from the formative study.

Key	Task	Gound Truth Ans.	Presented Ans.	Insuff.
FLOWCHART
yellowstone	What are hikes in Yellowstone that have been recommended by at least three different people with kids and are highly rated on TripAdvisor (an average from 4.5/5 from at least 50 reviews)?	Mystic Falls Trail Fountain Paint Pot Trout Lake Trail Lone Star Geyser	1. Fairy Falls Trail 2. Trout Lake Trail	Yes
dog	The dog genome was first mapped in 2004 and has been updated Several times since. What is the link to the files that were most recent in May 2020?	https://ftp.ensembl.org/pub/ release-100/fasta/ canis_lupus_familiaris/dna/	https://www.ncbi.nlm.nih.gov /datasets/genome/ GCF_000002285.5/	No
andew	On which social media platform does Andrew Ng have the most followers?	Linked-In	Linked-In	Yes
CITATION
karting	Which paintball places in Cologne, Germany are within a 10 minute walk from a karting track?	Adrenalinpark Köln	None available	Yes

Table 5. Tasks from the design probe study. The Insuff. column delineates if the answer “Insufficient” counts as correct.

Figure 7. Flowchart tasks. Keywords relate to Table 5 . Three columns for three flowchart tasks. Each has nodes with action items, some of which are highlighted in blue. Outcome nodes have a light blue background.

Figure 8. Citation tasks. Keywords relate to Table 5 . Three columns, each of which shows the citation description for the three citation tasks.

Figure 9. Specification tasks. Keywords relate to Table 5 . Three columns, each of which shows the specification for each of the specification tasks.

Participant	Rank 1 (high)	Rank 2	Rank 3	Rank 4
P6	Flowchart	Spec	Citation	Magentic-UI
P3	Flowchart	Spec	Magentic-UI	Citation
P1	Spec	Flowchart	Citation	Magentic-UI
P5	Spec	Flowchart	Citation	Magentic-UI
P4	Spec	Citation	Flowchart	Magentic-UI
P8	Spec	Citation	Flowchart	Magentic-UI

Table 6. Overall preference rankings from the design probes study. Manually ordered by similarity.

Participant	Rank 1 (high)	Rank 2	Rank 3	Rank 4
P7	Citation	Magentic-UI	Spec	Flowchart
P10	Citation	Magentic-UI	Flowchart	Spec
P12	Magentic-UI	Citation	Flowchart	Spec
P5	Magentic-UI	Spec	Flowchart	Citation
P6	Magentic-UI	Spec	Flowchart	Citation
P8	Spec	Flowchart	Citation	Magentic-UI

Table 7. Expected validation support rankings from the design probes study. Manually ordered by similarity. Only asked to P5-P12

Task	Ground Truth Ans.	Presented Ans.
SET 1
Which Fidelity international emerging markets equity mutual fund with $0 transaction fees had the lowest percentage increase between May 2019 to May 2024?	FPADX	FEDDX
What is the highest rated (according to IMDB) Isabelle Adjani feature film that is less than 2 hours and is available on Vudu (now called Fandango at Home) to buy or rent?	Nosferatu the Vampyre	No film fit the criteria.
How much will I save by getting annual passes for my family (2 adults, 1 kid age 5, 1 kid age 2) for the Seattle Children Museum, compared to buying daily tickets, if we visit 4 times in a year?	$61	$61
Which members of Fubo’s Management Team joined the company during the same year Fubo’s IPO happened?	Gina DiGioa	Gina DiGoia
SET 2

Table 8. Tasks from the control study.

Value	Descriptor
1	Not at all confident
2	Slightly confident
3	Somewhat confident
4	Moderately confident
5	Quite confident
6	Very confident

Table 9. Confidence scale. The scale values participants selected from for the question “How confident are you in your previous answer?”

Figure 10. Control study baseline interface. The content is near-identical Magentic-UI’s, but more visibly similar to the treatment condition. The display shows the task [A] and final answer [B], with a text description of actions [C] and a summarized final answer [D]. The screenshots are available [E], with one addition of a magnification feature [F]. Participants have five minutes [G] to determine if the output is correct [H]. This figure shows an interface display. The top bar has a timer countdown and a question ``Is this result correct'' with a dropdown selection and a submit button. There is a pink box that contains the task text at the top left, then a white box inset that contains the trace of actions, and then a final answer is briefly stated. Below this pink box is a white box containing a ``Final Answer'' summary with more detail. The right side has a ``Screenshots'' header indicating we are on screenshot ``14 of 14''. There is an icon of a magnifying glass and arrow keys. Below this header is the screenshot itself. — Figure 10. Control study baseline interface. The content is near-identical Magentic-UI’s, but more visibly similar to the treatment condition. The display shows the task [A] and final answer [B], with a text description of actions [C] and a summarized final answer [D]. The screenshots are available [E], with one addition of a magnification feature [F]. Participants have five minutes [G] to determine if the output is correct [H]. This figure shows an interface display. The top bar has a timer countdown and a question `Is this result correct'' with a dropdown selection and a submit button. There is a pink box that contains the task text at the top left, then a white box inset that contains the trace of actions, and then a final answer is briefly stated. Below this pink box is a white box containing a` Final Answer'' summary with more detail. The right side has a `Screenshots'' header indicating we are on screenshot` 14 of 14''. There is an icon of a magnifying glass and arrow keys. Below this header is the screenshot itself.

Human Oversight of AI

Human oversight of AI studies how people supervise, verify, and control AI decisions, especially as systems become more autonomous and consequential.

See all 180 papers

May 2, 2026

Less Interaction But More Explanation: A Communication Perspective on Agentic AI Interfaces

AI systems have long been expected to interact with users, answering questions, generating content, and continuing (social) conversations. Agentic AI, however, breaks from this expectation, as its primary objective is workflow execution on behalf of the users. If a system becomes more agentic, do users need less interaction with the system? Our answer is: less routine back-and-forth, but more communication for oversight and explanation, as agentic AI proactively acts, not just responds. Grounded in a communication perspective, we discuss how users perceive the communicative roles of AI systems (whether as the source of actions or merely a channel), and how this can shape trust. Because agentic AI can play multiple communicative roles, it can complicate this source perception and introduce potential risks. To address this, we propose three types of explanations that agentic AI needs to incorporate (action-process, uncertainty, and coordination), and suggest that customization affordances that allow users to decide when and which explanations they see may be key to preserving human agency as AI autonomy increases.

Eunchae Jang, S. Shyam Sundar

Media Effects Research Laboratory, Pennsylvania State University, Pennsylvania, USA · Center for Socially Responsible AI, Pennsylvania State University, Pennsylvania, USA

Explainable Artificial Intelligence Human-AI Interaction Trust in AI Human Oversight of AI Agentic AI

Ask PDF

Aug 28, 2026

When Agents Act Unwatched: The Reduced-Supervision Paradox in Agentic AI

Agentic AI is sold on a simple promise: the system keeps acting when the user stops watching. That promise creates an accountability inversion. As stepwise supervision recedes, verification does not disappear; it moves into the runtime infrastructure that defines authority, records action, interrupts execution, checks outcomes, and supports repair. We call this the reduced-supervision paradox. Using a 63-artifact audit, we examine its public visibility across 46 research papers and 17 engineering, documentation, security, and governance sources. We find that agents' action surfaces are far easier to reconstruct than the mechanisms needed to answer for their actions. Tool mediation and monitoring traces were clearly visible in 40 and 37 artifacts, whereas checkpoint placement was clearly visible in 6, validator independence in 4, recovery in 2, and contestability in 1. Three action paths show why this imbalance matters. A repository path can preserve rich diffs after a consequential change. A browser path can cross organizational boundaries faster than permissions travel. A service path can follow policy while leaving affected people without recourse. We argue that observability can become a substitute for accountability when it shifts verification onto users after meaningful intervention is no longer possible. Our action-path diagnostic instead asks whether a delegated action remains connected to authority, evidence, interruption, independent judgment, recovery, and challenge. The claim is deliberately bounded to public visibility; it does not establish the prevalence or effectiveness of undisclosed controls. We contribute an action-level account that recasts the harness from a technical wrapper into accountability infrastructure.

Hanjing Shi, Dominic DiFranzo

Department of Computer Science and Engineering Lehigh University, Bethlehem, PA, USA

AI Accountability AI Agent Governance AI Agent Monitoring AI Agent Auditing

Ask PDF

Mar 25, 2026

Willful Disobedience: Automatically Detecting Failures in Agentic Traces

AI agents are increasingly embedded in real software systems, where they execute multi-step workflows through multi-turn dialogue, tool invocations, and intermediate decisions. These long execution histories, called agentic traces, make validation difficult. Outcome-only benchmarks can miss critical procedural failures, such as incorrect workflow routing, unsafe tool usage, or violations of prompt-specified rules. This paper presents AgentPex, an AI-powered tool designed to systematically evaluate agentic traces. AgentPex extracts behavioral rules from agent prompts and system instructions, then uses these specifications to automatically evaluate traces for compliance. We evaluate AgentPex on 424 traces from

τ^2

-bench across models in telecom, retail, and airline customer service. Our results show that AgentPex distinguishes agent behavior across models and surfaces specification violations that are not captured by outcome-only scoring. It also provides fine-grained analysis by domain and metric, enabling developers to understand agent strengths and weaknesses at scale. The source code of AgentPex is available at https://github.com/microsoft/agentpex.

Reshabh K Sharma, Shraddha Barke, Benjamin Zorn

University of Washington Seattle, Washington, USA · Microsoft Research Redmond, Washington, USA

AI Agent Evaluation AI Agent Auditing

Ask PDF

Abstract

Human Oversight of AI

Explore similar work

Less Interaction But More Explanation: A Communication Perspective on Agentic AI Interfaces

When Agents Act Unwatched: The Reduced-Supervision Paradox in Agentic AI

Willful Disobedience: Automatically Detecting Failures in Agentic Traces