When AI Agents Act on Our Behalf: A Taxonomy of Human-Agent Interaction Failure
Organizations: Carnegie Mellon University, Pittsburgh, USA
Abstract
AI agents write and execute code, manipulate user interfaces, and transact on their users' behalf. We ask what forms failure takes in human-agent interaction, and how those forms change as agents act with greater autonomy. From 695 documented incidents in two public repositories, we developed 12 failure modes bottom-up within three interaction stages adapted from Amershi et al.'s guidelines, coding the autonomy at which each arose. What a person can do when an agent fails depends on how much the agent completes between points of user involvement--a property fixed at deployment. Therefore, the same model on the same task can fail differently under different configurations. All 12 modes appear at both low and high autonomy, and rising autonomy changes the form each failure takes. Users need controls between full delegation and full manual work, and neither incident records nor agent benchmarks reach the interaction where these failures occur.
Figures & tables
| Failure mode | What changes as autonomy rises | Examples | |
| Low | High | ||
| During Interaction | |||
| Unclear agent role: Users not knowing that they are interacting with an AI, or what role the agent is playing. | Interaction extends role ambiguity beyond uncertainty about who or what the agent is in a direct conversation to uncertainty about the capacity in which it is acting and the authority its actions appear to carry | Meta’s AI Studio bots posed as licensed therapists, “fabricating credentials and license numbers” ( AIAAIC, 2025k ; Cole, 2025 ) . | A GPT-3 bot answered questions on Reddit for a week undetected, its comments upvoted as human contributions ( AIAAIC, 2020 ; Heaven, 2020 ) . |
| Unclear provenance: Users not being able to tell where information comes from, what was generated or retrieved, or whether it was checked by a human. | Interaction increasingly involves agents retrieving, generating, transforming, or passing along information before reaching the user, making its origin and verification status harder for users to trace. | ChatGPT produced “entire Guardian articles and bylines that [the paper] never actually published” ( AIAAIC, 2023f ; Moran, 2023 ) . | GitHub Copilot reproduces open-source code “without attribution or notifying users of license requirements” ( AIAAIC, 2022 ; Wodecki, 2022 ) . |
| Limited visibility: Users not being able to tell what the agent is currently doing, has already done, or plans to do next. | Interaction shifts from visible turn-by-turn exchanges toward multi-step and background execution, leaving fewer points at which users can observe what is happening. | A Portland woman’s Alexa recorded and sent a private conversation to her husband’s employee, unnoticed ( AIAAIC, 2018 ) . | While Claude Cowork cleaned up a desktop, the user could not see which files were being removed, until a folder holding ~15 years of family photos disappeared ( AI Incident Database, 2026a ) . |
| Insufficient control: Users not being able to easily pause, redirect, or constrain the agent. | Interaction involves agents continuing to work toward a goal without user input at every intermediate step, reducing opportunities for users to intervene before the agent’s actions take effect. | A German man’s Alexa blasted music at 2am; unreachable from his phone, neighbors called police to unplug it ( AIAAIC, 2017 ) . | Despite an explicit freeze, Replit’s agent deleted a company founder’s production database without permission ( AIAAIC, 2025b ; Tyson, 2025 ; Lee, 2025 ) . |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Stage | Failure mode | Cohen’s Kappa |
| During interaction | Unclear agent role | 0.93 |
| Unclear provenance | 0.92 | |
| Limited visibility | 0.85 | |
| Insufficient control | 0.85 | |
| Boundary violation | 0.89 | |
| When wrong | Failure goes undetected | 0.80 |
Agent Failure Analysis
Agent failure analysis studies how and why AI agents make errors, traces causes through execution, and helps detect or prevent recurrence.
See all 159 papers