cs.HCMar 2, 2026
Save"When to Hand Off, When to Work Together": Understanding Concurrent Human-Agent Interaction in Shared Co-Creative Workspaces
Organizations: School of Computing, KAIST, Daejeon, Republic of Korea · Department of Computer Science, Purdue University, West Lafayette, Indiana, United States · Computer Science and Engineering, University of Michigan, Ann Arbor, Michigan, United States · Midjourney, San Francisco, California, United States · Mphora.ai, Monte Sereno, California, United States · SkillBench, Santa Barbara, USA
Abstract
Shared co-creative workspaces allow users to contribute while agents execute. Yet how users adjust their participation as their work and the agent's execution shape one another remains less understood. We conducted two design probe studies with professional designers. In Study 1 (N=10), participants identified opportunities to contribute by observing the agent's progress, but avoided intervention when the agent misinterpreted their edits. Therefore, we developed CLEO, a second probe that tracks user actions and selectively updates its plan. In Study 2 (N=12), we analyzed 255 interaction turns and stimulated recall interviews, identifying five action categories, ten codes, six triggers, and four enabling factors. Participants engaged in concurrent interaction during 33.73% of turns, expressing newly recognized design ideas or preferences through edits and changing the division of work by completing pending subtasks or independently developing copies of unfinished outputs. We present a decision model and five design implications for concurrent human-agent collaboration.
Figures & tables
Figure 1. An example of collaborative friction in human-agent collaboration within a shared workspace. Two stacked canvas sketches show conflicting edits to a contact form. At the top, the agent reports setting a two-column layout and updating its content, while the user considers a three-column version. A blue downward arrow shows the user changing the form to three columns in the bottom sketch. The agent then reports that the update conflicts with its original plan and says it will fix it. A green upward arrow shows the agent reverting to the two-column layout. Blue and green cursors identify the user and agent; red spiky marks highlight the conflict around the modified form.
Figure 2. Interaction flow of the first probe system. (a) The user sends a voice message (b) while selecting canvas elements for context. (c) The agent generates an action plan based on the voice transcription with selections and iterates through ReAct cycles. (d-f) Each iteration’s progress is reflected immediately on the canvas as tools execute during the act stage. Panels a and b show a user requesting color variations by voice while selecting a portfolio header on the canvas. Panel c shows a plan generated from this input followed by a repeating cycle: reason to select tools, act by executing canvas operations, and obtain feedback on plan completion. Panels d through f show the visible results with an agent character and progress messages. The first iteration duplicates the header, the second changes the color of the first header, and the final iteration presents three color variations.
Figure 3. Flexible co-creation scenario with Cleo . (a) The user invokes Cleo by calling its name. (b) While Cleo is acting on the canvas, the user can (c) work concurrently on the same task, or (d) intervene directly with additional instructions at any time. Panel a shows the user calling CLEO and requesting a portfolio header. Panel b shows the first iteration's result: a frame and a Portfolio title. Two alternative branches illustrate the second iteration. In panel c, the user changes the title to blue while the agent works; the agent then adds menu items and decorative lines in blue. In panel d, the user calls CLEO and requests a serif font; the resulting header contains the menu, decorative lines, and updated serif text. User characters and blue cursors mark the user's contributions, while rabbit characters and speech bubbles show the agent's responses.
Figure 4. Agent structure of Cleo . Three modules have been updated from the first probe pipeline (Appendix A.4 ): User Change Detection Module, Attribution Change Module, and Plan Update Module. The top pipeline runs from User Message through User Change Detection, Chat History and Canvas Context, the Plan Module, the ReAct Agent, and the Message Module to a Final Message. The expanded ReAct flow below first asks whether a design action is required. If not, it goes directly to the Message Module. If yes, tools execute, producing interim messages and tool-call results. These results and canvas context feed the Attribution Change Module, which connects to the Feedback Module and branches to the Plan Update Module. A plan-fulfillment check sends completed work to the Message Module or repeats the loop, up to ten iterations. Pink boxes highlight User Change Detection, Attribution Change, and Plan Update.
Figure 5. How Cleo achieves context awareness. (a) The user invokes the agent. (b) It recalls past observations and detects user updates against the current state. (c) It monitors concurrent user actions while executing its plan, prioritizes interventions, and updates its plan as needed. The figure demonstrates how the system achieves context awareness through three components. In (a), the user invokes the agent by issuing a voice command to add icons to the footer of a portfolio canvas. In (b), the User Change Detection Module compares the agent's previous observation — a simple footer with email and contact fields — against the current canvas state, detecting that the user has updated the footer content and added contact information. In (c), while the agent is actively creating icons on the canvas, the Attribution Change Module detects that the user concurrently changed the layout alignment to vertical. This intervention triggers the Plan Update Module, allowing the agent to revise its plan accordingly.
| Category | Code | Definition | % of Turns |
| (H) Hands-off | Full delegation | Completely disengaging from the agent’s work to focus on other tasks. | 69.41% |
| (O) Observational | Observational monitoring | Watching the agent’s execution without intervening. | 70.20% |
| (T) Terminating | Execution termination | Stopping the agent’s work before completion. | 7.06% |
| (D) Directive | Instruction-based steering | Providing verbal guidance to adjust the agent’s approach. | 26.67% |
| Switching tasks | Assigning a new task unrelated to the current execution. | ||
| (C) Concurrent | Intermediate result appropriation | Copying and using the agent’s partial outputs while it continues working. | 33.73% |
Table 1. Five categories and ten codes identified through qualitative analysis of user action patterns. See Figure 14 in the appendix for illustrations of each interaction. The table defines ten user action codes in five categories and reports the percentage of 255 turns containing each category. Hands-off (H), 69.41%: full delegation, in which users disengage from the agent's work to focus on other tasks. Observational (O), 70.20%: monitoring execution without intervening. Terminating (T), 7.06%: stopping execution before completion. Directive (D), 26.67%: instruction-based steering or switching the agent to an unrelated task. Concurrent (C), 33.73%: intermediate result appropriation, copying partial outputs for use elsewhere; artifact takeover, duplicating work-in-progress for independent editing; in-situ co-editing, editing the same subtask and artifact alongside the agent; opportunistic takeover, completing a pending subtask while the agent handles another; and demonstration-based steering, showing desired changes through direct editing. Multiple categories can occur within a turn, so their percentages sum to more than 100%.
| Trigger | (C) | (D) | (T) | % of Turns |
|---|---|---|---|---|
| Idea Spark from Agent’s Work-in-Progress | 4.31% | |||
| Need for Early Outcome Visibility | 14.51% | |||
| Readiness for Fine-grained Detailing | 9.80% | |||
| Misaligned Task Interpretation | 10.59% | |||
| Execution Quality Drop | 15.29% | |||
| Emerging New Task for Agent | 15.29% |
Table 2. Six triggers that prompt user actions across the \themetag concurrentC , \themetag directD , and \themetag terminateT user action categories. Rows list six triggers, with checkmarks indicating the action categories associated with each and a percentage of all 255 turns in which the trigger appeared at least once. C means Concurrent, D Directive, and T Terminating. Idea Spark from Agent's Work-in-Progress: C, 4.31%. Need for Early Outcome Visibility: C, 14.51%. Readiness for Fine-grained Detailing: C, 9.80%. Misaligned Task Interpretation: C, D, and T, 10.59%. Execution Quality Drop: C, D, and T, 15.29%. Emerging New Task for Agent: D and T, 15.29%.
| Enabling Factor (Category) | Value (Code) | (C) | (D) | (T) | (O) | (H) |
| Has mental model | ||||||
| (a) Mental Model of Agent’s Task Capability | No mental model | |||||
| User task significantly more important | ||||||
| User task moderately more important / Similar importance | ||||||
| (b) Task Importance: User vs. Agent | No user task | |||||
| Verbal explanation easier | ||||||
Table 3. Four categories and eleven codes of enabling factors across action categories. The table groups eleven enabling-factor codes into four categories. Checkmarks associate each code with Concurrent (C), Directive (D), Terminating (T), Observational (O), or Hands-off (H) actions. Mental model of the agent's task capability: having a mental model is associated with all five action categories; having no mental model is associated with O. Importance of the user's task relative to the agent's task: a significantly more important user task is associated with H; a moderately more important or similarly important user task with D, T, and O; and no user task with C, T, and O. Preferred intervention modality: verbal explanation being easier is associated with D, direct manipulation being easier with C, and uncertainty about how to intervene with T. Expected agent response: expecting success with direct collaboration is associated with C, expecting the agent to understand verbal guidance with D, and believing the agent is incapable of the task with T. For the modality and response categories, O and H are marked as no intervention.
Figure 6. Decision model of human-agent co-creative collaboration A decision flowchart on the left is paired with four loop insets on the right. Decision a asks whether the user has a mental model of the agent's task capability. No leads to observation and an end node; yes proceeds to decision b, comparing task importance to select hands-off interaction or observation. During observation, decision c checks for a trigger. Without one, observation continues; with one, enabling factors at d lead to Concurrent interaction, a green box labeled Direct interaction, No intervention, or Terminating intervention. Decision e checks completion: completed work ends, while unfinished work goes to the task-change check at f. A changed task returns to the mental-model check; an unchanged task returns to the importance check. Termination also ends the flow. Insets g through j highlight six loops: full delegation, continuous observation, three intervention variants for concurrent, directive, or no intervention, and task redirection.
Figure 7. Proposed adaptive process visibility (DI1). The same ongoing execution appears as individual operations for close oversight (left) or grouped milestones for parallel work (right). Both views track progress without requiring step-by-step approval. Two views of the same portfolio task contrast levels of process detail. The left view lists Create frame, Align title, and Adjust spacing as separate completed operations, followed by Add palette. The right view nests the same three operations inside one Layout ready milestone, followed by Explore palette. Matching blue backgrounds and repeated operation icons show the grouping; peach backgrounds connect the palette work across the views. The labels read Track individual operations and Track milestone progress. Checkmarks indicate completed work, not requests for user approval. Named user cursors and rabbit agent characters identify the collaborators.
Figure 8. Proposed agent responses (DI2): apply edits at the intended scope, preserve parallel work, and update remaining tasks after user contributions. Dashed blue arrows show the agent’s inference of collaboration needs. Three rows show concurrent user actions and proposed agent responses. The top row contrasts two example scopes for the same spacing edit to the third of three cards. This card only leads to Preserve this edit, with only the third card changed. Use for similar cards leads to Apply to similar cards, with all three cards changed. The middle row shows artifact takeover: the user refines a copy while the agent continues the original, with separate blue and green workflows and named collaborators. The bottom row shows opportunistic takeover: the user completes Add icons while the agent works on Build layout. Add icons is then marked Done by user above the remaining plan, which lists Build layout and Choose colors. Dashed blue arrows connect user actions to responses based on the agent's inference of collaboration needs. A legend distinguishes user actions and workflow, agent workflow, and inferred signal.
Figure 9. Proposed anticipatory pause (DI3): viewport focus and cursor approach could prompt the agent to pause between operations and expose an editable state. The dashed blue arrow represents the agent’s inference of possible user intervention. The left canvas shows a blue viewport outline around the agent's output and a solid blue arrow tracing the user's cursor from a faded previous position toward the artifact. Labels read Combine cues, Possible intervention, and Viewport focus + cursor approach. A dashed blue arrow between canvases represents the agent's inference of possible user intervention from these combined cues. The right canvas shows an editable intermediate artifact and a workflow strip where a solid green arrow connects the completed Align title operation to a paused state before the next step. The rabbit agent says it will pause so the user can make adjustments. A legend distinguishes user action, agent workflow, and inferred signal.
Figure 10. Proposed spatial controls (DI4): placing the agent near an element requests shared attention (left); moving it elsewhere requests independent work (right). Speech bubbles express the intended meaning of each placement. Two canvases show a user dragging a rabbit agent character. On the left, the user addresses CLEO and asks it to focus here, moving the agent near a selected portfolio element. On the right, the user asks for independent work elsewhere and moves the agent from the user's work area to a separate exploration area. Solid blue arrows connect faded prior positions to new positions and are identified as user actions in the legend. User characters identify the speakers, and named hand cursors show the drag actions.
Figure 11. Proposed proactive assistance (DI5). Dashed blue arrows show how the agent could infer collaboration preferences from past user behavior to inform a color-exploration offer that the user can accept or defer. On the left, three cards show palette observation, delegated color work, and agent positioning across interactions. Solid blue arrows depict user actions, while a gray timeline indicates their accumulation over time. Dashed blue arrows connect the history to a collaboration-context box stating that the user often delegates color exploration, then connect this inferred context to a new design. On the right, the rabbit agent offers to explore color variations before a user request. Try it and Later buttons let the user decide whether to proceed. The legend distinguishes user action from inferred signal.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12. Agent structure of the first probe system. Upon receiving user input, the Plan Module generates an action plan. The ReAct Agent then iterates through a cycle of reasoning with tool selection, execution, summary, and feedback evaluation until the plan is fulfilled or the maximum of 10 iterations is reached. The Message Module generates the final response, reflecting both the agent’s actions and the user’s original message, and delivers it to the user. Two levels of a flow diagram show the first probe's architecture. At the top, the User Message enters Chat History and Canvas Context, followed by the Plan Module, ReAct Agent, Message Module, and Final Message. The expanded ReAct loop below first asks whether a design action is required. If not, execution proceeds directly to the Message Module. If an action is required, the agent selects from the Tool List, produces an interim message, and obtains Tool Call Results. These results pass through the Summary Module and then the Feedback Module, which also receives Canvas Context. A second decision asks whether the plan is fulfilled. Yes leads to the Message Module; No follows a dashed return arrow to the first decision, with a maximum of ten iterations. The Message Module generates the final response reflecting the agent's actions and the original user request.
Figure 13. Visualization of interaction logs. (a) The overall task process is segmented into periods when the agent is activated and deactivated, using the timestamps of agent task initiation (triggered by user requests) and task completion. (b) Each segment is visualized with labeled interaction events. (c) Example of a visualized segment during agent activation. (d) Example of a visualized segment during agent deactivation. Four panels show how interaction logs are segmented and visualized. In (a), orange markers identify agent task starts triggered by user requests, and blue markers identify task completions along the whole task timeline. Dashed guides separate the timeline into agent-activated intervals between these markers and the remaining agent-deactivated intervals. In (b), the legend maps purple speech bubbles to initial requests or additional feedback, blue rectangles to user action descriptions, green rectangles to agent action descriptions, and pause and play icons to pause and resume events. In (c), an activated segment begins with a user request. Agent action blocks above the timeline overlap user action blocks below it. A pause icon, an additional request, and a resume icon precede further agent and user actions. In (d), a deactivated segment contains three successive user action blocks and no agent action blocks.
Figure 14. Visual representation of user action patterns. Ten schematic action patterns are grouped into five categories. Beret-wearing characters represent users, rabbits represent agents, and gray shapes represent work or subtasks. Hands-off shows Full delegation: the user attends to another task while the agent works. Observational shows Observational monitoring: the user watches without intervening. Terminating shows Execution termination: the user stops the agent before completion. Directive contains Instruction-based steering, where verbal guidance changes the agent's approach, and Switching tasks, where the user assigns an unrelated new task. Concurrent contains five distinct patterns. Intermediate result appropriation shows the user copying part of the agent's output into their own work while the agent continues. Artifact takeover shows a separate copy of the agent's work-in-progress that the user edits independently. In-situ co-editing places user and agent together on the same subtask and artifact. Opportunistic takeover shows the user completing pending Subtask C while the agent handles other subtasks. Demonstration-based steering shows a direct edit labeled Demo feeding into the agent's subsequent corrected behavior.
| Combination | Count | % |
|---|---|---|
| (H) | 76 | 29.80% |
| (O) | 36 | 14.12% |
| (O) + (H) | 28 | 10.98% |
| (O) + (C) | 14 | 5.49% |
| (O) + (D) | 7 | 2.75% |
| (O) + (T) | 3 | 1.18% |
Table 4. Turn-level distribution of action category combinations across 255 turns. The table lists 16 combinations of action categories across 255 turns, with a count and percentage for each combination. H means Hands-off, O Observational, C Concurrent, D Directive, and T Terminating. A combination identifies the categories present within a turn, not their order. Hands-off only is the most frequent combination; every other listed combination includes Observational. The rows are: Hands-off only, 76 turns (29.80%); Observational only, 36 (14.12%); Observational and Hands-off, 28 (10.98%); Observational and Concurrent, 14 (5.49%); Observational and Directive, 7 (2.75%); Observational and Terminating, 3 (1.18%); Observational, Concurrent, and Hands-off, 22 (8.63%); Observational, Directive, and Hands-off, 14 (5.49%); Observational, Concurrent, and Directive, 14 (5.49%); Observational, Concurrent, and Terminating, 2 (0.78%); Observational, Hands-off, and Terminating, 3 (1.18%); Observational, Concurrent, Hands-off, and Terminating, 3 (1.18%); Observational, Concurrent, Directive, and Hands-off, 26 (10.20%); Observational, Directive, Hands-off, and Terminating, 2 (0.78%); Observational, Concurrent, Directive, and Terminating, 2 (0.78%); and all five categories, 3 (1.18%).