cs.HCOct 3, 2026
SaveVoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents
Organizations: Tongji University Shanghai, China · Grinnell College Grinnell, Iowa, United States · Hunan University Changsha, Hunan, China · Southern University of Science and Technology Shenzhen, Guangdong, China
Abstract
People write and sketch while speaking to explain, organize, and develop content together. Inspired by these practices, we investigate how voice agents can use a canvas alongside speech in multi-turn conversations with users. We conducted a two-part formative study: an observational study of how pairs coordinated speech and boardwork, followed by a design workshop that informed a design space for speech-canvas interaction with voice agents. Building on these insights, we developed VoCa, a voice agent that coordinates speech with visual object creation, annotation, and attention guidance. A five-day deployment with 18 participants examined usability, experiences of speech-canvas interaction, patterns of use, and desired improvements. Participants' experiences highlighted opportunities for speech-canvas interaction in learning, work, and daily life, alongside challenges in coordinating what agents say and show in ways users can follow and influence. These findings inform how voice agents can use a canvas alongside speech in conversation.
Figures & tables
Figure 1. VoCa is a voice agent that coordinates speech with actions on the Canvas during multi-turn conversations. It creates and develops Visual Objects, adds Annotations, and directs attention to relevant content, allowing users and the agent to refer to, revisit, and continue developing shared content across turns. A left-to-right overview of a cat-versus-dog discussion. At upper left, the user asks for a comparison and the voice agent jointly plans speech and Canvas actions. A three-step sequence below shows the agent creating an empty comparison frame, filling it while speaking, and adding annotations. At right, the resulting Canvas contains a cat-versus-dog comparison and a cat preparation checklist; focus cues connect the spoken explanation to selected cells and notes.
Figure 2. Overview of the formative study, conducted over two days with six participants per day. Part 1 comprised an introduction, three discussion tasks completed in pairs on a shared digital whiteboard, and brief semi-structured interviews about observed whiteboard use. Part 2 introduced three application contexts and a concept video, followed by a six-person group discussion of possible agent canvas actions and desired interaction experiences. A left-to-right photographic flowchart shows five study steps: introduction, dyadic discussion, semi-structured interviews, scenarios and concept video, and group brainstorming. Brackets group the first three steps as an observational study and the last two as a design workshop. A side panel shows twelve participants across two days, with three pairs in the first part and one group of six in the second part on each day.
Figure 3. Design space for speech–canvas interaction with voice agents informed by the design workshop. D1–D2 describe canvas content and its evolution; D3–D6 describe agent actions, user interaction, temporal coordination, and agent initiative. Values within a dimension represent alternatives or capabilities that a system may combine. A diagram groups six dimensions into Canvas Content and Evolution and Interaction with the Canvas. Canvas Content includes five speech–canvas content mappings: record, extract, synthesize, restructure, and extend; visual forms include text, diagram, frame, annotation, media, and interactive components. Canvas Content Evolution includes append, revise, version, and branch. Agent Canvas Actions include attend, create, organize, annotate, revise, and remove or restore. User Interaction with the Canvas distinguishes user actions—select, navigate, manipulate, and edit—from context signals—viewport and gaze. Temporal Coordination includes canvas-first, speech-first, and synchronized speech–canvas timing, and input feedback that is absent, appears after the user's turn, or updates live. Agent Initiative includes acting on request, proactively suggesting an action for user acceptance, and proactively acting without per-action confirmation.
Figure 4. How VoCa works. The Speech–Canvas Action Planner uses the current input, System Prompt, and Dynamic Context to determine coordinated Speech and Canvas actions. A Speech-only Preface takes up the user’s input; Beats then bind Canvas actions to Speech spans. The Runtime Engine assembles Speech spans for streaming text-to-speech and updates Canvas content, resolves layout for new Canvas content, renders focus cues, and executes Camera movement. The current user input and committed Runtime results update Conversation History and the Current Canvas for subsequent turns. From left to right, a user speaks and may select or navigate the Canvas through the Voice–Canvas Interface. During speech, Live Capture incrementally updates a temporary Input Note and, when the turn ends, consolidates it into a user-attributed Turn Summary. A System Prompt and Dynamic Context, comprising Device Context, Conversation History, and the Current Canvas, inform the Speech–Canvas Action Planner. The Planner interprets the input, determines Speech and Canvas actions, and structures them into a Speech-only Preface followed by Beats containing Speech spans with optional Canvas actions. The Runtime Engine assembles Speech spans for streaming text-to-speech and executes Canvas content updates, layout, focus cues, and Camera movement. The current user input and committed Runtime results update Conversation History and the Current Canvas, which combine with Device Context to form the Dynamic Context for subsequent turns.
Figure 5. Canvas actions and Live Capture in VoCa . (1) Creating Visual Objects includes Notes, Visual Representations, and Discussion Frames. (2) Annotation adds persistent Comments, Badges, Visual Marks, and Frame Entries to existing content. (3) Attention Guidance uses temporary focus cues to point to current referents. (4) Live Capture incrementally updates an Input Note during speech and consolidates it into a user-attributed Turn Summary after the turn. Four numbered panels summarize \textsc{VoCa}'s Canvas interaction. Panel 1 shows a Note, several Visual Representation structures, and six example Discussion Frame arrangements. An inset presents six visual-form options from the formative design space, identifying text, diagram, frame, and annotation as selected for \textsc{VoCa}, while media and interactive objects remain unselected. Panel 2 shows examples of Comments, Highlights, underlines, circles, Badges, and other persistent Annotation marks. Panel 3 contrasts an ordinary diagram with the same diagram brought into visual focus. Panel 4 shows an Input Note becoming a Turn Summary.
Figure 6. Illustrative spatial arrangements that can arise through incremental placement: (a) Frame-centered expansion, (b) flow with branches, and (c) interconnected clusters. Positions and sizes are schematic. Three side-by-side layouts use blue for Notes, green for Visual Representations, and amber for Discussion Frames. Panel a has Frame 1 in the center surrounded by eight objects, all sourced from Frame 1. Objects 2 and 3 are on the right, 4 and 5 below, 6 and 7 on the left, and 8 and 9 above. Seven objects have internal-source links; Representation 4 references the whole Frame and has no line. Panel b contains no Discussion Frames. After objects 1 and 2, Notes 3 and 4 branch right and left from Representation 2, respectively. Note 5 continues below 2, followed by Representation 6, its right branch 7 and left branch 8, Note 9 below, and Representation 10. Panel c starts with Frame 1 and Representation 2 below it. Note 3 branches right from 2, and Representation 4 continues below 3. Frame 5 expands left from 4, followed by Note 6 to its left, Representation 7 below, and Note 8 below 7. Note 9 later branches right from Frame 1. Representation 10 branches right from the earlier Representation 4, followed by Note 11 below 10.
| Form | When used | Canvas result |
|---|---|---|
| Comment | Explanation, rationale, or question depends on a target. | Target-linked text with a lightweight leader. |
| Badge | Status or expressive cue should be recognizable at a glance. | Icon, number, or emoji on target. |
| Visual Mark | Existing content needs emphasis or a visible revision state. | Highlight, underline, circle, or strike on target. |
| Frame Entry | Contribution belongs in an established Frame location. | Compact text in a region, cell, axis, or plane position. |
Table 1. Persistent Annotation forms and their roles in VoCa .
Figure 7. Six moments from a multi-turn discussion with VoCa . Xiaolin and the agent move from framing three interests and project constraints, through tracing a social-feedback loop and comparing research opportunities, to selecting a persona-based direction and turning it into a proposal. Panels (a)–(f) proceed from top left to bottom right; in panel (f), Xiaolin continues the same Canvas on her phone. Six panels in two rows of three combine successive Canvas states with user and agent speech. Panels a through c show a Research Direction Frame, a habit–social-feedback loop, and a Social Feedback Design Note. Panels d through f show an opportunity representation for large-language-model social agents, the selected direction added to the Research Direction Frame, and an expanded Canvas containing the developed research proposal. The final panel also shows the proposal Notes continuing in a portrait phone view.
| Function | Model |
|---|---|
| Real-time voice (turn detection, transcript, interruption) | Qwen-Audio 3.0 Realtime Flash |
| Streaming transcription (Live Capture, manual voice) | Qwen3-ASR-Flash-Realtime |
| Live Capture | DeepSeek V4 Flash (thinking disabled; temperature 0) |
| Planner | GPT-5.6 (reasoning disabled; temperature 0) |
| Speech synthesis | Qwen-Audio 3.0 TTS Flash (word-level timestamps) |
Table 2. Models used in the VoCa prototype.
Figure 8. Post-deployment questionnaire responses from 18 participants for (a) speech–Canvas experience and (b) voice interaction. Segment widths indicate response counts. Disagreement extends left and agreement right; the middle response is centered on the dashed line. Item numbers follow Appendix C . Each item includes 18 responses. Q16 is negatively worded, so agreement indicates greater difficulty following both outputs. Two side-by-side diverging stacked bar charts show responses to six speech–Canvas experience items and six voice-interaction items. The five categories run from strongly disagree to strongly agree, with Uncertain in the middle. Each item includes eighteen responses. All eighteen respondents agree or strongly agree that spoken responses are relevant. For Canvas clarity as content grows, seven agree or strongly agree, five fall in the middle category, and six disagree or strongly disagree. For Q16, eight disagree that following both outputs is difficult, three select Uncertain, and seven agree or strongly agree.
| Purpose | Description | Sessions |
|---|---|---|
| Learning and understanding | Participants asked successive questions on topics such as mathematical proofs, language rules, philosophical ideas, and AI concepts; some also practiced speaking English or worked through example problems. The agent often drew concepts and their relations as diagrams, and participants sometimes selected earlier content when following up. | 33.1% (83) |
| Decision-making and planning | Participants made decisions, for example about gifts, travel, or study, weighing options against their preferences, budgets, or time, or developed plans such as schedules and task lists. Options and plans often appeared in Discussion Frames and were updated in later turns; Badges marked recommendations, cautions, or completed tasks. | 33.1% (83) |
| Information seeking and practical help | Participants looked up information, for example on weather or products, or sought practical help with a task at hand, such as finding a Photoshop function or dealing with food sticking to a pan. Steps, settings, or facts were mostly listed in Notes, and Annotations were often added in later turns. | 28.7% (72) |
| Emotional support | Participants talked through difficulties such as anxiety, rejection, loneliness, or pressures around study or work, seeking reassurance, help making sense of their feelings, or companionship. Notes or diagrams set out their situation or ways to cope, and Annotations added advice, in some Sessions including when to seek professional help. | 8.0% (20) |
| Entertainment and casual conversation | Participants engaged in activities such as chatting about favorite songs, role-playing with fictional personas, fortune-telling, and song-elimination games. Discussion Frames often tracked game rounds, tarot draws, or choices, and Notes held role-play lines and lists. | 6.8% (17) |
| Ideation and creative work | Participants worked on ideas and writing, such as developing product ideas, drafting and revising emails or stories, or working out how to make music or videos with AI. Written drafts appeared in Notes, with revised versions added on request, and Discussion Frames laid out workflows, in one Session dividing music-making between the participant and AI. | 6.0% (15) |
Table 3. Purposes of use identified through Session coding, ordered by number of Sessions. Values are percentage (count) of 251 Sessions. Purposes could overlap within a Session. Counts include unsuccessful attempts and Sessions that also involved system exploration; they describe recorded use, not task success.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9. P04 requested written Dutch examples and English equivalents; the agent created a Note and added Annotations. Vocabulary tables accompany a Dutch word-order diagram and a Note with three example sentences and Chinese meanings. Attached annotations provide English equivalents and word-order explanations.
Figure 10. P16 questioned arrows toward existentialism; the agent explained the relationships through speech and annotated the existing diagram. A diagram links rationalism and empiricism, Kant, Hegel, Marx, Nietzsche, and existentialism. Annotations distinguish critical responses and intellectual influences from direct inheritance.
Figure 11. P05 returned after approximately 35 hours and reported two completed activities; the agent added completion Badges to the existing Discussion Frame. A four-area task Frame organizes priorities, follow-up plans, everyday activities, and unconfirmed items. Completed Badges point to a recruitment poster task and dinner; other entries remain unmarked.
| Code | Board action | Description | Role in the discussion |
|---|---|---|---|
| B1 | Recording information | Recording selected terms, quantities, or constraints as standalone content. | Keeping information available for discussion and later reference. |
| B2 | Establishing a base structure | Drawing a basic figure, scene outline, grid, or structure around which discussion can develop. | Establishing a reference for objects, concepts, or spatial relationships. |
| B3 | Adding detail | Adding labels, attributes, explanations, or enlarged detail views to existing content. | Elaborating structures, rules, or local details. |
| B4 | Relating and grouping | Connecting or arranging elements to express relationships, sequence, membership, or grouping. | Making relationships and dependencies explicit. |
| B5 | Emphasizing and marking status | Highlighting individual or grouped content, or marking priority, certainty, completion, or another status. | Directing shared attention and marking the status of ideas or decisions. |
| B6 | Revising content | Correcting, updating, or removing information while largely retaining its overall organization. | Addressing errors, changed information, or revised constraints. |
Table 4. Board-action types observed in Part 1. Roles describe uses in the discussion, rather than measured effects or exclusive action–need mappings.
| Code | Coordination pattern | Speech–board relationship | Session coverage |
|---|---|---|---|
| C1 | Synchronous speech and boardwork | Writing or drawing content concurrently with speech about that content. | 18/18 |
| C2 | Visual updates prompted by questions or feedback | Adding, modifying, or reorganizing board content in response to a partner’s spoken question, challenge, or feedback. | 17/18 |
| C3 | Spoken reference to existing board content | Linking an utterance to a visible referent through deictic expressions coordinated with pointing. | 17/18 |
| C4 | Returning to earlier board content in conversation | Resuming discussion of earlier board content while reviewing it to elaborate, check, or reassess prior information. | 12/18 |
| C5 | Visual reorganization accompanying a discussion-stage transition | Reorganizing board content as talk moves into another stage, such as shifting from listing constraints to arranging a plan. | 10/18 |
| C6 | Visual framework first | Drawing a basic framework before a spoken explanation unfolds around it. | 6/18 |
Table 5. Speech–board coordination patterns in Part 1, ordered by decreasing session coverage.