VoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents
Authors: Yate Ge, Run Yuan, Yueran Qi, Wenjie He, Jiaqi Mo, Yangshuo Chen, Wenbin Zuo, Xiaohua Sun, +2 more
Organizations: Tongji University Shanghai, China · Grinnell College Grinnell, Iowa, United States · Hunan University Changsha, Hunan, China · Southern University of Science and Technology Shenzhen, Guangdong, China
People write and sketch while speaking to explain, organize, and develop content together. Inspired by these practices, we investigate how voice agents can use a canvas alongside speech in multi-turn conversations with users. We conducted a two-part formative study: an observational study of how pairs coordinated speech and boardwork, followed by a design workshop that informed a design space for speech-canvas interaction with voice agents. Building on these insights, we developed VoCa, a voice agent that coordinates speech with visual object creation, annotation, and attention guidance. A five-day deployment with 18 participants examined usability, experiences of speech-canvas interaction, patterns of use, and desired improvements. Participants' experiences highlighted opportunities for speech-canvas interaction in learning, work, and daily life, alongside challenges in coordinating what agents say and show in ways users can follow and influence. These findings inform how voice agents can use a canvas alongside speech in conversation.
Figures & tables
Figure 1. VoCa is a voice agent that coordinates speech with actions on the Canvas during multi-turn conversations. It creates and develops Visual Objects, adds Annotations, and directs attention to relevant content, allowing users and the agent to refer to, revisit, and continue developing shared content across turns. A left-to-right overview of a cat-versus-dog discussion. At upper left, the user asks for a comparison and the voice agent jointly plans speech and Canvas actions. A three-step sequence below shows the agent creating an empty comparison frame, filling it while speaking, and adding annotations. At right, the resulting Canvas contains a cat-versus-dog comparison and a cat preparation checklist; focus cues connect the spoken explanation to selected cells and notes.
Figure 2. Overview of the formative study, conducted over two days with six participants per day. Part 1 comprised an introduction, three discussion tasks completed in pairs on a shared digital whiteboard, and brief semi-structured interviews about observed whiteboard use. Part 2 introduced three application contexts and a concept video, followed by a six-person group discussion of possible agent canvas actions and desired interaction experiences. A left-to-right photographic flowchart shows five study steps: introduction, dyadic discussion, semi-structured interviews, scenarios and concept video, and group brainstorming. Brackets group the first three steps as an observational study and the last two as a design workshop. A side panel shows twelve participants across two days, with three pairs in the first part and one group of six in the second part on each day.
Figure 3. Design space for speech–canvas interaction with voice agents informed by the design workshop. D1–D2 describe canvas content and its evolution; D3–D6 describe agent actions, user interaction, temporal coordination, and agent initiative. Values within a dimension represent alternatives or capabilities that a system may combine. A diagram groups six dimensions into Canvas Content and Evolution and Interaction with the Canvas. Canvas Content includes five speech–canvas content mappings: record, extract, synthesize, restructure, and extend; visual forms include text, diagram, frame, annotation, media, and interactive components. Canvas Content Evolution includes append, revise, version, and branch. Agent Canvas Actions include attend, create, organize, annotate, revise, and remove or restore. User Interaction with the Canvas distinguishes user actions—select, navigate, manipulate, and edit—from context signals—viewport and gaze. Temporal Coordination includes canvas-first, speech-first, and synchronized speech–canvas timing, and input feedback that is absent, appears after the user's turn, or updates live. Agent Initiative includes acting on request, proactively suggesting an action for user acceptance, and proactively acting without per-action confirmation.
Figure 4. How VoCa works. The Speech–Canvas Action Planner uses the current input, System Prompt, and Dynamic Context to determine coordinated Speech and Canvas actions. A Speech-only Preface takes up the user’s input; Beats then bind Canvas actions to Speech spans. The Runtime Engine assembles Speech spans for streaming text-to-speech and updates Canvas content, resolves layout for new Canvas content, renders focus cues, and executes Camera movement. The current user input and committed Runtime results update Conversation History and the Current Canvas for subsequent turns. From left to right, a user speaks and may select or navigate the Canvas through the Voice–Canvas Interface. During speech, Live Capture incrementally updates a temporary Input Note and, when the turn ends, consolidates it into a user-attributed Turn Summary. A System Prompt and Dynamic Context, comprising Device Context, Conversation History, and the Current Canvas, inform the Speech–Canvas Action Planner. The Planner interprets the input, determines Speech and Canvas actions, and structures them into a Speech-only Preface followed by Beats containing Speech spans with optional Canvas actions. The Runtime Engine assembles Speech spans for streaming text-to-speech and executes Canvas content updates, layout, focus cues, and Camera movement. The current user input and committed Runtime results update Conversation History and the Current Canvas, which combine with Device Context to form the Dynamic Context for subsequent turns.
Figure 5. Canvas actions and Live Capture in VoCa . (1) Creating Visual Objects includes Notes, Visual Representations, and Discussion Frames. (2) Annotation adds persistent Comments, Badges, Visual Marks, and Frame Entries to existing content. (3) Attention Guidance uses temporary focus cues to point to current referents. (4) Live Capture incrementally updates an Input Note during speech and consolidates it into a user-attributed Turn Summary after the turn. Four numbered panels summarize \textsc{VoCa}'s Canvas interaction. Panel 1 shows a Note, several Visual Representation structures, and six example Discussion Frame arrangements. An inset presents six visual-form options from the formative design space, identifying text, diagram, frame, and annotation as selected for \textsc{VoCa}, while media and interactive objects remain unselected. Panel 2 shows examples of Comments, Highlights, underlines, circles, Badges, and other persistent Annotation marks. Panel 3 contrasts an ordinary diagram with the same diagram brought into visual focus. Panel 4 shows an Input Note becoming a Turn Summary.
Figure 6. Illustrative spatial arrangements that can arise through incremental placement: (a) Frame-centered expansion, (b) flow with branches, and (c) interconnected clusters. Positions and sizes are schematic. Three side-by-side layouts use blue for Notes, green for Visual Representations, and amber for Discussion Frames. Panel a has Frame 1 in the center surrounded by eight objects, all sourced from Frame 1. Objects 2 and 3 are on the right, 4 and 5 below, 6 and 7 on the left, and 8 and 9 above. Seven objects have internal-source links; Representation 4 references the whole Frame and has no line. Panel b contains no Discussion Frames. After objects 1 and 2, Notes 3 and 4 branch right and left from Representation 2, respectively. Note 5 continues below 2, followed by Representation 6, its right branch 7 and left branch 8, Note 9 below, and Representation 10. Panel c starts with Frame 1 and Representation 2 below it. Note 3 branches right from 2, and Representation 4 continues below 3. Frame 5 expands left from 4, followed by Note 6 to its left, Representation 7 below, and Note 8 below 7. Note 9 later branches right from Frame 1. Representation 10 branches right from the earlier Representation 4, followed by Note 11 below 10.
Form
When used
Canvas result
Comment
Explanation, rationale, or question depends on a target.
Target-linked text with a lightweight leader.
Badge
Status or expressive cue should be recognizable at a glance.
Icon, number, or emoji on target.
Visual Mark
Existing content needs emphasis or a visible revision state.
Highlight, underline, circle, or strike on target.
Frame Entry
Contribution belongs in an established Frame location.
Compact text in a region, cell, axis, or plane position.
Table 1. Persistent Annotation forms and their roles in VoCa .
Figure 7. Six moments from a multi-turn discussion with VoCa . Xiaolin and the agent move from framing three interests and project constraints, through tracing a social-feedback loop and comparing research opportunities, to selecting a persona-based direction and turning it into a proposal. Panels (a)–(f) proceed from top left to bottom right; in panel (f), Xiaolin continues the same Canvas on her phone. Six panels in two rows of three combine successive Canvas states with user and agent speech. Panels a through c show a Research Direction Frame, a habit–social-feedback loop, and a Social Feedback Design Note. Panels d through f show an opportunity representation for large-language-model social agents, the selected direction added to the Research Direction Frame, and an expanded Canvas containing the developed research proposal. The final panel also shows the proposal Notes continuing in a portrait phone view.
DeepSeek V4 Flash (thinking disabled; temperature 0)
Planner
GPT-5.6 (reasoning disabled; temperature 0)
Speech synthesis
Qwen-Audio 3.0 TTS Flash (word-level timestamps)
Table 2. Models used in the VoCa prototype.
Figure 8. Post-deployment questionnaire responses from 18 participants for (a) speech–Canvas experience and (b) voice interaction. Segment widths indicate response counts. Disagreement extends left and agreement right; the middle response is centered on the dashed line. Item numbers follow Appendix C . Each item includes 18 responses. Q16 is negatively worded, so agreement indicates greater difficulty following both outputs. Two side-by-side diverging stacked bar charts show responses to six speech–Canvas experience items and six voice-interaction items. The five categories run from strongly disagree to strongly agree, with Uncertain in the middle. Each item includes eighteen responses. All eighteen respondents agree or strongly agree that spoken responses are relevant. For Canvas clarity as content grows, seven agree or strongly agree, five fall in the middle category, and six disagree or strongly disagree. For Q16, eight disagree that following both outputs is difficult, three select Uncertain, and seven agree or strongly agree.
Purpose
Description
Sessions
Learning and understanding
Participants asked successive questions on topics such as mathematical proofs, language rules, philosophical ideas, and AI concepts; some also practiced speaking English or worked through example problems. The agent often drew concepts and their relations as diagrams, and participants sometimes selected earlier content when following up.
33.1% (83)
Decision-making and planning
Participants made decisions, for example about gifts, travel, or study, weighing options against their preferences, budgets, or time, or developed plans such as schedules and task lists. Options and plans often appeared in Discussion Frames and were updated in later turns; Badges marked recommendations, cautions, or completed tasks.
33.1% (83)
Information seeking and practical help
Participants looked up information, for example on weather or products, or sought practical help with a task at hand, such as finding a Photoshop function or dealing with food sticking to a pan. Steps, settings, or facts were mostly listed in Notes, and Annotations were often added in later turns.
28.7% (72)
Emotional support
Participants talked through difficulties such as anxiety, rejection, loneliness, or pressures around study or work, seeking reassurance, help making sense of their feelings, or companionship. Notes or diagrams set out their situation or ways to cope, and Annotations added advice, in some Sessions including when to seek professional help.
8.0% (20)
Entertainment and casual conversation
Participants engaged in activities such as chatting about favorite songs, role-playing with fictional personas, fortune-telling, and song-elimination games. Discussion Frames often tracked game rounds, tarot draws, or choices, and Notes held role-play lines and lists.
6.8% (17)
Ideation and creative work
Participants worked on ideas and writing, such as developing product ideas, drafting and revising emails or stories, or working out how to make music or videos with AI. Written drafts appeared in Notes, with revised versions added on request, and Discussion Frames laid out workflows, in one Session dividing music-making between the participant and AI.
6.0% (15)
Table 3. Purposes of use identified through Session coding, ordered by number of Sessions. Values are percentage (count) of 251 Sessions. Purposes could overlap within a Session. Counts include unsuccessful attempts and Sessions that also involved system exploration; they describe recorded use, not task success.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9. P04 requested written Dutch examples and English equivalents; the agent created a Note and added Annotations. Vocabulary tables accompany a Dutch word-order diagram and a Note with three example sentences and Chinese meanings. Attached annotations provide English equivalents and word-order explanations.
Figure 10. P16 questioned arrows toward existentialism; the agent explained the relationships through speech and annotated the existing diagram. A diagram links rationalism and empiricism, Kant, Hegel, Marx, Nietzsche, and existentialism. Annotations distinguish critical responses and intellectual influences from direct inheritance.
Figure 11. P05 returned after approximately 35 hours and reported two completed activities; the agent added completion Badges to the existing Discussion Frame. A four-area task Frame organizes priorities, follow-up plans, everyday activities, and unconfirmed items. Completed Badges point to a recruitment poster task and dinner; other entries remain unmarked.
Code
Board action
Description
Role in the discussion
B1
Recording information
Recording selected terms, quantities, or constraints as standalone content.
Keeping information available for discussion and later reference.
B2
Establishing a base structure
Drawing a basic figure, scene outline, grid, or structure around which discussion can develop.
Establishing a reference for objects, concepts, or spatial relationships.
B3
Adding detail
Adding labels, attributes, explanations, or enlarged detail views to existing content.
Elaborating structures, rules, or local details.
B4
Relating and grouping
Connecting or arranging elements to express relationships, sequence, membership, or grouping.
Making relationships and dependencies explicit.
B5
Emphasizing and marking status
Highlighting individual or grouped content, or marking priority, certainty, completion, or another status.
Directing shared attention and marking the status of ideas or decisions.
B6
Revising content
Correcting, updating, or removing information while largely retaining its overall organization.
Addressing errors, changed information, or revised constraints.
Appendix
Table 4. Board-action types observed in Part 1. Roles describe uses in the discussion, rather than measured effects or exclusive action–need mappings.
Code
Coordination pattern
Speech–board relationship
Session coverage
C1
Synchronous speech and boardwork
Writing or drawing content concurrently with speech about that content.
18/18
C2
Visual updates prompted by questions or feedback
Adding, modifying, or reorganizing board content in response to a partner’s spoken question, challenge, or feedback.
17/18
C3
Spoken reference to existing board content
Linking an utterance to a visible referent through deictic expressions coordinated with pointing.
17/18
C4
Returning to earlier board content in conversation
Resuming discussion of earlier board content while reviewing it to elaborate, check, or reassess prior information.
12/18
C5
Visual reorganization accompanying a discussion-stage transition
Reorganizing board content as talk moves into another stage, such as shifting from listing constraints to arranging a plan.
10/18
C6
Visual framework first
Drawing a basic framework before a spoken explanation unfolds around it.
6/18
Appendix
Table 5. Speech–board coordination patterns in Part 1, ordered by decreasing session coverage.
Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular interface for interacting with such systems. Speech input introduces an additional failure point: transcription errors can alter task-critical entities, constraints, or targets before the agent begins reasoning, while conventional ASR metrics do not directly measure whether the information required for successful execution has been preserved. We introduce Talk2Agent, a benchmark for evaluating how effectively voice interfaces convey human-spoken instructions to LLM-based computer-use agents. Talk2Agent builds human-spoken versions of tasks from WildClawBench and OSWorld and evaluates a range of voice interfaces, including dedicated ASR models, audio-capable LLMs, contextual biasing, and LLM-based ontology repair. Because repeatedly executing long-horizon computer-use tasks is costly and stochastic, we further propose an execution-free, task-conditioned evaluation framework that projects the original task grader onto prompt-addressable intentions and measures how much task-relevant information is retained after the voice interface. On WildClawBench, Talk2Agent's execution-free native projection provides a practical, execution-grounded measure of voice-interface quality, correlating with downstream task completion and improving Pearson correlation by 0.246 over WER/CER on 32 hours of real human speech.
Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1
Department of Electronic Engineering, Tsinghua University · University of Cambridge
Agentic speech systems must exchange more than text, including speaker identity, timing, phonetic information, and behavioral annotations. Yet these signals are often produced in incompatible tool-specific formats, making agent-to-agent handoff fragile. We present OneVoice, a lightweight, JSON-native intermediate representation that provides a shared semantic structure for speech pipelines. OneVoice organizes heterogeneous speech evidence into validated session records with stable identifiers, layered transcripts, explicit timing relationships, and provenance information. We evaluate OneVoice in two complementary multi-agent workflows covering event aggregation and acoustic-phonetic temporal linking across three language models. Compared with implicit agent-defined handoffs, OneVoice substantially improves aggregation reliability and the preservation of temporal relationships, demonstrating the value of an explicit speech-specific representation for agent communication.
Vipul Charugundla, Dancheng Liu, Jinjun Xiong
University at Buffalo · University of Texas at San Antonio
Speech agents are reactive and dyadic: they speak when spoken to, and to one person at a time. We ask what it takes for a speech agent to instead take part in a conversation among several people and speak up only when it can help. We introduce Jarvis, a real-time proactive speech agent that audibly participates in multi-party human conversations. Grounded in a document shared beforehand, Jarvis follows the discussion and intervenes when the group misses or misstates a fact and does not correct itself within a few turns. We make three contributions: a problem setting based on epistemic breakdowns that makes proactive intervention measurable, realized as CHI-180-proactive, a synthetic multi-party dataset seeded with known gaps, errors, and self-corrections; a proactive backbone that harnesses a small, open-weight model with deterministic checks and grounds every claim in a source sentence; and interaction techniques for taking the floor in live speech and showing the cited evidence on screen. On CHI-180-proactive, Jarvis is correct on most events it addresses and stays silent 97% of the time when the group resolves an issue itself. A live study with 23 participants confirms these trends with real-time interventions.
Seunghyun Oh, Hirotaka Hiraki, Shuyue Stella Li +2
Paul G. Allen School of Computer Science and Engineering, University of Washington, WA, USA · Graduate School of Interdisciplinary Information Studies, The University of Tokyo, Bunkyo, Tokyo, Japan