Equal participation in co-located discussion is important for effective collaboration, yet people often hold back when they anticipate negative interpersonal or professional consequences, especially when raising a point requires voicing it themselves. We present SecondVoice, a mixed-reality system that enables people to speak up through an embodied virtual proxy. By separating what is said from who says it, SecondVoice brings hesitant points into the live spoken discussion without putting the speaker on the spot. Using a private overlay, users specify their intent through a structured specification process rather than composing a full utterance. The system reformulates the input and voices it into the conversation through the proxy. We characterize a design space of participation channels under social risk. In a preliminary within-subject study (N = 16), we compare the complete SecondVoice system with an anonymous text-board channel across two group discussion tasks. Half of participants reported using SecondVoice for a point they did not say aloud, compared with 18.8% for the text board. Proxy-delivered points entered the spoken floor and were followed by multi-turn group engagement, which we did not observe after text-board posts. Participants described the channel as situationally valuable but identified tradeoffs around timing, ownership, and trust in reformulation.
Figures & tables
Figure 1. A group ranks survival items after a shipwreck. Everyone agrees food belongs at the top; one participant thinks the mirror is more important but does not want to be the one to say so (A). The system privately surfaces the active topic with stance options (B). She indicates disagreement, selects a communicative move, and confirms a point through the structured specification interface (C). A shared embodied proxy raises her point on the spoken floor without naming the author (D). Four-panel illustration of the SecondVoice system, depicting a Lost at Sea survival-ranking task. Left panel, labeled Hesitation: four people wearing MR headsets sit around a table; one participant speaks aloud that food should be near the top and the mirror seems useless, while another participant has a thought bubble considering that the mirror could help get rescued but everyone is focused on food. Center panel, labeled Private Specification: SecondVoice surfaces a pulse asking Food should be top?'' with three stance buttons; below, the participant sees the specification interface with four communicative move options (Agree, Challenge, Question, Suggest) with Challenge selected, and a preview reading The mirror could be used to signal rescuers.'' The proxy figure, rendered in blue, sits at the table. Right panel, labeled Public Delivery: the proxy figure raises its hand and speaks the reformulated point about the mirror reflecting sunlight to help signal rescuers; another participant responds ``Wait, that's true\ldots''\del{ indicating the group has taken up the point}.
Channel
Direct exposure
Specificity
Spoken floor
Direct speech
High
High
Yes
Anonymous text board / backchannel
Lower
Medium–High
No
Reactions / polls
Lower
Low
No
Facilitation / nudging
Medium
Low
Indirectly
Autonomous agent / surrogate
Lower for user
Variable
Yes
SecondVoice / bounded proxy
Lower for author
Medium–High
Yes
Table 1. Participation channels under social risk. Existing channels tend to offer either lower exposure outside the spoken floor or spoken-floor entry at the cost of direct self-voicing. The underserved region is a channel that remains expressive, lowers direct exposure, and still enters the spoken floor. A low-fidelity comparison table positioning participation channels by exposure, specificity, and whether they enter the spoken floor.
Figure 2. The SecondVoice interaction flow. (A) The system monitors the discussion and may initiate contact through a pulse or nudge. (B) System-elicited input: a pulse surfaces the current discussion topic with stance options (agree, question, disagree); a nudge asks whether the participant wants to share a thought. (C) User-initiated input: the participant opens the structured specification interface, selects a communicative move, anchors it to a discussion topic, and browses candidate points. After the participant confirms the point, the system reformulates it against the current discussion. (D) The proxy raises its hand and delivers the specified point on the spoken floor. Four-part figure showing the SecondVoice system. Top-left (A): the proxy sits at a table during discussion. Bottom-left (B, purple border): two system-elicited panels—a pulse showing the current topic with three stance buttons and a nudge asking whether the participant wants to share, with Yes and No buttons. Bottom-right (C, yellow border): the full structured specification interface with four attitude icons, three-step selection (communicative move, discussion topic, candidate points), and a text preview with Submit button. Top-right (D): the proxy raises its hand to deliver the point. Arrows show the flow from A to B to C to D.
Figure 3. The three-step structured specification interface. (1) Select a communicative move: question, challenge, agree, or suggest. (2) Anchor the move to an active discussion topic. (3) Browse and refine candidate points generated for the selected move–topic pair; a refresh button produces alternatives. Three vertically stacked panels showing the specification interface. Panel 1 shows four move buttons (question selected). Panel 2 adds three discussion-topic options with one highlighted and a text preview panel. Panel 3 adds candidate point options with a refresh button, and the preview panel shows the accumulated specification.
Figure 4. Backend pipeline architecture. Top: a periodic monitoring loop uses GPT-4o to extract discussion state and per-user stance summaries; a rule-based bookkeeping layer tracks question cues and pin rankings without LLM calls. Bottom: when a user opens the specification interface, GPT-4o-mini generates the initial three-step specification screen (communicative move × discussion topic × candidate points). A sentence-embedding model supports anchored refresh and contribution deduplication. Context-aware delivery reformulates the specified point into proxy speech via GPT-4o without naming its author; post-delivery feedback can inform later reformulations. Two-part pipeline diagram. The top half shows a periodic monitoring loop: speech-to-text feeds into an observation pass and profile update, with a rule-based bookkeeper accumulating context. The bottom half shows the user-triggered contribution flow: structured option generation, anchored refresh and similarity checks, and context-aware proxy delivery with a post-delivery feedback loop.
Prior
Speak
Hold
Unsaid-use
ID
Gender
VR/MR
up
back
SV
ATB
P1
F
Occ.
4
4
No
No
P2
F
1–2
2
4
Yes
No
P3
M
Occ.
4
2
No
No
P4
F
1–2
3
4
Yes
Yes
P5
F
Occ.
3
4
No
No
Table 2. Participant profile. Speak up and Hold back summarize two screening items about usually speaking up and sometimes holding back relevant ideas (1 = strongly disagree, 5 = strongly agree). VR/MR abbreviations are Occ. for occasionally, Freq. for frequently, and 1–2 for once or twice. Unsaid-use SV and ATB give each participant’s questionnaire answer to whether they used that condition’s channel for something they did not say aloud. Two participants completed the screening questionnaire twice; the later response is reported. Table listing the 16 participants (P1 to P16) with, for each, self-reported gender, prior VR or MR experience, two five-point screening ratings for usually speaking up and sometimes holding back relevant ideas, and yes or no answers for whether they used the SecondVoice channel and the anonymous text board for something they did not say aloud.
Figure 5. Percentage of participants who reported using the channel for something they did not say aloud. Half of participants reported using SecondVoice for an unsaid point, compared with 18.8% for the anonymous text board (exact McNemar p=.0625 , N=16 ). Grouped bar chart comparing the two conditions on the share of participants who reported using that channel for something they did not say aloud. The SecondVoice bar is higher than the anonymous text-board bar.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6. Lost at Sea task interface. Participants drag 15 salvaged items into a survival-importance ranking. An expert baseline is used for scoring. Screenshot of the survival ranking task interface showing 15 items including sextant, shaving mirror, water, mosquito netting, food rations, sea charts, seat cushion, oil and gas mixture, radio, shark repellent, rum, nylon rope, chocolate bars, fishing kit, and a map of the Pacific Ocean, presented as a drag-and-drop ranking list.
Figure 7. University President Selection task interface. Each participant sees shared candidate profiles and holds additional private facts not visible to other group members. Screenshot of the hidden-profile candidate selection task showing three candidates (A, B, C) with attributes including educational background, leadership style, campus experience, and interpersonal qualities. Some information is shared across participants while other facts are privately assigned.
Figure 8. Response distributions for the first eight primary Likert items, comparing SecondVoice (SV) and the anonymous text board (ATB). Items use a 7-point scale (1 = strongly agree, 7 = strongly disagree). No Likert item reached significance after Benjamini–Hochberg correction across the 13-item family (Sec. 5.3) ( N=16 ). Diverging stacked bar charts for eight primary Likert questionnaire items, comparing SecondVoice and anonymous text board conditions.
Figure 9. Response distributions for the remaining five primary Likert items and the two binary items, comparing SecondVoice (SV) and the anonymous text board (ATB). The two binary items were analyzed separately from the Likert family, with exact McNemar tests (Sec. 5.3) ( N=16 ). Diverging stacked bar charts for five primary Likert questionnaire items and horizontal bar charts for two binary items, comparing SecondVoice and anonymous text board conditions.
For noisy real-world environments such as those in open public spaces, spoken dialogue systems for both autonomous robots and avatars should be carefully designed to provide enhanced speech signals. These signals can be used either for speech recognition or, in the case of an avatar system, transmitted as clean speech to a remote operator. This work proposes an audio system that can be used for both these scenarios and was demonstrated as a proof-of-concept at the 2025 World Expo in Osaka. The first scenario is an attentive listening system with the android ERICA, and the second is a conversation support system with mobile Teleco robots, with one of them acting as an avatar for a remote operator. Both systems feature multi-party conversation and use a single multi-channel microphone array. We describe how our audio system not only enhances the speech of multiple speakers in a noisy environment, but provides a form of spatial audio which allows for more immersiveness in avatar-based conversational interactions.
Divesh Lala, Yogeeswaran Muthukumaran, Vincent Fernandes +10
1Kyoto University Graduate School of Informatics · 2Osaka University, Graduate School of Engineering Science · 3National University of Singapore +1
Speech agents are reactive and dyadic: they speak when spoken to, and to one person at a time. We ask what it takes for a speech agent to instead take part in a conversation among several people and speak up only when it can help. We introduce Jarvis, a real-time proactive speech agent that audibly participates in multi-party human conversations. Grounded in a document shared beforehand, Jarvis follows the discussion and intervenes when the group misses or misstates a fact and does not correct itself within a few turns. We make three contributions: a problem setting based on epistemic breakdowns that makes proactive intervention measurable, realized as CHI-180-proactive, a synthetic multi-party dataset seeded with known gaps, errors, and self-corrections; a proactive backbone that harnesses a small, open-weight model with deterministic checks and grounds every claim in a source sentence; and interaction techniques for taking the floor in live speech and showing the cited evidence on screen. On CHI-180-proactive, Jarvis is correct on most events it addresses and stays silent 97% of the time when the group resolves an issue itself. A live study with 23 participants confirms these trends with real-time interventions.
Seunghyun Oh, Hirotaka Hiraki, Shuyue Stella Li +2
Paul G. Allen School of Computer Science and Engineering, University of Washington, WA, USA · Graduate School of Interdisciplinary Information Studies, The University of Tokyo, Bunkyo, Tokyo, Japan
People write and sketch while speaking to explain, organize, and develop content together. Inspired by these practices, we investigate how voice agents can use a canvas alongside speech in multi-turn conversations with users. We conducted a two-part formative study: an observational study of how pairs coordinated speech and boardwork, followed by a design workshop that informed a design space for speech-canvas interaction with voice agents. Building on these insights, we developed VoCa, a voice agent that coordinates speech with visual object creation, annotation, and attention guidance. A five-day deployment with 18 participants examined usability, experiences of speech-canvas interaction, patterns of use, and desired improvements. Participants' experiences highlighted opportunities for speech-canvas interaction in learning, work, and daily life, alongside challenges in coordinating what agents say and show in ways users can follow and influence. These findings inform how voice agents can use a canvas alongside speech in conversation.
Yate Ge, Run Yuan, Yueran Qi +7
Tongji University Shanghai, China · Grinnell College Grinnell, Iowa, United States · Hunan University Changsha, Hunan, China +1