EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams

Heeseung Kim

cs.CLOct 8, 2026

EgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams

Authors: Heeseung Kim

Organizations: Department of AI, University of Seoul

Abstract

Wearable augmented reality (AR) assistants are moving toward continuous real-world interaction, where they perceive the user's activity through first-person video and audio and provide timely spoken guidance without being explicitly asked. While proactive video assistants, spoken dialog systems, and egocentric task understanding have each advanced rapidly, existing systems do not address the joint problem of deciding when to speak and what to say from continuous first-person streams. We introduce EgoVoice, a framework for training and evaluating proactive egocentric spoken assistants. From HoloAssist video recordings of real human instructors, we construct clean audio streams through source separation and speech resynthesis, and convert each video session into a format where the model must decide at each moment whether to remain silent or provide spoken guidance. We fine-tune an omni-modal LLM with our data, and further improve its proactive intervention behavior with direct preference optimization. Experiments across closed and open-source models show that existing systems rarely produce well-timed, meaningful proactive interventions, while EgoVoice yields clear improvements in intervention timing, content relevance, and human preference over the zero-shot backbone.

Figures & tables

Figure 1: Overview of proactive egocentric spoken assistance. An AR assistant receives first-person video and audio (left), decides whether to remain silent or intervene (center), and provides spoken guidance (right).

Figure 2: Overview of the EgoVoice framework. We process HoloAssist recordings into clean training data (left, Section 3.1 ), fine-tune MiniCPM-o 4.5 to learn listen-or-speak behavior (top right, EgoVoice-SFT), then refine intervention timing with DPO (bottom right, EgoVoice-DPO, Section 3.3 ).

Figure 3: The streaming interface of EgoVoice ( R101-2Aug-DSLR ). Each chunk carries one video frame and one second of audio, and the model emits Listen or Speak for each. On Speak chunks the LLM emits text tokens, and the speech-token decoder turns them into S3 speech tokens that are decoded into the waveform the user hears.

Model	Onset Timing			Content	LLM Judge (1–5)				End Timing
Model	Prec.	Rec.	F1	Cosine	Correct.	Prompt.	Effic.	Overall	end-vs-act (s)
Gemini Live	.379	.019	.022	.448	2.09	2.39	2.67	1.94	$-$ 3.20 / $-$ 2.26
gpt-realtime	.285	.024	.030	.500	2.14	2.60	2.52	2.10	$-$ 7.40 / $-$ 7.46
gpt-realtime-2	.256	.019	.024	.456	2.28	2.40	2.10	2.14	$-$ 16.1 / $-$ 16.8
ProAssist †	.480	.158	.195	.594	2.34	2.63	2.99	2.06	–
+ ASR/TTS cascade	.479	.158	.195	.564	2.13	2.50	2.87	1.92	$-$ 0.44 / $-$ 1.56

Table 1: Results on 170 benchmark sessions ( text-asr ).

\dagger

indicates a text-only baseline, evaluated on text-llm .

\ddagger

indicates the human instructor, whose utterances are the reference for all systems. End-vs-act is reported as mean / median on the action-linked subset (

T{=}5

), where positive values mean the model typically finishes speaking before the user acts; per-intervention distributions are in Appendix A.6.2 . Best in bold , second best underlined .

SFT design choices
Variant	Onset Timing			Content	LLM Judge (1–5)				End Timing
Variant	Prec.	Rec.	F1	Cosine	Correct.	Prompt.	Effic.	Overall	end-vs-act (s)
EgoVoice-SFT	.467	.235	.274	.684	2.41	2.64	2.64	2.23	+1.57 / +1.52
w/o recipe	.398	.223	.240	.621	1.91	2.14	2.15	1.81	+2.32 / +2.09
w/o detach	.435	.246	.276	.680	2.38	2.61	2.45	2.18	+2.19 / +1.86
DPO variants (all from EgoVoice-SFT)

Table 2: Ablation results ( text-asr ). Metrics are computed following the same protocol as Table 1 . EgoVoice-DPO uses length-normalized DPO with

\beta=0.1

. Each variant differs from this reference in only the listed factor.

Comparison	Win	Tie	Lose	p -val
SFT vs MiniCPM	45.6%	23.0%	31.3%	0.01
DPO vs MiniCPM	45.3%	21.7%	33.0%	0.02
DPO vs SFT	36.1%	30.6%	33.3%	0.60

Table 3: Pairwise human preference evaluation results.

Model	Guidance	Narration	Other
Human instructor	76.7	5.6	17.8
MiniCPM-o 4.5	30.1	21.0	49.0
EgoVoice-SFT	65.7	5.4	29.0
EgoVoice-DPO	64.2	5.5	30.3

Table 4: Distribution of utterance types on the 170 benchmark sessions (%). Each utterance is classified as proactive guidance, third-person scene narration, or other. The first row is the reference instructor speech.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Sessions
Original HoloAssist (train + val)	1,673
After source separation (VAD filter)	1,532
After TTS/WER filter + duration ( $\geq$ 60s)	1,419
Final train split	1,249
Final eval split	170

Appendix

Table 5: Dataset attrition through the construction pipeline. Filters: (i) Silero VAD with leak_final=0 to remove source-separation residue, (ii) TTS/WER quality + minimum duration (

\geq

60s). The eval split consists of the surviving sessions from the original HoloAssist evaluation split and is not further filtered.

Figure 4: Why naive TTS synthesis on video-text dialog data is insufficient. The ground truth instructor (top) times utterances to finish before the user’s next action. Naive TTS (middle) generates longer speech that overlaps with the user’s action. Shifting the onset earlier (bottom) avoids the action overlap but collides with the previous utterance and delivers guidance before the user has seen the relevant context. This motivates our choice to start from HoloAssist, where real instructor speech timing is already aligned with user actions, and to resynthesize speech with a strict duration constraint that preserves this alignment.

	Hyperparameter	SFT	DPO
LoRA	Rank ( $r$ ) / Alpha ( $\alpha$ ) / Dropout	32 / 64 / 0.05
	Target modules	All linear (LLM + TTS)	All linear (LLM only)
	TTS decoder	Trained	Frozen
Loss	Objective	$\mathcal{L}_{\text{LM}}+\lambda\mathcal{L}_{\text{S3}}$	Length-norm. DPO
Loss	Key parameter	$\lambda=0.1$ , detach = True	$\beta=0.1$
Optimization	Learning rate	2e-4	5e-6

Appendix

Table 6: Training hyperparameters for SFT and DPO.

Panel of Figure 5 — Figure 5: System prompt used for LLM-as-a-judge evaluation. The model is forced to emit a strict JSON schema with five keys: analysis , correctness , promptness , efficiency , overall (latter four integers 1-5).

Panel of Figure 6 — Figure 6: Prompt used for action-linked utterance classification with majority voting (strict-majority over 5 independent calls at temperature $1.0$ ; ties resolved by the most frequent label). Response is enforced as strict JSON {is_directive $\in$ {yes, no, ambiguous}, rationale} .

Panel of Figure 7 — Figure 7: Prompt used for utterance type classification. The label is the strict majority over 5 independent calls.

Model	Onset Timing			Content	LLM Judge (1–5)				End Timing
Model	Prec.	Rec.	F1	Cosine	Correct.	Prompt.	Effic.	Overall	end-vs-act (s)
Gemini Live	.364	.019	.022	.453	2.24	2.39	2.82	2.06	$-$ 3.65 / $-$ 3.62
gpt-realtime	.285	.024	.030	.499	2.24	2.70	2.54	2.20	$-$ 7.40 / $-$ 7.46
gpt-realtime-2	.256	.019	.024	.457	2.32	2.30	2.22	2.14	$-$ 16.0 / $-$ 16.8
ProAssist †	.480	.158	.195	.594	2.34	2.63	2.99	2.06	–
+ ASR/TTS cascade	.479	.158	.195	.585	2.35	2.63	2.97	2.07	$-$ 0.44 / $-$ 1.56

Appendix

Table 7: Results on 170 benchmark sessions ( text-llm ).

\dagger

indicates a text-only baseline. On this channel the cascade row is ProAssist reading ASR transcripts of the user’s speech instead of ground-truth transcripts, so the two rows differ only in that input. Notation otherwise follows Table 1 .

Figure 8: Human evaluation interface on Amazon Mechanical Turk. Evaluators watch two versions of the same egocentric clip with different assistant speech overlays and synchronized captions, then select which assistant (A or B) provides more helpful and timely guidance, or indicate a tie.

Model	n	Duration	Start-vs-action (s)	End-vs-action (s)
Gemini Live	3	6.64 / 6.48	+3.44 / +3.04	$-$ 3.20 / $-$ 2.26
gpt-realtime	30	11.73 / 10.84	+4.34 / +4.05	$-$ 7.40 / $-$ 7.46
gpt-realtime-2	22	20.39 / 20.20	+4.33 / +4.29	$-$ 16.1 / $-$ 16.8
ProAssist + ASR/TTS cascade	92	6.38 / 6.20	+5.94 / +4.82	$-$ 0.44 / $-$ 1.56
MiniCPM-o 4.5	99	9.87 / 6.00	+5.18 / +4.70	$-$ 4.69 / $-$ 0.40
EgoVoice-SFT	255	2.99 / 2.32	+4.55 / +4.38	+1.57 / +1.52

Appendix

Table 8: Decomposition of end-vs-action on the action-linked subset (

T{=}5

), in seconds, as mean / median.

n

is the number of matched interventions, which differs across models.

Figure 9: End-vs-action for every matched action-linked intervention (

T{=}5

, text-asr ), in seconds. Each dot is one intervention; blue means the model finished speaking before the user began the linked action, orange means it was still speaking. The vertical tick marks the median and

n

is the number of matched interventions. Values outside the plotted range are clipped to it.

Model	Utt.	Words	$\leq$ 3 words
Gemini Live	0.3	24.2	2.3%
gpt-realtime	0.9	37.7	0.0%
gpt-realtime-2	0.8	51.6	0.0%
ProAssist †	2.2	16.6	0.8%
+ ASR/TTS cascade	2.1	16.7	1.9%
MiniCPM-o 4.5	3.4	38.2	8.2%

Appendix

Table 9: Utterance statistics on the text-llm channel: utterances per session, mean words per utterance, and the share of utterances of three words or fewer.

Comparison	$\boldsymbol{n}$	Diff	p
Onset F1
DPO vs SFT	170	$+$ 0.050	0.0007
DPO vs MiniCPM	170	$+$ 0.211	6.0e $-$ 19
SFT vs MiniCPM	170	$+$ 0.161	1.2e $-$ 13
DPO vs ProAssist	170	$+$ 0.129	1.1e $-$ 08
SFT vs ProAssist	170	$+$ 0.079	7.9e $-$ 05

Appendix

Table 10: Paired Wilcoxon signed-rank tests over sessions. Onset F1 is defined for all 170 sessions, while judge scores exist only for sessions in which both models speak, so

n

varies. Comparisons against ProAssist use text-llm on both sides.

Panel of Figure 10 — Figure 10: Two interventions that succeed. Each panel follows Figure 3 , with the instructor’s own speech added at the bottom (shaded) and the start of the user’s linked action marked. EgoVoice-DPO speaks at the same moment as the instructor, says the same thing, and finishes before the user acts. The system prompt lists the task steps in third person, so the model has to choose which step is due, phrase it as an instruction, and time it.

Panel of Figure 11 — Figure 11: Two interventions that end too late. The onset and the content are right, but the model’s speech runs well past the instructor’s and is still playing when the user begins the action.

Panel of Figure 12 — Figure 12: Two interventions that guide the wrong step. The model delivers an instruction that belongs elsewhere in the task: in the first, one the instructor gave 14 seconds earlier; in the second, one the model itself gave 17 seconds earlier. In both cases the instructor has already moved on.

Datasets
Artifact	License	URL
HoloAssist	CDLAv2	https://holoassist.github.io
Multilingual LibriSpeech	CC BY 4.0	https://www.openslr.org/94
Backbone & Training
MiniCPM-o 4.5	apache-2.0	https://huggingface.co/openbmb/MiniCPM-o-4_5
Data Construction

Appendix

Table 11: All artifacts used in this work with their licenses and sources.

Explore similar work

Sep 7, 2026cs.CV

Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision

We present our submission to the EgoProactive track of the ECCV 2026 Wearable AI Challenge, which ranked first in the large-model division and second in the <=2B division. The task requires a wearable assistant to decide after each eight-second segment of egocentric video whether to intervene or remain silent. Our approach has two main components. First, we reformulate intervention timing as single-token classification. Rather than generating either

interrupt

<utterance> or

silent

, the model predicts yes or no, and we derive the decision from the renormalised probabilities of these two tokens. This formulation improved macro-F1 by 0.249 and G-mean by 0.30 over free-form generation. Second, because labelled data were limited to the released validation set, we generated additional supervision using a tool-calling video agent that inspects each clip and assigns intervention timestamps. A narration-only alternative was four times larger and ten times cheaper, but transferred worse than supervision from an unrelated real corpus, suggesting that visual grounding is more important than annotation volume for this task.

Logesh Kumar Umapathi

Team Ambient

Egocentric Video Understanding Proactive Assistance

Ask PDF

Jul 13, 2026cs.CV

Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

When should an intelligent assistant speak up without being asked? Continuous egocentric video offers rich, evolving context that enables a new form of assistance: one that is proactive rather than merely reactive. Yet existing approaches either wait passively for user queries or treat every detected event as requiring a response, without considering the user's history, current activity, or whether assistance would actually be welcome. We reframe proactive assistance as a context-dependent decision problem: the agent must not only perceive what is happening, but reason over accumulated temporal context to determine when and whether to intervene. To this end, we present Vinci2, a proactive egocentric assistance system that advances the on-device assistant Vinci from reactive response toward proactivity. On the evaluation side, we present EgoServe, the first large-scale benchmark for proactive assistance in continuous egocentric video. EgoServe comprises over 3,000 service instances organized along 4 temporal memory horizons, ranging from immediate safety alerts to long-term habit coaching, across 10 service categories. On the modeling side, we propose EgoMemo, a training-free, memory-augmented agent that maintains three complementary memory representations: multi-scale temporal summaries, a semantic knowledge graph, and visual embedding archives. At each timestep, EgoMemo performs retrieval-augmented reasoning to determine whether assistance is warranted and, if so, produces contextually grounded responses. Experiments demonstrate that EgoMemo establishes strong baselines on EgoServe while remaining competitive on existing egocentric benchmarks. Our benchmark and code are publicly available at Vinci2.

Gong Sitong, Tianyu Yan, Caixin Kang +6

Dalian University of Technology · Alaya Lab · The University of Tokyo

Egocentric Video Understanding Proactive Assistance Memory-Augmented Agents

Ask PDF

Sep 21, 2026eess.AS

Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction

Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M

^{2}

-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns whether, when, and how the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of

τ

-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.

Lujia Bao, Qian Chen, Luyao Cheng +15

Alibaba Token Foundry, Alibaba Group

Tool-Augmented Language Model Agents Speech Foundation Models Audio Reasoning LLM Agents Conversational Agents Full-Duplex Spoken Dialogue Systems

Ask PDF