cs.CLOct 8, 2026
SaveEgoVoice: Proactive Spoken Assistance from Egocentric Multimodal Streams
Organizations: Department of AI, University of Seoul
Abstract
Wearable augmented reality (AR) assistants are moving toward continuous real-world interaction, where they perceive the user's activity through first-person video and audio and provide timely spoken guidance without being explicitly asked. While proactive video assistants, spoken dialog systems, and egocentric task understanding have each advanced rapidly, existing systems do not address the joint problem of deciding when to speak and what to say from continuous first-person streams. We introduce EgoVoice, a framework for training and evaluating proactive egocentric spoken assistants. From HoloAssist video recordings of real human instructors, we construct clean audio streams through source separation and speech resynthesis, and convert each video session into a format where the model must decide at each moment whether to remain silent or provide spoken guidance. We fine-tune an omni-modal LLM with our data, and further improve its proactive intervention behavior with direct preference optimization. Experiments across closed and open-source models show that existing systems rarely produce well-timed, meaningful proactive interventions, while EgoVoice yields clear improvements in intervention timing, content relevance, and human preference over the zero-shot backbone.
Figures & tables
Figure 1: Overview of proactive egocentric spoken assistance. An AR assistant receives first-person video and audio (left), decides whether to remain silent or intervene (center), and provides spoken guidance (right).
Figure 2: Overview of the EgoVoice framework. We process HoloAssist recordings into clean training data (left, Section 3.1 ), fine-tune MiniCPM-o 4.5 to learn listen-or-speak behavior (top right, EgoVoice-SFT), then refine intervention timing with DPO (bottom right, EgoVoice-DPO, Section 3.3 ).
Figure 3: The streaming interface of EgoVoice ( R101-2Aug-DSLR ). Each chunk carries one video frame and one second of audio, and the model emits Listen or Speak for each. On Speak chunks the LLM emits text tokens, and the speech-token decoder turns them into S3 speech tokens that are decoded into the waveform the user hears.
| Model | Onset Timing | Content | LLM Judge (1–5) | End Timing | |||||
| Prec. | Rec. | F1 | Cosine | Correct. | Prompt. | Effic. | Overall | end-vs-act (s) | |
| Gemini Live | .379 | .019 | .022 | .448 | 2.09 | 2.39 | 2.67 | 1.94 | 3.20 / 2.26 |
| gpt-realtime | .285 | .024 | .030 | .500 | 2.14 | 2.60 | 2.52 | 2.10 | 7.40 / 7.46 |
| gpt-realtime-2 | .256 | .019 | .024 | .456 | 2.28 | 2.40 | 2.10 | 2.14 | 16.1 / 16.8 |
| ProAssist † | .480 | .158 | .195 | .594 | 2.34 | 2.63 | 2.99 | 2.06 | – |
| + ASR/TTS cascade | .479 | .158 | .195 | .564 | 2.13 | 2.50 | 2.87 | 1.92 | 0.44 / 1.56 |
Table 1: Results on 170 benchmark sessions ( text-asr ). indicates a text-only baseline, evaluated on text-llm . indicates the human instructor, whose utterances are the reference for all systems. End-vs-act is reported as mean / median on the action-linked subset ( ), where positive values mean the model typically finishes speaking before the user acts; per-intervention distributions are in Appendix A.6.2 . Best in bold , second best underlined .
| Variant | Onset Timing | Content | LLM Judge (1–5) | End Timing | |||||
|---|---|---|---|---|---|---|---|---|---|
| Prec. | Rec. | F1 | Cosine | Correct. | Prompt. | Effic. | Overall | end-vs-act (s) | |
| SFT design choices | |||||||||
| EgoVoice-SFT | .467 | .235 | .274 | .684 | 2.41 | 2.64 | 2.64 | 2.23 | +1.57 / +1.52 |
| w/o recipe | .398 | .223 | .240 | .621 | 1.91 | 2.14 | 2.15 | 1.81 | +2.32 / +2.09 |
| w/o detach | .435 | .246 | .276 | .680 | 2.38 | 2.61 | 2.45 | 2.18 | +2.19 / +1.86 |
| DPO variants (all from EgoVoice-SFT) | |||||||||
Table 2: Ablation results ( text-asr ). Metrics are computed following the same protocol as Table 1 . EgoVoice-DPO uses length-normalized DPO with . Each variant differs from this reference in only the listed factor.
| Comparison | Win | Tie | Lose | p -val |
|---|---|---|---|---|
| SFT vs MiniCPM | 45.6% | 23.0% | 31.3% | 0.01 |
| DPO vs MiniCPM | 45.3% | 21.7% | 33.0% | 0.02 |
| DPO vs SFT | 36.1% | 30.6% | 33.3% | 0.60 |
Table 3: Pairwise human preference evaluation results.
| Model | Guidance | Narration | Other |
|---|---|---|---|
| Human instructor | 76.7 | 5.6 | 17.8 |
| MiniCPM-o 4.5 | 30.1 | 21.0 | 49.0 |
| EgoVoice-SFT | 65.7 | 5.4 | 29.0 |
| EgoVoice-DPO | 64.2 | 5.5 | 30.3 |
Table 4: Distribution of utterance types on the 170 benchmark sessions (%). Each utterance is classified as proactive guidance, third-person scene narration, or other. The first row is the reference instructor speech.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Sessions |
| Original HoloAssist (train + val) | 1,673 |
| After source separation (VAD filter) | 1,532 |
| After TTS/WER filter + duration ( 60s) | 1,419 |
| Final train split | 1,249 |
| Final eval split | 170 |
Table 5: Dataset attrition through the construction pipeline. Filters: (i) Silero VAD with leak_final=0 to remove source-separation residue, (ii) TTS/WER quality + minimum duration ( 60s). The eval split consists of the surviving sessions from the original HoloAssist evaluation split and is not further filtered.
Figure 4: Why naive TTS synthesis on video-text dialog data is insufficient. The ground truth instructor (top) times utterances to finish before the user’s next action. Naive TTS (middle) generates longer speech that overlaps with the user’s action. Shifting the onset earlier (bottom) avoids the action overlap but collides with the previous utterance and delivers guidance before the user has seen the relevant context. This motivates our choice to start from HoloAssist, where real instructor speech timing is already aligned with user actions, and to resynthesize speech with a strict duration constraint that preserves this alignment.
| Hyperparameter | SFT | DPO | |
|---|---|---|---|
| LoRA | Rank ( ) / Alpha ( ) / Dropout | 32 / 64 / 0.05 | |
| Target modules | All linear (LLM + TTS) | All linear (LLM only) | |
| TTS decoder | Trained | Frozen | |
| Loss | Objective | Length-norm. DPO | |
| Key parameter | , detach = True | ||
| Optimization | Learning rate | 2e-4 | 5e-6 |
Table 6: Training hyperparameters for SFT and DPO.
Figure 5: System prompt used for LLM-as-a-judge evaluation. The model is forced to emit a strict JSON schema with five keys: analysis , correctness , promptness , efficiency , overall (latter four integers 1-5).
Figure 6: Prompt used for action-linked utterance classification with majority voting (strict-majority over 5 independent calls at temperature ; ties resolved by the most frequent label). Response is enforced as strict JSON {is_directive {yes, no, ambiguous}, rationale} .
Figure 7: Prompt used for utterance type classification. The label is the strict majority over 5 independent calls.
| Model | Onset Timing | Content | LLM Judge (1–5) | End Timing | |||||
| Prec. | Rec. | F1 | Cosine | Correct. | Prompt. | Effic. | Overall | end-vs-act (s) | |
| Gemini Live | .364 | .019 | .022 | .453 | 2.24 | 2.39 | 2.82 | 2.06 | 3.65 / 3.62 |
| gpt-realtime | .285 | .024 | .030 | .499 | 2.24 | 2.70 | 2.54 | 2.20 | 7.40 / 7.46 |
| gpt-realtime-2 | .256 | .019 | .024 | .457 | 2.32 | 2.30 | 2.22 | 2.14 | 16.0 / 16.8 |
| ProAssist † | .480 | .158 | .195 | .594 | 2.34 | 2.63 | 2.99 | 2.06 | – |
| + ASR/TTS cascade | .479 | .158 | .195 | .585 | 2.35 | 2.63 | 2.97 | 2.07 | 0.44 / 1.56 |
Table 7: Results on 170 benchmark sessions ( text-llm ). indicates a text-only baseline. On this channel the cascade row is ProAssist reading ASR transcripts of the user’s speech instead of ground-truth transcripts, so the two rows differ only in that input. Notation otherwise follows Table 1 .
Figure 8: Human evaluation interface on Amazon Mechanical Turk. Evaluators watch two versions of the same egocentric clip with different assistant speech overlays and synchronized captions, then select which assistant (A or B) provides more helpful and timely guidance, or indicate a tie.
| Model | n | Duration | Start-vs-action (s) | End-vs-action (s) |
| Gemini Live | 3 | 6.64 / 6.48 | +3.44 / +3.04 | 3.20 / 2.26 |
| gpt-realtime | 30 | 11.73 / 10.84 | +4.34 / +4.05 | 7.40 / 7.46 |
| gpt-realtime-2 | 22 | 20.39 / 20.20 | +4.33 / +4.29 | 16.1 / 16.8 |
| ProAssist + ASR/TTS cascade | 92 | 6.38 / 6.20 | +5.94 / +4.82 | 0.44 / 1.56 |
| MiniCPM-o 4.5 | 99 | 9.87 / 6.00 | +5.18 / +4.70 | 4.69 / 0.40 |
| EgoVoice-SFT | 255 | 2.99 / 2.32 | +4.55 / +4.38 | +1.57 / +1.52 |
Table 8: Decomposition of end-vs-action on the action-linked subset ( ), in seconds, as mean / median. is the number of matched interventions, which differs across models.
Figure 9: End-vs-action for every matched action-linked intervention ( , text-asr ), in seconds. Each dot is one intervention; blue means the model finished speaking before the user began the linked action, orange means it was still speaking. The vertical tick marks the median and is the number of matched interventions. Values outside the plotted range are clipped to it.
| Model | Utt. | Words | 3 words |
|---|---|---|---|
| Gemini Live | 0.3 | 24.2 | 2.3% |
| gpt-realtime | 0.9 | 37.7 | 0.0% |
| gpt-realtime-2 | 0.8 | 51.6 | 0.0% |
| ProAssist † | 2.2 | 16.6 | 0.8% |
| + ASR/TTS cascade | 2.1 | 16.7 | 1.9% |
| MiniCPM-o 4.5 | 3.4 | 38.2 | 8.2% |
Table 9: Utterance statistics on the text-llm channel: utterances per session, mean words per utterance, and the share of utterances of three words or fewer.
| Comparison | Diff | p | |
| Onset F1 | |||
| DPO vs SFT | 170 | 0.050 | 0.0007 |
| DPO vs MiniCPM | 170 | 0.211 | 6.0e 19 |
| SFT vs MiniCPM | 170 | 0.161 | 1.2e 13 |
| DPO vs ProAssist | 170 | 0.129 | 1.1e 08 |
| SFT vs ProAssist | 170 | 0.079 | 7.9e 05 |
Table 10: Paired Wilcoxon signed-rank tests over sessions. Onset F1 is defined for all 170 sessions, while judge scores exist only for sessions in which both models speak, so varies. Comparisons against ProAssist use text-llm on both sides.
Figure 10: Two interventions that succeed. Each panel follows Figure 3 , with the instructor’s own speech added at the bottom (shaded) and the start of the user’s linked action marked. EgoVoice-DPO speaks at the same moment as the instructor, says the same thing, and finishes before the user acts. The system prompt lists the task steps in third person, so the model has to choose which step is due, phrase it as an instruction, and time it.
Figure 11: Two interventions that end too late. The onset and the content are right, but the model’s speech runs well past the instructor’s and is still playing when the user begins the action.
Figure 12: Two interventions that guide the wrong step. The model delivers an instruction that belongs elsewhere in the task: in the first, one the instructor gave 14 seconds earlier; in the second, one the model itself gave 17 seconds earlier. In both cases the instructor has already moved on.
| Artifact | License | URL |
|---|---|---|
| Datasets | ||
| HoloAssist | CDLAv2 | https://holoassist.github.io |
| Multilingual LibriSpeech | CC BY 4.0 | https://www.openslr.org/94 |
| Backbone & Training | ||
| MiniCPM-o 4.5 | apache-2.0 | https://huggingface.co/openbmb/MiniCPM-o-4_5 |
| Data Construction | ||
Table 11: All artifacts used in this work with their licenses and sources.