Omni-Life: Benchmarking Omnimodal Real-Time AI Assistants in People's Daily Lives

Mengze Hong; Ziwei Shang; Di Liang; Zijian Li; Liang Zhu; Hongyu Shan; Robin Zhou; Minlong Peng

cs.CLNov 2, 2026

Omni-Life: Benchmarking Omnimodal Real-Time AI Assistants in People's Daily Lives

Authors: Mengze Hong, Ziwei Shang, Di Liang, Zijian Li, Liang Zhu, Hongyu Shan, Robin Zhou, Minlong Peng,

Organizations: Hong Kong Polytechnic University · Today AI · Fudan University · Beijing University of Posts and Telecommunications

Abstract

Real-time omnimodal assistants are expected to support people in everyday life, but audiovisual understanding alone does not establish whether they can provide timely and appropriate help. We introduce Omni-Life, a benchmark grounded in real-world user needs, deployment scenarios, and device-dependent observations. It combines controlled video synthesis for hazardous situations and edge cases with annotated real-world videos under a shared temporal protocol. The benchmark contains 400 videos: 290 requiring assistance, with 555 required intervention events, and 110 requiring silence throughout to test whether models can withhold unnecessary responses when no assistance is needed. We evaluate twelve configurations featuring mainstream real-time omnimodal models, jointly assessing event coverage, response content, timing, and unwanted output. At most 4.48% of videos requiring assistance are handled completely correctly. To examine practical deployment strategies, we study task reminders and response-latency tolerance. Reminders can improve recall by 17.12 percentage points but add approximately 4.6 unwanted responses per minute of reference silence. Increasing latency tolerance by 10s improves recall by at most 5.95 percentage points. These results show that existing models remain insufficient to deliver reliable assistance in real-life applications, calling for tailored model optimization in instruction following, timely proactive intervention, and withholding responses when assistance is not needed.

Figures & tables

Figure 1: Illustration of real-life use cases for omnimodal real-time AI assistants.

Benchmark	Task focus	Video source	Proactive response	Response timing	User groups	Hazard alerts	Silence-only videos
StreamingBench ( Lin et al., 2026 )	Streaming video QA	YouTube	✓	✓	✗	✗	✗
OVO-Bench ( Niu et al., 2025 )	Online video QA	Curated + YouTube	✓	✓	✗	✗	✗
OmniMMI ( Wang et al., 2025c )	Multimodal interaction	Curated + YouTube	✓	✓	✗	✓	✗
ProactiveVideoQA ( Wang et al., 2025b )	Proactive video QA	Public datasets	✓	✓	✗	✓	✗
PhoStream ( Lu et al., 2026b )	Mobile assistance	Web + curated + recordings	✓	✓	✗	✗	✗
OmniInteract ( Lu et al., 2026c )	Spoken interaction	Recordings + curated	✓	✓	✗	✗	✗

Table 1: Comparison of benchmark tasks and evaluation protocols.

Figure 2: Five scenario domains and twelve service groups. Counts are videos; rings encode separate dimensions with schematic angles.

Figure 3: Construction pipeline of Omni-Life. Support needs guide scenario design and video sourcing through controlled synthesis or selection from public real-world sources. Temporal annotations define response windows, behavior labels, and reference content, and provide correct options for turn-based QA.

Statistic	Synthetic	Real-world	Total
Videos	277	123	400
Intervention	184	106	290
Remain Silent	93	17	110
Intervention nodes	255	300	555
Inform	139	223	362
Warn	92	57	149

Table 2: Omni-Life dataset statistics.

Model	Serving / audio input
Open-weight and Locally deployed Models
Open-weight models, locally deployed
MiniCPM-o 4.5 ( Cui et al., 2026 )	Native duplex / audio encoder
MOSS-VL ( Wang et al., 2026b )	Streaming service / SenseVoice
MOSS Preview ( Wang et al., 2026a )	Streaming service / SenseVoice
Venus-Omni ( Group, 2026 )	Native duplex / audio encoder

Table 3: Evaluated local and API-based model interfaces.

	Streaming				Turn-based QA
	Full Correct $\uparrow$	Intervention Recall $\uparrow$	Intervention F1 $\uparrow$	Silence Success $\uparrow$	Joint Acc. $\uparrow$	Content Acc. $\uparrow$	Timing Acc. $\uparrow$
Human Baseline
Human	53.10	62.34	64.49	91.82	60.38	82.13	66.13
Open-weight Models
MOSS Preview	0.00	1.08	1.09	2.73	11.00	45.00	29.00
AURA	0.00	12.61	1.63	0.91	24.00	65.75	31.00

Table 4: Results on Omni-Life (%; higher is better,

\uparrow

). Excluding the human baseline, the highest scores are marked in green and the second-best scores are marked in lighter green

Model	On-time		Late		No later reply	Others
Model	Correct $\uparrow$	Wrong $\downarrow$	Correct $\uparrow$	Wrong $\downarrow$	No later reply	Others
Open-weight Models
MiniCPM-o 4.5	1.4	2.5	1.4	2.3	82.9	9.4
MOSS-VL	5.8	10.1	2.5	8.5	66.1	7.0
MOSS Preview	1.1	9.0	0.7	3.6	82.3	3.2
Venus-Omni	2.0	7.7	1.4	3.1	80.5	5.2

Table 5: Response outcomes for all 555 required interventions, reported separately for each model (%). The highest value in each Correct column is bold .

Model	Daily living & services $\uparrow$ 95 videos [0.3ex]	Health & caregiving $\uparrow$ 70 videos [0.3ex]	Travel & mobility $\uparrow$ 40 videos [0.3ex]	Education & work $\uparrow$ 41 videos [0.3ex]	Sports & leisure $\uparrow$ 44 videos [0.3ex]
Open-weight Models
MiniCPM-o 4.5	1.1	0.0	2.5	2.4	2.3
MOSS-VL	6.3	2.9	2.5	4.9	4.5
MOSS Preview	0.0	0.0	0.0	0.0	0.0
Venus-Omni	3.2	1.4	2.5	2.4	0.0
Gander	3.2	1.4	0.0	4.9	0.0

Table 6: Full Correct (%, higher is better,

\uparrow

) across five scenarios, evaluated on videos requiring intervention. Column headers report video counts. The highest value in each column is bold .

	Preventive $\uparrow$ 68 nodes	Corrective / remedial $\uparrow$ 157 nodes	Routine support $\uparrow$ 330 nodes
Open-weight Models
JoyAI	30.88	26.75	14.55
AURA	8.82	14.01	12.73
MOSS-VL	1.47	7.01	6.06
DuplexOmni (S1)	8.82	3.18	4.24
Venus-Omni	0.00	1.91	2.42

Table 7: Intervention Recall by primary service function (%, higher is better,

\uparrow

). Column headers give the number of required interval nodes. The highest value is bold .

Failure type	Observable behavior	Sessions (%)
F1: Intervention control	Misses required assistance, responds outside its window, or speaks when no intervention is needed.	3,223 (68.96)
F2: Situational grounding	Misidentifies a current object, attribute, location, relation, or risk state.	148 (3.17)
F3: Temporal tracking	Misreports accumulated counts, event order, historical states, or completion status.	219 (4.69)
F4: Task fulfillment	Produces relevant speech but fails to complete the requested assistance.	450 (9.63)
F5: Unusable output	Produces timestamps, empty tags, nonsemantic strings, or an incomplete essential answer.	1,064 (22.76)

Table 8: Observable failures across 4,674 completed sessions. Categories may co-occur, and a flagged session may also contain useful responses.

Figure 4: Two on-time failures in cup counting (A) and post-fall assistance in a synthetic video (B), shown with source frames, condensed references, and translated outputs. Green intervals mark reference windows; red dots mark response onsets.

Model	Completed	Control $\downarrow$	Grounding $\downarrow$	Tracking $\downarrow$	Fulfillment $\downarrow$	Unusable $\downarrow$	Failed $\downarrow$
Open-weight Models
JoyAI	400	97.50	16.50	19.75	47.75	56.50	0
AURA	400	28.00	6.25	8.25	14.75	96.25	0
MOSS-VL	400	66.25	1.50	4.00	7.50	1.75	0
DuplexOmni (S1)	370	94.86	0.81	0.81	8.65	0.54	30
Venus-Omni	400	67.50	0.50	2.75	6.75	5.50	0

Table 9: Model-wise failure prevalence (%, lower is better,

\downarrow

) among completed sessions. Completed and Failed give session counts out of 400 attempts and the lowest value in each column is bold .

Input strategy	MiniCPM-o 4.5	MOSS-VL	Venus-Omni	Gemini 3.8 Live	Qwen3.5 Flash
Required events handled correctly and on time (%) $\uparrow$
Original task text	1.44	5.77	1.98	3.42	4.14
Structured task text	1.44	6.31	4.32	2.52	3.60
Repeat full task every second	5.41	6.85	13.51	12.43	3.42
Repeat full task every 5 seconds	1.80	14.05	10.27	20.54	3.42
General reminder every second	2.88	3.78	6.67	11.71	3.24

Table 10: Input design and periodic task reminders. Upper: required events handled correctly and on time (%, higher is better,

\uparrow

). Lower: unwanted responses per minute of reference silence (lower is better,

\downarrow

). Within each model, the highest values in the upper section are shown in bold , and the lowest values in the lower section are shown in bold .

Figure 5: Effect of task instruction modality on Intervention Recall (top; higher is better) and unwanted responses per minute of reference silence (bottom; lower is better).

Figure 6: Intervention Recall (%) as reference-window ends are extended. Rows are ordered by the gain at $\delta=10$ s, shown in percentage points (pp) by the right-hand bars.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Ten dataset examples across five scenario domains, pairing synthetic (left) and real-world (right) videos. Each case shows three source frames in temporal order, bilingual task queries, reference responses, and intervals of required assistance and silence.

User / service group	Cases	Nodes	MiniCPM-o 4.5	MOSS-VL	MOSS Preview	Venus- Omni	Gander	AURA
Adults	172	229	3.06	4.37	0.44	1.75	1.75	14.41
Infants and toddlers	26	34	0.00	0.00	0.00	0.00	0.00	2.94
Schoolchildren	36	31	0.00	3.23	0.00	3.23	3.23	9.68
Older adults	29	58	1.72	8.62	1.72	1.72	0.00	22.41
Visual impairment	27	46	0.00	2.17	4.35	0.00	0.00	2.17
Hearing impairment	13	17	0.00	11.76	0.00	5.88	0.00	17.65

Appendix

Table 11: Population and service-group Intervention Recall (%), part 1 of 2. Cases and required nodes are fixed across models; failed sessions remain in the node denominators.

User / service group	Cases	Nodes	JoyAI	DuplexOmni (S1)	Gemini 3.8 Live	Qwen3.5 Plus	Qwen3.5 Flash	Qwen3.8 Flash
Adults	172	229	21.40	3.06	5.68	3.49	4.37	3.49
Infants and toddlers	26	34	8.82	0.00	0.00	2.94	2.94	2.94
Schoolchildren	36	31	35.48	3.23	3.23	3.23	3.23	3.23
Older adults	29	58	27.59	6.90	0.00	0.00	0.00	0.00
Visual impairment	27	46	13.04	6.52	0.00	6.52	2.17	4.35
Hearing impairment	13	17	35.29	0.00	0.00	0.00	5.88	5.88

Appendix

Table 12: Population and service-group Intervention Recall (%), part 2 of 2. Cases and required nodes are fixed across models; failed sessions remain in the node denominators.

Model	Daily living & services $\uparrow$	Health & caregiving $\uparrow$	Travel & mobility $\uparrow$	Education & work $\uparrow$	Sports & leisure $\uparrow$
Intervention nodes	144	146	77	82	106
MiniCPM-o 4.5	0.69	0.68	1.30	1.22	3.77
MOSS-VL	9.72	4.79	3.90	7.32	1.89
MOSS Preview	2.08	0.68	1.30	0.00	0.94
Venus-Omni	3.47	0.68	3.90	2.44	0.00
Gander	2.08	0.68	0.00	2.44	0.00

Appendix

Table 13: Intervention Recall by scenario (%, higher is better,

\uparrow

). Intervention-node counts give the denominators, including failed sessions. Bold marks the best value in each column.

Model	Inform 362 nodes	Warn 149 nodes	Feedback 29 nodes	Answer 15 nodes
MiniCPM-o 4.5	1.66	0.67	0.00	6.67
MOSS-VL	5.25	8.05	0.00	6.67
MOSS Preview	1.66	0.00	0.00	0.00
Venus-Omni	1.93	2.01	0.00	6.67
Gander	1.10	0.00	0.00	13.33
AURA	11.88	15.44	6.90	13.33

Appendix

Table 14: Intervention Recall (%) by reference action.

Task input	Intervention (%)		Full Correct (%)	Silence Success (%)	Unwanted / min	Failed
	Recall	F1	Full Correct (%)	Silence Success (%)	Unwanted / min	Failed
MiniCPM-o 4.5
Written	1.44	2.63	1.38	95.45	0.14	0
Spoken	1.44	2.34	1.03	78.18	0.44	0
MOSS-VL
Written	5.77	7.06	4.48	82.73	1.41	0

Appendix

Table 15: Complete results for written and spoken task instructions on 400 videos per condition.

Panel of Figure 8 — Figure 8: Source frames and session timeline: DuplexOmni (S1) on asm_boom_feedback_01 .

Window (s)	Action	Required content (English reference)
$[0,22)$	wait	Remain silent.
$[22,27)$	feedback	The excavator arm is on backwards; it should bend toward the rear.
$[27,60)$	wait	Remain silent.

Appendix

Table 16: Reference windows and required content for asm_boom_feedback_01 .

Panel of Figure 9 — Figure 9: Source frames and session timeline: AURA on orbit_mug_06 .

Window (s)	Action	Required content (English reference)
$[0,1.5)$	wait	Remain silent.
$[1.5,5.5)$	inform	You’re holding a stainless steel thermos.
$[5.5,19.7)$	wait	Remain silent.

Appendix

Table 17: Reference windows and required content for orbit_mug_06 .

Panel of Figure 10 — Figure 10: Source frames and session timeline: MOSS-VL on robot_host_rest .

Window (s)	Action	Required content (English reference)
$[0,6)$	wait	Remain silent.
$[6,11)$	inform	Welcome, please feel free to take a seat.
$[11,30.08)$	wait	Remain silent.

Appendix

Table 18: Reference windows and required content for robot_host_rest .

Panel of Figure 11 — Figure 11: Source frames and session timeline: AURA on roof_leak .

Window (s)	Action	Required content (English reference)
$[0,2)$	wait	Remain silent.
$[2,5)$	inform	Water is leaking near a light fixture in the kitchen ceiling.
$[5,22)$	wait	Remain silent.
$[22,25)$	inform	Several containers and cloths on the floor are catching the leaking water.
$[25,72.365)$	wait	Remain silent.

Appendix

Table 19: Reference windows and required content for roof_leak .

Multimodal Agent Evaluation

See all 32 papers

OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs
Aug 21, 2026Xianyun Sun, Chaoyou Fu, Zhengye Zhang +6Multimodal Large Language Models AI Agent Benchmarks
Nanjing University · Nankai University · University of Waterloo
Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams
Oct 26, 2026Yinsong Yan, Ling Wang, Yuxuan Wang +11Multimodal Agent Evaluation MLLM Evaluation
The Hong Kong Polytechnic University · Alibaba Token Hub, Alibaba Group · Shanghai Jiaotong University +2
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
May 26, 2026Xudong Lu, Xueying Li, Annan Wang +8Multimodal Agents Multimodal Understanding
CUHK MMLab · SJTU · NTU +3

Aug 21, 2026J/K move · Enter open · S save

OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

Xianyun Sun, Chaoyou Fu, Zhengye Zhang, Feiyang Duan+5

Nanjing University · Nankai University · University of Waterloo

Multimodal Large Language Models AI Agent Benchmarks Multimodal Agents Multi-Turn Dialogue Evaluation

Open paper AskPDF arXiv

Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.

Abstract

Multimodal Agent Evaluation

Explore similar work

OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs

Omni-InteractScope: Benchmarking Proactive and Agentic Interaction in Continuous Omnimodal Streams

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants