EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports

Yu-Chien Tang; Yu-Hsiang Liu; An-Zi Yen

cs.CLAug 26, 2026

EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports

Authors: Yu-Chien Tang, Yu-Hsiang Liu, An-Zi Yen

Organizations: Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan

Abstract

VLMs are increasingly positioned as daily assistants that perceive first-person environments, follow user dialogue, and decide how to help. Existing egocentric benchmarks mainly evaluate visual understanding in isolation, leaving open whether models can arbitrate between visual evidence and user-provided language when the two are helpful, irrelevant, or conflicting. We introduce EgoArgus, a human-annotated dataset for evaluating egocentric assistants on understanding and decision tasks in five dialogue-video daily scenarios. Our results demonstrate that it is still challenging for current VLMs as reliable egocentric assistants, which requires identifying which modality is trustworthy and deciding when intervention is warranted. Deeper analysis also shows that existing modality bias mitigation methods are quite restricted to enhance performance, providing insights to aid practioners into the deployment of current VLMs as daily assistants.

Figures & tables

Figure 1: Figure 1: Overview of EgoArgus. EgoArgus covers five assistance scenarios (multimodal grounded, contradictory, video-grounded on-topic, video-grounded off-topic, text-grounded) and a no-assistance case, each evaluated across three capabilities: context understanding, next action prediction, and intervention decision.

Figure 2: Results on understanding part of EgoArgus. (a) The overall accuracy of each VLM. (b) The results divided by different tasks. (c) The accuracy of VLMs on the proposed 5 scenarios.

Scenario	EgoPlan	QaEgo4D	MIntRec	VISTA	Total
Multimodal	500	500	0	116	1,116
Contradictory	500	500	0	104	1,104
Off-topic	1,000	1,000	0	118	2,118
On-topic	1,000	1,000	0	102	2,102
Text-grounded	0	0	978	110	1,088
No assistance	0	0	0	239	239

Table 1: EgoArgus composition in the current evaluated split. EgoPlan-Bench2, QaEgo4D, and MIntRec form the real-video MCQA part; VISTA contributes 789 evaluated Assist-Step rows across the same five scenarios plus no-assistance cases.

Model	Input	F1 $\uparrow$	MAE $\downarrow$	$\Delta t$	Scenario F1 $\uparrow$						Mode error $\downarrow$
					Multi.	Contra.	Vid on	Vid off	Text	No asst.	Safety FN	Non-saf. FN	No-asst. FP
Gemini 3.1 Flash Lite	Full-video	0.8117	2.7420	-2.4970	0.9395	0.9042	0.8049	0.7431	0.9463	0.3885	0.1445	0.2443	0.3173
MiMo V2 Omni	Full-video	0.7500	2.6340	-2.3420	0.9194	0.8681	0.8171	0.7046	0.7031	0.3051	0.1992	0.3969	0.2620
Qwen3.5 Plus 02-15	Full-video	0.7232	2.5050	-1.9280	0.9340	0.9101	0.7249	0.6087	0.4444	0.4418	0.3008	0.4695	0.1144
InternVL3.5 4B	Sampled	0.5979	3.7030	-0.4040	0.9091	0.8587	0.8571	0.1935	0.0909	0.2483	0.4141	0.5687	0.3506
Cosmos Reason2 8B	Sampled	0.5572	2.7840	-1.7310	0.7283	0.7577	0.6131	0.5165	0.1897	0.3095	0.4023	0.7595	0.1070

Table 2: Model performance and error analysis on VISTA. Rows are sorted by overall assistance-decision F1. We report overall F1, timing MAE, signed timing error

\Delta t

(s), scenario-level F1 across the six VISTA scenarios, and mode-specific errors. Safety/non-safety FN are missed interventions; no-assist FP is false-alarm rate. Bold marks each model’s strongest scenario score or lowest mode-specific error.

Figure 3: Representative model error signatures on intervention decision. We additionally evaluate Gemini-3.1-Pro as the strong-model reference.

Model	Input	Multimodal	Contradictory	Off-topic	On-topic	Text-grounded	Mean
Qwen3.5-2B	Full	64.0	16.0	48.5	50.5	73.5	50.5
	Drop distractor	64.0	57.0	49.5	49.5	73.0	58.6
InternVL3.5-4B	Full	73.5	13.0	50.0	49.5	76.0	52.4
	Drop distractor	73.5	56.0	49.0	49.0	77.0	60.9
Cosmos-Reason2-8B	Full	77.0	10.5	49.0	54.5	81.0	54.4
	Drop distractor	77.0	51.5	50.0	50.0	82.5	62.2

Table 3: Full-input and oracle distractor-removal accuracy (%) for four VLMs on the same 1,000 balanced examples.

Model	#L	Jump	Pre	Post	Peak
Qwen3.5	24	L16 (.67)	49.5	94.5	97.5@L23
InternVL3.5	36	L25 (.69)	60.0	96.0	99.0@L34
Cosmos-R2	36	L25 (.69)	69.5	93.5	98.5@L31
Molmo2	36	L25 (.69)	65.5	97.0	100.0@L31

Table 4: Cross-model layer-wise linear-probe summary for four VLMs. Jump reports the layer and fractional decoder depth; pre, post, and peak values are probe accuracies (%).

Figure 4: Detailed layer-wise probing diagnostics for Qwen3.5-2B. Answer-preference information is weakly decodable through layer 15, then jumps sharply at layer 16 across all five scenarios.

Figure 5: Qwen3.5-2B visual-token and dialogue-token representations in the linear-probe weight space. The two modalities are mixed in shallow layers and become separated in middle-to-late layers.

Scenario	$\epsilon=-10$	$\epsilon=-5$	$\epsilon=5$	$\epsilon=10$
Multimodal	67.5	67.3	68.2	36.8
Contradictory	15.9	15.9	16.5	18.5
Off-topic	44.4	44.1	44.2	26.6
On-topic	47.7	47.5	46.8	27.6
Text-grounded	74.1	74.4	72.6	38.7

Table 5: Results under uniform visual-token attention reweighting. The intervention is sensitive to

\epsilon

and does not uniformly improve all scenarios.

Scenario	Base	NaPO	$\Delta$
Multimodal	69.4	67.6	-1.8
Contradictory	18.4	15.7	-2.7
Off-topic	48.4	44.4	-4.0
On-topic	50.7	47.5	-3.2
Text-grounded	73.7	75.4	+1.6
Overall	51.3	48.9	-2.5

Table 6: Results before and after NaPO-style preference alignment on the understanding part real-video MCQA.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: The annotation instructions for human annotators.

Figure 7: An annotated decision-part example from a safety assistance case. The VISTA-generated video (F1–F5) shows a user placing a metal travel mug into a microwave, while the dialogue gives no sign of any problem. Recognizing the hazard requires the video, since only the visual content reveals that the container is metal. Annotators label the earliest useful intervention time (6.625,s), the issue to warn about, and the assistance to provide, including the microwave-safe alternatives visible in the scene.

Benchmark / Dataset	FP video	Dialogue	Decision	Timing	Steps	No-assist
PROASSIST	✓	✓
HoloAssist	✓	✓
Ego-EXTRA	✓	✓
LifeEval	✓	✓
ESTP-Bench	✓			✓
ProactiveVideoQA				✓

Appendix

Table 7: Comparison with related benchmarks. We compare representative benchmarks along six axes central to VISTA. ✓ , , and denote primary coverage, partial/adjacent coverage, and no primary coverage, respectively.

Figure 8: Composition of the decision part. (A) Examples by scene, stacked by assistance mode (safety, non-safety, no-assistance). (B) Assist-required examples by event type, split into safety and non-safety.

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Can Argus Judge Them All? Comparing VLMs Across Domains
Jun 23, 2025Harsh Joshi, Gautam Siddharth Kashyap, Rafiq Ali +5VLM Evaluation VLM Robustness
Bharati Vidyapeeth, New Delhi, India · Macquarie University, Sydney, Australia · DSEU-Okhla, New Delhi, India +3
EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards
Jun 30, 2026Siddhant Panpatil, Arth Singh, Mijin Koo +3VLM Robustness Egocentric Video Understanding
AIM Intelligence · Seoul National University
AgentVQA: A Unified Benchmark for Agentic Visual Understanding
Oct 22, 2026Advait Gupta, Sachin Konan, Tom Yin +2VLM Evaluation Agent Benchmarks
University of Maryland, College Park · Princeton University

Jun 23, 2025J/K move · Enter open · S save

Can Argus Judge Them All? Comparing VLMs Across Domains

Harsh Joshi, Gautam Siddharth Kashyap, Rafiq Ali, Ebad Shabbir+4

Bharati Vidyapeeth, New Delhi, India · Macquarie University, Sydney, Australia · DSEU-Okhla, New Delhi, India+3

VLM Evaluation VLM Robustness Multimodal Foundation Models Cross-Dataset Generalization

Open paper AskPDF arXiv

Vision-Language Models (VLMs) are increasingly used in industry VLM applications such as retrieval systems, content generation platforms, and decision-support workflows, where model selection is commonly guided by benchmark rankings. These rankings are largely determined by retrieval, captioning, and reasoning downstream tasks; however, models with similar task performance often show substantially different behavior across datasets. This creates a Capability-Reliability Gap between benchmark performance and observed model stability. We present ARGUS-EVAL, a capability-reliability-oriented evaluation framework for VLMs that characterizes model behavior through Benchmark Capability P(M), Cross-Dataset Consistency CDC(M), Robustness Retention RR(M), and Efficiency E(M). We evaluate CLIP, BLIP, LXMERT, Gemma-3-4B, and Qwen-2.5VL-3B-Instruct across retrieval, captioning, and reasoning downstream tasks. The results reveal notable differences between capability-oriented and reliability-oriented rankings. Qwen-2.5VL-3BInstruct achieves the strongest overall capability (R@1 = 82.7%, BLEU-4 = 47.2%, CIDEr = 141.6, CDC = 0.91), whereas CLIP records the lowest latency (31 ms) and memory footprint (0.9 GB).

Abstract

VLM Evaluation

Explore similar work

Can Argus Judge Them All? Comparing VLMs Across Domains

EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards

AgentVQA: A Unified Benchmark for Agentic Visual Understanding