EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AI

Junlong Li; Junxi Li; Jianjun Gao; Chen Cai; Lap-Pui Chau; Yi Wang

cs.CVAug 25, 2026

EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AI

Authors: Junlong Li, Junxi Li, Jianjun Gao, Chen Cai, Lap-Pui Chau, Yi Wang

Organizations: Department of EEE, The Hong Kong Polytechnic University · School of EEE, Nanyang Technological University

Abstract

The majority of our everyday activities are procedural and consist of sequences of interdependent steps. However, existing benchmarks for Visual Agents and Visual Language Models (VLMs) overlook the evaluation of their procedural comprehension ability from an egocentric visual perspective, particularly for detecting procedural errors, a critical capability for everyday assistance. To bridge this gap, the EgoErrorVQA task is firstly proposed for egocentric procedural comprehension with explicit procedural errors modeling. Besides, we develop a user-friendly evaluator agent based on the Agent2Agent (A2A) protocol, enabling rigorous and standardized evaluation of visual agents through VQA-based interaction. A range of models are evaluated using both open-ended and multiple-choice questions, revealing persistent weaknesses in handling procedural errors and error types. Moreover, we introduce Ego-ADR, an Adaptive Decoupled Reasoning framework that decouples complex procedural reasoning to enhance models' understanding of procedural errors. It achieves consistent performance gains over the selected baselines and attains state-of-the-art results on several metrics under comparable settings. Code: https://github.com/z1oong/EgoErrorVQA

Figures & tables

Benchmark	Videos	QA-pairs	Procedural	Aciton Label	Error Centric	LLM Scoring	Multiple Scenarios	Open- end Question	Multiple- choice Question
EgoVQA ( Fan, 2019 )	16	600	${\color[rgb]{1,0,0}\times}$	${\color[rgb]{1,0,0}\times}$	${\color[rgb]{1,0,0}\times}$	${\color[rgb]{1,0,0}\times}$	✓	✓	✓
AssistQ ( Wong et al., 2022 )	100	531	✓	✓	${\color[rgb]{1,0,0}\times}$	${\color[rgb]{1,0,0}\times}$	✓	${\color[rgb]{1,0,0}\times}$	✓
EgoTaskQA ( Jia et al., 2022 )	2K	40K	✓	✓	${\color[rgb]{1,0,0}\times}$	${\color[rgb]{1,0,0}\times}$	✓	${\color[rgb]{1,0,0}\times}$	✓
EgoPlan-Bench ( Chen et al., 2023 )	-	4939	✓	✓	${\color[rgb]{1,0,0}\times}$	${\color[rgb]{1,0,0}\times}$	✓	${\color[rgb]{1,0,0}\times}$	✓
EgoSchema ( Mangalam et al., 2023 )	-	5063	${\color[rgb]{1,0,0}\times}$	${\color[rgb]{1,0,0}\times}$	${\color[rgb]{1,0,0}\times}$	${\color[rgb]{1,0,0}\times}$	✓	${\color[rgb]{1,0,0}\times}$	✓
VidEgoThink ( Cheng et al., 2024a )	195	600	${\color[rgb]{1,0,0}\times}$	✓	${\color[rgb]{1,0,0}\times}$	✓	✓	✓	${\color[rgb]{1,0,0}\times}$

Table 1: Comparison between EgoErrorVQA and common video benchmarks. Videos represents the number of original videos.

Benchmark	Source	Clips	Correct samples	Error samples	Q-Len $/$ A-Len	QA pairs
Open-end	Cap	960	632	328	12.86 $/$ 15.57	3560
	Oops	215	175	40
	Tent	184	124	60
	Assem	446	341	105
Multiple-choice	Cap	1000	660	340	- $/$ 1.38	1857
	Oops	215	175	40

Table 2: Statistics of EgoErrorVQA. Cap, Oops, Tent, Assem represents CaptainCook4D, EgoOops, Epic-tent, and Assembly101. Q-Len, A-Len is the average length of questions and answers, measured in words.

Figure 1: Overview workflow of EgoErrorVQA. In the QA-pairs Generation stage, it outlines the overall procedure for constructing QA-pairs and highlights the stages and roles of human involvement. Subsequently, the Open-end VQA Evaluation and Multiple-choice VQA Evaluation sections illustrate the overall pipelines for those two evaluation tasks, respectively.

Figure 2: (a), (b), (c), and (d) respectively show the proportions of correct and incorrect samples in open-end VQA and multiple-choice VQA. (e) shows the proportion of QA-pairs in the open-end VQA that are generated by the LLM versus those added by human annotator. (f) shows the distribution of each error type.

Figure 3: Overall Architecture of Ego-ADR. In steps (a), (b), and (c), the upper path represents deep decoupling, whereas the lower path represents shallow decoupling.

Method	frames	Cook	Tent	Toy	Hand	Avg-Sim.
Human	-	3.79	3.89	3.91	3.47	3.77
GPT-4o	8f	2.97	3.14	3.46	3.5	3.17
GPT-4o-mini	8f	2.84	3.18	3.33	3.33	3.05
Gemini-2.5-flash	8f	2.82	2.48	2.89	3.39	2.87
Open-Source 7B / 8B
LLaVA-OneVision	8f	2.85	3.17	3.82	3.34	3.18

Table 3: Models’ performance on open-end VQA. Cook denotes results on Cooking, Tent on Tent assembly, Toy on Toy car assembly, Hand on Handicraft activities. Avg-Sim. denotes the sample-size-weighted average over all tasks. Details of Human performance shown in Appendix C .

Method	frames	Acc	Pre	Recall	F1	C	Om	UA	WA	WOb	WOr
Human	-	85.9	82.6	65.5	73.1	-	-	-	-	-	-
GPT-4o	8f	57.9	17.5	25.6	20.8	75.0	26.8	70.5	9.6	23.3	13.6
GPT-4o-mini	8f	28.5	9.1	45.5	15.1	31.8	3.6	1.6	30.6	15.6	46.9
Gemini-2.5-flash	8f	51.4	11.1	16.5	13.3	68.3	8.9	32.8	13.4	13.3	12.4
Open-Source 7B / 8B
LLaVA-OneVision	8f	64.5	5.5	3.3	4.2	92.2	1.8	19.7	0	0	0

Table 4: Performance of multiple-choice VQA. C, Om, UA, WA, WOb, WOr means Accuracy of Correct, Omission, Unintended and Unnecessary Action, Wrong Action, Wrong Object, and Wrong Order, respectively. Other not mentioned error types are all zero. Green arrows indicate improvements and relative percentages over the baseline model. Gray bars show Ego-ADR results, where darker gray denotes state-of-the-art performance under the same parameter count and input frames. Details of Human performance shown in Appendix C .

Method	Level	TP	Precision	Recall	F1
Qwen2-VL	Deep	172 $\uparrow$	11.2 $\uparrow$	67.5 $\uparrow$	19.3 $\uparrow$
	Shallow	84	10.6	21.1	14.1
	CoT	75	8.2	21.7	11.9
Qwen2.5-VL	Deep	183 $\uparrow$	11.3 $\uparrow$	80.3 $\uparrow$	19.8 $\uparrow$
	Shallow	76	7.1	22.7	10.8
	CoT	134	8.2	77.9	14.8

Table 5: Ablation experiments to demonstrate the necessity of adaptive decoupling, where Deep denotes deep decoupling and Shallow denotes shallow decoupling. Green arrows indicate the best performance.

Match	Narration	Precision	Recall	F1
${\color[rgb]{1,0,0}\times}$	${\color[rgb]{1,0,0}\times}$	10.1	97.3	18.2
${\color[rgb]{1,0,0}\times}$	✓	11.1	78.7	19.4
✓	${\color[rgb]{1,0,0}\times}$	12.4	24.3	16.5
✓	✓	11.3	80.3	19.8

Table 6: Ablation experiments to demonstrate the effectiveness of decoupled reasoning, where Match denotes key-step match and Narration denotes video narration.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: The quantities corresponding to the 31 tasks included in EgoErrorVQA are shown, with the values on the left indicating the number of QA-pairs in Open-end VQA, and the right representing the number of samples, i.e., QA-pairs, in Multiple-choice VQA.

Figure 5: Word cloud of EgoErrorVQA dataset visualizes the relative frequencies of words.

Figure 6: Examples of each Error Type in four selected datasets.

Qwen3-VL-8B-Instruct	Video-LLaVA	EgoGPT	Vinci
Q: What mistake did I make when using the switch? A: You connected switch S2 and lamp L1 in parallel and then wired S2 into the series circuit of the motor and battery box… GT: I used switch S1 instead of switch S2, which is incorrect based on the instructions. Qwen: 2 Qwen Reason: The answer describes a different mistake than the one provided in the ground truth. It suggests using S1 instead of S2, which is not mentioned in the model’s answer. Deepseek: 4 Deepseek Reason: The model’s answer is mostly correct, but it could be more clear and concise. It provides a detailed explanation of the mistake, but it could be more concise for a general audience.	Q: Did I use the wrong part when I attach track to chassis? A: Yes, you did. You used a wheel instead of a track. GT: No, I used the correct part. Qwen: 1 Qwen Reason: The model’s answer is completely incorrect as it states that a wheel was used instead of a track, while the ground truth indicates that the correct part was used. Deepseek: 2 Deepseek Reason: The model’s answer is partially correct. It correctly identifies that a wheel was used instead of a track, but it also contains a minor error by stating that the wheel was used instead of a track. The ground truth is that the correct part was used, so the model’s answer is mostly wrong.	Q: What went wrong when I tried to detach the bumper from the body? A:The screw was not properly loosened, making it difficult to remove. GT: This action was unnecessary because the bumper was already properly attached to the body. Qwen: 1 Qwen Reason: The model’s answer does not match the ground truth. The ground truth states that detaching the bumper was unnecessary, while the model suggests a specific technical issue with the screw. Deepseek: 3 Deepseek Reason: The model’s answer is partially correct. It correctly identifies the issue with the screw, but it does not fully address the ground truth that the bumper was already properly attached to the body.	Q: Did I discard both ends of the tortilla without any leftovers? A:You did discard both ends of the tortilla without any leftovers. GT: Yes, I discarded both ends of the tortilla completely. Qwen: 5 Qwen Reason: The model’s answer is a perfect match to the ground truth. Deepseek: 4 Deepseek Reason: The model’s answer is mostly correct, but it lacks the specific detail of discarding both ends of the tortilla completely. It could be interpreted as partially correct or mostly correct.

Appendix

Table 7: Performances on Open-end VQA is presented along with its scoring results.

Figure 7: Performance comparison among different agents and models under four metrics in Multiple-choice VQA evaluation.

Coefficient	A vs B	A vs C	B vs C	All
Cohen’s Kappa	0.716	0.771	0.646	0.711
Fleiss’ Kappa	-	-	-	0.710
Pearson	0.950	0.959	0.934	-

Appendix

Table 8: Results of three coefficients calculated among annotators A, B, and C. Fleiss’ Kappa represents the overall agreement of the three annotators’ ratings.

Figure 8: A heatmap of open-end VQA evaluation results.

Figure 9: Heatmaps of the Pearson and Spearman correlations, intuitively illustrating the inter-rater relationships. Spearman correlation is a non-parametric statistic that quantifies the strength and direction of a monotonic association between two variables. In addition, it shows the score distribution for each rater, where Human Final Score denotes the final score representing the human evaluation standard obtained via the voting mechanism described above, and LLM Judges Avg Score denotes the average score of the two judge models.

VLM Reasoning

Vision-language model reasoning studies how multimodal systems use visual evidence and language to answer questions, plan, and explain.

See all 853 papers

EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent
Jul 15, 2026Junlong Li, Junxi Li, Yuxiang Yang +3Egocentric Video QA Video QA
The Hong Kong Polytechnic University Hong Kong, China
AgentVQA: A Unified Benchmark for Agentic Visual Understanding
Oct 22, 2026Advait Gupta, Sachin Konan, Tom Yin +2VLM Evaluation Agent Benchmarks
University of Maryland, College Park · Princeton University
Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA
May 14, 2026Guanhua Chen, Yutong Yao, Shenghe Sun +5Visual Question Answering Multimodal Reasoning
NLP2CT Lab, Department of Computer and Information Science, University of Macau · Centre for Cognitive and Brain Sciences, University of Macau · Department of Electrical and Computer Engineering, University of Macau

Jul 15, 2026J/K move · Enter open · S save

EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent

Junlong Li, Junxi Li, Yuxiang Yang, Wenbin Zou+2

The Hong Kong Polytechnic University Hong Kong, China

Egocentric Video QA Video QA Egocentric Video Understanding LLM Agent Skill Learning AI Agent Benchmarks

Open paper AskPDF arXiv

Most daily activities are inherently procedural. However, existing evaluations for egocentric video understanding seldom address procedural understanding and largely overlook complex key-step-level reasoning under the widely used video question answering (VQA) paradigm for MLLMs. Such capabilities are crucial for building procedural AI assistants deployable on wearable devices. To bridge this gap, we introduce the Egocentric Procedural Understanding VQA task (EgoProceVQA), which systematically evaluates egocentric procedural reasoning abilities of current MLLMs and agents through six types of key-step-centric questions. Furthermore, we develop EgoProceGen, a data generation platform that efficiently constructs QA data tailored to different question types. Based on this platform, we build a benchmark with 3,600 questions, four common procedural scenarios, and 31 everyday procedural tasks. Evaluations on EgoProceVQA show that existing MLLMs and agents still have substantial room for improvement in procedural understanding. Therefore, we further propose EgoProceAgent, a self-skill-exploration agentic framework. We design a generic tool library for procedural understanding and a standardized sub-skill library shared across tools and models, enabling self-exploration without ground-truth supervision. By exploring how to compose and select sub-skills, the agent discovers effective skill strategies for diverse problems, and attains state-of-the-art performance among open-source models on multiple tasks. Together, our benchmark, generation platform, and agentic framework establish a unified foundation for EgoProceVQA. Project page: https://z1oong.github.io/EgoProceVQA/.

Abstract

VLM Reasoning

Explore similar work

EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA