ObjChangeVR: Object State Change Reasoning from Continuous Egocentric Views in VR Environments

Shiyi Ding; Shaoen Wu; Ying Chen

cs.CVFeb 27, 2026

ObjChangeVR: Object State Change Reasoning from Continuous Egocentric Views in VR Environments

Authors: Shiyi Ding, Shaoen Wu, Ying Chen

Organizations: Kennesaw State University · Pennsylvania State University

Abstract

Recent advances in multimodal large language models (MLLMs) offer a promising approach for natural language-based scene change queries in virtual reality (VR). Prior work on applying MLLMs for object state understanding has focused on egocentric videos that capture the camera wearer's interactions with objects. However, object state changes may occur in the background without direct user interaction, lacking explicit motion cues and making them difficult to detect. Moreover, no benchmark exists for evaluating this challenging scenario. To address these challenges, we introduce ObjChangeVR-Dataset, specifically for benchmarking the question-answering task of object state change. We also propose ObjChangeVR, a framework that combines viewpoint-aware and temporal-based retrieval to identify relevant frames, along with cross-view reasoning that reconciles inconsistent evidence from multiple viewpoints. Extensive experiments demonstrate that ObjChangeVR significantly outperforms baseline approaches across multiple MLLMs.

Figures & tables

Figure 1: Illustration of the question-answering task for object state change reasoning. Given a query frame and a question about object change, we retrieve several relevant frames from the egocentric frame sequence and leverage visual evidence from the retrieved frames to produce an answer and an explanation.

Figure 2: Overview of the ObjChangeVR-Dataset and the proposed ObjChangeVR framework.

VR scene	Scale	# scene sections	# target objects
Villa interior	Small	9	421
Restaurant	Small	7	62
Market	Small	5	52
Museum	Large	6	134
Viking village	Large	8	60

Table 1: Statistics of ObjChangeVR-Dataset.

Model	Method	Short traj.			Long traj.			All traj.
		EM @0.8	Macro F1	Weighted F1	EM @0.8	Macro F1	Weighted F1	EM @0.8	Macro F1	Weighted F1
GPT-4o	Caption-CLIP	0.529	0.595	0.588	0.528	0.522	0.537	0.529	0.574	0.570
	Image-CLIP	0.616	0.649	0.644	0.558	0.572	0.585	0.592	0.619	0.620
	Viewpoint-Retrieval	0.623	0.659	0.657	0.570	0.584	0.599	0.601	0.631	0.635
	ObjChangeVR	0.822	0.830	0.837	0.652	0.661	0.669	0.754	0.770	0.774
GPT-4o mini	Caption-CLIP	0.416	0.395	0.357	0.537	0.449	0.468	0.464	0.420	0.400

Table 2: Comparison of ObjChangeVR and different relevant frame retrieval methods across three MLLMs.

Model	Method	Short traj.			Long traj.			All traj.
		EM @0.8	Macro F1	Weighted F1	EM @0.8	Macro F1	Weighted F1	EM @0.8	Macro F1	Weighted F1
GPT-4o	CoT-SC	0.754	0.745	0.754	0.607	0.623	0.631	0.695	0.702	0.708
	ObjChangeVR w/o TCV	0.745	0.737	0.747	0.590	0.611	0.620	0.683	0.699	0.700
	ObjChangeVR	0.822	0.830	0.837	0.652	0.661	0.669	0.754	0.770	0.774
GPT-4o mini	CoT-SC	0.537	0.536	0.518	0.524	0.536	0.549	0.536	0.545	0.532
	ObjChangeVR w/o TCV	0.527	0.528	0.511	0.536	0.536	0.549	0.531	0.541	0.529

Table 3: Comparison of ObjChangeVR and variants with different reasoning methods across three MLLMs.

Metric	Case	CoT-SC	ObjChangeVR
[email protected]	Cons.	0.748	0.795
	Incons.	0.637	0.709
	All	0.695	0.754
Macro-F1	Cons.	0.520	0.749
	Incons.	0.551	0.624
	All	0.702	0.770

Table 4: Performance of ObjChangeVR under consistent and inconsistent intermediate answers.

Metric	Traj.	$k$
Metric	Traj.	1	2	3	5	7	9
EM @0.8	Short	0.691	0.774	0.822	0.776	0.711	0.672
	Long	0.542	0.638	0.652	0.598	0.554	0.532
	All	0.631	0.720	0.754	0.705	0.648	0.616
Macro- F1	Short	0.787	0.832	0.830	0.823	0.812	0.801
	Long	0.612	0.656	0.661	0.656	0.648	0.651

Table 5: Performance of ObjChangeVR across different numbers of retrieved frames (

k

Variant	Zero-shot	Few-shot
EM	0.731	0.754
Macro-F1	0.734	0.770
Weighted-F1	0.740	0.774

Table 6: Comparison between the ObjChangeVR variants with zero-shot and few-shot prompting.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Scene	Scale	Scene section
Villa interior	Small	First-floor kitchen, first-floor dining room, first-floor living room, first-floor balcony, first-floor lounge, first-floor storage room, second-floor bathroom, second-floor master bedroom with bed and wardrobe, second-floor study room with bookshelf and desk
Restaurant	Small	Dining area, men’s restroom, women’s restroom, entrance, checkout counter, pickup area, bar counter
Market	Small	Fish shop, dessert shop, butcher shop, weapon shop, vegetable stall
Museum	Large	Ground floor main exhibition hall, ground floor small exhibition room, ground floor corridor exhibition area, mezzanine staircase area, second floor corridor exhibition area, second floor balcony
Viking village	Large	Large house, small house, barrel storage, blacksmith workshop, dock, shipbuilding area, gate and tower, fenced area with multiple items

Appendix

Table 7: Scene sections in ObjChangeVR-Dataset.

Dataset	Virtual env.-based	Natural language Q&A	Egocentric	Object state change	Camera pose	Avg. duration	Scene type
ScanQA ( Azuma et al., 2022 )	✗	✓	✗	✗	✓	N/A	Indoor
MMScan ( Lyu et al., 2024 )	✗	✓	✗	✗	✓	N/A	Indoor
VLA-3D ( Zhang et al., 2024 )	✓	✗	✗	✗	✗	N/A	Indoor
SIMMC-VR ( Wu et al., 2023 )	✓	✓	✓	✗	✓	$\sim$ 2 min	Indoor
ChangeSim ( Park et al., 2021 )	✓	✗	✗	✓	✓	N/A	Indoor
Causal-VidQA ( Li et al., 2022 )	✗	✓	✗	✓	✗	$>$ 9 s	Indoor/Outdoor

Appendix

Table 8: Comparison of the ObjChangeVR-Dataset with representative 3D and video-based benchmarks.

Method	Caption-CLIP	Image-CLIP	Viewpoint-Retrieval	CoT-SC	ObjChangeVR w/o TCV	ObjChangeVR
[email protected]	0.5302	0.5934	0.6026	0.6958	0.6840	0.7588
[email protected]	0.5302	0.5934	0.6026	0.6958	0.6840	0.7566
[email protected]	0.5286	0.5924	0.6014	0.6950	0.6828	0.7540
[email protected]	0.5214	0.5860	0.5926	0.6774	0.6656	0.7368
[email protected]	0.5080	0.5678	0.5738	0.6626	0.6508	0.7182

Appendix

Table 9: Sensitivity analysis of EM@

\tau

under different similarity thresholds.

Method	[email protected]	Macro-F1	Weighted-F1
ObjChangeVR	0.8200	0.8070	0.8307

Appendix

Table 10: Performance of ObjChangeVR on the object addition dataset.

Figure 3: Proportion of questions (out of 5,000) with consistent and inconsistent intermediate answers across different

k

Method	Caption-CLIP	Image-CLIP	Viewpoint-Retrieval	CoT-SC	ObjChangeVR w/o TCV	ObjChangeVR
Retrieval (s)	0.035	0.261	0.003	0.004	0.004	0.003
Reasoning (s)	2.383	3.372	3.194	8.249	2.814	9.562
Total (s)	5.887	3.653	3.323	8.256	2.821	9.566

Appendix

Table 11: Inference latency breakdown of different methods.

Method	CI95% ([email protected])
Caption-CLIP	[0.5140, 0.5424]
Image-CLIP	[0.5788, 0.6064]
Viewpoint-Retrieval	[0.5886, 0.6158]
CoT-SC	[0.6812, 0.7076]
ObjChangeVR w/o TCV	[0.6696, 0.6950]
ObjChangeVR	[0.7428, 0.7670]

Appendix

Table 12: 95% confidence intervals (CI95%) of [email protected] obtained via 1,000-sample bootstrap resampling over all question–answer pairs.

Panel of Table 13 — Table 13: Instruction of question-answer pair generation.

Panel of Table 14 — Table 14: An example of question-answer pair generation.

Panel of Table 15 — Table 15: A case of consistent intermediate reasoning.

Panel of Table 16 — Table 16: A case of inconsistent intermediate reasoning.

Visual Reasoning

Visual reasoning studies how AI systems interpret and combine visual evidence to solve spatial, relational, compositional, and physical problems.

See all 162 papers

StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning
May 5, 2026Xiaowen Sun, Matthias Kerzel, Mengdi Li +3Physical Reasoning VLM Adaptation
Department of Informatics, University of Hamburg, Germany · King Abdullah University of Science and Technology, Saudi Arabia
EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning
Oct 3, 2026Yutong Li, Molin Wang, Xiaotong Li +3Video Reasoning VLM Evaluation
Fudan University, Shanghai, China · Xi’an Jiaotong-Liverpool University, Suzhou, China
SAVER: Selective Auditing of Verbal Evidence for Error Recovery in VLM Change Reasoning
Aug 24, 2026Youdi LiSelf-Correction in VLMs VLM Reasoning
Panasonic Connect Co., Ltd. Tokyo, Japan

May 5, 2026J/K move · Enter open · S save

StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning

Xiaowen Sun, Matthias Kerzel, Mengdi Li, Xufeng Zhao+2

Department of Informatics, University of Hamburg, Germany · King Abdullah University of Science and Technology, Saudi Arabia

Physical Reasoning VLM Adaptation Robotic Affordance Inference VLMs for Robotics

Open paper AskPDF arXiv

Vision-language models have demonstrated strong performance across robotic perception and instruction-following tasks. However, they still struggle with precise spatial reasoning, particularly in predicting object locations and fine-grained object states. We propose StateVLM, a vision-language model designed to learn fine-grained object representations, including object localization and grasp-relevant region prediction. We introduce a joint training objective that integrates an auxiliary regression loss (ARL) with the standard causal language modeling (CLM) objective to improve numerical reasoning and spatial understanding. To evaluate whether models can move beyond category-level grounding toward state-aware spatial understanding, we introduce an open-source benchmark, Object State Affordance Reasoning (OSAR), comprising 1,172 scenes with 7,746 individual objects and their corresponding bounding boxes. Empirical experiments on the Referring Expression Comprehension datasets demonstrate that integrating ARL improves model performance compared with CLM only. Experiments on the OSAR benchmark further demonstrate that StateVLM with ARL achieves an average improvement of 5.2% over models trained with CLM only. These results show that ARL is particularly beneficial for the complex affordance reasoning tasks that require object state understanding. The joint training objective in StateVLM demonstrates that integrating an auxiliary regression objective into VLM training improves numerical reasoning, and the OSAR benchmark provides a new testbed for understanding object state affordances in robotics.

Abstract

Visual Reasoning

Explore similar work

StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning

EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning

SAVER: Selective Auditing of Verbal Evidence for Error Recovery in VLM Change Reasoning