Memory-Guided View Refinement for Dynamic Human-in-the-loop EQA

Xin Lu; Rui Li; Xun Huang; Weixin Li; Chuanqing Zhuang; Jiayuan Li; Zhengda Lu; Jun Xiao

cs.CVMar 10, 2026

Memory-Guided View Refinement for Dynamic Human-in-the-loop EQA

Authors: Xin Lu, Rui Li, Xun Huang, Weixin Li, Chuanqing Zhuang, Jiayuan Li, Zhengda Lu, Jun Xiao,

Organizations: University of Chinese Academy of Sciences, China · Zhongguancun Academy, China · Beihang University, China · Ministry of Education of China, Xiamen University, China ·

Abstract

Embodied Question Answering (EQA) has traditionally been evaluated in temporally stable environments where visual evidence can be accumulated reliably. However, in dynamic, human-populated scenes, human activities and occlusions introduce significant perceptual non-stationarity: task-relevant cues are transient and view-dependent, while a store-then-retrieve strategy over-accumulates redundant evidence and increases inference cost. This setting exposes two practical challenges for EQA agents: resolving ambiguity caused by viewpoint-dependent occlusions, and maintaining compact yet up-to-date evidence for efficient inference. To enable systematic study of this setting, we introduce DynHiL-EQA, a human-in-the-loop EQA dataset with two subsets: a Dynamic subset featuring human activities and temporal changes, and a Static subset with temporally stable observations. To address the above challenges, we present DIVRR (Dynamic-Informed View Refinement and Relevance-guided Adaptive Memory Selection), a training-free framework that couples relevance-guided view refinement with selective memory admission. By verifying ambiguous observations before committing them and retaining only informative evidence, DIVRR improves robustness under occlusions while preserving fast inference with compact memory. Extensive experiments on DynHiL-EQA and the established HM-EQA dataset demonstrate that DIVRR consistently improves over existing baselines in both dynamic and static settings while maintaining high inference efficiency.

Figures & tables

Figure 1: Overview of the evidence verification process in DIVRR . Numerical values denote question-conditioned relevance scores $s_{t}$ . Purple boxes indicate observations admitted into the Relevance-guided Memory via Adaptive Admission Control . At waypoint 7, DIVRR triggers View Refinement and performs multi-view augmentation to select the most relevant viewpoint before memory commitment. The final answer is then generated by conditioning on the compact memory, successfully grounding the response to the question.

Dataset	Simulator	Source	Real Scenes	Dynamic Scenes	GT observation	Creation
EQA-v1 [ 10 ]	House3D	SUNCG	✗	✗	✗	Rule-Based
MP3D-EQA [ 33 ]	MINOS	MP3D	✓	✗	✓	Rule-Based
MT-EQA [ 40 ]	House3D	SUNCG	✗	✗	✓	Rule-Based
IQA [ 13 ]	AI2THOR	–	✗	✗	✗	Rule-Based
EXPRESS-Bench [ 17 ]	Habitat	HM3D	✓	✗	✓	VLMs
HM-EQA [ 24 ]	Habitat	HM3D	✓	✗	✗	VLMs

Table 1: Comparison of our dataset with existing EQA datasets.

Figure 2: The construction process of the DynHiL-EQA dataset.

Figure 3: Overview of the DynHiL-EQA dataset statistics.

Figure 4: The architectural overview of the DIVRR framework, illustrating the coupling of relevance-guided view refinement via multi-view augmentation and selective memory admission.

Figure 5: Comparison of exploration trajectories. Unlike coverage-centric strategies, DIVRR actively interrogates task-relevant viewpoints to resolve occlusions, leading to verified evidence acquisition.

	all			Dynamic			Static
Method	Accu(%) $\uparrow$	Mem $\downarrow$	Time $\downarrow$	Accu(%) $\uparrow$	Mem $\downarrow$	Time $\downarrow$	Accu(%) $\uparrow$	Mem $\downarrow$	Time $\downarrow$
Explore-EQA	46.3	–	5.247	42.0	–	5.501	50.5	–	4.993
Fine-EQA	48.1	–	5.970	45.0	–	5.795	51.3	–	6.145
Graph-EQA	49.2	21.6	13.576	44.2	17.3	14.234	54.2	25.8	12.917
MemoryEQA	41.2	56.0	27.767	29.8	73.6	30.133	52.5	38.5	24.827
DIVRR	56.6	9.5	5.745	55.1	4.5	5.700	58.2	14.4	5.790

Table 2: Comparative analysis on DynHiL-EQA splits ( all / Dynamic / Static ). Accuracy (

\uparrow

), Memory (

\downarrow

), and Latency (

\downarrow

). DIVRR achieves a strong accuracy–efficiency trade-off under occlusions and non-stationary observations.

Method	Accu	Time	Mem	Step
Explore-EQA	54.6	4.36	–	0.75
Fine-EQA	58.2	5.38	–	0.70
Graph-EQA	60.4	9.14	6.04	0.40
MemoryEQA	56.6	27.22	33.26	0.60
DIVRR	63.8	4.66	2.51	0.64

Table 3: Zero-shot performance on the static HM-EQA dataset. Accuracy (

\uparrow

), Latency (

\downarrow

), Memory (

\downarrow

), and Sensing Steps (

\downarrow

Category	Variant	all		Dynamic		Static
		Accu(%) $\uparrow$	Mem $\downarrow$	Accu(%) $\uparrow$	Mem $\downarrow$	Accu(%) $\uparrow$	Mem $\downarrow$
Module Ablation	baseline	46.3	–	42.0	–	50.5	–
	Base+AM	51.9	6.5	47.6	2.6	56.0	10.4
	Base+AM+VR	53.5	10.3	50.6	5.4	55.3	15.2
	Base+AM+VR+RD (full)	56.6	9.5	55.1	4.5	58.2	14.4
Exploration Strategy	RE	33.9	10.1	33.2	5.1	34.5	14.5

Table 4: Ablation of DIVRR modules, exploration backbones, and VLM backbones on DynHiL-EQA. Splits: all / Dynamic / Static .

Figure 6: Perceptual performance profile: Accuracy (left) and Intersection (right) across question categories on DynHiL-EQA.

Embodied QA

Embodied question answering studies how agents use perception, memory, and action to answer questions grounded in the physical world.

See all 39 papers

Informative Viewpoint Selection for Episodic-Memory Embodied Question Answering using Omnidirectional Images
Sep 27, 2026Kaname Kitamura, Asako KanezakiEmbodied QA Open-Domain QA
Department of Computer Science, Institute of Science Tokyo, Tokyo, Japan · RIKEN, Japan · Tohoku University, Sendai, Japan
FAST-EQA: Efficient Embodied Question Answering with Global and Local Region Relevancy
Feb 17, 2026Haochen Zhang, Nirav Savaliya, Faizan Siddiqui +1Embodied QA Embodied Navigation
Carnegie Mellon University, USA · Honda Research Institute USA
Hierarchical Floorplan-Guided Vision-Language Exploration for Embodied Question Answering
Sep 22, 2026Albert Gassol Puigjaner, Kostas AlexisEmbodied Navigation Embodied QA
Autonomous Robots Lab, Norwegian University of Science and Technology (NTNU), Trondheim, Norway

Sep 27, 2026J/K move · Enter open · S save

Informative Viewpoint Selection for Episodic-Memory Embodied Question Answering using Omnidirectional Images

Kaname Kitamura, Asako Kanezaki

Department of Computer Science, Institute of Science Tokyo, Tokyo, Japan · RIKEN, Japan · Tohoku University, Sendai, Japan

Embodied QA Open-Domain QA

Open paper AskPDF arXiv

Embodied Question Answering (EQA) requires agents to answer natural language questions about surrounding environments from visual observations. In this work, we focus on open-vocabulary episodic-memory EQA (EM-EQA), where an agent answers free-form questions using recorded observation histories. Omnidirectional images are promising for this task, as they provide wide field-of-view observations that can capture surrounding context without requiring explicit camera rotations. However, omnidirectional images introduce two challenges for EQA: (i) equirectangular projection causes severe geometric distortion that degrades vision-language model (VLM) recognition accuracy, and (ii) feeding equirectangular images directly into VLMs introduces excessive irrelevant background information, reducing answer accuracy and increasing the visual-token burden. To address these challenges, we propose a viewpoint selection method for EM-EQA using omnidirectional images. Our method converts equirectangular observations into perspective views via cubemap projection, estimates question-conditioned relevance with fine-tuned BLIP-2, and selects informative and diverse viewpoints through diversity-aware greedy selection. Experiments on the Habitat-Matterport 3D (HM3D) subset of OpenEQA show that our method achieves state-of-the-art model performance among the reported model results with equirectangular observations. Moreover, after removing rotation views, which reduces observation frames by 65.5%, our method largely maintains its answer accuracy.

Abstract

Embodied QA

Explore similar work

Informative Viewpoint Selection for Episodic-Memory Embodied Question Answering using Omnidirectional Images

FAST-EQA: Efficient Embodied Question Answering with Global and Local Region Relevancy

Hierarchical Floorplan-Guided Vision-Language Exploration for Embodied Question Answering