JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments

Zhixi Cai; Fucai Ke; Sukai Huang; Maria Garcia de la Banda; Peter J. Stuckey; Gholamreza Haffari; Hamid Rezatofighi

cs.AISep 28, 2026

JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments

Authors: Zhixi Cai, Fucai Ke, Sukai Huang, Maria Garcia de la Banda, Peter J. Stuckey, Gholamreza Haffari, Hamid Rezatofighi

Organizations: Monash University

Abstract

In complex embodied visual reasoning scenarios, an agent often has only a limited field of view, and the evidence needed to answer a question may be distributed across time, viewpoint, and interacting objects. A model may therefore give a plausible answer without ever observing the relevant object, time, or view that supports it. Current visual reasoning benchmarks largely evaluate passive observations and final answers, overlooking settings that require active reasoning and evidence acquisition. We introduce JRDB-AVR, a benchmark derived from existing real-world JRDB robotics data through a structured question-generation engine that turns this gap into an explicit evaluation: an embodied agentic system receives a visual reasoning question, requests bounded observations by timestamp and viewing angle, and is evaluated on both the final answer and the grounded visual evidence supporting it. The benchmark contains diverse questions over multiple real-world environments involving temporal search, viewpoint selection, and human-oriented compositional reasoning. We also introduce JRDB-AVR-Agent, a reference active reasoning agentic method that maintains an explicit observation-grounded graph-based world model and answers through solving. Experiments reveal a substantial gap between answer accuracy and evidence accuracy in current baselines, showing that current VLMs can produce unsupported correct answers and that active evidence-aware evaluation is necessary for embodied visual reasoning. Code and benchmark are available at https://github.com/ControlNet/JRDB-AVR.

Figures & tables

Figure 1: Overview of JRDB-AVR. Unlike passive visual reasoning, where relevant image/video observations are pre-selected and only the final answer is evaluated, JRDB-AVR requires an agent to actively observe across time and view angle, then evaluates both the answer and visual evidence.

Benchmark	Temporal	Spatial/View	Domain	Camera	Active Obs.	Evidence	Evaluation
GQA ( Hudson and Manning, 2019 )	$\times$	✓	Real	Static	$\times$	$\times$	Answer
CLEVRER ( Yi et al., 2019 )	✓	$\times$	Sim	Static	$\times$	$\times$	Answer
STAR ( Wu et al., 2021 )	✓	✓	Real	Static	$\times$	$\times$	Answer
MindCube ( Wang et al., 2025 )	$\times$	✓	Real	Static	$\times$	$\times$	Answer
VIEW2SPACE ( Ke et al., 2026 )	$\times$	✓	Sim	Static	$\times$	BBox	Answer
JRDB-Reasoning ( Jahangard et al., 2026 )	✓	✓	Real	Moving	$\times$	$\times$	Answer

Table 1: Comparison with related visual reasoning benchmarks. We compare whether each benchmark requires temporal reasoning, spatial or viewpoint reasoning, the data domain and camera setting, whether observations can be actively selected, the form of localized visual evidence, and the final evaluation target. Time + View denotes active selection of both timestamp and viewing angle, while Time + BBox denotes evidence localized by timestamp and target bounding box.

Figure 2: Dataset generation pipeline for JRDB-AVR. Starting from JRDB panoramic videos and annotations, we organize object boxes, tracks, poses, relations, social groups, actions, etc into temporal scene graphs. Question generators search these scene graphs for active reasoning patterns and produce candidate questions, which are retained only after they pass evidence, necessity, uniqueness, and manual quality checks.

Name	Description	Type	Evaluation Protocol
Chain 2-hop	Ask about the target with 2-hop entity relations.	Multiple choice	Correct if the selected choice matches the ground truth.
Chain 3-hop	Extend the relational path to three hops before querying the target.
Chain fork-join	Merge two relational branches from a shared entity before querying the target.
Unique anchor	Locate a target from conditions and ask about a later state or action in another temporal location.
Long range	Track temporally distant consequences or long-range dependencies.
Chain hybrid	Combine a spatial relation with temporal and viewpoint search to locate the target timestamp and view angle.	Timestamp and viewpoint	Correct if the predicted timestamp is within $\pm 1$ second of the ground truth and the wrapped viewpoint error is within $\pm 20^{\circ}$ .

Table 2: For each of the seven generators currently present in JRDB-AVR, the table provides a brief description of the generator, its question type and the metrics protocol of each question type.

Figure 3: Pipeline of JRDB-AVR-Agent. The question is first converted into a structured graph plan that specifies the anchor, intermediate, and target entities. During interaction, the agent selects tool actions from the current state, requests observations when needed, and writes observation-grounded entities, attributes, and relations into an explicit world model. The final answer is produced by solving the graph state and returning both the answer and supporting visual evidence.

Backbone VLM	Method	Answer $\uparrow$	Evidence $\uparrow$	Combined $\uparrow$	Hallucination $\downarrow$
Gemma-4 E2B (5B)	Monolithic VLM	29.12	0.05	0.05	99.83
	Chain-of-Thought	25.31	0.05	0.05	99.81
	Search-Recognize-Pipeline	26.69	4.00	0.81	96.96
	ReAct	29.50	5.67	1.91	93.54
Gemma-4 E4B (8B)	Monolithic VLM	29.50	0.14	0.14	99.52
	Chain-of-Thought	31.22	0.29	0.10	99.69

Table 3: Quantitative comparison on the JRDB-AVR. All scores are percentages. Hallucination rate is the conditional rate of answer-correct predictions without correct evidence, computed as

(\mathrm{Answer}-\mathrm{Combined})/\mathrm{Answer}

, where lower is better.

Active	World Model	Answer $\uparrow$	Evidence $\uparrow$	Combined $\uparrow$	Hallucination $\downarrow$
$\times$	$\times$	25.31	6.86	2.72	89.25
$\times$	✓	26.83	8.06	3.62	86.51
✓	$\times$	37.66	18.22	9.20	75.57
✓	✓	38.51	27.50	14.54	62.25

Table 4: Ablation study of JRDB-AVR-Agent. All scores are percentages. Active denotes active observation. World Model denotes the explicit observation-grounded world model. Hallucination is as before.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Count	Template(s)	Answer mode	Active reasoning requirement
Action boundary	67	start, span	Frame identification / temporal localization	Localize the onset of an action or recover its full temporal interval instead of answering from a single static frame.
Chain 2-hop	76	action	Multiple choice	Follow a unique two-hop relational path from the anchor person to the final target before reading out the answer.
Chain 3-hop	222	action	Multiple choice	Extend the path to three hops, so the answer depends on correctly grounding an additional intermediate person.
Chain fork-join	32	action	Multiple choice	Ground two branches from the same anchor and merge them at a shared target, requiring branch consistency rather than a single linear chain.
Chain hybrid	51	action search	Frame-viewpoint pair	Combine a spatial chain with temporal search and viewpoint selection, then return both the decisive frame and the best supporting viewpoint.
Long range	30	chain, consequence, gap	Multiple choice / numeric value	Recover a target over a larger temporal gap and either predict a later action/consequence or a temporally defined numeric gap.

Appendix

Table 5: Generator details for the seven VQA generator families used in JRDB-AVR. Counts refer to the selected 2,098-question benchmark used in the paper.

Generator	Count	Answer $\uparrow$	Evidence $\uparrow$	Combined $\uparrow$	Hallucination $\downarrow$
Action boundary	67	37.31	56.72	23.88	36.00
Chain family	330	41.82	8.48	5.45	86.96
Chain hybrid	51	11.76	5.88	0.00	100.00
Long range	30	50.00	3.33	0.00	100.00
Unique anchor	1,620	38.52	31.30	16.73	56.57

Appendix

Table 6: JRDB-AVR-Agent breakdown by high-level generator family. All scores are percentages. Hallucination rate is computed as

(\mathrm{Answer}-\mathrm{Combined})/\mathrm{Answer}

, where lower is better.

Explore similar work

ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI
Jun 16, 2026Hong Yang, Basura FernandoEmbodied Artificial Intelligence Diagnostic Benchmark
Centre for Frontier AI Research Agency for Science, Technology and Research Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore
Position: The Systemic Lack of Agency in Visual Reasoning
Jun 11, 2026Yizhao Huang, Haoyang Chen, Shiqin Wang +6Visual Reasoning Multiple Vision Tasks
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University · Hubei Key Laboratory of Multimedia and Network Communication Engineering · Zhongguancun Academy, Beijing, China. +2
Learning Adaptive Reasoning Paths for Efficient Visual Reasoning
Apr 16, 2026Yixu Huang, Tinghui Zhu, Muhao ChenVisual Reasoning Recent Vision-Language Models
♠Fudan University · ♣University of California, Davis

Jun 16, 2026, cs.ROJ/K move · Enter open · S save

ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI

Hong Yang, Basura Fernando

Centre for Frontier AI Research Agency for Science, Technology and Research Singapore · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore

Embodied Artificial Intelligence Diagnostic Benchmark Recent Vision-Language Models

Open paper AskPDF arXiv

Generalist embodied agents require more than object recognition: they must reason about spatial relations, actions, procedures, human intentions, environmental constraints, and commonsense consequences from situated visual observations. Yet existing visual and embodied question answering benchmarks often provide limited control over the reasoning dependencies being tested, making it difficult to distinguish grounded embodied reasoning from shortcut-driven visual or linguistic pattern matching. We present ERQA-Plus, a diagnostic benchmark for reasoning in embodied AI. ERQA-Plus contains 1,766 question-answer instances grounded in 711 robot-centric images and organized according to a structured taxonomy spanning perceptual, action-centric, social-interaction, navigation-environmental, and contextual commonsense reasoning. The dataset is constructed using a multi-stage generation and validation pipeline that combines taxonomy-guided question generation, automatic quality judging, iterative revision, and human assessment to improve visual grounding, answer validity, and reasoning quality. We benchmark representative general-purpose vision-language models and embodied models, including LLaVA-NeXT-8B, Prismatic-7B, MiniCPM-V-4.5-8B, Qwen3-VL, RoboRefer-8B, and RoboBrain2.5-8B. Although the strongest model, Qwen3-VL-32B, achieves 83.4% overall accuracy and 61.4 SBERT score, category-level results reveal persistent weaknesses in spatial reasoning, procedural reasoning, event prediction, and intention inference. ERQA-Plus therefore provides a fine-grained evaluation framework for measuring not only whether embodied agents answer correctly, but also which forms of embodied reasoning they can and cannot perform reliably. The dataset is available https://huggingface.co/datasets/huggingdas/erqa-plus and the project page at https://github.com/LUNAProject22/erqa-plus.

Abstract

Explore similar work

ERQA-Plus: A Diagnostic Benchmark for Reasoning in Embodied AI

Position: The Systemic Lack of Agency in Visual Reasoning

Learning Adaptive Reasoning Paths for Efficient Visual Reasoning