Answering Without Arriving: An Evidence Audit for Embodied Question Answering

Shubh Garg; Divyansh Pandey; Niyati Rawal; Vedant Khandelwal; Amit Sheth

cs.RONov 1, 2026

Answering Without Arriving: An Evidence Audit for Embodied Question Answering

Authors: Shubh Garg, Divyansh Pandey, Niyati Rawal, Vedant Khandelwal, Amit Sheth

Organizations: Indian AI Research Organisation (IAIRO), GIFT City, Gujarat · Independent Researcher · BITS Pilani · AI Institute, UofSC (AIISC), 1112 Greene St., Columbia

Abstract

Embodied question answering asks an agent to navigate, observe, and then answer questions about what it saw, yet answer accuracy alone cannot tell whether any of this happened. A plausible answer can come from the question, the room label, or the answer choices, and a correct answer does not show that the supporting observation was ever made. We present a multi-room benchmark whose 200 test episodes and 800 questions are stratified by route extent, with an evaluation ladder that holds the questions fixed while varying only the evidence supplied to the answerer, from text-only probes through closed-loop agents to replay of the reference trajectory. Questions are generated from symbolic records of rooms, visible objects, and frames, so every answer can be traced to the observations that support it. A closed-loop vision-language agent reaches no target in 200 episodes yet answers 45.5% of questions, within 0.1 points of the same model given no images; replaying the reference trajectory raises the same model to 57.4%, so the evidence exists along the route but is not gathered. A family-level audit then shows that access to evidence is not the same as use of it. Four of eight families are largely answerable from text, one cites frames that cannot establish its own answer, and the oracle conditions reach 59.5% on Rooms-visited by counting the supplied images. Evidence acquisition, answer coverage, and shortcut sensitivity must therefore be established before an accuracy gain can be credited to memory.

Figures & tables

Figure 1: Benchmark and evaluation pipeline. (A) Episode generation produces a reference route and a privileged symbolic record; the example strip shows a bed visible at $t{=}0$ and $t{=}1$ and gone by $t{=}2$ . (B) Symbolic templates write multiple-choice questions from that strip, with the navigation objective withheld. The question shown is answerable from the full trajectory, but its cited frames $\{0,1\}$ omit the later absence, the evidence-selection defect of Section 4 . (C) Schema, target-mention, and symbolic checks link each question to the frames supporting it. (D) The ladder holds the 800 questions and their format fixed and varies only the evidence reaching the answerer, which never includes the privileged record.

Constant	Value	Effect
stop_dist	1.5 m	instance-based success radius
min_spawn_dist_mult	$3.0\times$ (4.5 m)	minimum spawn–target distance
min_spawn_dist_mult_same_room	$1.5\times$ (2.25 m)	same-room episodes
visibility_distance	6.0 m	simulator visibility cutoff
min_bbox_area_frac	0.005	minimum target frame coverage
perception_min_bbox_frac	0.0005	minimum area to log an object as visible

Table 1: Generation and execution settings. Values are recorded per episode in the released data.

Model	Revision	Used for	Params
Qwen/Qwen3-VL-4B-Instruct	ebb281ec	E0b, E0, E2, E3, E5	4B
cheryyunl/MomaGraph-R1	d447423d	E2, E4	8B
Qwen/Qwen2.5-VL-7B-Instruct	cc594898	E2	7B

Table 2: Model revisions. Inference uses greedy decoding and unquantised 16-bit weights.

Family	$n$	Opt.-only (%)	Blind (%)	Screening
Persistence	131	51.9	61.1	Probe-sensitive
Global-count	120	31.7	33.3	Low probe scores
Rooms-visited	116	32.8	28.4	Low probe scores
Existence	113	31.0	57.5	Probe-sensitive
Presence-change	102	49.0	36.3	Evidence mismatch
Room Identification	80	28.7	65.0	Probe-sensitive

Table 3: Text-only probe accuracy by question family. Probe-sensitive families show strong performance without images. The screening describes observed probe behaviour and evidence requirements.

Family	$n$	E0b	E0	E3	E2 (MG)	E2 (Qw)	E5
Persistence	131	51.9	61.1	61.1	66.4	68.7	69.5
Global-count	120	31.7	33.3	27.5	32.5	30.0	30.5
Rooms-visited	116	32.8	28.4	32.8	67.2	67.2	68.1
Existence	113	31.0	57.5	40.7	68.1	73.5	71.4
Presence-change	102	49.0	36.3	61.8	33.3	40.2	39.6
Room Identification	80	28.7	65.0	46.3	68.8	66.2	65.4

Table 4: Per-family answer accuracy (%).

n

gives the test-set family size. E0b, E0, E3, and E5 use Qwen3-VL-4B; E2 uses MomaGraph-R1 (MG) or Qwen3-VL-4B (Qw). E5 accuracy uses its scored questions. E4 answer coverage is detailed in Appendix I .

Figure 2: Rooms-visited accuracy (

n=116

). The horizontal 33.0% line is the pooled test-set chance reference. The 59.5% line is a content-free rule that answers with the number of supplied images. Oracle scores exceed this rule, but the excess does not isolate visual reasoning. E0, E3, and E5 use Qwen3-VL-4B; E2 uses MomaGraph-R1.

Rung	Metric	S1	S2	S3	S4
E3 (closed-loop)	EQA (%)	49.5	44.5	50.0	38.0
E3 (closed-loop)	$\Delta$ vs chance	$+16.2$	$+11.1$	$+17.8$	$+4.8$
E5 (oracle replay)	EQA (%)	61.5	52.5	60.0	55.5
E5 (oracle replay)	$\Delta$ vs chance	$+28.2$	$+19.1$	$+27.8$	$+22.3$
E5 $-$ E3 (points)		$+12.0$	$+8.0$	$+10.0$	$+17.5$
E3 navigation	SR	0/50	0/50	0/50	0/50

Table 5: Accuracy and navigation by route-length stratum (50 designated test episodes each).

\Delta

subtracts the stratum-specific chance value (0.322–0.334). Navigation entries give the successful and scored episodes for each condition.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Train	Val	Test
Episodes	5,000	500	200
Distinct houses	1,359	151	53
Frame cache coverage	206,899/206,899	21,982/21,982	8,335/8,335

Appendix

Table 6: Benchmark statistics. Question counts are for the 800-question test split; the per-family probe screening derived from these questions is in Table 3 .

	gold room count
cited frames	2	3	4	total
1	6	–	–	6
2	30	16	1	47
3	–	18	24	42
4	–	–	21	21
total	36	34	46	116

Appendix

Table 7: Cited-frame count against gold room count for the 116 Rooms-visited questions. The distribution never exceeds the diagonal, so the frame count is a lower bound on the answer, and equals it in 69 cases (bold). Cells are shaded by count.

Rung	Stratum	Stuck	Not vis.	Fmt. (%)	Inv. (%)
E1 (random)	S1	0	47	54	46
	S2	0	48	40	60
	S3	0	50	40	60
	S4	0	50	48	52
E3 (closed-loop)	S1	42	8	100	0
	S2	45	5	100	0

Appendix

Table 8: Termination and action-format statistics by stratum. Each stratum contains 50 designated episodes. Fmt. denotes action-format compliance; Inv. denotes the invalid-action rate. Reached, Stuck, and Not vis. are termination counts, shaded from white (0) to lilac (50).

Comparison	Shared components	Changed components
E0b / E0	Model and answer choices	Question text and target-room label
E3 / E5	Model and navigation objective	Executed trajectory and QA frame selection
E2 models	Questions and cited-frame protocol	Answering model and its input processing
E3 / E4	Simulator and instance scorer	Model, local graph input, prompts, and QA trigger
MomaGraph QA configurations	Answering model	Single/multiple frames and QA path

Appendix

Table 9: Interpretation of the principal ladder comparisons. Shared components identify the comparison’s scope; changed components define the intervention that each score comparison represents.

Rung	Model	SR	SPL	Parse	EQA (%)
E0b (options-only)	Qwen3-VL-4B	–	–	1.000	39.0
E0 (blind)	Qwen3-VL-4B	–	–	0.995	45.4
E1 (immediate-Done)	–	0/200	0.000	0.00	n/a
E1 (random action)	–	0/200	0.000	0.00	n/a
E3 (closed-loop)	Qwen3-VL-4B	0/200	0.000	1.000	45.5
E4 (spec. closed-loop)	MomaGraph-R1	0/200	0.000	0.16	n/a (42.2 on parsed)

Appendix

Table 10: Full baseline ladder. SR is navigation success rate, SPL is success weighted by path length, Parse is answer coverage, and EQA is answer accuracy. The designated set contains 200 episodes and 800 questions. E5 SR and answer metrics use replay-scored episodes and questions. E4 accuracy is shown on parsed responses.

Panel of Figure 3 — Figure 3: Training sample: frame 0 (left) and frame 1 (middle) are cited; frame 6 (right) is not. The bed appears in the cited evidence and is absent from the final view. This training example illustrates how the trajectory, question, and selected evidence are linked.

Field	Interpretation
episode_id	Episode key for navigation and per-episode metrics
question_id	Question key for raw and parsed predictions
reached_target	Outcome of the target-instance success check
path_length_replayed_m	Executed path, including in the closed-loop log
path_length_optimal_m	Reference path distance recorded for replay
final_dist_to_target_m	Distance to the target at termination

Appendix

Table 11: Fields of the navigation and prediction records. Paths and distances are in metres; counts are integers. Episode and question identifiers are interpreted within their run and split.

Embodied QA

Embodied question answering studies how agents use perception, memory, and action to answer questions grounded in the physical world.

See all 43 papers

FAST-EQA: Efficient Embodied Question Answering with Global and Local Region Relevancy
Feb 17, 2026Haochen Zhang, Nirav Savaliya, Faizan Siddiqui +1Embodied QA Embodied Navigation
Carnegie Mellon University, USA · Honda Research Institute USA
Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering
Jul 23, 2026Zikui Cai, Kaushal Janga, Tan Dat Dao +15Agent Memory Embodied QA
University of Maryland, College Park. · The University of Texas at Austin. · University of Illinois Urbana-Champaign.
Before Answering: Evidence Sufficiency under Size-Matched Memory Construction
Sep 26, 2026Joyanta Jyoti Mondal, Md. Shifatul Ahsan Apurba, Mridul Banik +2Selective QA Memory-Augmented Agents
Department of Computer and Information Sciences, University of Delaware, USA · Department of Biomedical Informatics and Data Science, University of Alabama at Birmingham, USA · Luddy School of Informatics, Computing, and Engineering, Indiana University Indianapolis, USA +2

Feb 17, 2026J/K move · Enter open · S save

FAST-EQA: Efficient Embodied Question Answering with Global and Local Region Relevancy

Haochen Zhang, Nirav Savaliya, Faizan Siddiqui, Enna Sachdeva

Carnegie Mellon University, USA · Honda Research Institute USA

Embodied QA Embodied Navigation Memory-Augmented Agents

Open paper AskPDF arXiv

Embodied Question Answering (EQA) combines visual scene understanding, goal-directed exploration, spatial and temporal reasoning under partial observability. A central challenge is to confine physical search to question-relevant subspaces while maintaining a compact, actionable memory of observations. Furthermore, for real-world deployment, fast inference time during exploration is crucial. We introduce FAST-EQA, a question-conditioned framework that (i) identifies likely visual targets, (ii) scores global regions of interest to guide navigation, and (iii) employs Chain-of-Thought (CoT) reasoning over visual memory to answer confidently. FAST-EQA maintains a bounded scene memory that stores a fixed-capacity set of region-target hypotheses and updates them online, enabling robust handling of both single and multi-target questions without unbounded growth. To expand coverage efficiently, a global exploration policy treats narrow openings and doors as high-value frontiers, complementing local target seeking with minimal computation. Together, these components focus the agent's attention, improve scene coverage, and improve answer reliability while running substantially faster than prior approaches. On HMEQA and EXPRESS-Bench, FAST-EQA achieves state-of-the-art performance, while performing competitively on OpenEQA and MT-HM3D.

Abstract

Embodied QA

Explore similar work

FAST-EQA: Efficient Embodied Question Answering with Global and Local Region Relevancy

Beyond Episodic Evaluation: Memory Architectural Bottlenecks in Sequential Embodied Question Answering

Before Answering: Evidence Sufficiency under Size-Matched Memory Construction