SafeVantage: Vantage-Aware Memory for Reliable Embodied Decisions

Sean Hardesty Lewis; Zuyi Guo; Benwang Chen; Zirui Li; Hongyi Lin; Heye Huang

cs.CVSep 29, 2026

SafeVantage: Vantage-Aware Memory for Reliable Embodied Decisions

Authors: Sean Hardesty Lewis, Zuyi Guo, Benwang Chen, Zirui Li, Hongyi Lin, Heye Huang

Organizations: Massachusetts Institute of Technology · Korea Advanced Institute of Science and Technology · Nanyang Technological University · Tsinghua University

Abstract

Reliable embodied decisions under partial observability require informative observations and sufficient supporting evidence. However, semantic scores alone do not reveal which viewpoints justify a claim or where additional evidence should be acquired. We introduce SafeVantage, a vantage-aware semantic memory and active acquisition framework that retains each claim's supporting views, camera poses, and estimated target location, keeping positive support distinct from search coverage. A learned candidate-observability model uses claim-grounded geometry to predict target visibility at reachable viewpoints. These predictions guide view selection through expected reduction in terminal decision loss, accounting for travel cost and geometrically distinct corroboration. A calibrated head then combines support, spatial consistency, and coverage to produce Yes, No, or Abstain decisions. We evaluate SafeVantage on a category-presence benchmark spanning 232 unseen ProcTHOR houses and 7,424 paired episodes per method and action budget. Compared with validation-selected equal-budget baselines, SafeVantage achieves macro-F1 gains of 24.7% and 12.0% at eight and twelve actions, respectively, with lower risk and higher answer rates at both budgets and 31.7% less travel at eight actions. Equal-input HM3D experiments show lower selective risk under fixed observations, while controlled ScanNet interventions show that restoring supporting views improves downstream VLM answers. Ablations further support the contribution of candidate observability to decision quality and acquisition efficiency. Results demonstrate the value of claim-level viewpoint evidence for connecting semantic memory, active acquisition, and reliable decision-making. Code is available at https://safevantage.github.io

Figures & tables

Fig. 2: Viewpoint support and search coverage provide complementary evidence. Navigable viewpoints differ in the target evidence they reveal. Supporting observations provide positive evidence, whereas partial or context-only observations do not justify absence. Retained views and target geometry inform subsequent acquisition and selective decisions.

Budget	Method	Risk $\downarrow$	Macro-F1 $\uparrow$	Answer $\uparrow$	Travel $\downarrow$
8	SafeVantage	0.384	0.604	0.581	4.94
	Random	0.427	0.434	0.350	19.12
	Nearest frontier	0.398	0.491	0.394	3.46
	Coverage gain	0.422	0.406	0.305	6.31
	Category-room prior †	0.396	0.484	0.377	7.23
	Detector-conf. gain	0.407	0.446	0.333	5.94

TABLE I: Equal-budget acquisition-to-decision performance on 232 held-out ProcTHOR houses. Lower risk and travel are better; higher macro-F1 and answer rate are better. Bold marks the best value within each budget. Travel is measured in meters. † represents the primary comparison policy.

Fig. 3: One held-out episode for “Is there a paper towel in the room?” Both policies start from the same start with an 8-action budget and a positive ground-truth label. Category–room prior does not observe the target, answers NO , and travels 13.7 m; SafeVantage acquires 4 supporting views, answers YES , and travels 5.4 m.

Method	Cov.	Acc.	FP	Recall	Risk	$R_{9}$	$R_{@\mathrm{cov}}$	AURC
SafeVantage	0.436	0.962	0.059	0.584	0.524	0.538	0.538	0.649
Qwen2 caption-RAG	0.565	0.881	0.240	0.692	0.715	0.638	0.685	0.715
ConceptGraphs	0.432	0.920	0.123	0.552	0.617	0.594	0.707	0.726
VLMaps	0.541	0.916	0.162	0.689	0.618	0.656	0.910	0.850

TABLE II: Equal-input HM3D experiments.

R_{9}

repeats calibration across nine folds;

R_{@\mathrm{cov}}

matches coverage. Lower risk and AURC are better.

reader	evidence set	F1	$\Delta$ vs absent	scene CI
Qwen2-VL	target absent	0.2483	–	–
Qwen2-VL	strongest 1	0.3565	+0.1082	[-0.0181, 0.2290]
Qwen2-VL	strongest 4	0.4410	+0.1927	[0.0691, 0.2977]
Qwen2-VL	complete	0.4540	+0.2057	[0.0752, 0.3133]
Qwen2.5-VL	target absent	0.1888	–	–
Qwen2.5-VL	complete	0.3215	+0.1327	[0.0513, 0.2053]

TABLE III: ScanNet supporting-view intervention. Intervals compare restored conditions with target-absent histories.

Fig. 4: Removing vantage-aware state lowers macro-F1. Bars show full policy minus grouped removal on 96 validation houses; whiskers are paired 95% intervals.

	8 actions		12 actions
Variant	F1 $\uparrow$	Loss (pp) $\downarrow$	F1 $\uparrow$	Loss (pp) $\downarrow$
Full SafeVantage	0.5307	–	0.6149	–
$-$ claim provenance	0.4753	+5.54	0.5826	+3.23
$-$ target-ray alignment	0.4749	+5.58	0.5819	+3.30
$-$ yaw diversity	0.5311	$-0.04$	0.6162	$-0.13$
$-$ base geometry	0.5255	+0.52	0.6167	$-0.18$

TABLE IV: Component ablations on 96 ProcTHOR validation houses. Loss is the macro-F1 drop from the full model (pp).

Explore similar work

Aug 8, 2026cs.AI

Explore, Map, Remember, Decide: Are Embodied VLMs Ready for Safety-Critical Scenarios?

Theory of Space framework (ToS) assesses the spatial understanding of curiosity-driven Vision-Language Models (VLMs) under partial observability. As AI techniques are increasingly applied to safety-critical scenarios, it is crucial to understand whether VLMs possess robust spatial memory and make reliable decisions. In this paper, we assess whether VLMs' decisions are based on physical evidence or are corrupted by visual-language biases, if their memory processes align with human cognitive patterns, and how they respond to environmental hazards. We extend the ToS framework into a safety-critical, goal-driven pipeline, named Explore, Map, Remember, and Decide (EMRD). We then quantify Exploration Competence (Explore) through metrics of environmental coverage and temporal efficiency, assess Spatial Fidelity (Map), evaluate, with a suite of psychological metrics, Memory Persistence (Remember), and measure, using focal-point metrics, Cognitive Decision-Making (Decide). Our results show that in terms of decision-making capabilities, VLMs frequently select evacuation points based on pre-trained textual priors while lacking the spatial grounding to justify their choices. We also show that spatial reasoning degrades in low-light conditions, but it is not affected by texture and colour tampering. Our findings suggest that VLM memory fundamentally diverges from human cognition, creating unpredictable risks of misalignment.

Gabriele La Malfa, Nitay Alon, Emanuele La Malfa +2

King’s College London, UK · Massachusetts Institute of Technology, USA · University of Oxford, UK +2

Recent Vision-Language Models Spatial Reasoning Spatial Memory Safety-Critical Scenarios Cognitive Science Exploration+1

Ask PDF

Sep 17, 2026cs.RO

VABench: Measuring Embodied Spatial Intelligence through Visual Demonstrations, Active Perception, and Metric Control

Spatial intelligence requires more than describing object locations. Under incomplete observation, models must identify and acquire missing evidence, interpret it in a common spatial frame, and act on it. We introduce VA-Bench to evaluate the complete observe-reason-act-revise loop. General-purpose MLLMs learn procedural context from RGB-only demonstrations, actively select camera viewpoints, issue metric Cartesian commands, and revise them from execution feedback. Models receive no privileged object poses, oracle trajectories, or learned action heads. A fixed model-agnostic controller executes only model-specified targets. VA-Bench contains 14 base task families (11 single-arm and three dual-arm), seven held-out geometry/layout variants, and a long-horizon five-object composition track. We evaluate 12 primary model conditions in three independent runs over the same 20 physically verified seeds per base task, reporting terminal success, nine trajectory-level behavioral diagnostics, and subtask progress. First, the best-performing model scores 100.0% on target localization and 78.9% on spatial relations in the annotated run. Its three-run macro-average task success is only 53.93+/-3.17%. Second, active camera control significantly improves task success over passive multi-view observation. In one matched comparison, success rises from 27.86% to 57.50%. Third, held-out geometric transfer can reduce task success by over 30 percentage points. No model completes a strict long-horizon episode, despite substantial partial progress. VA-Bench thus tests whether general-purpose MLLMs can turn visual demonstrations and actively acquired evidence into successful embodied action.

Zhongbo Zhang, Jiayi Jin, Yifan Wang +4

Dalian University of Technology · Nanyang Technological University

Embodied Artificial Intelligence Trajectory-Level Credit Multimodal Large Language Models

Ask PDF

Jun 6, 2026cs.CV

Remember with Confidence: Uncertainty Quantification for Spatio-temporal Memory with Probabilistic Guarantees

Long-horizon robot operation requires spatio-temporal memory to record the environment state and recall it for downstream reasoning. Scene graphs and retrieval-augmented systems ground VLM descriptions to persistent 3D entities with rich semantic descriptions. However, VLM captions are noisy and viewpoint-inconsistent, and existing systems treat them as an oracle with no mechanism to detect unreliable stored descriptions. We introduce object-level semantic uncertainty for multi-view VLM memory: a score that measures object-centric cross-view semantic scatter of captions and identifies semantically unresolved objects. Then, we include our uncertainty scores in an advanced spatial-semantic memory system, that we dub UQ-DAAAM. UQ-DAAAM uses this score to actively refine uncertain objects under a fixed query budget by selecting high-quality views and fusing the resulting multi-view captions into a single object description. We also derive probabilistic guarantees showing that higher-quality candidate views (as selected by our approach) are more likely to reduce uncertainty. Our experiments show that uncertainty quantification can make embodied 4D memory systems more reliable and more effective. In particular, on the OC-NaVQA benchmark, UQ-DAAAM achieves substantially larger uncertainty reduction and better spatio-temporal question answering performance than baselines.

Harry Zhang, Nicolas Gorlo, Luca Carlone

MIT

Spatial Memory Semantic Geometry Uncertainty Quantification Recent Vision-Language Models

Ask PDF