cs.RONov 1, 2026

Do Embodied VLMs Actually Use Their Scene-Graph Memory?

Authors: Shubh Garg, Divyansh Pandey, Vedant Khandelwal, Niyati Rawal, Amit Sheth

Organizations: Indian AI Research Organisation (IAIRO), GIFT City, Gujarat · UPAI Labs · AI Institute, UofSC (AIISC), 1112 Greene St., Columbia · BITS Pilani

Abstract

Embodied agents that answer questions about their route need a memory of what they saw, and scene graphs are a natural form for it. A vision-language model (VLM) can receive such a graph as text or through a graph encoder, but accuracy alone cannot show whether the graph holds the episode's facts, reaches the model, or is actually relied on. In this diagnostic study, we introduce a graph-swap test for the last of these: each episode is given another episode's graph, and we count how many answers change. On an embodied question-answering benchmark with routes through two to seven rooms, we use it to compare ways of delivering scene-graph memory to a fine-tuned VLM. As text, the memory is used strongly, adding 30 points on route questions. As a graph, it is not, and we trace why: the encoder preserves the information, but standard pooled and late fusion discard it. Giving each node its own input token recovers part of it, a gain of 8.3 points on route questions that the swap test confirms comes from the graph. The test also exposes a shortcut that accuracy hides: when the same facts appear in the text, the model ignores its graph. The test thus offers a cheap check of whether a VLM uses its scene-graph memory, and our analysis points to how that memory should be delivered.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LVMem: Image-Augmented Memory for Long-Horizon Question Answering

    Oct 19, 2026Victoria Khoruzhaya, Maksim Kalmykov, Tuong Do Minh +3Embodied QALong-Term Memory for Embodied Agents

  2. EmbodiedLGR: Integrating Lightweight Graph Representation and Retrieval for Semantic-Spatial Memory in Robotic Agents

    Apr 20, 2026Paolo Riva, Leonardo Gargani, Matteo Frosi +1Graph Retrieval-Augmented GenerationEmbodied QA

  3. Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

    Sep 29, 2026L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5VLM EvaluationVisual Question Answering