Organizations: Indian AI Research Organisation (IAIRO), GIFT City, Gujarat · UPAI Labs · AI Institute, UofSC (AIISC), 1112 Greene St., Columbia · BITS Pilani
Embodied agents that answer questions about their route need a memory of what they saw, and scene graphs are a natural form for it. A vision-language model (VLM) can receive such a graph as text or through a graph encoder, but accuracy alone cannot show whether the graph holds the episode's facts, reaches the model, or is actually relied on. In this diagnostic study, we introduce a graph-swap test for the last of these: each episode is given another episode's graph, and we count how many answers change. On an embodied question-answering benchmark with routes through two to seven rooms, we use it to compare ways of delivering scene-graph memory to a fine-tuned VLM. As text, the memory is used strongly, adding 30 points on route questions. As a graph, it is not, and we trace why: the encoder preserves the information, but standard pooled and late fusion discard it. Giving each node its own input token recovers part of it, a gain of 8.3 points on route questions that the swap test confirms comes from the graph. The test also exposes a shortcut that accuracy hides: when the same facts appear in the text, the model ignores its graph. The test thus offers a cheap check of whether a VLM uses its scene-graph memory, and our analysis points to how that memory should be delivered.
Figures & tables
Fig. 1: Our scene-graph memory and three ways to give it to a VLM, with where its information is lost. Our pipeline (left) turns the agent’s frames into per-frame scene graphs and merges them into one episode graph, which reaches the model as memory text (blue), through a graph encoder (R-GCN), or both. The standard graph route (dashed) combines a mean-pooled vector added at the input and a gated cross-attention after the last layer. Our per-node route writes one token per node into the prompt. Numbered tags: the four findings of Sec. I , placed on the part of the pipeline each concerns.
Type
n
Gold answer
Example test question
Route questions: when things were seen (291)
Order
73
whether type X is first seen before or after type Y
Did the robot first see a sink before or after it first saw a houseplant? [before, after]
Presence change
102
whether a type seen in an earlier strip frame is absent from the strip’s last frame
Was a statue visible at some point but then gone from view later? [yes, no]
Rooms visited
116
number of distinct room labels among the strip frames
How many different rooms did the robot enter during its run? [2, 3, 4, 5]
Set questions: what was where (509)
Persistence
131
number of distinct room labels whose frames show the type
In how many different rooms did the robot see a painting? [2, 3, 4]
TABLE I: Question types, gold answers and examples. Options are in brackets. Every gold answer is computed from the visible-object records and room labels of a strip of six sampled frames.
Objects
Rooms
Episode memory
P
R
F1
Frame
Count
Edit
Obj.–room
Order
65.7
49.2
56.3
71.8
199
1.30
45.5
78.4
TABLE II: Quality of our scene graphs on the test split (200 episodes, the 8,135 frames the answering model may use), scored against the simulator’s visible-object records. P, R, F1: object types per frame (%). Frame: frames with the right room label (%). Count: episodes with the right number of visited rooms (of 200). Edit: edit distance between the remembered and the true room sequence (lower is better). Obj.–room: recall of (object type, room) pairs (%). Order: pairs of types first seen in the right order (%).
Fig. 2: What our scene graphs find on the test split (8,135 frames, scored by object type against the simulator’s records). (a) Precision and recall of each object type visible in at least 300 frames and kept by the rules; dot area: frames in which the type is visible; gray curves: equal F1. (b) Object precision and recall by route stratum (one to four or more door crossings).
Arm
Memory
Text
Graph route
Seeds
A
No memory
–
–
2
B
Text
✓
–
2
C
Graph
–
standard (pooled + late)
1
D
Text + graph
✓
standard (pooled + late)
1
C
Graph
–
per-node tokens (ours)
2
D
Text + graph
✓
per-node tokens (ours)
2
TABLE III: The arms. All are MomaGraph-R1 fine-tuned with LoRA on the same data and schedule; they differ only in how the episode KG reaches the model. Seeds: training seeds reported.
Gain over no memory
Arm, memory
Route
Acc.
All
Route
Set
A No memory
–
64.8
ref.
B Text
–
75.9
+11.06 *
+30.07 *
+0.20
C Graph
standard
65.9
+1.62
+3.44
+0.59
C Graph
per-node
67.1
+2.31 *
+8.25 *
− 1.08
D Text + graph
standard
77.0
+12.75 *
+27.49 *
+4.32 *
TABLE IV: Test results by memory route: accuracy on the 800 test questions (%) and the gain over no memory in accuracy points over all, route (291) and set (509) questions; * marks a paired 95% interval that excludes zero (intervals in Table XI ). Text, no memory and per-node models: two seeds; standard graph models: seed 0, compared with no memory seed 0. Shaded: our per-node route. Last row: text from perfect scene graphs, an upper bound.
Fig. 4: Gain over no memory (seed 0), in accuracy points. Wide pale bar: route questions; narrow bar: all questions; * marks a paired 95% interval that excludes zero. Of the graph-only models (shaded), only the per-node one clears zero, and only on route questions.
Route: when things were seen
Set: what was where
Model
Memory, graph route
S
Order
Pres.
Visited
Route
Persist
Room
Exist
Count
Global
Set
All
73
102
116
291
131
80
113
65
120
509
800
Chance (question-weighted)
50.0
50.0
28.9
41.6
30.0
28.2
25.1
28.6
28.6
28.1
33.0
Language only, no image
Qwen3-4B [ 41 ]
–
–
39.7
51.0
30.2
39.9
38.2
70.0
61.1
26.2
32.5
45.4
43.4
Zero-shot VLMs, evidence frames
TABLE V: Accuracy (%) by question type on the 800 test questions. Top: reference models that are not trained on the benchmark; they answer from the question’s evidence frames (the language-only model sees no image). Bottom: fine-tuned models that answer from memory with a blank image. The two regimes do not share an evidence protocol and are not one leaderboard. S : seeds averaged. Green (red): the paired 95% interval of the difference to the same backbone’s no-memory model excludes zero, with seeds matched. Bold: best in the column within each block. Shaded rows: our per-node route. Chance is question-weighted.
Fig. 5: Answers that change from right to wrong (left) or wrong to right (right), on route and set questions. Top: each model against no memory. Middle: the two per-node models with another episode’s graph or text. Bottom: the same final checkpoint evaluated twice, the noise floor.
Fig. 6: Accuracy lost to each swap, per-node models: each arrow runs from the step-3,000 snapshot (open circle, gray value) to the final checkpoint (arrowhead, bold value). * marks a paired 95% interval that excludes zero. The text-swap cost grows during training; the graph-swap cost of the text + graph model stays near zero.
Fig. 7: Gain over no memory by question type. Bars: text and the per-node graph model (two seeds) and the standard graph model (seed 0); thin line: paired 95% interval; pale bar: the interval includes zero. Diamonds: text from perfect scene graphs (two seeds; open: the interval includes zero). Shaded rows: route and set averages.
Fig. 8: Gain over no memory by number of door crossings (strata S1 to S4, 50 test episodes each), over all questions (left) and route questions (right). Bands: paired 95% intervals for text and the two graph-only models. Text, perfect-graph text and per-node models: two seeds; standard graph: seed 0 against no memory seed 0.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Stratum
Eps.
Doors
Rooms
Frames
Path
Objects
Repeat
S1
50
1
2
20.4 (10–40)
6.6
28.0
0
S2
50
2
2–3
33.6 (18–58)
11.2
43.5
2
S3
50
3
4
46.4 (26–87)
16.1
46.2
10
S4
50
4–6
5–7
66.2 (34–116)
22.9
68.2
50
Appendix
TABLE VI: Test split by route stratum. Doors: door crossings on the route. Rooms: distinct rooms on the route. Frames: mean (range). Path: walked length (m). Objects: distinct object instances seen per episode. Repeat: episodes passing two rooms of the same type.
Fig. 9: Test-split composition. (a) Questions per type. (b) Options per question; the mix of two, three and four options puts uniform-guess accuracy at 33.0%, above the 25% of four options.
Fig. 10: Stored 256×256 frames from one episode outside the test split (31 frames, living room → kitchen). Left: an early view in the living room. Middle: the view at the room transition, with the kitchen ahead. Right: a late view near the end of the route.
Backbone
MomaGraph-R1 (Qwen2.5-VL-7B), bf16
Adapter
LoRA r=16 , α=32 , dropout 0.05, on q, k, v, o projections
navigation-action cross-entropy on every step; answer-letter cross-entropy on the final step; answer frames oversampled 8× ; answer weight 1.0
Answer image
blank (all episodic information comes from memory)
Graph encoder
2-layer R-GCN over relation types; node features = 64-d name embedding + state and function features; hidden 128, output 256; projected to the VLM width (3,584) with Linear + LayerNorm
Appendix
TABLE VII: Training and evaluation settings, shared by every run.
Encoder taken from
Raw
Random
Trained
Graph, standard route (527)
0.495
0.856
0.807
Graph, per-node route (532)
0.495
0.834
0.834
Appendix
TABLE VIII: Linear-probe AUC for first-sighting order (8,000 test pairs).
Step 3,000
Step 6,000 (final)
Graph, per-node
Text + graph, per-node
Graph, per-node
Text + graph, per-node
Type
n
own
graph sw.
own
graph sw.
text sw.
both sw.
own
graph sw.
own
graph sw.
text sw.
both sw.
Order
73
52.1
49.3
76.7
76.7
47.9
47.9
54.8
56.2
84.9
83.6
43.8
43.8
Presence change
102
52.9
52.0
87.3
86.3
46.1
49.0
58.8
60.8
93.1
94.1
42.2
42.2
Rooms visited
116
68.1
50.9
77.6
74.1
52.6
52.6
66.4
47.4
81.9
81.0
47.4
48.3
Persistence
131
81.7
79.4
81.7
79.4
80.9
79.4
78.6
80.9
80.9
80.2
79.4
80.2
Appendix
TABLE IX: Graph-swap and text-swap test by question type (%), at the step-3,000 snapshots and at the final step-6,000 checkpoints. Swapped conditions score each episode with another episode’s graph, text, or both.
Memory
Graph route
Seed 0
Seed 1
No memory
–
514 (64.2)
523 (65.4)
Text
–
608 (76.0)
606 (75.8)
Graph
standard
527 (65.9)
–
Text + graph
standard
616 (77.0)
–
Graph
per-node
532 (66.5)
542 (67.8)
Text + graph
per-node
610 (76.2)
620 (77.5)
Appendix
TABLE X: Correct answers of 800, our scene graphs. No memory reads no scene graph.
Comparison
All questions (800)
Route questions (291)
Set questions (509)
Each model − no memory (two seeds where both have two; otherwise seed 0)
Text − no memory, seed 0
+11.75
* [+8.88, +14.75]
+29.55
* [+23.57, +35.49]
+1.57
[ − 1.31, +4.44]
Text − no memory, two seeds
+11.06
* [+8.50, +13.62]
+30.07
* [+24.66, +35.42]
+0.20
[ − 2.17, +2.54]
Graph (standard) − no memory
+1.62
[ − 0.25, +3.50]
+3.44
[ − 0.34, +7.09]
+0.59
[ − 1.61, +2.79]
Graph (per-node) − no memory, two seeds
+2.31
* [+0.56, +4.06]
+8.25
* [+4.47, +12.03]
− 1.08
[ − 2.75, +0.59]
Text + graph (standard) − no memory
+12.75
* [+9.75, +15.88]
+27.49
* [+21.48, +33.45]
+4.32
* [+1.15, +7.60]
Appendix
TABLE XI: Every paired difference used in the paper, in accuracy points with 95% episode-bootstrap intervals (* marks an interval that excludes zero). Two-seed rows average per-question correctness over seeds first.
Path
Contents
Supports
results/e1_e4_test_comparison.txt
per-node models vs all others, with intervals; per type
graph and text + graph, standard route, vs references
Tables IV , X
test_predictions/v2r_kg/
no memory and text (two seeds), graph and text + graph with both routes
Tables IV , X ; Fig. 5
training_logs/<run>/
full training logs: arguments, losses, gradient norms, validation accuracy and letter distribution every 500 steps, gate values
Table XIV
launchers/
exact launch scripts for every run
Table VII
Appendix
TABLE XII: Contents of log_corl/ and the results each file supports.
Stage
GPUs
Wall-clock time
Re-render frames at 512 px (AI2-THOR, Vulkan)
1
7.4 min for the 8,335 test frames
Scene graphs (MomaGraph-R1, 512 px)
2
train (206,899 frames) 17.8 h; validation 2.3 h; test 45 min; about 0.6 s per frame per GPU
Fine-tune, MomaGraph-R1
1
about 17–30 h per run
Test evaluation (800 questions)
1
23–36 min
Appendix
TABLE XIII: Hardware and wall-clock time per stage. Fine-tuning covers the runs of Table XIV .
Memory
Graph route
Seed
Steps
Val. last
Val. best (step)
Letter
Gate
Test
No memory (reads no scene graph)
No memory
–
0
6,000
67.3
69.8 (5k)
0.47
–
514
No memory
–
1
6,000
67.8
68.3 (2.5k)
0.46
–
523
Our scene graphs
Text
–
0
6,000
79.0
80.3 (5.5k)
0.44
–
608
Text
–
1
6,000
80.0
80.0 (6k)
0.41
–
606
Appendix
TABLE XIV: Training runs reported in this paper, in the order they started. Validation: answer accuracy (%) on 400 validation questions at the last validation pass and at the best one; letter: largest share of one answer letter at any validation pass. Gate: final late-fusion ∣tanhg∣ (standard graph route only). Test: correct of 800 at the final checkpoint.
Graph-based agent memory organizes an agent’s experiences, facts, and procedures as linked structures for persistent, evidence-aware retrieval and reuse.
Reliable language-guided perception is essential for embodied agents, where questions about objects, events, and their locations must be answered from long observation histories. Unlike conventional visual question answering, realistic robotic settings require reasoning over extended temporal horizons and persistent scene memory. We build upon the view-based memory paradigm and propose an efficient framework for long-horizon spatio-temporal question answering. While existing view-based systems typically store either text captions or visual embeddings, we extend the ReMEmbR-style memory with image paths that link each stored description to its original visual content. To make the stored annotations more informative, we replace uniform sampling during memory construction with an optical-flow-based frame selection strategy (PSFR) and store an object-centric descriptive scene representation instead of plain captions. We further equip a VLM-based agent with an adaptive retrieval tool that returns the image associated with the most relevant caption, allowing the agent to decide when visual inspection is needed. On the OC-NaVQA benchmark, LVMem outperforms ReMEmbR on binary and positional metrics with both backbones and matches or exceeds a 4D scene graph DAAAM baseline on binary accuracy, while operating on a lightweight view-based memory that requires no explicit graph construction.
Victoria Khoruzhaya, Maksim Kalmykov, Tuong Do Minh +3
Moscow Independent Research Institute of Artificial Intelligence, Moscow, Russia · Cognitive AI Systems Lab, Moscow, Russia · Peking University, Beijing, China
As the world of agentic artificial intelligence applied to robotics evolves, the need for agents capable of building and retrieving memories and observations efficiently is increasing. Robots operating in complex environments must build memory structures to enable useful human-robot interactions by leveraging the mnemonic representation of the current operating context. People interacting with robots may expect the embodied agent to provide information about locations, events, or objects, which requires the agent to provide precise answers within human-like inference times to be perceived as responsive. We propose the Embodied Light Graph Retrieval Agent (EmbodiedLGR-Agent), a visual-language model (VLM)-driven agent architecture that constructs dense and efficient representations of robot operating environments. EmbodiedLGR-Agent directly addresses the need for an efficient memory representation of the environment by providing a hybrid building-retrieval approach built on parameter-efficient VLMs that store low-level information about objects and their positions in a semantic graph, while retaining high-level descriptions of the observed scenes with a traditional retrieval-augmented architecture. EmbodiedLGR-Agent is evaluated on the popular NaVQA dataset, achieving state-of-the-art performance in inference and querying times for embodied agents, while retaining competitive accuracy on the global task relative to the current state-of-the-art approaches. Moreover, EmbodiedLGR-Agent was successfully deployed on a physical robot, showing practical utility in real-world contexts through human-robot interaction, while running the visual-language model and the building-retrieval pipeline locally.
Paolo Riva, Leonardo Gargani, Matteo Frosi +1
Department of Electronics, Information, and Bioengineering (DEIB), Politecnico di Milano, Milan, Italy
Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.
L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5