Visual Language Models (VLMs) have increasingly become the main paradigm for understanding indoor scenes, but they still struggle with metric and spatial reasoning. Current approaches rely on end-to-end video understanding or large-scale spatial question answering fine-tuning, inherently coupling perception and reasoning. In this paper, we investigate whether decoupling perception and reasoning leads to improved spatial reasoning. We propose an agentic framework for static 3D indoor scene reasoning that grounds an LLM in an explicit 3D scene graph (3DSG). Rather than ingesting videos directly, each scene is represented as a persistent 3DSG constructed by a dedicated perception module. To isolate reasoning performance, we instantiate the 3DSG from ground-truth annotations. The agent interacts with the scene exclusively through structured geometric tools that expose fundamental properties such as object dimensions, distances, poses, and spatial relationships. The results we obtain on the static split of VSI-Bench provide an upper bound under ideal perceptual conditions on the spatial reasoning performance, and we find that it is significantly higher than previous works, by up to 16%, without task specific fine-tuning. Compared to base VLMs, our agentic variant achieves significantly better performance, with average improvements between 33% to 50%. These findings indicate that explicit geometric grounding substantially improves spatial reasoning performance, and suggest that structured representations offer a compelling alternative to purely end-to-end visual reasoning.
Figures & tables
Figure 1: RieMind: Geometry-grounded framework for spatial agentic scene understanding, and in this figure, it is reasoning over the relative direction of objects in a scene, while performing allocentric-egocentric frame transformations.
Figure 2: Reasoning component of the spatial agent framework, containing the different types of tools, and general agent architecture following the MCP.
Node Type
Name / Identifier
Class (Count)
BuildingNode
Building-0
residential
FloorNode
Floor-0
0
RoomNode
Room-0
room_0
ObjectNode
Cabinet-0 …Cabinet-15
cabinet (16)
ObjectNode
Chair-0 …Chair-2
chair (3)
ObjectNode
Oven-0, Oven-1
oven (2)
Table 1: Example of the scene graph context provided to the agent that is integrated into the system prompt. This scene context is going to be dependent on the actual scene, as each will contain different contents and structure.
Figure 3: Exemplifying the different types of explicit grounding through a relative direction question. For any question posed to the agent, it will need to disambiguate the actual entity, and this grounds its reasoning. For this question, it then needs to obtain positions, orientations, and create the specific egocentric frame with its own axis convention. It can then perform the projection of the desired object into this new egocentric frame. All these tools and processes ground the reasoning sequence of the agent, and force it to stay focused on these basic geometric objects.
Qwen2.5-VL-7B [ 1 ]
GPT-4o [ 15 ]
Question Type
Base
Agent + Tools
Base
Agent + Tools
Object Count
40.9
89.7 (+48.8)
46.2
85.1 (+38.9)
Absolute Distance
14.8
90.3 (+75.5)
5.3
93.2 (+87.9)
Object Size
43.4
93.6 (+50.2)
43.8
96.5 (+52.7)
Room Size
10.7
31.9 (+21.2)
38.2
83.5 (+45.3)
Relative Distance
38.6
44.5 (+5.9)
37.0
85.6 (+48.6)
Table 2: Comparison of performance on VSI-Bench between base VLMs (without tools), and their LLM counterpart as an agent with our tool framework. Improvements are shown in parentheses next to the Agent + Tools scores.
Question Type
Number of Questions
Score
Tools
Average
Median
Object Count
544
89.7
1.05
1
Absolute Distance
799
90.3
2.14
2
Object Size
927
93.6
2.15
2
Room Size
282
31.9
2.35
2
Relative Distance
703
44.5
2.28
2
Table 3: Score vs. reasoning complexity by question type for Qwen2.5-VL-7B on VSI-Bench. Relative direction questions require a 5-tool pipeline ( sg_search → loc_get_position → loc_orientation → loc_build_frame → loc_project ), while simpler types need only 1–2 tools. Qwen2.5-VL-7B frequently fails to complete the full pipeline, resulting in lower accuracy.
Models
Avg.
Obj. Count
Abs. Dist.
Obj. Size
Room Size
Rel. Dist.
Rel. Dir.
Proprietary
GPT-4o
35.3
46.2
5.3
43.8
38.2
37.0
41.3
Gemini-1.5 Flash
44.5
49.8
30.8
53.5
54.4
37.7
41.0
Gemini-1.5 Pro
48.7
56.2
30.9
64.1
43.6
51.3
46.3
Open-Source
InternVL3-78B
50.7
71.2
53.7
44.4
39.5
55.9
39.5
Table 4: Evaluation on the static questions of VSI-Bench, the comparison includes proprietary, open-source, and fine-tuned models. Our agentic framework achieves state-of-the-art performance across all subtasks, where bold text denotes best performance, and underline denotes second best. These results should still be understood as an upper-bound on spatial understanding performance.
3D spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and motion to understand and act in three-dimensional scenes.