See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay

Ashish Baghel; Paras Chopra

cs.AIMar 12, 2026

See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay

Authors: Ashish Baghel, Paras Chopra

Organizations: Lossfunk

Abstract

Vision-Language Models (VLMs) excel at describing visual scenes, yet struggle to translate perception into precise, grounded actions. We investigate whether providing VLMs with both the visual frame and the symbolic representation of the scene can improve their performance in interactive environments. We evaluate three state-of-the-art VLMs across Atari games, VizDoom, and AI2-THOR, comparing frame-only, frame with self-extracted symbols, frame with ground-truth symbols, and symbol-only pipelines. Our results indicate that all models benefit when the symbolic information is accurate. However, when VLMs extract symbols themselves, performance becomes dependent on model capability and scene complexity. We further investigate how accurately VLMs can extract symbolic information from visual inputs and how noise in these symbols affects decision-making and gameplay performance. Our findings reveal that symbolic grounding is beneficial in VLMs only when symbol extraction is reliable, and highlight perception quality as a central bottleneck for future VLM-based agents.

Figures & tables

Figure 1: Frame + Ground-Truth Symbols pipeline.

Figure 3: Frame + Self-Extracted Symbols pipeline.

Figure 4: Symbol-only pipeline.

Frame + Self-Extracted Symbols Prompt
You are an expert [game] player analyzing a game frame.

Game controls:
- Action 0: NOOP, Action 1: FIRE, ...

Current frame analysis:

Table 1: Example prompt structure for Frame + Self-Extracted Symbols pipeline (abbreviated).

Model	Pipeline	Pong	Breakout	Space Inv.
Claude-4-Sonnet	F+S-GT	$-1.0$	$12.0$	$175.0$
	F	$-16.0$	$0.0$	$80.0$
	S-GT	$-14.0$	$0.0$	$90.0$
	F+S-self	$-3.0$	$12.0$	$150.0$
Gemini-2.5-Pro	F+S-GT	$-1.0$	$12.0$	$170.0$
	F	$-7.0$	$7.0$	$95.0$

Table 2: Average reward over 600 frames across pipelines. Abbreviations: F = Frame-only; S-GT = Symbol-only; F+S-self = Frame + Self-Extracted Symbols; F+S-GT = Frame + Ground-Truth Symbols.

Figure 5: Claude-4-Sonnet performance across games, normalized relative to each model’s F+S-GT score (see Table 2 for absolute values). Claude shows consistent improvement with self-extracted symbols across all games, achieving near-optimal performance in Breakout.

Model	F1 Score	IoU
Claude-4-Sonnet	0.715	0.533
Gemini-2.5-Pro	0.189	0.202
GPT-4o	0.124	0.128

Table 3: Object detection metrics across models, averaged over 100 curated evaluation frames per game (see Section 3.3 ).

Model	F	F+S-self	F+S-GT	S-GT
Claude-4-Sonnet	5	9	14	12
GPT-4o	12	8	13	3
Gemini-2.5-Pro	11	4	13	12

Table 4: VizDoom defend_the_center performance (kills over 600 frames). Higher is better.

Model	F	F+S-self	F+S-GT	S-GT
Claude-4-Sonnet	$-1.0$	$2.0$	$\mathbf{11.0}$	$3.0$
GPT-4o	$7.0$	$9.0$	$\mathbf{9.0}$	$-1.0$
Gemini-2.5-Pro	$5.0$	$1.0$	$\mathbf{10.0}$	$-3.0$

Table 5: AI2-THOR kitchen task performance (cumulative reward over 300 frames). Higher is better.

Figure 6: Claude-4-Sonnet performance on complex 3D environments, normalized relative to F+S-GT score (see Tables 4 and 5 for absolute values). Claude improves significantly in VizDoom (35%

\rightarrow

64%) with self-extracted symbols.

Figure 7: GPT-4o performance on complex 3D environments, normalized relative to F+S-GT score (see Tables 4 and 5 for absolute values). GPT-4o matches the F+S-GT baseline in AI2-THOR with self-extracted symbols, demonstrating that accurate symbol extraction enables near-optimal performance.

F1 Score (higher indicates better performance)
Game	160 $\times$ 210	640 $\times$ 840	720 $\times$ 720	1280 $\times$ 720
Pong	0.28	0.52	0.55	0.58
Breakout	0.31	0.68	0.70	0.71
Space Inv.	0.35	0.71	0.73	0.75
Average	0.31	0.64	0.66	0.68

Table 6: Detection quality across input resolutions for Claude-4-Sonnet (averaged 100 frames per game).

Figure 8: Detection quality (F1 score) across resolutions and games. The native resolution (

160\times 210

) produces lower detection accuracy (F1

<

0.35), while higher resolutions enable effective symbolic grounding (F1

>

0.65).

Game	Model	$\sigma$ =0.0	$\sigma$ =0.1	$\sigma$ =0.2	$\sigma$ =0.3	$\sigma$ =0.4
Pong	Claude-4-Sonnet	-0.6	-2.2	-2.2	-2.2	-2.2
Pong	GPT-4o	-2.7	-3.0	-3.4	-3.8	-2.8
Breakout	Claude-4-Sonnet	5.0	4.3	3.4	3.4	2.8
Breakout	GPT-4o	5.0	4.0	3.0	2.3	2.6
Space Inv.	Claude-4-Sonnet	53	57	57	88	52
Space Inv.	GPT-4o	86	43	69	66	62

Table 7: Performance under coordinate noise (mean reward across 10 seeds).

Figure 9: Noise robustness analysis for Breakout. Both Claude-4-Sonnet and GPT-4o degrade significantly with even low noise (

\sigma=0.1

). Bars mark 95% confidence intervals over 10 seeds.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Gemini-2.5-Pro performance breakdown, normalized relative to F+S-GT score (see Table 2 for absolute values). Gemini benefits in simpler games but struggles in Space Invaders.

Figure 11: GPT-4o performance breakdown, normalized relative to F+S-GT score (see Table 2 for absolute values). GPT-4o shows mixed results, with symbolic information sometimes harming performance.

Figure 12: Noise robustness analysis for Pong. Performance shows rapid degradation similar to Breakout.

Figure 13: Noise robustness analysis for Space Invaders. High variance due to dense alien formations where random firing can still score points.

Vision-Language Agents

Vision-language agents use visual and language models to perceive environments, reason about goals, and take actions through interactive interfaces.

See all 45 papers

SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes
May 29, 2026Tianhui Liu, Jie Feng, Zhiheng Zheng +6Vision-Language Agents Visual Spatial Reasoning
1The Hong Kong University of Science and Technology (Guangzhou) · 2Zhongguancun Academy · 3Tsinghua University +1
VLMs for Videogame Data Annotation
Aug 6, 2026Katrin Schmid, Iuri FrosioVLM Evaluation
NVIDIA, Australia · NVIDIA, Italy
A Heart Means Recursion: Exploring Visual Meaning and Communication in VLMs
Oct 27, 2026Alexa R. Tartaglini, Zhengxuan Wu, Thaddäus Wiedemer +3Emergent Communication VLM Interpretability
Google DeepMind · Stanford University

May 29, 2026J/K move · Enter open · S save

SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes

Tianhui Liu, Jie Feng, Zhiheng Zheng, Shengyuan Wang+5

1The Hong Kong University of Science and Technology (Guangzhou) · 2Zhongguancun Academy · 3Tsinghua University+1

Vision-Language Agents Visual Spatial Reasoning 3D Spatial Reasoning Spatial Reasoning Benchmarks Visually Grounded Reasoning Spatial Reasoning+2

Open paper AskPDF arXiv

Humans can effortlessly perceive spatial layouts, form cognitive representations, reason about spatial relations, and translate such reasoning into actions in everyday 3D environments. Although recent vision-language models (VLMs) have shown promising performance on observation-conditioned spatial perception and reasoning tasks, it remains unclear whether they can build coherent spatial understanding, act upon it, and refine their actions through multi-turn feedback. To study this problem, we introduce SpatialAct, a simulator-grounded benchmark for probing action-conditioned spatial reasoning in 3D scenes. Starting from the most challenging setting, Multi-turn Interactive Refinement, we further design its decomposed counterpart, Single-step Error Detection and Fix, together with five fundamental spatial ability tasks to diagnose the underlying causes of model failures. Experiments reveal a clear reasoning-to-action gap: current VLMs can perform well on isolated spatial reasoning tasks, but struggle to maintain coherent spatial beliefs and produce reliable actions during multi-turn feedback, substantially underperforming humans. These results suggest that current VLM agents still lack robust spatial state tracking under action-induced environment changes, even when low-level control is abstracted away.

Abstract

Vision-Language Agents

Explore similar work

SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes

VLMs for Videogame Data Annotation

A Heart Means Recursion: Exploring Visual Meaning and Communication in VLMs