AtariBench: Evaluating Spatiotemporal Reasoning of Vision-Language Models in Atari Gameplay

Mingjia Huo; Yao Fu; Bo Chang; Yaqing Wang; Dawei Zhu; Dylan Zhang; Xinyang Yi; Lichan Hong

cs.CVNov 5, 2026

AtariBench: Evaluating Spatiotemporal Reasoning of Vision-Language Models in Atari Gameplay

Authors: Mingjia Huo, Yao Fu, Bo Chang, Yaqing Wang, Dawei Zhu, Dylan Zhang, Xinyang Yi, Lichan Hong,

Organizations: University of California San Diego · Google DeepMind · Peking University · University of Illinois Urbana-Champaign

Abstract

Vision-Language Models (VLMs) have achieved strong performance on static visual understanding benchmarks, yet their ability to act in dynamic environments remains less well understood. We introduce AtariBench, a controlled benchmark that evaluates VLMs across 20 Atari 2600 games, requiring models to make sequential decisions from high-temporal-resolution gameplay streams. By decoupling fast in-game dynamics from wall-clock inference speed, AtariBench focuses on whether models can perceive rapid visual state changes, reason over spatiotemporal dynamics, and ground actions in the current observation. Our experiments reveal substantial limitations in state-of-the-art VLMs: models often fail to infer object motion and direction, exhibit brittle sequential decision-making, and over-rely on textual interaction history despite newly appended visual feedback. We further introduce a separability analysis to identify which in-game scores provide reliable signals of model competence, and use the benchmark to study how memory design and prompting strategies affect performance. Overall, AtariBench exposes a persistent gap between passive visual perception and active decision-making, providing a lightweight and rigorous testbed for developing VLMs that can reason and act in dynamic environments.

Figures & tables

Figure 1: Workflow of AtariBench. At each turn, the harness assembles game instructions, interaction history, and the current frame into a prompt; the VLM returns a thought and an action sequence, which is executed in ALE, and the resulting frames and feedback are added to the history.

Figure 2: Fast temporal dynamics in Assault . The enemy projectile appears at $t=0.1$ s and already hits the player at $t=0.3$ s, motivating high temporal resolution and 10 Hz control.

Shooter
Game	Primary Skill	#Goals	Action	Reward Density	Visual Load	Tempo
AirRaid	Aiming & Avoidance	2	H+A	Dense	Low	High
Assault	Aiming & Avoidance	3	H+A	Dense	Low	High
BeamRider	Aiming & Avoidance	2	H+A	Dense	High	High
DemonAttack	Aiming & Avoidance	2	H+A	Dense	Low	High
LaserGates	Multi-task Navigation	2	B+A	Dense	High	High

Table 1: AtariBench game taxonomy. #Goals: number of distinct in-game goals. Action: A=fire/attack/throw, H=horizontal, V=vertical, B=both horizontal and vertical. Reward density and visual load are qualitative summaries. Tempo reflects the degree of timing precision required for competent play.

Figure 3: Model performance ranks across the 20 AtariBench games used in the main analysis, including the standard structured-history settings and the Gemini 3.1 Pro append-only (AO) settings. Game-specific RL agents and a random-action baseline are also included. Rows are ordered by average per-game rank; raw reward values are reported in Appendix B.1 .

Figure 4: Game separability analysis across the 20 AtariBench games. Bars show the best-model-vs-random effect size, points show the top-quartile-vs-bottom-quartile effect size, and the right-hand matrix marks whether each of the three criteria passes. Games are strongly separable when at least two criteria pass and weakly separable when exactly one passes.

Figure 5: Append-only versus structured-history performance on three representative games for Gemini 2.5 Flash, Gemini 2.5 Pro, and Gemini 3 Flash. Bars show mean total reward, and error bars show standard error over runs. All models use default thinking.

Model (reasoning)	Perception	Motion	Decision	Overall
Gemini 2.5 Flash (off)	88.0%	51.0%	48.0%	62.3%
Gemini 2.5 Flash (on)	85.0%	47.0%	65.0%	65.7%
Gemini 2.5 Pro (default)	93.0%	54.0%	79.0%	75.3%
Gemini 3 Flash (low)	98.0%	64.0%	88.0%	83.3%
Gemini 3 Flash (high)	99.0%	63.0%	82.0%	81.3%
Gemini 3.1 Pro (low)	100.0%	69.0%	82.0%	83.7%

Table 2: Model accuracy on 60 manually curated diagnostic questions, with five queries per question.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Game	Protocol	Runs	Mean $\pm$ SE	SD
DemonAttack	Game-over termination	30	$70.0\pm 6.6$	$36.3$
DemonAttack	Fixed 30 seconds	10	$41.0\pm 4.3$	$13.7$
Phoenix	Game-over termination	30	$171.3\pm 19.5$	$106.6$
Phoenix	Fixed 30 seconds	10	$198.0\pm 27.4$	$86.6$

Appendix

Table 3: Gemini 2.5 Flash reward variability under original game-over termination versus the current fixed 30-second protocol. We report mean reward

\pm

standard error and per-run standard deviation (SD).

	Phoenix (neutral)				Phoenix (survival prompt)
Model	Reward	Survival	Neutral	Avg	Reward	Survival	Neutral	Avg
	seeking	focused		Reward	seeking	focused		Reward
Gemini 2.5 Flash	76	19	5	177.9	68	25	7	195.4
Gemini 2.5 Pro	59	34	7	239.5	80	16	4	243.8

Appendix

Table 4: Legacy strategy-sensitivity diagnostic under game-over termination. We report the number of reasoning traces classified as reward-seeking or survival-focused out of 100 sampled traces, along with the average reward at termination.

Figure 6: Example evaluation prompt for Breakout.

Figure 7: Raw model rewards across the 20 AtariBench games used in the main analysis, including the Gemini 3.1 Pro append-only settings shown in Fig. 3 . The first row reports rewards for game-specific RL agents on 16 games, with “-” indicating unavailable results. Cell values preserve each game’s raw reward scale, while colors are normalized within each game. Avg Rank is computed as in Fig. 3 .

Model	Game	Full feedback	No feedback
Gemini 3.1 Pro	Breakout	$4.5\pm 0.2$	$4.7\pm 0.3$
	Air Raid	$225.0\pm 23.9$	$215.0\pm 29.6$
	Pacman	$31.6\pm 5.4$	$26.7\pm 3.1$
	Fishing Derby	$-8.4\pm 1.8$	$-8.6\pm 1.4$
Gemini 2.5 Flash	Breakout	$2.3\pm 0.3$	$1.5\pm 0.3$
	Air Raid	$255.0\pm 45.3$	$282.5\pm 30.1$

Appendix

Table 5: Textual feedback ablation. Scores are mean

\pm

SE over ten seeds.

Figure 8: Breakout example with Gemini 2.5 Flash in which textual life-loss feedback makes the reset state explicit. After the harness reports that a new life has begun, the model selects start , which serves the next ball and resumes play. Without textual feedback, the model stalls and fails to press start to proceed.

Model	Game	1 action	1–5 actions	1–10 actions
Gemini 3 Flash	Air Raid	$295.0\pm 33.3$	$340.0\pm 36.7$	$300.0\pm 31.0$
	Breakout	$4.6\pm 0.2$	$4.5\pm 0.2$	$3.7\pm 0.3$
	Fishing Derby	$-6.6\pm 2.1$	$-8.4\pm 1.7$	$-7.2\pm 1.4$
	Pacman	$17.4\pm 4.0$	$13.6\pm 3.1$	$11.9\pm 2.8$
GPT-5.4 Mini	Air Raid	$175.0\pm 18.3$	$265.0\pm 39.8$	$217.5\pm 41.2$
	Breakout	$3.4\pm 0.2$	$3.8\pm 0.3$	$3.9\pm 0.3$

Appendix

Table 6: Reward under different allowed action-chunk lengths (mean

\pm

SE over ten seeds). Variable-length conditions permit the model to choose shorter sequences at each turn.

Figure 9: In Name This Game, GPT-5.4 ( high ) states that no shark is visible despite the white shark in the current frame (10.30 s; enlarged at right).

Figure 10: In this Freeway case, GPT-5.4 ( high ) describes a historical car position as current; with history removed and the remaining input and model settings unchanged, the erroneous relation is no longer stated (see Appendix D.2 ).

Figure 11: Breakout example of Gemini 3.1 Pro where the model misidentifies the ball’s motion direction.

Figure 12: Assault text-history-overreliance case in GPT-5.4 Mini. After an earlier rewarded upward shot, the model says recent shots are hitting despite updated frames showing the remaining alien left of its firing lane. Two more upward shots yield no reward.

Figure 13: Example of action stagnation under append-only prompting in Breakout. Despite receiving updated frames and feedback, the model repeatedly outputs only noop actions; we observe this behavior in Gemini 2.5 Flash, Gemini 2.5 Pro, and Gemini 3 Flash.

Figure 14: Original VLM-judge failures (left,

n=106

) and paired spatial-relation re-queries (right,

n=28

). Current-frame-only inputs remove history images and associated text while retaining the current frame, game instructions, and model settings. Responses without a spatial-relation statement are re-queried with identical input; the first response stating a relation is assessed, whether correct or incorrect. Correct relations need not explicitly repair the original claim. The matched error odds ratio (OR; current only versus complete input) has an exact 95% confidence interval.

Capability	Prompt text before interleaved frames
Perception: detection	This is the Atari game Fishing Derby. How many yellow fish are in the game? Choose from 4, 5, 6, or 7.
Perception: spatial reasoning	This is the Atari game Fishing Derby. In the last frame, focus on the deepest fish in the left half of the sea and the left player’s fishing line. Is the left player’s fishing line currently to the left or to the right of that fish? Choose from left or right.
Motion: target object	This is the Atari game Fishing Derby. Focus on the deepest fish in the left half of the sea. Which direction is that fish moving? Choose from left or right.
Motion: controlled object	This is the Atari game Fishing Derby. Focus on the left player’s fishing line. Across the timestamped sequence, what is the fishing line’s overall movement direction? Choose only one from upright, upleft, downleft, downright, up, down, left, right, or still.
Decision: short	This is the Atari game Fishing Derby. Focus on the deepest fish in the left half of the sea and the left player’s fishing line. To catch that deepest fish, which direction should the left player’s fishing line move next? Choose from left or right.
Decision: long	This is the Atari game Fishing Derby. Action Planning Guidelines: [complete Fishing Derby guide included in the model prompt; omitted here, approximately 500 tokens of instruction]. Using the frame sequence, focus on the deepest fish in the left half of the sea and the left player’s fishing line. First identify that target fish and the current position of the left player’s fishing line, then infer their motion. To catch that fish, which direction should the left player’s fishing line move next? Choose from left or right.

Appendix

Table 7: Diagnostic question prompts for each capability group, using Fishing Derby as an example. The long decision prompt additionally prepends the game-specific guidelines.

Setting	State-estimation failures	Planning failures with correct state	Other	Propagation
GPT-5.4 (high)	37/50 (74%)	3/50 (6%)	10/50 (20%)	26/37 (70.3%)
GPT-5.4 (low)	29/50 (58%)	12/50 (24%)	9/50 (18%)	20/29 (69.0%)

Appendix

Table 8: Failure types among 50 failed turns per setting. State estimation, planning with a substantially correct state estimate, and other failures are mutually exclusive. Propagation is measured only among state-estimation failures.

Model setting	Turns	Failures	Text Overreliance Failures	Overreliance Rate
GPT-5.4 (high)	60	21	2	9.5%
Gemini 3.1 Pro (low)	60	20	0	0.0%
GPT-5.4 Mini (low)	60	24	5	20.8%
Gemini 2.5 Flash (off)	60	43	10	23.3%
Two stronger settings	120	41	2	4.9%
Two weaker settings	120	67	15	22.4%

Appendix

Table 9: Text-history overreliance in sampled gameplay turns. Failures counts the observable failures identified by the judge, and Overreliance Rate is the percentage of these failures attributed to text-history overreliance.

Setting	Runs	Calls	Input tok.	Output tok.	Thinking tok.	Est. cost
Ours	200	27,938	140.81M	2.85M	17.18M	$522

Appendix

Table 10: Estimated Gemini 3.1 Pro API cost for 200 AtariBench runs (20 games

\times

10 runs).

Temporal Reasoning in VLMs

See all 42 papers

See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay
Mar 12, 2026Ashish Baghel, Paras ChopraVision-Language Agents Vision-Language Grounding
Lossfunk
Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?
Oct 13, 2026Qingan Liu, Juntong Feng, Yue Zhu +6VLM Evaluation Visually Grounded Reasoning
Dalian University of Technology · Nanyang Technological University
Can Vision-Language Models Solve the Shell Game?
Mar 9, 2026Tiedong Liu, Wee Sun LeeVisual Object Tracking Temporal Video Understanding
National University of Singapore

Mar 12, 2026J/K move · Enter open · S save

See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay

Ashish Baghel, Paras Chopra

Lossfunk

Vision-Language Agents Vision-Language Grounding VLM Reasoning Game-Playing Agents

Open paper AskPDF arXiv

Vision-Language Models (VLMs) excel at describing visual scenes, yet struggle to translate perception into precise, grounded actions. We investigate whether providing VLMs with both the visual frame and the symbolic representation of the scene can improve their performance in interactive environments. We evaluate three state-of-the-art VLMs across Atari games, VizDoom, and AI2-THOR, comparing frame-only, frame with self-extracted symbols, frame with ground-truth symbols, and symbol-only pipelines. Our results indicate that all models benefit when the symbolic information is accurate. However, when VLMs extract symbols themselves, performance becomes dependent on model capability and scene complexity. We further investigate how accurately VLMs can extract symbolic information from visual inputs and how noise in these symbols affects decision-making and gameplay performance. Our findings reveal that symbolic grounding is beneficial in VLMs only when symbol extraction is reliable, and highlight perception quality as a central bottleneck for future VLM-based agents.

Abstract

Temporal Reasoning in VLMs

Explore similar work

See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay

Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?

Can Vision-Language Models Solve the Shell Game?