GameVerse: Can Vision-Language Models Learn from Video-based Reflection?

Kuan Zhang; Dongchen Liu; Qiyue Zhao; Jinkun Hou; Xinran Zhang; Qinlei Xie; Miao Liu; Yiming Li

cs.CVMar 1, 2026

GameVerse: Can Vision-Language Models Learn from Video-based Reflection?

Authors: Kuan Zhang, Dongchen Liu, Qiyue Zhao, Jinkun Hou, Xinran Zhang, Qinlei Xie, Miao Liu, Yiming Li

Organizations: College of AI, Tsinghua University, China.

Abstract

Human gameplay is a visually grounded interaction loop in which players act, reflect on failures, and watch tutorials to refine strategies. Can Vision-Language Models (VLMs) also learn from video-based reflection? We present GameVerse, a comprehensive video game benchmark that enables a reflective visual interaction loop. Moving beyond traditional fire-and-forget evaluations, it uses a novel reflect-and-retry paradigm to assess how VLMs internalize visual experience and improve policies. To facilitate systematic and scalable evaluation, we also introduce a cognitive hierarchical taxonomy spanning 15 globally popular games, dual action space for both semantic and GUI control, and milestone evaluation using advanced VLMs to quantify progress. Our experiments show that VLMs benefit from video-based reflection in varied settings, and perform best by combining failure trajectories and expert tutorials-a training-free analogue to reinforcement learning (RL) plus supervised fine-tuning (SFT).Our project page is available at https://gameverse-bench.github.io/ . Our code is available at https://github.com/THUSI-Lab/GameVerse .

Figures & tables

Figure 1: Humans improve gameplay by reflecting on failures and consulting expert tutorials. Our benchmark mimics this process, enabling agents to learn from video-based reflection. Humans show the largest gains, while all models benefit to varying degrees.

Benchmarks	Game Env.	Taxonomy	# Games	Diff. Tiers	Major Modality	Privilege-Free	Paradigm	Action	Scalability
GameArena ( Hu et al., 2025b )	1D Text World	✗	3	✗	Text-Centric	✓	Fire-and-Forget	Semantic	✓
BALROG ( Paglieri et al., 2025 )	1D Text + 2D Grid	✗	6	✓	Text-Centric $\dagger$	Scaffold	Fire-and-Forget	Semantic	✗
LVLM-Play ( Wang et al., 2025b )	2D Grid	✗	6	✓	Vision-Centric	✓	Fire-and-Forget	Semantic	✓
VideoGameBench ( Zhang et al., 2025a )	2D Retro Arcade	✗	10	✗	Vision-Centric	✓	Fire-and-Forget	GUI	✓
FlashAdventure ( Ahn et al., 2025 )	2D Web Flash	✗	34	✗	Vision-Centric	✓	Fire-and-Forget	GUI	✗
V-MAGE ( Zheng et al., 2025 )	2D Retro Arcade	Comm. Genre	5	✓	Vision-Centric	✓	Fire-and-Forget	Semantic	✓

Table 1: G ame V erse ( GV ) distinguishes itself through a Cognitive Taxonomy rather than commercial genres ( Comm.Genre ), Dual Action Space , and a Visually-Grounded Interaction Loop allowing agents to reflect and retry. Mixed features selecting games spanning various eras and genres. Tag

\dagger

denotes using scaffolds/APIs to extract ground-truth state text, with vision as an auxiliary role.

Figure 2: Overview of GameVerse , which effectively probes the capability boundaries of VLMs in diverse video game worlds. GameVerse supports dual action space, enables human-like reflection by integrating failure and tutorial videos, and delivers process score.

Game	Trials ( $N$ ) / Time ( $T$ )	Eval. Mode	Feature & Capabilities Evaluated
Markov Grid
Tic-Tac-Toe	10 / 40 sec	$A_{S},A_{G}$	Classic zero-sum grid game; tests basic adversarial logical reasoning and turn-taking planning .
Baba Is You	5 / 1.5 min	$A_{S},A_{G}$	Puzzle with manipulable rules; tests out-of-the-box thinking and flexible logical inference .
2048	5 / 10 min	$A_{S},A_{G}$	Stochastic number puzzle; tests probabilistic planning and spatial foresight to manage grid constraints.
Non-Real-Time Linear
Maze	10 / 45 sec	$A_{S},A_{G}$	Navigational puzzle; tests virtual perception and spatial pathfinding algorithms.

Table 2: Overview of the 15 games in GameVerse. We detail the number of trials (

N

), average time per round (

T

), supported evaluation modes (Semantic

A_{S}

and GUI

A_{G}

), and a concise summary of the game content and the specific cognitive capabilities evaluated.

Model

Markov Grid

Non-Real-time Linear

Non-Real-time Non-Linear

Real-time Linear

Real-time Non-Linear

Avg

TicTacToe

Baba

2048

Maze

AngryBird

Slay

Attorney

Civilization

Scene

Snake

PvZ

Horizon

Metro

Genshin

RDR 2

Rank

Three Baselines

Random

—

26.3

\pm

19.3

10.8

\pm

7.6

1.2

\pm

0.9

21.3

\pm

2.7

0.0

\pm

0.0

\pm

0.0

\pm

0.0

\pm

0.0

\pm

0.0

7.0

\pm

9.1

0.0

\pm

0.0

\pm

0.0

\pm

0.0

\pm

0.0

\pm

0.0

16.5

Human Expert

—

98.9

\pm

2.6

100.0

\pm

0.0

77.1

\pm

16.4

100.0

\pm

0.0

85.3

\pm

7.2

99.4

\pm

1.2

100.0

\pm

0.0

100.0

\pm

0.0

94.2

\pm

6.3

100.0

\pm

0.0

100.0

\pm

0.0

98.1

\pm

4.4

100.0

\pm

0.0

100.0

\pm

0.0

100.0

\pm

0.0

1.6

Human Rookie

85.1

\pm

19.3

83.4

\pm

15.2

15.9

\pm

4.1

99.3

\pm

1.2

46.4

\pm

10.3

34.2

\pm

7.1

77.4

\pm

27.2

54.1

\pm

17.2

70.4

\pm

5.2

93.2

\pm

10.4

89.3

\pm

10.8

73.9

\pm

9.1

91.2

\pm

15.4

63.3

\pm

18.2

58.4

\pm

21.3

4.1

Yes

99.4

\pm

1.9

100.0

\pm

0.0

47.3

\pm

18.2

100.0

\pm

0.0

74.2

\pm

5.4

81.3

\pm

14.1

100.0

\pm

0.0

96.4

\pm

8.9

90.1

\pm

9.3

100.0

\pm

0.0

97.9

\pm

4.4

85.3

\pm

3.1

98.4

\pm

4.2

88.2

\pm

18.4

83.1

\pm

21.2

2.2

Table 3: Performance of 7 VLMs on 15 GameVerse games in GUI mode, with and without Video-based Reflection (VR). Green / Red cells indicate performance gain/drop after applying VR. Bold represents the top performance across all models and human baselines.

Figure 3: Top: Average performance across 5 cognitive and 3 difficulty categories. Bottom: Average improvement across models.

Figure 4: Left: Average improvement across 5 cognitive category tasks. Right: Overall improvement statistics of 120 trials.

Model	VR	TicTacToe		AngryBird		Slay the Spire		Plants vs. Zombies
Model	VR	GUI	Semantic	GUI	Semantic	GUI	Semantic	GUI	Semantic
Random	—	26.3 $\pm$ 19.3	30.4 $\pm$ 24.1	0.0 $\pm$ 0.0	7.4 $\pm$ 6.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0
Qwen3-VL-8B	No	53.3 $\pm$ 15.4	51.7 $\pm$ 13.6	19.6 $\pm$ 8.3	37.2 $\pm$ 11.7	11.6 $\pm$ 0.0	11.7 $\pm$ 0.0	33.4 $\pm$ 4.1	54.2 $\pm$ 5.8
Qwen3-VL-8B	Yes	51.8 $\pm$ 14.6	67.0 $\pm$ 12.9	31.2 $\pm$ 10.8	63.2 $\pm$ 15.1	7.4 $\pm$ 8.4	5.4 $\pm$ 5.2	51.3 $\pm$ 21.4	65.2 $\pm$ 11.3
GPT-4o	No	58.6 $\pm$ 4.2	66.8 $\pm$ 10.2	14.3 $\pm$ 9.0	49.3 $\pm$ 13.0	2.0 $\pm$ 0.0	25.3 $\pm$ 10.2	32.2 $\pm$ 12.4	42.7 $\pm$ 16.0
GPT-4o	Yes	64.2 $\pm$ 9.8	64.3 $\pm$ 8.0	15.4 $\pm$ 12.3	61.5 $\pm$ 17.6	2.0 $\pm$ 0.0	29.4 $\pm$ 8.1	36.4 $\pm$ 15.2	53.4 $\pm$ 7.8

Table 4: Performance of 4 VLMs on 4 games of GameVerse, evaluated with and without Video-base Reflection( VR ) in two modes.

Figure 5: Performance of 4 VLMs on 2 games of GameVerse in semantic action mode with three latency settings. Reactive model is average performance of GPT-4o and Qwen3-VL-8B. Reasoning model is average performance of Seed-1.8 and Gemini-2.5-Pro.

Model	VR	TicTacToe	AngryBird	Plants vs. Zombies	Avg
GPT-4o-mini weaker model	No	29.2 $\pm$ 15.1	54.6 $\pm$ 6.2	53.2 $\pm$ 25.6	45.6
	Self-F	43.1 $\pm$ 4.3	57.4 $\pm$ 13.4	52.1 $\pm$ 18.0	50.8
	Self-T	49.2 $\pm$ 13.7	73.2 $\pm$ 8.2	63.3 $\pm$ 13.1	61.9
	Self	52.3 $\pm$ 16.0	75.8 $\pm$ 10.6	68.6 $\pm$ 14.6	65.5
	Other	45.0 $\pm$ 7.2	77.1 $\pm$ 7.0	55.4 $\pm$ 7.2	59.1
Qwen3-VL-32B stronger model	No	67.0 $\pm$ 15.3	48.5 $\pm$ 11.3	67.6 $\pm$ 26.0	61.0

Table 5: Performance comparison of 2 VLMs on 3 games in semantic mode. The VR settings are categorized into: No , Self-F , Self-T , Self , and Other (the alternative VLM).

Panel of Figure 6 — Figure 6: Screenshots of Gemini-2.5-Pro playing 2048 and trajectories of Qwen3-VL-8B playing angry birds level 2.

Panel of Figure 7 — Figure 7: Top: GameVerse results breakdown of Qwen3-VL-32B. Bot : Failure cases about 4 types of errors in GameVerse.

Appendix figures & tables85 assets

Supplementary material from the paper’s appendix.

Appendix

Action Type	Description	Parameters
Mouse Movement
MOVE_TO	Move to position	x , y (int, req): Target coordinates (window relative)
MOVE_BY	Move relatively	dx , dy (int, req): Offset; duration (float, opt): Seconds
Mouse Click
CLICK	Click mouse	x , y (opt); button (left/right/middle); num_clicks
RIGHT_CLICK	Right click	x , y (int, opt): Click coordinates

Appendix

Table 6: GUI Action Types Summary

Figure 8: Prompt of reflecting experience from failure and tutorial

Game	Video Type	Link	Selected Time
Tic-Tac-Toe	Skill Tutorial	https://www.youtube.com/shorts/ZBZcnImNmhk	00:00:59
Baba Is You	Walkthrough	https://www.bilibili.com/video/BV1wb411p7ja/	00:00:38
2048	Skill Tutorial	https://www.bilibili.com/video/BV1VH4y1K7Gr/	00:03:02
Maze	Skill Tutorial	https://www.bilibili.com/video/BV1pp4y197fY/	00:03:37
Angry Birds	Walkthrough	https://www.bilibili.com/video/BV1b54y1s7pe/	00:01:31
Slay the Spire	Skill Tutorial	https://www.bilibili.com/video/BV1RVdLYyEFf/	00:09:36

Appendix

Table 7: List of longplay walkthrough links for games in GameVerse.

Figure 9: Prompt of milestone extraction and milestone matching

Figure 10: 9 milestones are extracted by Gemini-3-pro in the walkthrough video of Ace Attorney.

Game	Video Length	# Milestone	Hallucination Rate	Representation Score
Baba Is You	00:00:38	3 ( L 0), 5 ( L 1)	12.5%	0.92
Ace Attorney	00:21:50	9	0%	1.00
Civilization VI	00:19:23	12	17%	0.81
Scene Investigators	00:27:26	17	11%	0.95
Plants vs. Zombies	00:05:51	7 ( L 1), 6( L 2), 6( L 4)	5%	0.78
Genshin Impact	00:25:51	7	0%	0.95

Appendix

Table 8: The breakdown of the extracted milestones. L means level. Hallucination rate is defined as

\frac{\#inaccurate~~milestones}{\#total~~milestones}

. Representation score is defined as how representative the milestones are to significant game nodes, ranging in

[0,1]

. The score is evaluated by 10 human video game experts

Game	Agent gameplay video (discrete input frames)			Human gameplay video (continuous frame sequences)
Game	VLM eval.	human eval.	Percentage Diff. $\downarrow$	VLM eval.	human eval.	Percentage Diff. $\downarrow$
Baba Is You	0.68 $\pm$ 0.15	0.74 $\pm$ 0.10	8.1%	0.98 $\pm$ 0.06	1.00 $\pm$ 0.00	2.0%
Ace Attorney	0.29 $\pm$ 0.17	0.33 $\pm$ 0.19	12.1%	1.00 $\pm$ 0.00	1.00 $\pm$ 0.00	0%
Civilization VI	0.04 $\pm$ 0.08	0.04 $\pm$ 0.08	0.0%	0.92 $\pm$ 0.11	0.96 $\pm$ 0.09	4.2%
Scene Investigators	0.05 $\pm$ 0.04	0.05 $\pm$ 0.04	0.0%	0.90 $\pm$ 0.09	0.90 $\pm$ 0.09	0%
Plants vs. Zombies	0.34 $\pm$ 0.15	0.38 $\pm$ 0.13	10.5%	1.00 $\pm$ 0.00	0.98 $\pm$ 0.04	2.0%

Appendix

Table 9: The average VLM-evaluated and human-evaluated matching scores on agent play video and human play video with percentage difference across seven games with VL.

Human	Markov Grid			Non-Real-time Linear			Non-Real-time Non-Linear			Real-time Linear			Real-time Non-Linear			Total
Human	TicTacToe	Baba	2048	Maze	AngryBird	Slay	Attorney	Civ VI	Scene	Snake	PvZ	Horizon	Metro	Genshin	RDR 2	Total
Human Rookie	10	3	10	10	5	4	4	6	4	4	3	3	8	2	4	80
Total trials	100	3	50	30	15	4	4	6	4	12	9	9	24	2	4	276
Human Expert	6	6	5	10	4	7	2	3	3	3	3	2	2	2	2	60
Total trials	60	6	25	30	12	7	2	3	3	9	9	6	6	2	2	182

Appendix

Table 10: The statistics of human baselines for each game.

Game Factors	Markov Grid			Non-Real-time Linear			Non-Real-time Non-Linear			Real-time Linear			Real-time Non-Linear
Game Factors	TicTacToe	Baba	2048	Maze	AngryBird	Slay	Attorney	Civ VI	Scene	Snake	PvZ	Horizon	Metro	Genshin	RDR 2
Env. Complexity	1	2	2	1	3	4	4	5	5	2	4	4	3	5	5
Action Space	1	1	1	1	3	3	3	5	5	1	2	3	4	5	5
Rule Difficulty	1	2	3	1	2	4	3	5	5	2	3	3	4	5	5
Reaction Time	1	1	1	1	1	1	1	1	1	2	3	5	4	4	5
Content Scale	1	2	3	1	1	3	3	3	5	1	3	3	3	4	5

Appendix

Table 11: Analysis of game factors across GameVerse. The factors are evaluated on a 5-point scale , where 1 represents easy and 5 represents hard .

Model	GUI	GUI VR.	Semantic	Semantic VR.
Qwen3-VL-8B	$53.0\pm 15.0$	$52.0\pm 15.0$	$52.0\pm 14.0$	$67.0\pm 13.0$
Qwen3-VL-32B	$71.0\pm 17.0$	$63.0\pm 7.0$	$-$	$-$
GPT-4o-mini	$34.0\pm 17.0$	$43.0\pm 12.0$	$-$	$-$
GPT-4o	$59.0\pm 4.0$	$64.0\pm 10.0$	$67.0\pm 10.0$	$64.0\pm 8.0$
Seed-1.8	$\mathbf{92.0\pm 3.0}$	$\mathbf{100.0\pm 0.0}$	$-$	$-$
Gemini-2.5-Flash	$88.0\pm 16.0$	$95.0\pm 12.0$	$94.0\pm 9.0$	$\mathbf{100.0\pm 0.0}$

Appendix

Table 12: Tic-Tac-Toe Raw Scores

Model	GUI	GUI VR.
Gemini-2.5-Flash	$80.0\pm 0.0$	$80.0\pm 0.0$
Gemini-2.5-Pro	$80.0\pm 0.0$	$80.0\pm 0.0$
Qwen3-VL-8B	$60.0\pm 0.0$	$70.0\pm 8.2$
Qwen3-VL-32B	$73.3\pm 9.4$	$80.0\pm 0.0$
GPT-4o	$61.0\pm 1.4$	$60.0\pm 0.0$
GPT-4o-mini	$60.0\pm 0.0$	$66.7\pm 11.5$

Appendix

Figure 16: Raw scores in Baba Is You

Figure 17: Typical errors in Baba Is You

Figure 18: Qwen3-VL-8B reflection improvement example

Model	GUI	GUI VR.
Qwen3-VL-8B	$875.6\pm 417.9$	$1014.9\pm 238.8$
Qwen3-VL-32B	$1273.6\pm 636.8$	$1213.9\pm 736.3$
GPT-4o-mini	$258.7\pm 79.6$	$278.6\pm 238.8$
GPT-4o	$756.2\pm 179.1$	$676.6\pm 218.9$
Seed-1.8	$1810.9\pm 1034.8$	$3621.8\pm 1671.6$
Gemini-2.5-Flash	$1552.2\pm 417.9$	$2268.6\pm 756.2$

Appendix

Figure 21: 2048 Raw Scores

Figure 22: 2048 typical errors and reflection improvements

Model	GUI	GUI VR.
Qwen3-VL-8B	$80.0\pm 45.0$	$89.0\pm 31.0$
Qwen3-VL-32B	$100.0\pm 0.0$	$100.0\pm 0.0$
GPT-4o-mini	$77.0\pm 43.0$	$91.0\pm 26.0$
GPT-4o	$80.0\pm 44.0$	$85.0\pm 39.0$
Seed-1.8	$88.0\pm 21.0$	$100.0\pm 0.0$
Gemini-2.5-Flash	$100.0\pm 0.0$	$100.0\pm 0.0$

Appendix

Table 13: Maze Raw Scores

Model	GUI	GUI VR.	Semantic	Semantic VR.
Qwen3-VL-8B	26068 $\pm$ 11039	41496 $\pm$ 14364	49476.0 $\pm$ 15561	84056 $\pm$ 20083
Qwen3-VL-32B	55328 $\pm$ 9842	79002 $\pm$ 14497	-	-
GPT-4o-mini	22211 $\pm$ 15029	17689 $\pm$ 13566	-	-
GPT-4o	19019 $\pm$ 11970	20482 $\pm$ 16359	65569 $\pm$ 17290	81795 $\pm$ 23408
Seed-1.8	39102 $\pm$ 9443	76076 $\pm$ 16891	-	-
Gemini-2.5-Flash	12502 $\pm$ 10906	13566 $\pm$ 8512	60914 $\pm$ 12236	90972 $\pm$ 20482

Appendix

Table 14: Raw Score of Angry Birds

Model	Baseline	Expert VR.	Failure VR.	Joint VR.
GPT-4o-mini	$55.0\pm 6.0$	$73.0\pm 8.0$	$57.0\pm 13.0$	$\mathbf{77.0\pm 7.0}$
Qwen3-VL-32B	$50.0\pm 11.0$	$48.0\pm 12.0$	$54.0\pm 13.0$	$\mathbf{74.0\pm 10.0}$

Appendix

Table 15: Performance comparison of Expert vs. Failure reflection

Model	GUI	Semantic	GUI VR.	Semantic VR.
Gemini-2.5-Flash	$1\pm 0$	$6\pm 0$	$1\pm 0$	$8.0\pm 0.0$
Gemini-2.5-Pro	$2.0\pm 1.73$	$19.75\pm 11.5$	$4\pm 0.0$	$14.0\pm 4.0$
Qwen3-VL-8B	$6\pm 0$	$6\pm 0$	$3.33\pm 4.0$	$2.25\pm 2.5$
Qwen3-VL-32B	$4.2\pm 1.92$	–	$5.75\pm 0.5$	–
GPT-4o	$1\pm 0$	$13.5\pm 5.0$	$1\pm 0$	$14.0\pm 4.0$
GPT-4o-mini	$1\pm 0$	–	$1\pm 0$	–

Appendix

Figure 31: raw scores in slay the spire

Figure 32: slay the spire typical errors

Figure 33: Example of Qwen3-VL-32B Reflection Improvement

No.	Milestone Title	No.	Milestone Title
1	Identify Defendant	6	Reveal Weapon’s Secret
2	Identify Victim	7	Sound the Clock
3	Identify Cause of Death	8	Solve the Time Gap
4	Expose Time Contradiction	9	Not Guilty Verdict
5	Expose TV Logic Hole

Appendix

Table 16: Milestones of Ace Attorney

Model	GUI	GUI VR.
Qwen3-VL-8B	$7.0\pm 6.0$	$26.0\pm 19.0$
Qwen3-VL-32B	$17.0\pm 6.0$	$37.0\pm 6.0$
GPT-4o-mini	$0.0\pm 0.0$	$0.0\pm 0.0$
GPT-4o	$18.0\pm 6.0$	$33.0\pm 11.0$
Seed-1.8	$33.0\pm 0.0$	$59.0\pm 6.0$
Gemini-2.5-Flash	$33.0\pm 11.0$	$29.0\pm 25.0$

Appendix

Figure 37: Raw scores in Ace Attorney

Figure 38: Screenshot of Civilization VI

No.	Milestone Title	No.	Milestone Title
1	Found First City	7	Place Campus District
2	Select First Technology	8	Optimize Military Policy
3	Tribal Village Discovery	9	Declare Surprise War
4	Meet Rival Civilization	10	Capture Enemy Settler
5	Discover Natural Wonder	11	Found City with Captured Unit
6	Found Second City	12	Capture Capital & Victory

Appendix

Table 17: Milestones of Civilization VI

Figure 39: Civilization VI zero-shot agent prompt

Figure 40: Civilization VI memory agent prompt

Model	Zero-shot	Memory	Zero-shot VR.	Memory VR.
Gemini-2.5-Flash	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$
Gemini-2.5-Pro	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$
Qwen3-VL-8B	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$
Qwen3-VL-32B	$19.4\pm 4.0$	$8.3\pm 0.0$	$16.7\pm 0.0$	$8.3\pm 0.0$
GPT-4o	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$
GPT-4o-mini	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$

Appendix

Table 18: Scores in Civilization VI (Mean

\pm

Std)

Figure 42: Screenshot of Scene Investigators

Figure 43: Scene Investigators inference prompt

Figure 44: Scene Investigators review prompt

Model	Zeroshot	Memory	Memory VR.
Gemini-2.5-Flash	–	$2.78\pm 3.9$	$8.3\pm 0.0$
Gemini-2.5-Pro	$0.0\pm 0.0$	$0.0\pm 0.0$	$0.0\pm 0.0$
Qwen3-VL-8B	$5.6\pm 3.93$	$5.6\pm 3.93$	$5.6\pm 3.93$
Qwen3-VL-32B	$8.3\pm 0.0$	$8.3\pm 0.0$	$8.3\pm 0.0$
GPT-4o	–	$0.0\pm 0.0$	$0.0\pm 0.0$
GPT-4o-mini	–	$2.78\pm 3.9$	$2.78\pm 3.9$

Appendix

Figure 45: raw scores in Scene Investigators

Figure 47: Qwen3-VL-32B memory improvement example

Figure 48: Qwen3-VL-8B reflection improvement example

Model	GUI	GUI VR.
Gemini-2.5-Flash	$2.00\pm 1.80$	$1.00\pm 0.81$
Gemini-2.5-Pro	$2.38\pm 3.00$	$6.75\pm 3.20$
Qwen3-VL-8B	$0.25\pm 0.71$	$0.00\pm 0.00$
Qwen3-VL-32B	$4.38\pm 2.61$	$3.25\pm 1.71$
GPT-4o	$0.00\pm 0.00$	$0.00\pm 0.00$
GPT-4o-Mini	$0.00\pm 0.00$	$0.00\pm 0.00$

Appendix

Table 19: Main Experiment Results(w/ vs. w/o reflection).

Model	10s	20s	Stop
Qwen3-VL-32B	$4.38$	$4.75$	$5.33$
GPT-4o	$0.00$	$0.50$	$0.00$
Gemini-2.5-Pro	$2.38$	$6.50$	$7.33$
Seed-1.8	$0.75$	$0.86$	$9.00$

Appendix

Table 20: Latency Analysis

Figure 52: Qwen3-VL-32B reflection content

Figure 53: Screenshot of Plant vs. Zombies

Level 1-1		Level 1-2		Level 1-4
No.	Milestone Title	No.	Milestone Title	No.	Milestone Title
1	Planting First Peashooter	1	First Economic Setup	1	Level Start
2	First Sun Collection	2	First Enemy Wave Warning	2	First Economic Setup
3	Planting Second Peashooter	3	First Defense Deployment	3	First Defensive Action
4	First Zombie Appearance	4	Huge Wave Alert	4	Countering Buckethead
5	Defeating First Zombie	5	Final Wave Start	5	Huge Wave Event

Appendix

Table 21: Milestones of Plants vs. Zombies (Levels 1-1, 1-2, and 1-4)

Figure 54: Plants vs. Zombies GUI prompt

Figure 55: Plants vs. Zombies semantic prompt

Model	GUI	GUI VR.	Semantic	Semantic VR.
Qwen3-VL-8B	33.4 $\pm$ 4.1	51.3 $\pm$ 21.4	54.2 $\pm$ 5.8	65.2 $\pm$ 11.3
Qwen3-VL-32B	41.2 $\pm$ 12.6	48.3 $\pm$ 19.0	-	-
GPT-4o-mini	27.6 $\pm$ 14.0	22.4 $\pm$ 10.4	-	-
GPT-4o	32.2 $\pm$ 12.4	36.4 $\pm$ 15.2	42.7 $\pm$ 16.0	53.4 $\pm$ 7.8
Seed-1.8	26.1 $\pm$ 8.2	22.1 $\pm$ 10.3	-	-
Gemini-2.5-Flash	51.1 $\pm$ 13.2	55.3 $\pm$ 10.4	70.2 $\pm$ 8.5	81.8 $\pm$ 17.0

Appendix

Table 22: Model Accuracy Comparison: GUI vs. Semantic (with and without VR)

Figure 56: Plant vs. Zombies Typical Errors

Figure 57: Screenshot of Forza Horizon 5

Model	GUI	GUI VR.
Gemini-2.5-Flash	$1.333\pm 0.577$	$1.333\pm 0.58$
Gemini-2.5-Pro	$1.0\pm 0.0$	$1.25\pm 0.5$
Qwen3-VL-8B	$1.0\pm 0.0$	$1.0\pm 0.0$
Qwen3-VL-32B	$1.0\pm 0.0$	$1.0\pm 0.0$
GPT-4o	$0.75\pm 0.5$	$1.0\pm 0.0$
GPT-4o-Mini	$1.0\pm 0.0$	$1.0\pm 0.0$

Appendix

Table 23: Raw Score of Forza Horizon 5

Model	GUI	GUI VR.
Gemini-2.5-Flash	$74.5\pm 19.29$	$78.0\pm 46.4$
Gemini-2.5-Pro	$57.4\pm 15.09$	$87.2\pm 40.62$
Qwen3-VL-8B	$39.33\pm 23.51$	$41.25\pm 20.09$
Qwen3-VL-32B	$26.5\pm 19.15$	$52.8\pm 22.28$
GPT-4o	$58.0\pm 11.42$	$68.0\pm 13.78$
GPT-4o-mini	$23.0\pm 15.6$	$34.0\pm 12.20$

Appendix

Table 24: Raw Performance on Mini Metro

Milestones

1. Basic Operation: Follow Paimon’s instructions to learn the basic operations.

2. Unlock The Statue of The Seven: Interact with the first Statue of The Seven at Starfell Lake.

3. First Combat Encounter: The player encounters an enemy and completes the combat tutorial using the newly acquired skills.

4. Encounter with Dvalin: The player enters the Whispering Woods and triggers the plot where the dragon Dvalin interacts with the bard.

5. Meet Amber: The player encounters Amber, a scout knight of the Knights of Favonius, on the way to Mondstadt.

6. Clear the Hilichurl camp: The player clears the nearby Hilichurl camp to proceed further.

Appendix

Table 25: Milestones of Genshin Impact

Figure 64: Genshin Impact zero-shot prompt

Figure 65: Genshin Impact memory agent prompt

Model	Zero-shot	Memory	Zero-shot + VR.	Memory + VR.
Gemini-2.5-Flash	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$
Gemini-2.5-Pro	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$
Qwen3-VL-8B	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$
Qwen3-VL-32B	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$
GPT-4o	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$
GPT-4o-mini	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$	$14.3\pm 0.0$

Appendix

Table 26: Scores in Genshin Impact (Mean

\pm

Std)

Figure 67: Screenshot of Red Dead Redemption 2

Model	Zeroshot	Memory	Memory VR.
Gemini-2.5-Flash	$7.0\pm 5.0$	$10.0\pm 0.0$	$13.0\pm 5.0$
Gemini-2.5-Pro	$10.0\pm 0.0$	$10.0\pm 0.0$	$10.0\pm 0.0$
Qwen3-VL-8B	$7.0\pm 5.0$	$7.0\pm 5.0$	$7.0\pm 5.0$
Qwen3-VL-32B	$10.0\pm 8.0$	$7.0\pm 5.0$	$10.0\pm 8.0$
GPT-4o	$7.0\pm 5.0$	$10.0\pm 0.0$	$10.0\pm 0.0$
GPT-4o-mini	$7.0\pm 5.0$	$10.0\pm 0.0$	$10.0\pm 0.0$

Appendix

Figure 69: Scores in Red Dead Redemption 2

Figure 71: Gemini-2.5-Flash reflection improvement example

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics
Jun 8, 2026Mingxian Lin, Shengju Qian, Yuqi Liu +9VLM Evaluation Game-Playing Agents
1The University of Hong Kong · 2LIGHTSPEED · 3The Chinese University of Hong Kong +1
Do Vision Language Models Understand Human Engagement in Games?
Mar 19, 2026Ziyi Wang, Qizan Guo, Rishitosh Singh +1VLM Evaluation Video-Language Models
Arizona State University
Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning
Jul 2, 2026Liyan Tang, Fangcong Yin, Greg DurrettVisually Grounded Reasoning Vision-Language Models
The University of Texas at Austin · New York University

Jun 8, 2026J/K move · Enter open · S save

OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics

Mingxian Lin, Shengju Qian, Yuqi Liu, Yi-Hua Huang+8

1The University of Hong Kong · 2LIGHTSPEED · 3The Chinese University of Hong Kong+1

VLM Evaluation Game-Playing Agents LLM Agent Self-Improvement LLM Agent Skill Learning AI Agent Benchmarks

Open paper AskPDF arXiv

Vision-language model (VLM) agents are increasingly deployed in interactive game environments. Yet game benchmarks for VLM agents typically report a single first-attempt score per (agent, game) pair, focus on single-agent Solo play, and lack unified protocols for evaluating heterogeneous agent classes (commercial VLMs, open-weight VLMs, and specialized game policies) on the same footing. We address these gaps with OmniGameArena, a real-time benchmark of twelve newly built Unreal Engine 5 games spanning Solo (7), PvP (3), and Coop (2) with unified action interfaces, and the Improvement Dynamics Curve (IDC), an agentic-reflection harness in which a tool-using reflector LLM autonomously refines a bounded skill prompt across multiple rounds. Beyond cold-start leaderboard scores, IDC exposes two additional observables for each (agent, game) pair: how the score evolves across reflection rounds, and how the learned skill behaves on held-out task variants. We report these observables for twelve VLM agents on the cold-start leaderboard and four top agents under IDC.

Abstract

Vision-Language Models

Explore similar work

OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics

Do Vision Language Models Understand Human Engagement in Games?

Visually Grounded Self-Reflection for Vision-Language Models via Reinforcement Learning