Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training

Wenxuan Shen; Dongna Jin; Dongping Chen

cs.CVAug 25, 2026

Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training

Authors: Wenxuan Shen, Dongna Jin, Dongping Chen

Abstract

Video games provide a scalable source of training data for video world models, offering diverse environments, complex interactions, and abundant in-the-wild gameplay videos. However, raw gameplay footage entangles the game world with screen-space interfaces, introducing game-specific biases and irrelevant dynamics that hinder world-model training. To address this problem, we introduce GameUI-Taxonomy and G2WEngine, a full-stack framework that formalizes gameplay UI grounding and removal. G2WEngine automatically extracts reusable UI assets from real gameplay videos and synthesizes temporally coherent UI overlays on clean footage. Using this engine, we construct Game2World, comprising 96K synthetic paired videos with precise reconstruction targets and 1,079 in-the-wild clips from 303 games for realistic evaluation. Its asset library contains 5,132 verified UI elements across 21 taxonomy categories, collected from 1,010 representative gameplay frames. Based on Game2World, we propose GameCleaner, a mask-free gameplay UI removal model that combines multimodal semantic understanding with video editing capabilities. Unlike mask-based methods, GameCleaner directly identifies and removes diverse HUD elements while preserving the underlying scene content and temporal dynamics. In a controlled pilot, world models trained on UI-free gameplay improve overall VideoReward by 6.83% over those trained on UI-overlaid data. On UI-removal evaluation, GameCleaner achieves an average AAR of 95.36 on synthetic videos, outperforming the strongest temporal mask baseline by 57.3%, and obtains the best in-the-wild AAR of 80.05 with 99.8 background preservation. These results demonstrate the scalable potential of transforming Internet gameplay videos into high-quality world-model training data. Code, dataset, and model will be available at https://github.com/Dongping-Chen/Game2World.

Figures & tables

Figure 1: Overview of G2WEngine , a scalable data engine that transforms in-the-wild gameplay videos into world-model-ready data through HUD asset extraction, temporally coherent UI synthesis, and downstream support for HUD understanding, removal, and clean gameplay generation.

Table 2

Figure 2: Overview of GameUI-Taxonomy , which organizes gameplay HUD elements into categories based on their functional roles and rendering behaviors, including spatial information, player status, goals, messages, actions, and external overlays.

Figure 3: Overview of the Game2World-W dataset. The dataset provides temporally annotated in-the-wild gameplay videos, and evaluates UI removal through an MLLM-as-a-judge protocol with element-level bounding-box checklists.

Figure 4: Statistics of the Game2World UI asset taxonomy.

Figure 5: Model architecture of our GameCleaner .

Figure 6: Qualitative study for UI removal. GameCleaner w. reference image substantially reduces UI overlay while faithfully retaining the original video content. Frame and box level evaluation case study are provided in the Appendix.

Model	Mask	Ref P.T.	Overall	Synthetic					In-the-Wild
				AAR	UI	Clean	BG	Uncertain $\downarrow$	AAR	UI	Clean	BG	Uncertain $\downarrow$
VACE 1.3B ( Jiang et al., 2025 )	✔	✘	69.37	59.11	72.29	52.14	96.50	0.26	30.40	38.28	27.13	91.45	0.29
EffectErase ( Fu et al., 2026 )	✔	✘	59.68	56.17	94.56	48.75	84.45	0.07	23.55	92.00	20.42	74.55	0.09
VACE 14B ( Jiang et al., 2025 )	✔	✘	56.87	27.86	33.12	27.12	99.60	0.37	5.92	7.68	5.37	94.10	0.11
Aurora ( Yu et al., 2026 )	✘	✘	37.57	60.62	66.68	59.14	24.45	0.20	51.19	55.87	49.41	14.00	0.16
LTX-2 ( HaCohen et al., 2026 )	✘	✘	17.21	43.25	50.23	43.00	2.25	0.38	20.62	33.02	19.16	2.70	0.36

Table 3: Main comparison on the synthetic and in-the-wild benchmarks. Ref P.T. denotes reference-based post-training. Overall is computed as

(\mathrm{AAR}_{S}+\mathrm{BG}_{S}+\mathrm{AAR}_{W}+\mathrm{BG}_{W})/4

, where

S

and

W

denote the synthetic and wild benchmarks.

Model	Mask	Ref P.T.	Synthetic				In-the-Wild
			Frame-Macro	Box-Micro	Track-All	Track-Clean	Frame-Macro	Box-Micro	Track-All	Track-Clean
VACE 1.3B ( Jiang et al., 2025 )	✔	✘	72.26	75.29	74.58	56.09	37.33	31.69	33.88	20.29
EffectErase ( Fu et al., 2026 )	✔	✘	94.55	95.00	94.87	41.77	91.85	93.29	92.93	16.12
VACE 14B ( Jiang et al., 2025 )	✔	✘	33.10	32.41	31.74	26.37	6.57	3.79	5.25	4.35
Aurora ( Yu et al., 2026 )	✘	✘	66.66	67.21	65.87	56.80	56.09	52.67	53.44	56.80
LTX-2 ( HaCohen et al., 2026 )	✘	✘	50.28	61.74	63.13	54.65	32.01	30.61	31.34	15.76

Table 4: Fine-grained UI Removal results on the synthetic and in-the-wild benchmarks. Ref P.T. means reference based post train.

Figure 7: Qualitative study of the in-the-wild subset.

Figure 8: Qualitative study of the synthetic subset.

Figure 9: Ablation study of data scaling and reference drop ratio during training. We find out that larger and more diverse dataset show generalizable result on in-the-wild evaluation and have not converge yet.

Metric	ICC(A,1)	ICC(A,5)	Kripp. $\alpha$	Kendall $W$
AAR	0.713	0.926	0.712	0.772
UI	0.724	0.929	0.722	0.777
Clean	0.681	0.914	0.679	0.748
BG	0.834	0.962	0.833	0.566
Unc. $\downarrow$	0.582	0.874	0.580	0.705

Table 5: Inter-rater reliability on the in-the-wild benchmark.

Set	Resolution	Synthetic			In-the-Wild
		AAR	BG	Uncertain $\downarrow$	AAR	BG	Uncertain $\downarrow$
Set 1	$800{\times}448$	31.03	95.15	0.24	18.27	97.42	0.19
	$1280{\times}720$	28.12	97.47	0.46	16.43	98.35	0.24
	$1920{\times}1080$	32.02	95.76	0.44	16.20	95.46	0.09
Set 2	$800{\times}448$	86.19	97.47	0.03	33.47	99.23	0.09
	$1280{\times}720$	86.08	99.04	0.22	38.49	99.79	0.19

Table 6: Effect of inference resolution on the synthetic and in-the-wild benchmarks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Definition
map_radar	Geographic mini-map or symbolic proximity/scan radar.
navigation_compass	Compass, heading scale, bearing strip, or directional readout.
player_status	Health, armor, stamina, resources, level, portrait, or player-state cluster.
teammate_roster	Party/team member list or teammate status cluster.
weapon_ammo	Equipped weapon, ammunition, reload, or firing-mode widget.
inventory	Hotbar, quick slots, backpack, item grid, or inventory panel.

Appendix

Table 7: Complete taxonomy definitions used for UI annotation and synthesis.

Role	Taxonomy	Aspect	$w_{\max}$	$h_{\max}$	Priority	$p_{\mathrm{motion}}$	Motion frac. $(x,y)$
minimap	map_radar	0.65–1.65	0.185	0.290	80	0.25	(0.0012, 0.0008)
compass_bar	navigation_compass	1.80–18.00	0.390	0.065	70	0.42	(0.0018, 0.0008)
health	player_status	1.35–7.50	0.245	0.110	85	0.20	(0.0010, 0.0007)
stamina	player_status	1.35–8.50	0.205	0.060	84	0.24	(0.0010, 0.0008)
weapon_bar	inventory	1.45–8.50	0.360	0.175	75	0.28	(0.0012, 0.0007)
crosshair	crosshair	0.40–2.50	0.055	0.085	100	0.55	(0.0018, 0.0018)

Appendix

Table 8: Spatial and motion priors for persistent UI roles.

Style	Formation	Opacity scale	UI scale	Transient density	Role-specific scales
Adventure	Adventure	0.94	1.00	0.95	minimap 1.06, health 1.06, weapon bar 1.00
Tactical	Tactical	1.00	0.96	1.20	minimap 1.02, health 0.92, weapon bar 1.03, crosshair 1.14
RPG	RPG	0.92	1.04	1.10	minimap 1.08, health 1.08, weapon bar 1.00
Survival	Adventure	0.88	0.92	0.75	minimap 0.95, health 0.98, weapon bar 0.90, crosshair 0.92
Arena	Tactical	1.02	1.02	1.35	minimap 0.98, health 0.86, weapon bar 1.08, crosshair 1.22

Appendix

Table 9: UI style presets in the released renderer.

Taxonomy	Probability	Maximum
map_radar	0.68	1
player_status	0.72	1
inventory	0.58	1
navigation_compass	0.48	1
crosshair	0.64	1
weapon_ammo	0.54	1

Appendix

Table 10: Persistent element spawn policy.

Event category	Weight	Max.
world_context	0.65	3
objective_status	0.45	2
interaction_prompt	0.55	2
screen_notice	0.60	3
ability_status	0.35	2
teammate_roster	0.20	2

Appendix

Table 11: Popup event-category policy. Under the five-second profile, these values serve as relative weights after the total popup count is drawn.

Event category	Asset taxonomy mixture
objective_status	objective_status : 0.75; subtitle : 0.25
interaction_prompt	interaction_prompt : 0.75; menu_controls : 0.25
screen_notice	screen_notice : 0.65; performance_debug : 0.15; watermark : 0.10; other_hud : 0.10
ability_status	ability_status : 0.75; vehicle_status : 0.25
teammate_roster	teammate_roster : 0.60; chat_comms : 0.25; stream_overlay : 0.15
match_status	match_status : 1.00

Appendix

Table 12: Default mapping from event categories to source-asset taxonomies.

Category	$w_{\max}$	$h_{\max}$	Duration (s)	Fade (s)	Opacity	Anchors
objective_status	0.340	0.105	2.3–5.8	0.35	0.88	lower-center, upper-center, center-right
interaction_prompt	0.185	0.120	1.4–3.8	0.22	0.86	center-right, center-left, near-center, upper-center
screen_notice	0.300	0.130	2.0–4.5	0.28	0.91	top-right stack, top-left stack
ability_status	0.145	0.100	4.0–9.5	0.55	0.90	lower-left stack, upper-right stack, above weapon
teammate_roster	0.230	0.150	3.2–7.0	0.40	0.88	left-mid, right-mid, top-left stack
match_status	0.155	0.090	1.0–2.6	0.20	0.88	near-center, center-right, upper-center

Appendix

Table 13: Transient category profiles. The implementation also stores category-specific aspect-ratio bounds and drift fractions.

Animation	Weight	Implementation
fade	0.22	Smooth, independently sampled entrance and exit fades.
instant	0.13	Immediate appearance and hard disappearance.
slide	0.16	Horizontal entrance with a smaller exit displacement and opacity ramp.
wipe_trail	0.18	Irregular left-to-right scanline reveal/removal with deterministic high-frequency modulation.
pop	0.09	Ease-out-back scale overshoot, settle, and shrink.
border_arc	0.22	Enter from one frame border, traverse a noisy parabolic arc, and exit through a different border.

Appendix

Table 14: Popup animation weights and behaviors.

Field	Purpose
Case identifier	Uniquely associates a model output with its source example.
Dataset split	Distinguishes synthetic and in-the-wild evaluation cases.
Source video	UI-containing video supplied to the removal model.
Clean reference	Available only for synthetic cases and retained for controlled analysis.
Annotation	Frame-level synthetic boxes or human-annotated in-the-wild trajectories.
Video metadata	Width, height, frame rate, frame count, and annotation format.

Appendix

Table 15: Information required to define a benchmark case.

Run	Exact generation prompt
General Video Edit Models	This is a gameplay video. Please remove the UI and UI elements from it without affecting the gameplay footage.
GameCleaner	Remove the gameplay UI from the video.

Appendix

Table 16: Exact video-editing prompts recorded in the frozen evaluation protocol.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 264 papers

Oct 27, 2026

GameVerse: A Minute-Scale Gameplay Dataset for Long-Horizon Interactive World Modeling

Recent advances in interactive world models have made it possible to generate controllable, dynamic environments from action inputs. Yet current models struggle to maintain coherence beyond roughly one minute of rollout, a limitation rooted not in model capacity but in the scarcity of long-horizon training data. Existing gameplay video datasets typically consist of clips under ten seconds, with captions that describe whole clips in coarse and undifferentiated text, which is insufficient for controllable long-horizon generation. We address this with GameVerse, a minute-scale gameplay video dataset purpose-built for long-horizon interactive world modeling. GameVerse contains 69.7K clips spanning 1,655 hours and 349.6 million frames across diverse first- and third-person gameplay, including 1,358 hours of continuous two-minute sequences, an order of magnitude longer than prior datasets. Clips pass through a unified pipeline of scene-consistent segmentation, quality filtering, camera pose estimation, and event-based caption annotation. Each clip is paired with a temporally stable camera trajectory that remains coherent across minutes, together with captions that separate static scene context from dynamic foreground events and align to event boundaries rather than fixed time windows. Comprehensive analysis shows that GameVerse surpasses existing public datasets in scale, diversity, and annotation quality. Experiments further demonstrate that training on GameVerse substantially extends the temporal horizon of interactive world models.

Kang He, Wenshuo Peng, Chuanhao Li +7

Alaya Lab · Shanghai Innovation Institute · School of Cyber Science and Engineering, Wuhan University +1

Multimodal Dataset Interactive World Models Video World Models

Ask PDF

Aug 18, 2025

Matrix-game 2.0: An open-source, real-time, and streaming interactive world model

Recent advances in interactive video generations have demonstrated diffusion model's potential as world models by capturing complex physical dynamics and interactive behaviors. However, existing interactive world models depend on bidirectional attention and lengthy inference steps, severely limiting real-time performance. Consequently, they are hard to simulate real-world dynamics, where outcomes must update instantaneously based on historical context and current actions. To address this, we present Matrix-Game 2.0, an interactive world model generates long videos on-the-fly via few-step auto-regressive diffusion. Our framework consists of three key components: (1) A scalable data production pipeline for Unreal Engine and GTA5 environments to effectively produce massive amounts (about 1200 hours) of video data with diverse interaction annotations; (2) An action injection module that enables frame-level mouse and keyboard inputs as interactive conditions; (3) A few-step distillation based on the casual architecture for real-time and streaming video generation. Matrix Game 2.0 can generate high-quality minute-level videos across diverse scenes at an ultra-fast speed of 25 FPS. We open-source our model weights and codebase to advance research in interactive world modeling.

Xianglong He, Chunli Peng, Zexiang Liu +16

Skywork AI

Action-Conditioned Video Generation Real-Time Interactive Video Generation Action-Conditioned World Models Streaming Video Generation Autoregressive Video Generation Interactive World Models

Ask PDF

Oct 20, 2026

Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds

Moving from controllable videos to genuinely playable worlds requires persistent world state, executable game rules, and low-latency interaction. We outline a roadmap for the next generation of Matrix-Game that separates game execution from visual generation through an executable game harness, and introduce Matrix-Game 4.0 as the first system built on this design. A lightweight game engine maintains world state and executes interaction rules, a multimodal coding agent translates player ideas into game code and dynamic scene captions, and a renderer based on autoregressive video model transforms engine-rendered low-poly proxies into detailed visuals, conditioned on dynamic captions and memory references. This framework provides continuous guidance for visual generation and enables natural-language creation and modification of scenes and gameplay. Our renderer is trained on aligned low-poly and high-fidelity video pairs and distilled into a causal generator that generates in four-frame chunks with one or two denoising steps. Together with targeted inference optimizations, Matrix-Game 4.0 enables 480p, 60 FPS gameplay on a single NVIDIA RTX5090 with a key-press-to-display latency of 112ms. Single-step generation achieves 83FPS steady-state throughput with 18.2GiB of peak GPU memory usage. With fixed prompts, steady-state throughput also reaches 67FPS on an RTX5080 and 58FPS on an RTX5070Ti. We demonstrate customizable shooter and driving games with engine-enforced mechanics, and release code, model weights, and low-poly game assets to support research and creation of playable game worlds.

Jihai Zhang, Runjia Qian, Kaiqin Kong +15

SkyworkAI · The Chinese University of Hong Kong · UC San Diego +4

Real-Time Interactive Video Generation Autoregressive Video Generation Controllable Video Generation Causal Video Generation

Ask PDF

Abstract

Video World Models

Explore similar work

GameVerse: A Minute-Scale Gameplay Dataset for Long-Horizon Interactive World Modeling

Matrix-game 2.0: An open-source, real-time, and streaming interactive world model

Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds