Generative World Renderer

Zheng-Hui Huang; Zhixiang Wang; Jiaming Tan; Ruihan Yu; Yidan Zhang; Bo Zheng; Yu-Lun Liu; Yung-Yu Chuang

cs.CVApr 2, 2026

Generative World Renderer

Authors: Zheng-Hui Huang, Zhixiang Wang, Jiaming Tan, Ruihan Yu, Yidan Zhang, Bo Zheng, Yu-Lun Liu, Yung-Yu Chuang,

Organizations: Alaya Studio, Shanda AI Research Tokyo · National Taiwan University · The University of Tokyo · National Yang Ming Chiao Tung University

Abstract

Scaling generative inverse and forward rendering to real-world scenarios is bottlenecked by the limited realism and temporal coherence of existing synthetic datasets. To bridge this persistent domain gap, we introduce a large-scale, dynamic dataset curated from visually complex AAA games. Using a novel dual-screen stitched capture method, we extracted 4M continuous frames (720p/30 FPS) of synchronized RGB and five G-buffer channels across diverse scenes, visual effects, and environments, including adverse weather and motion-blur variants. This dataset uniquely advances bidirectional rendering: enabling robust in-the-wild geometry and material decomposition, and facilitating high-fidelity G-buffer-guided video generation. Furthermore, to evaluate the real-world performance of inverse rendering without ground truth, we propose a novel VLM-based assessment protocol measuring semantic, spatial, and temporal consistency. Experiments demonstrate that inverse renderers fine-tuned on our data achieve superior cross-dataset generalization and controllable generation, while our VLM evaluation strongly correlates with human judgment. Combined with our toolkit, our forward renderer enables users to edit styles of AAA games from G-buffers using text prompts.

Figures & tables

Figure 1: We present a large-scale dataset curated from game engines to support scalable generative world rendering. The dataset provides high-resolution RGB videos with aligned G-buffers, covering continuous and dynamic scenes, long temporal trajectories, and diverse visual conditions.

Figure 2: Motivation. Existing approaches, such as DiffusionRenderer, are primarily trained on synthetic datasets, struggling to capture (a) complex reflection and illumination effects, (b) real scene elements (e.g., humans and cars), (c) fine-grained visual details, (d) dynamic motion, and (e) long-range temporal dependencies in long video sequences. By contrast, the proposed dataset provides high-fidelity, scene-level supervision with rich geometry, appearance, and temporal dynamics, enabling scalable generative video inverse rendering that generalizes more effectively to real-world scenarios.

Figure 3: Pipeline. Stage I : We curate video sequences containing RGB frames and five corresponding G-buffer channels from commercial game engines. Buffer interception is performed using ReShade, which allows us to capture intermediate rendering outputs at runtime. Because a single rendering pass exposes thousands of heterogeneous and largely irrelevant buffers, an automated filtering procedure is required to identify valid candidates. To disambiguate target G-buffers from irrelevant render targets, we utilize RenderDoc for offline inspection to define filtering rules based on metadata invariants. By manually validating the semantic accuracy of the remaining buffers, we iteratively optimize these rules into robust signatures that ensure consistent runtime identification. As a final verification step, we re-render the RGB frames from the collected G-buffers using a deferred shading pipeline and check for pixel-level consistency with the original RGB outputs. Stage II : We annotate meta-information for each sequence and filter out unsatisfactory frames based on quality criteria. Stage III : We synthesize motion blur to enhance temporal realism and better match real-world capture conditions.

	Depth						Normals		Albedo				Metallic		Roughness
	Abs Rel $\downarrow$	RMSE $\downarrow$	RMSE log $\downarrow$	$\delta<1.25$ $\uparrow$	$\delta<1.25^{2}$ $\uparrow$	$\delta<1.25^{3}$ $\uparrow$	AngularError $\downarrow$	[email protected] ∘ $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	si-PSNR $\uparrow$	si-LPIPS $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$
RGB $\leftrightarrow$ X	-	-	-	-	-	-	78.05 ∘	0.035	8.74	0.619	20.11	0.626	0.510	0.503	0.349	0.313
DNF	0.862	0.026	0.918	0.361	0.610	0.762	53.21 ∘	0.065	13.84	0.702	15.59	0.701	0.245	0.183	0.566	0.543
DR	1.118	0.030	0.723	0.267	0.496	0.684	45.01 ∘	0.110	17.53	0.648	19.90	0.646	0.230	0.134	0.281	0.237
Ours	0.697	0.023	0.430	0.609	0.761	0.852	42.57 ∘	0.150	16.44	0.628	21.44	0.635	0.104	0.024	0.266	0.218

Table 1: Quantitative evaluation of inverse rendering on black myth video dataset. DNF denotes DNF-intrinsic, and DR denotes DiffusionRenderer. Note that RGB

\leftrightarrow

X does not output depth.

	Depth					Albedo
	RMSE $\downarrow$	RMSE log $\downarrow$	$\delta<1.25\uparrow$	$\delta<1.25^{2}\uparrow$	$\delta<1.25^{3}\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	si-PSNR $\uparrow$	si-LPIPS $\downarrow$
RGB $\leftrightarrow$ X	-	-	-	-	-	8.69	0.613	16.69	0.595
DNF-Intrinsic	0.249	1.090	0.371	0.590	0.710	13.16	0.546	13.68	0.535
DiffusionRenderer	0.268	0.911	0.331	0.560	0.707	14.87	0.505	17.46	0.497
Ours	0.220	0.745	0.478	0.649	0.776	15.40	0.486	17.80	0.491

Table 2: Quantitative evaluation of inverse rendering on the Sintel dataset.

Table 6

	Depth				Albedo
Method	RMSE log $\downarrow$	$\delta\!<\!1.25\uparrow$	$\delta\!<\!1.25^{2}\uparrow$	$\delta\!<\!1.25^{3}\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$	si-PSNR $\uparrow$	si-LPIPS $\downarrow$
Ours	0.773	0.467	0.649	0.756	15.73	0.513	17.37	0.513
Ours (w/ motion blur)	0.745	0.478	0.649	0.776	15.40	0.486	17.80	0.491

Table 5: Ablation study on motion blur.

Figure 4: Qualitative comparison of inverse rendering on in-the-wild data. (Top to bottom: albedo, normal, depth, metallic, roughness). Our method significantly outperforms DiffusionRenderer in inverse rendering. It produces cleaner albedo with better delighting, artifact-free geometry, and robust material predictions that effectively resist complex outdoor illumination and atmospheric disruptions like smoke.

Figure 5: Qualitative comparison of inverse rendering on in-the-wild data.

Figure 6: Application of relighting on in-the-wild data.

Figure 7: Application of game editing. We can craft the game styles ( e.g . , lighting, weather, and visual effect) using the G-buffers from an AAA game as conditions.

Figure 8: Visual results of ablation study.

Figure A.1 : Prompt used for VLM evaluation of metallic prediction.

Inverse Rendering

Inverse rendering estimates scene geometry, materials, lighting, and other causes of images from observed visual or sensor data.

See all 69 papers

Aug 27, 2026

Magpie: Real-Time World Renderer for Interactive Games

Modern game development relies heavily on conventional graphics pipelines. High-quality visual content requires modeling, material authoring, animation, lighting, effects, and runtime optimization, making asset production expensive and extending the development cycle of game prototypes. Recently, video foundation models are beginning to change film and video production, but games differ from linear media, they require not only continuous and realistic imagery, but also stable and reproducible gameplay rules, object states, and interaction outcomes. We present Magpie, a real-time generative world-rendering system for interactive games. Magpie separates gameplay execution from visual generation. Designers define scenes and rules in a game engine. At runtime, the Game Engine resolves player actions and maintains world state, while an independent Render Server generates visual output from white-box frames produced by the engine. Magpie provides a system-level implementation path for applying generative models to real-time game rendering. It preserves gameplay designability and reproducibility, and reduces the dependence of early game prototypes on complete visual assets.

Xiaoyu Zhan, Xinyu Wang, Xiaohong Zhang +6

Mogo AI Ltd. · Nanjing University · University of Southampton

Real-Time Interactive Video Generation

Ask PDF

Jul 21, 2026

Generative World Renderer at the Speed of Play

Generative world renderer AlayaRenderer receives structured world states exported from physics engines and synthesizes RGB frames. Unlike models that generate frames from text/control-hints prompts, AlayaRenderer preserves scene structure without altering the underlying world dynamics. This demonstrates an alternative path toward interactive world modeling and user-controllable play. However, the original AlayaRenderer is too computationally expensive for real-time deployment. This technical report introduces AlayaRenderer-Flash, a real-time-oriented generative forward world renderer that pushes AlayaRenderer from 0.56 FPS to 31.54 FPS, reaching the speed of play. AlayaRenderer-Flash reformulates the original renderer as a few-step autoregressive streaming model and introduces lightweight distilled codecs for efficient latent encoding and frame reconstruction. It retains the teacher model's G-buffer and text-prompt interfaces while enabling continuous rendering over input streams of unbounded length. We evaluate AlayaRenderer-Flash on G-buffer streams across content preservation, temporal consistency, cross-window stability, prompt controllability, and runtime efficiency. Our results show that AlayaRenderer-Flash substantially reduces inference cost while preserving the core rendering capabilities of the teacher model. By integrating AlayaRenderer-Flash with a physics engine, we build a fully playable generative world running at 30 FPS.

Guixu Lin, Zheng-Hui Huang, Siqi Yang +3

1Alaya Lab · University of California, Merced

Real-Time Interactive Video Generation Streaming Video Generation Video Generation Autoregressive Video Generation Interactive World Models Video World Models

Ask PDF

Oct 22, 2026

AnyGameRenderer: Entity-Consistent Generative Rendering Across Games and an Entity-Level Benchmark

AI coding agents can now build small playable games in minutes, but the results usually look like programmer art: gray blockouts, colored geometric primitives, and flat semantic blobs. We present AnyGameRenderer, a video-to-video generative renderer that turns such footage into high-quality visuals with characters, motion detail, and effects. Throughout, it keeps the layout, camera, and every entity's position and timing intact. Our AnyGameRenderer generalizes across a wide range of genres that agents build, such as tower defense, auto-battlers, top-down survivors, turn-based battles, kart racing, and crowd runners. AnyGameRenderer takes plain video as its game-state input, with no complex representations such as engine G-buffers, per-object class labels, or low-poly proxies. Gray blockouts, colored primitives, semantic maps, and wireframes all work as input. It also preserves entity correspondence: group counts stay correct, and entities appear and disappear on the right frame. We implement AnyGameRenderer by LoRA fine-tuning MiniMax-H3 ref2va, a 33B reference-video-conditioned video generation model. Training uses ~10k paired clips from three sources: (i) multi-stream rendering data from 38 agent-made games; (ii) two-stream rendering data from Unreal Engine, with programmatically controlled crowds, count changes, and entity scales; and (iii) real gameplay recordings, with pairs reconstructed via 3D-box detection. To evaluate the entity correspondence, we propose an entity-level benchmark of 59 held-out cases with 753 entities, scored against ground-truth boxes. Compared with the base model, AnyGameRenderer better preserves entity correspondence: temporal match rises from 50.9% to 62.9%, missed count changes fall from 34.7% to 21.3%, and fewer entities are hallucinated or left unrendered. Project page: https://alaya-lab.github.io/AnyGameRenderer.

Ruicong Liu, Liangyang Ouyang, Fengbo Lan +2

Alaya Lab

Video Generation Video-Reference Video Generation Temporal Consistency in Video Generation Geometric Consistency in Video Generation

Ask PDF

Abstract

Inverse Rendering

Explore similar work

Magpie: Real-Time World Renderer for Interactive Games

Generative World Renderer at the Speed of Play

AnyGameRenderer: Entity-Consistent Generative Rendering Across Games and an Entity-Level Benchmark