cs.CVOct 22, 2026

AnyGameRenderer: Entity-Consistent Generative Rendering Across Games and an Entity-Level Benchmark

Authors: Ruicong Liu, Liangyang Ouyang, Fengbo Lan, Zhixiang Wang, Kaipeng Zhang

Organizations: Alaya Lab

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

AI coding agents can now build small playable games in minutes, but the results usually look like programmer art: gray blockouts, colored geometric primitives, and flat semantic blobs. We present AnyGameRenderer, a video-to-video generative renderer that turns such footage into high-quality visuals with characters, motion detail, and effects. Throughout, it keeps the layout, camera, and every entity's position and timing intact. Our AnyGameRenderer generalizes across a wide range of genres that agents build, such as tower defense, auto-battlers, top-down survivors, turn-based battles, kart racing, and crowd runners. AnyGameRenderer takes plain video as its game-state input, with no complex representations such as engine G-buffers, per-object class labels, or low-poly proxies. Gray blockouts, colored primitives, semantic maps, and wireframes all work as input. It also preserves entity correspondence: group counts stay correct, and entities appear and disappear on the right frame. We implement AnyGameRenderer by LoRA fine-tuning MiniMax-H3 ref2va, a 33B reference-video-conditioned video generation model. Training uses ~10k paired clips from three sources: (i) multi-stream rendering data from 38 agent-made games; (ii) two-stream rendering data from Unreal Engine, with programmatically controlled crowds, count changes, and entity scales; and (iii) real gameplay recordings, with pairs reconstructed via 3D-box detection. To evaluate the entity correspondence, we propose an entity-level benchmark of 59 held-out cases with 753 entities, scored against ground-truth boxes. Compared with the base model, AnyGameRenderer better preserves entity correspondence: temporal match rises from 50.9% to 62.9%, missed count changes fall from 34.7% to 21.3%, and fewer entities are hallucinated or left unrendered. Project page: https://alaya-lab.github.io/AnyGameRenderer.

Figures & tables

Figure 1: AnyGameRenderer turns crude gameplay into polished game visuals. Top: one checkpoint handles diverse genres and input types, and renders one input in many styles. Insets show the crude input. Bottom: an agent writes the game and the engine outputs a blockout, which AnyGameRenderer renders into the final visuals. This allows style switching and skips modeling, texturing and animation.
Figure 2: Overview of AnyGameRenderer. (a) Paired data from three sources. Agent-made games use multi-stream rendering, which pairs each game render with several aligned crude streams such as blockouts and semantic maps. Unreal Engine scenes use two-stream rendering, which pairs a full render with blockouts. Real gameplay recordings are converted into blockouts by detecting 3D bounding boxes. (b) One training sample: the crude clip is the input video <Video 1> , the target’s first frame serves as an optional anchor, and a six-segment caption defines which real entity each crude shape represents and what must be preserved. (c) Text, <Video 1> , anchor, and noisy target tokens form one sequence for the frozen MiniMax-H3 DiT. Only the LoRA adapters on self-attention and feed-forward layers are trained, and the model renders the input gameplay in the prompted style.
Figure 3: Three failures of the zero-shot base model. Red dashed boxes mark the errors. Left: input primitives are left unrendered; the cyan cylinders stay in the output instead of becoming characters. Middle: the input has only two blocks, but the model adds an extra creature that does not exist in the input. Right: the input is a top-down view, but the model renders a frontal group shot.
SourceCrude inputClipsShare
Agent-made games (38)primitive / coarse / semantic passes3,22131%
Unreal Enginegray blockout with colored character blocks3,91038%
Real recordingsdetected 3D boxes as solid / wireframe / cylinder / ellipsoid proxies3,19931%
Total 10,330≈\approx 14.8 h
Table 1: Training data mixture. All clips are 124 frames at 24 fps and 1344×7681344{\times}768 , with captions generated from each source’s metadata.
  TemporalPositionUnrenderedHalluc.On-timeNeverMissed count
SplitModelmatch ↑\uparrowmatch ↑\uparrowrate ↓\downarrowrate ↓\downarrowappear. ↑\uparrowappears ↓\downarrowchanges ↓\downarrow
All (59)Zero-shot50.957.14.152.155.926.034.7
Ours62.961.92.946.067.620.221.3
GT floor80.979.30.134.193.60.70.0
Multi-stream rendering (34)Zero-shot47.656.44.555.450.928.440.7
Ours60.760.63.251.264.023.523.7
Table 2: Entity-sync results (%) on the held-out benchmark : four main metrics (left) and three secondary statistics (right). GT floor (gray) runs the ground-truth RGB video through the same detection and matching pipeline.
Figure 4: Zero-shot MiniMax-H3 versus AnyGameRenderer on held-out cases , with the same seed, steps, and caption. The zero-shot model leaves crude shapes unrendered (rows 1–4), drops the input viewpoint for out-of-domain styles (rows 5–6: a top-down blockout becomes a frontal anime scene or a first-person voxel scene), and adds caption-named characters although the input has only two blocks (row 7). AnyGameRenderer fixes all three. For rows 5–6, the “GT” column is the original in-domain render and serves only as a layout reference.
Figure 5: Entity-sync metrics by genre. Zero-shot MiniMax-H3 (orange) versus AnyGameRenderer (blue) on the held-out benchmark, grouped into first-person shooter, top-down, third-person, crowd, and vehicle games.
More in this topic

Video Generation

Video generation studies models that create and edit moving images from text, images, audio, and structured controls.

See all 363 papers

Explore similar work

CardsList
Aug 27, 2026

Magpie: Real-Time World Renderer for Interactive Games

Modern game development relies heavily on conventional graphics pipelines. High-quality visual content requires modeling, material authoring, animation, lighting, effects, and runtime optimization, making asset production expensive and extending the development cycle of game prototypes. Recently, video foundation models are beginning to change film and video production, but games differ from linear media, they require not only continuous and realistic imagery, but also stable and reproducible gameplay rules, object states, and interaction outcomes. We present Magpie, a real-time generative world-rendering system for interactive games. Magpie separates gameplay execution from visual generation. Designers define scenes and rules in a game engine. At runtime, the Game Engine resolves player actions and maintains world state, while an independent Render Server generates visual output from white-box frames produced by the engine. Magpie provides a system-level implementation path for applying generative models to real-time game rendering. It preserves gameplay designability and reproducibility, and reduces the dependence of early game prototypes on complete visual assets.

Xiaoyu Zhan, Xinyu Wang, Xiaohong Zhang +6

Mogo AI Ltd. · Nanjing University · University of Southampton

Real-Time Interactive Video Generation
AskPDF
Apr 2, 2026

Generative World Renderer

Scaling generative inverse and forward rendering to real-world scenarios is bottlenecked by the limited realism and temporal coherence of existing synthetic datasets. To bridge this persistent domain gap, we introduce a large-scale, dynamic dataset curated from visually complex AAA games. Using a novel dual-screen stitched capture method, we extracted 4M continuous frames (720p/30 FPS) of synchronized RGB and five G-buffer channels across diverse scenes, visual effects, and environments, including adverse weather and motion-blur variants. This dataset uniquely advances bidirectional rendering: enabling robust in-the-wild geometry and material decomposition, and facilitating high-fidelity G-buffer-guided video generation. Furthermore, to evaluate the real-world performance of inverse rendering without ground truth, we propose a novel VLM-based assessment protocol measuring semantic, spatial, and temporal consistency. Experiments demonstrate that inverse renderers fine-tuned on our data achieve superior cross-dataset generalization and controllable generation, while our VLM evaluation strongly correlates with human judgment. Combined with our toolkit, our forward renderer enables users to edit styles of AAA games from G-buffers using text prompts.

Zheng-Hui Huang, Zhixiang Wang, Jiaming Tan +6

Alaya Studio, Shanda AI Research Tokyo · National Taiwan University · The University of Tokyo +1

Inverse RenderingControllable Video GenerationVideo Generation Evaluation
AskPDF
Oct 20, 2026

Matrix-Game 4.0: Harnessing Video Models for Real-Time Playable Game Worlds

Moving from controllable videos to genuinely playable worlds requires persistent world state, executable game rules, and low-latency interaction. We outline a roadmap for the next generation of Matrix-Game that separates game execution from visual generation through an executable game harness, and introduce Matrix-Game 4.0 as the first system built on this design. A lightweight game engine maintains world state and executes interaction rules, a multimodal coding agent translates player ideas into game code and dynamic scene captions, and a renderer based on autoregressive video model transforms engine-rendered low-poly proxies into detailed visuals, conditioned on dynamic captions and memory references. This framework provides continuous guidance for visual generation and enables natural-language creation and modification of scenes and gameplay. Our renderer is trained on aligned low-poly and high-fidelity video pairs and distilled into a causal generator that generates in four-frame chunks with one or two denoising steps. Together with targeted inference optimizations, Matrix-Game 4.0 enables 480p, 60 FPS gameplay on a single NVIDIA RTX5090 with a key-press-to-display latency of 112ms. Single-step generation achieves 83FPS steady-state throughput with 18.2GiB of peak GPU memory usage. With fixed prompts, steady-state throughput also reaches 67FPS on an RTX5080 and 58FPS on an RTX5070Ti. We demonstrate customizable shooter and driving games with engine-enforced mechanics, and release code, model weights, and low-poly game assets to support research and creation of playable game worlds.

Jihai Zhang, Runjia Qian, Kaiqin Kong +15

SkyworkAI · The Chinese University of Hong Kong · UC San Diego +4

Real-Time Interactive Video GenerationAutoregressive Video GenerationControllable Video GenerationCausal Video Generation
AskPDF