WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation

Jisu Nam; Yicong Hong; Chun-Hao Paul Huang; Feng Liu; JoungBin Lee; Jiyoung Kim; Siyoon Jin; Yunsung Lee

cs.CVMar 17, 2026

WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation

Authors: Jisu Nam, Yicong Hong, Chun-Hao Paul Huang, Feng Liu, JoungBin Lee, Jiyoung Kim, Siyoon Jin, Yunsung Lee,

Organizations: KAIST · Adobe Research · MAUM AI

Abstract

Recent advances in video diffusion transformers have enabled interactive gaming world models that allow users to explore generated environments over extended horizons. However, existing approaches struggle with precise action control and long-horizon 3D consistency. Most prior works treat user actions as abstract conditioning signals, overlooking the fundamental geometric coupling between actions and the 3D world, whereby actions induce relative camera motions that accumulate into a global camera pose within a 3D world. In this paper, we establish camera pose as a unifying geometric representation to jointly ground immediate action control and long-term 3D consistency. First, we define a physics-based continuous action space and represent user inputs in the Lie algebra to derive precise 6-DoF camera poses, which are injected into the generative model via a camera embedder to ensure accurate action alignment. Second, we use global camera poses as spatial indices to retrieve relevant past observations, enabling geometrically consistent revisiting of locations during long-horizon navigation. To support this research, we introduce a large-scale dataset comprising 3,000 minutes of authentic human gameplay annotated with camera trajectories and textual descriptions. Extensive experiments show that our approach substantially outperforms state-of-the-art interactive gaming world models in action controllability, long-horizon visual quality, and 3D spatial consistency.

Figures & tables

Figure 1: Teaser (Best viewed in color and zoomed in): WorldCam is an interactive 3D gaming model that enables precise action control under challenging keyboard and mouse inputs (top), supports long-horizon interactions (middle), and preserves consistent 3D geometry across viewpoints (bottom). Time (seconds at 20 FPS) is visualized in the top-left of each frame, while keyboard and mouse inputs are shown in the bottom-left and bottom-right, respectively. The red box highlights consistent 3D geometry in revisited views.

Figure 2: Overall architecture. WorldCam converts user actions into camera poses in Lie algebra and conditions a progressive autoregressive video transformer on these camera poses for precise action control. Retrieved long-term memory latents and camera poses from the memory pool enforce 3D consistency of the generated world, while short-term memory with an attention sink stabilizes long-horizon generation.

	The Matrix	Matrix-game	Matrix-game 2.0	Genie	Yan	GameCraft	CameraCtrl	MotionCtrl	WorldCam
Action control	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\checkmark$
3D consistency	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
Long-horizon inference	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\checkmark$

Table 1: Comparison with existing interactive gaming world models and camera-controlled models.

Figure 3: Dataset samples and statistics: (a) Example gameplay frames annotated with camera trajectories, and text captions. (b) Distribution of video durations. (c) Distribution of linear velocities $(v_{x},v_{y},v_{z})$ . (d) Distribution of angular velocities $(\omega_{x},\omega_{y},\omega_{z})$ . The dataset captures diverse and authentic human gameplay behaviors for training interactive gaming world models.

Figure 4: Qualitative comparison with recent interactive gaming world models: Compared to prior works, WorldCam faithfully follows user actions and maintains coherent 3D scene structure with high visual fidelity over long horizons.

	Action Controllability			Visual Quality
Method	RPE ${}_{\text{trans}}\downarrow$	RPE ${}_{\text{rot}}(^{\circ})\downarrow$	RPE ${}_{\text{camera}}\downarrow$	Avg. $\uparrow$	Aesth. $\uparrow$	Subj. Cons. $\uparrow$	Bg. Cons. $\uparrow$	Img. $\uparrow$	Temp. $\uparrow$	Motion. $\uparrow$
Yume ( Mao et al., 2025 )	0.111	2.222	0.137	0.774	0.476	0.741	0.892	0.600	0.955	0.986
Matrix-Game 2.0 ( He et al., 2025 )	0.098	1.656	0.119	0.766	0.457	0.741	0.843	0.633	0.937	0.981
GameCraft ( Li et al., 2025a )	0.086	1.146	0.100	0.781	0.464	0.804	0.850	0.626	0.958	0.986
WorldCam	0.080	0.696	0.086	0.844	0.508	0.896	0.959	0.752	0.964	0.984

Table 2: Quantitative comparison with recent interactive gaming world models in terms of action controllability and visual quality. All methods are evaluated under the same action trajectories with 200-frame video generation.

Method	PSNR $\uparrow$	LPIPS $\downarrow$	MEt3R $\downarrow$	DINO Sim. $\uparrow$	Sharpness $\uparrow$
Real Videos	-	-	-	-	577
Yume ( Mao et al., 2025 )	16.03	0.5629	0.0905	0.4545	95
Matrix-Game 2.0 ( He et al., 2025 )	13.66	0.4997	0.0662	0.6153	179
GameCraft ( Li et al., 2025a )	14.27	0.5749	0.0489	0.5960	201
WorldCam	16.69	0.3277	0.0342	0.8884	656

Table 3: Quantitative results for long-horizon 3D consistency.

Method	RPE ${}_{\text{trans}}\downarrow$	RPE ${}_{\text{rot}}(^{\circ})\downarrow$	RPE ${}_{\text{camera}}\downarrow$
CameraCtrl ( He et al., 2024 )	0.071	0.943	0.083
MotionCtrl ( Wang et al., 2024b )	0.080	1.271	0.102
WorldCam	0.026	0.386	0.030

Table 4: Comparison with camera-controlled models under 16-frame generation.

Method	Action Controllability $\uparrow$	Visual Quality $\uparrow$	3D Consistency $\uparrow$
Yume ( Mao et al., 2025 )	2.47	2.83	1.44
Matrix-2.0 ( He et al., 2025 )	3.78	3.42	2.75
GameCraft ( Li et al., 2025a )	2.55	3.34	3.36
WorldCam	4.31	4.44	4.36

Table 5: Human evaluation results. We report average user ratings on a scale of 1 (worst) to 5 (best).

Table 10

Method	Avg. $\uparrow$	Aesth. $\uparrow$	Subj. Cons. $\uparrow$	Bg. Cons. $\uparrow$	Img. $\uparrow$	Temp. $\uparrow$	Motion. $\uparrow$
w/o attention sink	0.840	0.511	0.876	0.955	0.751	0.961	0.984
with attention sink	0.841	0.502	0.883	0.956	0.753	0.964	0.984

Table 8: Ablation on the effect of attention sinks for visual quality.

Method	RPE ${}_{\text{trans}}\downarrow$	RPE ${}_{\text{rot}}(^{\circ})\downarrow$	RPE ${}_{\text{camera}}\downarrow$
Yume ( Mao et al., 2025 )	0.111	2.222	0.137
Matrix-Game 2.0 ( He et al., 2025 )	0.098	1.656	0.119
GameCraft ( Li et al., 2025a )	0.086	1.146	0.100
WorldCam (Linear)	0.093	0.962	0.102
WorldCam (Lie)	0.080	0.696	0.086

Table 9: Ablation study on action-to-camera mapping.

Method	PSNR $\uparrow$	LPIPS $\downarrow$	MEt3R $\downarrow$
Random	15.76	0.3645	0.041
Temporal	15.18	0.3867	0.040
Ours	16.42	0.3496	0.038

Table 10: Ablation study on long-term memory retrieval.

Figure 5: Qualitative Results (Best viewed in color and zoomed in): (a) WorldCam enables fine-grained action control coupled with simultaneous keyboard and mouse inputs. (b) WorldCam generates long-horizon videos exceeding 10 seconds (20 FPS) without error drift. Time (seconds at 20 FPS) is visualized in the top-left of each frame, while keyboard and mouse inputs are shown in the bottom-left and bottom-right, respectively.

Figure 6: Qualitative Results (Best viewed in color and zoomed in): Our model preserves the underlying 3D scene consistently over long-horizon beyond the denoising window. Time (seconds at 20 FPS) is visualized in the top-left of each frame, while keyboard and mouse inputs are shown in the bottom-left and bottom-right, respectively. The red box highlights consistent 3D geometry in revisited views.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Latent-wise AR	Steps	Frames/Chunk	Time/Chunk (s)	Time/Step (s)
Yume ( Mao et al., 2025 )	$\times$	4	32	2.91	0.73
GameCraft ( Li et al., 2025a )	$\times$	8	33	109.4	13.68
Matrix-Game 2.0 ( He et al., 2025 )	$\checkmark$	3	12	3.09	1.03
WorldCam	$\checkmark$	8	4	4.17	0.52

Appendix

Table 12: Inference time comparison. All methods are evaluated at a spatial resolution of

480\times 832

on a single H100 GPU.

Interactive World Models

Interactive world models predict and generate how environments change in response to actions, supporting simulation, planning, and control.

See all 114 papers

ActionRoPE: Action as Coordinates for Spatially Consistent Control in Planar-Camera Game World Models
Oct 19, 2026Zhiyang Deng, Yuzhe Wu, Zijun Lin +1Action-Conditioned Video Generation World Models
Tencent · National University of Singapore · Nanyang Technological University
DreamX-World 1.0: A General-Purpose Interactive World Model
Jun 15, 2026DreamX Team, Yancheng Bai, Rui Chen +20Camera-Controlled Video Generation Real-Time Interactive Video Generation
DreamX Team
Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory
Aug 30, 2026Runjia Qian, Zile Wang, Jihai Zhang +14Camera-Controlled Video Generation Real-Time Interactive Video Generation
Riemann Dynamics

Oct 19, 2026J/K move · Enter open · S save

ActionRoPE: Action as Coordinates for Spatially Consistent Control in Planar-Camera Game World Models

Zhiyang Deng, Yuzhe Wu, Zijun Lin, Yeying Jin

Tencent · National University of Singapore · Nanyang Technological University

Action-Conditioned Video Generation World Models Video World Models Rotary Positional Embeddings

Open paper AskPDF arXiv

Game world models have made rapid progress toward real-time interactive generation. Yet action control is still predominantly implemented as feature conditioning, leaving the model to learn from training data how each action should move the visual world. This formulation is poorly matched to a broad class of planar-camera worlds, such as top-down and isometric games. In these worlds, player motion induces a geometrically well-defined planar camera displacement. We observe that the key to spatially consistent control in such worlds is to represent the effect of each action directly in world coordinates. To this end, we introduce ActionRoPE, a parameter-free action interface that converts action-driven camera displacement into world-coordinate rotary positions. As a result, tokens corresponding to the same world location retain the same positional address across time. These world coordinates further expose a spatial asymmetry in generation: regions covered by the initial observation are geometrically known, while newly revealed regions remain to be synthesized. We therefore propose Coordinate-Conditioned Noise Scheduling (CNS), which assigns token-wise denoising schedules according to this known-new structure. It allows observed regions to stabilize earlier, while new content is generated with reference to them. Compared with four representative action-conditioning mechanisms, our approach follows the commanded motion more faithfully and reduces the revisit drift by 66% over the strongest baseline. The same interface transfers across video backbones from 1.3B to 33B parameters and to out-of-distribution scenes. Project Page: https://teawhite.cn/ActionRoPE/.

Abstract

Interactive World Models

Explore similar work

ActionRoPE: Action as Coordinates for Spatially Consistent Control in Planar-Camera Game World Models

DreamX-World 1.0: A General-Purpose Interactive World Model

Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory