Proxy2World: Learning to Generate Worlds From Lightweight Proxies without Seeing Them

Hongli Xu; Weilong Yan; Anbang Wang; Chunyu Zou; Siyu Hong; Jingwei Huang

cs.CVSep 28, 2026

Proxy2World: Learning to Generate Worlds From Lightweight Proxies without Seeing Them

Authors: Hongli Xu, Weilong Yan, Anbang Wang, Chunyu Zou, Siyu Hong, Jingwei Huang

Organizations: Tencent

Abstract

Lightweight scene proxies let creators control scene layout and motion while leaving room for imagination in appearance, lighting, and visual effects. However, a suitable proxy is not uniquely defined, making paired proxy-video data difficult to construct automatically at scale. We present Proxy2World, a controllable world model that learns these complementary capabilities from ordinary posed RGBD videos, without training on authored proxy-video pairs. The model jointly learns depth-conditioned RGB generation and joint RGBD generation through cross-modal flow matching. Learning both tasks enables proxy-camera hybrid denoising at inference to follow the proxy structure while producing natural, detailed visuals. We further introduce ProxyBench to evaluate this capability across a diverse set of scenes, camera trajectories, and subject motions. Experiments on ProxyBench show that Proxy2World achieves a better balance between structural adherence and visual quality than camera-controlled and geometry-conditioned methods, supported by quantitative metrics, VLM assessments, human evaluations and diverse qualitative results.

Figures & tables

Figure 1: Build the structure. Generate the world. Proxy2World transforms lightweight, interactive scene proxies into visually rich, structure-aligned worlds without paired proxy–video training data. Creators specify layouts and interactions through editable proxies (left), while the model generates detailed visuals (center) and variations in style, character, and lighting (right).

Figure 2: Overview of the Method. (A) Cross Modal Joint learning. We use a shared DiT to learn depth-conditioned RGB generation and joint RGBD generation, with camera-grid conditioning for both tasks. We train on ordinary posed RGBD videos without paired proxy–video supervision. (B) Proxy–camera hybrid denoising. We combine these learned capabilities during sampling: we first use proxy depth to establish scene structure, then jointly refine RGB and depth to enrich geometry and appearance. We retain camera-grid conditioning throughout. The refinement ratio $\tau$ controls the balance between proxy adherence and generative refinement.

Figure 3: (a) Diverse lightweight scene proxies. (b) Text descriptions and scripted interactions specify the intended scene behavior and subject motion. (c) Evaluation combines camera and spatial consistency metrics, VLM assessments of proxy adherence and refinement, and human preferences.

		Camera		Spatial Cons.	Proxy Evaluation
					Operator-based		VLM-based		Human
Method	Input	$\mathrm{nATE}_{t}\downarrow$	$\mathrm{nATE}_{r}\downarrow$	Reproj. $\downarrow$	Geo. Align $\uparrow$	Temp. Align $\uparrow$	Adh. WR $\uparrow$	Ref. WR $\uparrow$	Rank $\downarrow$
Camera-conditioned methods
Lyra 2.0 ( Shen et al., 2026 )	T	0.0418	0.0413	2.171	0.4965	0.7068	32.21	39.48	10
SCoPE / RAYPE ( Yin et al., 2026 )	T	0.1066	0.1229	4.338	0.4508	0.5780	29.51	47.06	9
FantasyWorld ( Dai et al., 2026 )	T	0.2966	0.3472	2.449	0.4711	0.6158	36.51	50.25	7

Table 1: Evaluation on ProxyBench. Input icons indicate text, reference image, camera trajectory, and proxy control; gray icons denote unused inputs. Trajectory errors are computed from VIPE-estimated poses, and reprojection error excludes dynamic objects. Adherence and refinement are evaluated through Gemini pairwise win rates (%). Human ranks reflect overall preference. Bold and underline indicate the best and second-best results within each conditioning group. All our variants share one checkpoint trained without paired proxy–RGB data.

Figure 4: Qualitative comparison on ProxyBench. Red boxes highlight the intended subject locations specified by the proxy, highlighting differences in subject placement and shape. Camera-conditioned methods can deviate from the prescribed subject placement, while depth-conditioned methods can retain coarse body shapes and produced distorted characters. Ours preserves the intended placement while generating more natural character shapes and detailed scene appearance.

Panel of Figure 5 — Figure 5: Effect of the refinement ratio and model components. (a) Varying $\tau$ with all other inputs and the random seed fixed illustrates the trade-off between proxy adherence and refinement. (b) Ablation of metric scale alignment and joint RGBD learning, evaluated through camera accuracy and human preference.

Explore similar work

ProxyUp: Training-Free Proxy-Conditioned Video Generation for Controllable Dynamics
Jul 4, 2026Zanwei Zhou, Jiazhong Cen, Jiemin Fang +7Camera-Conditioned Video Generation Video Generation
1Shanghai Jiao Tong University · 2Huawei Inc.
DreamX-World 1.0: A General-Purpose Interactive World Model
Jun 15, 2026DreamX Team, Yancheng Bai, Rui Chen +20Video World Models World Models
DreamX Team
World in World: Explore the World with World Models
Sep 11, 2026Chenxi Song, Yanming Yang, Chi ZhangVideo World Models Autoregressive Video Generation
Westlake AGI Lab · AGI Lab, Westlake University

Jul 4, 2026, cs.CVJ/K move · Enter open · S save

ProxyUp: Training-Free Proxy-Conditioned Video Generation for Controllable Dynamics

Zanwei Zhou, Jiazhong Cen, Jiemin Fang, Yumeng He+6

1Shanghai Jiao Tong University · 2Huawei Inc.

Camera-Conditioned Video Generation Video Generation Human Motion Generation Generative Models Video Latents Training-Free+1

Open paper AskPDF arXiv

Precise control over complex dynamics remains challenging for modern video generative models, as text prompts alone often cannot specify physically plausible, fine-grained motion and interactions. We introduce

\textit{proxy-conditioned video generation}

, where a coarse proxy video from physics-based simulation or real-world recording serves as a dynamics carrier to control foreground object motion. Given a proxy video and a text prompt, the goal is to synthesize a new video that preserves the proxy dynamics while generating novel content and plausible interactions aligned with the prompt. Since paired proxy-target videos are difficult to obtain, we propose

\textbf{ProxyUp}

, a training-free framework built on pretrained video generative models. ProxyUp first inverts the proxy video into an intermediate latent representation and applies

\textbf{region-wise latent noising}

, preserving motion-critical proxy latents while injecting noise into regions intended for text-driven regeneration. To mitigate the distribution mismatch and weak foreground-background coupling introduced by this heuristic latent composition, we further propose

\textbf{Stochastic Flow Relaxation (SFR)}

, which progressively relaxes the composed latent toward the model's learned distribution before ODE sampling. Experiments on both simulation and real-world proxies show that ProxyUp outperforms strong video editing and motion transfer baselines in dynamic fidelity and text alignment.

Abstract

Explore similar work

ProxyUp: Training-Free Proxy-Conditioned Video Generation for Controllable Dynamics

DreamX-World 1.0: A General-Purpose Interactive World Model

World in World: Explore the World with World Models