Automatically generating photorealistic and self-consistent appearances for untextured 3D models is a critical challenge in digital content creation. The advancement of large-scale video generation models offers a natural approach: directly synthesizing 360-degree turntable videos (TTVs), which can serve not only as high-quality dynamic previews but also as an intermediate representation to drive texture synthesis and neural rendering. However, existing general-purpose video diffusion models struggle to maintain strict geometric consistency and appearance stability across the full range of views, making their outputs ill-suited for high-quality 3D reconstruction. To this end, we introduce TAPESTRY, a framework for generating high-fidelity TTVs conditioned on explicit 3D geometry. We reframe the 3D appearance generation task as a geometry-conditioned video diffusion problem: given a 3D mesh, we first render and encode multi-modal geometric features to constrain the video generation process with pixel-level precision, thereby enabling the creation of high-quality and consistent TTVs. Building upon this, we also design a method for downstream reconstruction tasks from the TTV input, featuring a multi-stage pipeline with 3D-Aware Inpainting. By rotating the model and performing a context-aware secondary generation, this pipeline effectively completes self-occluded regions to achieve full surface coverage. The videos generated by TAPESTRY are not only high-quality dynamic previews but also serve as a reliable, 3D-aware intermediate representation that can be seamlessly back-projected into UV textures or used to supervise neural rendering methods like 3DGS. This enables the automated creation of production-ready, complete 3D assets from untextured meshes. Experimental results demonstrate that our method outperforms existing approaches in both video consistency and final reconstruction quality.
Figures & tables
Figure 1: We introduce TAPESTRY for high-fidelity 3D appearance generation by synthesizing a reconstructable Turntable Video through strong geometric conditioning. This highly consistent video then serves as a robust data source to create a final, high-quality asset.
Figure 2: An overview of the TAPESTRY architecture. (a) Geometry-guided video generation. Our method generates a 3D consistent Turntable Video by injecting multi-modal geometric conditions and reference context into a DiT-based video diffusion model. (b) Our progressive texturing pipeline. We iteratively generate TTVs from new, optimized viewpoints and fuse their projections via Texture Baking. Each pass is conditioned on previously generated textures to ensure global consistency, continuing until full coverage is achieved.
Figure 3: Qualitative results of TAPESTRY. Our method generates a consistent Turntable Video, from which a complete and high-fidelity textured asset is produced.
Figure 4: Qualitative comparison. As shown, the controllable video baselines suffer from significant appearance drift, the Janus problem. In contrast, our method maintains strong object identity and consistency throughout the rotation, effectively eliminating such artifacts.
Video Metrics
Method
PSNR (↑)
SSIM (↑)
LPIPS (↓)
FVD (↓)
Wan2.1-Fun (Depth)
23.48
0.904
0.084
277.9
Wan2.1-Fun (Canny)
24.32
0.916
0.074
235.0
Diffusion as Shader
23.21
0.906
0.090
295.7
Ours
25.79
0.924
0.066
189.9
Table 1: Quantitative comparison of generated Turntable Videos.
Texture Metrics
Method
FID (↓)
KID (↓)
CLIP-S (↑)
Time (↓)
Paint3D
43.47
36.63
0.8880
96s
HY3D-2.0
31.45
22.52
0.9165
33s
TEXGen
39.11
35.09
0.8906
15 s
MV-Adapter
30.68
25.73
0.9291
32s
Ours
26.90
16.21
0.9488
87s
Table 2: Quantitative comparison of generated textures. All metrics are evaluated on multi-view renderings.
Method
PSNR (↑)
SSIM (↑)
LPIPS (↓)
FVD (↓)
w.o normal
25.15
0.918
0.073
203.8
w.o position
25.66
0.921
0.069
191.1
Ours
25.79
0.924
0.066
189.9
Table 3: Quantitative ablation results on the Geometry Control Fusion module
Figure 6: The initial Iter.1 result has an incomplete texture due to self-occlusion. The w/o Inpainting baseline that generates a second TTV independently suffers from visible seams and color shifts.In contrast, our full pipeline with context-aware inpainting produces a seamless and globally consistent result.
Figure 7: Our consistent TTVs can supervise both traditional texturing and neural representations like 3DGS, with the latter better capturing volumetric details like fur.
Recent 3D generative models can synthesize high-quality geometry but often struggle to reproduce intricate textures from reference images, largely due to the scarcity of large-scale 3D training data with rich surface appearance. In contrast, visual generative models are trained on datasets several orders of magnitude larger and excel at modeling complex visual patterns. Motivated by this gap, we introduce Ink3D, a framework that bridges 3D generation with large-scale video generative models to synthesize extremely complex textures. Ink3D first reconstructs a white-mesh geometry using an off-the-shelf 3D generation model. It then employs OrbitPainter, a conditional video generative model, to produce dense orbit-scan videos capturing object appearance across viewpoints. To convert these views into coherent textures, we introduce TextureOptimizer, a neural baking module that integrates dense multi-view observations while mitigating geometry inconsistencies arising from video generation. By decoupling geometry and texture synthesis and leveraging large-scale pretrained video priors, Ink3D enables significantly richer and more faithful texture generation than prior approaches.
Yue Han, Chong Li, Zhening Liu +5
ZGCA & ZGCI · Zhejiang University · Microsoft Research +1
Generating complete digital twins from videos requires precise camera control, global scene coverage, and strict spatial-temporal consistency constraints that remain challenging for perspective video generators due to their limited field of view (FoV). Their narrow FoV forces long or multi-view trajectories, amplifying cross-view inconsistency and temporal drift. We argue that 360° video generation offers a natural solution: panoramic coverage simplifies trajectory design and provides a strong global context for maintaining coherence. We introduce Pantheon360: Taming Digital Twin Generation via 3D-Aware 360° Video Diffusion, a controllable 360° video generation framework that synthesizes high-fidelity videos from sparse 360° inputs. The key idea is an explicit 3D Cache, reconstructed from the input, which serves as a geometric scaffold for any user-defined camera path. This allows the diffusion model to focus on photorealistic texture refinement while the 3D Cache enforces global geometric consistency. Experiments show that Pantheon360 achieves superior visual quality and unmatched geometric coherence, enabling reliable and flexible 360° scene generation for downstream simulation and digital-twin applications.
Ting-Hsuan Chen, Ying-Huan Chen, Tao Tu +10
University of Southern California · National Yang Ming Chiao Tung University · Cornell University +1
Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing -- retrieving relevant past frames and establishing dense correspondences with the target viewpoints -- while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.