Organizations: Data Science & Artificial Intelligence Research Institute, China Unicom · Unicom Data Intelligence, China Unicom · School of Computer Science and Software Engineering, Southwest Petroleum University · DJI Technology Co.,Ltd. · Beijing Key Laboratory of Science Fiction Audio and Video Intelligent Processing
Current 2D-to-3D conversion methods achieve geometric accuracy but are artistically deficient, failing to replicate the immersive and emotionally resonant experience of professional 3D cinema. This is because geometric reconstruction paradigms mistake deliberate artistic intent, such as strategic zero-plane shifts for pop-out effects and local depth sculpting, for data noise or ambiguity. This paper argues for a new paradigm: Artistic Disparity Synthesis, shifting the goal from physically accurate disparity estimation to artistically coherent disparity synthesis. We propose Art3D, a preliminary framework exploring this paradigm. Art3D uses a dual-path architecture to decouple global depth parameters (macro-intent) from local artistic effects (visual brushstrokes) and learns from professional 3D film data via indirect supervision. We also introduce a preliminary evaluation method to quantify cinematic alignment. Experiments show our approach demonstrates potential in replicating key local out-of-screen effects and aligning with the global depth styles of cinematic 3D content, laying the groundwork for a new class of artistically-driven conversion tools.
Figures & tables
Figure 1: Artistic ambiguity in 3D film production. The same scene can exhibit distinct 3D perceptions under different artistic directions. To illustrate this, we simulate various 3D production choices and visualize their effects. From left to right, the three columns show the left view, right view, and the resulting red–cyan anaglyph (best viewed when zoomed in). Comparing the first and second rows, variations in the stereo camera’s baseline b and focal length f lead to different disparities, reflecting differences in the Mastery of Global Depth . The second and third rows show zero-plane ambiguity , where the perceived depth reference shifts from the foreground cow to the distant forest, often due to creative camera setup or post-production adjustment. The fourth row illustrates over-simple depth layering , where disparity remains nearly uniform, likely reflecting cost-efficient rather than artistic intent.
Figure 2: Art3D Framework. Our pipeline takes a 2D Left View as input. 1. Geometric Feature Extraction (Frozen): ‘DepthNet’ extracts geometric features and the inverse depth map, ‘geometric canvas ( iz )’. 2. Artistic Blueprint & Masks (Frozen, Data Construction Stage): During the data construction stage (no training or inference required), ‘StereoNet’ estimates the target artistic blueprint ( dL ) and provides a valid pixel mask ( Mvalid ) via its left-right consistency check. Simultaneously, ‘Lang-SAM’ analyzes the Left View to generate the local effects mask ( Mlocal ). The final global style mask ( Mglobal ) is then derived from valid regions that are not part of the local effects (i.e., Mvalid⋅(1−Mlocal) ). 3. CameraNet (Trainable): This network takes features and iz to synthesize the virtual camera parameters ( vs,vt ), which are pixel-level tensors , and a preliminary virtual right disparity map ( d^R ). The virtual left disparity map ( d^L ) is then constructed as iz×vs+vt . 4. Dual-Path Supervision: Our core ‘ LArt ’ loss guides the learning of vs,vt by comparing d^L with dL , weighted by Mglobal (for global style) and Mlocal (for local effects). An additional left-right consistency check loss further refines d^R . 5. Virtual Right View Synthesis: The final d^R , combined with the original Left View , can be used to generate the Virtual Right View via standard warping techniques. For more details on the loss functions, please refer to Sec. 3 .
Operation
Details
Description
Input
Features and inverse depth ( iz )
Input size = 512×512
Downsampling (x3)
Conv1x1 + ReLU + Conv3x3 + ReLU
Output size = 64×64 (encoder features)
Upsampling (x3)
Conv1x1 + ReLU + Deconv3x3 + ReLU
Output size = 512×512 (decoder features)
Output
Conv1x1 (out_channels = 3)
Virtual camera parameters vs and vt , and right disparity map d^R
Table 1: Overview of the CameraNet Architecture. This is the only component trained in our framework.
Figure 3: Qualitative analysis of Sculpting of Local Effects on 2D inputs. Row 1: Input image. Row 2: Ours (w/o Lpath(Mlocal) ), which fails to sculpt local pop-out effects. Row 3: Owl3D, which produces partial effects but lacks artistic consistency. Row 4: Art3D (Full Model) , which successfully sculpts strong and coherent pop-out effects. Best viewed zoomed in and with red–cyan anaglyph filters.
Figure 4: Analysis of Geometric Consistency via DDC-IoU. Each row presents three independent samples. For each sample, the Anaglyph 3D view is shown on top. Below it, the corresponding right disparity map ( dR or d^R ) is displayed alongside the right geometric canvas ( izR ) derived from that disparity: for ground truth, izR is obtained by feeding the true right image into DepthNet; for our estimate, izR is computed from the synthesized right view generated by warping the left image using d^R . Top Row (Original 3D Films): Visualizes the inconsistent quality of source data. The leftmost sample shows poor structural alignment ( DDC-IoU=0 ). The middle and rightmost samples are acceptable ( DDC-IoU≥0.8 ). Bottom Row (Our Art3D Output): Shows our model’s synthesized blueprints ( d^R ) for the same three scenes shown above. Our outputs consistently achieve high DDC-IoU scores (e.g., 0.85, 0.83, 0.89), demonstrating that artistic style is learned without corrupting the underlying geometry. Visually, the synthesized disparity maps are coherent and structurally sound.
Method
Immersion ↑
Visual Comfort ↑
Style Consistency ↑
Overall Preference ↑
Baseline (DA-V2)
35.2%
41.5%
22.8%
20.0%
Art3D (Ours)
64.8%
58.5%
77.2%
80.0%
Table 5: User Study : Art3D vs. Depth-Anything-V2 (Geometric-only, N=50 videos, 25 participants)
3D Gaussian Splatting (3DGS) delivers high-fidelity real-time rendering but suffers from geometric and photometric degradations under sparse-view constraints. Current generative restoration approaches are often limited by insufficient temporal coherence, a lack of explicit spatial constraints, and a lack of large-scale training data, resulting in multi-view inconsistencies, erroneous geometric hallucinations, and limited generalization to diverse real-world artifact distributions. In this paper, we present ArtifactWorld, a framework that resolves 3DGS artifact repair through systematic data expansion and a homogeneous dual-model paradigm. To address the data bottleneck, we establish a fine-grained phenomenological taxonomy of 3DGS artifacts and construct a comprehensive training set of 107.5K diverse paired video clips to enhance model robustness. Architecturally, we unify the restoration process within a video diffusion backbone, utilizing an isomorphic predictor to localize structural defects via an artifact heatmap. This heatmap then guides the restoration through an Artifact-Aware Triplet Fusion mechanism, enabling precise, intensity-guided spatio-temporal repair within native self-attention. Extensive experiments demonstrate that ArtifactWorld achieves state-of-the-art performance in sparse novel view synthesis and robust 3D reconstruction. Code available at: https://github.com/fyting/ArtifactWorld.
Generating 3D-consistent novel views from 2D diffusion models remains a fundamental challenge in computer vision. In this work, we introduce a framework that enhances geometric alignment for monocular view synthesis under drastic camera changes by jointly generating a target view and its corresponding depth map. By conditioning the diffusion pipeline on the single source image, source depth, and target camera rays, our approach forces the model to inherently reason in 3D space even under extreme viewpoint shifts. To guarantee spatial fidelity, we propose a novel per-pixel depth consistency constraint, which verifies and enforces 3D consistency across generated views. Our framework establishes a robust pipeline for geometrically consistent view synthesis, yielding superior visual accuracy and spatial alignment compared to current state-of-the-art methods. With this work, we pave the road towards robust 3D world models capable of understanding complex environments from a single view image.
Ge Ya Luo, Daniel Severo, Brian Karrer +3
FAIR at Meta · Mila – Québec AI Institute · Université de Montréal +3
Per-scene optimization methods such as 3D Gaussian Splatting provide state-of-the-art novel view synthesis quality but extrapolate poorly to under-observed areas. Methods that leverage generative priors to correct artifacts in these areas hold promise but currently suffer from two shortcomings. The first is scalability, as existing methods use image diffusion models or bidirectional video models that are limited in the number of views they can generate in a single pass (and thus require a costly iterative distillation process for consistency). The second is quality itself, as generators used in prior work tend to produce outputs that are inconsistent with existing scene content and fail entirely in completely unobserved regions. To solve these, we propose a two-stage pipeline that leverages two key insights. First, we train a powerful bidirectional generative model with a novel opacity mixing strategy that encourages consistency with existing observations while retaining the model's ability to extrapolate novel content in unseen areas. Second, we distill it into a causal auto-regressive model that generates hundreds of frames in a single pass. This model can directly produce novel views or serve as pseudo-supervision to improve the underlying 3D representation in a simple and highly efficient manner. We evaluate our method extensively and demonstrate that it can generate plausible reconstructions in scenarios where existing approaches fail completely. When measured on commonly benchmarked datasets, we outperform all existing baselines by a wide margin, exceeding prior state-of-the-art methods by 1-3 dB PSNR.
Riccardo de Lutio, Tobias Fischer, Yen-Yu Chang +7
NVIDIA, USA · NVIDIA, Switzerland · ETHZ, Switzerland +4