IntrinSync: Joint Intrinsic Decomposition and Reciprocal Rendering

Intrinsic image decomposition aims to recover material and illumination factors from RGB observations, but real-world images entangle reflectance with illumination, visibility, shadows, and non-diffuse appearance. Recent single-image methods address this entanglement with a residual image formation model, decomposing RGB into albedo, diffuse shading, and a non-diffuse residual. However, applying such decomposition independently to consecutive frames lacks scene-level context, leading to inconsistent factor assignments, albedo drift, and leakage across views. Meanwhile, multi-view inverse-rendering methods recover properties, such as material, lighting, and geometry, but they rely on synthetic data due to the highly uncertain estimation, and do not generalize to supervision by real images. We present MVID, Multi-View Intrinsic image Decomposition, a feed-forward framework built on the residual image formation model. MVID builds a scene-level multi-view representation and decodes a view-consistent albedo together with coherent per-view shading and residual factors through a factor query adapter, while using the same image formation model for self-supervised RGB reconstruction on unlabeled real-world sequences. Experiments on indoor, real-world, and outdoor benchmarks show that MVID improves both per-frame decomposition quality and cross-view consistency over single-view intrinsic, generative intrinsic, and multi-view inverse-rendering baselines. The resulting view-stable factors support practical image-space applications, including multi-view consistent illumination editing and specularity removal.

Intrinsic image decomposition aims to estimate physically based rendering (PBR) parameters such as albedo, roughness, and metallicity from images. While recent methods achieve strong single-view predictions, applying them independently to multiple views of the same scene often yields inconsistent estimates, limiting their use in downstream applications such as editable neural scenes and 3D reconstruction. Video-based models can improve cross-frame consistency but require dense, ordered sequences and substantial compute, limiting their applicability to sparse, unordered image collections. We propose Geo-ID, a novel test-time framework that repurposes pretrained single-view intrinsic predictors to produce cross-view consistent decompositions by coupling independent per-view predictions through sparse geometric correspondences that form uncertainty-aware consensus targets. Geo-ID is model-agnostic, requires no retraining or inverse rendering, and applies directly to off-the-shelf intrinsic predictors. Experiments on synthetic benchmarks and real-world scenes demonstrate substantial improvements in cross-view intrinsic consistency as the number of views increases, while maintaining comparable single-view decomposition performance. We further show that the resulting consistent intrinsics enable coherent appearance editing and relighting in downstream neural scene representations.

Inverse rendering aims to recover both 3D geometry and physically meaningful material properties from images, enabling applications such as relighting and novel view synthesis. Optimization-based methods achieve high fidelity but require costly per-scene fitting, while image-space learning-based approaches often suffer from multi-view inconsistencies and lack an explicit 3D representation for stable novel view rendering. We present a feed-forward multi-view reconstruction framework for inverse rendering that directly predicts a structured 3D Gaussian representation with intrinsic material attributes. Each Gaussian primitive is parameterized by mean, normal, opacity, rotation, scale, albedo, metallic, and roughness, enabling a disentangled and physically grounded scene representation. Our model integrates priors from a material estimation network with a multi-view 3D reconstruction backbone, allowing joint prediction of geometry and reflectance parameters in a single forward pass. Experiments on synthetic and real-world datasets demonstrate improved multi-view consistency compared to 2D baselines, accurate material recovery, and stable novel view rendering. Our representation further supports physically-based relighting and more faithful modeling of view-dependent effects compared to existing RGB-based feed-forward reconstruction methods. Our project webpage is: https://poliik.github.io/invsplat/.