VIRGi: View-dependent Instant Recoloring of 3D Gaussians Splats
Authors: Alessio Mazzucchelli, Ivan Ojeda-Martin, Fernando Rivas-Manzaneque, Elena Garces, Adrian Penate-Sanchez, Francesc Moreno-Noguer
Organizations: Arquimea Research Center, San Cristóbal de la Laguna, Santa Cruz de Tenerife, 38320 · Universidad Polit`ecnica de Catalunya, Doctoral Degree in Automatic Control, Robotics and Vision, Carrer de Jordi Girona, 31, Les Corts, Barcelona, 08034, Spain · Volinga AI, San Cristóbal de la Laguna, Santa Cruz de Tenerife, 38320 · Universidad Polit´ecnica de Madrid, Programa de Doctorado en Autom´atica y Rob´otica, Calle de Jos´e Guti´errez Abascal 2, Madrid, 28006, Spain · Universidad Rey Juan Carlos, C/Tulip´an S/N, 28933 M´ostoles (Madrid), Spain · IUSANI, Universidad de Las Palmas de Gran Canaria, C/ Juan de Quesada 30, Las Palmas de Gran Canaria, 35001, Spain · Institut de Rob`otica i Inform`atica Industrial (IRI), CSIC-UPC, C/ Llorens i Artigas 4, Barcelona, 08028, Spain
3D Gaussian Splatting (3DGS) has recently transformed the fields of novel view synthesis and 3D reconstruction due to its ability to accurately model complex 3D scenes and its unprecedented rendering performance. However, a significant challenge persists: the absence of an efficient and photorealistic method for editing the appearance of the scene's content. In this paper we introduce VIRGi, a novel approach for rapidly editing the color of scenes modeled by 3DGS while preserving view-dependent effects such as specular highlights. Key to our method are a novel architecture that separates color into diffuse and view-dependent components, and a multi-view training strategy that integrates image patches from multiple viewpoints. Improving over the conventional single-view batch training, our 3DGS representation provides more accurate reconstruction and serves as a solid representation for the recoloring task. For 3DGS recoloring, we then introduce a rapid scheme requiring only one manually edited image of the scene from the end-user. By fine-tuning the weights of a single MLP, alongside a module for single-shot segmentation of the editable area, the color edits are seamlessly propagated to the entire scene in just two seconds, facilitating real-time interaction and providing control over the strength of the view-dependent effects. An exhaustive validation on diverse datasets demonstrates significant quantitative and qualitative advancements over competitors based on Neural Radiance Fields representations.
Figures & tables
Fig. 1: V I R Gi enables – by editing a single view – instant recolorings of one or multiple objects in scenes modeled with 3D Gaussian Splats. Our key idea of decomposing the reflectance of the scene into diffuse and specular components allows view-dependent consistent edits. Our separation can also be used to enhance or reduce material specularity.
Fig. 2: Comparison of recoloring methods with a ground truth recolor. Our method V I R Gi is the first one to propose a solution for 3DGS recoloring, outperforming one of the best methods in NeRFs recoloring [ 15 ] (IReNe). We propose several contributions that make our method robust and better than a baseline ( Vanilla- V I R Gi ) obtained by naïvely combining existing methods.
Fig. 3: Overview of V I R Gi . Our method comprises two main steps. In the initial step, we train a 3DGS architecture consisting of a Hashgrid f representing the geometry and two MLPs for color modeling. We separate the observed color into a diffuse term \textscMLPdiff , which solely relies on hashgrid features f , and a specular term \textscMLPspec , which incorporates the view direction θ . Our training strategy involves leveraging multiple views of the same scene point in a single batch, as opposed to using a single image per batch, resulting in improved PSNR compared to previous methods. In the second step, given an edited 2D image, we fine-tune the last layer of the diffuse MLP using a soft-segmentation mask α to achieve the target color.
Original image
Monoview
Multiview
Fig. 4: Specularity Comparison: Monoview vs. Multiview. Training with a single viewpoint poses challenges in learning specularities due to limited sample diversity, multiview training facilitates an easier understanding of specular effects.
Mip NeRF 360 [ 71 ]
LLFF [ 75 ]
NeRF Synthetic [ 2 ]
Method
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PaletteNeRF
22.24
0.720
0.278
24.19
0.800
0.187
29.92
0.960
0.038
RecolorNeRF
-
-
-
24.52
0.736
0.333
29.00
0.943
0.048
IReNe
26.80
0.714
0.292
26.51
0.823
0.136
30.33
0.959
0.035
Vanilla- V I R Gi
29.19
0.881
0.150
28.01
0.905
0.129
32.13
0.975
0.023
V I R Gi
29.74
0.886
0.146
29.10
0.915
0.119
31.65
0.971
0.027
TABLE I: Quantitative color edition results. Best , Second . Results using the recolored ground truth images from [ 15 ] .
Mip NeRF 360
LLFF
NeRF Synthetic
DC
MV
DS
PSNR ↑
PSNR ↑
PSNR ↑
✗
✗
✗
29.19
28.01
32.13
✗
✓
✗
29.45
28.54
31.90
✗
✓
✓
29.56
28.62
31.78
✓
✗
✗
28.58
27.66
30.93
✓
✓
✗
29.61
29.15
31.50
TABLE II: Ablation study. Recoloring results on all datasets of our ablated method . Best , Second . DC indicates if the output is decoupled or not. MV indicates if the model has been trained using our multi-view rasterization DS indicates if the diffuse’s network \textscMLPdiff output has been introduced into the soft segmentation network \textscMLPseg .
Dataset (PSNR ↑ )
Mip NeRF 360
LLFF
NeRF Synthetic
Mono-view C3DGS
28.96
25.63
35.60
Multi-view C3DGS
29.47
25.87
35.84
Mono-view V I R Gi
28.43
25.73
33.87
Multi-view V I R Gi
29.30
25.97
35.76
TABLE III: Multi-View VS Mono-View novel view synthesis . Training both the original C3DGS as well as V I R Gi using the proposed multi-view approach enhances the novel view synthesis results. We underline that V I R Gi also provides a decomposition of the image into diffuse and specular.
Fig. 5: Comparison between V I R Gi (upper row per case) and Gaussian Editor [ 68 ] (bottom row per case). Gaussian Editor requires the user to initialize a mask using SAM [ 76 ] before applying color edits based on provided prompts: “kitchen - make it magenta”, “bicycle - make it yellow”, and “bonsai - make it green”. Gaussian Editor exhibits several drawbacks, such as color bleeding into surrounding areas in the first two scenes and not understanding the scene structure when recoloring the third scene. Notably, Gaussian Editor takes an average of 612 seconds to process these scenes, while V I R Gi completes the task in just 2 seconds on average.
Fig. 6: Illumination editing with decoupled output. Top row: specular component reduced by applying a multiplicative factor of 0 . Bottom row: specular component increased by applying a factor of 1.5 .
Fig. 7: Qualitative results. Results on all scenes from the MipNeRF-360 dataset [ 71 ] . This dataset shows complex real scenes with a complete 360 rotation on all scenes. We show several examples of single and multiple color editions. The first column is the original image. The second column is the edit performed by the user and used to train V I R Gi . We also show the soft segmentation predicted by V I R Gi for each recolored image.
Fig. 8: Some qualitative results of the IReNe [ 15 ] dataset.
Fig. 9: Limitations of diffuse/specular modeling. First row Kitchen scene, second row Counter scene. Comparison between Ground Truth, Diffuse, Specular and Combined RGB render. Our simple diffuse/specular model struggles with complex or anisotropic materials. This can be seen in the Counter (bottom) scene, while the Kitchen (top) scene, especially the table, shows better results. Still, both yield solid RGB reconstructions. Third row - Fourth Row: Comparison of original, edited, and rendered views on specular failure cases. In the case of the water jug, although we are able to apply the recoloring, the result lacks uniformity: some areas of the jug appear more red than others, and the color bleeds onto objects visible through the jug’s transparent surface. For the reflective bowl, we manage to change its color as well, but the edits are less clean compared to those on non-reflective, opaque objects. Some regions of the bowl retain the original color, with only a slight green tint appearing in the edited output
Fig. 10: Limitations in segmentation. We show examples where issues in recoloring happened. These cases occur mainly with parts of the scene that were not present in the first edited image. In the first example, the notebook has been incorrectly recolored. In the second example, the palm tree has been incorrectly recolored. In the third example, the legs of the table have not been fully recolored to red. These cases were fixed by adding a second edited image. In most cases, such issues can be solved by adding an additional edited image, but in some cases, like the last example, certain edits cannot be achieved. In this particular case, the curtain is far from the camera, and not much viewpoint change is present in the dataset from which to learn the correct model.
Gaussian Splatting has recently become one of the most popular frameworks for photorealistic 3D scene reconstruction and rendering. While current rasterizers allow for efficient mappings of 3D Gaussian splats onto 2D camera views, this work focuses on mapping 2D image information (e.g. color, neural features or segmentation masks) efficiently back onto an existing scene of Gaussian splats. This 'opposite' direction enables applications ranging from scene relighting and stylization to 3D semantic segmentation, but also introduces challenges, such as view-dependent colorization and occlusion handling. Our approach tackles these challenges using the normal equation to solve a visibility-weighted least squares problem for every Gaussian and can be implemented efficiently with existing differentiable rasterizers. We demonstrate the effectiveness of our approach on scene relighting, feature enrichment and 3D semantic segmentation tasks, achieving up to an order of magnitude speedup compared to gradient descent-based baselines.
3D Gaussian Splatting (3D-GS) enables real-time 3D scene reconstruction but lacks robust segmentation for editing tasks such as object removal, extraction, and recoloring. Existing approaches that lift 2D segmentations to the 3D domain suffer from view inconsistencies and coarse masks. In this paper, we propose a novel framework that leverages the Segment Anything Model High Quality (SAM-HQ) to generate accurate 2D masks, addressing the limitations of the standard SAM in boundary fidelity and fine-structure preservation. To achieve robust 3D segmentation of any target object in a given scene, we introduce a prior-guided label reassignment method that assigns labels to 3D Gaussians by enforcing multiview consistency with learned priors. Our approach achieves state-of-the-art segmentation accuracy and enables interactive, real-time object editing while maintaining high visual fidelity. Qualitative results demonstrate superior boundary preservation and practical utility in Virtual Reality (VR) and robotics, advancing 3D scene editing.
Language-driven 3D Gaussian Splatting (3DGS) editing provides a more convenient approach for modifying complex scenes in VR/AR. Standard pipelines typically adopt a two-stage strategy: first editing multiple 2D views, and then optimizing the 3D representation to match these edited observations. Existing methods mainly improve view consistency through multi-view feature fusion, attention filtering, or iterative recalibration. However, they fail to explicitly address a more fundamental issue: the semantic correspondence between edited 2D evidence and 3D Gaussians. To tackle this problem, we propose TransSplat, which formulates language-driven 3DGS editing as a multi-view unbalanced semantic transport problem. Specifically, our method establishes correspondences between visible Gaussians and view-specific editing prototypes, thereby explicitly characterizing the semantic relationship between edited 2D evidence and 3D Gaussians. It further recovers a cross-view shared canonical 3D edit field to guide unified 3D appearance updates. In addition, we use transport residuals to suppress erroneous edits in non-target regions, mitigating edit leakage and improving local control precision. Qualitative and quantitative results show that, compared with existing 3D editing methods centered on enhancing view consistency, TransSplat achieves superior performance in local editing accuracy and structural consistency.
Yanhui Chen, Jiahong Li, Jingchao Wang +3
Guangdong University of Technology · Peking University