ControlGS: Conditioning Neural Gaussians for Downstream-Processing-Aware XR Rendering

Weikai Lin; Junjie Zhao; Carl Marshall; Sushant Kondguli; Yuhao Zhu

cs.CVSep 25, 2026

ControlGS: Conditioning Neural Gaussians for Downstream-Processing-Aware XR Rendering

Authors: Weikai Lin, Junjie Zhao, Carl Marshall, Sushant Kondguli, Yuhao Zhu

Organizations: University of Rochester, Rochester, USA · Reality Labs, Meta, Redmond, USA

Abstract

Extended Reality (XR) users do not directly perceive the output of a rendering engine. Instead, rendered images pass through a post-processing pipeline and the physical display-optics path before reaching the eye. Critically, the exact downstream processing can vary significantly at run time, influenced by, for instance, camera pose and display power budget. Traditional 3DGS methods either implicitly assume that this downstream pipeline preserves image quality or cannot adapt to downstream processing changes. To bridge this gap, we present ControlGS, an XR Gaussian rendering pipeline that optimizes end-to-end visual quality. ControlGS models and integrates the entire downstream processing, between the rendering output and the human eye, into the optimization objective. To adapt to downstream processing at run time, ControlGS dynamically generates Gaussian primitives conditioned upon the downstream processing parameters. Experiments show that ControlGS consistently improves end-to-end post-optics XR quality across different neural Gaussian backbones and datasets, with minimal overhead. Code is available at https://horizon-lab.org/controlgs/.

Figures & tables

Fig. 1. ControlGS enables downstream-processing–aware XR rendering. Top: a Gaussian-rendered image passes through post-processing and the display-optics path before forming the post-optics virtual image. Bottom: compared with Scaffold-GS and its supersampling-enhanced variant (SS-Scaffold-GS), ControlGS produces higher-quality virtual images under both native resolution with no power saving and $8{\times}$ downsampling with $80\%$ power saving.

Fig. 2. XR pipeline for Gaussian rendering. The rendered image from a Gaussian renderer passes through post-processing modules and the physical display-optics path before reaching the user’s eye. Thus, the user perceives the final eye-space image rather than the raw rendered image. For clarity, the figure places the perceived image on the right side of the optics, although an actual near-eye display forms a virtual image on the opposite side.

Fig. 3. Overview of ControlGS . (a) A neural Gaussian renderer decodes Gaussian attributes from pretrained feature anchors and rasterizes them into an image. (b) For each feature and downstream module, ControlGS learns a condition vector, projects it into the decoder hidden space, and scales it based on injection strength before injection (Eqn. 13 ). (c) The conditioned rendering result is passed through fixed XR downstream modules and the plug-in is trained with end-to-end supervision on the final output image (Eqn. 16 ).

Fig. 4. Average quality improvement of ControlGS over SS-Scaffold-GS on Mip-NeRF 360. Heatmaps show PSNR, SSIM, and LPIPS gains across downsample scale and display-power-saving settings, averaged over all scenes. ControlGS consistently improves end-to-end quality .

Fig. 5. Effect of the condition dimension on end-to-end quality. The x-axis is the total condition dimension across anti-aliasing resampling, lens correction, and display mapping. The y-axis reports PSNR, SSIM, and LPIPS on the Garden and Lego scenes.

Run-Time–Fixed Gaussian Splatting
Method / Dataset	Mip-NeRF 360			NeRF Synthetic
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
3DGS	18.15	0.478	0.579	21.96	0.840	0.166
SS-3DGS	18.93	0.496	0.558	23.36	0.869	0.146
Mip-Splatting	18.87	0.493	0.564	23.17	0.862	0.152
SS-Mip-Splatting	18.93	0.497	0.557	23.37	0.870	0.145

Table 1. End-to-end quality after the XR post-processing pipeline, averaged over all scenes and

20

resolution-power run-time settings. ControlGS here uses the MLP in SS-Scaffold-GS; results on other MLPs are in Supp. D.1 .

Method	ColorVideoVDP $\uparrow$	HDR-VDP-3 $\uparrow$	HDR-FLIP $\downarrow$	FovVideoVDP $\uparrow$
SS-Scaffold-GS	7.179	6.971	0.455	7.001
ControlGS	7.232	7.111	0.418	7.079

Table 2. Perceptual quality on Mip-NeRF 360.

Method / Dataset	Mip-NeRF 360		NeRF Synthetic
	FPS $\uparrow$	Storage (MB) $\downarrow$	FPS $\uparrow$	Storage (MB) $\downarrow$
SS-3DGS	214.9	647.51	764.0	58.51
SS-Scaffold-GS	337.9	180.53	920.5	15.13
ControlGS	268.9	235.36	702.7	19.72

Table 3. FPS and model storage.

Fig. 6. Qualitative end-to-end comparison on two representative scenes. We compare Scaffold-GS, its supersampled variant, and ControlGS under no downsampling/no power saving and under $8{\times}$ downsampling with $80\%$ display-power saving. Numbers report PSNR / SSIM / LPIPS for the shown view; bold indicates the best value among the three methods. Qualitative comparison of Scaffold-GS, SS-Scaffold-GS, and ControlGS on the Stump and Materials scenes under two XR run-time settings.

Fig. 7. Lens-aware condition injection under spatially varying aberrations. Left: the spatially varying PSFs (top) and injection-strength map (bottom). Middle and right: results without and with lens injection, respectively; the top and bottom rows show the rendered and post-optics images.

Setting	$\Delta$ PSNR $\uparrow$	$\Delta$ SSIM $\uparrow$	$\Delta$ LPIPS $\downarrow$
Canonical	+0.37	+0.028	-0.031
PSF $0.9\times$	+0.38	+0.028	-0.032
PSF $1.1\times$	+0.37	+0.026	-0.028
RGB wavelengths $-10$ nm	+0.37	+0.028	-0.030
RGB wavelengths $+10$ nm	+0.35	+0.027	-0.029
Gamma $2.3$	+0.36	+0.027	-0.031

Table 4. Robustness to downstream-model mismatch on Mip-NeRF 360. Values report the quality improvement of ControlGS over SS-Scaffold-GS.

Method	Mip-NeRF 360			NeRF Synthetic
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
ControlGS	19.33	0.524	0.528	23.59	0.871	0.137
w/o Lens	19.25	0.521	0.534	23.52	0.870	0.139
w/o Display Mapping	19.07	0.511	0.531	23.42	0.870	0.138
w/o Resampling	19.32	0.523	0.530	23.58	0.871	0.137
w/o State-Aware	19.05	0.513	0.526	23.40	0.870	0.136

Table 5. Ablation studies on Mip-NeRF 360 and NeRF Synthetic. Results are averaged over all scenes and downstream-pipeline settings.

Fig. 8. Condition-response visualization on Lego and Room. We vary one condition at a time and show the raw rendered outputs before the downstream pipeline. From top to bottom, the groups show the effects of anti-aliasing resampling, display mapping, and lens correction. For lens correction, we compare the original Scaffold-GS rendering with the ControlGS -conditioned rendering at the peripheral crops.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Object distance	$39.07$ mm
Lens thickness	$8.795$ mm
Lens material	PMMA $(n_{d}=1.492,\;v=57.4)$
Lens diameter	$34$ mm
Eye relief	$18$ mm
Projected virtual image distance	$667$ mm

Appendix

Table S1. Optical parameters used for the Google Cardboard lens simulation. Most parameters are taken from the public Google Cardboard specification, while the lens surface radii

R_{1}

and

R_{2}

are optimized in Zemax.

Fig. S1. Zemax setup for our optics simulation. Left: overview of the full optical system. Middle: zoom-in view of the spatial layout of the display, lens, and eye model. Right: lens parameter settings used in the simulation.

Fig. S2. Average quality improvement of ControlGS over SS-Scaffold-GS on NeRF Synthetic. Heatmaps show PSNR, SSIM, and LPIPS gains across downsample scale and display-power-saving settings, averaged over all scenes. ControlGS consistently improves end-to-end quality.

Method / Dataset	Mip-NeRF 360		NeRF Synthetic
	FPS $\uparrow$	Storage (MB) $\downarrow$	FPS $\uparrow$	Storage (MB) $\downarrow$
Scaffold-GS (No SS, 2x)	369.7	180.53	744.6	15.13
Scaffold-GS (No SS, 4x)	278.2	180.53	478.0	15.13
Scaffold-GS (No SS, 8x)	157.8	180.53	250.7	15.13

Appendix

Table S2. Supersampling impact. SS: Supersampling. No-SS: direct low-resolution rendering.

Fig. S3. Camera-based verification of the display–optics simulation against real captures. We display the input image on an AMOLED display and capture the result through a Google Cardboard lens using a calibrated Sony FE 24–70mm F2.8 GM II camera. Left: input image shown on the display. Middle: calibrated camera capture through the real lens. Right: simulated result from our display-optics model. Remaining color differences may come from the simplified emission spectrum model of the display, camera sensor response functions, and potential Image Signal Processor (ISP) post-processing, which are not fully accounted for in our simulation.

Fig. S4. Examples of dynamic runtime states. Top: as the camera moves farther from an object, the object occupies fewer pixels on the image plane, reducing its effective screen-space sampling rate. Bottom: the same Gaussian primitives can project to different screen-space locations under different camera poses. Because lens aberrations vary spatially, the same primitives should render differently at different projected locations to compensate for location-dependent optical aberrations. This requires dynamically changing the aberration-related state during rendering.

run-time -fixed Gaussian Splatting
Method / Dataset	Mip-NeRF 360			NeRF Synthetic
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
3DGS	18.15	0.478	0.579	21.96	0.840	0.166
SS-3DGS	18.93	0.496	0.558	23.36	0.869	0.146
Mip-Splatting	18.87	0.493	0.564	23.17	0.862	0.152
SS-Mip-Splatting	18.93	0.497	0.557	23.37	0.870	0.145

Appendix

Table S3. End-to-end quality across neural Gaussian backbones after the XR post-processing pipeline. Each cell averages all test views and all

20

resolution-power run-time settings. Rows without a prefix report the original backbone baseline, SS rows use the supersampling-enhanced baseline, and Control rows further add learned condition-injection adapters to the same frozen backbone. ControlGS consistently improves end-to-end quality across backbones.

Metric	Method	Mip-NeRF 360									Avg
Metric	Method	Bicycle	Bonsai	Counter	Flowers	Garden	Kitchen	Room	Stump	Treehill	Avg
PSNR $\uparrow$	3DGS	17.32	19.79	19.06	15.51	17.08	17.93	20.09	19.46	17.11	18.15
	SS-3DGS	18.14	20.86	19.72	16.48	17.93	18.50	20.74	20.27	17.77	18.93
	Mip-Splatting	18.17	20.68	19.62	16.42	17.88	18.41	20.58	20.25	17.81	18.87
	SS-Mip-Splatting	18.15	20.84	19.73	16.51	17.93	18.51	20.69	20.29	17.75	18.93
	CompGS	17.40	19.96	19.11	15.69	17.15	17.93	20.10	19.57	17.42	18.26

Appendix

Table S4. Per-scene end-to-end PSNR and SSIM on Mip-NeRF 360. Each scene cell averages all test views and all 20 resolution-power settings. Rows without prefix report the original backbone baseline without supersampling. SS rows use the enhanced supersampling baseline. Control rows add learned condition-injection adapters.

Metric	Method	Mip-NeRF 360									Avg
Metric	Method	Bicycle	Bonsai	Counter	Flowers	Garden	Kitchen	Room	Stump	Treehill	Avg
LPIPS $\downarrow$	3DGS	0.636	0.494	0.500	0.655	0.648	0.543	0.479	0.613	0.643	0.579
	SS-3DGS	0.612	0.479	0.484	0.621	0.625	0.525	0.467	0.583	0.621	0.558
	Mip-Splatting	0.616	0.484	0.489	0.633	0.632	0.532	0.471	0.590	0.626	0.564
	SS-Mip-Splatting	0.611	0.479	0.484	0.619	0.625	0.524	0.468	0.581	0.621	0.557
	CompGS	0.634	0.495	0.501	0.652	0.644	0.544	0.480	0.610	0.636	0.577

Appendix

Table S5. Per-scene end-to-end LPIPS on Mip-NeRF 360. Each scene cell averages all test views and all 20 resolution-power settings. Rows without prefix report the original backbone baseline without supersampling. SS rows use the enhanced supersampling baseline. Control rows add learned condition-injection adapters.

Metric	Method	NeRF Synthetic								Avg
Metric	Method	Chair	Drums	Ficus	Hotdog	Lego	Mater.	Mic	Ship	Avg
PSNR $\uparrow$	3DGS	19.86	19.03	23.43	22.95	20.47	22.58	25.14	22.20	21.96
	SS-3DGS	21.54	20.32	24.94	24.46	21.90	23.96	26.51	23.26	23.36
	Mip-Splatting	21.29	20.17	24.66	24.21	21.71	23.85	26.24	23.25	23.17
	SS-Mip-Splatting	21.54	20.32	24.94	24.47	21.90	23.97	26.51	23.28	23.37
	CompGS	19.91	18.68	21.01	22.99	20.43	21.71	25.11	21.94	21.47

Appendix

Table S6. Per-scene end-to-end PSNR and SSIM on NeRF Synthetic. Each scene cell averages all test views and all 20 resolution-power settings. Rows without prefix report the original backbone baseline without supersampling. SS rows use the enhanced supersampling baseline. Control rows add learned condition-injection adapters.

Metric	Method	NeRF Synthetic								Avg
Metric	Method	Chair	Drums	Ficus	Hotdog	Lego	Mater.	Mic	Ship	Avg
LPIPS $\downarrow$	3DGS	0.139	0.188	0.116	0.130	0.203	0.157	0.110	0.283	0.166
	SS-3DGS	0.118	0.161	0.097	0.114	0.175	0.138	0.096	0.265	0.146
	Mip-Splatting	0.124	0.169	0.102	0.120	0.184	0.143	0.101	0.271	0.152
	SS-Mip-Splatting	0.118	0.161	0.097	0.114	0.175	0.138	0.096	0.264	0.145
	CompGS	0.141	0.196	0.135	0.131	0.202	0.179	0.111	0.285	0.173

Appendix

Table S7. Per-scene end-to-end LPIPS on NeRF Synthetic. Each scene cell averages all test views and all 20 resolution-power settings. Rows without prefix report the original backbone baseline without supersampling. SS rows use the enhanced supersampling baseline. Control rows add learned condition-injection adapters.

Fig. S5. Additional qualitative comparisons on Bicycle, Bonsai, and Counter. Additional qualitative comparisons for Bicycle, Bonsai, and Counter using Scaffold-GS, SS-Scaffold-GS, and \projunder two XR run-time settings.

Fig. S6. Additional qualitative comparisons on Flowers, Garden, and Kitchen. Additional qualitative comparisons for Flowers, Garden, and Kitchen using Scaffold-GS, SS-Scaffold-GS, and \projunder two XR run-time settings.

Fig. S7. Additional qualitative comparisons on Room, Treehill, and Chair. Additional qualitative comparisons for Room, Treehill, and Chair using Scaffold-GS, SS-Scaffold-GS, and \projunder two XR run-time settings.

Fig. S8. Additional qualitative comparisons on Drums, Ficus, and Hotdog. Additional qualitative comparisons for Drums, Ficus, and Hotdog using Scaffold-GS, SS-Scaffold-GS, and \projunder two XR run-time settings.

Fig. S9. Additional qualitative comparisons on Lego, Ship, and Mic. Additional qualitative comparisons for Lego, Ship, and Mic using Scaffold-GS, SS-Scaffold-GS, and \projunder two XR run-time settings.

Fig. S10. Additional condition-response visualization on Bicycle and Bonsai. Additional condition-response visualization combining Bicycle and Bonsai scenes.

Fig. S11. Additional condition-response visualization on Counter and Flowers. Additional condition-response visualization combining Counter and Flowers scenes.

Fig. S12. Additional condition-response visualization on Garden and Kitchen. Additional condition-response visualization combining Garden and Kitchen scenes.

Fig. S13. Additional condition-response visualization on Stump and Treehill. Additional condition-response visualization combining Stump and Treehill scenes.

Fig. S14. Additional condition-response visualization on Chair and Drums. Additional condition-response visualization combining Chair and Drums scenes.

Fig. S15. Additional condition-response visualization on Ficus and Hotdog. Additional condition-response visualization combining Ficus and Hotdog scenes.

Fig. S16. Additional condition-response visualization on Materials and Mic. Additional condition-response visualization combining Materials and Mic scenes.

Explore similar work

RoofGS: Roofline-Guided End-to-End Acceleration of 3D Gaussian Splatting
Aug 16, 2026Yang Luo, Yan Gong, Yongsheng Gao +13D Gaussian Splatting 3DGS Acceleration
State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Harbin 150001, China
RGBX-Next: Towards Realistic Generative Rendering from G-Buffers
Aug 14, 2026Zheng Zeng, Marco Salvi, Lifan Wu +9Inverse Rendering Neural Rendering
University of California, Santa Barbara, NVIDIA, USA · NVIDIA, USA · NVIDIA, Czech Republic +2
EffGS: Efficient and High-Fidelity Gaussian Splatting
Sep 30, 2026Changbai Li, Shuo Yang, Yichen Yang +23D Gaussian Splatting 3D Gaussian Pruning
Beihang University · Nanyang Technological University

Aug 16, 2026, cs.CVJ/K move · Enter open · S save

RoofGS: Roofline-Guided End-to-End Acceleration of 3D Gaussian Splatting

Yang Luo, Yan Gong, Yongsheng Gao, Jie Zhao

State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Harbin 150001, China

3D Gaussian Splatting 3DGS Acceleration GPU Kernel Optimization Novel View Synthesis

Open paper AskPDF arXiv

3D Gaussian Splatting (3DGS) enables real-time novel-view synthesis but remains limited on GPUs at high resolutions. Through a stage-wise Roofline characterization, we identify two distinct hardware bottlenecks: global memory traffic dominates the front end, whereas instruction throughput limits rasterization. Guided by this analysis, we develop RoofGS, a rendering framework that applies bottleneck-specific optimizations rather than generic kernel acceleration. For the memory-bound front end, we design a resolution-adaptive quantized depth sorting key that compresses each key to 32 bits. For the compute-bound rasterizer, we introduce a range-aware bit-level fast exponential approximation tailored to the bounded exponent range after opacity culling, with a derived per-pixel error bound. These two core techniques are complemented by additional optimizations (kernel fusion, compact attribute storage, culling, dual-pixel evaluation) that additionally reduce memory traffic and improve instruction-level parallelism. Experiments show that RoofGS achieves a 10.1

\times

end-to-end speedup over 3DGS at 4K on an RTX 4090, increasing throughput from 61 to 616 FPS, with only a 0.028 dB PSNR loss.

Abstract

Explore similar work

RoofGS: Roofline-Guided End-to-End Acceleration of 3D Gaussian Splatting

RGBX-Next: Towards Realistic Generative Rendering from G-Buffers

EffGS: Efficient and High-Fidelity Gaussian Splatting