RLSplat: Resource-Linear Budget Control for Feed-forward 3D Gaussian Splatting

Injae Kim; Minseok Joo; Jihwan Park; Sehyung Kim; Minseong Bae; Hyunwoo J. Kim

cs.CVOct 20, 2026

RLSplat: Resource-Linear Budget Control for Feed-forward 3D Gaussian Splatting

Authors: Injae Kim, Minseok Joo, Jihwan Park, Sehyung Kim, Minseong Bae, Hyunwoo J. Kim

Organizations: KAIST · Korea University

Abstract

Feed-forward 3D Gaussian Splatting (FF3DGS) reconstructs 3D scenes from images in a single forward pass, making it well-suited to applications that must operate within limited resource budgets. To further improve its practicality, recent budget-controllable FF3DGS methods allow the number of Gaussians to be adjusted at inference, so that the output meets a given storage limit or rendering-latency constraint. However, these methods overlook the cost of inference itself: to produce a representation with the target number of Gaussians, they first predict a dense set of Gaussians and then prune or select from it, requiring nearly constant latency and peak memory regardless of the budget. We therefore propose RLSplat, a resource-linear budget-controllable FF3DGS framework whose representation size, inference latency, and peak memory all decrease linearly as the Gaussian budget is reduced. Instead of pruning or selection, RLSplat learns a Gaussian allocation policy that distributes the budget across local scene regions, so that exactly the requested number of Gaussians is generated. These Gaussians are then refined through resource-linear Gaussian decoding, in which each Gaussian attends only to nearby Gaussians found via an efficient hierarchical neighbor search, so that the decoding cost decreases linearly with fewer Gaussians. In zero-shot evaluation, RLSplat outperforms prior budget-controllable methods at every budget, surpassing the strongest baseline even with 4 $\times$ fewer Gaussians, while running up to 4.1 $\times$ faster with 9.6 $\times$ less peak memory at low budgets.

Figures & tables

Figure 1: Budget-controllable quality, memory, and latency trade-offs. (a) RLSplat achieves the highest PSNR across the evaluated total Gaussian budgets $B$ . (b) At the 2K budget, RLSplat uses $9.6\times$ less peak VRAM than F 4 Splat while attaining higher PSNR. (c) As $B$ is reduced, RLSplat’s inference latency decreases approximately linearly, whereas the latencies of F 4 Splat and EcoSplat remain nearly constant. In axis tick labels, $1\mathrm{K}=1{,}024$ .

Figure 2: Overview of RLSplat. Given uncalibrated context images, the geometry backbone predicts camera parameters and extracts anchor tokens $\{\mathbf{a}_{n}\}$ representing local 3D regions. The Gaussian allocation policy distributes the budget $B$ across anchors, assigning $k_{n}$ Gaussians to each anchor with $\sum k_{n}=B$ . Each anchor then spawns $k_{n}$ Gaussian tokens, which are decoded into $B$ Gaussians and locally refined using hierarchical neighbor search.

Method	8 views				16 views				24 views
Method	#GS $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	#GS $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	#GS $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
EcoSplat	32K	0.300	0.669	23.29	64K	0.282	0.689	23.86	96K	0.284	0.688	23.81
F 4 Splat	32K	0.240	0.718	24.50	64K	0.218	0.743	25.09	96K	0.211	0.753	25.31
RLSplat	32K	0.211	0.756	25.24	64K	0.196	0.778	25.75	96K	0.192	0.787	25.94
EcoSplat	16K	0.357	0.621	21.63	32K	0.324	0.654	22.63	48K	0.320	0.660	22.73
F 4 Splat	16K	0.269	0.694	24.12	32K	0.242	0.720	24.72	48K	0.233	0.731	24.95

Table 1: Budget-controllable methods on ACID (zero-shot from RE10K). Each method is evaluated at the same requested budget of 4K, 2K, and 1K Gaussians per context view; #GS is the resulting total per scene (1K

=

1024). Best per column within each budget in bold.

Method	8 views				16 views				24 views
Method	#GS $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	#GS $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	#GS $\downarrow$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
NoPoSplat	512K	0.268	0.672	22.84	1024K	0.294	0.644	22.14	1536K	0.307	0.630	21.78
SPFSplat	512K	0.215	0.736	24.89	1024K	0.214	0.743	25.07	1536K	0.218	0.744	25.00
AnySplat	470K	0.248	0.696	23.30	885K	0.236	0.720	23.88	1259K	0.234	0.727	24.04
RLSplat	32K	0.211	0.756	25.24	32K	0.205	0.773	25.63	24K	0.215	0.771	25.58

Table 2: Budget-fixed methods on ACID (zero-shot from RE10K). These methods emit a fixed, dense set of Gaussians determined by the input. RLSplat outperforms them on all metrics while using substantially fewer Gaussians. Best per column in bold.

Figure 4: Qualitative comparison under the same Gaussian budget. All methods use 8K Gaussians on RE10K. For each method, we show the rendered view and PSNR, a zoomed region and its local PSNR, the Gaussian centers in the highlighted region (red), and a $16{\times}16$ tile-wise map of the number of Gaussians. RLSplat places more Gaussians around the thin stair balusters (5,388 vs. 3,026 for F 4 Splat and 2,521 for EcoSplat), preserving the fine structure that the baselines blur.

Variant	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
w/o $\mathcal{L}_{\mathrm{offset}}$	0.394	0.630	22.31
w/o $\mathcal{L}_{\mathrm{offset}}$ , $\mathcal{L}_{\mathrm{vor}}$	0.479	0.547	20.33
w/o $\mathcal{L}_{\mathrm{proj}}$ †	0.731	0.007	5.80
w/o local refinement	0.425	0.607	21.88
Full model	0.353	0.643	22.84

Table 3: Architecture and loss ablations. † Without the projection loss, the model fails to reconstruct the scene.

Variant	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
Base †	0.277	0.708	24.18
Uniform	0.271	0.720	24.53
Edge-based	0.268	0.727	24.76
Policy	0.253	0.734	24.83

Table 4: Allocation strategy ablation. † Base model trained with random allocations, before fine-tuning.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

$C$	$\tau$	LPIPS $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$
5	1.0	0.268	0.722	24.68
3	1.0	0.273	0.717	24.57
7	1.0	0.269	0.720	24.63
5	0.8	0.271	0.718	24.60
5	1.2	0.270	0.719	24.61

Appendix

Table 6: GSPO hyperparameters. Group size

C

and rollout temperature

\tau

. The first row is our default.

Figure 8: More allocation policies (1/3). Raw (left pair) and smoothed (right pair) allocation maps on a log scale ( $0.25\times$ to $4\times$ ; red: more, blue: fewer). In these indoor scenes the policy shifts budget toward the view with more structure and away from near-blank walls and doorways.

Figure 9: More allocation policies (2/3). Same layout as Fig. 8 . Budget moves away from dark or under-exposed regions, such as the arch frame and the darkened hillside, and concentrates on the lit part of the scene or on the object.

Figure 10: More allocation policies (3/3). Same layout as Fig. 8 . When the two views are nearly identical, the budget stays close to an even split between them, and the policy acts within each view instead: it empties water and sky and spends the budget on buildings, vegetation and shorelines.

Figure 11: Qualitative comparison across Gaussian budgets (1/3).

Figure 12: Qualitative comparison across Gaussian budgets (2/3).

Figure 13: Qualitative comparison across Gaussian budgets (3/3).

3D Gaussian Splatting

3D Gaussian Splatting represents scenes as optimizable 3D Gaussian primitives for fast, photorealistic rendering from new viewpoints.

See all 1,019 papers

Budgeted-GS: Real-Time Large-Scale Gaussian Splatting via Factoring LOD
Oct 2, 2026Haipeng Wang3DGS Training 3D Gaussian Splatting
Neusoft
Matryoshka Gaussian Splatting
Mar 19, 2026Zhilin Guo, Boqiao Zhang, Hakan Aktas +113D Gaussian Splatting 3DGS Training
University of Cambridge · Chalmers University of Technology · Google
ParetoGS: High-Quality Continuous-Budget 3D Gaussian Splatting with Sparse Adapted Anchors
Oct 25, 2026Xiaobin Deng, Changyu Diao, Min Li +13D Gaussian Splatting 3DGS Compression
Zhejiang University

Oct 2, 2026J/K move · Enter open · S save

Budgeted-GS: Real-Time Large-Scale Gaussian Splatting via Factoring LOD

Haipeng Wang

Neusoft

3DGS Training 3D Gaussian Splatting 3DGS Acceleration 3DGS Compression

Open paper AskPDF arXiv

3D Gaussian Splatting achieves excellent visual quality with real-time rendering, but at the scale of entire cities it does not fit: a trained model carries millions of primitives and gigabytes of memory, and real-time rendering at high quality on a consumer GPU remains out of reach. We introduce Budgeted-GS, a post-hoc method that turns any trained 3DGS model into a factoring tree, a multi-resolution hierarchy of moment-matched aggregates. After a construction pass of a few seconds, a single quality parameter selects, for each view, the level of detail that fits the memory of the target device, so the same city-scale model serves GPUs with widely different memory capacities. When a new scene is to be trained, the same theory applies: instead of growing a full-sized model and compressing it afterwards, budget-centered training first measures how many primitives the scene needs and then trains the model directly at that size, avoiding the wasted effort of optimizing primitives that are later discarded. Both methods are grounded in a measurable capacity floor, a budget-error law derived from optimal transport in phase space; selection rules certified by recent covering theorems decide which primitives are redundant. The floor answers how many primitives a scene actually needs and how many can safely be given up. We validate the floor on 13 public scenes under a preregistered protocol, and exercise both methods from object scenes to an official city capture, rendering it at native 1920x1080, full SH, in real time on one consumer GPU.

Abstract

3D Gaussian Splatting

Explore similar work

Budgeted-GS: Real-Time Large-Scale Gaussian Splatting via Factoring LOD

Matryoshka Gaussian Splatting

ParetoGS: High-Quality Continuous-Budget 3D Gaussian Splatting with Sparse Adapted Anchors