Feed-forward 3D Gaussian Splatting (FF3DGS) reconstructs 3D scenes from images in a single forward pass, making it well-suited to applications that must operate within limited resource budgets. To further improve its practicality, recent budget-controllable FF3DGS methods allow the number of Gaussians to be adjusted at inference, so that the output meets a given storage limit or rendering-latency constraint. However, these methods overlook the cost of inference itself: to produce a representation with the target number of Gaussians, they first predict a dense set of Gaussians and then prune or select from it, requiring nearly constant latency and peak memory regardless of the budget. We therefore propose RLSplat, a resource-linear budget-controllable FF3DGS framework whose representation size, inference latency, and peak memory all decrease linearly as the Gaussian budget is reduced. Instead of pruning or selection, RLSplat learns a Gaussian allocation policy that distributes the budget across local scene regions, so that exactly the requested number of Gaussians is generated. These Gaussians are then refined through resource-linear Gaussian decoding, in which each Gaussian attends only to nearby Gaussians found via an efficient hierarchical neighbor search, so that the decoding cost decreases linearly with fewer Gaussians. In zero-shot evaluation, RLSplat outperforms prior budget-controllable methods at every budget, surpassing the strongest baseline even with 4× fewer Gaussians, while running up to 4.1× faster with 9.6× less peak memory at low budgets.
Figures & tables
Figure 1: Budget-controllable quality, memory, and latency trade-offs. (a) RLSplat achieves the highest PSNR across the evaluated total Gaussian budgets B . (b) At the 2K budget, RLSplat uses 9.6× less peak VRAM than F 4 Splat while attaining higher PSNR. (c) As B is reduced, RLSplat’s inference latency decreases approximately linearly, whereas the latencies of F 4 Splat and EcoSplat remain nearly constant. In axis tick labels, 1K=1,024 .
Figure 2: Overview of RLSplat. Given uncalibrated context images, the geometry backbone predicts camera parameters and extracts anchor tokens {an} representing local 3D regions. The Gaussian allocation policy distributes the budget B across anchors, assigning kn Gaussians to each anchor with ∑kn=B . Each anchor then spawns kn Gaussian tokens, which are decoded into B Gaussians and locally refined using hierarchical neighbor search.
Figure 3: Policy optimization.
Method
8 views
16 views
24 views
#GS ↓
LPIPS ↓
SSIM ↑
PSNR ↑
#GS ↓
LPIPS ↓
SSIM ↑
PSNR ↑
#GS ↓
LPIPS ↓
SSIM ↑
PSNR ↑
EcoSplat
32K
0.300
0.669
23.29
64K
0.282
0.689
23.86
96K
0.284
0.688
23.81
F 4 Splat
32K
0.240
0.718
24.50
64K
0.218
0.743
25.09
96K
0.211
0.753
25.31
RLSplat
32K
0.211
0.756
25.24
64K
0.196
0.778
25.75
96K
0.192
0.787
25.94
EcoSplat
16K
0.357
0.621
21.63
32K
0.324
0.654
22.63
48K
0.320
0.660
22.73
F 4 Splat
16K
0.269
0.694
24.12
32K
0.242
0.720
24.72
48K
0.233
0.731
24.95
Table 1: Budget-controllable methods on ACID (zero-shot from RE10K). Each method is evaluated at the same requested budget of 4K, 2K, and 1K Gaussians per context view; #GS is the resulting total per scene (1K = 1024). Best per column within each budget in bold.
Method
8 views
16 views
24 views
#GS ↓
LPIPS ↓
SSIM ↑
PSNR ↑
#GS ↓
LPIPS ↓
SSIM ↑
PSNR ↑
#GS ↓
LPIPS ↓
SSIM ↑
PSNR ↑
NoPoSplat
512K
0.268
0.672
22.84
1024K
0.294
0.644
22.14
1536K
0.307
0.630
21.78
SPFSplat
512K
0.215
0.736
24.89
1024K
0.214
0.743
25.07
1536K
0.218
0.744
25.00
AnySplat
470K
0.248
0.696
23.30
885K
0.236
0.720
23.88
1259K
0.234
0.727
24.04
RLSplat
32K
0.211
0.756
25.24
32K
0.205
0.773
25.63
24K
0.215
0.771
25.58
Table 2: Budget-fixed methods on ACID (zero-shot from RE10K). These methods emit a fixed, dense set of Gaussians determined by the input. RLSplat outperforms them on all metrics while using substantially fewer Gaussians. Best per column in bold.
Figure 4: Qualitative comparison under the same Gaussian budget. All methods use 8K Gaussians on RE10K. For each method, we show the rendered view and PSNR, a zoomed region and its local PSNR, the Gaussian centers in the highlighted region (red), and a 16×16 tile-wise map of the number of Gaussians. RLSplat places more Gaussians around the thin stair balusters (5,388 vs. 3,026 for F 4 Splat and 2,521 for EcoSplat), preserving the fine structure that the baselines blur.
Variant
LPIPS ↓
SSIM ↑
PSNR ↑
w/o Loffset
0.394
0.630
22.31
w/o Loffset , Lvor
0.479
0.547
20.33
w/o Lproj †
0.731
0.007
5.80
w/o local refinement
0.425
0.607
21.88
Full model
0.353
0.643
22.84
Table 3: Architecture and loss ablations. † Without the projection loss, the model fails to reconstruct the scene.
Variant
LPIPS ↓
SSIM ↑
PSNR ↑
Base †
0.277
0.708
24.18
Uniform
0.271
0.720
24.53
Edge-based
0.268
0.727
24.76
Policy
0.253
0.734
24.83
Table 4: Allocation strategy ablation. † Base model trained with random allocations, before fine-tuning.
Figure 9Table 10
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
C
τ
LPIPS ↓
SSIM ↑
PSNR ↑
5
1.0
0.268
0.722
24.68
3
1.0
0.273
0.717
24.57
7
1.0
0.269
0.720
24.63
5
0.8
0.271
0.718
24.60
5
1.2
0.270
0.719
24.61
Appendix
Table 6: GSPO hyperparameters. Group size C and rollout temperature τ . The first row is our default.
Figure 8: More allocation policies (1/3). Raw (left pair) and smoothed (right pair) allocation maps on a log scale ( 0.25× to 4× ; red: more, blue: fewer). In these indoor scenes the policy shifts budget toward the view with more structure and away from near-blank walls and doorways.
Figure 9: More allocation policies (2/3). Same layout as Fig. 8 . Budget moves away from dark or under-exposed regions, such as the arch frame and the darkened hillside, and concentrates on the lit part of the scene or on the object.
Figure 10: More allocation policies (3/3). Same layout as Fig. 8 . When the two views are nearly identical, the budget stays close to an even split between them, and the policy acts within each view instead: it empties water and sky and spends the budget on buildings, vegetation and shorelines.
Figure 11: Qualitative comparison across Gaussian budgets (1/3).
Figure 12: Qualitative comparison across Gaussian budgets (2/3).
Figure 13: Qualitative comparison across Gaussian budgets (3/3).
3D Gaussian Splatting achieves excellent visual quality with real-time rendering, but at the scale of entire cities it does not fit: a trained model carries millions of primitives and gigabytes of memory, and real-time rendering at high quality on a consumer GPU remains out of reach. We introduce Budgeted-GS, a post-hoc method that turns any trained 3DGS model into a factoring tree, a multi-resolution hierarchy of moment-matched aggregates. After a construction pass of a few seconds, a single quality parameter selects, for each view, the level of detail that fits the memory of the target device, so the same city-scale model serves GPUs with widely different memory capacities. When a new scene is to be trained, the same theory applies: instead of growing a full-sized model and compressing it afterwards, budget-centered training first measures how many primitives the scene needs and then trains the model directly at that size, avoiding the wasted effort of optimizing primitives that are later discarded. Both methods are grounded in a measurable capacity floor, a budget-error law derived from optimal transport in phase space; selection rules certified by recent covering theorems decide which primitives are redundant. The floor answers how many primitives a scene actually needs and how many can safely be given up. We validate the floor on 13 public scenes under a preregistered protocol, and exercise both methods from object scenes to an official city capture, rendering it at native 1920x1080, full SH, in real time on one consumer GPU.
The ability to render scenes at adjustable fidelity from a single model, known as level of detail (LoD), is crucial for practical deployment of 3D Gaussian Splatting (3DGS). Existing discrete LoD methods expose only a limited set of operating points, while concurrent continuous LoD approaches enable smoother scaling but often suffer noticeable quality degradation at full capacity, making LoD a costly design decision. We introduce Matryoshka Gaussian Splatting (MGS), a training framework that enables continuous LoD for standard 3DGS pipelines without sacrificing full-capacity rendering quality. MGS learns a single ordered set of Gaussians such that rendering any prefix, the first k splats, produces a coherent reconstruction whose fidelity improves smoothly with increasing budget. Our key idea is stochastic budget training: each iteration samples a random splat budget and optimises both the corresponding prefix and the full set. This strategy requires only two forward passes and introduces no architectural modifications. Experiments across four benchmarks and six baselines show that MGS matches the full-capacity performance of its backbone while enabling a continuous speed-quality trade-off from a single model. Extensive ablations on ordering strategies, training objectives, and model capacity further validate the designs.
Zhilin Guo, Boqiao Zhang, Hakan Aktas +11
University of Cambridge · Chalmers University of Technology · Google
Scalable 3D Gaussian Splatting allows one scene representation to serve different Gaussian-count budgets, yet its quality-budget curve may lag behind models optimized separately for each budget. Reducing the number of Gaussians changes both the selected members and the reconstruction tasks carried by the survivors; parameters inherited from a high-budget model may fit a low-budget representation poorly. We propose ParetoGS, which describes member changes and parameter evolution with a small set of adapted budget anchors and decodes intermediate budgets without separate optimization at every requested count. ParetoGS alternates pruning and fixed-member adaptation to build nested anchors with persistent identities, and encodes their parameter changes relative to the full model in a compact residual atlas. At decoding, it interpolates the states of shared members and gradually introduces the rendering contributions of new members, connecting sparse anchors into a continuous-budget representation. Across 13 scenes, ParetoGS attains better average PSNR, SSIM, and LPIPS with fewer Gaussians than the compared prefix methods at 10% of each method's own full model. From a shared full model, it also outperforms FlexGS-imp at its trained 1%, 5%, and 10% budgets and NanoGS-imp at the 1% and 0.1% budgets. On Garden, its non-anchor quality approaches an empirical reference from independently optimized GNS models at higher retention ratios and exceeds it at some lower-budget points. Across 13 scenes, encoding reduces mean additional state from 390.0 to 85.7 MB with an average PSNR change of 0.01 dB. Project page: https://xiaobin2001.github.io/ParetoGS-Web/