Localization and Neural Splatting on the Edge: An FPGA Scene Memory for Robots
Authors: Haohua Que, Mingkai Liu, Tianle Zhu, Handong Yao
Organizations: School of Environmental, Civil, Agricultural and Mechanical Engineering, College of Engineering, University of Georgia, Athens, GA 30602, USA
Resource-constrained robots and autonomous vehicles need scene memories that localize a camera and render nearby views without a desktop GPU. Scene-coordinate regression (SCR) and 3D Gaussian splatting (3DGS) fit because they avoid an SfM stage, but ACEsplat-style pipelines are GPU-centered, and moving them to an edge FPGA is hard because SCR is pose-sensitive and 3DGS rendering is irregular and capacity-sensitive. We present LENS, an edge scene-memory acceleration system on the AMD KR260 that brings an ACEsplat-style workload to a low-power FPGA/ARM platform. Rather than proposing a new scene model, LENS co-designs the two dominant kernels with the board data path: an INT16 SCR accelerator that preserves pose accuracy, and a bit-exact fixed-point ordered 3DGS renderer that rewrites alpha transmittance as a log-sum-exp recurrence and removes provably useless streamed Gaussians. All LENS latency, power, and quality results are board-measured. The SCR engine preserves DSAC* localization accuracy at 25.42 ms, the on-board renderer reaches 34.11 dB on Cambridge against ACEsplat's 33.20, novel-view synthesis on RealEstate10K reaches 27.84 dB against 26.11, and all ten Wayspots scenes reach the ACEsplat rendering target. LENS wins performance per watt over a Jetson Orin NX on every deployment task, by 1.9x to 5.7x at about a quarter of the power, and is faster in absolute latency on localization and Wayspots rendering. On-device training reaches ACEsplat-level quality but not the GPU's time or energy, marking the remaining boundary for a future training accelerator.
Figures & tables
Fig. 1: LENS system overview. Posed RGB frames enter the PL SCR engine, which produces scene-coordinate point clouds for CPU pose solving and Gaussian initialization. The ARM CPU performs pose solving and tiling, while the PL LSE render engine rasterizes fixed-point Gaussian streams for reconstruction and novel-view synthesis. All outputs are board-measured on the KR260.
Engine
LUT
FF
BRAM
DSP
Clock
SCR GEMM (2 × -pump)
2126
2044
0
5
400/800 MHz
LSE render array
32010
24063
24
672
300 MHz
XCK26 available
117120
234240
144
1248
TABLE I: Post-route XCK26 utilization (Vivado 2022.2). Both engines meet timing; the render array is the dominant DSP consumer.
Task
Dataset
ACE target
LENS
ACE/Orin NX
LENS lat.
Orin NX lat.
Power (W)
perf/W
Localization
Cambridge
0.42 / 24.9
0.44 / 25.2
0.42 / 25.8
25.4 ms
33.0 ms
3.5 / 14.0
5.2x
Localization
Wayspots rep.
63.8%
65.6%
63.3%
25.4 ms
33.0 ms
3.5 / 14.0
5.2x
Reconstruction
Cambridge
33.20
34.11
34.45
31.9 ms
16.5 ms
4.0 / 14.4
1.9x
Rendering
Wayspots
29.11
31.06
–
10.7 ms
16.5 ms
3.9 / 14.4
5.7x
NVS
RealEstate10K
26.11
27.84
28.21
31.0 ms
16.5 ms
4.0 / 14.4
1.9x
Training
Cambridge
33.20
34.96
35.24
11.3 h
41.5 s
4.0 / 12.6
0.003x
TABLE II: Deployment summary. Quality is PSNR except localization (rot°/trans cm or acc.@10 cm/5°); power is LENS/Orin watts. ACE/Orin NX is matched FP32 on Jetson when available. Training is outside deployment perf/W claims.
Scene
ACEsplat
FP32
LENS
Cambridge: median rot° / trans cm
ShopFacade
0.30 / 5.0
0.27 / 7.1
0.27 / 7.2
KingsCollege
0.40 / 27.8
0.48 / 28.6
0.48 / 32.5
OldHospital
0.56 / 28.9
0.60 / 33.9
0.58 / 28.6
StMarysChurch
0.66 / 20.8
0.59 / 18.4
0.69 / 19.0
GreatCourt
0.19 / 42.2
0.18 / 40.8
0.16 / 38.6
TABLE III: Localization accuracy of the INT16 SCR engine plus DSAC* against the FP32 software reference and ACEsplat. Cambridge: median rotation degrees / translation cm over 30 frames per scene. Wayspots: accuracy at 10 cm/5°. The INT16 board result tracks FP32 on every scene.
Low-rank gate: Cambridge mean localization
Rank ratio
MAC red.
mean (deg / cm)
1.0 (full)
1.00x
0.42 / 25.3
1.4
1.33x
0.43 / 24.9
1.6
1.49x
0.43 / 24.7
1.8
1.63x
0.43 / 26.8
2.2
1.90x
0.52 / 33.2
TABLE IV: SCR accuracy gates. Top: low-rank ablation on the encoder, with MAC reduction versus mean Cambridge localization. Bottom: precision gate for the SCR coordinate field over 103 relocalization frames. The accepted design is the last compression/precision point that preserves pose accuracy.
Static-view crossing protocol: three-view mean
Scene
ACEsplat
LENS
Lat. (ms)
perf/W
ShopFacade
31.80
34.03
32.9
1.81x
KingsCollege
31.88
32.97
33.1
1.79x
OldHospital
31.82
32.33
32.0
1.86x
StMarysChurch
33.92
34.46
33.7
1.76x
GreatCourt
36.57
36.75
27.7
2.14x
TABLE V: Cambridge static-view reconstruction. Top: KR260 LSE renderer, three-view mean at the crossing point; perf/W is relative to Orin NX. Bottom: reference-view PSNR check, with GPU/Orin NX in FP32 and LENS in integer LSE.
Fig. 2: Qualitative comparison on Cambridge Landmarks. Top: full renders (columns: ground truth, FP32 GPU, Jetson Orin NX, LENS integer render); the Orin NX render is produced on the Jetson itself and the LENS column is bit-exact to the integer emulator, with PSNR overlaid. Bottom: detail crops of the same region across the four columns, where the LENS render holds the fine structure of the FP32 and Orin NX renders.
Scene
LENS
Scene
LENS
Scene
LENS
bears
29.52
squarebench
30.07
cubes †
30.18
inscription
29.54
tendrils
29.27
lawn †
30.08
map
29.64
therock
30.18
statue †
35.02
wintersign †
37.11
mean 31.06 dB
TABLE VI: Wayspots reconstruction rendering PSNR (dB), board-measured and bit-exact to the emulator. Grid-init scenes are marked † .
Fig. 3: Qualitative reconstruction rendering on all ten Wayspots scenes. Each scene shows the ground truth and the LENS integer render (bit-exact to the emulator), with the board PSNR overlaid. Every scene exceeds the 29.11 dB ACEsplat average.
Scene
LENS
Scene
LENS
Scene
LENS
s0
22.80
s5
30.13
s10
26.53
s1
24.62
s6
28.46
s11
32.49
s2
24.09
s7
31.63
s12
26.08
s3
30.00
s8
26.23
s13
20.86
s4
31.87
s9
32.16
s14
29.67
mean 27.84 dB
TABLE VII: RealEstate10K two-view NVS PSNR (dB), board-measured and bit-exact to the emulator.
Fig. 4: Two-view novel-view synthesis on RealEstate10K. Columns: ground truth, FP32 GPU render, Jetson Orin NX render, LENS integer render.
Fig. 5: Analysis. (a) Integer precision recipe on ShopFacade; the 4096-entry covariance LUT is the dominant lever and the final configuration (star) clears the ACEsplat PSNR. (b) Stream compaction removes 24 to 60 percent of streamed Gaussians per scene with no change to any output bit. (c) On-device training crosses the ACEsplat target at iteration 240 (star) and reaches 34.96 dB, board-measured. (d) Performance per watt relative to the Jetson Orin NX across deployment tasks; bars above the baseline favor LENS.
Fig. 6: Analysis line plots, board-measured. (a) SCR-forward latency across stacked optimizations (full-rank RTL, then double-pumping, low rank, and both) falls from 55.3 to 25.4 ms, below the Jetson Orin NX (33 ms, dashed). (b) Render latency versus the fraction of streamed Gaussians the compaction removes, per Cambridge scene; GreatCourt sheds 60 percent and is the fastest. (c) On-device training boundary: a float forward and backward converges and crosses the ACEsplat PSNR at iteration 240, while differentiating the quantized render plateaus at 21.6 dB and a straight-through estimator plateaus at 17.3 dB.
Per-scene 3D Gaussian Splatting (3DGS) enables high-fidelity rendering, but practical robotic and AR scene capture pipelines often depend on external geometric initialization (e.g., SfM point clouds or depth estimates), which can be slow and brittle in on-site deployment. We present ACEsplat, a fast per-scene optimization framework that reconstructs 3D Gaussian representations from RGB images and camera poses only, without requiring external 3D priors (e.g., precomputed SfM models or supervised depth maps). ACEsplat uses a two-stage pipeline: (1) a self-supervised scene coordinate regression (SCR) module builds an internal geometry prior within 4--5 minutes; (2) SCR features and coordinate priors are fused by a lightweight Gaussian initialization head, followed by per-scene 3DGS optimization. On static-view rendering, ACEsplat achieves 29.11 dB PSNR on Wayspots with real-time SLAM poses and 33.20 dB on Cambridge Landmarks with SfM-refined poses. On RealEstate10K sparse-view novel view synthesis, it achieves competitive image fidelity under a challenging 2-view setting. ACEsplat completes scene-specific SCR mapping and 3DGS reconstruction within 15--25 minutes on a single GPU, making it a practical RGB+pose-only solution for rapid scene setup in robotics and mixed-reality applications.
Mingkai Liu, Haohua Que, Dikai Fan +7
Peking University · University of Georgia · Infinity Robotics +3
Novel view synthesis (NVS) is increasingly relevant for edge robotics, where compact and incrementally updatable 3D scene models are needed for SLAM, navigation, and inspection under tight memory and latency budgets. Variational Bayesian Gaussian Splatting (VBGS) enables replay-free continual updates for the 3DGS algorithm by maintaining a probabilistic scene model, but its high-precision computations and large intermediate tensors make on-device training impractical. We present a precision-adaptive optimization framework that enables VBGS training on resource-constrained hardware without altering its variational formulation. We (i) profile VBGS to identify memory/latency hotspots, (ii) fuse memory-dominant kernels to reduce materialized intermediate tensors, and (iii) automatically assign operation-level precisions via a mixed-precision search with bounded relative error. Across the Blender, Habitat, and Replica datasets, our optimised pipeline reduces peak memory from 9.44 GB to 1.11 GB and training time from ~234 min to ~61 min on an A5000 GPU, while preserving (and in some cases improving) reconstruction quality of the state-of-the-art VBGS baseline. We also enable for the first time NVS training on a commercial embedded platform, the Jetson Orin Nano, reducing per-frame latency by 19x compared to 3DGS.
Ivan Zaino, Matteo Risso, Daniele Jahier Pagliari +3
Feed-forward 3D Gaussian Splatting (FF3DGS) reconstructs 3D scenes from images in a single forward pass, making it well-suited to applications that must operate within limited resource budgets. To further improve its practicality, recent budget-controllable FF3DGS methods allow the number of Gaussians to be adjusted at inference, so that the output meets a given storage limit or rendering-latency constraint. However, these methods overlook the cost of inference itself: to produce a representation with the target number of Gaussians, they first predict a dense set of Gaussians and then prune or select from it, requiring nearly constant latency and peak memory regardless of the budget. We therefore propose RLSplat, a resource-linear budget-controllable FF3DGS framework whose representation size, inference latency, and peak memory all decrease linearly as the Gaussian budget is reduced. Instead of pruning or selection, RLSplat learns a Gaussian allocation policy that distributes the budget across local scene regions, so that exactly the requested number of Gaussians is generated. These Gaussians are then refined through resource-linear Gaussian decoding, in which each Gaussian attends only to nearby Gaussians found via an efficient hierarchical neighbor search, so that the decoding cost decreases linearly with fewer Gaussians. In zero-shot evaluation, RLSplat outperforms prior budget-controllable methods at every budget, surpassing the strongest baseline even with 4× fewer Gaussians, while running up to 4.1× faster with 9.6× less peak memory at low budgets.