Memory-efficient GPU pipelines for real-time non-line-of-sight reconstruction

Alfonso López-Ruiz; Diego Royo

cs.DCAug 28, 2026

Memory-efficient GPU pipelines for real-time non-line-of-sight reconstruction

Authors: Alfonso López-Ruiz, Diego Royo

Organizations: Universidad de Zaragoza–I3A, Zaragoza, Spain

Abstract

Non-line-of-sight (NLOS) imaging reconstructs scenes hidden around a corner from indirect light recorded by a single-photon avalanche diode (SPAD). A single reconstruction is a large inverse problem: billions of photon timestamps must be binned, moved through memory, transformed and inverted. As SPAD arrays raise acquisition throughput, reconstruction becomes the limiting stage. We rebuild the GPU execution of two established wave-based algorithms, f-k migration and phasor-fields, for both streaming and offline processing. On the phasor-fields side we assemble the ring-and-radius kernels of previous work once and offline, using the analytic Fourier transform of a ring, so the propagation kernel never exists in dense form at runtime, reducing the memory and bandwidth. We reorganize the pipeline of both algorithms with fused kernels, warp-level photon binning, batched transforms, CUDA graph replay, and FP16 storage applied only where it reduces the actual bottleneck. Our implementations are up to 42x faster than the reference streaming pipeline and up to 14x faster than the fastest published GPU baseline, all while using a fraction of the memory (down to 2.5%), enabling vastly larger and finer reconstructions on the same hardware, or comparable ones within a much lower memory budget. We report an ablation of each implementation choice and propose three denoising strategies enabled by the resulting frame budget for next-generation NLOS video processing.

Figures & tables

Figure 1: Performance overview on the office scene, reconstructed at two volume sizes: a dense $720\times 520\times 512$ , and a coarser $360\times 260\times 512$ which therefore shows fewer details in the reconstruction. For each, the left panel reports reconstruction time per frame and the right panel peak video memory, with the methods grouped into three families: backprojection ( Sun et al., 2026 ) , $f\text{--}k$ migration, and phasor-fields ( Nam et al., 2021 ; Mu et al., 2025 ) . Hatched bars are previous work and solid bars are ours. Backprojection is three to four orders of magnitude slower than the rest, so its bar runs off the top of the time panel. Against PttR, the fastest published GPU baseline, our $f\text{--}k$ is $2.7\times$ faster at the coarser volume and our phasor-fields about $14\times$ , in a fraction of the memory; at the dense volume neither phasor-fields baseline fits in the $16$ $\mathrm{GB}$ of our GPU at all ( OOM ).

Compared work	Model	Comparison use	Main limitation for comparison
Fast backprojection ( Arellano et al., 2017 )	Backprojection	Rerun (partially modified)	Different reconstruction model; costly for many ellipsoids
CUDA irregular-relay NLOS ( Sun et al., 2026 )	Backprojection	Rerun	Different reconstruction model
NLOS at 5 FPS ( Nam et al., 2021 )	Phasor-fields	Rerun	Real-time but low-resolution setting
FPGA-accelerated NLOS ( Liao et al., 2022 )	Phasor-fields	Reported only	Specialized hardware; code unavailable
Physics to the Rescue ( Mu et al., 2025 )	Phasor-fields / learned	Rerun	Highest FPS reported for single-depth reconstruction
$f\text{--}k$ migration ( Lindell et al., 2019 )	$f\text{--}k$ migration	Offline reference	Original offline implementation

Table 1: Previous NLOS imaging methods considered in our evaluation. We distinguish between baselines rerun in our hardware and methods included as reported literature values. Partially modified denotes code changes required to run the method on our data, mainly for loading HDF5 capture datasets. Every method is run with its own published reconstruction operator.

Figure 2: Illustration of the capture setup and the collected data. An ultrafast laser emits a pulse onto the visible relay wall, from which light scatters toward the hidden scene. The scattered light then returns to the relay wall and is captured by the SPAD array as third-bounce photons. The recorded measurements consist of individual photon arrival events, which can be aggregated into transient histograms or directly accumulated into temporal-frequency phasors for wave-based reconstruction methods such as phasor-fields.

Figure 3: Overview of our producer-consumer pipeline. Labels P and C indicate whether a module is a producer, a consumer, or both.

Figure 4: Histogram construction time for DMA and GPU-resident buffers using different photon counts. By default, histograms are built with 1.2M photons, matching the binary datasets of Nam et al. (2021) ; the 20M and 100M cases are generated by uniformly oversampling this baseline.

Figure 5: Two scenes reconstructed with and without padding using $f\text{--}k$ . The reconstruction on the left side does not show artifacts without padding, whereas the right one does; however, the hidden scene remains recognizable and the versions without padding were computed at higher speed.

Figure 6: Performance and video memory footprint of

f\text{--}k

with different padding and precision configurations, measured on a confocal dataset shaped as

256\times 256\times 256

Figure 8: Mean radial magnitude of the dense phasor-fields Fourier-domain kernel and of our radial approximation, averaged over temporal frequencies and depths. The comparison is one of shape rather than of exact values.

Figure 10: Denoising results from DDA on a dynamic capture, for temporal windows of $N_{w}$ frames. Each row is one frame of the sequence, with the corresponding pose rendered at the left; the leftmost column reconstructs that frame without denoising. All panels of a row show the same frame, and the inset magnifies the same region in every panel.

Figure 11: Merging reconstructed frames in the complex domain on a simulated scene. Weighting the coherent mean by $C_{i}(\mathbf{x}_{v})$ attenuates the incoherent background speckle while preserving the hidden surface, whose phase is consistent across the window.

Figure 12: Merging phasor-fields and $f\text{--}k$ migration with MoE. The inset magnifies the same region through a display tone curve ( $\gamma=2.2$ ).

Panel of Figure 13 — Figure 13: Ablation of implementation decisions: (a) the photon-binning path, (b) the shared-memory Stolt-remapping variant of $f\text{--}k$ migration, and (c) replaying a captured CUDA graph instead of launching each operation separately.

Approach	Mvox/s $\uparrow$	Time (s) $\downarrow$	VRAM (GB) $\downarrow$
RTX 4080 SUPER - Z : $256^{2}\times 512$ , $255$ frequencies, $30$ depths
Our $f\text{--}k$	4,846.80	$0.0069\pm 0.0000$	0.750
Our phasor-fields	9,577.03	$0.0035\pm 0.0000$	0.293
Phasor fields ( Nam et al., 2021 )	521.67	$0.0643\pm 0.0042$	4.389
Phasor fields ( Mu et al., 2025 )	1,242.38	$0.0270\pm 0.0023$	8.851
CUDA backprojection	1.21	$26.593\pm 0.377$	0.366

Table 3: Performance of fast reconstruction algorithms. Throughput is the processed data volume divided by reconstruction time: for the wave-based methods this is the input transient cube of each block header, of which the phasor-fields family propagates the listed frequency and depth counts, while backprojection uses its reconstructed volume,

256\times 256\times 491

voxels on Z and

360\times 260\times 512

on office. The frequency counts are chosen from the virtual wavelength and capped at the Nyquist limit of the temporal sampling. Green and red cells mark the best and worst value per metric within each dataset.

			Peak GPU memory usage ( $\mathrm{GB}$ )
Volume	$\Omega_{\mathrm{PF}}$	$d_{\mathrm{PF}}$	$f\text{--}k$	Phasor fields	MoE
$90\times 65\times 128$	29	13	0.031	0.005	0.035
$180\times 130\times 256$	58	26	0.188	0.026	0.234
$360\times 260\times 512$	115	51	1.496	0.208	1.872
$720\times 520\times 1024$	229	103	11.960	1.658	14.976

Table 4: Peak GPU memory usage for our reconstruction methods without padding .

\Omega_{\mathrm{PF}}

and

d_{\mathrm{PF}}

are the number of temporal frequencies and propagated depths used by the phasor-fields runs. Missing entries are reported as OOM . Precision:

f\text{--}k

in FP32 ; phasor fields and MoE in FP16 .

			Peak GPU memory usage ( $\mathrm{GB}$ )
Volume	$\Omega_{\mathrm{PF}}$	$d_{\mathrm{PF}}$	$f\text{--}k$	Phasor fields	MoE
$90\times 65\times 128$	29	13	0.133	0.009	0.143
$180\times 130\times 256$	58	26	1.064	0.072	1.147
$360\times 260\times 512$	115	51	8.477	0.567	9.170
$720\times 520\times 1024$	229	103	oom	4.530	oom

Table 5: Peak GPU memory usage for our reconstruction methods with padding .

\Omega_{\mathrm{PF}}

and

d_{\mathrm{PF}}

are the number of temporal frequencies and propagated depths used by the phasor-fields runs. Missing entries are reported as OOM . Precision:

f\text{--}k

in FP32 ; phasor fields and MoE in FP16 .

Figure 14: Stage-time decomposition for the main reconstruction pipelines.

Figure 15: Frame time and peak GPU memory for FP16 and FP32 configurations.

Panel of Figure 16 — Figure 16: Equivalence with reference implementation on the dynamic capture of Nam et al. (2021) . The sequence plot reports the mean absolute difference over time, while the frame panel shows the reference image, our reconstruction, FLIP metric and absolute difference. The reference for phasor-fields-based methods is Nam et al. (2021) ’s reconstruction; for $f\text{--}k$ , the reference is $f\text{--}k$ using FP32 . The two phasor-fields curves overlap.

Figure 17: The regions used in Table 6 . (a) The brightest 5% of pixels form the object (red contour), and the outer border forms the background (blue band). Their ratio is the contrast . (b) Sharpness is the mean gradient magnitude inside the same object contour.

Method	Padding	$\uparrow$ Object	$\downarrow$ Background	$\uparrow$ Contrast	$\uparrow$ Sharpness
Our $f\text{--}k$	no	0.615	0.0379	16.2	0.043
Our $f\text{--}k$	yes	0.656	0.0475	13.8	0.037
Our phasor-fields	no	0.405	0.0115	35.1	0.056
Our phasor-fields	yes	0.410	0.0079	51.6	0.056
Our MoE ( $\beta=0.3$ )	no	0.449	$7.4\cdot 10^{-6}$	60,981	0.053
Our MoE ( $\beta=0.3$ )	yes	0.479	0.0009	512	0.052

Table 6: Background suppression on nlosbox1 , in FP32 . Each method is reported with and without padding. Green and bold mark the best value in each column. The sweep over the pooling weight

\beta

is reported in the supplementary material.

Panel of Figure 18 — Figure 18: Offline throughput, in millions of voxels processed per second, for representative confocal and non-confocal datasets. Hatched bars are our implementations and solid bars the original MATLAB ones. Full timing and memory results are reported in the supplementary material.

			$\downarrow$ Time ( $\mathrm{s}$ )		$\uparrow$ Throughput (Mvox/ $\mathrm{s}$ )		$\downarrow$ Max. tracked memory ( $\mathrm{GB}$ )
Dataset	Dimensions	Algorithm	Ours	Original	Ours	Original	Ours	Original
Confocal – $f\text{--}k$ migration dataset ( Lindell et al., 2019 )
bike	512 $\times$ 512 $\times$ 512	$f\text{--}k$ migration	2.114 $\pm$ 0.03	146.624 $\pm$ 8.47	63.50	0.92	17.000	52.430
bike	512 $\times$ 512 $\times$ 512	Phasor fields	3.791 $\pm$ 0.03	30.872 $\pm$ 0.67	35.40	4.35	22.001	33.579
teaser	512 $\times$ 512 $\times$ 512	$f\text{--}k$ migration	2.016 $\pm$ 0.01	154.730 $\pm$ 14.77	66.59	0.87	17.000	52.433
teaser	512 $\times$ 512 $\times$ 512	Phasor fields	3.916 $\pm$ 0.08	31.458 $\pm$ 2.65	34.28	4.27	22.001	33.579

Table 7: Offline comparison against the original MATLAB implementations, showing the largest configuration measured for each dataset. This is a representative extract: every dataset was also evaluated at lower spatial and temporal resolutions, and those results, together with the per-resolution trends, are reported in full in the supplementary material. Throughput is the reconstructed volume divided by reconstruction time. For exhaustive captures, the dimensions column lists the original measurement grid before normal moveout correction; normal moveout correction produces a

31\times 31\times T

confocal transient, which is the volume used for the throughput calculation. Max. tracked memory reports the peak footprint of each implementation’s main buffers: host RAM for the original MATLAB code, and the peak of all tracked CUDA allocations for ours, which counts device buffers and pinned host staging together and is therefore not a pure video-memory figure.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 19: Photon binning time for a dynamic NLOS capture of shape

190\times 190\times 3002

(

x\times y\times t

), whose photons are accumulated directly into

208

temporal frequencies. The photon count varies slightly from frame to frame and typically stays near

1.2\cdot 10^{6}

Figure 20: Teaser dataset from ( Lindell et al., 2019 ) . The first and second columns correspond to measurements of 180 and 10 minutes, respectively. We downsample the former to $10^{6}$ and $5\cdot 10^{5}$ photons to show that the hidden scene remains recognizable with fewer captured photons. The first and second rows show our $f\text{--}k$ and phasor-fields reconstructions, respectively. The numbers within each reconstruction indicate the Peak Signal-to-Noise Ratio with respect to the most informed reconstruction, shown in the first column.

Panel of Figure 25 — Algorithm 1 $f\text{--}k$ migration ( Lindell et al., 2019 ) . Reconstruction pipeline following prior work. Pseudocode shown in a Python-like style. $N_{x}$ , $N_{y}$ , and $N_{z}$ denote the volume dimensions. $d_{\max}$ refers to the maximum distance, whereas $\mathrm{apt_{width}}$ is the physical size in meters of the scanned relay wall.

Panel of Figure 26 — Algorithm 2 Our CUDA-based $f\text{--}k$ migration pipeline.

Panel of Figure 27 — Algorithm 3 Phasor-fields as implemented by Nam et al. (2021) . Reconstruction pipeline following prior work. $P$ holds the sampled phasor field $\hat{\mathcal{P}}_{\omega}$ and $K$ the dense propagation kernel, both as defined in Section 4.3 of the main paper; $\widehat{P}$ denotes the stack of transformed slices $\widehat{P}_{c}$ produced by the first loop. $N_{x}$ , $N_{y}$ , and $N_{f}$ denote the relay-wall spatial dimensions and the number of temporal-frequency slices; $d_{\min}$ , $d_{\max}$ , and ${\Delta}d$ define the propagated-depth range, and $w$ weights each frequency slice.

Panel of Figure 28 — Algorithm 4 Our symmetry-aware phasor-fields dataflow. The dense kernel $K$ from Algorithm 3 is replaced by a radial kernel $K_{r}$ and a pixel-to-radius map $R$ .

Panel of Figure 22 — Figure 22: Per-configuration real-time comparison on the nlosbox1 dynamic capture. The top row shows a representative reconstruction, while the bottom row summarizes the runtime-memory trade-off across methods. Frame rates are derived from mean frame times, as in Table 2 of the main paper.

Figure 23: Image-space error on the office scene. Each row compares our reconstruction with its reference and reports FLIP and absolute error maps. The reference for the phasor-fields rows is Nam et al. (2021) ’s phasor-fields implementation, so the comparison isolates the radial kernel representation. For $f\text{--}k$ , the reference is $f\text{--}k$ using FP32 .

$\beta$	$\uparrow$ Object	$\downarrow$ Background	$\uparrow$ Contrast	$\uparrow$ Sharpness
0.15	0.520	$8.7\cdot 10^{-5}$	6,008	0.050
0.20	0.495	$3.8\cdot 10^{-5}$	12,993	0.051
0.25	0.471	$1.7\cdot 10^{-5}$	28,186	0.052
0.30	0.449	$7.4\cdot 10^{-6}$	60,981	0.053
0.35	0.428	$3.2\cdot 10^{-6}$	132,666	0.054
0.40	0.408	$1.4\cdot 10^{-6}$	291,566	0.055

Appendix

Table 8: Impact of the pooling weight

\beta

on the 100-frame nlosbox1 sequence (FP32, unpadded). Because every metric changes monotonically with

\beta

, there is no single optimal setting:

\beta=0.5

provides the strongest background rejection but also the weakest object signal. Bold text highlights our chosen operating point (

\beta=0.3

), which matches the MoE row in Table 6 of the main paper.

Panel of Table 9 — Table 9: Offline performance and memory comparison between our accelerated $f\text{--}k$ and phasor-fields implementations and their original MATLAB counterparts, across every measured resolution. Throughput is the reconstructed volume divided by reconstruction time. For exhaustive captures, the dimensions column lists the original measurement grid before normal moveout correction, which produces a $31\times 31\times T$ confocal transient volume. Max. tracked memory reports the peak footprint of each implementation’s main buffers: host RAM for the original MATLAB code, and the peak of all tracked CUDA allocations for ours, counting device buffers and pinned host staging together (see the discussion preceding these tables). Bold marks our value when it beats the baseline.

Panel of Table 10 — Table 10: Continuation of Table 9 .

			$\downarrow$ Time ( $\mathrm{s}$ )		$\uparrow$ Throughput (Mvox/ $\mathrm{s}$ )		$\downarrow$ Max. tracked memory ( $\mathrm{GB}$ )
Dataset	Dimensions	Algorithm	Ours	Original	Ours	Original	Ours	Original
Tal-generated dataset ( Royo and Luesia-Lahoz, 2024 ) , inspired by a Phasor Fields scene ( Liu et al., 2020 )
office	360 $\times$ 260 $\times$ 1500	$f\text{--}k$ migration	2.945 $\pm$ 0.05	507.808 $\pm$ 47.68	47.67	0.28	17.784	75.938
	360 $\times$ 260 $\times$ 1500	Phasor fields	5.414 $\pm$ 0.02	138.281 $\pm$ 13.87	25.93	1.02	23.023	66.424
	360 $\times$ 260 $\times$ 750	$f\text{--}k$ migration	0.308 $\pm$ 0.01	64.483 $\pm$ 7.22	227.77	1.09	8.893	37.969
	360 $\times$ 260 $\times$ 750	Phasor fields	0.458 $\pm$ 0.00	28.830 $\pm$ 1.34	153.27	2.43	11.510	33.194

Appendix

Table 11: Continuation of Table 10 .

Image Reconstruction

Image reconstruction recovers images, volumes, or scenes from incomplete, noisy, indirect, or compressed measurements using physical models and learned priors.

See all 287 papers

Non-line-of-sight imaging with arbitrary relay surface geometries via 3D Gaussian Transient Rendering
Jun 19, 2026Yi Wang, Ziyu Zhan, Yuran Wang +53D Reconstruction Image Reconstruction
Department of Precise Instrument, Tsinghua University, China · Department of Electrical Engineering, City University of Hong Kong, China · Yau Mathematical Science Center, Tsinghua University, China
A comprehensive study of time-of-flight non-line-of-sight imaging
Mar 10, 2026Julio Marco, Adrian Jarabo, Ji Hyun Nam +3Image Reconstruction Computational Imaging
Universidad de Zaragoza–I3A, Spain · University of Wisconsin–Madison, USA · Department of Data Science, Inha University, South Korea +1
Measurement-Adaptive Light-Cone Transform for Robust Non-Line-of-Sight Imaging
Oct 16, 2026Lianfang Wang, Xueying Liu, Junnan Dong +4Image Reconstruction Image Inverse Problems
School of Mathematical Sciences, Beijing Normal University, Beijing 100875, China · School of Physics, Nankai University, Tianjin 300071, China

Jun 19, 2026J/K move · Enter open · S save

Non-line-of-sight imaging with arbitrary relay surface geometries via 3D Gaussian Transient Rendering

Yi Wang, Ziyu Zhan, Yuran Wang, Hao Wang+4

Department of Precise Instrument, Tsinghua University, China · Department of Electrical Engineering, City University of Hong Kong, China · Yau Mathematical Science Center, Tsinghua University, China

3D Reconstruction Image Reconstruction

Open paper AskPDF arXiv

Imaging objects hidden outside the direct line of sight expands the effective field of view and is critical for applications such as autonomous driving and robotic perception. Despite impressive progress in time-of-flight (ToF)-based non-line-of-sight (NLOS) imaging, real-world deployment remains challenging because practical measurements are often collected over spatially limited, arbitrarily shaped relay regions-conditions that violate the planar-wall and dense-sampling assumptions made by most existing methods. To address these limitations, we propose a LOS-guided NLOS imaging pipeline that imposes no geometric assumptions on the relay surface and naturally supports both confocal and non-confocal configurations. Our method represents the hidden scene using 3D Gaussian primitives and couples them with an efficient, differentiable transient rendering model, enabling end-to-end optimization directly from measured transients. We validate our approach on real-world measurements from both a public dataset and a custom-built capture system. Across settings, our method achieves state-of-the-art reconstruction fidelity under spatially limited, sparsely sampled conditions, and significantly outperforms existing methods on complex, arbitrary relay surface geometries.

Abstract

Image Reconstruction

Explore similar work

Non-line-of-sight imaging with arbitrary relay surface geometries via 3D Gaussian Transient Rendering

A comprehensive study of time-of-flight non-line-of-sight imaging

Measurement-Adaptive Light-Cone Transform for Robust Non-Line-of-Sight Imaging