Organizations: School of Mathematical Sciences, Beijing Normal University, Beijing 100875, China · School of Physics, Nankai University, Tianjin 300071, China
Practical non-line-of-sight (NLOS) measurements are affected by noise and deviations from idealized light transport, which can degrade reconstruction with conventional light-cone transform (LCT) methods. We propose the Measurement-Adaptive Light-Cone Transform (MALCT), a physics-guided framework that retains the efficient Fourier-domain inversion structure of LCT. {Within a unified variational formulation, MALCT combines the analytical response with a shared learned response through measurement-conditioned weighting and predicts a measurement-conditioned, frequency-dependent regularization map.} An attention-enhanced spatio-temporal feature extraction module captures long-range temporal dependencies and local spatial correlations to support reconstruction under low-signal conditions. Experiments on synthetic data and real measurements from different NLOS systems demonstrate improved reconstruction quality. Controlled evaluations further assess robustness to temporal-response broadening and reduced acquisition time. MALCT also preserves scene structures with spatial sampling grids down to 32×32, supporting its potential for NLOS imaging under limited acquisition budgets.
Figures & tables
Figure 1: Overview of the proposed framework. (a) Confocal NLOS imaging configuration. (b) Conventional LCT reconstruction with a fixed analytical Fourier response H and scalar regularization. (c) MALCT reconstruction with a measurement-conditioned effective response HΘ(k;τ) and regularization map αΘ(k;τ) . The effective response combines the analytical response with a shared learned response through measurement-conditioned weighting.
Figure 2: The architecture of our proposed MALCT framework.
Figure 3: The spatio-temporal feature extraction module consists of a 3D UNet, temporal global feature extraction, spatial local feature extraction and fusion module.
Figure 4: Results for the USTC letter and Stanford Statue scenes, with the Statue acquired over 10 min. (a) Central slices of the jointly normalized inverse kernels learned from the USTC letter scene. (b) 64×64 albedo reconstructions using LCT with αHigh=10 or αLow=0.1 and MALCT with αLearned .
Figure 5: Evaluation under prescribed Gaussian IRF broadening. (a)1D inverse kernel profiles ( κ=1 ) for LCT and MALCT. (b) Intensity reconstructions with PSNR/SSIM values. (c) Resampled STFE transients and corresponding reconstruction-projections. (d) Absolute t - x residuals and relative errors.
Method
SNR = 10 dB
SNR = 8 dB
SNR = 5 dB
Intensity
Depth
Intensity
Depth
Intensity
Depth
PSNR ↑
SSIM ↑
RMSE ↓
MAD ↓
PSNR ↑
SSIM ↑
RMSE ↓
MAD ↓
PSNR ↑
SSIM ↑
RMSE ↓
MAD ↓
LCT
21.672
0.480
0.215
0.168
21.569
0.480
0.215
0.168
21.267
0.471
0.215
0.168
FK
26.452
0.677
0.112
0.059
26.414
0.726
0.112
0.059
26.268
0.617
0.111
0.059
PF
26.765
0.367
0.112
0.027
25.998
0.279
0.112
0.027
25.689
0.341
0.133
0.064
LFE
27.052
0.792
0.066
0.021
26.235
0.787
0.065
0.021
26.176
0.789
0.065
0.021
Table 1: Quantitative comparison under different noise levels. PSNR and SSIM are computed for reconstructed intensity images, while RMSE and MAD are computed for depth maps. The best and second-best results are shown in bold and underline , respectively.
Figure 6: Qualitative comparison of the reconstructed intensity images and the depth error maps, where the depth errors are visualized in the range [0,0.3] , within the ROI on the simulated dataset.
Metric
LCT
FK
PF
LFE
NLOST
LPP
MALCT
Time (s) ↓
0.05
0.06
0.09
0.04
0.42
0.25
0.12
Memory (GB) ↓
8.2
9.7
11.6
3.3
21.4
9.1
5.5
Table 2: Computational efficiency comparison in terms of inference time and GPU memory usage. The best and second-best results are shown in bold and underline , respectively.
Figure 7: Reconstructions from the Stanford dataset under 180-minute and 10-minute exposures.
Figure 8: Reconstructed hidden scenes and corresponding 2D albedo images from the USTC and Stanford 10-minute real-world datasets.
Figure 9: Reconstructed 2D albedo images from the Stanford 180-minute and USTC datasets under uniform spatial subsampling. Measurements are resized to 128×128 by nearest-neighbor interpolation and processed by the same model without retraining.
Figure 10: Ablation study on Dragon and Bike. (a) Configurations. All variants retain the 3D convolutional backbone; NoAtt removes G-TSA and L-SSA. Variants I-IV use fixed scalar regularization, V a globally learned scalar, and VI a measurement-conditioned frequency map. (b) Reconstructions for Variants I-VI and the model without LTV or LTik from 32×32 measurements. PSNR and SSIM are computed against the PF reference obtained from the original 512×128×128 , 180-min measurements.
Image reconstruction recovers images, volumes, or scenes from incomplete, noisy, indirect, or compressed measurements using physical models and learned priors.
Time-of-Flight non-line-of-sight (ToF NLOS) imaging techniques provide state-of-the-art reconstructions of scenes hidden around corners by inverting the optical path of indirect photons scattered by visible surfaces and measured by picosecond resolution sensors. The emergence of a wide range of ToF NLOS imaging methods with heterogeneous formulae and hardware implementations obscures the assessment of both their theoretical and experimental aspects. We present a comprehensive study of a representative set of ToF NLOS imaging methods by discussing their similarities and differences under common formulation and hardware. We first outline the problem statement under a common general forward model for ToF NLOS measurements, and the typical assumptions that yield tractable inverse models. We discuss the relationship of the resulting simplified forward and inverse models to a family of Radon transforms, and how migrating these to the frequency domain relates to recent phasor-based virtual line-of-sight imaging models for NLOS imaging that obey the constraints of conventional lens-based imaging systems. We then evaluate performance of the selected methods on hidden scenes captured under the same hardware setup and similar photon counts. Our experiments show that existing methods share similar limitations on spatial resolution, visibility, and sensitivity to noise when operating under equal hardware constraints, with particular differences that stem from method-specific parameters. We expect our methodology to become a reference in future research on ToF NLOS imaging to obtain objective comparisons of existing and new methods.
Julio Marco, Adrian Jarabo, Ji Hyun Nam +3
Universidad de Zaragoza–I3A, Spain · University of Wisconsin–Madison, USA · Department of Data Science, Inha University, South Korea +1
Imaging objects hidden outside the direct line of sight expands the effective field of view and is critical for applications such as autonomous driving and robotic perception. Despite impressive progress in time-of-flight (ToF)-based non-line-of-sight (NLOS) imaging, real-world deployment remains challenging because practical measurements are often collected over spatially limited, arbitrarily shaped relay regions-conditions that violate the planar-wall and dense-sampling assumptions made by most existing methods. To address these limitations, we propose a LOS-guided NLOS imaging pipeline that imposes no geometric assumptions on the relay surface and naturally supports both confocal and non-confocal configurations. Our method represents the hidden scene using 3D Gaussian primitives and couples them with an efficient, differentiable transient rendering model, enabling end-to-end optimization directly from measured transients. We validate our approach on real-world measurements from both a public dataset and a custom-built capture system. Across settings, our method achieves state-of-the-art reconstruction fidelity under spatially limited, sparsely sampled conditions, and significantly outperforms existing methods on complex, arbitrary relay surface geometries.
Yi Wang, Ziyu Zhan, Yuran Wang +5
Department of Precise Instrument, Tsinghua University, China · Department of Electrical Engineering, City University of Hong Kong, China · Yau Mathematical Science Center, Tsinghua University, China
Time-of-flight non-line-of-sight (NLOS) imaging recovers information from hidden objects by analyzing the time of flight of indirect photons scattered on a visible (relay) wall. Most methods make the simplifying assumption that photons travel exclusively three-bounce paths, thus ignoring other useful information encoded in higher-order photons (with, e.g., four- or five-bounce paths). We present a novel cascaded NLOS imaging approach that leverages higher-order information and allows imaging a broader range of single- and multi-corner scenarios. We combine ultra-fast laser scanning with recent time-gated 2D sensor arrays to capture the scene's impulse response on a visible relay wall. From the captured impulse response, our method computes an analogous virtual impulse response at any other hidden wall. This effectively allows us to concatenate a second, virtual NLOS imaging system that leverages higher-order illumination. We validate our cascaded imaging method both in simulation and with a real prototype, demonstrating NLOS imaging with fourth- and fifth-bounce illumination of objects in challenging orientations and hidden around two corners. We also analyze how wave-based NLOS imaging interacts with rough hidden walls, which explains and helps overcome existing visibility limitations. We further illustrate how to image hidden objects from different perspectives, thus observing previously unseen features, by relying on multiple hidden walls.
Diego Royo, María Peña, Forrest B. Peterson +3
Universidad de Zaragoza–I3A, Spain · University of Wisconsin–Madison, United States of America