Single-photon lidar records the arrival time of each returning photon, enabling 3D reconstruction from few views and scarce photons. However, leading neural methods such as Transientangelo require over an hour of non-convex per-scene optimization, and their accuracy degrades as photon flux decreases. We propose FirstCross, a network-free approach that instead solves a short sequence of convex problems on a voxel grid. For a solid object, the Poisson likelihood of a ray with given signal and background rates depends on the scene solely through its first crossing, the point where the ray first enters the object. Decomposing this likelihood yields two tractable terms. The first penalizes surfaces in front of detected photons and admits a tight convex closure, namely volume rendering with a running-maximum transmittance in place of NeRF's product transmittance. The second accounts for photons that no surface explains and admits a convex upper bound once the object is assumed to be thick behind each detected surface point. Compared with Transientangelo under its own real-data protocol, FirstCross achieves lower depth error across every test scene, view count, and flux level. With three views and 100 times fewer photons than at the protocol's lowest flux, our mean object depth error stays under half of Transientangelo's with all photons. Given our detected surface points, the best classical fusion has 1.3 to 1.8 times our three-view error across flux levels. On real room-scale captures with 1% of their photons, TransientNeRF has 2.8 to 7.7 times our error. FirstCross lets few-view single-photon scanners cut exposure to 1% and reconstruct an object in about a minute on one GPU.
Figures & tables
Figure 2: Overview of FirstCross on real data. FirstCross fits a field u by a short sequence of convex problems. (a) Photons of one view at 1 PPP in three arrival-time slices. (b) The likelihood of ray r depends on the scene only through its first crossing t⋆ (purple). Cross-sections across r (top) show almost no photons in free space, and the pooled photons of r (bottom) peak at t⋆ . The witness (dark blue), just behind the return, is assumed to lie inside the object. (c) Each of the 12 iterations puts the weights W (orange) where rays enter the current object, then updates u by a linear solve. The weight zr of ray r falls with its photon count. (d) The shape Ω={u>0} and a novel view.
Figure 3: The too-early cost and its two extensions. (a) The cost of a photon at tj against the first crossing t⋆ splits into too-early and too-late costs, capped at ω beyond tj±ε . (b, c) The too-early cost (darker is lower) of an eight-sample floater over the occupancies x1,x2 of its halves, both empty at the star. Gradient descent from a nearly opaque floater (dot) stalls for 20 000 steps under the product and empties it in 196 under the running maximum.
Flux (PPP)
Full
300
150
50
10
1
0.1
Scene
TA [ 22 ]
Ours
TA
Ours
TA
Ours
TA
Ours
TA
Ours
Ours
Ours
baskets
7.58
5.73
7.26
5.63
7.23
5.68
7.55
5.60
14.77
5.66
5.67
5.64
boots
0.92
0.42
0.86
0.41
0.83
0.41
33.21 †
0.41
0.62
0.42
0.43
0.58
chef
3.88
0.91
3.50
0.95
4.81
0.92
2.73
0.94
5.17
0.94
1.08
1.76
food
6.43
4.84
6.30
4.84
6.26
4.85
5.94
4.86
7.62
4.86
4.94
5.02
Mean
4.70
2.97
4.48
2.96
4.78
2.97
12.36 †
2.95
7.04
2.97
3.03
3.25
Table 1: Depth error (mm, ↓ ) on the test scenes of the Transientangelo (TA) protocol. Ours is lower than TA in every pair. TA uses its released checkpoints, none below 10 PPP, and 1 and 0.1 PPP are thinned from 10 PPP. Rows use three views unless stated, and the object depth MAE excludes the back wall. † The boots checkpoint misses most of the object here in at least half of the novel views, and ours is also lower without boots. The TransientNeRF row averages the per-scene errors rounded to 1 mm in [ 22 ] , with – where none is reported.
Figure 4: Mean object depth MAE against photon count over the three-view test scenes. Ours stays below Transientangelo’s full-flux error (dashed) down to fewer than 103 photons in total. Transientangelo at 50 PPP is omitted, as its boots checkpoint fails.
Figure 5: Novel views from full flux to 0.1 PPP, and transients on two test scenes. The red dot marks a pixel where Transientangelo has a floater or hole (depth error over 2 cm) at 10 PPP and both methods are within 5 mm at full flux. There, Transientangelo’s transient at 10 PPP is mostly late on baskets and missing on boots, whereas ours stays at the ground-truth peak. Transients, on a log scale, are scaled to the ground truth (measured at full flux). Grey lines mark powers of ten (horizontal) and 0.5 ns after the ground-truth peak (vertical).
Figure 6: Product transmittance against the single convex problem. At 0.1 PPP on the three-view test scenes, the product variant leaves floaters (orange) with masks, largest on chef and food. Without masks, it covers most of the view of boots, chef and food with spurious surfaces. Blue marks holes, and numbers are object depth MAEs.
Figure 7: Room-scale depth on the test rooms with three views. At 1% flux, TransientNeRF’s depth becomes noisy with spurious surfaces in every room (red frames), whereas ours stays near its full-flux depth. Per room, we show the novel view where our 1% flux error is the median, with one colour scale and white outside the benchmark mask. Numbers give the per-view mean depth MAE.
With masks
No masks
Method
Full
10
0.1
10
TV-L1 fusion [ 49 ]
1.18
1.61
2.03
–
Product transmittance
1.12
1.11
3.02
12.46
Single convex problem
0.91
0.92
1.35
0.91
FirstCross (Ours)
0.90
0.89
1.17
0.85
Table 2: Ablation study. Object depth MAE (mm, ↓ ) per flux level (PPP), averaged over the four three-view test scenes. Each variant changes one component and uses our solver. Product transmittance replaces the running maximum with NeRF’s product. TV-L1 fusion uses our detected returns. Best in bold, second best underlined, and – marks settings not run.
Full flux
1% flux
Room
Views
TN [ 23 ]
Ours
TN
Ours
c4floor2
3
0.435
0.439
3.973
0.525
5
0.338
0.459
3.891
0.508
material_building
3
0.382
0.705
2.135
0.325
5
0.319
0.718
1.767
0.315
physics_building3
3
0.768
1.209
3.303
1.193
Table 3: Mean depth MAE (m, ↓ ) over all novel views of the SP-TransientBench test rooms. Better of each pair in bold. TN (TransientNeRF) trains for the official 300k steps. On material_building, our error drops at 1% flux as the grid box around our returns shrinks. With the full-flux box, ours wins there (Sec. S6).
Partially transmissive screens and protective covers are common in robotic inspection, but they create mixed LiDAR returns from both the foreground material and the scene behind it. Conventional peak-based LiDAR usually discards weak hidden returns, while single-photon LiDAR records time-resolved histograms that preserve attenuated and overlapping echoes. However, existing transient reconstruction methods typically fit a single scene representation to the measured waveform. Under occlusion, weak or nearby foreground--hidden echoes can form a broad peak or subtle shoulder. Because such waveforms can also be explained by a displaced single surface or a thick density distribution, accurate transient fitting does not necessarily imply correct geometry. We propose a state-aware framework for foreground-view and hidden scene reconstruction from occluded single-photon histograms. For each ray, we estimate local echo evidence, identifying no reliable surface evidence, single-return evidence, or two returns. The inferred echo state routes supervision for a two-head neural field: all rays constrain waveform reconstruction, while reliable anchors provide geometry localization. We also introduce a real paired single-photon LiDAR occlusion dataset with occluded and clean captures at fixed poses. Experiments on a real dataset show improved hidden scene depth and point-cloud accuracy over baselines. Our results demonstrate single-photon layered reconstruction as a practical route for 3D perception through partially transmissive occluders.
Ziting Wen, Runrong Deng, Zili Zhang +5
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology · School of Mechatronic Engineering and Automation, Shanghai University · Department of Electrical and Computer Engineering, National University of Singapore +1
Single-view 3D reconstruction is a challenging task in computer vision due to information missing from the single input image. Generative model-based approaches can produce plausible 3D objects from a single image, thanks to data-driven priors learnt from rich and large-scale datasets. However, plausible generation does not guarantee fidelity to the geometry and appearance of the particular input object. Inspired by object perception in human vision, we propose OPERA, a framework that guides multi-view diffusion sampling with pretrained perception models through lightweight alignment modules. These modules are trained independently while both the generative and perception models remain frozen, allowing multiple signals to be combined at inference without joint fusion training. We evaluate OPERA on two single-view 3D reconstruction baselines using subsets of Google Scanned Objects and OmniObject3D. On the primary baseline, combined guidance reduces mean Chamfer Distance by 30.4% and 24.8%, respectively, relative to unguided reconstruction. We also compare our method with recent image-to-3D models. We provide in-depth analyses of the design choices and their effects across datasets and backbones. Our project page is at https://opera-3d.github.io/.
Y Huynh, Duc Thanh Nguyen, Mohamed Abdelrazek
Applied Artificial Intelligence Initiative Deakin University · School of Information Technology Deakin University
Single-photon cameras based on single-photon avalanche diode (SPAD) technology are gaining popularity for 3D sensing, thanks to their extreme sensitivity and time resolution. There are two key challenges with single-photon cameras that limit their widespread use: (i) they suffer from non-linear distortions called ''pile-up'' when operated in high-photon-flux conditions, and (ii) they generate a large volume of raw photon data, creating a severe data bottleneck at each sensor pixel. In this work, we show that while compressive capture techniques successfully mitigate data transfer challenges, they exacerbate the effects of dead-time distortion because they fail to retain sufficient information about the photon detection history to allow post-processing pile-up correction via existing methods. We propose a new computational-imaging method that combines free-running capture with an analysis-by-synthesis software pipeline to mitigate pile-up distortions. Our results with hardware emulations and full-scene and single-pixel simulations show that our method can reliably capture scene distance and reflectance over a wide range of illumination conditions. Our work will enable high-resolution SPAD cameras that are severely bandwidth-constrained to operate in real-world high-flux scenarios.