We introduce SpiderCam, an FPGA-based snapshot depth-from-defocus camera which produces 480x400 sparse depth maps in real-time at 32.5 FPS over a working range of 52 cm while consuming 624 mW of power in total. SpiderCam comprises a custom camera that simultaneously captures two differently focused images of the same scene, processed with a SystemVerilog implementation of depth from differential defocus (DfDD) on a low-power FPGA. To achieve state-of-the-art power consumption, we present algorithmic improvements to DfDD that overcome challenges caused by low-power sensors, and design a memory-local implementation for streaming depth computation on a device that is too small to store even a single image pair. We report the first sub-Watt total power measurement for passive FPGA-based 3D cameras in the literature.
Figures & tables
Name
Type
FPGA
Resolution
Core Power (W)
Real-World Quantitative Results
Min
Max
On-Device Working Range
Full System Power
Jin, 2014 [ 31 ]
Stereo
Kintex-6
640 × 480
1.93
2.67
none
none
Raj, 2014 [ 32 ]
DfD
Virtex-4 †
400 × 400
0.46
0.58
0.77–0.80 m ( 0.5 % err )
2 W + camera
Mattoccia, 2015 [ 44 ]
Stereo
Spartan-7
640 × 480
0.44
0.68
qualitative
2.5 W
Ttofis, 2015 [ 65 ]
Stereo
Kintex-7
1280 × 720
0.92
1.53
qualitative
2.8 W
Puglia, 2017 [ 55 ]
Stereo
Zynq Artix
1024 × 768
0.43
0.68
none
2 W
Table 1: Power consumption for passive FPGA-based depth imaging systems. “Core Power” numbers reflect only power consumed by logic for depth computation on aligned images; I/O and sensor power are ignored. Power estimates are derived using vendor tools and data from the corresponding works, normalized to a processing rate of 30 Mpix/sec across methods, see Sec. S8 for details. In addition to SOTA power efficiency in core power estimates, we are one of the very few systems to report full system power consumption and real-world depth performance (from noisy sensor captures rather than on existing datasets collected with large-baseline, high-power, pre-aligned image pairs). Note that we report two estimates for our method: a “normalized” estimate that handles more pixels per second on a less efficient FPGA for fairer comparisons with the literature, and a lower estimate reflecting the true parameters of our system. Additionally, we include Focal Split (Luo, 2025) [ 41 ] , which reports a similar method on a Raspberry Pi 5 with real-world depth and power performance. † This work uses an Xilinx Virtex-2 FPGA, but due to vendor tool unavailability, we estimate power for the more efficient Virtex-4.
Figure 1: Method overview. Our camera uses a beam splitter and a pair of differentially-defocused low power sensors to observe the same scene with an offset depth of field. Our algorithm processes these images to produce depth maps thresholded by confidence.
Figure 2: Captured images and depth maps computed on-device. Note the handling of continuous depths (vs. disparity levels), reflective textures, and transparent objects that active sensing would struggle with, as well as objects in motion due to snapshot capture.
Figure 3: Quantitative analysis on images from device. (a) Mean absolute error (MAE) of depth as a function of true depth, with working range bounds. Our method robustly outperforms the Focal Split algorithm [ 41 ] in the low power hardware setting, and accounting for spatial variation in our compact optics is vital. Purple dots show errors from depth maps computed on-device and demonstrate good agreement with off-line depth computations (rest of figure), which were performed in 32-bit floating point. (b) The MAE vs. depth curve of our method with dynamic sparsity values set by per-image confidence percentiles (grays) and the MAE (solid purple) and output pixel density (100-sparsity, dashed purple) resulting from the fixed confidence thresholds used in our method. (c) Working range increases as data is sparsified by per-image confidence percentiles.
Figure 4: Power/accuracy trade-offs for alternate algorithms. We estimate max core power on the ECP5 with manufacturer’s tools and evaluate working range and MAE from 0.4m to 1.0m offline on images collected by our hardware prototype. Adding computational components improves accuracy while increasing power cost. Our algorithm is shown with the purple star.
Figure 5: Resource Usage Across Methods. We compare the resource utilization of several configurations of our system as well as systems in the literature. “DS” and “SS” denote evaluation at 2 and 1 scales, respectively.
Figure S1 : General overview of the algorithm.
Figure S2 : Preprocessing module.
Figure S3 : Computation for the Nth scale.
Filter / Operation
Adders
True Mult.
Easy Mult.
Dividers
Total FLOPs
Gaussian 5×5 Filter
8
2
8
0
18
Downsampler Filter
2
0
4
0
6
Upsampler Filter
6
4
4
0
14
VN & WN
2
2
0
0
4
Cross Mult. w/ Weights (DX_DY_ENABLE = 1)
8
12
10
0
30
Cross Mult. w/ Weights (DX_DY_ENABLE = 0)
0
4
0
0
4
Table S1 : FLOP counts and arithmetic breakdown per operation. N refers to the number of scales. Total FLOPs is the sum of all arithmetic operations per filter or stage.
N (Scales)
DX_DY_ENABLE
Adders
True Mult.
Easy Mult.
Dividers
Total FLOPs
1
0
28
14
28
1
71
1
1
36
22
38
1
97
2
0
70
36
64
1
171
2
1
86
52
84
1
223
3
0
124
66
108
1
299
3
1
148
90
138
1
377
Table S2 : Aggregate per-pixel operation counts across N scales using the per-operation table and the updated per-scale rule (2× Gaussian, 2× Downsampler, (1+2×Nth) Upsamplers per scale, and 1× Cross Mult. with Weights per scale), VN & WN plus one final Add and Divide.
N (Scales)
DX_DY_ENABLE
Scale Buffers
Latency Buffers
Total Buffers
1
0
29
0
29
1
1
33
0
33
2
0
93
12
105
2
1
105
14
119
3
0
227
36
263
3
1
255
42
297
Table S3 : Aggregate buffered line requirements for N=1 – 3 scales, including buffers used in scales, buffers consumed by Latency Buffer, and Total buffers used. Each buffer is the size of one line of the image in FP16. For both DX_DY_ENABLE modes.
Name
Type
FPGA
Resolution
Stereo Disparity Levels
Mpix/sec
Estimated Core Power (W)
Measured Full-System Power
Resource Utilization
Reported
Min
Max
LUTs
REGs
BRAMs
DSPs
Jin, 2014 [ 31 ]
Stereo
Kintex-6
640 × 480
60
15.6
10.6
1.93
2.67
none
61000
61000
165
0
Raj, 2014 [ 32 ]
DfD
Virtex-4
400 × 400
n/a
48.0
none
0.46
0.58
2W + camera
10028
7609
29
50
Mattoccia, 2015 [ 44 ]
Stereo
Spartan-7
640 × 480
32
9.2
none
0.44
0.68
2.5W
23749
9030
67
0
Ttofis, 2015 [ 65 ]
Stereo
Kintex-7
1280 × 720
64
55.3
2.8
0.92
1.53
none
57492
71192
302
458
Puglia, 2017 [ 55 ]
Stereo
Zynq Artix
1024 × 768
64
23.6
0.17
0.43
0.68
2W
29057
15208
80
0
Table S4 : Extended version of Tab. 1 . This table includes the reported power from other works. We re-did calculations from other works to account for differing pixel processing rates - our analysis favors methods that process pixels above 30 Mpix/s and penalize works below 30 Mpix/s. We under-estimate power for all works except [ 55 ] . † The internal design of the Lattice ECP5 means that its resource utilization values can’t be directly compared to those of other FPGAs. Each LUT, BRAM, and DSP module on the ECP5 is smaller than on the 7-Series, so a design on the ECP5 will typically take more of these smaller resources.
Figure S4 : Heatmap comparison of our method with and without field curvature correction. We decompose an image in 16 radial regions (row 1) and show the predicted depth vs. true depth heatmaps (filtered with a 90% sparsity) for every radial regions and their combined region. We show that compared to using constant optical parameters (row 2), applying spatial variation (row 3) yields better depth predictions that are much more aligned with the diagonal, indicating better accuracy, in all radial zones. We also observe with the confidence filtering, radial zones in the center tend to perform better at longer depth ranges than the peripheral zones, which is well aligned with the observation of Petzval field curvature, where the focal planes are curved towards the lens.
Figure S5 : Ablation studies. (a) Heatmaps of Focal Split [ 41 ] and different variations of our method. (b) Predicted depth maps on the calibration dataset at three different depths from our front-parallel calibration dataset. We show that incorporating spatial variation, multiscale, and multi-derivative in our algorithm helps improve depth estimation in our lower power setting.
Figure S6 : Comparison of different confidence. We show the mean depth error vs. true depth curves at 90% sparsity on the calibration data using different confidence metrics. We observed an increase in the working range with VW (numerator in Eq. 3 ) as confidence compared with W2 or ∣W∣ , which were used in previous work [ 41 , 40 ] .
Component Description
Commercial off-the-shelf
Part number / path to design files on GitHub
Qty
FPGA development board
yes
LFE5UM5G-85F-EVN
1
Image sensor cables
yes
900-0150180325-ND
2
FT232H breakout board
yes
2264 (Adafruit)
1
16mm beamsplitter
yes
CCM5-BS016
1
M09 mounted lens
yes
AC050-008-A-ML
2
Lens adapter
yes
S05TM09
2
Table S5 : A list of all of the significant parts making up our system. Commercial off-the-shelf components can be purchased at a store, while non-commercial off-the-shelf parts are those which we designed for this project and can be found in our GitHub repository ( https://github.com/NUBIVlab/SpiderCam ) using the path names. To replicate our work, our design files must be used to order these from manufacturers.
Figure S7 : Optical assembly. This photograph shows the 3D printed mount that affixes the image sensor to the beamsplitter and aligns it to the lens. The image sensor, lens, and flat-flex cable connecting the optical assembly to the FPGA development board are visible.
Figure S8 : Depth maps computed on-device for front-parallel textured plane. This data was used to generate purple dots in Fig. 3 . Note that density naturally falls off away from the center of the working range. Our confidence estimate captures edge textures with sufficient focus, and field curvature shifts the effective working range across the image. Edge effects dominate error in sparse images.
Figure S9 : Depth maps computed on-device with their references for real scenes. Note MAE from top to bottom of 0.057 m, 0.058 m, and 0.063 m, consistent with data in Fig. S8
Figure S10 : Screenshot of representative power consumption trace. We demonstrate how we used the Otii Arc Pro to measure power consumption for Tab. S6 by measuring power over time. This screenshot shows power for our system measured in a dual-scale configuration with dx and dy filters at 32.5 FPS. Notably, the frame rate is evident from the frequency of fluctuations in power consumption as seen in this graph - during active processing of a frame, power consumption is 650 mW; in-between frames, it drops to 420 mW. Our average power is 611 mW, which is slightly lower than our demonstrated 624 mW due to having captured this trace when the FPGA was cold compared to an extended period of usage during the recording of our demonstration that allowed the chip to warm up to its stable temperature. At the higher warm temperature, resistive losses in the device rise as well, resulting in slightly higher overall power draw.
Work
Scales
Compute Ix,Iy
FPS
Overall power (mW)
[ 41 ]
-
-
2.1
4900
[ 55 ]
-
-
30
2000
[ 44 ]
-
-
>30
<2500
ours
2
yes
32.5
624
ours
2
no
32.5
562
ours
1
yes
32.5
489
Table S6 : Full system power consumption including image sensors, I/O, and FPGA, as reported in the literature and measured for several configurations of our method.
Scales
Ix,Iy
SV
WR (m)
MAE (m)
Core power (mW)
2
yes
yes
0.551
0.044
312
2
yes
no
0.311
0.058
280
2
no
yes
0.282
0.075
276
2
no
no
0.259
0.071
257
1
yes
yes
0.423
0.069
214
1
yes
no
0.210
0.076
195
Table S7 : Power-accuracy tradeoff for alternate algorithms. We report the working range, MAE, and core power in Tab. S7 .
Single-photon cameras based on single-photon avalanche diode (SPAD) technology are gaining popularity for 3D sensing, thanks to their extreme sensitivity and time resolution. There are two key challenges with single-photon cameras that limit their widespread use: (i) they suffer from non-linear distortions called ''pile-up'' when operated in high-photon-flux conditions, and (ii) they generate a large volume of raw photon data, creating a severe data bottleneck at each sensor pixel. In this work, we show that while compressive capture techniques successfully mitigate data transfer challenges, they exacerbate the effects of dead-time distortion because they fail to retain sufficient information about the photon detection history to allow post-processing pile-up correction via existing methods. We propose a new computational-imaging method that combines free-running capture with an analysis-by-synthesis software pipeline to mitigate pile-up distortions. Our results with hardware emulations and full-scene and single-pixel simulations show that our method can reliably capture scene distance and reflectance over a wide range of illumination conditions. Our work will enable high-resolution SPAD cameras that are severely bandwidth-constrained to operate in real-world high-flux scenarios.
Depth from Defocus (DfD) is the task of estimating a dense metric depth map from a focus stack. Unlike previous works overfitting to a certain dataset, this paper focuses on the challenging and practical setting of zero-shot generalization. We first propose a new real-world DfD benchmark ZEDD, which contains 8.3x more scenes and significantly higher quality images and ground-truth depth maps compared to previous benchmarks. We also design a novel network architecture named FOSSA. FOSSA is a Transformer-based architecture with novel designs tailored to the DfD task. The key contribution is a stack attention layer with a focus distance embedding, allowing efficient information exchange across the focus stack. Finally, we develop a new training data pipeline allowing us to utilize existing large-scale RGBD datasets to generate synthetic focus stacks. Experiment results on ZEDD and other benchmarks show a significant improvement over the baselines, reducing errors by up to 55.7%. The ZEDD benchmark is released at https://zedd.cs.princeton.edu. The code and checkpoints are released at https://github.com/princeton-vl/FOSSA.
Yiming Zuo, Hongyu Wen, Venkat Subramanian +5
Department of Computer Science, Princeton University
This work presents a lightweight super-resolution (LiteSR) neural network for depth and intensity images acquired from a consumer-grade single-photon avalanche diode (SPAD) array with a 48x32 spatial resolution. The proposed framework reconstructs high-resolution (HR) images of size 256x256. Both synthetic and real datasets are used for performance evaluation. Extensive quantitative metrics demonstrate high reconstruction fidelity on synthetic datasets, while experiments on real indoor and outdoor measurements further confirm the robustness of the proposed approach. Moreover, the SPAD sensor is interfaced with an Arduino UNO Q microcontroller, which receives low-resolution (LR) depth and intensity images and feeds them into a compressed, pre-trained deep learning (DL) model, enabling real-time SR video streaming. In addition to the 256x256 setting, a range of target HR resolutions is evaluated to determine the maximum achievable upscaling resolution (512x512) with LiteSR, including scenarios with noise-corrupted LR inputs. The proposed LiteSR-embedded system co-design provides a scalable, cost-effective solution to enhance the spatial resolution of current consumer-grade SPAD arrays to meet HR imaging requirements.
Zhenya Zang, Xingda Li, David Day Uei Li
Department of Biomedical Engineering, University of Strathclyde, 16 Richmond Street, Glasgow, G1 1XQ, United Kingdom