Authors: Holly Jackson, Caleb Adams, Ignacio Lopez-Francos, Benjamin Recht
Organizations: ⋆University of California, Berkeley, California, USA · † NASA Ames Research Center, Moffett Field, California, USA · ‡ KBR, Houston, Texas, USA
Though there exists a reasonable forward model for blur based on optical physics, recovering depth from a collection of defocused images remains a computationally challenging optimization problem. In this paper, we show that with contemporary optimization methods and reasonable computing resources, a global optimization approach to depth from defocus is feasible. Our approach rests on alternating minimization. When holding the depth map fixed, the forward model is linear with respect to the all-in-focus image. When holding the all-in-focus image fixed, the depth at each pixel can be computed independently, enabling embarrassingly parallel computation. We show that alternating between convex optimization and parallel grid search can effectively solve the depth-from-defocus problem at higher resolutions than current deep learning methods. We demonstrate our approach on benchmark datasets with synthetic and real defocus blur and show promising results compared to prior approaches. Our code is available at github.com/hollyjackson/dfd.
Figures & tables
Figure 1: Example focal stack from Make3D [ 18 , 19 ] , shown with synthetic defocus blur at five focus distances.
Figure 2: Illustration of a thin lens with aperture diameter D and focal length f . Parallel rays from infinity focus at f (green). A point at the focus distance ( Zf ) forms a sharp image on the sensor at distance s (blue). Points away from the focus plane form a circle of confusion with blur diameter b (orange).
Figure 3: Qualitative results on two NYUv2 examples. Top: ground-truth AIF images and depth maps. Bottom: reconstructed AIF images and depth maps. RMSEs (left to right) are 0.0731 and 0.0230.
Figure 4: Qualitative results on two Make3D examples. Top: ground-truth AIF images and resized depth maps. Bottom: reconstructed AIF images and depth maps. C2 RMSEs (left to right) are 5.439 and 0.243.
Method
RMSE ↓
AbsRel ↓
δ1↑
δ2↑
δ3↑
Supervised learning
DFF-FV [ 29 ]
0.285
0.470
0.956
0.979
0.988
DFF-DFV [ 29 ]
0.232
0.445
0.967
0.980
0.990
Self-supervised w/ AIF
Gur & Wolf [ 8 ]
0.649
0.184
0.720
0.887
0.951
Defocus-Net [ 11 ]
0.623
0.176
0.732
0.887
0.951
Table 1: Comparison of depth-from-defocus results on the NYUv2 test set [ 21 ] . Results for [ 14 , 23 , 8 , 11 ] are taken from [ 11 ] , and results for [ 20 , 29 ] from [ 20 ] . Lower is better for RMSE and AbsRel; higher is better for δ1 , δ2 , and δ3 .
C1
C2
Method
RMSE ↓
AbsRel ↓
RMSE ↓
AbsRel ↓
Self-supervised w/ AIF
Gur & Wolf [ 8 ]
7.474
0.262
9.248
0.269
Analytical
Ours
2.277
0.0663
5.630
0.0695
Table 2: Comparison of depth-from-defocus results on the Make3D test set. For [ 8 ] , we report their best RMSE (F6, rendered focal stack with six focus settings). Lower is better.
Figure 5: Reconstructed AIF images (left) and depth maps (right) for three scenes from the mobile phone dataset [ 23 ] .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S.1: Sparsity pattern of A , which is constructed based on the depth map Z and camera parameters. The example shown is for the Make3D data, where the (resized) images are 460 × 345. Here the maximum blur kernel size is set to 17 × 17.
Figure S.2: Example NYUv2 result with one of the highest RMSEs. (a) Ground-truth AIF image. (b) Ground-truth depth. (c) Reconstructed depth (RMSE = 1.397). (d) After post-processing (TV threshold = 0.4), reducing RMSE to 0.0604.
Figure S.3: Loss vs. alternating-minimization iteration for NYUv2 main-paper experiment (run for 40 epochs).
Figure S.4: Loss vs. alternating-minimization iteration for Make3D main-paper experiment (run for 5 epochs).
Figure S.5: Selection of qualitative results from the NYUv2 dataset. The top half of the examples (1 and 2) are from the test set, the bottom half of the examples (3 and 4) are from the train set. First row of each block shows the ground truth all-in-focus images and depth maps, and the second row of each block shows the reconstructed all-in-focus images and depth maps..
Figure S.6: Selection of qualitative results from the Make3D dataset. The top half of the examples (1 and 2) are from the test set, the bottom half of the examples (3 and 4) are from the train set. First row of each block shows the ground truth all-in-focus images and depth maps, and the second row of each block shows the reconstructed all-in-focus images and depth maps.
Method
RMSE ↓
AbsRel ↓
δ1↑
δ2↑
δ3↑
Supervised learning (DfD)
Song & Lee [ SongandLee2018 ]
0.154
0.028
-
-
-
Carvalho et al. [ 2 ]
0.144
0.036
-
-
-
Yang et al. (DFF-FV) [ 29 ]
0.285
0.470
0.956
0.979
0.988
Yang et al. (DFF-DFV) [ 29 ]
0.232
0.445
0.967
0.980
0.990
Supervised learning (mono. depth est.)
Appendix
Table S.1: Extended comparison of depth-from-defocus and monocular depth estimation results on the NYUv2 test set [ 21 ] . Results for [ 14 , 23 , 8 , 11 ] are recorded from [ 11 ] ; results for [ 20 , 29 ] from [ 20 ] ; results for [ Li2015 , Xu2017 , Fu2018 ] from [ 8 ] ; and results for [ Anwar2017 , Tang2021 , ChangandWetzstein2019 , Dong2022 , SongandLee2018 , 2 , Nazir2023 ] from [ Nazir2023 ] . Lower is better for RMSE and AbsRel; higher is better for δ1 , δ2 , and δ3 .
C1
C2
Method
RMSE ↓
AbsRel ↓
RMSE ↓
AbsRel ↓
Supervised learning
Li et al. [ Li2015 ]
7.120
0.278
10.27
0.279
Xu et al. (MS-CRF) [ Xu2017 ]
4.380
0.184
8.56
0.198
Fu et al. (DORN) [ Fu2018 ]
3.970
0.157
7.32
0.162
Nazir et al. [ Nazir2023 ]
4.178
0.153
6.132
0.170
Appendix
Table S.2: Extended comparison of depth-from-defocus and monocular depth estimation methods on the Make3D test set [ 18 , 19 ] . Results for [ Li2015 , 8 , Xu2017 , Fu2018 , Godard2017 , Zhou2017 , Wang2018 ] are recorded from [ 8 ] . For Gur and Wolf [ 8 ] , we report their best RMSE result (F6, rendered focal stack with six focus settings). Lower is better.
Figure S.7: Reconstructed all-in-focus images (top row) and depth maps (bottom row) for a selection of (vertically-oriented) mobile phone focal stacks from Suwajanakorn et al. [ 23 ] .
Figure S.8: Reconstructed all-in-focus images (top row) and depth maps (bottom row) for two (horizontally-oriented) mobile phone focal stacks (of a keyboard and rubber band balls) from Suwajanakorn et al. [ 23 ] .
Figure S.9: Reconstructed all-in-focus images (left column) and depth maps (right column) of three (aligned) focal stacks from Suwajanakorn et al. [ 23 ] .
Settings
Train Set
Test Set
T0
α
# epochs
RMSE ↓
AbsRel ↓
δ1↑
δ2↑
δ3↑
RMSE ↓
AbsRel ↓
δ1↑
δ2↑
δ3↑
200
1.05
40
0.110
8.11e-3
0.992
0.994
0.995
0.109
8.37e-3
0.992
0.994
0.995
50
2
5
0.189
0.0198
0.985
0.990
0.992
0.190
0.0211
0.984
0.989
0.991
10
2
5
0.348
0.0394
0.964
0.981
0.985
0.333
0.0386
0.967
0.982
0.986
Appendix
Table S.3: Performance of our method on NYUv2 under different parameter settings. T0 is the initial maximum number of iterations for the Nesterov gradient method; α is the multiplier that increases the number of Nesterov iterations at each alternating-minimization step; and # epochs is the number of alternating-minimization epochs.
Settings
Train Set
Test Set
Windowed MSE
C1
C2
C1
C2
T0
α
#epochs
Enabled?
Size
RMSE ↓
AbsRel ↓
RMSE ↓
AbsRel ↓
RMSE ↓
AbsRel ↓
RMSE ↓
AbsRel ↓
10
2
5
No
–
2.487
0.0850
7.302
0.0971
2.432
0.0751
7.924
0.0881
10
2
5
Yes
5
2.405
0.0781
5.729
0.0824
2.277
0.0663
5.630
0.0695
Appendix
Table S.4: Effect of windowed MSE on optimization over Make3D dataset. We report train and test performance under the C1/C2 evaluation protocols. When enabled, windowed MSE averages the per-pixel reconstruction error over a local window of a user-defined size, improving accuracy.
Depth from Defocus (DfD) is the task of estimating a dense metric depth map from a focus stack. Unlike previous works overfitting to a certain dataset, this paper focuses on the challenging and practical setting of zero-shot generalization. We first propose a new real-world DfD benchmark ZEDD, which contains 8.3x more scenes and significantly higher quality images and ground-truth depth maps compared to previous benchmarks. We also design a novel network architecture named FOSSA. FOSSA is a Transformer-based architecture with novel designs tailored to the DfD task. The key contribution is a stack attention layer with a focus distance embedding, allowing efficient information exchange across the focus stack. Finally, we develop a new training data pipeline allowing us to utilize existing large-scale RGBD datasets to generate synthetic focus stacks. Experiment results on ZEDD and other benchmarks show a significant improvement over the baselines, reducing errors by up to 55.7%. The ZEDD benchmark is released at https://zedd.cs.princeton.edu. The code and checkpoints are released at https://github.com/princeton-vl/FOSSA.
Yiming Zuo, Hongyu Wen, Venkat Subramanian +5
Department of Computer Science, Princeton University
Single-Image Defocus Deblurring (SIDD) aims to recover an all-in-focus image from a single defocused observation, but rigorous and reproducible evaluation remains challenging due to the scarcity of realistic, high-resolution datasets with well-aligned defocused/sharp pairs and standardized protocols. We build on RealDefocus, a benchmark derived from the real-world RealBokeh dataset originally proposed for Bokeh Rendering. RealDefocus provides paired defocused inputs and sharp ground truth images, predefined training/validation/test splits, and a unified evaluation framework for comparing image restoration and neural rendering approaches. We further outline a benchmarking protocol with cross-dataset validation to assess reconstruction quality and generalization. The project page is publicly available at: www.github.com/TimSeizinger/RealDefocus-Benchmark.
Tim Seizinger, Zhuyun Zhou, Radu Timofte
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
Single-image refocusing changes the focal plane and depth of field of a photograph after capture. It is inherently interactive: users try many focal points and blur strengths on one image, so each edit should be fast and change only the blur. Recent generative methods, however, rerun large diffusion models for every edit. We observe that the two sub-problems differ in kind: recovering the all-in-focus image is ill-posed and needs a generative prior, whereas rendering blur is largely determined by depth and the focal plane. We therefore propose FlashReFocus, which runs diffusion once per photograph and renders every edit with a lightweight network. To address the ill-posed restoration, a one-step diffusion deblurrer with a decoder refiner recovers fine detail efficiently while balancing fidelity and perceptual quality. For precise control, the renderer is conditioned on an explicit defocus map and uses defocus-bias attention, which lets pixels at similar defocus share information without interference from pixels at other depths; in-focus content stays unchanged, and editing the map yields tilt-shift and two focal planes. To further test generalization to unseen cameras, we collect 3CReal, a three-camera benchmark of 89 pairs. FlashReFocus achieves the best perceptual fidelity for zero-shot bokeh rendering on EBB! and 3CReal, refocuses 148.3x faster than GenRefocus, and takes about half a second per further edit, 462x faster.