Depth from Defocus via Direct Optimization

Holly Jackson; Caleb Adams; Ignacio Lopez-Francos; Benjamin Recht

cs.CVFeb 18, 2026

Depth from Defocus via Direct Optimization

Authors: Holly Jackson, Caleb Adams, Ignacio Lopez-Francos, Benjamin Recht

Organizations: ⋆University of California, Berkeley, California, USA · † NASA Ames Research Center, Moffett Field, California, USA · ‡ KBR, Houston, Texas, USA

Abstract

Though there exists a reasonable forward model for blur based on optical physics, recovering depth from a collection of defocused images remains a computationally challenging optimization problem. In this paper, we show that with contemporary optimization methods and reasonable computing resources, a global optimization approach to depth from defocus is feasible. Our approach rests on alternating minimization. When holding the depth map fixed, the forward model is linear with respect to the all-in-focus image. When holding the all-in-focus image fixed, the depth at each pixel can be computed independently, enabling embarrassingly parallel computation. We show that alternating between convex optimization and parallel grid search can effectively solve the depth-from-defocus problem at higher resolutions than current deep learning methods. We demonstrate our approach on benchmark datasets with synthetic and real defocus blur and show promising results compared to prior approaches. Our code is available at github.com/hollyjackson/dfd.

Figures & tables

Figure 1: Example focal stack from Make3D [ 18 , 19 ] , shown with synthetic defocus blur at five focus distances.

Figure 2: Illustration of a thin lens with aperture diameter $D$ and focal length $f$ . Parallel rays from infinity focus at $f$ (green). A point at the focus distance ( $Z_{f}$ ) forms a sharp image on the sensor at distance $s$ (blue). Points away from the focus plane form a circle of confusion with blur diameter $b$ (orange).

Figure 3: Qualitative results on two NYUv2 examples. Top: ground-truth AIF images and depth maps. Bottom: reconstructed AIF images and depth maps. RMSEs (left to right) are 0.0731 and 0.0230.

Figure 4: Qualitative results on two Make3D examples. Top: ground-truth AIF images and resized depth maps. Bottom: reconstructed AIF images and depth maps. C2 RMSEs (left to right) are 5.439 and 0.243.

Supervised learning
Method	RMSE $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	$\delta_{2}\uparrow$	$\delta_{3}\uparrow$
DFF-FV [ 29 ]	0.285	0.470	0.956	0.979	0.988
DFF-DFV [ 29 ]	0.232	0.445	0.967	0.980	0.990
Self-supervised w/ AIF
Gur & Wolf [ 8 ]	0.649	0.184	0.720	0.887	0.951
Defocus-Net [ 11 ]	0.623	0.176	0.732	0.887	0.951

Table 1: Comparison of depth-from-defocus results on the NYUv2 test set [ 21 ] . Results for [ 14 , 23 , 8 , 11 ] are taken from [ 11 ] , and results for [ 20 , 29 ] from [ 20 ] . Lower is better for RMSE and AbsRel; higher is better for

\delta_{1}

\delta_{2}

, and

\delta_{3}

	C1		C2
Method	RMSE $\downarrow$	AbsRel $\downarrow$	RMSE $\downarrow$	AbsRel $\downarrow$
Self-supervised w/ AIF
Gur & Wolf [ 8 ]	7.474	0.262	9.248	0.269
Analytical
Ours	2.277	0.0663	5.630	0.0695

Table 2: Comparison of depth-from-defocus results on the Make3D test set. For [ 8 ] , we report their best RMSE (F6, rendered focal stack with six focus settings). Lower is better.

Figure 5: Reconstructed AIF images (left) and depth maps (right) for three scenes from the mobile phone dataset [ 23 ] .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S.1: Sparsity pattern of $A$ , which is constructed based on the depth map $Z$ and camera parameters. The example shown is for the Make3D data, where the (resized) images are 460 $\times$ 345. Here the maximum blur kernel size is set to 17 $\times$ 17.

Figure S.2: Example NYUv2 result with one of the highest RMSEs. (a) Ground-truth AIF image. (b) Ground-truth depth. (c) Reconstructed depth (RMSE = 1.397). (d) After post-processing (TV threshold = 0.4), reducing RMSE to 0.0604.

Figure S.3: Loss vs. alternating-minimization iteration for NYUv2 main-paper experiment (run for 40 epochs).

Figure S.4: Loss vs. alternating-minimization iteration for Make3D main-paper experiment (run for 5 epochs).

Figure S.5: Selection of qualitative results from the NYUv2 dataset. The top half of the examples (1 and 2) are from the test set, the bottom half of the examples (3 and 4) are from the train set. First row of each block shows the ground truth all-in-focus images and depth maps, and the second row of each block shows the reconstructed all-in-focus images and depth maps..

Figure S.6: Selection of qualitative results from the Make3D dataset. The top half of the examples (1 and 2) are from the test set, the bottom half of the examples (3 and 4) are from the train set. First row of each block shows the ground truth all-in-focus images and depth maps, and the second row of each block shows the reconstructed all-in-focus images and depth maps.

Supervised learning (DfD)
Method	RMSE $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	$\delta_{2}\uparrow$	$\delta_{3}\uparrow$
Song & Lee [ SongandLee2018 ]	0.154	0.028	-	-	-
Carvalho et al. [ 2 ]	0.144	0.036	-	-	-
Yang et al. (DFF-FV) [ 29 ]	0.285	0.470	0.956	0.979	0.988
Yang et al. (DFF-DFV) [ 29 ]	0.232	0.445	0.967	0.980	0.990
Supervised learning (mono. depth est.)

Appendix

Table S.1: Extended comparison of depth-from-defocus and monocular depth estimation results on the NYUv2 test set [ 21 ] . Results for [ 14 , 23 , 8 , 11 ] are recorded from [ 11 ] ; results for [ 20 , 29 ] from [ 20 ] ; results for [ Li2015 , Xu2017 , Fu2018 ] from [ 8 ] ; and results for [ Anwar2017 , Tang2021 , ChangandWetzstein2019 , Dong2022 , SongandLee2018 , 2 , Nazir2023 ] from [ Nazir2023 ] . Lower is better for RMSE and AbsRel; higher is better for

\delta_{1}

\delta_{2}

, and

\delta_{3}

Supervised learning
	C1		C2
Method	RMSE $\downarrow$	AbsRel $\downarrow$	RMSE $\downarrow$	AbsRel $\downarrow$
Li et al. [ Li2015 ]	7.120	0.278	10.27	0.279
Xu et al. (MS-CRF) [ Xu2017 ]	4.380	0.184	8.56	0.198
Fu et al. (DORN) [ Fu2018 ]	3.970	0.157	7.32	0.162
Nazir et al. [ Nazir2023 ]	4.178	0.153	6.132	0.170

Appendix

Table S.2: Extended comparison of depth-from-defocus and monocular depth estimation methods on the Make3D test set [ 18 , 19 ] . Results for [ Li2015 , 8 , Xu2017 , Fu2018 , Godard2017 , Zhou2017 , Wang2018 ] are recorded from [ 8 ] . For Gur and Wolf [ 8 ] , we report their best RMSE result (F6, rendered focal stack with six focus settings). Lower is better.

Figure S.7: Reconstructed all-in-focus images (top row) and depth maps (bottom row) for a selection of (vertically-oriented) mobile phone focal stacks from Suwajanakorn et al. [ 23 ] .

Figure S.8: Reconstructed all-in-focus images (top row) and depth maps (bottom row) for two (horizontally-oriented) mobile phone focal stacks (of a keyboard and rubber band balls) from Suwajanakorn et al. [ 23 ] .

Figure S.9: Reconstructed all-in-focus images (left column) and depth maps (right column) of three (aligned) focal stacks from Suwajanakorn et al. [ 23 ] .

Settings			Train Set					Test Set
$T_{0}$	$\alpha$	# epochs	RMSE $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	$\delta_{2}\uparrow$	$\delta_{3}\uparrow$	RMSE $\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	$\delta_{2}\uparrow$	$\delta_{3}\uparrow$
200	1.05	40	0.110	8.11e-3	0.992	0.994	0.995	0.109	8.37e-3	0.992	0.994	0.995
50	2	5	0.189	0.0198	0.985	0.990	0.992	0.190	0.0211	0.984	0.989	0.991
10	2	5	0.348	0.0394	0.964	0.981	0.985	0.333	0.0386	0.967	0.982	0.986

Appendix

Table S.3: Performance of our method on NYUv2 under different parameter settings.

T_{0}

is the initial maximum number of iterations for the Nesterov gradient method;

\alpha

is the multiplier that increases the number of Nesterov iterations at each alternating-minimization step; and # epochs is the number of alternating-minimization epochs.

Settings					Train Set				Test Set
			Windowed MSE		C1		C2		C1		C2
$T_{0}$	$\alpha$	#epochs	Enabled?	Size	RMSE $\downarrow$	AbsRel $\downarrow$	RMSE $\downarrow$	AbsRel $\downarrow$	RMSE $\downarrow$	AbsRel $\downarrow$	RMSE $\downarrow$	AbsRel $\downarrow$
10	2	5	No	–	2.487	0.0850	7.302	0.0971	2.432	0.0751	7.924	0.0881
10	2	5	Yes	5	2.405	0.0781	5.729	0.0824	2.277	0.0663	5.630	0.0695

Appendix

Table S.4: Effect of windowed MSE on optimization over Make3D dataset. We report train and test performance under the C1/C2 evaluation protocols. When enabled, windowed MSE averages the per-pixel reconstruction error over a local window of a user-defined size, improving accuracy.

Depth Estimation

Depth estimation infers scene distance from images and sensors, enabling 3D reconstruction, navigation, and spatial understanding.

See all 198 papers

Zero-Shot Depth from Defocus
Mar 27, 2026Yiming Zuo, Hongyu Wen, Venkat Subramanian +5Metric Depth Estimation Depth Estimation
Department of Computer Science, Princeton University
The RealDefocus Benchmark for Defocus Deblurring
Jul 23, 2026Tim Seizinger, Zhuyun Zhou, Radu TimofteImage Deblurring Image Restoration
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
FlashReFocus: Interactive Image Refocusing in Seconds
Oct 16, 2026Ching-Heng Cheng, Chia-Ming Lee, Ming-Ching Chang +3Neural Rendering Diffusion Models for Image Restoration
National Cheng Kung University · State University of New York at Albany · National Yang Ming Chiao Tung University

Mar 27, 2026J/K move · Enter open · S save

Zero-Shot Depth from Defocus

Yiming Zuo, Hongyu Wen, Venkat Subramanian, Patrick Chen+4

Department of Computer Science, Princeton University

Metric Depth Estimation Depth Estimation

Open paper AskPDF arXiv

Depth from Defocus (DfD) is the task of estimating a dense metric depth map from a focus stack. Unlike previous works overfitting to a certain dataset, this paper focuses on the challenging and practical setting of zero-shot generalization. We first propose a new real-world DfD benchmark ZEDD, which contains 8.3x more scenes and significantly higher quality images and ground-truth depth maps compared to previous benchmarks. We also design a novel network architecture named FOSSA. FOSSA is a Transformer-based architecture with novel designs tailored to the DfD task. The key contribution is a stack attention layer with a focus distance embedding, allowing efficient information exchange across the focus stack. Finally, we develop a new training data pipeline allowing us to utilize existing large-scale RGBD datasets to generate synthetic focus stacks. Experiment results on ZEDD and other benchmarks show a significant improvement over the baselines, reducing errors by up to 55.7%. The ZEDD benchmark is released at https://zedd.cs.princeton.edu. The code and checkpoints are released at https://github.com/princeton-vl/FOSSA.

Abstract

Depth Estimation

Explore similar work

Zero-Shot Depth from Defocus

The RealDefocus Benchmark for Defocus Deblurring

FlashReFocus: Interactive Image Refocusing in Seconds