Differentiable Task-Driven Coded Aperture Camera Design with 3D Gaussian Scene Representations

Chengyang Yan; Donald G. Dansereau

cs.CVOct 20, 2026

Differentiable Task-Driven Coded Aperture Camera Design with 3D Gaussian Scene Representations

Authors: Chengyang Yan, Donald G. Dansereau

Organizations: Australian Centre for Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney

Abstract

Camera design heavily influences perception task performance, yet cameras are typically designed by domain experts through costly experimentation, in isolation from the tasks they serve. Existing joint-design methods either rely on differentiable simulation limited to pre-captured 2D images, restricting them to optics or image-processing pipeline design, or use non-differentiable camera simulators optimised with derivative-free or reinforcement-learning methods that scale poorly with the number of design parameters. We present a differentiable framework for jointly designing complex camera systems and downstream perception tasks. We use it to design two coded-aperture depth-from-defocus cameras, a monocular and a stereo configuration, by jointly optimising the coded aperture mask, camera parameters, and a depth estimation network end-to-end. Our framework represents scenes with 3D Gaussian Splatting reconstructed from real-world captures, providing a differentiable image formation process for gradient-based optimisation over design spaces intractable for prior differentiable approaches. Results demonstrate that our differentiable method improves over existing coded aperture design and non-differentiable camera design baselines, offering a step forward in automated and principled camera design for autonomous systems.

Figures & tables

Figure 1: We introduce a differentiable method that jointly optimises camera parameters and perception tasks, using 3DGS scenes reconstructed from real-world captures and a coded aperture simulation modelling depth-dependent optical blur. Camera parameters and the perception network are jointly updated via gradient-based optimisation of the task loss (red arrows), spanning geometric parameters (FoV, pixel count, multi-camera configurations) and high-dimensional optical parameters (coded aperture mask).

Figure 2: A camera captures the real-world scene (blue arrow), which 3DGS models as a 3D Gaussian representation. At each step, this representation is rendered with the current geometric camera parameters into pinhole RGB images and depth maps; the coded aperture simulation then convolves these RGB renders with depth-dependent PSFs computed from the coded mask, focus distance, and f-number via a physical optics model, followed by a physics-based noise model. The resulting renders are fed to the perception task network, and the task loss $l_{\text{task}}$ is backpropagated through both stages (red arrows) to jointly update all learnable parameters (green text).

Scenario	Method	Camera Parameters						Performance
		Coded Mask	Focus Dist.	f-num	Focal Length	Sensor Size	Pixel Size	All-in-Focus			Depth
		$M$	focus ( $m$ )	$f$	focal (mm)	$w\times h$ (mm)	$p$ ( $\mu$ m)	PSNR (dB) $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	RMSE (m) $\downarrow$	MAE (m) $\downarrow$	$\text{log}_{10}$ $\downarrow$	$\delta<1.25$ $\uparrow$
Noise $\times$ 100	Fixed Cam. + Phase + Ikoma et al. [ 16 ]	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	1.70 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	1.8 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.29 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.47 $\times$ 9.94 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	9.00 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	22.587	0.669	0.308	0.131	0.080	0.102	0.910
	Fixed Cam. + Amp. + Ikoma et al. [ 16 ]	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	1.70 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	1.8 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.29 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.47 $\times$ 9.94 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	9.00 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	23.543	0.696	0.292	0.126	0.080	0.098	0.917
	TaCOS + Amp. + Proposed Model	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	0.99 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	11.9 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	24.95 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.44 $\times$ 9.90 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	4.50 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	24.494	0.767	0.401	0.169	0.115	0.131	0.868
	Ours - Optim. Cam. + Amp. + Proposed Model	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	0.45 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	12.4 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	12.37 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.47 $\times$ 9.94 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	9.00 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	29.301	0.893	0.102	0.110	0.070	0.091	0.938

Table 1: Quantitative results for monocular camera design, comparing our joint camera, mask, and perception optimisation against a fixed-camera baseline (Ikoma et al. [ 16 ] , tested with their original Phase mask and with an Amp. (amplitude) mask matched to our settings) and a derivative-free joint optimiser (TaCOS [ 44 ] ). Optimised parameters are marked

\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{

\displaystyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\textstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptscriptstyle\bullet

}}}}}

, fixed parameters

\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{

\displaystyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\textstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptscriptstyle\bullet

}}}}}

; our method outperforms both baselines under two noise scenarios.

Figure 3: Qualitative comparison for the monocular experiment. Our jointly optimised camera, mask, and perception model produce sharp AiF images and accurate depth, while the fixed-camera baseline (Ikoma et al. [ 16 ] ) and the derivative-free joint optimiser (TaCOS [ 44 ] ) degrade noticeably in blurry regions.

Scenario	Method	Camera Parameters				Performance
		Coded Mask	Focus Dist.	f-num	Baseline	All-in-Focus			Depth
		$M$	focus ( $m$ )	$f$	$b$ (m)	PSNR (dB) $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	RMSE (m) $\downarrow$	MAE (m) $\downarrow$	$\text{log}_{10}$ $\downarrow$	$\delta<1.25$ $\uparrow$
Noise $\times$ 100	Fixed Cam. + CADS [ 12 ]	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	0.29 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	9.87 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	NA	26.397	0.759	0.301	0.244	0.135	0.188	0.797
	Ours - Monocular	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	0.29 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	16.0 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	NA	26.335	0.819	0.205	0.089	0.058	0.114	0.912
	Ours - Stereo	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	0.26/0.89 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	16.0/11.3 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	0.01 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	26.828	0.841	0.161	0.084	0.049	0.086	0.933
Noise $\times$ 200	Fixed Cam. + CADS [ 12 ]	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	0.28 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	9.87 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	NA	25.618	0.718	0.350	0.259	0.143	0.203	0.775

Table 2: Quantitative results for stereo camera design. The monocular row is our own strongest monocular configuration evaluated under the same narrow FoV, isolating the benefit of a second view; CADS [ 12 ] is the closest existing coded-aperture baseline, included despite its dual-pixel baseline being limited to the sub-mm aperture scale. Optimised parameters are marked

\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{

\displaystyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\textstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptscriptstyle\bullet

}}}}}

, fixed

\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{

\displaystyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\textstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptscriptstyle\bullet

}}}}}

; stereo values are given as Left / Right (CADS shares parameters across views). Our stereo method outperforms both, especially in AiF reconstruction.

Figure 4: Our method fuses photometric cues from a foreground-focused and a background-focused view for higher AiF quality and improves depth accuracy by leveraging disparity. Full comparisons with baselines are presented in the supplementary material.

Scenario	Method	Camera Parameters							Performance
		Coded Mask	Focus Dist.	f-num	Focal Length	Sensor Size	Pixel Size	Baseline	All-in-Focus			Depth
		$M$	focus ( $m$ )	$f$	focal (mm)	$w\times h$ (mm)	$p$ ( $\mu$ m)	$b$ (m)	PSNR (dB) $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	RMSE (m) $\downarrow$	MAE (m) $\downarrow$	$\text{log}_{10}$ $\downarrow$	$\delta<1.25$ $\uparrow$
Monocular	Fixed Mask	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	0.56 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	5.0 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	12.96 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.47 $\times$ 9.94 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	9.00 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	–	27.575	0.838	0.153	0.124	0.079	0.098	0.925
	Fixed Camera	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	1.70 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	1.8 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.29 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.47 $\times$ 9.94 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	9.00 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	–	24.397	0.740	0.290	0.124	0.081	0.095	0.920
	No Freq.	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	0.45 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	12.4 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.87 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.47 $\times$ 9.94 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	9.00 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	–	29.123	0.895	0.100	0.125	0.081	0.101	0.925
	Proposed	$\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	0.45 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	12.4 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	12.37 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	14.47 $\times$ 9.94 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	9.00 $\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{$ \displaystyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \textstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptstyle\bullet $}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{$ \scriptscriptstyle\bullet $}}}}}$	–	29.301	0.893	0.102	0.110	0.070	0.091	0.938

Table 3: Ablation study under Noise

\times

100. Monocular: a fixed human-designed mask [ 21 ] , a fixed camera (Ikoma et al. [ 16 ] configuration), and no frequency injection. Stereo: shared camera parameters, a shared mask, a conventional (uncoded) circular aperture, and no right-view warping. Optimised parameters are marked

\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{

\displaystyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\textstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptscriptstyle\bullet

}}}}}

, fixed

\mathchoice{\mathbin{\vbox{\hbox{\scalebox{2}{

\displaystyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\textstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptstyle\bullet

}}}}}{\mathbin{\vbox{\hbox{\scalebox{2}{

\scriptscriptstyle\bullet

}}}}}

; ‘–’ marks a parameter not applicable to that scenario. The results confirm that each component or design choice is a meaningful contributor to the proposed design.

Figure 5: Prototype of the optical system used for real-world validation and the real-world stereo results.

Figure 7: The stereo depth-from-defocus method. Per-view AiF proxy images are synthesised by fusing the Tikhonov-deconvolved focal stack using a differentiable Laplacian sharpness measure and passed to a frozen RAPIDFlow [ 29 ] network to estimate dense disparity. The disparity is then used to warp the right-view into the left-view coordinate frame via bilinear grid sampling. The left-view focal stack, the warped right-view focal stack, and a depth map derived from the disparity estimate are concatenated and fed into the monocular U-Net backbone with frequency injection, yielding the reconstructed AiF image and depth map for the left view.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Qualitative comparison for the monocular experiment under the Noise $\times$ 200 scenario. Our jointly optimised camera, mask, and perception model again produce sharp AiF images and accurate depth, while the fixed-camera baseline (Ikoma et al. [ 16 ] ) and the derivative-free joint optimiser (TaCOS [ 44 ] ) produce images with more defocus blur and lower depth accuracy in these regions.

Figure 9: Qualitative comparison of the proposed stereo method with a coded aperture dual-pixel camera method (CADS [ 12 ] ) and the monocular alternative. Our method achieves better AiF reconstruction by combining varying focus planes across the two views, and better depth estimation by exploiting disparity as an additional cue.

Figure 10: (a) The optimised coded aperture masks in both monocular and stereo experiments, the manufcatured masks, and their measured PSFs. (b) The captured coded aperture image, reconstructed AiF image, and estimated depth map for the monocular setup. The AiF and depth recovery on real hardware show plausible outcomes but with modest accuracy gap attributable to the sim-to-real domain shift and manufacturing tolerances.

Computational Imaging

Computational imaging combines optical and sensor design with algorithms to recover or infer scene information from indirect, compressed, or noisy measurements.

See all 123 papers

Beyond Bayer: Task-Optimal Sensor Co-Design for Robust Autonomous-Driving Segmentation
Jun 23, 2026Reeshad Khan, John GauchAutonomous Driving Perception Computational Imaging
University Of Arkansas, Fayetteville AR 72701, USA
EDoF-NeRF: extended depth-of-field neural radiance fields using a coded aperture camera
Jun 17, 2026Yoshiyuki Shirasaki, Ryoichi HorisakiNeural Rendering Neural Radiance Fields
Department of Information Physics and Computing, Graduate School of Information Science and Technology, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan
NavCrafter: Exploring 3D Scenes from a Single Image
Apr 3, 2026Hongbo Duan, Peiyu Zhuang, Yi Liu +5Novel View Synthesis Single-View 3D Reconstruction
Center for Artificial Intelligence and Robotics, Shenzhen International Graduate School, Tsinghua University, Shenzhen 518055, China · Peng Cheng Laboratory, 518108, China · School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University, China +1

Jun 23, 2026J/K move · Enter open · S save

Beyond Bayer: Task-Optimal Sensor Co-Design for Robust Autonomous-Driving Segmentation

Reeshad Khan, John Gauch

University Of Arkansas, Fayetteville AR 72701, USA

Autonomous Driving Perception Computational Imaging Hardware-Software Co-Design Semantic Segmentation

Open paper AskPDF arXiv

Robust perception underpins autonomous driving, and most recent progress comes from scaling the model-larger backbones, foundation models, and cooperative multi-agent fusion. We pursue a complementary, upstream question: what should the camera itself measure? Using a differentiable RAW-to-task pipeline, we decompose which sensor degrees of freedom benefit dense prediction. Learning the spectral colour-filter-array (CFA) weights is the dominant lever, improving mIoU by +0.017 (KITTI-360) and +0.023 (ACDC) over a fixed camera. In contrast, point-spread-function (optics) co-design is net-negative (-0.020 mIoU on KITTI-360) - a consequence of the data-processing inequality, which also bounds the task information that any downstream model, however large or cooperative, can recover. Noise co-optimisation is marginal, and counter to intuition enlarging the CFA tile beyond 2x2 consistently hurts, as the filters are confined to the rank three sRGB input. Because the intervention is at the sensor, the gains are model-agnostic; we validate robustness on ACDC's fog, night, rain, and snow, and conclude with a simple recipe: learn the 2x2 CFA weights and keep an identity PSF.

Abstract

Computational Imaging

Explore similar work

Beyond Bayer: Task-Optimal Sensor Co-Design for Robust Autonomous-Driving Segmentation

EDoF-NeRF: extended depth-of-field neural radiance fields using a coded aperture camera

NavCrafter: Exploring 3D Scenes from a Single Image