OmniRay: Camera-Agnostic Monocular 3D Reconstruction with Composite Ray Fields

Namitha Guruprasad; Abhay Yadav; Cheng Peng; Rama Chellappa; Anand Bhattad

cs.CVOct 22, 2026

OmniRay: Camera-Agnostic Monocular 3D Reconstruction with Composite Ray Fields

Authors: Namitha Guruprasad, Abhay Yadav, Cheng Peng, Rama Chellappa, Anand Bhattad

Organizations: Johns Hopkins University · University of Virginia

Abstract

Monocular models recover 3D geometry from a single image in one pass, but most assume a pinhole camera and produce distorted geometry on the wide-angle imagery produced by fisheye and panoramic lenses. Camera-aware models remove this assumption by predicting a per-pixel ray field. However, they represent different camera geometries using a single fixed basis to capture both the overall camera projection and its finer variations. We argue that camera geometry can be viewed as having a hierarchy of complexity for representing rays. We present OmniRay, a feed-forward model whose Ray Module is composed of three components: a monotone radial base that captures radially symmetric structure, a spherical-harmonic residual for smooth deviations from it, and a free-form residual for remaining local variation. These components are composed as successive rotations on the sphere, allowing the representation to accommodate diverse camera geometries. Given a single image, OmniRay predicts a ray direction and an up-to-scale radial distance for every pixel, from which a 3D point cloud is recovered without needing camera information at inference time. Extensive experiments across different camera configurations demonstrate that OmniRay establishes a new state-of-the-art in camera-agnostic monocular 3D reconstruction.

Figures & tables

Figure 1: OmniRay reconstructs 3D geometry from a single image across diverse camera geometries. A single model predicts an up-to-scale, camera-frame point cloud without camera intrinsics or camera-type input. Columns show inputs and reconstructions for pinhole, GoPro, fisheye, catadioptric, and panoramic cameras, respectively. Note that the GoPro and catadioptric camera configurations are withheld from training.

Figure 2: OmniRay Overview. Given a single image, a DINOv2 ViT-L/14 encoder produces global and dense features that drive two branches. The Ray Module progressively builds a camera-agnostic ray field from a monotone radial base, a spherical-harmonic residual, and a free-form residual. The Distance Module predicts a positive radial distance for each pixel. Combining the final ray $\mathbf{d}$ and distance $\rho$ yields the local pointmap $\mathbf{P}=\rho\,\mathbf{d}$ , recovering 3D geometry up to scale without camera intrinsics or a camera-type label.

Figure 3: Why combine the radial construction with the residual corrections? (a) The radial representation gives the same incidence angle to pixels that lie at the same radius in the base coordinates. (b) In the centered $2{:}1$ equirectangular projection, point A is closer to the center of the image than point B, yet has a higher incidence angle. A monotone radial function cannot accommodate such ordering under fixed coordinate transformation and orientation. (c) The spherical-harmonic and free-form residuals improve upon the base direction via iterative correction on the unit sphere.

		Angular error $({}^{\circ})\downarrow$
Representation	#Params	Fisheye	Equirectangular	Catadioptric
Pinhole	6	5.731	48.080	50.434
Double sphere	8	0.021	22.932	41.876
Radial ( $K=16$ )	24	0.002	22.802	0.104
Radial ( $K=32$ )	40	0.002	22.801	0.023
SH ( $L=7$ )	195	0.276	0.641	0.295

Table 1: Oracle representation fitting. Mean angular error in degrees after fitting each representation directly to ground-truth rays; lower is better. #Params denotes parameters per camera; the SH row adds three rotation parameters to the

3(L+1)^{2}=192

coefficients for the base direction, which the residual inherits from the radial base. Best results among compact representations are bold . The free field is shown separately as a high-capacity reference.

	Seen cameras		Held-out cameras				Held-out domains
	Aggregate		Catadioptric †		GoPro †		PanoCity †		CO3Dv2 †
Method	Ang. $\,({}^{\circ})\downarrow$	CD $\downarrow$	Ang. $\,({}^{\circ})\downarrow$	CD $\downarrow$	Ang. $\,({}^{\circ})\downarrow$	CD $\downarrow$	Ang. $\,({}^{\circ})\downarrow$	CD $\downarrow$	Ang. $\,({}^{\circ})\downarrow$	CD $\downarrow$
GeoCalib ( Veicht et al., 2024 )	30.997	–	119.662	–	14.126	–	47.132	–	3.713	–
AnyCalib ( Tirado-Garín & Civera, 2025 )	28.655	–	107.129	–	10.031	–	42.335	–	2.711	–
DA3 ( Lin et al., 2025a )	33.028	0.1999	121.566	1.0412	16.637	0.1097	50.533	0.3677	2.041	0.0288
UniK3D ( Piccinelli et al., 2025 )	6.336	0.0760	75.702	0.5948	6.342	0.0454	21.220	0.2120	2.676	0.0277

Table 2: Main benchmark. Results on seen cameras, held-out cameras, and held-out domains. Ang. denotes mean ray error in degrees; CD denotes median relative Chamfer distance, normalized by each scene’s mean point norm. Metrics are macro-averaged over test images, and all methods use per-image median scale alignment. Lower is better; best results are bold . AUG denotes random-lens augmentation, and RG denotes ray guidance.

\dagger

denotes exclusion from our training data; a dash indicates that the method does not predict scene geometry. Full per-camera and per-dataset results and additional metrics are provided in Sec. A.3 .

Figure 4: Qualitative reconstructions on unseen datasets. We compare OmniRay with Depth Anything 3 and UniK3D on PanoCity and CO3Dv2 across five camera configurations. Small visualizations show viewing-ray fields; larger renderings show the corresponding point clouds, with ground truth in the rightmost column. OmniRay better preserves scene structure in the panoramic and catadioptric examples without camera intrinsics or camera-type input.

Figure 5: Ablation. (a) Held-out catadioptric camera: predicted ray field and point cloud for UniK3D, OmniRay trained without random-lens augmentation, and the full model, against the ground truth. (b) Panoramic image: the radial base alone covers only part of the sphere (orange) and leaves the reconstruction torn, while the spherical-harmonic and free-form residuals complete the field (blue) and close the geometry.

Figure 6: In-the-wild reconstructions. Qualitative reconstructions on photographs collected from the internet, shown in order as a panorama, a fisheye image, and a pinhole image. Although these camera families are represented during training, the images come from different camera instances with unknown intrinsics and lens parameters. OmniRay requires no camera metadata at inference.

		Seen cameras			Held-out config †
Ray representation	Training	Panorama	Fisheye	Pinhole	Catadioptric	GoPro
Radial base	–	21.825	1.761	0.788	6.039	5.377
Spherical base	–	5.088	1.540	0.790	13.327	5.962
Free-form field	–	0.783	1.508	0.621	11.335	5.191
Radial + SH residual	Joint	0.796	1.784	0.581	1.879	6.566
Radial + SH residual	Staged	0.610	1.545	0.675	1.680	3.393

Table 3: Ray representation ablation. Mean angular ray error in degrees (

\downarrow

) at native resolution. Seen camera families are evaluated on held-out configurations;

\dagger

denotes configurations excluded from training. Only the Ray Module is trained and evaluated. Best results are bold .

Figure 7: Limitation: zero-shot in-the-wild reconstruction of an aerial panorama sourced from the internet. With the horizon near the top of the image, the predicted rays cover a bounded, forward-facing region.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Random-lens augmentation. Images rendered from a single panorama under cameras drawn from the sampling distribution of Sec. 4.2 . A camera is a viewing direction, a monotone radial profile and an imaged sensor region, so one family spans narrow perspective crops, wide-angle and fisheye views, off-center and anisotropic sensors, and annular domains. The projection varies continuously from sample to sample, so no fixed map from image position to ray direction fits more than one of them. Draws falling close to either held-out geometry are rejected and never enter training.

Figure 9: Qualitative reconstructions on seen dataset: 2D3DS

Figure 10: Qualitative reconstructions on seen dataset: 360Loc

Figure 11: Additional in-the-wild reconstructions.

		Ray	Radial Dist.		point cloud Quality
Camera	Method	ang ${}^{\circ}\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	CD $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	F@10cm $\uparrow$
pinhole	GeoCalib	2.196	–	–	–	–	–	–
	AnyCalib	1.043	–	–	–	–	–	–
	DA3	6.604	0.1508	0.862	0.0701	0.0862	0.0540	0.300
	UniK3D	3.619	0.1371	0.871	0.1058	0.1802	0.0315	0.356
	ours	0.155	0.0769	0.925	0.0242	0.0355	0.0130	0.533

Appendix

Table 4: Per-camera results, all metrics. Macro-averaged over test images. † geometry withheld from training. Baselines are median-scale-aligned per image. Acc, Comp and CD are medians of the relative distance, divided by each scene’s mean point norm.

\delta_{1}

\delta<1.25

. F@10cm is the point-cloud F-score at a

10

cm distance threshold, the harmonic mean of precision and recall over nearest-neighbor distances. A dash means the method predicts no scene geometry.

		Ray	Radial Dist.		point cloud Quality
Scenes	Method	ang ${}^{\circ}\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	CD $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	F@10cm $\uparrow$
2D3DS	GeoCalib	40.713	–	–	–	–	–	–
	AnyCalib	37.520	–	–	–	–	–	–
	DA3	43.775	0.2124	0.747	0.3614	0.1744	0.5483	0.224
	UniK3D	19.432	0.1272	0.864	0.1487	0.0981	0.1994	0.411
	ours	1.619	0.0568	0.958	0.0236	0.0226	0.0246	0.767

Appendix

Table 5: Per-dataset results, all metrics. Macro-averaged over test images. † unseen in training. Conventions as in Table 4 . CO3Dv2 contains perspective and fisheye imagery only, so its zero-shot axis is the domain rather than the camera, and its object-masked COLMAP ground truth covers roughly

12\%

of pixels.

		Ray	Radial Dist.		point cloud Quality
Split	Method	ang ${}^{\circ}\downarrow$	AbsRel $\downarrow$	$\delta_{1}\uparrow$	CD $\downarrow$	Acc $\downarrow$	Comp $\downarrow$	F@10cm $\uparrow$
all	GeoCalib	45.356	–	–	–	–	–	–
	AnyCalib	40.625	–	–	–	–	–	–
	DA3	47.457	0.2272	0.716	0.3501	0.2394	0.4608	0.091
	UniK3D	20.211	0.1974	0.785	0.1736	0.1718	0.1754	0.193
	ours	1.414	0.0701	0.940	0.0212	0.0240	0.0184	0.499

Appendix

Table 6: Aggregate results over the

4500

-sample benchmark. Conventions as in Table 4 . Seen cameras are perspective, fisheye and panoramic; the held-out geometries are the catadioptric mirror and the GoPro fisheye.

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 955 papers

OmniPoint: Universal Monocular Metric Pointcloud from Any Camera
Sep 8, 2026Botao Ye, Marc Pollefeys, Ming-Hsuan Yang +1Panoramic 3D Reconstruction 3D Reconstruction
Google DeepMind · ETH Zurich · Work done as an intern at Google DeepMind.
CAM3R: Camera-Agnostic Model for 3D Reconstruction
Mar 23, 2026Namitha Guruprasad, Abhay Yadav, Cheng Peng +13D Reconstruction Multi-View 3D Reconstruction
Johns Hopkins University, Baltimore, MD 21218, USA · University of Virginia, Charlottesville, VA 22904, USA
Any3R: Feedforward 3D Reconstruction from Any Camera Type
Oct 26, 2026Jingong Chen, Qingwen Zhang, Youming Deng +53D Reconstruction Multi-View 3D Reconstruction
South Dakota State University · Aalto University · KTH Royal Institute of Technology +3

Sep 8, 2026J/K move · Enter open · S save

OmniPoint: Universal Monocular Metric Pointcloud from Any Camera

Botao Ye, Marc Pollefeys, Ming-Hsuan Yang, Abhijit Kundu

Google DeepMind · ETH Zurich · Work done as an intern at Google DeepMind.

Panoramic 3D Reconstruction 3D Reconstruction Monocular 3D Reconstruction Monocular Depth Estimation

Open paper AskPDF arXiv

Recovering metric 3D geometry from monocular images is a fundamental computer vision task, yet current methods remain heavily fragmented by fixed camera model assumptions and inflexible input schemes. We present OmniPoint, a unified framework designed to generalize metric reconstruction across diverse imaging sensors, including pinhole, fisheye, and equirectangular projections, while accommodating varying geometric priors. To overcome projection rigidity, OmniPoint abandons conventional planar depth regression. It instead adopts a decoupled ray and distance representation alongside a decoupled training objective, explicitly separating the camera projection model from the scene structure. To address the severe scarcity of training data for alternative cameras, we introduce a bidirectional augmentation strategy that explicitly bridges labeled perspective data and unlabeled omnidirectional domains in 3D space. Furthermore, to seamlessly integrate optional inputs like camera intrinsics or sparse depth without destabilizing the network through feature distribution shifts, we propose a robust information injection mechanism. This mechanism utilizes learnable input state embeddings to resolve architectural ambiguity and applies vectorized Gaussian smoothing to densify irregular measurements. Extensive experiments demonstrate that OmniPoint achieves state-of-the-art zero-shot performance across multiple benchmarks, establishing a robust new standard for unified monocular 3D reconstruction.

Abstract

3D Reconstruction

Explore similar work

OmniPoint: Universal Monocular Metric Pointcloud from Any Camera

CAM3R: Camera-Agnostic Model for 3D Reconstruction

Any3R: Feedforward 3D Reconstruction from Any Camera Type