Fractal Autoregressive Depth Estimation with Continuous Token Diffusion

Jinchang Zhang; Xinrou Kang; Guoyu Lu

cs.CVMar 16, 2026

Fractal Autoregressive Depth Estimation with Continuous Token Diffusion

Authors: Jinchang Zhang, Xinrou Kang, Guoyu Lu

Organizations: Intelligent Vision and Sensing (IVS) Lab at SUNY Binghamton University

Abstract

Monocular depth estimation can benefit from autoregressive (AR) generation, but direct AR modeling is hindered by the modality gap between RGB and depth, inefficient pixel-wise generation, and instability in continuous depth prediction. We propose a Fractal Visual Autoregressive Diffusion framework that reformulates depth estimation as a coarse-to-fine, next-scale autoregressive generation process. A VCFR module fuses multi-scale image features with current depth predictions to improve cross-modal conditioning, while a conditional denoising diffusion loss models depth distributions directly in continuous space and mitigates errors caused by discrete quantization. To improve computational efficiency, we organize the scale-wise generators into a fractal recursive architecture, reusing a base visual AR unit in a self-similar hierarchy. We further introduce an uncertainty-aware robust consensus aggregation scheme for multi-sample inference to improve fusion stability and provide a practical pixel-wise reliability estimate. Experiments on standard benchmarks demonstrate strong performance and validate the effectiveness of the proposed design.

Figures & tables

Figure 1: Given an input RGB image $X\in\mathbb{R}^{H\times W\times 3}$ , we first extract multi-scale visual features $Z(X)$ using an image encoder and a multi-scale aggregation module, which serve as global conditioning throughout depth generation. The framework is organized as a fractal recursive hierarchy of four nested scale-wise generators $\{g_{4},g_{3},g_{2},g_{1}\}$ , which progressively recover depth representations from coarse to fine and output the final depth map $\hat{D}\in\mathbb{R}^{H\times W}$ . Specifically, $g_{4}$ starts from a coarse latent token and predicts an initial depth latent, which is then passed to $g_{3}$ , followed by $g_{2}$ , and finally $g_{1}$ . The top-left panel illustrates the fractal recursion pattern, while the bottom panel shows the concrete generation pipeline. At each scale, a VCFR block fuses image features with the current-scale depth latent/token to form a Visual-Depth Joint token for next-scale prediction. Within each module, depth is predicted via a visual autoregressive diffusion process, where a conditional denoising diffusion objective models the continuous depth distribution at the current scale; the noise predictor $\epsilon_{\theta}(x_{t}\mid t,z)$ takes diffusion timestep $t$ and condition vector $z$ and is optimized with diffusion loss.

Method	Architecture	AbsRel ↓	Sq Rel↓	RMSE↓	RMSE log↓	$\delta<1.25$ ↑	$\delta<1.25^{2}$ ↑	$\delta<1.25^{3}$ ↑
AdaBins [ 2 ]	E-B5+mini-ViT	0.067	0.190	2.960	0.088	0.949	0.992	0.998
DPT [ 27 ]	VIT-L	0.060	-	2.573	0.092	0.959	0.995	0.996
P3Depth [ 24 ]	ResNet-101	0.071	0.270	2.842	0.103	0.953	0.993	0.998
NeWCRFs [ 36 ]	Swin-Large	0.052	0.155	2.129	0.079	0.974	0.997	0.999
BinsFormer [ 20 ]	Swin-Large	0.052	0.151	2.098	0.079	0.974	0.997	0.999
PixelFormer [ 1 ]	Swin-Large	0.051	0.149	2.081	0.077	0.976	0.997	0.999

Table 1: A quantitative comparison of the KITTI dataset [ 10 ] .

Figure 2: Different stages of depth in a fractal framework.

Table 4

Method	Encoder	AbsRel ↓	Sq Rel↓	RMSE↓	log↓	$\delta<1.25$ ↑	$\delta<1.25^{2}$ ↑	$\delta<1.25^{3}$ ↑
AdaBins [ 2 ]	E-B5+mini-ViT	0.103	-	0.364	0.044	0.903	0.984	0.997
P3Depth [ 24 ]	ResNet-101	0.104	-	0.356	0.043	0.904	0.988	0.998
LocalBins [ 3 ]	E-B5	0.099	-	0.357	0.042	0.907	0.987	0.998
NeWCRFs [ 36 ]	Swin-Large	0.095	0.045	0.334	0.041	0.922	0.992	0.998
BinsFormer [ 20 ]	Swin-Large	0.094	-	0.330	0.040	0.925	0.991	0.997
PixelFormer [ 1 ]	Swin-Large	0.090	-	0.322	0.039	0.929	0.991	0.998

Table 4: A quantitative comparison on NYU dataset [ 30 ] .

Table 6

Figure 3: Visual results on NYU. From left to right: input images, depth estimation from ground truth, DiffusionDepth [ 8 ] , Repurposing Diffusion [ 17 ] , ours.

Figure 6: Visual results. From top to bottom; input images, ground truth, depth from discrete VQ tokens, and depth from continuous KL tokens..

Method	Marigold	Lotus	E2E-Mono	Monodepth2	MonoViT	DiffusiongDepth	Jasmine	Ours
MACs	133T	2.65T	2.65T	21.43G	25.63G	-	35.7G	132G
Runtime	9.88s	157ms	152ms	33ms	29ms	223ms	172ms	64ms

Table 7: MACs and runtime on input resolution

1024\times 320

AR	VAR	Diffusion	Fractal	VCFR	Cost	AbsRel $\downarrow$	Sq Rel $\downarrow$	RMSE $\downarrow$	$\delta<1.25$ $\uparrow$
✓					1.000	0.083	0.039	0.314	0.938
	✓				0.017	0.079	0.037	0.279	0.949
	✓	✓			0.720	0.063	0.020	0.235	0.975
	✓	✓	✓		0.045	0.058	0.013	0.212	0.982
	✓	✓	✓	✓	0.051	0.049	0.011	0.197	0.984

Table 8: Ablation study of the framework on the NYU dataset.

Depth Estimation

Depth estimation infers scene distance from images and sensors, enabling 3D reconstruction, navigation, and spatial understanding.

See all 198 papers

ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning
Jul 14, 2026Zijie Wang, Wei Zhang, Weiming Zhang +4Depth Estimation Monocular Depth Estimation
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, 510006, China · Shenzhen Loop Area Institute, Shenzhen, 518038, China · Baidu Inc., Shenzhen, 518066, China +3
ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning
Jul 2, 2026Xuanhua He, Jiaxin Xie, Mingzhe Zheng +1Video Diffusion Models Depth Estimation
The Hong Kong University of Science and Technology, Hong Kong SAR, China
Efficient and High-Quality Depth Estimation via Pixel-Space Diffusion with Linear Attention
Aug 31, 2026Bingde Liu, Wu Ran, Jinglei Zhang +2Monocular Depth Estimation Pixel-Space Diffusion Models
MoE Key Lab of Artificial Intelligence, Institute of AI, Shanghai Jiao Tong University, Shanghai, China · Zhiyuan College, Shanghai Jiao Tong University, Shanghai, China

Jul 14, 2026J/K move · Enter open · S save

ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning

Zijie Wang, Wei Zhang, Weiming Zhang, Xiao Tan+3

School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, 510006, China · Shenzhen Loop Area Institute, Shenzhen, 518038, China · Baidu Inc., Shenzhen, 518066, China+3

Depth Estimation Monocular Depth Estimation Autoregressive Generation

Open paper AskPDF arXiv

Diffusion models have recently become the dominant paradigm for monocular depth estimation (MDE). However, they implicitly assume that depth can be recovered as a globally smooth field through iterative denoising, which does not explicitly reflect the piecewise and scale-dependent organization of scene geometry. In practice, geometric structure emerges progressively across spatial scales, where coarse layout, surfaces, and boundaries are constructed in a hierarchical manner. Motivated by this observation, we introduce ARDepth, which formulates depth estimation as structured auto-regressive generation. Instead of recovering depth through global refinement, ARDepth progressively constructs depth representations as spatial resolution increases. To support this generative process, we introduce Scale-Progressive Conditioning (SPC) to inject multi-scale visual features at each generation stage, and Semantic-Aware Guidance (SAG) to provide scene-level semantic priors that enhance global structural consistency. Together, these designs enable the model to capture fine-grained local details while maintaining coherent global geometry. Empirical results demonstrate that our approach achieves strong performance and produces structurally consistent depth predictions across scales, validating auto-regressive generation as a promising alternative paradigm for geometric modeling.

Abstract

Depth Estimation

Explore similar work

ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning

ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning

Efficient and High-Quality Depth Estimation via Pixel-Space Diffusion with Linear Attention