StreamTTO: Efficient Online Test-Time Optimization for Video Depth Completion

Minseok Seo; Wonjun Lee; Jaehyuk Jang; Changick Kim

cs.CVMar 2, 2026

StreamTTO: Efficient Online Test-Time Optimization for Video Depth Completion

Authors: Minseok Seo, Wonjun Lee, Jaehyuk Jang, Changick Kim

Organizations: Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea

Abstract

Monocular depth foundation models generalize across diverse scenes, but recovering accurate metric depth consistent with a target sensor remains challenging under sensor variation and domain shift. Test-time optimization (TTO) guided by sparse depth addresses this limitation, yet independent per-frame adaptation incurs substantial computational cost. We propose StreamTTO, an online test-time optimization framework for video depth completion that reuses visual features and adaptation state. A frozen feature extractor with causal temporal attention incorporates past visual context into features cached for repeated decoder optimization. Causal Sliding-Window Optimization directly updates a shared depth decoder using cached features and sparse observations from recent frames, requiring only two passes over the window per incoming frame after initialization. Reusing recent observations and the adapted decoder substantially reduces optimization steps. We also introduce MS-Depth, comprising approximately 1.45M synchronized RGB--LiDAR frames from 40 hours of recordings. Its uninterrupted sequences capture transitions among indoor, underground, and outdoor environments under daytime and nighttime conditions, enabling evaluation of adaptation to changes in illumination, scene structure, and depth range. Experiments on KITTI, Bonn, NYUv2 Raw, TUM RGB-D, DDAD, and MS-Depth demonstrate competitive depth completion accuracy. On MS-Depth, StreamTTO achieves average processing speeds of approximately 15~~FPS with VGGT and 25~~FPS with MoGe-2 at an input resolution of 518 X 392.

Figures & tables

Figure 1: Depth accuracy and processing speed on MS-Depth TE06, with approximately 13,000 frames spanning indoor, outdoor, and underground environments. StreamTTO achieves the best accuracy among the evaluated methods at 15.2 FPS after initial adaptation.

Figure 2: An example MS-Depth test sequence repeatedly traversing indoor, underground, and outdoor environments before returning indoors. Sequences average approximately 20 minutes and are recorded continuously with a calibrated handheld camera–LiDAR rig. RGB images are paired with sparse and dense reference depth maps.

Figure 3: Overview of StreamTTO. A frozen ViT encoder with spatial and causal temporal attention extracts features using a key–value cache for past context. A separate feature cache stores the extracted features for batched optimization over a 3-frame causal window. Sparse depth observations supervise updates to the shared depth decoder, whose parameters are carried forward to subsequent frames. Only the decoder is optimized, avoiding repeated feature extraction.

Method	KITTI		Bonn		NYUv2 Raw		TUM RGB-D		DDAD		Speed
Method	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	Avg. FPS $\uparrow$
Depth Foundation Models
Depth Anything V2 Metric	4.1902	2.1348	0.2278	0.1380	0.1660	0.1060	0.2705	0.1410	15.7768	8.7141	26.68
UniDepthV2	2.9397	1.2993	0.1832	0.0811	0.1327	0.0698	0.2674	0.1265	13.2580	5.7289	37.47
Video Depth Anything Metric (online)	4.3396	2.3805	0.1992	0.1125	0.2513	0.1380	0.2793	0.1490	15.4656	7.8687	35.53
3D Foundation Models

Table 1: Quantitative comparison of depth foundation models, 3D foundation models, and depth completion methods on five public benchmarks. RMSE and MAE are reported in meters, excluding sparse-input observation locations from accuracy evaluation. Bold and underlined values indicate the best and second-best results across all methods, respectively, including ties.

MS-Depth
Method	Indoor				Outdoor				FPS $\uparrow$
	Underground		Aboveground		Day		Night
	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$
Depth Foundation Models
Depth Anything V2 Metric	2.3864	1.7842	1.7433	0.8399	2.7577	1.6428	2.4214	1.3059	59.52
UniDepthV2	1.4090	0.8418	1.6569	0.5920	3.6521	2.3448	2.9081	1.6813	83.16

Table 2: Results on the 12 MS-Depth test sequences excluding TE04. RMSE and MAE (m) exclude shared sparse-input locations. Depth and 3D foundation models use sparse-depth-based scale–shift alignment; depth completion methods receive no additional alignment.

Figure 4: Qualitative 3D comparisons for an indoor staircase and an outdoor building. Predicted depth is back-projected into 3D and colored using the corresponding RGB values. Reference geometry is shown in gray for comparison of scale and structure.

Table 3: StreamTTO ablations on MS-Depth TE04 with the VGGT backbone. RMSE (m) is reported as mean

\pm

standard deviation over 10 seeds. In the adaptation-target ablation, “full” denotes joint updates to the image encoder and depth decoder, while the spatial and causal temporal attention modules remain frozen. FLOPs (T/frame) include feature extraction and test-time adaptation for each configuration at an input resolution of

518\times 392

. Default settings are highlighted in gray: 6-frame causal KV context, full-decoder adaptation, warm-start with 2 passes,

W=3

, and 20 bootstrap updates.

Method	Indoor $\rightarrow$ Outdoor	Outdoor $\rightarrow$ Indoor	Indoor $\rightarrow$ Underground	Underground $\rightarrow$ Indoor	Outdoor $\rightarrow$ Underground	Underground $\rightarrow$ Outdoor	Overall	FPS $\uparrow$	Training FLOPs $\downarrow$ (T/frame)
W1 (6 updates)	$1.0363\pm 0.0600$	$1.5833\pm 0.0469$	$1.1038\pm 0.0116$	$0.9515\pm 0.0435$	$0.8663\pm 0.0338$	$0.7513\pm 0.0368$	$1.2146\pm 0.0360$	10.5	4.063
W3 (2 passes)	$\mathbf{0.9762\pm 0.0241}$	$\mathbf{1.4970\pm 0.0386}$	$\mathbf{1.0047\pm 0.0268}$	$\mathbf{0.8789\pm 0.0376}$	$\mathbf{0.7470\pm 0.0424}$	$\mathbf{0.6459\pm 0.0406}$	$\mathbf{1.1340\pm 0.0286}$	15.7	4.063

Table 4: Post-transition RMSE (m,

\downarrow

) over

[0,10)

s on 51 MS-Depth transition clips, excluding TE04. Results are reported as mean

\pm

standard deviation across 10 random seeds. W1 and W3 use matched decoder training FLOPs, which exclude frozen feature extraction.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Qualitative results on MS-Depth at 100-frame intervals, showing RGB images, ground-truth depth, and StreamTTO predictions in that order.

Figure 6: Qualitative results on MS-Depth at 100-frame intervals, showing RGB images, ground-truth depth, and StreamTTO predictions in that order.

	Optimization passes $R$
$W$	1	2	5	10	20	40
1	1.4609	1.3786	1.1952	0.9908	0.9245	0.9623
2	1.4583	1.3734	1.1847	0.9723	0.8895	0.9233
3	1.4577	1.3725	1.1831	0.9681	0.8753	0.9039
6	1.4588	1.3745	1.1883	0.9728	0.8615	0.8769
12	1.4623	1.3812	1.2028	0.9927	0.8645	0.8600

Appendix

Table 5: Window size and optimization passes under reset adaptation on MS-Depth TE04 with the VGGT backbone. The decoder and optimizer are reinitialized at every frame. Values are RMSE in meters (lower is better); underlined entries indicate the lowest RMSE in each row.

	Optimization passes $R$
$W$	1	2	5	10	20	40
1	0.9733	0.9314	0.9038	0.9781	0.9770	0.9758
2	0.9481	0.9004	0.8869	0.9514	1.5399	1.8283
3	0.9233	0.8694	0.8594	0.9704	0.9846	1.2465
6	0.9048	0.8551	0.8812	0.9746	1.0659	1.2872
12	0.8922	0.9544	1.0273	1.2603	1.4860	1.5117

Appendix

Table 6: Window size and optimization passes under warm-start adaptation on MS-Depth TE04 with the VGGT backbone. The decoder and optimizer states are retained across frames, with

R

passes per subsequent frame after initial bootstrap adaptation. Values are RMSE in meters (lower is better); underlined entries indicate the lowest RMSE in each row.

			RMSE (m) $\downarrow$
$W$	$R$	Configuration	Observed	Unobserved
3	2	Baseline	0.8162	0.8696
3	20	No modification	0.2623	0.9773
3	20	Lower constant LR	0.7454	0.9321
3	20	LR decay	0.4113	0.9955
3	20	Optimizer-only reset	0.2740	0.9869

Appendix

Table 7: Diagnostic experiments on the full MS-Depth TE04 development sequence. Observed and unobserved RMSE are evaluated at sparse-input locations and at the remaining valid evaluation pixels, respectively; underlined values indicate the lowest RMSE in each column. All window frames receive weight 1, except in the W12 reweighting variant, which assigns weight 5.5 to the current frame and 1 to each past frame. Periodic resets are applied every 250 frames throughout the sequence, immediately before adaptation at zero-based frame indices

250,500,750,1000,\ldots

. Optimizer-only reset clears the AdamW state while retaining the decoder; full reset additionally restores the pretrained decoder. Both retain the KV cache, cached features, and optimization window, without additional bootstrap updates. All frames, including those immediately following resets, contribute to the reported mean per-frame RMSE.

Initialization	$[0,1)$ s	$[1,3)$ s	$[3,10)$ s	$[0,10)$ s
Local reset + bootstrap	1.26	1.16	1.11	1.14
Full-stream warm-start	1.24	1.10	1.07	1.09

Appendix

Table 8: Post-transition RMSE (m,

\downarrow

) on 51 transition clips from the 12 MS-Depth evaluation sequences, excluding TE04. Local initialization resets the decoder and optimizer approximately four seconds before each transition and performs 20 bootstrap updates. Full-stream warm-start retains adaptation state from the beginning of each recording. Both configurations use identical visual features and sparse observations.

Adaptation strategy	0–5 min	5–10 min	10–15 min	$\geq$ 15 min	Overall
Continuous warm-start	1.06	1.14	1.05	1.11	1.09
Periodic reset	1.15	1.24	1.16	1.22	1.19
Periodic reset + bootstrap	1.05	1.16	1.08	1.14	1.11

Appendix

Table 9: Long-horizon adaptation on the 12 complete MS-Depth evaluation sequences, excluding TE04. RMSE (m,

\downarrow

) is reported by elapsed time and over the complete recordings. Periodic resets reinitialize the decoder and optimizer every 250 frames while retaining the KV and feature caches. The bootstrap variant performs 20 additional updates at each reset. All configurations use

W=3

and

R=2

after bootstrap.

Method	KITTI		Bonn		NYUv2 Raw		TUM RGB-D		DDAD		Speed
Method	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	Avg. FPS $\uparrow$
VGGT + Causal-Attn	3.0817	1.3334	0.1723	0.0792	0.1602	0.0832	0.2150	0.0988	12.7059	6.6541	17.52
VGGT + Causal-Attn + TestPromptDC	1.2449	0.3768	0.0559	0.0208	0.0584	0.0254	0.0550	0.0190	9.3939	2.6467	0.201
VGGT + CSWO	2.2404	0.8587	0.1330	0.0543	0.0992	0.0621	0.1425	0.0782	8.9450	4.2453	9.491
VGGT + StreamTTO	1.0223	0.3920	0.0592	0.0210	0.0403	0.0220	0.0738	0.0274	7.1448	2.3483	7.14
MoGe-2 + Causal-Attn	2.8130	1.2086	0.1641	0.0793	0.1194	0.0732	0.2414	0.1075	10.2409	4.1452	33.28

Appendix

Table 10: Contribution of the adaptation strategy and causal temporal attention across five public benchmarks. All variants receive identical RGB frames and sparse observations and follow the same evaluation protocol. Within each backbone group, Causal-Attn, Causal-Attn + TestPromptDC, and StreamTTO share the same pretrained causal-backbone initialization and 6-frame causal KV context. Causal-Attn performs no test-time parameter updates; TestPromptDC optimizes an input visual prompt independently per frame. CSWO applies window-based decoder adaptation without causal temporal attention or a temporal KV cache. Both CSWO and StreamTTO use

W=3

and

R=2

after bootstrap and retain decoder parameters and optimizer state across frames. CAPA-LoRA is excluded because it uses an offline clip-level protocol in our evaluation. RMSE and MAE are reported in meters, excluding sparse-input locations, following Table 1 . FPS is averaged over the five benchmarks at their dataset-specific input resolutions on one B200. Bold and underlined values indicate the best and second-best results within this table, including ties.

MS-Depth
Method	Indoor				Outdoor				FPS $\uparrow$
	Underground		Aboveground		Day		Night
	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$
VGGT + Causal-Attn	1.5222	0.9521	1.8435	0.6425	2.3414	1.4592	2.3252	1.3433	39.84
VGGT + Causal-Attn + TestPromptDC	1.0995	0.3174	1.2347	0.2536	1.8212	0.5253	1.7266	0.4693	0.608
VGGT + CSWO	1.1234	0.6227	1.4582	0.4388	1.6999	0.8930	1.7881	0.8691	19.46

Appendix

Table 11: Contribution of the adaptation strategy and causal temporal attention on the 12 MS-Depth test sequences excluding TE04. All variants receive identical RGB frames and sparse observations and follow the same evaluation protocol. Within each backbone group, Causal-Attn, Causal-Attn + TestPromptDC, and StreamTTO share the same pretrained causal-backbone initialization and 6-frame causal KV context. Causal-Attn performs no test-time parameter updates; TestPromptDC optimizes an input visual prompt independently per frame. CSWO applies window-based decoder adaptation without causal temporal attention or a temporal KV cache. Both CSWO and StreamTTO use

W=3

and

R=2

518\times 392

on one B200. Bold and underlined values indicate the best and second-best results within this table, including ties.

	Observations 3%		Observations 1%
Method	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$
Depth foundation models
Depth Anything V2 Metric	0.1660	0.1060	0.1661	0.1061
UniDepthV2	0.1327	0.0698	0.1328	0.0698
Video Depth Anything Metric (online)	0.2514	0.1380	0.2516	0.1381
3D foundation models

Appendix

Table 12: Depth completion under reduced observation density on NYUv2 Raw. Observation density is the fraction of valid reference-depth pixels provided as input. RMSE and MAE are reported in meters; lower is better. Input observation locations are excluded from evaluation at each density. StreamTTO uses the VGGT backbone. Bold and underlined values indicate the best and second-best results in each column, respectively, including ties.

	KITTI		Bonn		NYUv2 Raw		TUM RGB-D		DDAD
Method	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$	RMSE $\downarrow$	MAE $\downarrow$
TestPromptDC	1.1266	0.3441	0.0514	0.0190	0.0495	0.0229	0.0514	0.0163	8.2693	2.2167
TestPromptDC + warm-start	1.0994	0.3368	0.0507	0.0175	0.0460	0.0196	0.0512	0.0161	8.1666	2.1951
Error reduction	2.41%	2.12%	1.36%	7.89%	7.07%	14.41%	0.39%	1.23%	1.24%	0.97%
CAPA-LoRA (offline)	1.0902	0.2684	0.0566	0.0179	0.0546	0.0219	0.0560	0.0144	7.6213	1.7592
CAPA-LoRA + warm-start	1.0076	0.2374	0.0540	0.0156	0.0488	0.0185	0.0532	0.0123	7.5613	1.7009

Appendix

Table 13: Effect of warm-start adaptation on baseline methods, with StreamTTO included for comparison. All experiments follow the same evaluation settings as Table 1 .

Dataset	Environment	Sensing	RGB Resolution	Rate	Total Video Duration / Frame Count
KITTI	Outdoor Driving	Stereo RGB; 64-Channel LiDAR	Sequence- Dependent 2 2 2 KITTI’s rectified RGB resolutions are $1224\times 370$ , $1226\times 370$ , $1238\times 374$ , $1241\times 376$ , and $1242\times 375$ , as specified in the date-specific calibration files of the official raw release ( Geiger et al., 2013 ) .	10 Hz	1.3 h / 47,937 frames
Bonn	Indoor	ASUS Xtion Pro LIVE RGB-D	$640\times 480$	30 Hz	17 min / 30,022 frames
NYUv2 Raw	Indoor	Kinect RGB-D	$640\times 480$	20–30 Hz	407,024 frames
TUM RGB-D	Indoor	Kinect / ASUS Xtion	$640\times 480$	30 Hz	1 h 16 min / 129,824 frames (Public Pose GT: 49 min / 81,413 frames; Validation: 27 min / 48,411 frames)
DDAD	Outdoor Driving	6 RGB Cameras; Luminar-H2 LiDAR	$1936\times 1216$	10 Hz	28 min / 16,600 frames per camera (Train: 21 min / 12,650 frames; Validation: 7 min / 3,950 frames) Test: 3,080 frames
MS-Depth	Indoor / Outdoor / Underground	RGB; 128-Channel LiDAR; IMU	$2048\times 1536$	10 Hz	40 h / 1.45 M frames (Train 36 h / 1.3 M frames; Test 4 h / 148,115 frames)

Appendix

Table 14: Dataset specifications and video volume. Available group or split statistics are shown in parentheses. All video durations are approximate, using reported durations or estimates from frame counts and capture rates.

Dataset	Platform	RGB resolution	Rate	LiDAR channels	Duration (min/seq.)	Seq. $\geq$ 15 min	I/O seq.	Day / Night	Depth data	Env. info.
FusionPortableV2	Multiple	$1024\times 768$	20 Hz	128	1.3–11.6	0/27	6/27	24 / 3	Subset + tools	Sequence
Oxford Spires	Human- carried	$1440\times 1080$	5–20 Hz	64	3.5–21.3	3/24	11/24	24 / 0	Subset + tools	Sequence
MS-Depth (test)	Handheld	$2048\times 1536$	10 Hz	128	15.8–22.6	13/13	13/13	6 / 7	Input + reference	Temporal

Appendix

Table 16: Dataset characteristics relevant to continuous adaptation. Duration is per recording. Fractions denote qualifying/total sequences; Day/Night gives capture-time counts. I/O denotes indoor–outdoor recordings. MS-Depth uses the test split.

Dataset	Sequences	Pairs per sequence	Total pairs
KITTI	8	110	880
Bonn	5	110	550
NYUv2 Raw	8	110	880
TUM RGB-D	8	110	880
DDAD	50	50 or 100	3,950
Total	79		7,140

Appendix

Table 17: Public-benchmark evaluation subsets. Each sequence contributes one evaluation clip; DDAD clips contain all front-camera frames from their respective validation scenes.

Sequence ID	Start frame	End frame	Pairs
2011_09_26_drive_0005_sync	0000000005	0000000114	110
2011_09_26_drive_0013_sync	0000000005	0000000114	110
2011_09_26_drive_0023_sync	0000000005	0000000114	110
2011_09_26_drive_0036_sync	0000000005	0000000114	110
2011_09_26_drive_0095_sync	0000000005	0000000114	110
2011_09_29_drive_0026_sync	0000000005	0000000114	110

Appendix

Table 18: KITTI evaluation sequences and inclusive frame-ID intervals.

Sequence ID	Start depth timestamp	End depth timestamp	Pairs
rgbd_bonn_synchronous	1548340301.91798	1548340305.55515	110
rgbd_bonn_person_tracking	1548265882.51176	1548265886.14241	110
rgbd_bonn_crowd2	1548339894.59618	1548339898.23325	110
rgbd_bonn_crowd3	1548339959.43347	1548339963.07062	110
rgbd_bonn_balloon2	1548266533.17712	1548266536.81403	110

Appendix

Table 19: Bonn evaluation sequences and inclusive depth-timestamp intervals.

Sequence ID	Start depth timestamp	End depth timestamp	Pairs
basement_0001a	1316653609.773176	1316653613.405163	110
bookstore_0001a	1295525659.331413	1295525662.963355	110
cafe_0001a	1295527093.873179	1295527097.504396	110
classroom_0001a	1315404991.101447	1315404994.732725	110
kitchen_0003	1294887072.518566	1294887076.168351	110
office_0004	1294851491.914890	1294851495.546016	110

Appendix

Table 20: NYUv2 Raw evaluation sequences and inclusive depth-timestamp intervals.

Sequence ID	Start depth timestamp	End depth timestamp	Pairs
rgbd_dataset_freiburg1_desk	1305031453.374112	1305031457.342954	110
rgbd_dataset_freiburg1_desk2	1305031526.688356	1305031530.656445	110
rgbd_dataset_freiburg1_room	1305031910.771502	1305031914.407339	110
rgbd_dataset_freiburg2_desk	1311868164.373557	1311868168.115940	110
rgbd_dataset_freiburg2_desk_with_person	1311870427.207687	1311870431.247317	110
rgbd_dataset_freiburg2_large_with_loop	1311875572.417883	1311875576.219311	110

Appendix

Table 21: TUM RGB-D evaluation sequences and inclusive depth-timestamp intervals.

Frames per scene	Sample indices	Scene IDs
50	0–49	000151, 000152, 000154, 000157, 000159, 000162, 000166, 000172, 000174, 000177, 000182, 000183, 000191, 000192, 000193, 000194, 000195, 000196, 000197, 000198, 000199
100	0–99	000150, 000153, 000155, 000156, 000158, 000160, 000161, 000163, 000164, 000165, 000167, 000168, 000169, 000170, 000171, 000173, 000175, 000176, 000178, 000179, 000180, 000181, 000184, 000185, 000186, 000187, 000188, 000189, 000190

Appendix

Table 22: Complete DDAD validation-scene coverage for CAMERA_01 . Sample-index intervals are zero-based and inclusive.

$W$	Mean $q_{t}$	Median $q_{t}$	Frames with $q_{t}<1$	Mean $d_{t}$	Mean $s_{t}$
1	0.58	0.30	84%	13.9	0.55
3 (default)	0.55	0.29	86%	9.4	0.78
6	0.57	0.30	85%	6.1	0.86
12	1.05	0.46	70%	4.3	0.91

Appendix

Table 23: Decoder-gradient diagnostics on MS-Depth TE04 before current-frame adaptation. The ratio

q_{t}

compares warm-start and pretrained initializations, while

d_{t}

and

s_{t}

measure gradient difference and cosine similarity between consecutive windows evaluated at the same decoder parameters.

Explore similar work

Feb 16, 2026

Depth Completion as Parameter-Efficient Test-Time Adaptation

We introduce CAPA, a parameter-efficient test-time optimization framework that adapts pre-trained 3D foundation models (FMs) for depth completion, using sparse geometric cues. Unlike prior methods that train task-specific encoders for auxiliary inputs, which often overfit and generalize poorly, CAPA freezes the FM backbone. Instead, it updates only a minimal set of parameters using Parameter-Efficient Fine-Tuning (e.g. LoRA or VPT), guided by gradients calculated directly from the sparse observations available at inference time. This approach effectively grounds the foundation model's geometric prior in the scene-specific measurements, correcting distortions and misplaced structures. For videos, CAPA introduces sequence-level parameter sharing, jointly adapting all frames to exploit temporal correlations, improve robustness, and enforce multi-frame consistency. CAPA is model-agnostic, compatible with any ViT-based FM, and achieves state-of-the-art results across diverse condition patterns on both indoor and outdoor datasets. Project page: research.nvidia.com/labs/dvl/projects/capa.

Bingxin Ke, Qunjie Zhou, Jiahui Huang +5

NVIDIA · ETH Zürich

Depth Completion Test-Time Adaptation Vision Foundation Model Adaptation 3D Foundation Models

Ask PDF

Mar 11, 2026

Need for Speed: Zero-Shot Depth Completion with Single-Step Diffusion

We introduce Marigold-SSD, a single-step, late-fusion depth completion framework that leverages strong diffusion priors while eliminating the costly test-time optimization typically associated with diffusion-based methods. By shifting computational burden from inference to finetuning, our approach enables efficient and robust 3D perception under real-world latency constraints. Marigold-SSD achieves significantly faster inference with a training cost of only 4.5 GPU days. We evaluate our method across four indoor and two outdoor benchmarks, demonstrating strong cross-domain generalization and zero-shot performance compared to existing depth completion approaches. Our approach significantly narrows the efficiency gap between diffusion-based and discriminative models. Finally, we challenge common evaluation protocols by analyzing performance under varying input sparsity levels. Page: https://dtu-pas.github.io/marigold-ssd/

Jakub Gregorek, Paraskevas Pegios, Nando Metzger +3

DTU - Technical University of Denmark · Pioneer Centre for AI · ETH Zürich

Depth Completion Diffusion Models Diffusion Model Inference Acceleration

Ask PDF

Apr 8, 2026

VDPP: Video Depth Post-Processing for Speed and Scalability

Video depth estimation is essential for providing 3D scene structure in applications ranging from autonomous driving to mixed reality. Current end-to-end video depth models have established state-of-the-art performance. Although current end-to-end (E2E) models have achieved state-of-the-art performance, they function as tightly coupled systems that suffer from a significant adaptation lag whenever superior single-image depth estimators are released. To mitigate this issue, post-processing methods such as NVDS offer a modular plug-and-play alternative to incorporate any evolving image depth model without retraining. However, existing post-processing methods still struggle to match the efficiency and practicality of E2E systems due to limited speed, accuracy, and RGB reliance. In this work, we revitalize the role of post-processing by proposing VDPP (Video Depth Post-Processing), a framework that improves the speed and accuracy of post-processing methods for video depth estimation. By shifting the paradigm from computationally expensive scene reconstruction to targeted geometric refinement, VDPP operates purely on geometric refinements in low-resolution space. This design achieves exceptional speed (>43.5 FPS on NVIDIA Jetson Orin Nano) while matching the temporal coherence of E2E systems, with dense residual learning driving geometric representations rather than full reconstructions. Furthermore, our VDPP's RGB-free architecture ensures true scalability, enabling immediate integration with any evolving image depth model. Our results demonstrate that VDPP provides a superior balance of speed, accuracy, and memory efficiency, making it the most practical solution for real-time edge deployment. Our project page is at https://github.com/injun-baek/VDPP

Daewon Yoon, Injun Baek, Sangyu Han +2

Seoul National University · Samsung Electronics

Depth Estimation Monocular Depth Estimation Edge Inference

Ask PDF