Self-Improving 4D Perception via Self-Distillation

Nan Huang; Pengcheng Yu; Weijia Zeng; James M. Rehg; Angjoo Kanazawa; Haiwen Feng; Qianqian Wang

cs.CVApr 9, 2026

Self-Improving 4D Perception via Self-Distillation

Authors: Nan Huang, Pengcheng Yu, Weijia Zeng, James M. Rehg, Angjoo Kanazawa, Haiwen Feng, Qianqian Wang

Organizations: UIUC · Impossible Research · MPI for Intelligent Systems · UBC ·

Abstract

Large-scale multi-view reconstruction models have made remarkable progress, but most existing approaches still rely on fully supervised training with ground-truth 3D/4D annotations. Such annotations are expensive and particularly scarce for dynamic scenes, limiting scalability. We propose SelfEvo, a self-improving framework that continually improves pretrained multi-view reconstruction models using unlabeled videos. SelfEvo introduces a self-distillation scheme using spatiotemporal context asymmetry, enabling self-improvement for learning-based 4D perception without external annotations. We systematically study design choices that make self-improvement effective, including loss signals, forms of asymmetry, and other training strategies. Across eight benchmarks spanning diverse datasets and domains, SelfEvo consistently improves pretrained baselines and generalizes across base models (e.g. VGGT and $π^3$ ), with significant gains on dynamic scenes. Overall, SelfEvo achieves up to 36.5% relative improvement in video depth estimation and 20.1% in camera estimation, without using any labeled data. Project Page: https://self-evo.github.io/.

Figures & tables

Figure 2: We propose an annotation-free self-improving framework that continually post-trains pretrained multi-view reconstruction models using unlabeled videos. Our method forms an online self-distillation loop where a richer-context teacher provides stop-gradient pseudo targets to a student operating on reduced context, and is updated as an EMA of the student after each step.

Methods	OmniGeo				OmniVideo
	scale		scale&shift		scale		scale&shift
	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$
VGGT [ 59 ]	0.346	0.592	0.180	0.758	0.236	0.713	0.145	0.824
SelfEvo (VGGT)	0.278	0.703	0.124	0.867	0.181	0.792	0.113	0.876

Table 1: New-domain video depth evaluation on OmniGeo and OmniVideo [ 81 ] .

Methods	OmniGeo			OmniVideo			RealEstate10K
Methods	AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$	AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$	AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
VGGT	45.093	64.659	72.649	67.785	83.468	89.743	38.597	66.404	78.833
SelfEvo (VGGT)	58.271	79.034	87.285	75.420	89.132	93.945	48.765	73.324	83.359

Table 2: Camera estimation: new-domain results are reported on OmniGeo and OmniVideo [ 81 ] , and original-domain retention is evaluated on RealEstate10K [ 80 ] .

Method	Align	Sintel		Bonn		KITTI
Method	Align	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$
VGGT [ 59 ]	scale	0.300	0.641	0.055	0.971	0.074	0.960
SelfEvo (VGGT)	scale	0.283	0.707	0.046	0.977	0.047	0.981
VGGT [ 59 ]	scale& shift	0.227	0.684	0.049	0.974	0.059	0.961
SelfEvo (VGGT)	scale& shift	0.212	0.692	0.044	0.977	0.042	0.979

Table 3: Video Depth Estimation on original domains: Sintel [ 5 ] , Bonn [ 43 ] and KITTI [ 21 ] .

Model	Train/Eval: DROID				Train/Eval: BEDLAM2.0
	Video Depth				Camera			Video Depth
	scale		scale&shift		AUC@5 $\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$	scale		scale&shift
	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	AUC@5 $\uparrow$	AUC@15 $\uparrow$		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$
VGGT	0.306	0.597	0.294	0.645	77.22	91.79	95.84	0.110	0.921	0.086	0.934
SelfEvo (VGGT)	0.254	0.656	0.223	0.733	86.48	95.38	97.68	0.072	0.954	0.051	0.959
$\pi^{3}$	0.271	0.670	0.252	0.709	77.58	92.19	96.08	0.034	0.982	0.028	0.981

Table 4: Generality across base models and training data. We apply our self-improvement framework to VGGT and

\pi^{3}

on two training sources without using annotation. Left: self-improve on DROID and evaluate video depth on DROID. Right: self-improve on BEDLAM2.0 and evaluate video depth and camera on BEDLAM2.0.

Figure 3: Visual result on unseen-domain data, including animal motion, robotics and ego-centric.

Figure 4: Qualitative results for camera and geometry on in the wild videos.

Methods	DROID				HOI4D
	scale		scale&shift		scale		scale&shift
	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$
VGGT [ 59 ]	0.306	0.597	0.294	0.645	0.044	0.957	0.041	0.966
SelfEvo (VGGT)	0.268	0.659	0.237	0.724	0.030	0.988	0.031	0.987

Table 5: unseen domain evaluation. unseen domain evaluation for VGGT self-improved on OmniWorld-Game [ 81 ] and then evaluate on DROID [ 29 ] and HOI4D [ 35 ] .

Dataset	Methods	Video Depth				Camera
		scale		scale&shift		AUC@15 $\uparrow$	AUC@30 $\uparrow$
		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
OmniGeo	aug-stu	0.347	0.619	0.148	0.824	66.476	75.675
	aug-all	0.364	0.607	0.150	0.820	64.432	73.890
	cropping	0.384	0.605	0.147	0.829	64.262	75.990
	dropping	0.296	0.682	0.127	0.862	75.317	84.323

Table 6: Ablation of asymmetry mechanisms on VGGT. “aug-stu” augments student inputs only, while “aug-all” augments both the teacher and student inputs.

Dataset	Methods	Video Depth				Camera
		scale		scale&shift		AUC@15 $\uparrow$	AUC@30 $\uparrow$
		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
OmniGeo	keep-top	0.347	0.619	0.148	0.824	71.423	80.903
	keep-bottom	0.364	0.607	0.150	0.820	78.521	87.013
	random	0.278	0.703	0.124	0.867	79.034	87.285

Table 7: Ablation of frame selection strategy on VGGT.

Dataset	Mode	Video Depth				Camera
		scale		scale&shift		AUC@15 $\uparrow$	AUC@30 $\uparrow$
		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$
OmniGeo	offline	0.357	0.615	0.150	0.817	64.618	74.939
	online	0.278	0.703	0.124	0.867	79.034	87.285

Table 8: Ablation of online supervision. We self-improve on OmniWorld-Game [ 81 ] and evaluate on its benchmark OmniGeo.

Mode	Droid [ 29 ]				Bonn [ 43 ]
	scale		scale & shift		scale		scale & shift
	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$	Abs Rel $\downarrow$	$\delta<1.25$ $\uparrow$
Pretrain (baseline)	0.306	0.597	0.294	0.645	0.055	0.971	0.049	0.974
SFT (Offline)	0.281	0.658	0.264	0.692	0.081	0.895	0.073	0.940
SelfEvo (Online)	${0.285}$	0.622	$\mathbf{0.243}$	0.703	$\mathbf{0.049}$	0.973	$\mathbf{0.044}$	0.976

Table 9: Role of Online Supervision in OOD transfer after target domain adaptation. We compare Pretrain (baseline), SFT (Offline, fully supervised fine-tuning on the target domain) and SelfEvo (Online, unlabeled self-improvement).

Dataset	Methods	Video Depth				Camera
		scale		scale&shift		AUC@15 $\uparrow$	AUC@30 $\uparrow$
		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
OmniGeo	freeze-D	0.325	0.654	0.134	0.851	77.982	86.540
	freeze-A	0.370	0.613	0.162	0.793	62.270	71.497
	freeze-C&D	0.307	0.669	0.133	0.855	77.511	86.236
	train-all	0.292	0.696	0.125	0.865	78.704	86.807

Table 10: Ablation of training recipe on VGGT. freeze-A/C/D denotes freezing the aggregator, camera head, or depth head, respectively; freeze-C&D freezes both heads; train-all updates all modules. Our default is freeze-C (ours).

Dataset	Mode	Video Depth				Camera
		scale		scale&shift		AUC@15 $\uparrow$	AUC@30 $\uparrow$
		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$
OmniGeo	$+\mathcal{L}_{\text{feat}}$	0.283	0.696	0.125	0.867	78.935	87.310
	base	0.278	0.703	0.124	0.867	79.034	87.285

Table 11: Ablation of supervision form.

+\mathcal{L}_{\text{feat}}

add feature matching loss.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Context improves feedforward reconstruction quality. Starting from two temporally distant anchor frames, we progressively add intermediate frames as context and evaluate performance only on the anchors. As the number of input views increases, the overall covisibility increases, while both pointmap and pose errors decrease.

Dataset	Methods	Video Depth				Camera
		scale		scale&shift		AUC@15 $\uparrow$	AUC@30 $\uparrow$
		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
OmniVideo	aug-stu	0.211	0.759	0.125	0.857	86.602	92.303
	aug-all	0.202	0.740	0.129	0.851	85.970	91.851
	cropping	0.228	0.714	0.136	0.841	82.299	90.405
	dropping	0.173	0.810	0.111	0.879	87.611	92.480

Appendix

Table 12: OmniVideo counterpart of the asymmetry ablation on VGGT. “aug-stu” augments student inputs only, while “aug-all” augments both the teacher and student inputs.

Dataset	Methods	Video Depth				Camera
		scale		scale&shift		AUC@15 $\uparrow$	AUC@30 $\uparrow$
		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
OmniVideo	random	0.181	0.792	0.113	0.876	89.132	93.945
	keep-top	0.329	0.632	0.133	0.853	86.602	92.303
	keep-bottom	0.271	0.713	0.130	0.850	85.970	91.851
	probabilistic	0.268	0.698	0.126	0.866	82.299	90.405

Appendix

Table 13: OmniVideo counterpart of the frame selection ablation on VGGT.

Dataset	Mode	Video Depth				Camera
		scale		scale&shift		AUC@15 $\uparrow$	AUC@30 $\uparrow$
		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$
OmniVideo	offline	0.244	0.703	0.145	0.818	85.684	91.793
	online	0.181	0.792	0.113	0.876	89.132	93.945

Appendix

Table 14: OmniVideo counterpart of the online supervision ablation. We self-improve on OmniWorld-Game [ 81 ] and evaluate on its benchmark OmniVideo.

Dataset	Methods	Video Depth				Camera
		scale		scale&shift		AUC@15 $\uparrow$	AUC@30 $\uparrow$
		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$	AUC@15 $\uparrow$	AUC@30 $\uparrow$
OmniVideo	freeze-C(ours)	0.181	0.792	0.113	0.876	89.132	93.945
	freeze-D	0.193	0.761	0.118	0.869	89.281	93.983
	freeze-A	0.263	0.672	0.157	0.800	82.531	89.174
	freeze-C&D	0.197	0.752	0.116	0.873	88.468	93.517

Appendix

Table 15: OmniVideo counterpart of the training recipe ablation on VGGT. freeze-A/C/D denotes freezing the aggregator, camera head, or depth head, respectively; freeze-C&D freezes both heads; train-all updates all modules. Our default is freeze-C (ours).

Dataset	Mode	Video Depth				Camera
		scale		scale&shift		AUC@15 $\uparrow$	AUC@30 $\uparrow$
		Abs Rel $\downarrow$	$\delta<1.25\uparrow$	Abs Rel $\downarrow$	$\delta<1.25\uparrow$
OmniVideo	$+\mathcal{L}_{\text{feat}}$	0.182	0.783	0.114	0.873	89.146	93.840
	base	0.181	0.792	0.113	0.876	89.132	93.945

Appendix

Table 16: OmniVideo counterpart of the supervision form ablation.

+\mathcal{L}_{\text{feat}}

adds a feature matching loss.

Figure 6: Checkpoint-wise depth evaluation on the held-out OmniGeo benchmark. We retrospectively evaluate all training checkpoints of SelfEvo(VGGT) on OmniGeo after training is completed. Top row: scale alignment; bottom row: scale-and-shift alignment. Left: $\delta<1.25$ (higher is better); right: Abs Rel (lower is better). The horizontal gray line denotes the pretrained VGGT baseline. Across both alignment settings, self-improvement leads to substantial gains over the pretrained model throughout training, showing self-improvement on held-out depth benchmarks.

Figure 7: Checkpoint-wise depth evaluation on the held-out OmniVideo benchmark. We retrospectively evaluate all training checkpoints of SelfEvo(VGGT) on OmniVideo after training is completed. Top row: scale alignment; bottom row: scale-and-shift alignment. Left: $\delta<1.25$ (higher is better); right: Abs Rel (lower is better). The horizontal gray line denotes the pretrained VGGT baseline. The curves show clear and sustained improvement across checkpoints, with rapid gains in the early stage and mild saturation later, again supporting self-improvement on depth benchmarks.

Figure 8: Checkpoint-wise camera evaluation on held-out OmniGeo and OmniVideo benchmarks. We retrospectively evaluate all training checkpoints of SelfEvo(VGGT) after training is completed. Top row: OmniGeo; bottom row: OmniVideo. Left: AUC@15; right: AUC@30. Higher is better for all metrics. The horizontal gray line denotes the pretrained VGGT baseline. Camera estimation improves consistently throughout training on both held-out benchmarks, further demonstrating the self-improvement ability of our framework beyond depth prediction alone.

Figure 9: Qualitative results on additional training sources. We further apply SelfEvo to VGGT using two additional unlabeled video sources beyond our main setup: DynPose-100K and Egocentric-10K. The first three examples show results after training on a very small fully in-the-wild subset of DynPose-100K, using only 85 internet videos, while the last example shows results after training on egocentric data. In each pair, the left result is from the pretrained VGGT model and the right result is from SelfEvo(VGGT) .

Video Depth Estimation

See all 17 papers

Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models
Oct 1, 2026Xinhao Xiang, Weiyang Li, Zhijie Zheng +2Multi-View Depth Estimation 4D Reconstruction
IFM Lab, University of California, Davis, CA, USA · Genies Inc., CA, USA
4DVGGT-D: 4D Visual Geometry Transformer with Improved Dynamic Depth Estimation
May 12, 2026Ying Zang, Xuanyi Liu, Yidong Han +94D Reconstruction Depth Estimation
4Huzhou University · 2Peking University · 1KOKONI 3D, Moxin Technology +4
Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself
Apr 15, 2026Yuhang Dai, Xingyi YangFeed-Forward 3D Reconstruction Multi-View 3D Reconstruction
The Hong Kong Polytechnic University, Hong Kong SAR

Oct 1, 2026J/K move · Enter open · S save

Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models

Xinhao Xiang, Weiyang Li, Zhijie Zheng, Abhijeet Rastogi+1

IFM Lab, University of California, Davis, CA, USA · Genies Inc., CA, USA

Multi-View Depth Estimation 4D Reconstruction Dynamic Scene Reconstruction 4D Scene Understanding

Open paper AskPDF arXiv

Recent depth foundation models like Depth Anything 3 (DA3) achieve remarkable multi-view depth estimation but assume static 3D scenes, limiting their applicability to real-world dynamic environments. Existing training-free 4D methods like Easi3R and VGGT4D rely on correspondence-trained backbones whose attention encodes cross-frame matching, a property absent in depth-only models like DA3. We present Dyna3, a training-free framework that extends DA3 for 4D dynamic scene reconstruction without any fine-tuning. Our key insight is that DA3's cross-view features, though trained only for depth consistency, implicitly encode motion-discriminative signals when combined with best-match feature search across frames. Its static surfaces find consistent matches globally, while dynamic objects cannot. We further adopt vision-language models (VLM) to automatically generate scene-specific semantic prompts for SAM 3, enabling precise instance-level segmentation that distinguishes which objects move from what objects exist. For reconstruction, we decouple the scene into a cross-frame aligned static background and per-frame dynamic point clouds. Experiments on four datasets demonstrate that Dyna3 surpasses correspondence-trained methods with +5.5pp J-Mean over state-of-the-art VGGT4D on dynamic object segmentation, while achieving up to 13x faster pose estimation and 3x faster 4D reconstruction with 4 to 8x lower memory. Dyna3 could therefore enable much denser temporal sampling that prior methods cannot support.

Abstract

Video Depth Estimation

Explore similar work

Dyna3: VLM-Guided Training-Free 4D Reconstruction via Depth Foundation Models

4DVGGT-D: 4D Visual Geometry Transformer with Improved Dynamic Depth Estimation

Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself