SCOPE-4D: Endoscopic 4D Geometry Foundation Models

Chaoyi Zhou; Zhongpai Gao; Anwesa Choudhuri; Meng Zheng; Benjamin Planche; Run Wang; Terrence Chen; Siyu Huang

cs.CVOct 1, 2026

SCOPE-4D: Endoscopic 4D Geometry Foundation Models

Authors: Chaoyi Zhou, Zhongpai Gao, Anwesa Choudhuri, Meng Zheng, Benjamin Planche, Run Wang, Terrence Chen, Siyu Huang,

Organizations: United Imaging Intelligence, Boston, MA, USA · Clemson University, Clemson, SC, USA

Abstract

Geometric understanding supports endoscopic navigation and robotic assistance, but learning reliable endoscopic geometry faces two challenges: scarce geometric annotations and ambiguity between camera motion and tissue deformation. We present SCOPE-4D, an endoscopic 4D geometry foundation model that jointly predicts camera parameters, dense geometry, and 3D tissue trajectories from monocular RGB video in a single forward pass. Our curation and annotation pipeline constructs SCOPE-5K, a collection of approximately 5,000 clips spanning real and synthetic gastrointestinal endoscopy and laparoscopy. The collection provides rich geometric supervision and includes newly collected phantom and real-colonoscopy evaluation sets. Geometric supervised fine-tuning on SCOPE-5K learns endoscopic priors that improve camera and depth estimation. Common--Residual Motion (CRM) further constrains local deformation relative to common tissue movement. Together with geometric supervision, CRM and trajectory supervision further improve camera and depth estimation over geometric fine-tuning alone while enabling dense 3D tissue tracking. Evaluations on public and newly collected benchmarks demonstrate strong in-domain and out-of-domain geometry, superior 3D tracking, and more stable long-sequence colon reconstruction. A blinded user study further supports the perceived reconstruction quality on real clinical video. Together, these results demonstrate the value of large-scale endoscopic supervision and motion constraints for joint geometry estimation and tissue tracking.

Figures & tables

Figure 1: SCOPE-4D overview. (a) Our curation and annotation pipeline builds SCOPE-5K for geometric supervision and cross-domain evaluation. SCOPE-4D (b) achieves strong cross-domain geometry, (c) recovers tissue motion while preserving camera stability in a stationary-camera sequence, unlike SM4RT, and (d) enables stable long-sequence reconstruction.

Figure 2: Data annotation and SCOPE-4D. (a) SCOPE-5K curation and annotation. (b) Joint camera, geometry, and 3D tissue-track prediction. (c) Common–Residual Motion composes common tissue movement with local deformation.

	In-domain average					Hamlyn (OOD)				SCOPE-colon-phantom (OOD)
	Pose			Depth		Pose		Depth		Pose
Method	AUC@ $5^{\circ}\uparrow$	AUC@ $30^{\circ}\uparrow$	ATE $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$	ATE $\downarrow$	ARE $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$	ATE $\downarrow$	RTE $\downarrow$	ARE ${}^{*}\downarrow$
General-domain
VGGT	2.94	29.11	0.128	0.160	74.61	4.610	1.964	0.190	69.19	16.53	16.78	12.14
VGGT- $\Omega$	7.81	48.07	0.049	0.093	91.07	4.496	1.746	0.135	80.82	8.54	10.79	8.22
MapAnything	2.63	26.22	0.122	0.141	85.00	4.984	1.960	0.166	74.15	15.98	17.70	26.98

Table 1: Camera pose and depth across domains. Hamlyn pose measures camera stability; phantom ARE ∗ excludes roll. Gain compares our best variant per metric with the best competing method.

Figure 3: Qualitative comparison of 3D geometry reconstruction. Instrument regions are masked out. Highlighted regions show surface consistency and suppression of spurious camera motion.

	Sano							StereoMIS
	Tracking		Pose			Depth		Tracking		Pose			Depth
Method	EPE $\downarrow$	APD $\uparrow$	ATE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$	EPE $\downarrow$	APD $\uparrow$	ATE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
D4RT ∗	2.035	72.94	0.014	0.239	0.215	0.193	65.76	7.540	34.12	0.043	0.940	0.344	0.147	87.20
SpatialTrackerV2	0.933	90.58	0.017	0.223	0.125	0.145	78.76	6.870	38.98	0.049	0.677	0.195	0.107	95.24
4RC	0.523	94.72	0.006	0.100	0.062	0.186	66.15	6.266	44.28	0.039	0.481	0.146	0.114	90.42
Any4D	0.535	94.41	0.010	0.376	0.309	0.188	66.01	7.277	42.45	0.061	2.203	0.765	0.144	82.72

Table 2: 3D tracking and geometry on Sano and StereoMIS. D4RT ∗ : open-source reproduction.

Figure 4: 3D tissue tracking under different motion conditions. Instrument regions are masked out. Red boxes highlight the suppression of spurious motion.

Table 3: Data and motion ablations. (a) Geometry-only SFT by data source. (b) StereoMIS motion design. (c) SoM vs. CRM. Gain in (a): full dataset versus pretrained VGGT- $\Omega$ .

Method	ATE $\downarrow$	RTE $\downarrow$	RRE $\downarrow$
VGGT	93.21	23.65	15.11
VGGT- $\Omega$	85.92	21.67	31.21
SM4RT	88.95	21.29	16.98
Endo3R	77.87	14.75	22.92
SCOPE-4D SFT (Ours)	73.52	10.38	13.38
SCOPE-4D (Ours)	71.17	10.94	13.61

Table 8

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Data source	Domain		Data type		Annotations		Clips
Data source	GI	Laparo	Synthetic	Real	Depth	Camera	Clips
EndoMapper-Real ( Azagra et al., 2023 )	$\checkmark$	–	–	$\checkmark$	$\checkmark^{\dagger}$	$\checkmark^{\dagger}$	2,379
C3VD ( Bobrow et al., 2023 )	$\checkmark$	–	–	$\checkmark$	$\checkmark$	$\checkmark$	112
SimCol3D ( Rau et al., 2024 )	$\checkmark$	–	$\checkmark$	–	$\checkmark$	$\checkmark$	33
EndoMapper-Sim ( Azagra et al., 2023 )	$\checkmark$	–	$\checkmark$	–	$\checkmark$	$\checkmark$	5
Sano ( Nelson et al., 2026 )	–	$\checkmark$	$\checkmark$	–	$\checkmark$	$\checkmark$	1,938

Appendix

Table 6: Data composition and annotation coverage of SCOPE-5K. Annotations reflect those available in SCOPE-5K.

\dagger

denotes annotations generated by our pipeline.

Dataset	Collected		Experimental split			Sampling (%)
Dataset	Clips	Views	Train	Val.	Test	Sampling (%)
In-domain
SCARED	34	24,046	24	2	8	13.86
SimCol3D	33	37,833	22	2	9	13.25
C3VD	112	43,164	99	2	11	18.07
EndoMapper-Sim	5	1,610	3	1	1	3.01

Appendix

Table 7: Dataset sizes, splits, and sampling proportions.

Check	Criterion	Role
Registration	One significant reconstruction; at least 90% of frames registered	Hard gate
Integrity	Readable depth, consistent frame counts, finite cameras	Hard gate
Connectivity	Largest point-cloud component at least 90%	Hard gate
Coverage	Valid depth over at least 20% of clip pixels	Hard gate
Parallax	Median maximum sparse-point parallax at least $1.5^{\circ}$	Soft flag
Camera adoption	Dense-model camera change at most 1.5% translation and $1^{\circ}$ rotation	Soft flag

Appendix

Table 8: Clip-level geometry checks.

Stage	Clips	Frames
Raw videos (93 recordings)	–	3,973,969
Clean clips (Stage 1)	3,034	319,727
Accept	1,733	177,358
Borderline	657	70,501
Reject	644	71,868
Retained collection	2,379	246,981

Appendix

Table 9: EndoMapper-Real curation counts. Frame counts precede training conversion.

Method	Input	Output	Resolution	Supplied intrinsics	Test-time optimization
General 3D
VGGT ( Wang et al., 2025a )	RGB video	Depth, cameras, points	518	$\times$	$\times$
VGGT- $\Omega$ ( Wang et al., 2026 )	RGB video	Depth, cameras, points	512	$\times$	$\times$
MapAnything ( Keetha et al., 2026 )	RGB video	Depth, cameras, points	518	$\times$	$\times$
CUT3R ( Wang et al., 2025b )	RGB video	Depth, cameras, points	512	$\times$	$\checkmark$
General 4D

Appendix

Table 10: Compared methods.

Figure 6: User-study interface. Reference video, masked depth, interactive point clouds, and ranking controls for four anonymous methods. No rankings are selected.

	Pose			Depth
Method	AUC@ $5^{\circ}\uparrow$	AUC@ $30^{\circ}\uparrow$	ATE $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
VGGT	$10.31$	$64.78$	$0.047$	$0.047$	$99.15$
VGGT- $\Omega$	$17.87$	$76.51$	$0.030$	$0.038$	$99.36$
MapAnything	$6.10$	$56.83$	$0.050$	$0.071$	$95.00$
CUT3R	$0.35$	$11.80$	$0.174$	$0.135$	$81.43$
SpatialTrackerV2	$8.65$	$68.50$	$0.049$	$0.038$	$99.35$

Appendix

Table 11: Camera pose and depth estimation on SCARED.

	Pose			Depth
Method	AUC@ $5^{\circ}\uparrow$	AUC@ $30^{\circ}\uparrow$	ATE $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
VGGT	$0.44$	$10.82$	$0.288$	$0.181$	$68.03$
VGGT- $\Omega$	$6.69$	$52.96$	$0.072$	$0.093$	$92.77$
MapAnything	$0.22$	$7.39$	$0.258$	$0.172$	$71.11$
CUT3R	$0.09$	$11.28$	$0.223$	$0.459$	$32.57$
SpatialTrackerV2	$0.79$	$35.24$	$0.105$	$0.155$	$75.69$

Appendix

Table 12: Camera pose and depth estimation on SimCol3D.

	Pose			Depth
Method	AUC@ $5^{\circ}\uparrow$	AUC@ $30^{\circ}\uparrow$	ATE $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
VGGT	$7.37$	$50.19$	$0.097$	$0.196$	$65.00$
VGGT- $\Omega$	$18.54$	$70.34$	$0.042$	$0.062$	$97.05$
MapAnything	$9.37$	$60.34$	$0.056$	$0.144$	$78.64$
CUT3R	$0.51$	$22.06$	$0.138$	$0.406$	$38.45$
SpatialTrackerV2	$25.32$	$77.26$	$\underline{0.031}$	$0.076$	$94.10$

Appendix

Table 13: Camera pose and depth estimation on C3VD.

	Pose			Depth
Method	AUC@ $5^{\circ}\uparrow$	AUC@ $30^{\circ}\uparrow$	ATE $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
VGGT	$0.05$	$19.17$	$0.228$	$0.153$	$79.49$
VGGT- $\Omega$	$1.86$	$47.59$	$0.067$	$\underline{0.062}$	$99.51$
MapAnything	$0.04$	$8.53$	$0.265$	$0.088$	$94.00$
CUT3R	$0.03$	$21.75$	$0.120$	$0.301$	$49.62$
SpatialTrackerV2	$0.42$	$43.41$	$0.060$	$0.077$	$97.55$

Appendix

Table 14: Camera pose and depth estimation on EndoMapper-Sim.

	Pose			Depth
Method	AUC@ $5^{\circ}\uparrow$	AUC@ $30^{\circ}\uparrow$	ATE $\downarrow$	AbsRel $\downarrow$	$\delta_{1.25}\uparrow$
VGGT	$1.41$	$31.43$	$0.155$	$0.214$	$61.56$
VGGT- $\Omega$	$3.22$	$49.08$	$0.089$	$0.157$	$76.44$
MapAnything	$1.49$	$27.23$	$0.156$	$0.274$	$81.11$
CUT3R	$0.20$	$17.54$	$0.190$	$0.408$	$37.70$
SpatialTrackerV2	$3.03$	$54.16$	$0.097$	$0.194$	$68.16$

Appendix

Table 15: Camera pose and depth estimation on EndoMapper-Real.

	Sano				StereoMIS
Method	APD@1 $\uparrow$	APD@2 $\uparrow$	APD@5 $\uparrow$	APD@10 $\uparrow$	APD@1 $\uparrow$	APD@2 $\uparrow$	APD@5 $\uparrow$	APD@10 $\uparrow$
D4RT ∗	33.99	66.11	92.98	98.69	4.29	10.56	42.99	78.63
SpatialTrackerV2	77.81	88.97	96.50	99.03	6.87	18.56	50.80	79.70
4RC	86.91	94.26	98.08	99.63	12.25	26.19	56.17	82.50
Any4D	87.29	93.65	97.35	99.34	12.02	25.08	54.46	78.25
SM4RT	89.48	95.28	98.17	99.62	12.20	25.26	56.14	82.21

Appendix

Table 16: 3D tracking APD at multiple thresholds.

Variant	APD@1 $\uparrow$	APD@2 $\uparrow$	APD@5 $\uparrow$	APD@10 $\uparrow$
SoM (baseline)	11.22	28.89	63.67	82.55
w/ zero-motion anchor	12.73	33.55	66.33	87.83
Full CRM	19.02	37.30	66.89	87.30

Appendix

Table 17: Motion ablation on StereoMIS.

Figure 7: Additional 3D geometry comparisons.

Figure 8: Additional 3D geometry comparisons.

Figure 9: Additional 3D geometry comparisons.

Figure 10: Additional 3D geometry comparisons.

Figure 11: Additional 3D geometry comparisons.

Figure 12: Additional 3D geometry comparisons.

Figure 13: Additional 3D geometry comparisons.

Figure 14: Additional 3D geometry comparisons.

Figure 15: Additional depth comparisons.

Panel of Figure 16 — Figure 16: Long-sequence reconstruction on RealSynCol Colon 2.

Endoscopic 3D Reconstruction

See all 18 papers

Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation
Jun 15, 2026Hongchao Shu, Roger D. Soberanis-Mukul, Hao Ding +5Endoscopy Monocular Depth Estimation
Department of Computer Science, Johns Hopkins University, Baltimore, MD 21211 USA · Semaphor Surgical · Johnson & Johnson MedTech, Santa Clara, CA 95054 USA
4D Monocular Surgical Reconstruction under Arbitrary Camera Motions
Feb 19, 2026Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh +54D Reconstruction 3D Reconstruction
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China · State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China +3
SCOPE: Scale-Consistent One-Pass Estimation of 3D Geometry
Jun 19, 2026Zheng Zhang, Lihe Yang, Tianyu Yang +6Streaming 3D Reconstruction 3D Reconstruction
The University of Hong Kong, Hong Kong · Alibaba Group, Hong Kong · Alibaba Group, China +2

Jun 15, 2026J/K move · Enter open · S save

Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation

Hongchao Shu, Roger D. Soberanis-Mukul, Hao Ding, Morgan Ringel+4

Department of Computer Science, Johns Hopkins University, Baltimore, MD 21211 USA · Semaphor Surgical · Johnson & Johnson MedTech, Santa Clara, CA 95054 USA

Endoscopy Monocular Depth Estimation Vision Foundation Model Adaptation Medical Image Representation Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Accurate vision-based navigation in monocular endoscopy is difficult due to limited depth cues, weak tissue texture, non-rigid deformation, and substantial appearance variation across domains, all of which complicate pose estimation, depth prediction, and image-to-anatomy alignment. Although recent vision foundation models have shown promise, their learned representations often remain insufficiently geometry-consistent, hindering stable feature correspondence and limiting their reliability for downstream navigation tasks. We propose a unified framework for learning geometry-consistent and domain-robust image representations for monocular endoscopy. The framework combines a synthetic data pipeline that provides accurate geometric supervision with Hierarchy-Aware Geometry-Semantic Adaptation, a structured alternative to standard LoRA that inserts low-rank adapters selectively across the transformer hierarchy and couples them with layer-wise training objectives to encourage geometric correspondence in intermediate features and semantic consistency in deeper features. Experiments on public and proprietary datasets show improved geometric and semantic representation quality, leading to better performance on downstream navigation tasks including pose estimation and monocular depth estimation. The learned representations show favorable synthetic-to-real transfer on clinical bronchoscopy and provide a useful initialization for adaptation to sinus endoscopy and colonoscopy under limited supervision. The framework also shows favorable scaling with model size and training data. These results support hierarchy-aware, geometry-guided adaptation as a practical approach for endoscopic representation learning.

Abstract

Endoscopic 3D Reconstruction

Explore similar work

Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

SCOPE: Scale-Consistent One-Pass Estimation of 3D Geometry