NeRFscopy: Neural Radiance Fields for in-vivo Time-Varying Tissues from Endoscopy

Laura Salort-Benejam; Antonio Agudo

cs.CVFeb 17, 2026

NeRFscopy: Neural Radiance Fields for in-vivo Time-Varying Tissues from Endoscopy

Authors: Laura Salort-Benejam, Antonio Agudo

Organizations: Institut de Rob`otica i Inform`atica Industrial, CSIC-UPC, Spain

Abstract

Endoscopy is essential in medical imaging, used for diagnosis, prognosis and treatment. Developing a robust dynamic 3D reconstruction pipeline for endoscopic videos could enhance visualization, improve diagnostic accuracy, aid in treatment planning, and guide surgery procedures. However, challenges arise due to the deformable nature of the tissues, the use of monocular cameras, illumination changes, occlusions and unknown camera trajectories. Inspired by neural rendering, we introduce NeRFscopy, a self-supervised pipeline for novel view synthesis and 3D reconstruction of deformable endoscopic tissues from a monocular video. NeRFscopy includes a deformable model with a canonical radiance field and a time-dependent deformation field parameterized by SE(3) transformations. In addition, the color images are efficiently exploited by introducing sophisticated terms to learn a 3D implicit model without assuming any template or pre-trained model, solely from data. NeRFscopy achieves accurate results in terms of novel view synthesis, outperforming competing methods across various challenging endoscopy scenes.

Figures & tables

Figure 1: Overview of our NeRFscopy pipeline for time-varying NeRF from RGB endoscopy videos.

		PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
TECAB1	DPT [ 17 ]	26.096	0.767	0,248
	IID-SfmLearner [ 12 ]	26.054	0.761	0.246
	Depth-Anything [ 28 ]	25.743	0.750	0.263
TECAB2	DPT [ 17 ]	24.716	0.676	0.426
	IID-SfmLearner [ 12 ]	24.938	0.692	0.418
	Depth-Anything [ 28 ]	24.904	0.678	0.423

Table 1: Depth term evaluation on four real videos by using 3 monocular depth estimators (DPT [ 17 ] , IID-SfmLearner [ 12 ] and Depth-Anything [ 28 ] ) in our NeRFscopy.

Panel of Figure 2 — Figure 2: Qualitative evaluation of the depth regularizer on the lung lobectomy video. Columns show from left to right: arbitrary input frame, RGB rendered image, input depth estimation and depth rendered result. For depth initialization, we consider DPT [ 17 ] (top), IID-SfmLearner [ 12 ] (middle) or Depth-Anything [ 28 ] (bottom).

		PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
TECAB1	EndoNeRF [ 25 ]	25.791	0.742	0.255
	NeRFscopy baseline	25.743	0.750	0.263
	NeRFscopy w/ grad	25.874	0.757	0.250
	NeRFscopy w/ grad, smooth	25.811	0.750	0.255
	NeRFscopy w/ grad, smooth, tv	25.137	0.743	0.274
TECAB2	EndoNeRF [ 25 ]	24.954	0.685	0.419

Table 2: Ablation study of NeRFscopy and comparison with EndoNeRF [ 25 ] . Each prior in Eq. ( 5 ) is evaluated with the NeRFscopy baseline. Results shown as best and second best .

Panel of Figure 3 — Figure 3: Qualitative evaluation on real videos. Columns show from left to right: arbitrary input frame, RGB rendering, input depth estimation, depth rendering, and side 3D view. From top to bottom: TECAB1, TECAB2, lung lobectomy and bronchoscopy images.

	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
EndoNeRF [ 25 ]	29.831	0.925	0.081
EndoSurf [ 29 ]	34.993	0.954	0.113
LerPlane-32k [ 27 ]	35.504	0.935	0.083
EndoGaussian-monocular [ 13 ]	36.429	0.951	0.089
Ours	37.204	0.949	0.054

Table 3: Quantitative evaluation on the EndoNeRF dataset . The table reports PSNR, SSIM and LPIPS for EndoNeRF [ 25 ] , EndoSurf [ 29 ] , LerPlane-32k [ 27 ] and EndoGaussian-monocular [ 13 ] ; and our NeRFscopy method. Results shown as best and second best .

Panel of Figure 4 — Figure 4: Rendered results for five consecutive frames. Novel views highlighted in red. From top to bottom: TECAB1, TECAB2, lung lobectomy and bronchoscopy images.

Endoscopic 3D Reconstruction

See all 18 papers

Endo-NeRF++: Uncertainty-Aware Neural Rendering with Multi-Resolution Hash Encoding for Dynamic Surgical Scene Reconstruction
Jul 30, 2026Gousia Habib, Laura Ruotsalainen3D Reconstruction Neural Rendering
Department of Computer Science, University of Helsinki, Finland
4D Monocular Surgical Reconstruction under Arbitrary Camera Motions
Feb 19, 2026Jiwei Shan, Zeyu Cai, Cheng-Tai Hsieh +54D Reconstruction 3D Reconstruction
Department of Mechanical and Automation Engineering and T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong. · Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, China · State Key Laboratory of Robotics and Intelligent Systems, Shenyang, China +3
ExtraGS: Enhancing Endoscopic View Extrapolation via Diffusion-Guided 3D Gaussian Splatting
Jul 14, 2026Cheng-Tai Hsieh, Jiwei Shan, Han Fang +63D Gaussian Splatting Endoscopy
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, and the Shanghai Key Laboratory of Navigation and Location-Based Services, Shanghai, China · Department of Mechanical and Automation Engineering and the T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong · Global College, Shanghai Jiao Tong University, Shanghai, China +1

Jul 30, 2026J/K move · Enter open · S save

Endo-NeRF++: Uncertainty-Aware Neural Rendering with Multi-Resolution Hash Encoding for Dynamic Surgical Scene Reconstruction

Gousia Habib, Laura Ruotsalainen

Department of Computer Science, University of Helsinki, Finland

3D Reconstruction Neural Rendering Dynamic Scene Reconstruction Uncertainty Sampling Endoscopic 3D Reconstruction Surgical Robotics

Open paper AskPDF arXiv

Reconstructing dynamic surgical scenes is crucial for robot-assisted minimally invasive surgery; however, it continues to be difficult because of tissue deformation, occlusions, specular reflections, and restricted viewpoints. In this study, we introduce Endo-NeRF++, a neural rendering framework that accounts for uncertainty in the reconstruction of dynamic surgical scenes. Expanding on EndoNeRF, the suggested approach incorporates multi-resolution hash-grid encoding, temporal feature merging, and uncertainty-informed adaptive sampling to enhance reconstruction accuracy and temporal coherence in deformable endoscopic scenes.The multi-resolution hash-grid representation within the framework effectively captures both coarse and fine anatomical details, while temporal feature blending ensures stable reconstruction during tissue deformation and surgical tool occlusions. Additionally, uncertainty-driven adaptive sampling assigns more samples to uncertain areas to enhance rendering quality and geometric coherence. Experiments on robotic surgical video sequences demonstrate that the proposed uncertainty-guided adaptive sampling improves PSNR by up to 1.22dB (4.3%), increases SSIM by up to 5.3%, and reduces LPIPS by up to 55.1% compared with the EndoNeRF baseline.

Abstract

Endoscopic 3D Reconstruction

Explore similar work

Endo-NeRF++: Uncertainty-Aware Neural Rendering with Multi-Resolution Hash Encoding for Dynamic Surgical Scene Reconstruction

4D Monocular Surgical Reconstruction under Arbitrary Camera Motions

ExtraGS: Enhancing Endoscopic View Extrapolation via Diffusion-Guided 3D Gaussian Splatting