Organizations: Department of Information Physics and Computing, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Department of Veterinary Neurology and Oncology, Faculty of Agriculture, Tottori University, Tottori, Japan · Department of Ophthalmology, Tokyo Medical University, Tokyo, Japan
Intraoperative scene perception in ophthalmic microsurgery is limited by instrument occlusion: when a tool enters the high-magnification field, it conceals tissue from both the surgeon and the perception components of robot-assisted systems that process the video. Generative inpainting can remove the occluder but replaces the missing image content with synthesis, which is unsuitable when visual provenance matters. Many video inpainters also depend on future frames and cannot operate online. We present TransGHOST, a causal reference-guided reconstruction pipeline whose non-generative safe mode operates in real time. Every restored value is obtained by resampling a recorded frame using geometric transforms fitted only to visible regions. The system provides per-pixel confidence and provenance and marks unsupported pixels as abstentions. We evaluate TransGHOST on a new veterinary clinical corpus of canine cataract surgery with pixel-level instrument and anatomy labels. Only the occluded region is scored, and each scoring target is verified to differ from the reference supplied to the system. Safe mode reaches 31.89 dB masked PSNR and 0.915 masked SSIM at 51.7 ms median and 85.5 ms 95th-percentile latency. On a pre-declared held-out cohort using instrument cutouts excluded from segmentation training, it reaches 29.83 dB masked PSNR, while matched-mask evaluation on validation shows a 3.45 dB advantage over LaMa. Region-aware reference selection improves masked PSNR by 3.60 dB over recency when candidate references can themselves be occluded. Reference-age analysis further defines the operating interval for the brief, repeated occlusions targeted by TransGHOST.
Figures & tables
Figure 1: TransGHOST reconstructs the masked region from an aligned causal reference. Safe mode uses geometric resampling and returns confidence, abstention and provenance layers.
Figure 2: Dataset preparation: (a) an annotated surgical frame with labelled corneal boundaries and instruments, and (b) further annotated frames from the training corpus.
Split
Real
Synthetic
Total images
Train
20,000
40,000
60,000
Test
4,000
6,000
10,000
Table 1: Corpus size after each expansion stage. The real column counts recorded frames carrying hand or confidence-gated pseudo labels and the synthetic column counts semi-synthetic composites. Source videos are assigned to a split before expansion, so no derived frame crosses a split boundary. This is the segmentation partition and is separate from the restoration evaluation sets of Section 5.4 .
Figure 3: Training triplets from three development videos, one per instrument. Each removal mask Mttr unites the instrument mask (white) with one to four random rectangles or circles (blue), which may overlap one another and the instrument. The mask is binary, and the two colours only distinguish its parts. The reference It−30 is the frame 30 frames earlier, and the dotted outline marks the mask region in it. In these examples the instrument has moved away from its own masked region.
Figure 4: Regional reference selection over the causal buffer. The overlap rule selects the newest candidate whose instrument mask leaves the target region visible.
Figure 5: Real-time-constrained TransRef-inspired baseline. Reference features enter through Ref-PA modules and a transfer decoder regenerates the masked stream at 256 px.
Table 2: Segmentation benchmark on the 1,760-frame hand-labelled set. Both models are scored on mask metrics at their best checkpoint, matching Figure 6 . Latency is model inference alone, measured on the same GPU and separately from the deployed segmentation stage timed in Section 6.8 .
Figure 6: Mask F1 against detection confidence for (a) YOLOv8 and (b) YOLOv11, from the validation run of each fine-tuned model. Thin curves are the five classes with enough annotated frames for a stable sweep and the bold curve aggregates all classes, with its maximum given in each panel title.
Figure 7: Instrument segmentation by the deployed model on validation frames, with per-instance confidences. The model predicts the merged instrument channel of Section 4.2 .
Ground-truth masks
Predicted masks (deployed)
Fill rule or stage
PSNR ↑
SSIM ↑
MAE ↓
PSNR ↑
SSIM ↑
MAE ↓
Bnd. MAE ↓
FP MAE ↓
Reference-free and unaligned controls evaluated with ground-truth masks
Occluded input (no restoration)
13.43
0.485
46.21
-
Telea inpainting (no reference)
28.47
0.867
7.64
-
Navier-Stokes inpainting (no reference)
28.58
0.867
7.43
-
Unaligned reference transfer (unaligned)
30.26
0.863
6.73
-
Table 3: Ablation of reference guidance and alignment on 178 validation samples. All rows use the same samples, masks and evaluation code.
Figure 8: Safe-mode results on six validation samples, labelled (a) to (f), each set on its own tinted strip of four panels. Within a strip the panels are the occluded frame, the reference selected for it, the restored composite produced by the deployed configuration, and a twofold magnification of the restored region. The solid outline traces the instrument mask recorded for that sample. The dotted outline marks that same boundary at identical image coordinates on the other three panels, where the region is unobstructed in the reference and the instrument no longer appears in the output.
Table 4: Masked-region fidelity on matched samples. Learned baselines use ground-truth masks. Rows with predicted masks show deployed TransGHOST. Bnd. denotes boundary MAE. LPIPS is reported for validation.
Unobstructed references
Occluded references
Selection rule
PSNR
Abst.
PSNR
Δ
SSIM
Abst.
Newest valid reference (recency)
33.42
0.000
26.44
-
0.854
0.154
Composite quality score
n/a
n/a
27.14
+0.70
0.862
0.134
Least region difference [ 28 ]
n/a
n/a
26.41
-0.03
0.857
0.145
Least mask overlap (ours)
33.37
0.000
30.04
+3.60
0.904
0.037
Table 5: Reference-selection rules on unobstructed and controlled-occlusion candidate sets. Abst. denotes the proportion of reconstruction pixels marked as abstentions.
Distribution
Candidates hiding
Partial-support
Masked PSNR
Abstention
the region
samples
recency
overlap
gain
recency
overlap
Light
16%
6
31.29
32.63
+1.34
0.043
0.009
Medium
41%
27
26.44
30.04
+3.60
0.154
0.037
Heavy
64%
53
23.36
27.76
+4.40
0.219
0.068
Table 6: Reference-selection gains under light, medium and heavy distributions of controlled occlusion.
Stage
PSNR ↑
Δ
SSIM ↑
MAE ↓
Bnd. MAE ↓
FP MAE ↓
Global affine registration only
28.25
-
0.861
6.76
3.83
6.63
+ residual translation
29.13
+ 0.89
0.883
5.81
3.26
5.50
+ sub-pixel refinement
29.26
+ 1.01
0.885
5.71
3.19
5.36
+ per-block local offsets
29.83
+ 1.58
0.893
5.01
2.69
4.34
Table 7: Held-out ablation on 101 samples from three videos excluded from training, using instrument cutouts from the test split.
Figure 9: Masked fidelity against reference age. The shaded band spans the classical reference-free controls of Table 3 , and the dashed series is the fixed-cohort control of Section 6.7 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Selection policy
PSNR ↑
Δ
SSIM ↑
MAE ↓
Nearest valid reference (ours)
33.42
-
0.933
2.93
Random valid reference
31.06
-2.36
0.910
4.64
Oldest valid reference
29.19
-4.23
0.888
6.37
Oracle on pre-alignment agreement
33.35
-0.07
0.931
3.02
Appendix
Table B1: Reference choice with alignment, masks and samples held fixed.
Variant
PSNR ↑
Δ
p95 (ms) ↓
p99 (ms) ↓
Multi-reference evaluation set, one reference per candidate lag
Single nearest reference (ours)
33.42
-
96
120
Median fusion of 5 references
33.39
−0.03
109
744
Per-block choice among 5 references
33.42
0.00
109
343
Base validation set, deployed configuration
Deployed configuration (ours)
31.89
-
99
118
Appendix
Table C1: Design-selection experiments comparing the reported configuration with measured alternatives.
Figure D1: End-to-end latency percentiles for three implementations of the alignment search on the validation set. The median changes by several milliseconds, whereas the 99th percentile decreases by approximately one quarter.
Configuration
PSNR ↑
p50 (ms) ↓
p95 (ms) ↓
p99 (ms) ↓
Initial measurement
Without per-block local offsets
29.25
57.6
87.2
100.5
Full configuration
29.83
61.0
102.7
116.9
Confirmatory measurement, sparse band sampling
Without per-block local offsets
29.26
52.1
83.1
90.0
Full configuration
29.83
55.9
86.1
101.2
Appendix
Table D1: Initial and confirmatory end-to-end latency on the held-out cohort.
All available samples
Fixed cohort
Lag
n
PSNR ↑
SSIM ↑
MAE ↓
Bnd. MAE ↓
PSNR ↑
Δ vs lag 1 [95% CI]
1
178
33.42
0.933
2.93
1.58
33.37
-
2
171
31.95
0.920
3.75
2.04
31.94
−1.43[−2.93,−1.21]
3
166
31.20
0.910
4.35
2.38
31.16
−2.22[−4.75,−1.95]
5
150
30.12
0.896
5.47
2.96
30.17
−3.20[−7.02,−3.04]
8
120
28.86
0.884
6.85
3.65
28.93
−4.45[−7.89,−3.96]
Appendix
Table E1: Reference-age analysis at 29.97 fps. The fixed-cohort block contains samples available at every lag. Bracketed values are 95 per cent intervals from a clip-clustered paired bootstrap against lag 1.
Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.
Real-time reconstruction of deformable surgical scenes is vital for advancing robotic surgery, improving intraoperative guidance, and enabling automation. Recent methods achieve dense reconstructions from da Vinci robotic surgery videos, with Gaussian Splatting (GS) offering real-time performance via graphics acceleration. However, reconstruction quality in occluded regions remains limited, and depth accuracy has not been fully assessed, as benchmarks like EndoNeRF and StereoMIS lack 3D ground truth. We propose Diff2DGS, a two-stage framework for reliable 3D reconstruction of occluded surgical scenes. First, a diffusion-based video module with temporal priors inpaints tissue occluded by instruments with high spatiotemporal consistency. Second, we adapt 2D Gaussian Splatting (2DGS) with a Learnable Deformation Model (LDM) to capture dynamic tissue deformation and anatomical geometry, and introduce adaptive depth weight to improve geometric fidelity. We further extend evaluation beyond image-quality metrics by performing quantitative depth analysis on the SCARED dataset. Diff2DGS outperforms state-of-the-art methods in both appearance and geometry, reaching 38.02 dB PSNR on EndoNeRF and 34.40 dB on StereoMIS. Our experiments also show that optimizing image quality alone does not necessarily ensure accurate 3D geometry.The code is available at https://diff2dgs.github.io/.
We present a real-time, hybrid framework for occlusion handling and depth-aware tracking in minimally invasive surgeries (MIS). Occlusions caused by surgical instruments often obscure the region of interest (ROI), affecting surgical precision. To address this, our approach integrates enhanced You Only Look Once (YOLO) based detection, Particle Filtering (PF), and monocular depth estimation using Multiple Depth Estimation Accuracy with Single Network (MiDaS). Occlusion is detected using a distance-based threshold between instruments and ROI. Experimental results demonstrate robust instrument detection with a confidence score of 0.85, effective occlusion event handling, and accurate depth prediction with an MAE of 0.34 cm and RMSE of 0.53 cm. Independent phantom-based validation against stereo depth measurements yielded an MAE of 0.65 cm and RMSE of 0.66 cm. The framework further achieved an occlusion detection accuracy of 81.95% while maintaining real-time performance at 38.7 FPS. Preliminary surgeon feedback indicates that the proposed visualization framework is clinically relevant and holds promise for future intraoperative image-guided navigation and decision support in MIS.
Anuska Sarkar, Partha Acharya, Vishnu Nandury +2
School of Medical Science and Technology, Indian Institute of Technology Kharagpur, Kharagpur 721302 India · Computer Science Department, Arizona State University, Tempe, Arizona 85287 USA · Narayana Hospital, Kolkata 700107 India
Objective assessment of robotic surgery uses instrument kinematics, which must be reconstructed when only video is available. We introduce a kinematic reconstruction network for estimating instrument position, orientation and jaw angle from monocular video. Our visual representation combines global attention pooling of frozen DINOv3 features with local pooling at instrument landmarks from fine-tuned SAM 3.1 masks. Our shared Transformer encoder and temporal convolutional heads integrate this representation with mask geometry, monocular depth and visual state estimates from arm-specific multilayer regression networks. Our position branch predicts displacement magnitude and direction separately to preserve traveled distance. We fit trajectories to predicted state observations and motion increments by differentiable weighted least squares, expressing quaternion observations relative to cumulative predicted rotations to obtain a quadratic orientation objective. We evaluate reconstruction across 2,802 Open-H episodes. Compared with LiveMAE on the main Open-H benchmark, our method reduces path-length mean absolute error from 0.45 to 0.34,cm and increases temporal mean average precision for motion segmentation from 44.54% to 54.44%.