Organizations: Department of Information Physics and Computing, Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan · Department of Veterinary Neurology and Oncology, Faculty of Agriculture, Tottori University, Tottori, Japan · Department of Ophthalmology, Tokyo Medical University, Tokyo, Japan
Intraoperative scene perception in ophthalmic microsurgery is limited by instrument occlusion: when a tool enters the high-magnification field, it conceals tissue from both the surgeon and the perception components of robot-assisted systems that process the video. Generative inpainting can remove the occluder but replaces the missing image content with synthesis, which is unsuitable when visual provenance matters. Many video inpainters also depend on future frames and cannot operate online. We present TransGHOST, a causal reference-guided reconstruction pipeline whose non-generative safe mode operates in real time. Every restored value is obtained by resampling a recorded frame using geometric transforms fitted only to visible regions. The system provides per-pixel confidence and provenance and marks unsupported pixels as abstentions. We evaluate TransGHOST on a new veterinary clinical corpus of canine cataract surgery with pixel-level instrument and anatomy labels. Only the occluded region is scored, and each scoring target is verified to differ from the reference supplied to the system. Safe mode reaches 31.89 dB masked PSNR and 0.915 masked SSIM at 51.7 ms median and 85.5 ms 95th-percentile latency. On a pre-declared held-out cohort using instrument cutouts excluded from segmentation training, it reaches 29.83 dB masked PSNR, while matched-mask evaluation on validation shows a 3.45 dB advantage over LaMa. Region-aware reference selection improves masked PSNR by 3.60 dB over recency when candidate references can themselves be occluded. Reference-age analysis further defines the operating interval for the brief, repeated occlusions targeted by TransGHOST.
Figures & tables
Figure 1: TransGHOST reconstructs the masked region from an aligned causal reference. Safe mode uses geometric resampling and returns confidence, abstention and provenance layers.
Figure 2: Dataset preparation: (a) an annotated surgical frame with labelled corneal boundaries and instruments, and (b) further annotated frames from the training corpus.
Split
Real
Synthetic
Total images
Train
20,000
40,000
60,000
Test
4,000
6,000
10,000
Table 1: Corpus size after each expansion stage. The real column counts recorded frames carrying hand or confidence-gated pseudo labels and the synthetic column counts semi-synthetic composites. Source videos are assigned to a split before expansion, so no derived frame crosses a split boundary. This is the segmentation partition and is separate from the restoration evaluation sets of Section 5.4 .
Figure 3: Training triplets from three development videos, one per instrument. Each removal mask Mttr unites the instrument mask (white) with one to four random rectangles or circles (blue), which may overlap one another and the instrument. The mask is binary, and the two colours only distinguish its parts. The reference It−30 is the frame 30 frames earlier, and the dotted outline marks the mask region in it. In these examples the instrument has moved away from its own masked region.
Figure 4: Regional reference selection over the causal buffer. The overlap rule selects the newest candidate whose instrument mask leaves the target region visible.
Figure 5: Real-time-constrained TransRef-inspired baseline. Reference features enter through Ref-PA modules and a transfer decoder regenerates the masked stream at 256 px.
Table 2: Segmentation benchmark on the 1,760-frame hand-labelled set. Both models are scored on mask metrics at their best checkpoint, matching Figure 6 . Latency is model inference alone, measured on the same GPU and separately from the deployed segmentation stage timed in Section 6.8 .
Figure 6: Mask F1 against detection confidence for (a) YOLOv8 and (b) YOLOv11, from the validation run of each fine-tuned model. Thin curves are the five classes with enough annotated frames for a stable sweep and the bold curve aggregates all classes, with its maximum given in each panel title.
Figure 7: Instrument segmentation by the deployed model on validation frames, with per-instance confidences. The model predicts the merged instrument channel of Section 4.2 .
Ground-truth masks
Predicted masks (deployed)
Fill rule or stage
PSNR ↑
SSIM ↑
MAE ↓
PSNR ↑
SSIM ↑
MAE ↓
Bnd. MAE ↓
FP MAE ↓
Reference-free and unaligned controls evaluated with ground-truth masks
Occluded input (no restoration)
13.43
0.485
46.21
-
Telea inpainting (no reference)
28.47
0.867
7.64
-
Navier-Stokes inpainting (no reference)
28.58
0.867
7.43
-
Unaligned reference transfer (unaligned)
30.26
0.863
6.73
-
Table 3: Ablation of reference guidance and alignment on 178 validation samples. All rows use the same samples, masks and evaluation code.
Figure 8: Safe-mode results on six validation samples, labelled (a) to (f), each set on its own tinted strip of four panels. Within a strip the panels are the occluded frame, the reference selected for it, the restored composite produced by the deployed configuration, and a twofold magnification of the restored region. The solid outline traces the instrument mask recorded for that sample. The dotted outline marks that same boundary at identical image coordinates on the other three panels, where the region is unobstructed in the reference and the instrument no longer appears in the output.
Table 4: Masked-region fidelity on matched samples. Learned baselines use ground-truth masks. Rows with predicted masks show deployed TransGHOST. Bnd. denotes boundary MAE. LPIPS is reported for validation.
Unobstructed references
Occluded references
Selection rule
PSNR
Abst.
PSNR
Δ
SSIM
Abst.
Newest valid reference (recency)
33.42
0.000
26.44
-
0.854
0.154
Composite quality score
n/a
n/a
27.14
+0.70
0.862
0.134
Least region difference [ 28 ]
n/a
n/a
26.41
-0.03
0.857
0.145
Least mask overlap (ours)
33.37
0.000
30.04
+3.60
0.904
0.037
Table 5: Reference-selection rules on unobstructed and controlled-occlusion candidate sets. Abst. denotes the proportion of reconstruction pixels marked as abstentions.
Distribution
Candidates hiding
Partial-support
Masked PSNR
Abstention
the region
samples
recency
overlap
gain
recency
overlap
Light
16%
6
31.29
32.63
+1.34
0.043
0.009
Medium
41%
27
26.44
30.04
+3.60
0.154
0.037
Heavy
64%
53
23.36
27.76
+4.40
0.219
0.068
Table 6: Reference-selection gains under light, medium and heavy distributions of controlled occlusion.
Stage
PSNR ↑
Δ
SSIM ↑
MAE ↓
Bnd. MAE ↓
FP MAE ↓
Global affine registration only
28.25
-
0.861
6.76
3.83
6.63
+ residual translation
29.13
+ 0.89
0.883
5.81
3.26
5.50
+ sub-pixel refinement
29.26
+ 1.01
0.885
5.71
3.19
5.36
+ per-block local offsets
29.83
+ 1.58
0.893
5.01
2.69
4.34
Table 7: Held-out ablation on 101 samples from three videos excluded from training, using instrument cutouts from the test split.
Figure 9: Masked fidelity against reference age. The shaded band spans the classical reference-free controls of Table 3 , and the dashed series is the fixed-cohort control of Section 6.7 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Selection policy
PSNR ↑
Δ
SSIM ↑
MAE ↓
Nearest valid reference (ours)
33.42
-
0.933
2.93
Random valid reference
31.06
-2.36
0.910
4.64
Oldest valid reference
29.19
-4.23
0.888
6.37
Oracle on pre-alignment agreement
33.35
-0.07
0.931
3.02
Appendix
Table B1: Reference choice with alignment, masks and samples held fixed.
Variant
PSNR ↑
Δ
p95 (ms) ↓
p99 (ms) ↓
Multi-reference evaluation set, one reference per candidate lag
Single nearest reference (ours)
33.42
-
96
120
Median fusion of 5 references
33.39
−0.03
109
744
Per-block choice among 5 references
33.42
0.00
109
343
Base validation set, deployed configuration
Deployed configuration (ours)
31.89
-
99
118
Appendix
Table C1: Design-selection experiments comparing the reported configuration with measured alternatives.
Figure D1: End-to-end latency percentiles for three implementations of the alignment search on the validation set. The median changes by several milliseconds, whereas the 99th percentile decreases by approximately one quarter.
Configuration
PSNR ↑
p50 (ms) ↓
p95 (ms) ↓
p99 (ms) ↓
Initial measurement
Without per-block local offsets
29.25
57.6
87.2
100.5
Full configuration
29.83
61.0
102.7
116.9
Confirmatory measurement, sparse band sampling
Without per-block local offsets
29.26
52.1
83.1
90.0
Full configuration
29.83
55.9
86.1
101.2
Appendix
Table D1: Initial and confirmatory end-to-end latency on the held-out cohort.
All available samples
Fixed cohort
Lag
n
PSNR ↑
SSIM ↑
MAE ↓
Bnd. MAE ↓
PSNR ↑
Δ vs lag 1 [95% CI]
1
178
33.42
0.933
2.93
1.58
33.37
-
2
171
31.95
0.920
3.75
2.04
31.94
−1.43[−2.93,−1.21]
3
166
31.20
0.910
4.35
2.38
31.16
−2.22[−4.75,−1.95]
5
150
30.12
0.896
5.47
2.96
30.17
−3.20[−7.02,−3.04]
8
120
28.86
0.884
6.85
3.65
28.93
−4.45[−7.89,−3.96]
Appendix
Table E1: Reference-age analysis at 29.97 fps. The fixed-cohort block contains samples available at every lag. Bracketed values are 95 per cent intervals from a clip-clustered paired bootstrap against lag 1.
Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.
School of Medical Science and Technology, Indian Institute of Technology Kharagpur, Kharagpur 721302 India · Computer Science Department, Arizona State University, Tempe, Arizona 85287 USA · Narayana Hospital, Kolkata 700107 India