Toward Trustworthy Robot-Assisted Sliding Palpation for Shallow Vessel Localisation with a Calibrated Digital Twin

Piotr Blaszyk; Wen Fan; Kaizhong Deng; Daniel Elson; Dandan Zhang

cs.ROAug 29, 2026

Toward Trustworthy Robot-Assisted Sliding Palpation for Shallow Vessel Localisation with a Calibrated Digital Twin

Authors: Piotr Blaszyk, Wen Fan, Kaizhong Deng, Daniel Elson, Dandan Zhang

Organizations: Imperial College London, UK

Abstract

Reliable localisation of shallow subsurface vessels is important for safe robot-assisted venous access and vessel-aware manipulation, but collecting diverse tactile data on physical hardware is costly, time-consuming, and can degrade soft vision-based tactile sensors. We present a robot-assisted sliding-palpation framework in which a calibrated digital twin generates labelled tactile sequences, reducing reliance on real-world data. The twin models sensor-vessel contact, is calibrated against real palpation trajectories using Bayesian-optimisation-based domain adaptation, and is randomised over sliding direction and contact conditions. A spatio-temporal graph neural network trained on simulated marker trajectories performs per-node vessel classification and produces a human-verifiable top-view localisation map through 2D-to-3D-to-2D geometric projection. We evaluate three datasets: Sim, Silicone, and Meat, the latter a raw-meat phantom with vessel models at nominal depths of 0 to 30 mm, using four train-to-test configurations: Sim to Sim, Sim to Silicone, Sim to Meat, and Meat to Silicone. The calibrated twin achieves a simulated-to-real marker-alignment mean absolute error of 0.50 mm at deepest contact across four canonical interactions. After reprojection onto a 1 mm top-view grid, predicted vessel pixels lie on average 1.05 to 5.49 mm from the nearest true vessel pixel across the four models, with 1.05 to 1.31 mm for all except Sim to Meat. The larger error for Sim to Meat reflects the greater domain shift and current limit of simulation transfer. These results demonstrate progress toward trustworthy tactile palpation through calibrated simulation, interpretable localisation, and transparent cross-domain evaluation. Code, model weights, and data are publicly available on GitHub and Zenodo.

Figures & tables

Figure 1: The working principle of the ViTacTip sensor used in this work. This figure was inspired by the work of Fan et al. [ 12 ] .

Figure 2: Vessel-detection pipeline. Domain adaptation calibrates the sensor Young’s modulus and sensor–vessel contact stiffness, while domain randomisation varies the sliding direction and sensor–vessel contact parameters. Training data are generated using the DiffTactile-based digital twin, and an ST-GNN performs per-node binary classification on the marker graph (magenta: vessel; cyan: no vessel). Robot kinematics are then used to project these predictions into a top-view vessel map (white: predicted vessel). Real ViTacTip trajectories collected on the silicone vascular phantom (Real Scenario #1) and the raw-meat phantom (Real Scenario #2) provide labelled tactile videos for evaluating the ST-GNN.

Project	Target	Sensor	Palpation	Approach	BP	Train	Test
MiniTac [ 25 ]	T	VBTS	T	M	n/a	R	P, EV
DIGIT Pinki [ 8 ]	T	VBTS	T	M	n/a	R	P, EV
Bewley et al. [ 3 ]	T	VBTS	T	EE	n/a	R	P
Beasley et al. [ 2 ]	BV	Capacitive	S	EE	Y	n/a	P, IV
Hampson et al. [ 18 ]	BV	Tactile array	T	EE	Y	n/a	P
3D CNN [ 7 ]	BV	NIR	none	EE	N	R	IV

Table 1: Comparison of related work: part 1

Project	Model	Accuracy [%]	Error [mm]
MiniTac [ 25 ]	MLP, SVM	100	n/a
DIGIT Pinki [ 8 ]	RN18, VMAE	98, 100	n/a
Bewley et al. [ 3 ]	LR, GKDE	95, 70	1
Beasley et al. [ 2 ]	MMD, THR, LR	n/a	1.3
Hampson et al. [ 18 ]	MM, GC	n/a	0.2, 0.6, 6.0
3D CNN [ 7 ]	3D CNN	87	n/a

Table 2: Comparison of related work: part 2

$L$	FG IoU $\uparrow$	AP $\uparrow$
1	$0.114\pm 0.002$	$0.209\pm 0.005$
3	$0.197\pm 0.006$	$0.201\pm 0.001$
5	$\mathbf{0.238\pm 0.001}$	$\mathbf{0.324\pm 0.001}$
7	$0.214\pm 0.029$	$0.311\pm 0.006$

Table 3: Temporal-window ablation: Sim

\rightarrow

Silicone trained with a clip of

L

frames. Arrows give the desirable direction; bold marks the best value per column and underline the runner-up.

L=5

is used throughout the rest of this manuscript

Panel of Figure 3 — Figure 3: (a) ST-GNN architecture. Raw node features (marker, global) and edge features (spatial, temporal, global-spatial, global-temporal) are projected to a shared embedding space and processed by 3 GINE (graph isomorphism network with edge support) convolution layers, each in a residual unit with dropout and batch normalisation, giving a 3-hop receptive field. The input and all 3 layer representations are concatenated by skip connections (late fusion) and an MLP head outputs one classification value per marker node. (b) Graph entity types and raw input features. Each frame has marker nodes and one global node; within a frame markers connect via spatial edges and to the global node via global-spatial edges; across adjacent frames each marker connects to itself via temporal edges and the global nodes via global-temporal edges.

Panel of Figure 4 — Figure 4: Alignment between simulated (red) and real (green) marker positions after domain adaptation for four canonical interactions, each shown at the point of deepest contact: (a) press, (b) twist about the $z$ -axis, (c) twist about the $x$ -axis, and (d) slide.

Panel of Figure 5 — Figure 5: (a),(b) Example Silicone video frame with 2 vessels, (a) without and (b) with annotations: large red/green circles are manually annotated points and the small-circle line is the vessel centreline from a manually specified orientation. (c) Meat ground-truth labels (red: vessel present, green: absent) from robot kinematics; note the minor misalignment with the actual metal straw.

Panel of Figure 6 — Figure 6: (a)–(d) Precision–recall curves on the test sets, over five training seeds per model: the thick line is the mean curve and the shaded band is $\pm 1$ standard deviation across seeds.

Panel of Figure 7 — Figure 7: Vessel localisation maps (2D $\rightarrow$ 3D $\rightarrow$ 2D geometric projection; 1 $\times$ 1 mm per pixel; green: TP, blue: FP, red: FN, black: TN). (a) Sim $\rightarrow$ Sim, one simulated slide across the single vessel; the sensor slides horizontally across the vertical vessel. (b) Sim $\rightarrow$ Meat on the trial with two metal straws beneath two medallions; the sensor slides horizontally across the vertical straws. (c) Sim $\rightarrow$ Silicone and (d) Meat $\rightarrow$ Silicone on the silicone phantom, which has four vessel inclusions: five vertical slides are performed, side by side, across the roughly horizontal vessels, against the video-annotation ground truth. Each model is thresholded at its own operating point (Sec. 5.3 ).

Model	TP $\uparrow$	FP $\downarrow$	FN $\downarrow$	TN $\uparrow$	MCC $\uparrow$	F1 $\uparrow$	Prec. $\uparrow$	Rec. $\uparrow$	FG IoU $\uparrow$	BG IoU $\uparrow$	AP $\uparrow$	$\bar{d}$ $\downarrow$
Video-frame space (per marker)
Sim $\rightarrow$ Sim	3569	15169	38	66949	0.39	0.32	0.19	0.99	0.19	0.81	0.50	n/a
Sim $\rightarrow$ Silicone	909	2070	818	11443	0.30	0.39	0.31	0.53	0.24	0.80	0.33	n/a
Sim $\rightarrow$ Meat	1493	7134	317	18869	0.29	0.29	0.17	0.82	0.17	0.72	0.23	n/a
Meat $\rightarrow$ Silicone	1600	8448	127	5065	0.20	0.27	0.16	0.93	0.16	0.37	0.35	n/a
Top-view map space (per pixel)

Table 4: Segmentation statistics of the best-of-five instance of each model, pooled once over all its test data: per marker in video-frame space (threshold 0.5) and per pixel on the top-view map (chosen operating point). Arrows give the desirable direction; per column and within each space, bold marks the best value and underline the runner-up (counts are not highlighted, as the test sets differ in size)

Explore similar work

Aug 18, 2026cs.RO

Transferable Tool-Tissue Contact Detection from Stereo Depth in Robot-Assisted Surgery

Reliable tool--tissue contact detection can support interaction-aware control and downstream force estimation in robot-assisted surgery. Most existing methods learn a contact classifier from RGB appearance, which is hard to generalize. In this work, we use the depth image generated from a stereo pair to give more information about tool--tissue contact. For each depth frame, we localize a spatially supported minimum-distance patch around the tool boundary and reduce it to a single scalar,

-\log_{10}|d|

; this signal rises and falls in step with ground-truth contact. We formalize this observation with a fully supervised two-state hidden Markov model. We fit this model as a six-fold leave-one-session-out (LOSO) ensemble on six palpation sessions against a single silicone cup-like phantom, with the decision threshold selected from the pooled out-of-fold predictions. It is evaluated on four held-out sessions of three categories: 1. same task on same phantom; 2. same task on different phantom; 3. different task on different phantom. This model reaches held-out macro F1

0.927

and AUPRC

0.980

. We further compare against a reproduction of an RGB-based contact classifier from prior work. This RGB-based model achieves high performance on the first category (F1

0.965

), but substantially lower performance on the other two, resulting in macro F1

0.320

across all four sessions. These results indicate that the tool--tissue distance is a strong, transferable cue for contact detection in robot-assisted surgery.

Mingyeung Wu, Zhonghao Zhang, Hao Yang +2

Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, United States · Department of Computer Science, Vanderbilt University, Nashville, TN, United States

Surgical Robotics

Ask PDF

Oct 5, 2026cs.RO

Demo: Vision-Language Model-Guided Online Calibration of an Electromagnetic Digital Twin

An electromagnetic (EM) digital twin gives mobile robots wireless situational awareness but depends on material conductivities that change with the environment. Online calibration faces initialization sensitivity and measurement travel costs. We demonstrate a vision-language model (VLM)-guided framework using a Unitree G1 robot and NVIDIA Sionna, with two VLM calls: material classification maps visible materials through ITU-R P.2040 to conductivity priors for Sionna's gradient descent on accumulated received signal strength (RSS) measurements; waypoint planning selects the next measurement location online using residual RSS calibration error and image coverage. In a real indoor scenario, the framework achieves a normalized mean absolute conductivity error of

1.74\times10^{-4}

within 20 m of travel; random initialization never converges, while random waypoints require over twice the travel.

Zerui Kang, Yishen Lim, Zhouyou Gu +4

Singapore University of Technology and Design Singapore, Singapore · Inha University Incheon, Republic of Korea

Wireless Communications Digital Twins

Ask PDF

Sep 18, 2026cs.RO

Tracker-Free Robotic Ultrasound Calibration with a Spherical-Marker Phantom and Threshold-Free Center Localization

Robotic ultrasound (US) calibration is essential for accurately relating US images to the robot coordinate system, but accurate and automated calibration remains challenging because existing methods often require complex phantoms or external 3D trackers. In this work, we develop a tracker-free robotic US calibration framework using a spherical-marker phantom and a highly automated perception pipeline for sphere-center localization. The proposed threshold-free image-processing method localizes the spherical feature based on each US image's intensity distribution, eliminating hand-tuned intensity thresholds and improving robustness across imaging settings without per-system retuning. Multi-pose observations of the spherical fiducial are then used to estimate the US-to-EE transformation without external tracking or prior localization of the sphere center in the robot base frame. We validate the proposed framework on a robotic US platform through repeated sphere scans and further assess the calibrated system using geometrically distinct phantoms with known CAD models. Across 12 cross-validation folds, single-marker calibration achieved sphere-center accuracy and precision of

1.75\pm0.51

mm and

0.85\pm0.11

mm, respectively, compared with

1.72\pm0.51

mm and

0.84\pm0.12

mm for three-marker calibration. Across the three reconstruction sets, the single-marker calibration yielded pooled post-registration point-to-surface MAE

\pm

SD values of

0.41\pm0.35

mm for the cone and

0.50\pm0.41

mm for the triangular prism, closely matching the three-marker results of

0.40\pm0.35

mm and

0.48\pm0.40

mm, respectively.

Kyoungmo Koo, Guangshen Ma, Xueding Wang +1

Department of Robotics, University of Michigan, 2505 Hayward St, Ann Arbor, MI 48109, USA · Department of Biomedical Engineering, University of Michigan, 1150 Medical Center Dr, Ann Arbor, MI 48109, USA · Department of Ophthalmology and Visual Sciences, University of Michigan Medical School, 1000 Wall St, Ann Arbor, MI 48105, USA

Robotic Ultrasound Hand-Eye Calibration Medical Image Localization

Ask PDF