PinPoint: Monocular Needle Pose Estimation for Robotic Suturing via Stein Variational Newton and Geometric Residuals

Jesse F. d'Almeida; Tanner Watts; Susheela Sharma Stern; James Ferguson; Alan Kuntz; Robert J. Webster

cs.ROMar 24, 2026

PinPoint: Monocular Needle Pose Estimation for Robotic Suturing via Stein Variational Newton and Geometric Residuals

Authors: Jesse F. d'Almeida, Tanner Watts, Susheela Sharma Stern, James Ferguson, Alan Kuntz, Robert J. Webster

Organizations: Department of Mechanical Engineering, Vanderbilt University, TN, USA. · Department of Electrical and Computer Engineering and Department of Computer Science, Vanderbilt University, TN, USA. · The Robotics Center and the Kahlert School of Computing, University of Utah, Salt Lake City, UT, USA.

Abstract

Reliable estimation of surgical needle 3D position and orientation is essential for autonomous robotic suturing, yet existing methods operate almost exclusively under stereoscopic vision. In monocular endoscopic settings, common in transendoscopic and intraluminal procedures, depth ambiguity and rotational symmetry render needle pose estimation inherently ill-posed, producing a multimodal distribution over feasible configurations, rather than a single, well-grounded estimate. We present PinPoint, a probabilistic variational inference framework that treats this ambiguity directly, maintaining a distribution of pose hypotheses rather than suppressing it. PinPoint combines monocular image observations with robot-grasp constraints through analytical geometric likelihoods with closed-form Jacobians. This framework enables efficient Gauss-Newton preconditioning in a Stein Variational Newton inference, where second-order particle transport deterministically moves particles toward high-probability regions while kernel-based repulsion preserves diversity in the multimodal structure. On real needle-tracking sequences, PinPoint reduces mean translational error by 80% (down to 1.00 mm) and rotational error by 78% (down to 13.80°) relative to a particle-filter baseline, with substantially better-calibrated uncertainty. On induced-rotation sequences, where monocular ambiguity is most severe, PinPoint maintains a bimodal posterior 84% of the time, almost three times the rate of the particle filter baseline, correctly preserving the alternative hypothesis rather than committing prematurely to one mode. Suturing experiments in ex vivo tissue demonstrate stable tracking through intermittent occlusion, with average errors during occlusion of 1.34 mm in translation and 19.18° in rotation, even when the needle is fully embedded.

Figures & tables

Fig. 1: Conceptual illustration of the proposed monocular needle tracking framework. (Top) Image and robot-based observations define an error over the needle pose estimate, which is iteratively solved for with PinPoint. The isometric robot view highlights the inherent depth ambiguity of monocular needle pose estimation, where multiple 3D configurations can produce similar image projections. (Bottom) PinPoint uses Stein Variational Inference, which represents the posterior over needle poses using a set of interacting particles. Particles are deterministically transported toward high-probability regions of the posterior while repulsive interactions prevent collapse and preserve multimodal hypotheses. The maximum a posteriori (MAP) particle corresponds to the most likely needle pose estimate.

Fig. 2: Sources of information used for monocular needle pose estimation and the geometric reach of their corresponding residuals: (i) image keypoints at the needle tip and tail, (ii) needle backbone segmentation, (iii) the C-Lock rigidly fixes the needle at a known grasp point, and (iv) the C-Lock enforces a perpendicular relationship between the gripper axis and the needle tangent at the grasp location.

Fig. 3: The needle is rigidly fixed to the checkerboard such that the transform between the centers can be measured, enabling ground truth needle pose measurements, for calibration and accuracy tests.

Fig. 4: Distribution of translational and rotational pose estimation errors for PF and PinPoint across motion speeds. Each panel shows a half-violin with overlaid scatter for PF (pink, right) and PinPoint (blue, left), with white circles indicating the median. Stacked bars below each panel show the fractional contribution of each component axis (X: green, Y: orange, Z: purple) to total error. In both (Left) slow-speed trials and (Right) normal-speed trials, we show the rotational and translational errors.

Likelihood Type	Std. Dev.
Anchor (x, y, z)	$(0.41,\,0.44,\,1.00)$ mm
Perpendicular	$0.138$
Conic	$1.235$ px
Keypoints (tip $u,v$ ; tail $u,v$ )	$(1.006,\;1.012;\;0.921,\;0.680)$ px

TABLE I: Empirical standard deviations of observation likelihoods computed from calibration data.

Method	Speed	Metric	X	Y	Z	Total	Unc (p95)	NLL-S	IS
PF	Slow	Trans (mm)	$2.81\pm 2.63$	$2.05\pm 1.86$	$2.31\pm 1.76$	$4.45\pm 3.33$	$8.42$	$6.76$	$6.62$
		Rot (deg)	$12.88\pm 9.77$	$19.51\pm 12.00$	$57.35\pm 52.44$	$65.99\pm 44.74$	$119.91$	$79.10$	$33.98$
	Normal	Trans (mm)	$2.02\pm 1.88$	$2.89\pm 1.92$	$2.71\pm 2.99$	$5.04\pm 3.26$	$9.70$	$6.87$	$8.40$
		Rot (deg)	$12.93\pm 9.75$	$18.87\pm 14.15$	$57.78\pm 42.27$	$63.27\pm 38.54$	$112.56$	$22.00$	$30.94$
PinPoint	Slow	Trans (mm)	$0.14\pm 0.13$	$0.10\pm 0.09$	$0.55\pm 0.40$	$\mathbf{0.60\pm 0.39}$	$\mathbf{1.20}$	$\mathbf{1.86}$	2.426
		Rot (deg)	$8.62\pm 6.92$	$12.09\pm 8.19$	$3.32\pm 1.97$	$16.92\pm 7.74$	$30.80$	4.82	24.15

TABLE II: Comparison of PF and SVN tracking performance under slow and normal needle motion. Component-wise mean

\pm

standard deviation and total error are reported for translation and rotation, along with 95th-percentile uncertainty (Unc), negative log-likelihood score (NLL-S), and interval score (IS). Lower is better for all metrics.

Fig. 5: A bar graph breakdown of the error degradation about the translational and rotational under the given ablation.

Ablation	Translation (mm)				Rotation (deg)
	Mean $\pm$ Std	Unc	NLL-S	IS	Mean $\pm$ Std	Unc	NLL-S	IS
All (Full Model)	0.75 $\pm$ 0.37	1.37	1.40	3.03	9.87 $\pm$ 4.60	18.29	4.82	24.15
No Dense Image	0.81 $\pm$ 0.48	1.80	1.49	4.36	12.83 $\pm$ 8.95	31.55	5.43	40.06
1 Sparse Keypoint	1.66 $\pm$ 1.80	3.94	3.13	4.98	16.10 $\pm$ 13.40	47.49	22.13	45.99
No Sparse Keypoints	1.96 $\pm$ 1.72	4.30	4.57	5.87	28.73 $\pm$ 21.80	60.52	52.02	58.49
No Image	4.53 $\pm$ 2.41	10.14	9.51	5.98	31.55 $\pm$ 43.06	143.99	11.49	59.53

TABLE III: Ablation study evaluating the contribution of individual likelihood terms to needle pose estimation accuracy using the SVN method. Mean

\pm

std error is reported along with 95% uncertainty coverage (Unc), negative log-likelihood score (NLL-S), and interval score (IS). Lower is better for all metrics.

Fig. 6: PinPoint correctly resolves pose ambiguity that confounds the particle filter. (Left) A suturing needle undergoing rotation presents a visually symmetric appearance, inducing a bimodal posterior over needle pose. PinPoint and PF estimates are shown overlaid on the endoscopic image. (Right, Top Row) PinPoint represents the posterior as a two-component GMM with well-separated modes in PCA space; the estimate $T_{est}$ is located within the same mode as the ground truth $T_{gt}$ , while correctly maintaining the alternative hypothesis. (Right, Bottom Row) The PF fits a two-component mixture but fails to separate the hypotheses, as both components heavily overlap in PCA space, producing a diffuse, ambiguous distribution.

Fig. 7: Graph of translation (blue) and rotation (orange) errors during suturing, with representative images shown at key frames. A graph of the occlusion level is graphed below, with icons illustrating the levels of none, partial, and heavy occlusion. The graph illustrates how occlusion increases during piercing until the tip emerges on the other side of the tissue. Occlusion then decreases as the needle is retracted until it fully exits the tissue.

Object Pose Estimation

Object pose estimation recovers an object's 3D position and orientation from images, depth, point clouds, or contact sensing.

See all 184 papers

PICO: Projection-Informed Consistency Optimisation for 6DoF Surgical Tool Pose Estimation
Sep 25, 2026Lucy Fothergill, Pietro Valdastri, Dominic Jones +1Object Pose Estimation Surgical Robotics
School of Computer Science, University of Leeds. · School of Electronic and Electrical Engineering, University of Leeds.
SutureFormer: Learning Surgical Trajectories via Goal-conditioned Offline RL in Pixel Space
Mar 19, 2026Huanrong Liu, Chunlin Tian, Tongyu Jia +8Offline RL Trajectory Prediction
Faculty of Science and Technology, University of Macau, Macau, China · University of Macau Advanced Research Institute in Hengqin, Zhuhai, China · Senior Department of Urology, The Chinese PLA General Hospital, Beijing, China +4
Reeling It In: Flexible Needle Pick Up via Thread Manipulation for Autonomous Suturing
Jul 28, 2026Emma Huang, Zih-Yun Chiu, Neelay Joglekar +2Robotic Grasping Robotic Manipulation
Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, CA 92093 USA. · Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218 USA. · CMU Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213 USA. +1

Sep 25, 2026J/K move · Enter open · S save

PICO: Projection-Informed Consistency Optimisation for 6DoF Surgical Tool Pose Estimation

Lucy Fothergill, Pietro Valdastri, Dominic Jones, Duygu Sarikaya

School of Computer Science, University of Leeds. · School of Electronic and Electrical Engineering, University of Leeds.

Object Pose Estimation Surgical Robotics Surgical Video Understanding

Open paper AskPDF arXiv

Purpose: Accurate 6 DoF pose estimation of surgical tools is critical for automa- tion, robotic proprioception, and safe interaction with the tissue operated on. Kinematics-based approaches suffer from accumulated errors due to the cable- driven nature of robotic arms, while vision-based methods often rely on external markers or trackers. Although more recent vision-based advances have been pro- posed, these two-stage pose estimation methods often lack real-time robustness due to accumulated errors and computational overhead. Methods: We propose a novel end-to-end trainable model, PICO. Our model employs a multi-task learning architecture to predict segmentation and depth maps, alongside regression of translation and rotation parameters. We define two proxy tasks that enforce geometric consistency in both 2D and 3D spaces, improving accuracy and robustness. For this, we propose a projection loss, and a point-to-point loss. Results: We evaluate our method on the SurgRIPE dataset, benchmarking its performance against state-of-the-art approaches using standard 6DoF pose esti- mation metrics. Our results demonstrate consistently strong performance across all four subsets, specifically in rotation, ranking second even under occlusion. It also demonstrates comparable translational performance, remaining competitive, especially in occluded cases. Conclusion: PICO demonstrates the effectiveness of multi-task learning and geometry-aware proxy tasks for robust and reliable surgical tool pose estimation, especially in occluded scenarios, highlighting potential for future applications.

Abstract

Object Pose Estimation

Explore similar work

PICO: Projection-Informed Consistency Optimisation for 6DoF Surgical Tool Pose Estimation

SutureFormer: Learning Surgical Trajectories via Goal-conditioned Offline RL in Pixel Space

Reeling It In: Flexible Needle Pick Up via Thread Manipulation for Autonomous Suturing