cs.CVMar 1, 2026

ConVibNet: Needle Detection during Continuous Insertion via Frequency-Inspired Features

Authors: Jiamei Guo, Zhehao Duan, Maria Neiiendam, Dianye Huang, Nassir Navab, Zhongliang Jiang

Organizations: Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich, Munich, Germany. · Technical University of Denmark, Kongens Lyngby, Denmark. · Munich Center for Machine Learning (MCML), Munich, Germany. · The University of Hong Kong, Hong Kong, China.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Purpose: Ultrasound-guided needle interventions are widely used in clinical practice, but their success critically depends on accurate needle placement, which is frequently hindered by the poor and intermittent visibility of needles in ultrasound images. Existing approaches remain limited by artifacts, occlusions, and low contrast, and often fail to support real-time continuous insertion. To overcome these challenges, this study introduces a robust real-time framework for continuous needle detection. Methods: We present ConVibNet, an extension of VibNet for detecting needles with significantly reduced visibility, addressing real-time, continuous needle tracking during insertion. ConVibNet leverages temporal dependencies across successive ultrasound frames to enable continuous estimation of both needle tip position and shaft angle in dynamic scenarios. To strengthen temporal awareness of needle-tip motion, we introduce a novel intersection-and-difference loss that explicitly leverages motion correlations across consecutive frames. In addition, we curated a dedicated dataset for model development and evaluation. Results: The performance of the proposed ConVibNet model was evaluated on our dataset, demonstrating superior accuracy compared to the baseline VibNet and UNet-LSTM models. Specifically, ConVibNet achieved a tip error of 2.80+-2.42 mm and an angle error of 1.69+-2.00 deg. These results represent a 0.75 mm improvement in tip localization accuracy over the best-performing baseline, while preserving real-time inference capability. Conclusion: ConVibNet advances real-time needle detection in ultrasound-guided interventions by integrating temporal correlation modeling with a novel intersection-and-difference loss, thereby improving accuracy and robustness and demonstrating high potential for integration into autonomous insertion systems.

Figures & tables

Figure 1: Frequency analysis of the US image sequence. (a) Selected pixel locations for analysis in the tt -th frame, with the yellow transparent line indicating the needle at an insertion angle of 30°. (b) ∼\sim (e) Spectrograms corresponding to the needle tip, shaft, surrounding tissue, and background.
Figure 2: Overview of the proposed method. (a) Architecture of ConVibNet. (b) Illustration of how the intersection and difference masks are computed.
Figure 3: Overview of the data acquisition workflow. (a) Hardware setup and connections (see Sec. 2.2.1 ). (b) US image sequence annotation process (see Sec. 2.2.2 ).
Insertion AngleTotalTraining SetValidation SetTest Set
15°50/1038040/86295/8255/926
30°56/2296544/181576/22986/2510
Values are presented in the format: num of videos/num of sequences .
Table 1: Overview of the Curated Dataset
MethodsTip Err. (mm)Angle Err. (°)Suc. Rate (%)
VibNet Huang et al. (2025) w/o DHT3.55±2.653.55\pm 2.651.59±2.131.59\pm 2.1363.763.7
UNet-LSTM Belikova et al. (2021)3.60±2.753.60\pm 2.751.59±1.78\mathbf{1.59\pm 1.78}62.762.7
ConVibNet † (ours)2.80±2.42\mathbf{2.80\pm 2.42}1.69±2.001.69\pm 2.0079.6\mathbf{79.6}
The angle error and tip error are described in the format of mean ±\pm std.
† Parameter settings for the proposed ConVibNet: α=0.5\alpha=0.5 and β=0.02\beta=0.02 .
Err.: error; Suc. success.
Table 2: Needle detection performance compared with baseline models
Figure 4: Example comparison of predicted segmentation and post-processed outputs
ConVibNetTip Err. (mm)Angle Err. (°)Suc. Rate (%)
Frameworksα\alphaβ\beta
w/o Linter\mathcal{L}_{\text{inter}} and Ldiff\mathcal{L}_{\text{diff}}0.000.000.000.003.55±2.653.55\pm 2.651.59±2.131.59\pm 2.1363.763.7
w/ Linter\mathcal{L}_{\text{inter}} , w/o Ldiff\mathcal{L}_{\text{diff}}0.250.250.000.003.66±2.853.66\pm 2.851.87±2.341.87\pm 2.3466.966.9
0.500.500.000.002.93±2.542.93\pm 2.541.78±2.241.78\pm 2.2481.4\mathbf{81.4}
0.750.750.000.003.49±2.903.49\pm 2.901.86±2.241.86\pm 2.2477.077.0
w/ Linter\mathcal{L}_{\text{inter}} and Ldiff\mathcal{L}_{\text{diff}}0.500.500.010.012.83±2.452.83\pm 2.451.76±2.121.76\pm 2.1280.580.5
Table 3: Ablation study on the intersection and difference loss
More in this topic

Ultrasound Imaging

Ultrasound imaging uses sound waves to visualize anatomy and function, with AI improving image reconstruction, diagnosis, measurement, and scan guidance.

See all 150 papers

Explore similar work

CardsList
Apr 22, 2026

A Vision-Language-Action Model for Adaptive Ultrasound-Guided Needle Insertion and Needle Tracking

Ultrasound (US)-guided needle insertion is a critical yet challenging procedure due to dynamic imaging conditions and difficulties in needle visualization. Many methods have been proposed for automated needle insertion, but they often rely on hand-crafted pipelines with modular controllers, whose performance degrades in challenging cases. In this paper, a Vision-Language-Action (VLA) model is proposed for adaptive and automated US-guided needle insertion and tracking on a robotic ultrasound (RUS) system. This framework provides a unified approach to needle tracking and needle insertion control, enabling real-time, dynamically adaptive adjustment of insertion based on the obtained needle position and environment awareness. To achieve real-time and end-to-end tracking, a Cross-Depth Fusion (CDF) tracking head is proposed, integrating shallow positional and deep semantic features from the large-scale vision backbone. To adapt the pretrained vision backbone for tracking tasks, a Tracking-Conditioning (TraCon) register is introduced for parameter-efficient feature conditioning. After needle tracking, an uncertainty-aware control policy and an asynchronous VLA pipeline are presented for adaptive needle insertion control, ensuring timely decision-making for improved safety and outcomes. Extensive experiments on both needle tracking and insertion show that our method consistently outperforms state-of-the-art trackers and manual operation, achieving higher tracking accuracy, improved insertion success rates, and reduced procedure time, highlighting promising directions for RUS-based intelligent intervention.

Yuelin Zhang, Qingpeng Ding, Longxiang Tang +2

Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong · Shenzhen International Graduate School, Tsinghua University, China +1

Robotic InsertionRobotic UltrasoundVision-Language-Action ModelsVLMs for Robotics
AskPDF
Sep 16, 2026

STUNet-Fusion: Spatiotemporal Needle-Tip Localization in Ultrasound Video via Multi-Channel Motion Fusion

Needle-tip localization in ultrasound remains challenging because the needle may appear weak, discontinuous, or partially invisible, while imaging artifacts and anatomical structures can produce similar responses. To address this problem, we propose STUNet-Fusion, a spatiotemporal framework for needle-tip localization in ultrasound videos. The proposed method formulates the input as a tri-channel spatio-temporal fusion tensor, comprising grayscale appearance, grid-based motion feature, and raw frame difference. A shared ResNet-34 encoder extracts spatial features, ConvLSTM integrates temporal dependencies, and a U-Net decoder reconstructs a dense probability heatmap. The final coordinates are extracted via a soft-argmax operation to achieve sub-pixel localization accuracy. Experimental results demonstrate that this spatiotemporal fusion strategy significantly improves localization robustness compared to conventional baselines.

Chia-Chi Hsu, Chia-Hsuan Hsu, Che-Chou Shen

National Taiwan University of Science and Technology, Taiwan · National Yang Ming Chiao Tung University, Taiwan

Medical Image LocalizationUltrasound ImagingFeature Fusion
AskPDF
Mar 24, 2026

PinPoint: Monocular Needle Pose Estimation for Robotic Suturing via Stein Variational Newton and Geometric Residuals

Reliable estimation of surgical needle 3D position and orientation is essential for autonomous robotic suturing, yet existing methods operate almost exclusively under stereoscopic vision. In monocular endoscopic settings, common in transendoscopic and intraluminal procedures, depth ambiguity and rotational symmetry render needle pose estimation inherently ill-posed, producing a multimodal distribution over feasible configurations, rather than a single, well-grounded estimate. We present PinPoint, a probabilistic variational inference framework that treats this ambiguity directly, maintaining a distribution of pose hypotheses rather than suppressing it. PinPoint combines monocular image observations with robot-grasp constraints through analytical geometric likelihoods with closed-form Jacobians. This framework enables efficient Gauss-Newton preconditioning in a Stein Variational Newton inference, where second-order particle transport deterministically moves particles toward high-probability regions while kernel-based repulsion preserves diversity in the multimodal structure. On real needle-tracking sequences, PinPoint reduces mean translational error by 80% (down to 1.00 mm) and rotational error by 78% (down to 13.80°) relative to a particle-filter baseline, with substantially better-calibrated uncertainty. On induced-rotation sequences, where monocular ambiguity is most severe, PinPoint maintains a bimodal posterior 84% of the time, almost three times the rate of the particle filter baseline, correctly preserving the alternative hypothesis rather than committing prematurely to one mode. Suturing experiments in ex vivo tissue demonstrate stable tracking through intermittent occlusion, with average errors during occlusion of 1.34 mm in translation and 19.18° in rotation, even when the needle is fully embedded.

Jesse F. d'Almeida, Tanner Watts, Susheela Sharma Stern +3

Department of Mechanical Engineering, Vanderbilt University, TN, USA. · Department of Electrical and Computer Engineering and Department of Computer Science, Vanderbilt University, TN, USA. · The Robotics Center and the Kahlert School of Computing, University of Utah, Salt Lake City, UT, USA.

Object Pose EstimationSurgical Instrument TrackingSurgical Robotics
AskPDF