ConVibNet: Needle Detection during Continuous Insertion via Frequency-Inspired Features

Jiamei Guo; Zhehao Duan; Maria Neiiendam; Dianye Huang; Nassir Navab; Zhongliang Jiang

cs.CVMar 1, 2026

ConVibNet: Needle Detection during Continuous Insertion via Frequency-Inspired Features

Authors: Jiamei Guo, Zhehao Duan, Maria Neiiendam, Dianye Huang, Nassir Navab, Zhongliang Jiang

Organizations: Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich, Munich, Germany. · Technical University of Denmark, Kongens Lyngby, Denmark. · Munich Center for Machine Learning (MCML), Munich, Germany. · The University of Hong Kong, Hong Kong, China.

Abstract

Purpose: Ultrasound-guided needle interventions are widely used in clinical practice, but their success critically depends on accurate needle placement, which is frequently hindered by the poor and intermittent visibility of needles in ultrasound images. Existing approaches remain limited by artifacts, occlusions, and low contrast, and often fail to support real-time continuous insertion. To overcome these challenges, this study introduces a robust real-time framework for continuous needle detection. Methods: We present ConVibNet, an extension of VibNet for detecting needles with significantly reduced visibility, addressing real-time, continuous needle tracking during insertion. ConVibNet leverages temporal dependencies across successive ultrasound frames to enable continuous estimation of both needle tip position and shaft angle in dynamic scenarios. To strengthen temporal awareness of needle-tip motion, we introduce a novel intersection-and-difference loss that explicitly leverages motion correlations across consecutive frames. In addition, we curated a dedicated dataset for model development and evaluation. Results: The performance of the proposed ConVibNet model was evaluated on our dataset, demonstrating superior accuracy compared to the baseline VibNet and UNet-LSTM models. Specifically, ConVibNet achieved a tip error of 2.80+-2.42 mm and an angle error of 1.69+-2.00 deg. These results represent a 0.75 mm improvement in tip localization accuracy over the best-performing baseline, while preserving real-time inference capability. Conclusion: ConVibNet advances real-time needle detection in ultrasound-guided interventions by integrating temporal correlation modeling with a novel intersection-and-difference loss, thereby improving accuracy and robustness and demonstrating high potential for integration into autonomous insertion systems.

Figures & tables

Figure 1: Frequency analysis of the US image sequence. (a) Selected pixel locations for analysis in the $t$ -th frame, with the yellow transparent line indicating the needle at an insertion angle of 30°. (b) $\sim$ (e) Spectrograms corresponding to the needle tip, shaft, surrounding tissue, and background.

Figure 2: Overview of the proposed method. (a) Architecture of ConVibNet. (b) Illustration of how the intersection and difference masks are computed.

Figure 3: Overview of the data acquisition workflow. (a) Hardware setup and connections (see Sec. 2.2.1 ). (b) US image sequence annotation process (see Sec. 2.2.2 ).

Insertion Angle	Total	Training Set	Validation Set	Test Set
15°	50/10380	40/8629	5/825	5/926
30°	56/22965	44/18157	6/2298	6/2510
Values are presented in the format: num of videos/num of sequences .

Table 1: Overview of the Curated Dataset

The angle error and tip error are described in the format of mean $\pm$ std.
Methods	Tip Err. (mm)	Angle Err. (°)	Suc. Rate (%)
VibNet Huang et al. (2025) w/o DHT	$3.55\pm 2.65$	$1.59\pm 2.13$	$63.7$
UNet-LSTM Belikova et al. (2021)	$3.60\pm 2.75$	$\mathbf{1.59\pm 1.78}$	$62.7$
ConVibNet † (ours)	$\mathbf{2.80\pm 2.42}$	$1.69\pm 2.00$	$\mathbf{79.6}$
† Parameter settings for the proposed ConVibNet: $\alpha=0.5$ and $\beta=0.02$ .
Err.: error; Suc. success.

Table 2: Needle detection performance compared with baseline models

Figure 4: Example comparison of predicted segmentation and post-processed outputs

ConVibNet			Tip Err. (mm)	Angle Err. (°)	Suc. Rate (%)
Frameworks	$\alpha$	$\beta$	Tip Err. (mm)	Angle Err. (°)	Suc. Rate (%)
w/o $\mathcal{L}_{\text{inter}}$ and $\mathcal{L}_{\text{diff}}$	$0.00$	$0.00$	$3.55\pm 2.65$	$1.59\pm 2.13$	$63.7$
w/ $\mathcal{L}_{\text{inter}}$ , w/o $\mathcal{L}_{\text{diff}}$	$0.25$	$0.00$	$3.66\pm 2.85$	$1.87\pm 2.34$	$66.9$
	$0.50$	$0.00$	$2.93\pm 2.54$	$1.78\pm 2.24$	$\mathbf{81.4}$
	$0.75$	$0.00$	$3.49\pm 2.90$	$1.86\pm 2.24$	$77.0$
w/ $\mathcal{L}_{\text{inter}}$ and $\mathcal{L}_{\text{diff}}$	$0.50$	$0.01$	$2.83\pm 2.45$	$1.76\pm 2.12$	$80.5$

Table 3: Ablation study on the intersection and difference loss

Ultrasound Imaging

Ultrasound imaging uses sound waves to visualize anatomy and function, with AI improving image reconstruction, diagnosis, measurement, and scan guidance.

See all 150 papers

A Vision-Language-Action Model for Adaptive Ultrasound-Guided Needle Insertion and Needle Tracking
Apr 22, 2026Yuelin Zhang, Qingpeng Ding, Longxiang Tang +2Robotic Insertion Robotic Ultrasound
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong · Shenzhen International Graduate School, Tsinghua University, China +1
STUNet-Fusion: Spatiotemporal Needle-Tip Localization in Ultrasound Video via Multi-Channel Motion Fusion
Sep 16, 2026Chia-Chi Hsu, Chia-Hsuan Hsu, Che-Chou ShenMedical Image Localization Ultrasound Imaging
National Taiwan University of Science and Technology, Taiwan · National Yang Ming Chiao Tung University, Taiwan
PinPoint: Monocular Needle Pose Estimation for Robotic Suturing via Stein Variational Newton and Geometric Residuals
Mar 24, 2026Jesse F. d'Almeida, Tanner Watts, Susheela Sharma Stern +3Object Pose Estimation Surgical Instrument Tracking
Department of Mechanical Engineering, Vanderbilt University, TN, USA. · Department of Electrical and Computer Engineering and Department of Computer Science, Vanderbilt University, TN, USA. · The Robotics Center and the Kahlert School of Computing, University of Utah, Salt Lake City, UT, USA.

Apr 22, 2026J/K move · Enter open · S save

A Vision-Language-Action Model for Adaptive Ultrasound-Guided Needle Insertion and Needle Tracking

Yuelin Zhang, Qingpeng Ding, Longxiang Tang, Chengyu Fang+1

Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong · Shenzhen International Graduate School, Tsinghua University, China+1

Robotic Insertion Robotic Ultrasound Vision-Language-Action Models VLMs for Robotics

Open paper AskPDF arXiv

Ultrasound (US)-guided needle insertion is a critical yet challenging procedure due to dynamic imaging conditions and difficulties in needle visualization. Many methods have been proposed for automated needle insertion, but they often rely on hand-crafted pipelines with modular controllers, whose performance degrades in challenging cases. In this paper, a Vision-Language-Action (VLA) model is proposed for adaptive and automated US-guided needle insertion and tracking on a robotic ultrasound (RUS) system. This framework provides a unified approach to needle tracking and needle insertion control, enabling real-time, dynamically adaptive adjustment of insertion based on the obtained needle position and environment awareness. To achieve real-time and end-to-end tracking, a Cross-Depth Fusion (CDF) tracking head is proposed, integrating shallow positional and deep semantic features from the large-scale vision backbone. To adapt the pretrained vision backbone for tracking tasks, a Tracking-Conditioning (TraCon) register is introduced for parameter-efficient feature conditioning. After needle tracking, an uncertainty-aware control policy and an asynchronous VLA pipeline are presented for adaptive needle insertion control, ensuring timely decision-making for improved safety and outcomes. Extensive experiments on both needle tracking and insertion show that our method consistently outperforms state-of-the-art trackers and manual operation, achieving higher tracking accuracy, improved insertion success rates, and reduced procedure time, highlighting promising directions for RUS-based intelligent intervention.

Abstract

Ultrasound Imaging

Explore similar work

A Vision-Language-Action Model for Adaptive Ultrasound-Guided Needle Insertion and Needle Tracking

STUNet-Fusion: Spatiotemporal Needle-Tip Localization in Ultrasound Video via Multi-Channel Motion Fusion

PinPoint: Monocular Needle Pose Estimation for Robotic Suturing via Stein Variational Newton and Geometric Residuals