STUNet-Fusion: Spatiotemporal Needle-Tip Localization in Ultrasound Video via Multi-Channel Motion Fusion

Chia-Chi Hsu; Chia-Hsuan Hsu; Che-Chou Shen

cs.CVSep 16, 2026

STUNet-Fusion: Spatiotemporal Needle-Tip Localization in Ultrasound Video via Multi-Channel Motion Fusion

Authors: Chia-Chi Hsu, Chia-Hsuan Hsu, Che-Chou Shen

Organizations: National Taiwan University of Science and Technology, Taiwan · National Yang Ming Chiao Tung University, Taiwan

Abstract

Needle-tip localization in ultrasound remains challenging because the needle may appear weak, discontinuous, or partially invisible, while imaging artifacts and anatomical structures can produce similar responses. To address this problem, we propose STUNet-Fusion, a spatiotemporal framework for needle-tip localization in ultrasound videos. The proposed method formulates the input as a tri-channel spatio-temporal fusion tensor, comprising grayscale appearance, grid-based motion feature, and raw frame difference. A shared ResNet-34 encoder extracts spatial features, ConvLSTM integrates temporal dependencies, and a U-Net decoder reconstructs a dense probability heatmap. The final coordinates are extracted via a soft-argmax operation to achieve sub-pixel localization accuracy. Experimental results demonstrate that this spatiotemporal fusion strategy significantly improves localization robustness compared to conventional baselines.

Figures & tables

Figure 1: Overview of STUNet-Fusion. Ten three-channel fused frames are processed independently by a shared ResNet-34 encoder. The resulting feature sequence is integrated by ConvLSTM and decoded into a needle-tip heatmap using final-frame encoder skip connections.

Panel of Figure 2 — Figure 2: Example of the three input channels used in STUNet-Fusion. From left to right: grayscale appearance $I_{t}$ , grid-based motion feature $G_{t}$ , and raw frame difference $D_{t}$ .

Method	MLE (mm) $\downarrow$	MedLE (mm) $\downarrow$	RMSE (mm) $\downarrow$	SDR@2 mm (%) $\uparrow$
U-Net [ 11 ]	44.50	8.92	68.62	13.4
Time-aware DNN [ 7 ]	41.17	3.52	58.14	26.7
STUNet-Fusion	0.86	0.81	0.99	98.2

Table 1: Best observed video-level performance of the baseline methods and the proposed STUNet-Fusion model.

Method	MLE (mm) $\downarrow$	MedLE (mm) $\downarrow$	RMSE (mm) $\downarrow$	SDR@2 mm (%) $\uparrow$
U-Net [ 11 ]	$57.00\pm 6.84$	$30.00\pm 36.92$	$79.20\pm 7.54$	$1.84\pm 3.97$
Time-aware DNN [ 7 ]	$55.38\pm 8.87$	$26.60\pm 38.99$	$79.15\pm 11.46$	$11.50\pm 8.03$
STUNet-Fusion	$\mathbf{5.26\pm 3.02}$	$\mathbf{1.25\pm 0.25}$	$\mathbf{19.99\pm 11.34}$	$\mathbf{76.38\pm 10.81}$

Table 2: Mean

\pm

standard deviation of the baseline comparison results.

Model	MLE (mm) $\downarrow$	MedLE (mm) $\downarrow$	RMSE (mm) $\downarrow$	SDR@2 mm (%) $\uparrow$
No Skip Connection	4.70	1.07	21.35	80.5
No ConvLSTM	61.54	27.92	87.84	33.5
Appearance Only	2.66	1.16	8.87	82.1
Full Model (STUNet-Fusion)	0.86	0.81	0.99	98.2

Table 3: Best video-level performance of each ablation variant.

Model	MLE (mm) $\downarrow$	MedLE (mm) $\downarrow$	RMSE (mm) $\downarrow$	SDR@2 mm (%) $\uparrow$
No Skip Connection	$19.95\pm 10.32$	$1.73\pm 0.50$	$47.68\pm 15.07$	$58.48\pm 16.48$
No ConvLSTM	$74.53\pm 9.45$	$60.26\pm 31.92$	$100.24\pm 8.44$	$24.90\pm 6.50$
Appearance Only	$28.37\pm 33.59$	$15.01\pm 33.72$	$46.53\pm 31.86$	$40.92\pm 26.19$
Full Model (STUNet-Fusion)	$\mathbf{5.26\pm 3.02}$	$\mathbf{1.25\pm 0.25}$	$\mathbf{19.99\pm 11.34}$	$\mathbf{76.38\pm 10.81}$

Table 4: Mean

\pm

standard deviation of the ablation results averaged over 15 test videos.

Panel of Figure 3 — Figure 3: Representative needle-tip localization results. Predicted heatmaps are overlaid on the ultrasound frames to indicate the localized tip positions.

Medical Image Localization

Medical image localization finds anatomical landmarks, lesions, organs, or devices in scans and videos to support diagnosis and image-guided care.

See all 101 papers

ConVibNet: Needle Detection during Continuous Insertion via Frequency-Inspired Features
Mar 1, 2026Jiamei Guo, Zhehao Duan, Maria Neiiendam +3Ultrasound Imaging
Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich, Munich, Germany. · Technical University of Denmark, Kongens Lyngby, Denmark. · Munich Center for Machine Learning (MCML), Munich, Germany. +1
Seeing Globally, Refining Locally: Global Visual Guidance and Local Ultrasound Cues for Robust Freehand 3-D Ultrasound Reconstruction
Jul 14, 2026Yameng Zhang, Zhongyu Chen, Dianye Huang +3Object Pose Estimation Ultrasound Imaging
Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR, China · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China
A Vision-Language-Action Model for Adaptive Ultrasound-Guided Needle Insertion and Needle Tracking
Apr 22, 2026Yuelin Zhang, Qingpeng Ding, Longxiang Tang +2Robotic Insertion Robotic Ultrasound
Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Hong Kong · Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong · Shenzhen International Graduate School, Tsinghua University, China +1

Mar 1, 2026J/K move · Enter open · S save

ConVibNet: Needle Detection during Continuous Insertion via Frequency-Inspired Features

Jiamei Guo, Zhehao Duan, Maria Neiiendam, Dianye Huang+2

Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich, Munich, Germany. · Technical University of Denmark, Kongens Lyngby, Denmark. · Munich Center for Machine Learning (MCML), Munich, Germany.+1

Ultrasound Imaging

Open paper AskPDF arXiv

Purpose: Ultrasound-guided needle interventions are widely used in clinical practice, but their success critically depends on accurate needle placement, which is frequently hindered by the poor and intermittent visibility of needles in ultrasound images. Existing approaches remain limited by artifacts, occlusions, and low contrast, and often fail to support real-time continuous insertion. To overcome these challenges, this study introduces a robust real-time framework for continuous needle detection. Methods: We present ConVibNet, an extension of VibNet for detecting needles with significantly reduced visibility, addressing real-time, continuous needle tracking during insertion. ConVibNet leverages temporal dependencies across successive ultrasound frames to enable continuous estimation of both needle tip position and shaft angle in dynamic scenarios. To strengthen temporal awareness of needle-tip motion, we introduce a novel intersection-and-difference loss that explicitly leverages motion correlations across consecutive frames. In addition, we curated a dedicated dataset for model development and evaluation. Results: The performance of the proposed ConVibNet model was evaluated on our dataset, demonstrating superior accuracy compared to the baseline VibNet and UNet-LSTM models. Specifically, ConVibNet achieved a tip error of 2.80+-2.42 mm and an angle error of 1.69+-2.00 deg. These results represent a 0.75 mm improvement in tip localization accuracy over the best-performing baseline, while preserving real-time inference capability. Conclusion: ConVibNet advances real-time needle detection in ultrasound-guided interventions by integrating temporal correlation modeling with a novel intersection-and-difference loss, thereby improving accuracy and robustness and demonstrating high potential for integration into autonomous insertion systems.

Abstract

Medical Image Localization

Explore similar work

ConVibNet: Needle Detection during Continuous Insertion via Frequency-Inspired Features

Seeing Globally, Refining Locally: Global Visual Guidance and Local Ultrasound Cues for Robust Freehand 3-D Ultrasound Reconstruction

A Vision-Language-Action Model for Adaptive Ultrasound-Guided Needle Insertion and Needle Tracking