Incision trajectory tracing for electrosurgical navigation by CNN-based knife contacting frames extraction method
Authors: Yu Chun Wang, Kaixu Chen, Naoto Ienaga, Yoshihiro Kuroda
Organizations: Degree Programs in Systems and Information Engineering, University of Tsukuba, Tsukuba, 305-8573, Ibaraki, Japan · Center for Computational Sciences, University of Tsukuba, Tsukuba, 305-8577, Ibaraki, Japan · Institute of Systems and Information Engineering, University of Tsukuba, Tsukuba, 305-8573, Ibaraki, Japan
Background and Objective: Image-guided surgical navigation has been actively studied because of its advantage of identifying subsurface targets and critical structures, whereas it requires incision trajectories to update the preoperative three-dimensional model dynamically during the surgery. The novelty of this study is the thermal feature distinguishment of whether the electric tools contacting the tissue by Convolutional Neural Network (CNN), and the extraction of the knife contacting frames, to form incision trajectories which can meet with the requirement during the surgery. Methods: This study firstly verified that CNN can classify the thermal images of electric knife and ultrasonic cutter operations separately, and can raise the accuracy of the incision trajectories derived from the connection of the thermal intensity centroid of the frames predicted by CNN as contacting. Results: Our results obtained by employing the electric knife not only reveal a remarkably high accuracy 97.2 % in CNNs identification, but also can achieve an error reduction as high as more than 2.5 times of the incision trajectory prediction as compared to those proceeded in the conventional method. Besides electric knife, the results obtained by employing another electric tool, ultrasonic cutter, reveal a high accuracy up to 93.7 %. Conclusion: In this study, we ensured the possibility of CNN in distinguishing electric tools contacting with the tissue, and confirmed that the proposed method has not only overcome the problem of missing trajectories which usually occurs in the convolutional long-short term memory method but also achieved a remarkable improvement of the accuracy with less limitation.
Figures & tables
Figure 1: Cavitron Ultrasonic Surgical Aspirator (CUSA) left thermal diffusion on the tissue acquired by thermal camera. The temperature on the graph is measured in degrees Celsius.
Figure 2: Proposed Method: 1. After input the thermal images, each frame was classified to whether the electric cutting device was “contacting" or “not contacting" the tissue by the CNN. 2.The frames predicted as “contacting" the tissue were extracted. 3.The extracted frames were processed by Fast Fourier Transform (FFT) to enhance the contact area apart from the other part of the electric cutting device. 4. The hottest spots, the tip of the electric cutting device, on each of the extracted frame acquired out by an intensity centroid method. 5. The hottest spots from each frame were connected to form and output as a trajectory.
Figure 3: examples of FFT post-processing: (a) The example of electric knife’s frame labeled as “contacting" before FFT post-processing. (b) The example of the electric knife’s frame after FFT post processing. (c) The example of ultrasonic cutter’s frame labeled as “contacting" before FFT post-processing. (d) The example of the ultrasonic cutter’s frame after FFT post processing.
Figure 4: Examples of intensity centroid method: The white pixels with the green boundary shows the hottest pixels of the thermal image, and the pixel with the red boundary shows the hottest point calculate by the intensity centroid method by setting the threshold at the highest temperature at the frame.
Labeled as contacting
Labeled as not contacting
Frames
2,141 (36.1%)
3,791 (63.9%)
Table 1: Electric Knife’s trajectories dataset
Figure 5: Examples of frame labeled by employing the electric knife: (Top row) The examples of frames labeled as “contacting" and (Bottom row) the examples of frames labeled as “not contacting.
Labeled as contacting
Labeled as not contacting
Frames
2,967 (48.3%)
3,177 (51.7%)
Table 2: Ultrasonic cutter’s trajectories dataset
Figure 6: The hottest point of the ultrasonic cutter is not at the tip of the knife
Figure 7: Examples of frame labeled by employing ultrasonic cutter: (Top row)The examples of frames labeled as “contacting" and (Bottom row) the examples of frames labeled as “not contacting.
Figure 8: The example of DTW. The red pixels represent the ground truth trajectory, the blue pixels represent the prediction trajectory the orange circles represent the overlap of the pre-determined trajectory in red pixels and the prediction trajectory in blue pixels, and the green lines are the minimum distances between each blue and red point.
Figure 9: The confusion matrix of the CNN prediction result by employing electric knife, where the top-left number represents the true positives (TP), with the value in parentheses indicating the true positive ratio. The bottom-left number shows the false positives (FP). The top-right number displays the false negatives (FN). Finally, the bottom-right number denotes the true negatives (TN).
Figure 10: Output from gradient-based attribution by employing electric knife: An example of (Left) the input thermal image, (Right) the output from gradient-based attribution shows the model focusing on the feature at the area of the tip of electric knife and the diffusion area. The density of 0 to 1 represent the positive relevance of the area.
Prediciton Method
Ave DTW [px]
ConvLSTM
1.46±0.82
Intensity Centroid
2.19±1.34
Proposed w/o FFT
0.55±0.13
Proposed w/ FFT
0.75±0.22
Table 3: Electric knife’s Incision Trajectory’s DTW. Values are mean ± std.
Figure 11: Examples of output by employing electric knife: Examples of output from (a) Ground truth. (b) The output obtained from ConvLSTM method. The red circles point out the missing trajectories compared to ground truth. (c)The output of intensity centroid of the electric knife without CNN applied. The blue circles point out the false detection trajectories compared to ground truth. (d) The proposed method without the step of FFT post-processing. (e) The proposed method with the step of FFT post-processing
Figure 12: The confusion matrix of the CNN prediction result by employing ultrasonic cutter
Prediciton Method
Ave DTW [px]
Intensity Centroid
16.29±4.19
Proposed w/o FFT
5.95±3.67
Proposed w/ FFT
1.7±0.76
Table 4: Ultrasonic cutter’s Incision Trajectory’s DTW, Values are mean ± std.
Figure 13: Output from gradient-based attribution by employing ultrasonic cutter: An example of (Left) the input thermal image, (Right) the output from gradient-based attribution shows the model focusing on the feature at the area of the tip of ultrasonic cutter and the diffusion area. The density of 0 to 1 represent the positive relevance of the area.
Figure 14: Examples of output by employing electric knife: (a) The ground truth trajectories (b)The output of intensity centroid of the ultrasonic cutter without CNN applied. The blue circles point out the false detection trajectories compared to ground truth. (c)The proposed method’s prediction trajectories without FFT post-processing. The red circles point out the false connecting point due to the higher temperature on the ultrasonic cutter body. (d)The proposed method’s prediction trajectories with FFT post-processing
Figure 15: Examples of the false prediction images by employing electric knife: First category, the prediction was one frame earlier or later than the labeled (a)contact moment and(b) the disengagement moment. Second category, (c)The electric knife’s position at the edge of the image, which was predicted as “no contacting”. Third category, (d) The electric knife was out of focus, which was predicted as ”contacting”.
Fold
Accuracy [%]
Fold0
97.2
Fold1
96.7
Fold2
98.2
Fold3
97.5
Fold4
97.1
Average
97.3
Table 5: Electric knife’s "contacting" and "no contacting" cross-validation result.
Fold
Accuracy [%]
Fold0
93.7
Fold1
93.6
Fold2
90.9
Fold3
94.5
Fold4
91.0
Average
92.7
Table 6: Ultrasnoic cutter’s "contacting" and "no contacting" cross-validation result.
Figure 16: Examples of the false prediction images by ultrasonic cutter: First category, the prediction was one frame earlier or later than the labeled (a) contact moment and (b) the disengagement moment. Second category, (c)The angle of edge view of the ultrasonic cutter. Third category, (d) The ultrasonic cutter was a little out of focus, so the image couldn’t show a clear edge of blade.
Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, Maryland · Department of Pathology, Johns Hopkins Hospital, Baltimore, Maryland
Faculty of Science and Technology, University of Macau, Macau, China · University of Macau Advanced Research Institute in Hengqin, Zhuhai, China · Senior Department of Urology, The Chinese PLA General Hospital, Beijing, China +4