Detection and Classification of Cetacean Echolocation Clicks using Image-based Object Detection Methods applied to Advanced Wavelet-based Transformations

Christopher Hauer

Figure 1

Figure 1: Six example images of the waveform and spectrogram of different clicks and echoes in isolation. The differing SNR values can happen due to the distance of the emitted click to the hydrophone and the directionality of clicks. The spectrogram was generated with a segment size of 16 samples and a hop of 8 samples. The number of samples are very low when compared to the more commonly used 1024 segment size and 512 hop. This is necessary due to the small window size of only 384 samples (2 milliseconds using 192kHz). This explanation also holds for other pictures.

Figure 2: Three example images of click and echo pairs. Image (a) shows a usual click and echo pair, with a positive starting amplitude click and a negative starting amplitude echo. Due to a weak initial phase, the echo in image (b) appears to have a positive starting amplitude. In Image (c), due to the strong impulse compared to the initial amplitude, it looks like the click starts with a negative amplitude and echo with a positive amplitude.

Figure 3: An example of the waveform, spectrogram and labels in Audacity. The equidistant click echo pairs marked within green boxes are part of a burst. The click and echo pairs marked with orange boxes are also easy to distinguish, but not part of the same burst as the ones marked with green. The events marked with red boxes are labeled, but difficult to identify and potentially wrong.

Figure 4: Example of a low-frequency click [image (a)], a high-frequency click [image (b)] and an ultrasonic-frequency click [image (c)].

Figure 5: Examples of a reverberated echo in image (a) and a frequency shifted reverberated echo in image (b). The echo in image (b) has a higher peak and average energy than the click.

Figure 6: Three examples of possible noises. Image (a) shows mechanical or electrical interference. Image (b) depicts a click and echo pair from a dolphin, image (c) from a pilot whale.

Figure 7: This figure was generated using Audacity. It displays the process of the first order Dirac-like impulse detection (FOD). Image (a) shows an example of an original signal. Image (b) displays the absolute values of the first order of the signal. Image (c) is a moving average over the signal, which is used to increase or reduce a threshold function. Image (d) shows the noise-reduced Dirac-like remains of the absolute values of the first order signal. Some of the remains are so short that they do not appear on the spectrogram.

Figure 8: Three examples of the same click and echo with different spectrogram window sizes. Image (a) shows the waveform and spectrogram of a signal with an FFT window size of 64. With this window size, the click and echo can still be distinguished, although there is a poor tradeoff between frequency and time resolution. Image (b) uses a window size of 128 samples. With this larger window, it becomes difficult to differentiate between the click and the echo, and pinpointing their exact start or end is challenging. Finally, Image (c) uses a window size of 256 samples, where the click and echo have merged due to the reduced time resolution, thus making them indistinguishable.

Figure 9: Examples of wavelets taken from Wikipedia [ 59 ] The first step of the Haar continuous wavelet transformation of image (a) was used for the first order gradient conversion and first order Dirac-like event detection (FOD). The Mexican Hat wavelet from image (b) was used to generate the scalogram. The Mexican Hat is the smallest symmetrical wavelet in terms of used samples.

Figure 10: An example of a Scalogram with its respective waveform. The Dirac-like impulses can be easily seen and tracked as the outlying cones protruding from the top of the image. The representation can be particularly useful for a click detector.

Figure 11: A visual representation of the Rosenblatt perceptron using TikZ [ 9 ] . The inputs X are multiplied by their corresponding trainable weights W, and the weighted sum is calculated. This sum is then passed through an activation function to produce the final output.

Figure 12: Depiction of the Sigmoid(a), Tanh(b) and ReLU(c) activation functions taken from Wikipedia.

Figure 13: A depiction of a small fully connected deep neural network taken from stackexchange [ 23 ] , where every output from a previous layer is connected to the input of each perceptron in the following layer.

Figure 14: Depiction of a 2D convolution with a stride of 1 and no padding. The original 6x6 image is processed into a smaller 4x4 representation using a 3x3 kernel. The image was taken from zaforf GitHub [ 60 ] .

Figure 15: Depiction of a typical deep CNN model. In the first layer, the original image is transformed into multiple representative feature maps, every feature map is a convolution of the original image over a kernel filter. This step is repeated iteratively, the subsequent 2D convolutions in deeper layers are stacked on the feature maps generated by earlier layers until the feature map is flattened into a 1D array and further processed using a fully connected neural network. The image was taken from Park explanation of CNN models [ 26 ] .

Figure 16: A visualization of an example decision tree based on the iris dataset using sklearn and graphviz. This visualization was taken from the scikit-learn examples [ 47 ] .The Iris dataset is a well-known introductory dataset used for classifying three species of flowers (Setosa, Versicolor, and Virginica) based on the measurements of their petal and sepal width and length.

Figure 17: Depiction of the ANIMAL-SPOT model architecture and spectrograms of some of the species’ target calls taken from the ANIMAL-SPOT publication by Bergler et al. [ 3 ]

Figure 18: Three example depictions of the adjusted ANIMAL-SPOT input. To simplify the viewing, the waveform (top), CWT (middle) and spectrogram (bottom) images were stacked vertically and not over the color channels. A channel stacked example can be found in Figure 23 .

Figure 19: An example of how YOLO predicts bounding boxes based on a grid of cells. The image is taken from the YOLO publication by Redmon et al. [ 44 ] . The model divides the image into a grid and for each grid cell it predicts bounding boxes, confidence for those boxes, and class probabilities.

Figure 20: A depiction of the YOLO post-processing using the FOD. The images were made in Audacity. Figure (a) shows a scene of a low SNR signal with the corresponding YOLO bounding boxes. Due to overlap, the bounding boxes merged into a large event which includes both the click and echo. Through FOD peak detection and FOD peak grouping shown in Figure (b), the click and echo are extracted from the merged event, as can be seen in Figure (c).

Figure 21: Diagram of the random forest classifier model, showing the construction of multiple decision trees and the aggregation of their results. The example was taken from freesion [ 32 ]

Label	Train	Train Percentage	Evaluation	Evaluation Percentage
LF	119	3.3	115	4.0
HF	3491	96.6	2742	95.8
US	3	0.1	6	0.2
Clicks	3613	51.7	2863	53.8
Echo	3381	48.3	2459	46.2
All	6994		5322

Table 1: Summary of the provided labeled data. ”LF” represents low-frequency clicks (below 5 kHz), ”HF” refers to high-frequency clicks (between 5 kHz and 40 kHz), and ”US” corresponds to ultrasonic-frequency clicks (above 40 kHz).

Figure 22: A depiction on how multiple hand labeled annotations can exist in one input window, and how a hand labeled annotation can be split into multiple train entries in Audacity. On the top is the waveform of the two 5 millisecond input windows. The first text track displays the hand labeled annotations. The second text track shows the input windows. The last text track shows the train entries.

Figure 23: Example of an SCWTSPEC input image. The image has dimensions of 960x960 pixels, with the red channel containing the waveform, the green channel containing the continuous wavelet transform (CWT), and the blue channel containing the spectrogram.

	9dB	10dB	13dB	15dB	17dB	20dB	30dB	40dB
Detection	47786	38887	19902	13607	9442	5673	672	103
TP	13686	12650	9776	8185	6416	4354	617	96
FP	34100	26237	10126	5422	3026	1319	55	7
Precision	28.64	32.53	49.12	60.15	67.95	76.74	91.81	93.20
All	6994	6994	6994	6994	6994	6994	6994	6994
Found	6472	6218	5419	4617	3799	2709	437	79

Table 2: Results of the PAMGuard click detector on the train data with different decibel(dB) thresholds. The sum of all detections is the number of all PAMGuard detections using the threshold. The number of true positives (TP) is the number of PAMGuard detections that were within an event annotation. The number of false positives (FP) is the number of PAMGuard detections outside an event annotation. Precision is the percentage of correct predictions over all predictions. All is the number of annotations. The number of Found annotations is the number of event annotations that were found by at least one PAMGuard click detection. The number of Missed annotations is the number of annotations that have no PAMGuard click detection. The Recall is the percentage of found annotations over all annotations. The best overall accuracy was achieved with the 15 dB threshold, where PAMGuard identified 66.0% of annotations with an accuracy of 60.2%, resulting in an overall annotation accuracy of 39.7%. The experiment was performed on the first data set with 6994 annotations later used for training.

	FOD
Detection	4834
TP	4231
FP	603
Precision	87.52
All	5322
Found	3232

Table 3: Summary of results from the FOD-only mathematical approach. The Detection row represents the total number of FOD detections. True Positives (TP) are the FOD detections that correctly match an event annotation, while False Positives (FP) are detections that do not match any annotation. Precision is the proportion of correct predictions (TP) over total detections. All refers to the total number of annotations, with Found indicating annotations that were detected by at least one FOD, and Missed representing annotations with no FOD detections. Recall measures the proportion of found annotations over all annotations. Overall, 60.7% of annotations were correctly found with an accuracy of 87.5%, resulting in a final precision of 53.1%. The experiment was performed on the later improved dataset with 5322 event annotations.

window size	20ms/4ms
window hop	10ms/2ms
lr	10e-5
beta1	0.5
lr patience	8
lr decay	0.5
early stopping	20

Table 4: The settings used to train the ANIMAL-SPOT model. The window size specifies the size of the input images in milliseconds. The window hop represents the advancement time between consecutive windows in milliseconds. The learning rate (lr) is the preset learning rate. The adam optimizer (Beta1) controls the exponential decay rate of the first moment. Learning rate patience (lr patience) is the number of epochs without improvement on the validation set before the learning rate starts decaying. The learning rate decay (lr decay) factor determines the decay applied after the specified patience. Early stopping (early stopping) defines the number of epochs after which training stops if no improvement is observed, to prevent overfitting. Batch size is the number of images in each batch. The number of frequency bins (n freq bins) is the number of bins used to represent the given frequency range, from fmin to fmax. Number of FFT points (n fft) refers to the FFT window size in samples, and hop is the FFT hop size. The convolutional kernel size is the size of the initial square convolution in the ResNet architecture. Sampling rate is the rate at which the input signal is sampled. fmin and fmax represent the lower and upper-frequency thresholds for the ANIMAL-SPOT input image, respectively. Augmentation and min-max normalization were applied during training.

Figure 24: Depiction of the ANIMAL-SPOT 20ms results in Audacity. Overall, the network was able to learn how to differentiate events from noise, but due to the large window size, the results are not usable for further progression. This depiction displays the 20ms ANIMAL-SPOT window results in a lower click density situation.

Figure 25: Depiction of the ANIMAL-SPOT 20ms blocking problem in Audacity. The windows all merge into one large event, which makes it difficult to differentiate multiple events from each other.

	0.10	0.20	0.30	0.40	0.50	0.60	0.70	0.80	0.90
Detection	32294	24483	19918	16572	13885	11636	9652	7662	5520
TP	11411	10814	10240	9679	9065	8364	7638	6620	5155
FP	20883	13669	9678	6893	4820	3272	2014	1042	365
Precision	35.33	44.16	51.41	58.40	65.28	71.88	79.13	86.40	93.38
All	5322	5322	5322	5322	5322	5322	5322	5322	5322
Found	5149	5011	4903	4799	4685	4505	4288	3936	3316

Table 5: This table presents the results of the ANIMAL-SPOT 4ms approach at varying thresholds from 0.1 to 0.9. The TP (true positives) row shows the number of ANIMAL-SPOT windows that correspond to an event, and the FP (false positives) row indicate the number of ANIMAL-SPOT windows that detected events without annotations. Finally, the Precision row provides the detection accuracy at each threshold. The Detection row shows the total number of events for each threshold, while the Found row indicate the number of events detected by ANIMAL-SPOT. The Missed row represents the number of events that were missed by ANIMAL-SPOT. The Recall measures the proportion of found annotations over all annotations. Overall, ANIMAL-SPOT achieved its best accuracy of 63.9% (86.4% label accuracy and 73.9% detection accuracy) with a threshold of 80. The experiment was performed on the improved evaluation file with 5322 annotations.

Figure 26: Depiction of the ANIMAL-SPOT 4ms results in Audacity. Overall, the 4ms results are more precise than the 20ms results. Yet, the windows are still too large to differentiate between clicks and echoes. This depiction displays the 4ms ANIMAL-SPOT window results in a lower click density situation.

Figure 27: This image was made in Audacity. Unlike the 20ms windows, the 4ms windows do not block into one event, so while it is better in differentiating events, it still blocks clicks and echo together. This depiction displays the 4ms ANIMAL-SPOT window results in a higher click density situation.

	SIGNAL	CWT	SPEC	SCWT	SSPEC	CWTSPEC	SCWTSPEC
Detection	3545	5982	8678	5168	8359	5487	6255
TP Partial	3118	4433	4970	3997	4894	4221	4675
TP Full	2120	3998	4352	3478	4294	3697	4110
TP Partial%	87.95	74.11	57.27	77.34	58.54	76.92	74.74
TP Full%	59.80	66.83	50.14	67.29	51.36	67.37	65.70
FP	427	1549	3708	1171	3465	1266	1580

Table 6: Results of the representation combination experiment. The columns represent different combinations of image representations: SIGNAL (signal representation), CWT (continuous wavelet transformation), SPEC (spectrogram), SCWT (signal and continuous wavelet), SSPEC (signal and spectrogram), CWTSPEC (continuous wavelet transformation and spectrogram), and SCWTSPEC (all three representations combined). The Detection row indicates the number of detected events, while TP Partial and TP Full describe the true positive detections with overlaps of more than 20% and 90% with the hand annotated events, respectively. TP Partial% and TP Full% show the percentage of partial and full detections over all detections. FP describes the number of false positives, detected windows that have an overlap below 20% or no overlap. FP% is the percentage of false positives over all detection. All describes the number of all hand annotated events, Partial and Full describe the hand annotated events with overlaps of more than 20% and 90% with the detections. Partial% and Full% show the percentage of annotated events with partial and full overlaps over all annotated events. Missed describes the number of annotations with less than 20% detection overlap, or no overlap. Missed% is the percentage of missed annotations over all annotations. SCWTSPEC achieved the best partial and full overlap percentages for ground truth labels. This proves that every representation adds a noticeable improvement to the model. This early experiment was performed on the first evaluation dataset with 5063 event annotations. An experiment on the improved dataset was deemed unnecessary.

	13	14	15	16	17	19
Detection	4487	4491	4515	4520	4496	4504
TP Partial	3018	3068	3115	3183	3250	3337
TP Full	2723	2753	2780	2835	2876	2916
TP Partial%	67.26	68.31	68.99	70.42	72.28	74.08
TP Full%	60.68	61.30	61.57	62.72	63.96	64.74
FP	1469	1423	1400	1337	1246	1167

Table 7: Results of the confidence threshold experiment. The table shows the performance of the YOLO model at different confidence threshold values (ranging from 13 to 19 from a confidence between 1 and 100) for event detection. The results are portioned the same as in Table 6 . The experiment demonstrates that adjusting the confidence threshold serves as a tradeoff between false positives and the number of detected events, with higher thresholds increasing partial overlaps in both YOLO detections and ground truth events. This experiment was also performed on the first evaluation dataset with 5063 event annotations instead of the later improved dataset with 5322. A redo of this experiment was deemed unnecessary.

Figure 28: Example depiction of a split event. The windows end and start in the middle of the click, as such three events were generated instead of one.

Detection	6416
TP Partial	5103
TP Full	2967
TP Partial%	79.53
TP Full%	46.24
FP	1313
FP%	20.46

Table 8: The experiment was performed on the improved evaluation data. As such, there are 5322 instead of 5063 Ground truth labels. The bounding boxes were provided with a confidence threshold of 0.15 (see Table 7 ). The results are partitioned the same as in Table 6 . The FOD splitting increased the number of events from 4487 to 6416. In addition, the split bounding boxes were less likely to have a full overlap. Due to the FOD confidence threshold adjustment, the overall findings of the labels were partially reduced from 94.0% to 89.8%, but the accuracy of the detector rose from 69.0% to 79.5% leading to a reduction of false positives. These changes increased the overall accuracy from 63.0% to 71.4%.

Random Forest Classifier input vector
start
end
confidence
length
number FOD
minimum energy

Table 9: The start and end represent the event’s beginning and end times. The confidence represents the confidence of the YOLO detection for the bounding box. The length describes the event’s duration, and the number FOD refers to the number of FOD peaks within the bounding box. Additional features include the minimum and maximum energy levels, the mean energy, and the max FOD peak, which represents the strongest FOD peak. The FOD direction reflects the phase of the strongest FOD peak, and the strongest frequency identifies the frequency bin with the highest energy. Finally, the interarrival time measures the gap between the current and next bounding box.

	confidence	15	20	25	30	35	40
YOLO detection	Detection	6416	6070	5793	5559	5360	5181
	TP Partial	5103	4947	4809	4691	4571	4472
	TP Full	2967	2861	2770	2688	2608	2544
	TP Partial%	79.53	81.49	83.01	84.38	85.27	86.31
	TP Full%	46.24	47.13	47.81	48.35	48.65	49.10
	FP	1313	1123	984	868	789	709

Table 10: Results of the optimization experiments. The experiment was performed on the improved dataset with 5322 labeled events. The rows are described identical to Table 6 . The confidence threshold were increased to see if a better recall can be achieved. Clicks and echoes were also separated to see how the model confidence threshold would change the overlap with the annotations. This table only shows the first half of the experiment, from the confidence threshold 15 to 40. The second half from 45 to 75 is displayed in Table 11 . The 30 confidence threshold gave the best recall from the experiment. As such, a threshold of 30 was used for the rest of this study.

	confidence	45	50	55	60	65	70	75
YOLO detection	Detection	5024	4887	4770	4664	4556	4457	4362
	TP Partial	4391	4304	4224	4156	4082	4013	3953
	TP Full	2490	2438	2387	2326	2290	2262	2236
	TP Partial%	87.40	88.07	88.55	89.10	89.59	90.03	90.62
	TP Full%	49.56	49.88	50.04	49.87	50.26	50.75	51.26
	FP	633	583	546	508	474	444	409

Table 11: Second half of the experiment from Table 10 .

Confidence	Event Correlation	Click correlation	Echo correlation
15	94.51	97.84	84.87
20	94.79	98.24	85.20
25	95.04	98.37	85.86
30	95.06	98.48	84.36
35	95.03	98.00	85.37
40	94.92	98.10	86.34

Table 12: The event rate, click rate and echo rate correlation results based on the final CLICK-SPOT model confidence threshold. Overall, the correlation between the found click events and the annotated click events is higher than between the echoes. The best results were achieved with a confidence threshold of 30.

Figure 29: Extract of the Atlantic white sided Dolphins track in Audacity. The first text labels show the results of the unoptimized approach. The second text label shows the optimized approach.

Figure 30: Extract of the pilot whale track in Audacity. The first text labels show the results of the unoptimized approach. The second text label shows the optimized approach.

Figure 31: Extract of the sperm whale track in Audacity. The first text labels show the results of the unoptimized approach. The second text label shows the optimized approach.

Detection and Classification of Cetacean Echolocation Clicks using Image-based Object Detection Methods applied to Advanced Wavelet-based Transformations

Abstract

Figures & tables

Sound Event Detection

Classifying bioacoustic data without individual call annotations using temporal convolutional networks and feature extractors

Large-scale bioacoustic detection using semantic segmentation: a deep learning framework applied to fin whale calls in ocean-bottom seismometer recordings

Hearing the Ocean: Bio-inspired Gammatone-CNN framework for Robust Underwater Acoustic Target Classification

Abstract

Sound Event Detection

Explore similar work

Classifying bioacoustic data without individual call annotations using temporal convolutional networks and feature extractors

Large-scale bioacoustic detection using semantic segmentation: a deep learning framework applied to fin whale calls in ocean-bottom seismometer recordings

Hearing the Ocean: Bio-inspired Gammatone-CNN framework for Robust Underwater Acoustic Target Classification