Audio fingerprinting methods identify audio based on its content, even when the content has been possibly corrupted. Despite their applicability and usage in the speech domain, research and advancements in audio fingerprinting traditionally focus on music and are almost exclusively evaluated on music. We provide insights into speech-oriented audio fingerprinting by benchmarking the best performing audio fingerprinting models on speech data while taking into consideration the effects of noise, reverberation, temporal changes, pitch modifications, and lossy compression. Our findings indicate that audio fingerprinting methods based on self-supervised learning outperform classical approaches on speech data, while the performance of these models on speech is inferior compared to the performance that they achieve on music. Results also include an analysis of false positives, showing that fingerprinting methods are more likely to match with an audio of the same speaker rather than an audio of the same text, when mismatches occur.
Figures & tables
Figure 1: Runtime of fingerprinting methods shown at logarithmic scale. The orange line indicates the median value.
fingerprints
fingerprint size (bits)
total size (MB)
audfprint
1,509,822
20
3.59
GraFPrint
1,107,864
4096
540
PTC
378,248
4096
184
Table 1: Memory footprint per fingerprinting method for storing the fingerprints of VCTK speech corpus without metadata. Note that for audfprint a single hash is considered a fingerprint.
audfprint
GraFPrint
PTC
Hit rate (%)
27.82±5.18
77.21±4.88
47.56±5.56
Table 2: Retrieval rates when the query sample has been distorted by artificially synthesized reverbant conditions.
Figure 2: Retrieval rates when the query sample has been corrupted by background noise or coloured noise.
audfprint
GraFPrint
PTC
Hit rate (%)
91.00±3.33
80.89±4.55
96.69±2.02
Table 3: Retrieval rates when the query sample has been temporally offset from the reference sample.
Figure 3: Retrieval rates when the query sample’s playback rate has been modified, resulting in either a sped up or slowed down version of the audio.
audfprint
GraFPrint
PTC
Downshift 1 semitone
0.00±0.00
97.26±1.94
52.84±5.82
Downshift 0.5 semitones
0.09±0.58
97.87±1.87
93.11±3.11
Upshift 0.5 semitones
0.08±0.55
98.10±1.94
95.04±2.71
Upshift 1 semitone
0.00±0.00
97.98±1.65
41.98±5.69
Table 4: Retrieval rates when the query sample has been pitch-shifted.
Figure 4: Retrieval rates of fingerprinting methods when the query sample has been quantized and downsampled by a lossy compression format or codec.
Figure 5: Mismatch analysis: fingerprinting methods are more likely to match with the same speaker, rather than the same text.
Background noise
5 dB
10 dB
15 dB
20dB
Speech
audfprint
33.6±5.4
51.7±5.7
64.2±5.5
72.9±5.1
GraFPrint
59.9±5.6
76.5±4.8
84.4±4.1
88.4±3.8
PTC
35.9±5.4
50.7±5.7
67.1±5.4
82.9±4.2
Music
audfprint
63.4±5.4
76.2±4.9
85.3±4.3
89.1±3.8
GraFPrint [ 4 ]
98.3
99.3
99.6
99.9
Table 5: Comparison of audio fingerprinting retrieval rates between music and speech data.