Authors: Hongjie Chen, Hanyu Meng, Huimin Zeng, Ryan A. Rossi, Lie Lu, Josh Kimball
Organizations: Dolby Laboratories, Atlanta, USA · The University of New South Wales, Sydney, Australia · University of Illinois Urbana-Champaign, Champaign, USA · Adobe Research, San Jose, USA · Dolby Laboratories, San Francisco, USA
Audio fingerprinting converts audio to much lower-dimensional representations, allowing distorted recordings to still be recognized as their originals through similar fingerprints. Existing deep learning approaches rigidly fingerprint fixed-length audio segments, thereby neglecting temporal dynamics during segmentation. To address limitations due to this rigidity, we propose Variable-Length Audio FingerPrinting (VLAFP), a novel method that supports variable-length fingerprinting. To the best of our knowledge, VLAFP is the first deep audio fingerprinting model capable of processing audio of variable length, for both training and testing. Our experiments show that VLAFP outperforms existing state-of-the-arts in live audio identification and audio retrieval across three real-world datasets.
Figures & tables
Figure 1. Limitations of fixed-length audio fingerprinting. The top row shows a 3 -second excerpt (containing both interlude and verse) in its original form and with a 1.5× speed-up. The middle and bottom rows compare fixed-length segmentation and variable-length segmentation. Fixed-length segmentation suffers from three issues: Loss of Natural Boundaries. Segments cut across semantic units ((a)-2), complicating interpretation. Redundant or Noisy Context. Segments oversimplify ((a)-1) or combine too much information ((a)-2). Distortion Incompatibility. Time-stretch prevents exact matching; no subfigure in (b) aligns perfectly with (a)’s. Variable-length segmentation addresses them through segments aligned with semantic boundaries.
Figure 2. The architecture of our proposed VLAFP model. (a) Initial Projection: Audio A in its spectrogram representation is projected through a linear layer. (b) Inter-frame Self-Attention: Multi-head self-attention layers learn inter-frame relationships. (c) Frame-to-segment Cross-Attention: Multi-head cross-attention layers model the frame-to-segment relationships. (d) Segment Embedding Initialization: Replicas of segment embeddings are initialized through frame-to-segment pooling. (e) Fingerprint Summarization: Replicas of segment embeddings are aggregated and L2-normalized to generate an audio fingerprint z .
Task
Fingerprint database source
Query (distorted)
Segmentation
Distortion
Objective
Dataset
Metric
VL
FL (1 sec)
TS
BG
IR
CBR
a commercial
distorted broadcast
VLAFP
baselines
✓
✓
✓
identify commercial
all
precision, recall, F1
DTR
dummy + original target
distorted target
all methods
✓
✓
retrieve original
FMA
top-1 hit rate
Table 1. Setup of Commercial-Broadcast Retrieval (CBR) and Dummy-Target Retrieval (DTR) .
Figure 3. Diagram of Commercial-Broadcast Retrieval (CBR).
Figure 4. Diagram of Dummy-Target Retrieval (DTR).
Method
FMA
LibriSpeech
AudioSet
Precision
Recall
F1
Precision
Recall
F1
Precision
Recall
F1
wav2vec2
5.76
32.78
9.79
3.88
26.09
6.75
5.32
40.63
9.40
HuBERT
4.87
98.94
9.29
4.53
15.48
7.01
6.96
25.25
10.91
AST
9.62
30.88
14.68
3.80
25.03
6.59
12.80
26.24
17.21
AMG
25.81
31.37
28.32
22.51
28.82
25.28
17.48
27.72
21.44
NAFP
75.84
64.23
69.55
44.74
33.37
38.23
55.95
34.32
42.54
Table 2. CBR results on FMA , LibriSpeech , and AudioSet . Values are reported as percentages ( % ) for precision , recall , and F1-score , with the best scores highlighted in bold and second best in italics .
Table 7Table 8
Figure 5. Segment count and average length.
Figure 6. Segment length distribution.
Segmentation Threshold θ
VLAFP on FMA
Precision
Recall
F1
0
79.28
71.63
75.26
1
81.00
70.15
75.19
2
74.86
65.24
69.72
3
74.16
61.31
67.12
4
71.99
64.95
68.29
Table 7. CBR results of VLAFP on FMA when trained with different segmentation threshold θ∈{0,1,2,3,4,+∞} . Smaller θ (0 or 1) with more shorter segments yields better results.
Segmentation Method
VLAFP on FMA
Precision
Recall
F1
( main )
81.00
70.15
75.19
No Silence
76.29
67.86
71.83
Pelt
69.84
60.31
64.73
Waveform
78.1
68.60
73.07
Table 8. CBR results when trained with different segmentation methods, including our proposed segmentation method ( main ) and three other methods. main yields the best results.
Figure 7. DTR results of VLAFP on FMA with different θ .
Figure 8. DTR results on FMA with different segmentations.
Augmentation
VLAFP on FMA
TS
BG
IR
Precision
Recall
F1
No Augmentation
26.54
24.62
25.54
✓
51.68
39.87
45.01
✓
66.16
60.70
63.31
✓
76.27
66.17
70.86
✓
✓
67.95
63.45
65.62
Table 9. CBR results on FMA with different augmentations: time stretching (TS), background mixing (BG), and impulse response convolution (IR). The best results are obtained when using all augmentation types.
#pos
VLAFP on FMA
Precision
Recall
F1
1
76.38
66.46
71.07
2
74.93
67.63
71.09
3
76.38
64.56
69.97
4
75.41
65.49
70.10
5
74.55
63.77
68.74
Table 10. CBR results of VLAFP on FMA when trained with different numbers of positive samples #pos∈{1,2,3,4,5} per anchor. Fewer positive samples yield better result.
Figure 9. DTR results on FMA with different augmentations.
Figure 10. DTR results on number of positive samples.
Symbol
Meaning
Symbol
Meaning
Symbol
Meaning
a
audio signal
n
audio length
z
audio fingerprint
A
audio in time-frequency rep.
T
audio frame length
F
number of frequency bins
d
feature dimension
dh
head dimension
H
number of attention heads
L
number of blocks
h0
initial embedding
hl,h~l
frame-level embedding
xql
query variable
xkvl
key/value variable
sl
seg.-level embedding
a
anchor sample
a+
positive sample to a
P(a)
set of positive samples
Table 11. A summary of notations.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Split
#Audio
Duration (seconds / hours)
Type
Min.
Med.
Avg.
Max.
Total
FMA
Train
10,000
6.2
30.0
30.0
30.0
83.3 h
Music
Test
500
30.0
30.0
30.0
30.0
4.2 h
LibriSpeech
Train
5,459
1.4
13.8
12.3
17.3
25.9 h
Speech
Test
2,620
1.3
5.8
7.4
35.0
5.4 h
AudioSet
Train
10,000
1.0
9.9
10.0
10.0
27.5 h
General
Appendix
Table 12. A summary of dataset statistics. The train / test split is configured using the original dataset splits.
Figure 11. Details of database construction and broadcast construction & query in Commercial-Broadcast Retrieval (CBR).
Figure 12. Details of database construction and query in Dummy-Target Retrieval (DTR).
Figure 13. A sensitivity study of hyperparameter sensitivity of VLAFP. Each row is for a specific number of heads H∈{8,16,32} and each column is for a specific hidden dimension d∈{64,128,256} . In each figure, each line is for a specific learning rate η∈{10−5,10−4,10−3} , and each dot is for a specific number of blocks L∈{4,8,16} . The average accuracy across query duration is reported. See Sec. D for detailed discussions.
Method
dim d
CBR Commercial
CBR Broadcast
DTR Dummy
DTR Target
#seg
size
#seg
size
#seg
size
#seg
size
wav2vec2
1568
15k
87M
289k
1.7GB
581k
3.6GB
30k
177MB
HuBERT
1568
15k
87M
289k
1.7GB
581k
3.6GB
30k
177MB
AST
527
15k
29M
289k
581MB
581k
1.3GB
30k
60MB
AMG
128
15k
7M
289k
141MB
581k
299MB
30k
15MB
NAFP
128
15k
7M
289k
141MB
581k
299MB
30k
15MB
Appendix
Table 13. A comparison of storage across methods for CBR and DTR on the FMA dataset. The best (lowest) values are highlighted in bold and second best in italics .
Audio fingerprinting methods identify audio based on its content, even when the content has been possibly corrupted. Despite their applicability and usage in the speech domain, research and advancements in audio fingerprinting traditionally focus on music and are almost exclusively evaluated on music. We provide insights into speech-oriented audio fingerprinting by benchmarking the best performing audio fingerprinting models on speech data while taking into consideration the effects of noise, reverberation, temporal changes, pitch modifications, and lossy compression. Our findings indicate that audio fingerprinting methods based on self-supervised learning outperform classical approaches on speech data, while the performance of these models on speech is inferior compared to the performance that they achieve on music. Results also include an analysis of false positives, showing that fingerprinting methods are more likely to match with an audio of the same speaker rather than an audio of the same text, when mismatches occur.
Kemal Altwlkany, Elmedin Selmanovic, Sead Delalić +1
Infobip · Faculty of Science, University of Sarajevo
This work presents POLARIS, a training-free audio fingerprinting system that selects landmarks from a locally normalized saliency field and groups them into sparse fingerprints using Delaunay triangulation. To deal with query distortion, POLARIS adds fingerprints from two-hop Delaunay neighborhoods only at query time, without enlarging the reference index. An adaptive configuration applies this expansion only when the original fingerprints do not produce a confident match. We evaluate POLARIS on synthetic distortions from the public PEX Hard Medium benchmark, excluding queries with pitch or tempo shifts, and on a new benchmark of real re-recorded music. POLARIS achieves the best performance among the evaluated training-free methods on both benchmarks. On the real recordings, its adaptive configuration also outperforms the neural NMFP baseline with a comparable measured query time and a smaller logical reference payload. Code, dataset, and instructions for reproducing all experiments are available at https://github.com/JihengLi/POLARIS.git.
Jiheng Li
Department of Computer Science, Vanderbilt University, Nashville, TN, 37235
Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, downstream finetuning has now reached a state of diminishing returns. In this paper, we shift the focus to pretraining and propose a novel recipe that combines bottleneck masked embedding prediction with flow-matching based spectrogram reconstruction. The outcome, Alethia, is the first foundational audio encoder for various voice deepfake detection and localization tasks. We evaluate on 5 different tasks with 56 benchmark datasets, and note Alethia significantly outperforms state-of-the-art SFMs with superior robustness to real-world perturbations and zero-shot generalization to unseen domains (e.g., singing deepfakes). We also demonstrate the limitation of discrete targets in masked token prediction, and show the importance of continuous embedding prediction and generative pretraining for capturing deepfake artifacts.