Self-Supervised Diffusion Models for Handwritten Signature Verification via Ruin Reconstruction
Authors: Peirong Zhang, Zhenhua Yang, Jiaxin Zhang, Moises Diaz, Miguel A. Ferrer, Umapada Pal, Wei Zhou, Lianwen Jin
Organizations: School of Electronic and Information Engineering, South China University of Technology, Guangzhou, China. · Instituto Universitario para el Desarrollo Tecnológico y la Innovación en Comunicaciones, Universidad de Las Palmas de Gran Canaria, Las Palmas, Spain. · Computer Vision and Pattern Recognition Unit, Indian Statistical Institute, Kolkata 700108, India. · School of Computer Science and Informatics, Cardiff University, Cardiff, UK.
Handwritten signature verification (HSV) is a critical biometric technique in legal and forensic authentication, whose efficacy hinges on the discriminability of signature representations. Current HSV methods predominantly adopt supervised learning to enhance representation quality, leaving self-supervised learning largely unexplored. We propose SigDiffuser, a diffusion-based self-supervised framework for signature representation and verification. At its core, SigDiffuser introduces a novel pretext task termed Reconstruction from Ruins (RFR), which deliberately ruins'' signatures through severe crumpling, obliterating their structural integrity and writing styles. A visual encoder is employed to capture crucial handwriting patterns from intact signature regions, guiding the denoiser to reconstruct the ruined'' portions and learning meaningful representations. Additionally, we introduce a pyramidal feature aggregation module (PFAM) to bridge the visual encoder and denoiser, enhancing representation quality via hierarchical aggregation and modulation. Extensive experiments on multilingual signature datasets demonstrate that SigDiffuser significantly outperforms both state-of-the-art SSL methods and HSV models. We further show that increasing the crumpling severity on signatures effectively improves representation quality. Notably, our work demonstrates that extreme destruction, coupled with self-supervised diffusion models, can yield superior signature representation compared to traditional mild deformations, paving new avenues for HSV and other biometric domains. Code will be publicly available.
Figures & tables
Fig. 1 : Qualitative and quantitative (Structural Similarity Index (SSIM)) comparisons between MIM and our proposed RFR. (a) MIM exhibits blurred, oversimplified structures that fail to recover complex strokes, resulting in suboptimal representations. (b) In contrast, RFR faithfully reconstructs stroke details and character structures despite severe crumpling, achieving higher SSIM metrics. Although not exact replicas, the fidelity suffices for capturing effective signature representation.
Fig. 2 : Overall framework of SigDiffuser. (a) Signature data construction for RFR. We apply crumpling on the signature image I to obtain Ic and segment square patches in different positions to acquire Iorg , Isty , and Icor for RFR pretraining. (b) Detailed model architecture and self-supervised pretraining process. The visual encoder encodes the intact region Isty into latent representations. These features are hierarchically modulated to the denoiser to assist in reconstructing the severely crumpled portion Icor , thereby enforcing the visual encoder to learn meaningful representations.
Fig. 3 : Downstream model architecture for HSV. (a) Training. The encoder pretrained on the RFR pretext task is transferred to the downstream model for HSV fine-tuning. (b) Testing. All signatures are input to the model to extract corresponding feature vectors, which are then used for verification.
Dataset
Language
#Writer
#Writer Training
#Writer Testing
#Genuine Sig./writer
#Forged Sig./writer
#Split
#Training
#Testing
#Total
BHSig260-B [ 20 ]
Bengali
100
50
50
24
30
5
2,700
2,700
5,400
BHSig260-H [ 20 ]
Hindi
160
100
60
24
30
5
5,400
3,240
8,640
HanSig [ 21 ]
Chinese
238
213
25
20
20
10
31,800
3,600
35,400
UTSig [ 22 ]
Persian
115
60
55
27
45
5
4,320
3,960
8,280
CEDAR [ 23 ]
English
55
50
5
24
24
11
2,400
240
2,640
TABLE I : Dataset details and data splitting. #Writer Training and #Writer Testing respectively denote the number of writers for training and testing. Sig. denotes signature. /writer denotes per writer.
Method
Venue
Linear Probing
Downstream Fine-tuning
BHSig260-B
BHSig260-H
BHSig260-B
BHSig260-H
EER ↓
FAR ↓
FRR ↓
Acc ↑
EER ↓
FAR ↓
FRR ↓
Acc ↑
EER ↓
FAR ↓
FRR ↓
Acc ↑
EER ↓
FAR ↓
FRR ↓
Acc ↑
Contrastive SSRL Methods
SimCLR [ 60 ]
ICML’20
34.96 ( ± 0.57)
34.96 ( ± 0.57)
34.96 ( ± 0.57)
65.04 ( ± 0.57)
35.55 ( ± 0.54)
35.56 ( ± 0.54)
35.54 ( ± 0.54)
64.45 ( ± 0.54)
8.28 ( ± 1.52)
8.28 ( ± 1.53)
8.28 ( ± 1.52)
91.72 ( ± 1.52)
6.36 ( ± 1.23)
6.37 ( ± 1.22)
6.36 ( ± 1.23)
93.63 ( ± 1.22)
SimCLR* [ 60 ]
ICML’20
39.59 ( ± 0.98)
39.58 ( ± 0.99)
39.59 ( ± 0.98)
60.42 ( ± 0.99)
38.35 ( ± 0.68)
38.35 ( ± 0.69)
38.36 ( ± 0.67)
61.65 ( ± 0.68)
9.67 ( ± 1.58)
9.67 ( ± 1.58)
9.67 ( ± 1.58)
90.33 ( ± 1.58)
8.54 ( ± 1.53)
8.54 ( ± 1.54)
8.54 ( ± 1.53)
91.46 ( ± 1.54)
BYOL [ 61 ]
NeurIPS’20
13.51 ( ± 1.69)
13.52 ( ± 1.69)
13.51 ( ± 1.70)
86.48 ( ± 1.69)
27.08 ( ± 0.90)
27.08 ( ± 0.90)
27.08 ( ± 0.90)
72.92 ( ± 0.90)
5.86 ( ± 0.73)
5.86 ( ± 0.73)
5.85 ( ± 0.73)
94.14 ( ± 0.73)
7.11 ( ± 1.67)
7.11 ( ± 1.67)
7.11 ( ± 1.67)
92.89 ( ± 1.67)
TABLE II : Linear probing and downstream fine-tuning results on BHSig260-B [ 20 ] and BHSig260-H [ 20 ] . The best and second-best results are marked in bold and with underlines , with the same annotations throughout. Superscripts *, † , ⋆ indicate models utilizing their original backbones: ResNet-50 [ 53 ] , a modified 10-layer VGG [ 58 ] , and ConvNeXt-Small [ 59 ] , respectively.
Table 6Table 7
Method
Venue
EER ↓
FAR ↓
FRR ↓
Acc ↑
SigNet-F [ 40 ]
PR’17
5.92
-
-
-
FMPNet [ 51 ]
ICFHR’18
12.17
12.08
12.17
87.87
IDN [ 6 ]
CVPR’19
3.62
5.87
2.17
-
SDINet [ 67 ]
AAAI’21
1.75
0.73
3.42
-
MSDN [ 43 ]
PR’21
6.74
-
-
-
CaP [ 68 ]
ICPR’21
4.34
4.34
4.34
-
TABLE VII : Performance comparison on CEDAR [ 23 ] .
#Line
Case
BHSig260-B
BHSig260-H
EER ↓
FAR ↓
FRR ↓
Acc ↑
EER ↓
FAR ↓
FRR ↓
Acc ↑
1
Baseline
5.77
5.78
5.76
94.23
5.95
5.96
5.95
94.04
2
Baseline w/ RFR-based SSRL; w/ Icor
2.60
2.60
2.60
97.40
5.18
5.19
5.18
94.82
3
Baseline w/ RFR-based SSRL; w/ Icor ; w/ PFAM
1.58
1.58
1.57
98.43
2.59
2.59
2.58
97.41
4
Baseline w/ RFR-based SSRL; w/o Icor ; w/ PFAM
4.34
4.34
4.33
95.66
3.31
3.31
3.31
96.69
5
Baseline w/ MIM-based SSRL; w/ Icor ; w/ PFAM
3.39
3.39
3.39
96.61
4.13
4.13
4.13
95.87
TABLE VIII : Ablation study on BHSig260-B [ 20 ] and BHSig260-H [ 20 ] . SSRL denotes self-supervised representation learning.
Source
Model
Venue
Target
BHSig260-B
BHSig260-H
CEDAR
HanSig
EER ↓
Acc ↑
ΔAcc↓
EER ↓
Acc ↑
ΔAcc↓
EER ↓
Acc ↑
ΔAcc↓
EER ↓
Acc ↑
ΔAcc↓
BHSig260-B
SigNet [ 41 ]
arXiv’17
33.89
86.11
0.00
-
52.78
33.33
-
50.00
36.11
-
-
-
IDN [ 6 ]
CVPR’19
-
95.32
0.00
-
74.30
21.02
-
50.00
45.32
-
-
-
CaP [ 68 ]
ICDAR’21
3.96
96.04
0.00
-
69.59
26.45
-
66.29
29.75
-
-
-
ILNNeck [ 70 ]
IJCNN’22
5.72
94.28
0.00
-
79.21
15.07
-
80.34
13.94
-
-
-
TABLE IX : Cross-dataset validation. Models are trained on the source dataset and tested on the target dataset. ΔAcc denotes the accuracy drop from the best in-domain accuracy, where smaller values indicate better cross-dataset generalization.
TABLE X : Comparison of corruption degrees on BHSig260-B [ 20 ] and BHSig260-H [ 20 ] . (a) Setting 1. We use α to control the crumpling severity, ranging from 0 (no crumpling) to 27,108 (highly intensified). (b) Setting 2. We use existing data-augmentation techniques to control the crumpling degree: None (SimAN [ 19 ] ), Mild (CMT-Co [ 64 ] ), and Full (our RFR).
Fig. 4 : Visualized comparison of varying corruption degrees.
Fig. 5 : Grad-CAM [ 74 ] visualizations of downstream models. While MAE produces sharp responses on local, isolated strokes, SigDiffuser highlights broader, discriminative structures like stroke connections and turns, capturing rich personal writing styles.
Fig. 6 : t-SNE visualizations of downstream HSV embeddings (partial samples shown; full results in Appendix H). We use the Silhouette score [ 75 ] to quantify the separation degree (higher is better). Compared to the baseline and MAE, SigDiffuser forms more compact writer-level clusters with significantly higher scores, demonstrating superior feature separability.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Model
EER ↓
FAR ↓
FRR ↓
Acc ↑
Existing SOTA Model
CWA [ 45 ]
2.53
2.95
2.14
97.27
Ours
Split-1
1.59
1.59
1.59
98.41
Split-2
2.45
2.44
2.45
97.55
Split-3
2.23
2.23
2.22
97.77
Appendix
TABLE XI : Per-split performance on BHSig260-B [ 20 ] .
Model
EER ↓
FAR ↓
FRR ↓
Acc ↑
Existing SOTA Model
TransOSV [ 5 ]
3.24
3.24
3.24
-
Ours
Split-1
3.13
3.13
3.13
96.87
Split-2
3.31
3.31
3.31
96.69
Split-3
3.44
3.44
3.44
96.56
Appendix
TABLE XII : Per-split performance on BHSig260-H [ 20 ] .
Model
EER ↓
FAR ↓
FRR ↓
Acc ↑
Existing SOTA Model
MGRNet [ 21 ]
9.93
11.85
7.69
-
CWA [ 45 ]
9.47
6.61
16.23
90.26
Ours
Split-1
5.23
5.23
5.25
94.76
Split-2
7.36
7.36
7.36
92.64
Appendix
TABLE XIII : Per-split performance on HanSig [ 21 ] .
Model
EER ↓
FAR ↓
FRR ↓
Acc ↑
Existing SOTA Model
UTSig-S1 [ 22 ]
29.71
21.29
39.27
-
HTCSigNet [ 47 ]
28.93
28.92
28.96
71.08
Ours
Split-1
24.22
24.22
24.23
75.78
Split-2
20.50
20.51
20.50
79.49
Appendix
TABLE XIV : Per-split performance on UTSig [ 22 ] .
Model
EER ↓
FAR ↓
FRR ↓
Acc ↑
Existing SOTA Model
IDN [ 6 ]
3.62
5.87
2.17
-
SDINet [ 67 ]
1.75
0.73
3.42
-
Ours
Split-1
1.59
1.60
1.59
98.40
Split-2
3.26
3.26
3.26
96.74
Appendix
TABLE XV : Per-split performance on CEDAR [ 23 ] .
Architecture
BHSig260-B
BHSig260-H
EER ↓
FAR ↓
FRR ↓
Acc ↑
EER ↓
FAR ↓
FRR ↓
Acc ↑
Concatenation
1.77
1.77
1.77
98.23
3.83
3.83
3.83
96.17
Cross-Attention
4.78
4.78
4.77
95.22
7.11
7.11
7.11
92.89
Hierarchical Bottleneck
1.97
1.98
1.97
98.03
2.27
2.27
2.27
97.73
PFAM
1.58
1.58
1.57
98.43
2.59
2.59
2.59
97.41
Appendix
TABLE XVI : Comparison of different multi-scale feature aggregation architectures used for SigDiffuser on BHSig260-B [ 20 ] and BHSig260-H [ 20 ] . PFAM denotes the pyramidal feature aggregation module of SigDiffuser.
Method
BHSig260-B
BHSig260-H
EER ↓
FAR ↓
FRR ↓
Acc ↑
EER ↓
FAR ↓
FRR ↓
Acc ↑
w/o Diffusion
6.85
6.85
6.86
93.15
5.67
5.67
5.67
94.33
w/ Diffusion
1.58
1.58
1.57
98.43
2.59
2.59
2.59
97.41
Appendix
TABLE XVII : Performance comparison of SigDiffuser pretrained with versus without diffusion modeling on BHSig260-B [ 20 ] and BHSig260-H [ 20 ] .
#Epoch
BHSig260-B
BHSig260-H
EER ↓
FAR ↓
FRR ↓
Acc ↑
EER ↓
FAR ↓
FRR ↓
Acc ↑
300
2.13
2.13
2.14
97.87
3.19
3.19
3.19
96.81
500
1.90
1.90
1.90
98.10
3.21
3.21
3.21
96.79
700
2.14
2.13
2.14
97.86
2.63
2.63
2.63
97.37
800
1.58
1.58
1.57
98.43
2.59
2.59
2.59
97.41
Appendix
TABLE XVIII : Performance comparison under different numbers of pretraining epochs in the RFR task on BHSig260-B [ 20 ] and BHSig260-H [ 20 ] .
Fig. 7 : Full t-SNE visualizations [ 80 ] of downstream HSV embeddings. We use the Silhouette score [ 75 ] to quantify the compactness and separation (higher is better). SigDiffuser yields more compact writer-level feature clusters with significantly larger Silhouette scores than the baseline and MAE, validating its effectiveness in learning separable writer-level representations.
l
BHSig260-B
BHSig260-H
EER ↓
FAR ↓
FRR ↓
Acc ↑
EER ↓
FAR ↓
FRR ↓
Acc ↑
32
4.92
4.92
4.91
95.08
4.44
4.44
4.44
95.56
48
3.07
3.08
3.07
96.93
4.69
4.70
4.69
95.30
64
2.38
2.38
2.38
97.62
3.38
3.38
3.38
96.62
96
1.58
1.58
1.57
98.43
2.59
2.59
2.59
97.41
128
1.66
1.67
1.66
98.34
4.02
4.02
4.03
95.98
Appendix
TABLE XIX : Performance comparison of varying signature patch size (side length) l used in the RFR task on BHSig260-B [ 20 ] and BHSig260-H [ 20 ] .
Crumpling Effect
BHSig260-B
BHSig260-H
EER ↓
FAR ↓
FRR ↓
Acc ↑
EER ↓
FAR ↓
FRR ↓
Acc ↑
Forged Signature
3.04
3.04
3.04
96.96
4.09
4.09
4.09
95.91
Vanilla Crumpling
1.58
1.58
1.57
98.43
2.59
2.59
2.59
97.41
Appendix
TABLE XX : Comparison between using forged signatures (natural destruction) and RFR’s severe crumpling as destruction methods on BHSig260-B [ 20 ] and BHSig260-H [ 20 ] .
Fig. 8 : Reconstruction visualizations of SigDiffuser on different datasets. Isty denotes the intact signature patches used as the style reference, which is input to the visual encoder. Icor denotes the corrupted signature patch. Iorg denotes the clean version of Icor , which is also the reconstruction target of the model. Igen denotes the reconstructed signature patches.
Fig. 9 : Visualization of false rejection and false acceptance examples of SigDiffuser.