Evaluating the quality of process models discovered from event logs is a fundamental challenge in process mining. Traditional evaluation relies on conformance metrics such as fitness and precision, which are highly sensitive to noise and deviations in event logs and therefore often fail to give a consistent assessment of discovery algorithms. We introduce a model-centric evaluation framework that combines node embeddings with Procrustes alignment to enable cross-graph comparability, since embeddings derived from different graphs reside in distinct vector spaces. The proposed PetriNetGraph method transforms Petri net process models into graph representations compatible with embedding algorithms. Comparing the node embeddings of ground-truth and discovered models through a parameterized cosine distance yields a unified and bounded metric for evaluating process discovery across datasets and methods. We evaluate the framework under static and context-aware scenarios to examine how prior knowledge of activity existence influences the assessment of structural similarity. The framework also integrates a compliance checking mechanism that localizes deviations at the activity level and an entropy-regularized node distance aggregation function that quantifies global deviations across the model. Experiments on nine publicly available event logs show that the approach provides a reliable embedding-based conformance indicator that reflects the behavior of fitness and precision. Experiments on synthetic logs with controlled noise levels confirm that it is substantially more robust to log deviations than traditional conformance metrics, PetriNet2Vec, and its dynamic-window and Procrustes-aligned variants, while remaining sensitive to process model degradation. All implementation details and datasets are publicly available.
Figures & tables
Figure 1Figure 2
Figure 2: Detailed overview of embedding-based distance for process model discovery method evaluation
Figure 3: Static process model conformance checking results vs. cosine distance
Figure 4: 3D components by PCA Heuristics Miner on Road Traffic Fines dataset
Figure 5: Context-aware process model conformance checking results vs. cosine distance
Figure 6: Discovered process model for Road Traffic Fines dataset
Dataset
Method
Simplicity
BPIC_2020_Request_for_Payment
ILP Miner
0.245
Inductive Miner
0.556
Road Traffic
ILP Miner
0.245
Inductive Miner
0.636
BPIC_2012
Heuristics Miner
0.654
Split Miner
0.818
Table 1: Simplicity metrics for process models discovered using different methods across datasets. Higher values indicate more readable models. Note: Simplicity metric as defined in [ 36 ] .
Figure 7: 3D Components by PCA ILP miner on 2020 Domestic Declaration
Figure 8: Distance and conformance metrics evaluated on fixed models across varying noise levels in the event logs.
Metric
CV @ ( ε↓ in Lε )
CV @ ( ε↑ in Lε )
CV @ ( ε↑ in Mε )
Fitness
0.000000
0.237964
0.000000
Precision
0.124164
0.121690
0.126832
PetriNet2Vec [ 9 ]
0.120372
0.227511
0.105085
PetriNet2Vec+PA
0.109452
0.162488
0.256595
PetriNet2Vec+DW
0.095438
0.096008
0.524864
Our Work
0.077518
0.074094
0.678094
Table 2: Comparison of coefficient of variation (CV) values under different noise conditions in Lε and Mε . (PA: Procrustes Alignment, DW: Dynamic Window)
Figure 9: Evaluation of distance and conformance metrics using noise-free event logs with process models discovered under varying noise levels. ( Mε )
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset / Method
λ
DWeighted
# Matched
# Unmatched
Total residual
Per-anchor
bpic_2012
GNN Miner
0
0.00766
12
14
6.206
0.517
0.5
0.11635
12
14
6.206
0.517
1
0.12079
12
14
6.206
0.517
10
0.12468
12
14
6.206
0.517
inf
0.12510
12
14
6.206
0.517
Appendix
Table 3: Context-aware model distance of node embeddings, grouped by dataset and method, with varying λ .
Dataset / Method
λ
DWeighted
# Matched
# Unmatched
Total residual
Per-anchor
bpic_2012
GNN Miner
0
0.00942
12
14
5.431
0.453
0.5
0.12431
12
14
5.431
0.453
1
0.12893
12
14
5.431
0.453
10
0.13306
12
14
5.431
0.453
inf
0.13351
12
14
5.431
0.453
Appendix
Table 4: Static model distance of node embeddings, grouped by dataset and method, with varying λ .
Figure 10: Context-aware process model conformance checking results vs. cosine distance
Figure 11: Static process model conformance checking results vs. cosine distance
Scenario
d
# Matched
# Unmatched
Total residual
Per-anchor residual
context_aware
12
12
41
4.843588
0.403632
context_aware
32
12
41
4.048280
0.337357
context_aware
64
12
41
4.909243
0.409104
context_aware
128
12
41
4.534481
0.377873
static
12
11
42
4.653875
0.423080
static
32
11
42
4.713240
0.428476
Appendix
Table 5: Parameter sensitivity analysis on BPIC_2020_Permit_log using ILP Miner, reported for different embedding dimensions d .
Table 18
Context-aware
Static
min_count
Trace pairs
Truth pairs
Trace pairs
Truth pairs
20
1,365,674
1,944,968
2,527,037
2,427,754
30
1,368,732
1,943,188
2,530,069
2,427,626
50
1,364,554
1,946,840
2,524,866
2,428,312
100
1,356,886
1,940,308
2,527,260
2,430,278
150
1,363,682
1,939,820
2,528,345
2,427,590
Appendix
Table 8: Average number of Word2Vec embedding pairs across min_count values (Sepsis).
Figure 12: Wall-clock runtime breakdown across the nine benchmark datasets for the two evaluation scenarios
Figure 13: Runtime scaling behavior of the proposed framework