In real-world applications, being able to explain why a prediction is made is often as important - if not more so - than achieving high accuracy. However, existing methods for sequential event prediction typically prioritize accuracy at the expense of interpretability. Neuro-symbolic approaches offer a promising direction to improve interpretability by learning symbolic rules, but extending them to next-event prediction is challenging due to the presence of temporal dependencies. We introduce RULES, a neuro-symbolic model that learns sequential rules directly from data, and combine it with a residual network to form RUNES (Rule Network for Event Sequences). RULES accurately recovers ground truth rules from synthetic data and discovers compact, interpretable rule sets on real data. RUNES combines these rules additively with a residual network, achieving competitive accuracy while providing transparent access to active rules and how residuals adjust predictions.
Figures & tables
Figure 1: Architecture overview of Runes . (a) Runes is trained in two phases: Rules first learns sequential rules, after which a residual network is trained with the Rules network frozen. (b) Rules represents rules in its layers: Neurons in the first layer encode event-time predicates. Neurons in the second layer select a subset of these predicates and combine them into conjunctions (antecedents). The output layer sums the softmax logits of all active rules to produce the final prediction.
Figure 2: Rules reliably recovers predictive rules from synthetic data across varying signal-to-noise ratios. Results are averaged over 10 rule sets and corresponding datasets per setting; error bars indicate one empirical standard deviation. (a) Test accuracy compared to baselines; Rules∗ denotes Rules with embedded ground-truth antecedents. (b) Similarity between the rule set R discovered by Rules and the ground truth R∗ .
Rules
Runes
PPM
CatBoost
XGBoost
Transformer
Datasets
∣R∣
Acc.
Acc.
∣R∣
Acc.
∣R∣
Acc.
∣R∣
Acc.
Acc.
Holidays
26
97.57
100.00
14
100.00
1129
100.00
1455
100.00
100.00
Chorales
33
21.00
37.00
180
31.40
11596
37.10
34462
37.72
39.35
Commits
51
19.60
18.31
80
18.11
380
15.55
83965
17.03
20.10
Rolling Mill
77
34.13
70.44
9583
67.64
13809
65.64
640723
68.84
71.24
Amazon
42
6.53
17.55
993
15.16
13123
11.79
5149683
15.37
17.61
Table 1: Test accuracy (Acc.) in % and number of rules ( ∣R∣ ) on real-world datasets. Best accuracy results are in bold , second best are underlined . Runes performs close to the black-box transformer.
Figure 4
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Illustration of temperature annealing for different temperature-scaled functions. (a) The sigmoid function σ(x/τ) converges to the Heaviside step function as τ→0+ . The dashed curve shows a version with temperature-dependent bias: σ(x/τ−log(τ)) as used in Equation 7 . (b) The soft interval membership function π(x;a,b,τ) converges to the hard interval function I(x∈[a,b]) , shown here with a=2 and b=6 . (c) The temperature-scaled softplus function τ⋅log(1+exp(x/τ)) converges to the ReLU function max(0,x) .
Figure 5: Example symbolic rule set derived from the weights of Rules . Black highlights retained weights after discretization; gray marks discarded weights. Top table : Event-time predicate neurons encoding (computer case,[3,15]) , (computer case,[1,43]) , (computer processor,[2,21]) , and (power supply unit,[4,9]) . Bottom table : (Post-ReLU) conjunction weights ( left ) and mean-centered output weights ( right ). Positive conjunction weights indicate the inclusion of a predicate in the antecedent; for example, antecedent neuron 2 combines predicate neuron 2 (computer case,[1,43]) and predicate neuron 3 (power supply unit,[2,21]) . Bottom : Symbolic rule set discretized with θvar=0.9 , θquant=0.875 , and θinc=0.9 . The rule of antecedent neuron 3 is discarded because its maximum logit is not in the top (1−87.5) % (top 2); it also fails to meet the variation threshold θvar=0.9 . The rule of neuron 2 includes two consequent events because its highest logit (headphones) is in the top 2, and its second-highest (phone case) satisfies the inclusion threshold θinc .
Figure 6: Per-batch computation times for Rules and the transformer across varying sequence lengths L and vocabulary sizes D . Rules (S) denotes the model with symbolic (boolean) hidden layers. We report mean and one standard deviation over 32 batches of size 32; variability is small, so error bars are barely visible. Both models are largely insensitive to the vocabulary size. For larger L , the expected quadratic scaling of Rules becomes visible, while the symbolic variant achieves substantially faster inference.
Figure 7: Different models using symbolic rules (denoted by ⇒ ) to predict the next event x4 . Markov models base predictions on empirical event counts following a fixed-order context and do not capture time information beyond order. Decision trees define prediction rules through split conditions and leaf weights. While they can incorporate time features, these are not explicitly tied to specific events (e.g., x2 vs. t1 ), and splits for categorical variables often rely on subset partitions (e.g., x_{2}\in\left\{{\color[rgb]{0,0.447,0.698}a},\text{{\color[rgb]{0.96,0.624,0}b}}\right\} ). In contrast, Rules learns compact neuro-symbolic rules that directly associate specific events with delay conditions (e.g., x2 and t2 ), enabling transparent and temporally grounded predictions.
Datasets
N
Ltot
Lmin
Lmax
D
Regular
Holidays
1
1200
–
–
14
✗
Chorales
100
4693
26
86
20
✗
Commits
1
28879
–
–
119
✓
Rolling Mill
1000
53788
24
142
556
✓
Amazon
4909
1760607
10
5057
993
✗
Appendix
Table 3: Statistics of the real-world datasets. Columns show: Number of sequences ( N ), total number of events ( Ltot ), minimum and maximum sequence lengths ( Lmin and Lmax ), vocabulary size ( D ), and whether sequences are regularly sampled.
Datasets
I
J
ω
Batch Size
Warm-up
θquant
θinc
Holidays
200
45
365
64
100
0.85
0.85
Chorales
200
64
48
30
300
0.85
0.85
Commits
300
100
24
64
60
0.95
0.90
Rolling Mill
500
200
64
64
100
0.90
0.90
Amazon
500
200
20
64
200
0.80
0.90
Appendix
Table 4: Selected hyperparameters of Rules on real-world datasets. Columns show: I (number of temporal predicate neurons), J (number of antecedent neurons), ω (maximum lookback window), batch size, warm-up epochs, θquant (quantile threshold), and θinc (inclusion threshold).
Figure 8: Rules improves with an increasing number of training samples N . We show the mean results over 10 rule sets and corresponding datasets per synthetic setting. Error bars mark the empirical standard deviation. (a) Test accuracy compared to baseline models. (b) Similarity between the rule set R discovered by Rules and the ground truth R∗ .
Figure 9: Rules performs well on synthetic data for varying vocabulary sizes D . We show the mean results over 10 rule sets and corresponding datasets per synthetic setting. Error bars mark the empirical standard deviation. (a) Test accuracy compared to baseline models. (b) Similarity between the rule set R discovered by Rules and the ground truth R∗ .
Figure 10: Extended results on synthetic data for varying signal-to-noise parameters (rtarg,rcons) . We show the mean results over 10 rule sets and corresponding datasets per synthetic setting. Error bars mark the empirical standard deviation. (a) Test accuracy compared to baseline models. (b) Similarity between the rule set R discovered by Rules and the ground truth R∗ .
Datasets
Rules
Runes
PPM
CatBoost
XGBoost
Transformer
Holidays
0.053
0.000
0.022
0.000
0.001
0.004
Chorales
2.404
1.897
1.993
1.967
1.880
1.820
Commits
4.161
4.212
5.459
4.200
4.547
4.330
Rolling Mill
2.581
1.174
1.321
1.399
1.247
1.072
Amazon
5.799
4.829
5.318
5.429
5.270
4.817
Avg. Rank
5.0
2.2
4.8
3.4
3.2
2.2
Appendix
Table 5: Test cross-entropy on real-world datasets (lower is better). The best results are in bold , the second best are underlined .
Rules
PPM
CatBoost
XGBoost
Datasets
∣R∣
∣A∣
Acc. ↑
∣R∣
∣A∣
Acc. ↑
∣R∣
∣A∣
Acc. ↑
∣R∣
∣A∣
Acc. ↑
Holidays
26
1.2
97.57
14
1.0
100.00
1129
4.0
100.00
1455
3.2
100.00
Chorales
33
4.2
21.00
180
1.9
31.40
11596
6.0
37.10
34462
3.7
37.72
Commits
51
2.6
19.60
80
1.0
18.11
380
6.0
15.55
83965
3.8
17.03
Rolling Mill
77
4.1
34.13
9583
2.6
67.64
13809
6.0
65.64
640723
3.4
68.84
Amazon
42
1.7
6.53
993
1.0
15.16
13123
6.0
11.79
5149683
4.9
15.37
Appendix
Table 6: Number of rules ( ∣R∣ ), average antecedent size ( ∣A∣ ), and test accuracy (Acc.) in % on real-world datasets for models that can be fully decomposed into symbolic rules. Rules consistently produces compact rule sets with small antecedents.
Table 7: Influence of the Residual Network for Individual Datasets. Counts are normalized within the rule activity groups. The bottom rows show the fraction of predictions that fall within the respective groups and the total number of testing events.
Figure 11: Entropy of the output distribution of Rules (x-axis) versus the symmetric KL divergence to Runes (y-axis) on test splits of real-world datasets, considering events with at least one active rule. Plots are grouped by prediction outcome: both models correct (✓✓), only Runes correct (correct change, ✗ → ✓), only Rules correct (incorrect change, ✓ → ✗), both incorrect (✗✗) with the same prediction, and both incorrect with different predictions (✗ → ✗). For clarity, density plots are shown for Rolling Mill and Amazon. Subcaptions report the total number of events per dataset, and the legends indicate the fraction corresponding to each case.
Before
After
Difference
Datasets
Acc.
Avg. ∣A∣
Avg. ∣Δi∣
Acc.
Avg. ∣A∣
Avg. ∣Δi∣
Acc.
Avg. ∣A∣
Avg. ∣Δi∣
Holidays
96.12
3.1 ± 1.8
107.6 ± 51.9
97.57
1.2 ± 0.4
8.5 ± 14.0
+ 1.45
− 1.99
− 99.03
Chorales
22.22
6.5 ± 3.6
16.7 ± 0 7.5
21.00
4.2 ± 2.3
5.1 ± 0 6.3
− 1.22
− 2.28
− 11.59
Commits
11.72
3.2 ± 2.2
6.9 ± 0 4.9
19.60
2.6 ± 1.8
2.8 ± 0 3.4
+ 7.88
− 0.54
− 0 4.13
Rolling Mill
43.10
7.0 ± 4.0
25.2 ± 0 9.7
34.13
4.1 ± 2.2
4.1 ± 0 4.9
− 8.97
− 2.95
− 21.04
Amazon
7.55
1.7 ± 0.9
6.2 ± 0 6.7
6.53
1.7 ± 0.9
3.3 ± 0 5.3
− 1.02
− 0.00
− 0 2.88
Appendix
Table 8: Symbolic processing of rules extracted from Rules reduces antecedent complexity. The table shows Rules before and after symbolic post-processing of rules (Appendices B.2 – B.4 ). It reports test accuracy for next-event prediction (Acc.) in %, average antecedent size ( ∣A∣ ), and average delay interval width ( ∣Δi∣=[δmin,δmax] ) across rules. Standard deviations for antecedent sizes and delay intervals are included. The final column shows the absolute change after post-processing. Post-processing accuracy refers to Rules after embedding the processed antecedents into its hidden layers and retraining only the output layer.
Figure 12: Symbolic processing of rules extracted from Rules reduces the number of active rules per prediction. The figure shows histograms of the number of active rules per next-event prediction on the test split of real-world datasets.
Rules (no refinement)
Rules
Runes
Transformer
Holidays
0.147 ± 0.082
0.054 ± 0.001
0.000 ± 0.000
0.005 ± 0.000
Chorales
2.627 ± 0.065
2.404 ± 0.002
1.931 ± 0.029
1.841 ± 0.024
Commits
8.107 ± 0.671
4.186 ± 0.018
4.191 ± 0.017
4.250 ± 0.073
Rolling Mill
2.468 ± 0.104
2.580 ± 0.001
1.183 ± 0.010
1.069 ± 0.003
Amazon
5.811 ± 0.045
5.801 ± 0.002
4.850 ± 0.016
4.834 ± 0.015
Appendix
Table 9: Mean and standard deviation of the test cross-entropy (lower is better) across training stages of Runes on real-world datasets. “No refinement” refers to Rules without symbolically processed rules.
Rules (no refinement)
Rules
Runes
Transformer
Holidays
90.98 ± 4.05
97.57 ± 0.00
100.00 ± 0.00
100.00 ± 0.00
Chorales
21.41 ± 0.59
21.15 ± 0.10
37.16 ± 0.72
39.17 ± 0.49
Commits
12.85 ± 1.27
17.20 ± 1.60
17.07 ± 0.99
20.58 ± 0.85
Rolling Mill
37.79 ± 3.74
34.13 ± 0.00
70.20 ± 0.34
71.49 ± 0.33
Amazon
6.72 ± 0.65
6.52 ± 0.04
17.36 ± 0.14
17.49 ± 0.10
Appendix
Table 10: Mean and standard deviation of the test accuracy in % (higher is better) across training stages of Runes on real-world datasets. “No refinement” refers to Rules without symbolically processed rules.
Rules
Sqs → Rules
Datasets
NLL ↓
Acc. ↑
NLL ↓
Acc. ↑
Holidays
0.053
97.57
0.643
71.29
Chorales
2.404
21.00
2.518
12.95
Commits
4.161
19.60
4.344
13.67
Rolling Mill
2.581
34.13
1.987
54.21
Appendix
Table 11: Comparison of externally mined rules and end-to-end learning. We report test cross-entropy (NLL) and accuracy (Acc.) in %. Sqs → Rules denotes Rules with antecedents mined via Sqs .
Runes
Runes (independent)
Datasets
NLL ↓
Acc. ↑
NLL ↓
Acc. ↑
Holidays
0.000 ± 0.000
100.00 ± 0.00
0.000 ± 0.000
100.00 ± 0.00
Chorales
1.931 ± 0.029
37.16 ± 0.72
2.234 ± 0.035
33.69 ± 0.61
Commits
4.191 ± 0.017
17.07 ± 0.99
5.655 ± 0.124
20.71 ± 1.12
Rolling Mill
1.183 ± 0.010
70.20 ± 0.34
1.386 ± 0.004
70.00 ± 0.30
Amazon
4.850 ± 0.016
17.36 ± 0.14
5.760 ± 0.015
15.05 ± 0.05
Appendix
Table 12: Runes benefits from training the residual network with added rule logits. We report mean and standard deviation of test cross-entropy (NLL) and accuracy (Acc.) in % on real-world datasets over 4 random seeds for residual network training. “Independent” denotes Runes with independently trained rule and residual networks. Best results within one standard deviation are in bold .
Lookback Window
Rules
CatBoost
XGBoost
Transformer
12
18.32
17.81
16.68
19.71
24
19.60
15.55
17.03
20.10
36
13.65
15.46
16.26
20.23
48
12.68
17.34
16.24
20.14
Appendix
Table 13: Test accuracy in % for different lookback windows ω on Commits.
Modern self-supervised predictive architectures excel at capturing complex statistical correlations from high-dimensional data but lack mechanisms to internalize verifiable human logic, leaving them susceptible to spurious correlations and shortcut learning. Conversely, traditional rule-based inference systems offer rigorous, interpretable logic but suffer from discrete boundaries and NP-hard combinatorial explosion. To bridge this divide, we propose a bidirectional neuro-symbolic framework centered around Rule-informed Joint-Embedding Predictive Architectures (RiJEPA). In the first direction, we inject structured inductive biases into JEPA training via Energy-Based Constraints (EBC) and a multi-modal dual-encoder architecture. This fundamentally reshapes the representation manifold, replacing arbitrary statistical correlations with geometrically sound logical basins. In the second direction, we demonstrate that by relaxing rigid, discrete symbolic rules into a continuous, differentiable logic, we can bypass traditional combinatorial search for new rule generation. By leveraging gradient-guided Langevin diffusion within the rule energy landscape, we introduce novel paradigms for continuous rule discovery, which enable unconditional joint generation, conditional forward and abductive inference, and marginal predictive translation. Empirical evaluations on both synthetic topological simulations and a high-stakes clinical use case confirm the efficacy of our approach. Ultimately, this framework establishes a powerful foundation for robust, generative, and interpretable neuro-symbolic representation learning.
Aggregate performance on continuous-time dynamic graphs (CTDGs) combines, in a single score, the portion attributable to known temporal regularities and the additional predictive power of neural models. This study separates the two at the query level. We construct a mechanism-constrained predictor that uses pair recurrence, recency and history position, renewal patterns, and short sequential transitions while learning the compatibility within each mechanism. Across four CTDG datasets, this predictor recovers a substantial portion of the performance of strong neural baselines, and the recovered performance quickly saturates with a small, dataset-specific set of explicit mechanisms. Neural residuals concentrate on queries for which the positive and negative candidates have similar mechanism-execution profiles. Allowing conditional interactions among mechanisms is more effective than simply reweighting their existing contributions. Conditioning the contribution of one mechanism on the execution state of another recovers 54.9-73.2% of the original neural-only queries and improves overall paired accuracy on all four datasets. Although the magnitude of the effect varies across datasets, these results show that the performance gap of neural CTDG models need not be treated solely as an opaque difference in representational capacity. At least part of the gap is localized to queries with similar candidate execution profiles and can be functionally explained by conditional coordination among known, low-dimensional mechanisms.
Minwoo Yu, Young-guk Ha
Smart Computing Laboratory, Department of Computer Science & Engineering, Konkuk University, Seoul 05029, Republic of Korea
Predictive modeling on sequential event data is critical for fraud detection and healthcare monitoring. Existing data-driven approaches learn correlations from historical data but fail to incorporate domain-specific sequential constraints and logical rules governing event relationships, limiting accuracy and regulatory compliance. For example, healthcare procedures must follow specific sequences, and financial transactions must adhere to compliance rules. We present a neuro-symbolic approach integrating domain knowledge as differentiable logical constraints using Logic Networks (LTNs). We formalize control-flow, temporal, and payload knowledge using Linear Temporal Logic and first-order logic. Our key contribution is a two-stage optimization strategy addressing LTNs' tendency to satisfy logical formulas at the expense of predictive accuracy. The approach uses weighted axiom loss during pretraining to prioritize data learning, followed by rule pruning that retains only consistent, contributive axioms based on satisfaction dynamics. Evaluation on four real-world event logs shows that domain knowledge injection significantly improves predictive performance, with the two-stage optimization proving essential knowledge (without it, knowledge can severely degrade performance). The approach excels particularly in compliance-constrained scenarios with limited compliant training examples, achieving superior performance compared to purely data-driven baselines while ensuring adherence to domain constraints.
Fabrizio De Santis, Gyunam Park, Francesco Zanichelli
University of Parma, Parma, Italy · Eindhoven University of Technology, Eindhoven, Netherlands