What Comes Next and Why: Interpretable Next-Event Prediction with Neuro-Symbolic Rules

Tim Nico Bauerschmidt; Isabel Valera; Jilles Vreeken

Figure 4: Illustration of temperature annealing for different temperature-scaled functions. (a) The sigmoid function

\sigma(x/\tau)

converges to the Heaviside step function as

\tau\to 0^{+}

. The dashed curve shows a version with temperature-dependent bias:

\sigma(x/\tau-\log(\tau))

as used in Equation 7 . (b) The soft interval membership function

\pi(x;a,b,\tau)

converges to the hard interval function

\mathbb{I}(x\in[a,b])

, shown here with

a=2

and

b=6

. (c) The temperature-scaled softplus function

\tau\cdot\log(1+\exp(x/\tau))

converges to the ReLU function

\max(0,x)

.

Figure 5: Example symbolic rule set derived from the weights of Rules . Black highlights retained weights after discretization; gray marks discarded weights. Top table : Event-time predicate neurons encoding

(\textit{computer case},[3,15])

,

(\textit{computer case},[1,43])

,

(\textit{computer processor},[2,21])

, and

(\textit{power supply unit},[4,9])

. Bottom table : (Post-ReLU) conjunction weights ( left ) and mean-centered output weights ( right ). Positive conjunction weights indicate the inclusion of a predicate in the antecedent; for example, antecedent neuron 2 combines predicate neuron 2

(\textit{computer case},[1,43])

and predicate neuron 3

(\textit{power supply unit},[2,21])

. Bottom : Symbolic rule set discretized with

\theta_{\mathrm{var}}=0.9

,

\theta_{\mathrm{quant}}=0.875

, and

\theta_{\mathrm{inc}}=0.9

. The rule of antecedent neuron 3 is discarded because its maximum logit is not in the top

(1-87.5)

% (top 2); it also fails to meet the variation threshold

\theta_{\mathrm{var}}=0.9

. The rule of neuron 2 includes two consequent events because its highest logit (headphones) is in the top 2, and its second-highest (phone case) satisfies the inclusion threshold

\theta_{\mathrm{inc}}

.

Figure 6: Per-batch computation times for Rules and the transformer across varying sequence lengths

L

and vocabulary sizes

D

. Rules (S) denotes the model with symbolic (boolean) hidden layers. We report mean and one standard deviation over 32 batches of size 32; variability is small, so error bars are barely visible. Both models are largely insensitive to the vocabulary size. For larger

L

, the expected quadratic scaling of Rules becomes visible, while the symbolic variant achieves substantially faster inference.

Figure 7: Different models using symbolic rules (denoted by

\Rightarrow

) to predict the next event

x_{4}

. Markov models base predictions on empirical event counts following a fixed-order context and do not capture time information beyond order. Decision trees define prediction rules through split conditions and leaf weights. While they can incorporate time features, these are not explicitly tied to specific events (e.g.,

x_{2}

vs.

t_{1}

), and splits for categorical variables often rely on subset partitions (e.g.,

x_{2}\in\left\{{\color[rgb]{0,0.447,0.698}a},\text{{\color[rgb]{0.96,0.624,0}b}}\right\}

). In contrast, Rules learns compact neuro-symbolic rules that directly associate specific events with delay conditions (e.g.,

x_{2}

and

t_{2}

), enabling transparent and temporally grounded predictions.

Datasets	$N$	$L_{\mathrm{tot}}$	$L_{\mathrm{min}}$	$L_{\mathrm{max}}$	$D$	Regular
Holidays	1	1200	–	–	14	✗
Chorales	100	4693	26	86	20	✗
Commits	1	28879	–	–	119	$\checkmark$
Rolling Mill	1000	53788	24	142	556	$\checkmark$
Amazon	4909	1760607	10	5057	993	✗

Appendix

Table 3: Statistics of the real-world datasets. Columns show: Number of sequences (

N

), total number of events (

L_{\mathrm{tot}}

), minimum and maximum sequence lengths (

L_{\mathrm{min}}

and

L_{\mathrm{max}}

), vocabulary size (

D

), and whether sequences are regularly sampled.

Datasets	$I$	$J$	$\omega$	Batch Size	Warm-up	$\theta_{\mathrm{quant}}$	$\theta_{\mathrm{inc}}$
Holidays	200	45	365	64	100	0.85	0.85
Chorales	200	64	48	30	300	0.85	0.85
Commits	300	100	24	64	60	0.95	0.90
Rolling Mill	500	200	64	64	100	0.90	0.90
Amazon	500	200	20	64	200	0.80	0.90

Appendix

Table 4: Selected hyperparameters of Rules on real-world datasets. Columns show:

I

(number of temporal predicate neurons),

J

(number of antecedent neurons),

\omega

(maximum lookback window), batch size, warm-up epochs,

\theta_{\mathrm{quant}}

(quantile threshold), and

\theta_{\mathrm{inc}}

(inclusion threshold).

Figure 8: Rules improves with an increasing number of training samples

N

. We show the mean results over 10 rule sets and corresponding datasets per synthetic setting. Error bars mark the empirical standard deviation. (a) Test accuracy compared to baseline models. (b) Similarity between the rule set

\mathbf{R}

discovered by Rules and the ground truth

\mathbf{R}^{*}

.

Figure 9: Rules performs well on synthetic data for varying vocabulary sizes

D

. We show the mean results over 10 rule sets and corresponding datasets per synthetic setting. Error bars mark the empirical standard deviation. (a) Test accuracy compared to baseline models. (b) Similarity between the rule set

\mathbf{R}

discovered by Rules and the ground truth

\mathbf{R}^{*}

.

Figure 10: Extended results on synthetic data for varying signal-to-noise parameters

(r_{\mathrm{targ}},r_{\mathrm{cons}})

. We show the mean results over 10 rule sets and corresponding datasets per synthetic setting. Error bars mark the empirical standard deviation. (a) Test accuracy compared to baseline models. (b) Similarity between the rule set

\mathbf{R}

discovered by Rules and the ground truth

\mathbf{R}^{*}

.

Datasets	Rules	Runes	PPM	CatBoost	XGBoost	Transformer
Holidays	0.053	0.000	0.022	0.000	0.001	0.004
Chorales	2.404	1.897	1.993	1.967	1.880	1.820
Commits	4.161	4.212	5.459	4.200	4.547	4.330
Rolling Mill	2.581	1.174	1.321	1.399	1.247	1.072
Amazon	5.799	4.829	5.318	5.429	5.270	4.817
Avg. Rank	5.0	2.2	4.8	3.4	3.2	2.2

Appendix

Table 5: Test cross-entropy on real-world datasets (lower is better). The best results are in bold , the second best are underlined .

Datasets	$\left\lvert\mathbf{R}\right\rvert$	$\left\lvert A\right\rvert$	Acc. $\uparrow$	$\left\lvert\mathbf{R}\right\rvert$	$\left\lvert A\right\rvert$	Acc. $\uparrow$	$\left\lvert\mathbf{R}\right\rvert$	$\left\lvert A\right\rvert$	Acc. $\uparrow$	$\left\lvert\mathbf{R}\right\rvert$	$\left\lvert A\right\rvert$	Acc. $\uparrow$
	Rules			PPM			CatBoost			XGBoost
Holidays	26	1.2	97.57	14	1.0	100.00	1129	4.0	100.00	1455	3.2	100.00
Chorales	33	4.2	21.00	180	1.9	31.40	11596	6.0	37.10	34462	3.7	37.72
Commits	51	2.6	19.60	80	1.0	18.11	380	6.0	15.55	83965	3.8	17.03
Rolling Mill	77	4.1	34.13	9583	2.6	67.64	13809	6.0	65.64	640723	3.4	68.84
Amazon	42	1.7	6.53	993	1.0	15.16	13123	6.0	11.79	5149683	4.9	15.37

Appendix

Table 6: Number of rules (

\left\lvert\mathbf{R}\right\rvert

), average antecedent size (

\left\lvert A\right\rvert

), and test accuracy (Acc.) in % on real-world datasets for models that can be fully decomposed into symbolic rules. Rules consistently produces compact rule sets with small antecedents.

Table 7: Influence of the Residual Network for Individual Datasets. Counts are normalized within the rule activity groups. The bottom rows show the fraction of predictions that fall within the respective groups and the total number of testing events.

Figure 11: Entropy of the output distribution of Rules (x-axis) versus the symmetric KL divergence to Runes (y-axis) on test splits of real-world datasets, considering events with at least one active rule. Plots are grouped by prediction outcome: both models correct (✓✓), only Runes correct (correct change, ✗

\to

✓), only Rules correct (incorrect change, ✓

\to

✗), both incorrect (✗✗) with the same prediction, and both incorrect with different predictions (✗

\to

✗). For clarity, density plots are shown for Rolling Mill and Amazon. Subcaptions report the total number of events per dataset, and the legends indicate the fraction corresponding to each case.

	Before			After			Difference
Datasets	Acc.	Avg. $\left\lvert A\right\rvert$	Avg. $\left\lvert\Delta_{i}\right\rvert$	Acc.	Avg. $\left\lvert A\right\rvert$	Avg. $\left\lvert\Delta_{i}\right\rvert$	Acc.	Avg. $\left\lvert A\right\rvert$	Avg. $\left\lvert\Delta_{i}\right\rvert$
Holidays	96.12	3.1 $\pm$ 1.8	107.6 $\pm$ 51.9	97.57	1.2 $\pm$ 0.4	8.5 $\pm$ 14.0	$+$ 1.45	$-$ 1.99	$-$ 99.03
Chorales	22.22	6.5 $\pm$ 3.6	16.7 $\pm$ 0 7.5	21.00	4.2 $\pm$ 2.3	5.1 $\pm$ 0 6.3	$-$ 1.22	$-$ 2.28	$-$ 11.59
Commits	11.72	3.2 $\pm$ 2.2	6.9 $\pm$ 0 4.9	19.60	2.6 $\pm$ 1.8	2.8 $\pm$ 0 3.4	$+$ 7.88	$-$ 0.54	$-$ 0 4.13
Rolling Mill	43.10	7.0 $\pm$ 4.0	25.2 $\pm$ 0 9.7	34.13	4.1 $\pm$ 2.2	4.1 $\pm$ 0 4.9	$-$ 8.97	$-$ 2.95	$-$ 21.04
Amazon	7.55	1.7 $\pm$ 0.9	6.2 $\pm$ 0 6.7	6.53	1.7 $\pm$ 0.9	3.3 $\pm$ 0 5.3	$-$ 1.02	$-$ 0.00	$-$ 0 2.88

Appendix

Table 8: Symbolic processing of rules extracted from Rules reduces antecedent complexity. The table shows Rules before and after symbolic post-processing of rules (Appendices B.2 – B.4 ). It reports test accuracy for next-event prediction (Acc.) in %, average antecedent size (

\left\lvert A\right\rvert

), and average delay interval width (

\left\lvert\Delta_{i}\right\rvert=\left\lvert[\delta^{\mathrm{min}},\delta^{\mathrm{max}}]\right\rvert

) across rules. Standard deviations for antecedent sizes and delay intervals are included. The final column shows the absolute change after post-processing. Post-processing accuracy refers to Rules after embedding the processed antecedents into its hidden layers and retraining only the output layer.

Figure 12: Symbolic processing of rules extracted from Rules reduces the number of active rules per prediction. The figure shows histograms of the number of active rules per next-event prediction on the test split of real-world datasets.

	Rules (no refinement)	Rules	Runes	Transformer
Holidays	0.147 $\pm$ 0.082	0.054 $\pm$ 0.001	0.000 $\pm$ 0.000	0.005 $\pm$ 0.000
Chorales	2.627 $\pm$ 0.065	2.404 $\pm$ 0.002	1.931 $\pm$ 0.029	1.841 $\pm$ 0.024
Commits	8.107 $\pm$ 0.671	4.186 $\pm$ 0.018	4.191 $\pm$ 0.017	4.250 $\pm$ 0.073
Rolling Mill	2.468 $\pm$ 0.104	2.580 $\pm$ 0.001	1.183 $\pm$ 0.010	1.069 $\pm$ 0.003
Amazon	5.811 $\pm$ 0.045	5.801 $\pm$ 0.002	4.850 $\pm$ 0.016	4.834 $\pm$ 0.015

Appendix

Table 9: Mean and standard deviation of the test cross-entropy (lower is better) across training stages of Runes on real-world datasets. “No refinement” refers to Rules without symbolically processed rules.

	Rules (no refinement)	Rules	Runes	Transformer
Holidays	90.98 $\pm$ 4.05	97.57 $\pm$ 0.00	100.00 $\pm$ 0.00	100.00 $\pm$ 0.00
Chorales	21.41 $\pm$ 0.59	21.15 $\pm$ 0.10	37.16 $\pm$ 0.72	39.17 $\pm$ 0.49
Commits	12.85 $\pm$ 1.27	17.20 $\pm$ 1.60	17.07 $\pm$ 0.99	20.58 $\pm$ 0.85
Rolling Mill	37.79 $\pm$ 3.74	34.13 $\pm$ 0.00	70.20 $\pm$ 0.34	71.49 $\pm$ 0.33
Amazon	6.72 $\pm$ 0.65	6.52 $\pm$ 0.04	17.36 $\pm$ 0.14	17.49 $\pm$ 0.10

Appendix

Table 10: Mean and standard deviation of the test accuracy in % (higher is better) across training stages of Runes on real-world datasets. “No refinement” refers to Rules without symbolically processed rules.

	Rules		Sqs $\to$ Rules
Datasets	NLL $\downarrow$	Acc. $\uparrow$	NLL $\downarrow$	Acc. $\uparrow$
Holidays	0.053	97.57	0.643	71.29
Chorales	2.404	21.00	2.518	12.95
Commits	4.161	19.60	4.344	13.67
Rolling Mill	2.581	34.13	1.987	54.21

Appendix

Table 11: Comparison of externally mined rules and end-to-end learning. We report test cross-entropy (NLL) and accuracy (Acc.) in %. Sqs

\to

Rules denotes Rules with antecedents mined via Sqs .

	Runes		Runes (independent)
Datasets	NLL $\downarrow$	Acc. $\uparrow$	NLL $\downarrow$	Acc. $\uparrow$
Holidays	0.000 $\pm$ 0.000	100.00 $\pm$ 0.00	0.000 $\pm$ 0.000	100.00 $\pm$ 0.00
Chorales	1.931 $\pm$ 0.029	37.16 $\pm$ 0.72	2.234 $\pm$ 0.035	33.69 $\pm$ 0.61
Commits	4.191 $\pm$ 0.017	17.07 $\pm$ 0.99	5.655 $\pm$ 0.124	20.71 $\pm$ 1.12
Rolling Mill	1.183 $\pm$ 0.010	70.20 $\pm$ 0.34	1.386 $\pm$ 0.004	70.00 $\pm$ 0.30
Amazon	4.850 $\pm$ 0.016	17.36 $\pm$ 0.14	5.760 $\pm$ 0.015	15.05 $\pm$ 0.05

Appendix

Table 12: Runes benefits from training the residual network with added rule logits. We report mean and standard deviation of test cross-entropy (NLL) and accuracy (Acc.) in % on real-world datasets over 4 random seeds for residual network training. “Independent” denotes Runes with independently trained rule and residual networks. Best results within one standard deviation are in bold .

Lookback Window	Rules	CatBoost	XGBoost	Transformer
12	18.32	17.81	16.68	19.71
24	19.60	15.55	17.03	20.10
36	13.65	15.46	16.26	20.23
48	12.68	17.34	16.24	20.14

Appendix

Table 13: Test accuracy in % for different lookback windows

\omega

on Commits.

What Comes Next and Why: Interpretable Next-Event Prediction with Neuro-Symbolic Rules

Abstract

Figures & tables

Appendix

Next-Event Prediction

Knowledge, Rules and Their Embeddings: Two Paths towards Neuro-Symbolic JEPA

Where Does Neural Advantage Arise in Continuous-Time Dynamic Graph Prediction?

Neuro-Symbolic Learning for Predictive Process Monitoring via Two-Stage Logic Tensor Networks with Rule Pruning

Abstract

Next-Event Prediction

Explore similar work

Knowledge, Rules and Their Embeddings: Two Paths towards Neuro-Symbolic JEPA

Where Does Neural Advantage Arise in Continuous-Time Dynamic Graph Prediction?

Neuro-Symbolic Learning for Predictive Process Monitoring via Two-Stage Logic Tensor Networks with Rule Pruning