Foundation Models for Event Prediction Across Heterogeneous Event Logs

Fabian Spaeh; Sean Jaffe; Shandian Zhe; Bin Shen

cs.LGOct 13, 2026

Foundation Models for Event Prediction Across Heterogeneous Event Logs

Authors: Fabian Spaeh, Sean Jaffe, Shandian Zhe, Bin Shen

Organizations: Celonis

Abstract

Event logs are ubiquitous across real-world domains, including business processes, manufacturing workflows, healthcare, and many other operational systems. Yet event logs can differ substantially in their event semantics, label spaces, interaction patterns, and temporal scales, making it difficult to build a single foundation model that generalizes directly to previously unseen systems. We address this challenge with the Foundation Event Predictor (FEP), a simple and scalable in-context architecture that performs next-event prediction through learned, event-level nonparametric inference. Causal self-attention encodes context and target histories into key and query representations, respectively, while the observed next context events are encoded as values. Cross-attention then performs learned, relevance-weighted aggregation of next-event outcomes over all context transitions, allowing the entire nonparametric prediction procedure to be expressed as standard attention operations and efficiently parallelized with modern kernels such as FlashAttention. To promote cross-system generalization, we introduce permutation-robust embedding learning and random fixed embeddings to decouple event IDs from persistent semantics, together with empirical quantile transforms and robust normalization to handle diverse temporal scales. We pretrain FEP on 696 real-world event datasets spanning heterogeneous event systems and evaluate it on 50 held-out datasets unseen during pretraining. Across these datasets, FEP generalizes effectively to new event systems without parameter adaptation, frequently approaching the performance of strong models trained separately on each target dataset while substantially outperforming existing foundation-model baselines, particularly on challenging prediction tasks.

Figures & tables

Figure 1: The architecture of Foundation Event Predictor (FEP). Context histories and observed next events form keys and values, respectively, while target histories form queries for cross-attention-based nonparametric prediction.

Table 1: Generalization performance by dataset difficulty, averaged over five runs across 50 test datasets. The best results among local methods and foundation models are highlighted in bold. FEP (Quantile) uses permutation training with the empirical quantile transform, while FEP (Robust log) uses random fixed embeddings with robust log-time normalization; all four combinations are reported in Appendix B . FEP-Local is the architecture-matched local counterpart of FEP (Quantile), trained separately on each target dataset. FIM-PP supports at most 22 activities; we therefore retain the 21 most frequent activities and merge the rest into an “other” activity. FIM-PP (FT) denotes FIM-PP further fine-tuned on the training set of each target dataset. FM-v2 is omitted from NMAE because it predicts remaining case time, while Transformer Hawkes and IFTPP are omitted because their NMAEs exceed

10{,}000

across all difficulty levels.

Method	MAE score	Accuracy score
Transformer	$1644$	$\boldsymbol{1664}$
GRU	$\boldsymbol{1690}$	$1631$
Neural Hawkes	$1518$	$1619$
Transformer Hawkes	$1377$	$1422$
IFTPP	$1169$	$1540$
FEP (Local)	$1641$	$1436$

Table 2: Overall pairwise ratings across 50 test datasets, reported as Bradley–Terry scores (higher is better). Scores are computed jointly from pairwise comparisons among all local, foundation, and fine-tuned models.

Figure 2: Dataset-wise comparison between FEP and the best locally trained model. Colors indicate empirical dataset difficulty, and the dashed line denotes equal performance. The shaded region indicates the side of the diagonal where FEP performs better.

Panel of Figure 3 — Figure 3: Ablation results for context size and model dimension.

Table 3: Ablation results for the number of pretraining datasets and model depth.

\infty

denotes NMAE larger than

10{,}000

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

		Latency (ms)		Queries/s		Peak memory (GiB)
# Context seq.	# Context K/V tokens	Standard	FA2	Standard	FA2	Standard	FA2
8	792	2.118	2.182	472.2	458.2	0.039	0.038
16	1,584	2.124	2.185	470.7	457.6	0.042	0.040
32	3,168	2.120	2.191	471.7	456.5	0.050	0.045
64	6,336	2.112	2.188	473.5	457.0	0.064	0.055
128	12,672	2.121	2.196	471.5	455.5	0.094	0.073

Appendix

Table 4: Inference efficiency and scalability of standard attention and FlashAttention-2 (FA2) for cached-context nonparametric inference. Each context sequence and query trace has length 100, yielding 99 history-to-next-event transitions or predictions. Peak memory includes model parameters, cached context representations, query execution, and output predictions.

# Context seq. (tokens)	Latency reduction	Throughput increase	Memory reduction
2,048 (203K)	19.3%	23.9%	42.1%
4,096 (406K)	27.8%	38.5%	42.8%
8,192 (811K)	34.7%	53.0%	43.3%
12,288 (1.2M)	37.5%	60.0%	43.4%
16,384 (1.6M)	39.1%	64.1%	43.5%

Appendix

Table 5: Relative improvement of FlashAttention-2 over standard attention at large context sizes. Positive values indicate improvement.

Setting	Accuracy (%)	NMAE
Random fixed	$67.77\pm 0.03$	$0.80\pm 0.00$
Learned + permutation	$\boldsymbol{67.95\pm 0.03}$	$\boldsymbol{0.79\pm 0.00}$
Learned, no permutation	$64.14\pm 0.03$	$0.83\pm 0.00$

Appendix

Table 6: Effect of event-ID representation under the ablation protocol.

Table 7: Performance of all four FEP configurations across dataset difficulty levels. The configurations combine two inter-arrival-time transformations (quantile and robust log) with two event-ID representation strategies (permutation training and random fixed embeddings).

Figure 4: Dataset-wise comparison of FEP (Quantile) and FEP-Local on the 50 held-out datasets. Colors denote dataset difficulty, and the dashed line indicates equal performance. Points below the diagonal favor FEP for accuracy, whereas points above the diagonal favor FEP for NMAE. The FEP-favoring region is shaded in both panels.

	Medical Appointments		Beijing Air Quality		NASA SMAP/MSL
Method	Accuracy	NMAE	Accuracy	NMAE	Accuracy	NMAE
Transformer	$\boldsymbol{0.96\pm 0.00}$	$0.30\pm 0.00$	$0.76\pm 0.02$	$\boldsymbol{0.68\pm 0.01}$	$0.76\pm 0.01$	$1.04\pm 0.01$
GRU	$\boldsymbol{0.96\pm 0.00}$	$0.30\pm 0.01$	$\boldsymbol{0.80\pm 0.02}$	$\boldsymbol{0.68\pm 0.01}$	$\boldsymbol{0.79\pm 0.00}$	$\boldsymbol{0.99\pm 0.00}$
Neural Hawkes	$\boldsymbol{0.96\pm 0.00}$	$0.31\pm 0.01$	$0.78\pm 0.01$	$0.72\pm 0.02$	$0.79\pm 0.00$	$1.04\pm 0.05$
Trans. Hawkes	$\boldsymbol{0.96\pm 0.00}$	$0.33\pm 0.03$	$0.74\pm 0.02$	$0.77\pm 0.06$	$0.26\pm 0.06$	$1.32\pm 0.19$
IFTPP	$\boldsymbol{0.96\pm 0.00}$	$\infty$	$0.76\pm 0.01$	$0.79\pm 0.02$	$0.78\pm 0.01$	$1.51\pm 0.55$

Appendix

Table 8: Accuracy and NMAE on six representative datasets, with two selected from each difficulty level. Results are reported analogously to Table 1 ; NMAE values above

10{,}000

are denoted by

\infty

Table 9: Summary of the real-world datasets used for pretraining and evaluation, grouped by domain. Trace length and duration denote the mean number of events and total duration per case, respectively; all values are unweighted averages of per-dataset statistics within each domain. Environmental monitoring is entirely held out from the pretraining pool , providing evaluation on a previously unseen domain.

Dataset and source	#Cases	#Act.	Length	Duration
Environmental monitoring
Beijing Air Quality ( Chen , 2017 )	12	5	4 387	1 218
Hourly PM2.5 observations from 12 Beijing stations. Values are discretized into six AQI categories and only transitions between categories are retained.
EPA AQS Ozone ( EPA AQS )	1 225	6	186.1	205.6
Hourly ozone measurements from US EPA monitoring sites. Concentrations are mapped to six AQI categories and only changes in category are emitted as events.
EPA AQS PM2.5 ( EPA AQS )	1 029	6	57.60	290.9

Appendix

Table 10: Details of the 50 real-world test datasets. Trace length and duration (in days) are averaged over all traces within each dataset.

Figure 5: Interarrival times, trace durations, activity counts, and trace lengths per difficulty in the test datasets, and among the pretraining datasets. Interarrival times, trace durations, and trace lengths are per-dataset averages.

Next-Event Prediction

See all 16 papers

Sep 21, 2026

Universal Multi-Modal Traceformer: Integrating Heterogeneous Context for Process Event Prediction

Event logs arise in a wide range of real-world processes, capturing not only event activities and timestamps but also multi-modal contextual information. Existing event-sequence models, including many temporal point process approaches, primarily model event activities and timestamps while overlooking heterogeneous context, such as numerical measurements, categorical attributes, textual descriptions, and metadata associated with individual events and entire traces. In this paper, we propose Universal Multi-Modal Traceformer (UMT), a unified framework for incorporating heterogeneous process context into next-event prediction. Built on a Transformer backbone, UMT introduces a universal feature encoder that maps diverse feature types into a shared representation space and handles contextual information at both the event and trace levels. UMT further develops a per-event Perceiver module that dynamically weights contextual features and adaptively integrates them into event-token representations. To accommodate the heavy-tailed and potentially multi-modal distribution of inter-arrival times, UMT represents each interval at multiple temporal scales and jointly predicts the corresponding scale-specific quantities. Experiments on 13 real-world event logs show that UMT improves both next-event activity and time prediction over existing approaches.

Fabian Spaeh, Jingxing Fang, Shandian Zhe +1

Celonis

Event Time Prediction Predictive Process Monitoring Multimodal Fusion Next-Event Prediction

Ask PDF

Oct 8, 2026

KITE: A Knowledge-Injected Temporal Encoder for Time-Series Event Forecasting

Forecasting rare, high-cost events in multivariate time-series systems is hard because their precursors are seldom observed, so a purely data-driven model has little to learn from. The engineers who run these systems, however, hold knowledge that precedes such events, information absent from the scarce labels. We call this a semantic prior and inject it into a self-supervised event forecaster through two switchable mechanisms, a covariate mechanism and an attention-bias mechanism, each the identity at its null, so one architecture serves every prior type with no per-dataset architecture search. A prior helps when it supplies information the observed window does not already contain: in a short context an informative prior beats no prior in-domain (pooled h-AUROC over the no-prior baseline: a periodicity clock +14%, p<0.001; a conservation residual +47% at a 0.25 label fraction), and a working prior transfers out-of-domain (+53%), substituting for feature transfer, while priors the window already recovers stay inert and the attention-bias mechanism shows no measurable gain. KITE is a switchable, attributable route to contextual conditioning, and a substrate for scaling toward cross-domain, pretrained time-series foundation world models that carry their governing structure explicitly.

Jonas Petersen, Pratik Jawahar, Riccardo Maggioni +6

ETH Zurich · Forgis AG · University of Vienna

Rare-Event Prediction Time Series Forecasting Multivariate Time Series Forecasting Self-Supervised Time Series Representation Learning

Ask PDF

Apr 23, 2026

Promoting Simple Agents: Ensemble Methods for Event-Log Prediction

We compare lightweight automata-based models (n-grams) with neural architectures (LSTM, Transformer) for next-activity prediction in streaming event logs. Experiments on synthetic patterns and five real-world process mining datasets show that n-grams with appropriate context windows achieve comparable accuracy to neural models while requiring substantially fewer resources. Unlike windowed neural architectures, which show unstable performance patterns, n-grams provide stable and consistent accuracy. While we demonstrate that classical ensemble methods like voting improve n-gram performance, they require running many agents in parallel during inference, increasing memory consumption and latency. We propose an ensemble method, the promotion algorithm, that dynamically selects between two active models during inference, reducing overhead compared to classical voting schemes. On real-world datasets, these ensembles match or exceed the accuracy of non-windowed neural models with lower computational cost.

Benedikt Bollig, Matthias Függer, Thomas Nowak +1

Université Paris-Saclay, CNRS, ENS Paris-Saclay, LMF, Gif-sur-Yvette, France · Institut Universitaire de France, Paris, France · Direction Générale de l’Armement, Paris, France

Next-Action Prediction Ensemble Learning Process Mining Language Modeling Next-Event Prediction

Ask PDF

Abstract

Next-Event Prediction

Explore similar work

Universal Multi-Modal Traceformer: Integrating Heterogeneous Context for Process Event Prediction

KITE: A Knowledge-Injected Temporal Encoder for Time-Series Event Forecasting

Promoting Simple Agents: Ensemble Methods for Event-Log Prediction