cs.LGOct 13, 2026

Foundation Models for Event Prediction Across Heterogeneous Event Logs

Authors: Fabian Spaeh, Sean Jaffe, Shandian Zhe, Bin Shen

Organizations: Celonis

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Event logs are ubiquitous across real-world domains, including business processes, manufacturing workflows, healthcare, and many other operational systems. Yet event logs can differ substantially in their event semantics, label spaces, interaction patterns, and temporal scales, making it difficult to build a single foundation model that generalizes directly to previously unseen systems. We address this challenge with the Foundation Event Predictor (FEP), a simple and scalable in-context architecture that performs next-event prediction through learned, event-level nonparametric inference. Causal self-attention encodes context and target histories into key and query representations, respectively, while the observed next context events are encoded as values. Cross-attention then performs learned, relevance-weighted aggregation of next-event outcomes over all context transitions, allowing the entire nonparametric prediction procedure to be expressed as standard attention operations and efficiently parallelized with modern kernels such as FlashAttention. To promote cross-system generalization, we introduce permutation-robust embedding learning and random fixed embeddings to decouple event IDs from persistent semantics, together with empirical quantile transforms and robust normalization to handle diverse temporal scales. We pretrain FEP on 696 real-world event datasets spanning heterogeneous event systems and evaluate it on 50 held-out datasets unseen during pretraining. Across these datasets, FEP generalizes effectively to new event systems without parameter adaptation, frequently approaching the performance of strong models trained separately on each target dataset while substantially outperforming existing foundation-model baselines, particularly on challenging prediction tasks.

Figures & tables

Figure 1: The architecture of Foundation Event Predictor (FEP). Context histories and observed next events form keys and values, respectively, while target histories form queries for cross-attention-based nonparametric prediction.
Table 1: Generalization performance by dataset difficulty, averaged over five runs across 50 test datasets. The best results among local methods and foundation models are highlighted in bold. FEP (Quantile) uses permutation training with the empirical quantile transform, while FEP (Robust log) uses random fixed embeddings with robust log-time normalization; all four combinations are reported in Appendix B . FEP-Local is the architecture-matched local counterpart of FEP (Quantile), trained separately on each target dataset. FIM-PP supports at most 22 activities; we therefore retain the 21 most frequent activities and merge the rest into an “other” activity. FIM-PP (FT) denotes FIM-PP further fine-tuned on the training set of each target dataset. FM-v2 is omitted from NMAE because it predicts remaining case time, while Transformer Hawkes and IFTPP are omitted because their NMAEs exceed 10,00010{,}000 across all difficulty levels.
MethodMAE scoreAccuracy score
Transformer164416441664\boldsymbol{1664}
GRU1690\boldsymbol{1690}16311631
Neural Hawkes1518151816191619
Transformer Hawkes1377137714221422
IFTPP1169116915401540
FEP (Local)1641164114361436
Table 2: Overall pairwise ratings across 50 test datasets, reported as Bradley–Terry scores (higher is better). Scores are computed jointly from pairwise comparisons among all local, foundation, and fine-tuned models.
Figure 2: Dataset-wise comparison between FEP and the best locally trained model. Colors indicate empirical dataset difficulty, and the dashed line denotes equal performance. The shaded region indicates the side of the diagonal where FEP performs better.
Figure 3: Ablation results for context size and model dimension.
Table 3: Ablation results for the number of pretraining datasets and model depth. ∞\infty denotes NMAE larger than 10,00010{,}000 .
Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

  Latency (ms)Queries/sPeak memory (GiB)
# Context seq.# Context K/V tokensStandardFA2StandardFA2StandardFA2
87922.1182.182472.2458.20.0390.038
161,5842.1242.185470.7457.60.0420.040
323,1682.1202.191471.7456.50.0500.045
646,3362.1122.188473.5457.00.0640.055
12812,6722.1212.196471.5455.50.0940.073
Appendix
Table 4: Inference efficiency and scalability of standard attention and FlashAttention-2 (FA2) for cached-context nonparametric inference. Each context sequence and query trace has length 100, yielding 99 history-to-next-event transitions or predictions. Peak memory includes model parameters, cached context representations, query execution, and output predictions.
# Context seq. (tokens)Latency reductionThroughput increaseMemory reduction
2,048 (203K)19.3%23.9%42.1%
4,096 (406K)27.8%38.5%42.8%
8,192 (811K)34.7%53.0%43.3%
12,288 (1.2M)37.5%60.0%43.4%
16,384 (1.6M)39.1%64.1%43.5%
Appendix
Table 5: Relative improvement of FlashAttention-2 over standard attention at large context sizes. Positive values indicate improvement.
SettingAccuracy (%)NMAE
Random fixed67.77±0.0367.77\pm 0.030.80±0.000.80\pm 0.00
Learned + permutation67.95±0.03\boldsymbol{67.95\pm 0.03}0.79±0.00\boldsymbol{0.79\pm 0.00}
Learned, no permutation64.14±0.0364.14\pm 0.030.83±0.000.83\pm 0.00
Appendix
Table 6: Effect of event-ID representation under the ablation protocol.
Table 7: Performance of all four FEP configurations across dataset difficulty levels. The configurations combine two inter-arrival-time transformations (quantile and robust log) with two event-ID representation strategies (permutation training and random fixed embeddings).
Figure 4: Dataset-wise comparison of FEP (Quantile) and FEP-Local on the 50 held-out datasets. Colors denote dataset difficulty, and the dashed line indicates equal performance. Points below the diagonal favor FEP for accuracy, whereas points above the diagonal favor FEP for NMAE. The FEP-favoring region is shaded in both panels.
 Medical AppointmentsBeijing Air QualityNASA SMAP/MSL
MethodAccuracyNMAEAccuracyNMAEAccuracyNMAE
Transformer0.96±0.00\boldsymbol{0.96\pm 0.00}0.30±0.000.30\pm 0.000.76±0.020.76\pm 0.020.68±0.01\boldsymbol{0.68\pm 0.01}0.76±0.010.76\pm 0.011.04±0.011.04\pm 0.01
GRU0.96±0.00\boldsymbol{0.96\pm 0.00}0.30±0.010.30\pm 0.010.80±0.02\boldsymbol{0.80\pm 0.02}0.68±0.01\boldsymbol{0.68\pm 0.01}0.79±0.00\boldsymbol{0.79\pm 0.00}0.99±0.00\boldsymbol{0.99\pm 0.00}
Neural Hawkes0.96±0.00\boldsymbol{0.96\pm 0.00}0.31±0.010.31\pm 0.010.78±0.010.78\pm 0.010.72±0.020.72\pm 0.020.79±0.000.79\pm 0.001.04±0.051.04\pm 0.05
Trans. Hawkes0.96±0.00\boldsymbol{0.96\pm 0.00}0.33±0.030.33\pm 0.030.74±0.020.74\pm 0.020.77±0.060.77\pm 0.060.26±0.060.26\pm 0.061.32±0.191.32\pm 0.19
IFTPP0.96±0.00\boldsymbol{0.96\pm 0.00}∞\infty0.76±0.010.76\pm 0.010.79±0.020.79\pm 0.020.78±0.010.78\pm 0.011.51±0.551.51\pm 0.55
Appendix
Table 8: Accuracy and NMAE on six representative datasets, with two selected from each difficulty level. Results are reported analogously to Table 1 ; NMAE values above 10,00010{,}000 are denoted by ∞\infty .
Table 9: Summary of the real-world datasets used for pretraining and evaluation, grouped by domain. Trace length and duration denote the mean number of events and total duration per case, respectively; all values are unweighted averages of per-dataset statistics within each domain. Environmental monitoring is entirely held out from the pretraining pool , providing evaluation on a previously unseen domain.
Dataset and source#Cases#Act.LengthDuration
Environmental monitoring
Beijing Air Quality ( Chen , 2017 )1254 3871 218
Hourly PM2.5 observations from 12 Beijing stations. Values are discretized into six AQI categories and only transitions between categories are retained.
EPA AQS Ozone ( EPA AQS )1 2256186.1205.6
Hourly ozone measurements from US EPA monitoring sites. Concentrations are mapped to six AQI categories and only changes in category are emitted as events.
EPA AQS PM2.5 ( EPA AQS )1 029657.60290.9
Appendix
Table 10: Details of the 50 real-world test datasets. Trace length and duration (in days) are averaged over all traces within each dataset.
Figure 5: Interarrival times, trace durations, activity counts, and trace lengths per difficulty in the test datasets, and among the pretraining datasets. Interarrival times, trace durations, and trace lengths are per-dataset averages.
More in this topic

Next-Event Prediction

See all 16 papers

Explore similar work

CardsList
Sep 21, 2026

Universal Multi-Modal Traceformer: Integrating Heterogeneous Context for Process Event Prediction

Event logs arise in a wide range of real-world processes, capturing not only event activities and timestamps but also multi-modal contextual information. Existing event-sequence models, including many temporal point process approaches, primarily model event activities and timestamps while overlooking heterogeneous context, such as numerical measurements, categorical attributes, textual descriptions, and metadata associated with individual events and entire traces. In this paper, we propose Universal Multi-Modal Traceformer (UMT), a unified framework for incorporating heterogeneous process context into next-event prediction. Built on a Transformer backbone, UMT introduces a universal feature encoder that maps diverse feature types into a shared representation space and handles contextual information at both the event and trace levels. UMT further develops a per-event Perceiver module that dynamically weights contextual features and adaptively integrates them into event-token representations. To accommodate the heavy-tailed and potentially multi-modal distribution of inter-arrival times, UMT represents each interval at multiple temporal scales and jointly predicts the corresponding scale-specific quantities. Experiments on 13 real-world event logs show that UMT improves both next-event activity and time prediction over existing approaches.

Fabian Spaeh, Jingxing Fang, Shandian Zhe +1

Celonis

Event Time PredictionPredictive Process MonitoringMultimodal FusionNext-Event Prediction
AskPDF
Oct 8, 2026

KITE: A Knowledge-Injected Temporal Encoder for Time-Series Event Forecasting

Forecasting rare, high-cost events in multivariate time-series systems is hard because their precursors are seldom observed, so a purely data-driven model has little to learn from. The engineers who run these systems, however, hold knowledge that precedes such events, information absent from the scarce labels. We call this a semantic prior and inject it into a self-supervised event forecaster through two switchable mechanisms, a covariate mechanism and an attention-bias mechanism, each the identity at its null, so one architecture serves every prior type with no per-dataset architecture search. A prior helps when it supplies information the observed window does not already contain: in a short context an informative prior beats no prior in-domain (pooled h-AUROC over the no-prior baseline: a periodicity clock +14%, p<0.001; a conservation residual +47% at a 0.25 label fraction), and a working prior transfers out-of-domain (+53%), substituting for feature transfer, while priors the window already recovers stay inert and the attention-bias mechanism shows no measurable gain. KITE is a switchable, attributable route to contextual conditioning, and a substrate for scaling toward cross-domain, pretrained time-series foundation world models that carry their governing structure explicitly.

Jonas Petersen, Pratik Jawahar, Riccardo Maggioni +6

ETH Zurich · Forgis AG · University of Vienna

Rare-Event PredictionTime Series ForecastingMultivariate Time Series ForecastingSelf-Supervised Time Series Representation Learning
AskPDF
Apr 23, 2026

Promoting Simple Agents: Ensemble Methods for Event-Log Prediction

We compare lightweight automata-based models (n-grams) with neural architectures (LSTM, Transformer) for next-activity prediction in streaming event logs. Experiments on synthetic patterns and five real-world process mining datasets show that n-grams with appropriate context windows achieve comparable accuracy to neural models while requiring substantially fewer resources. Unlike windowed neural architectures, which show unstable performance patterns, n-grams provide stable and consistent accuracy. While we demonstrate that classical ensemble methods like voting improve n-gram performance, they require running many agents in parallel during inference, increasing memory consumption and latency. We propose an ensemble method, the promotion algorithm, that dynamically selects between two active models during inference, reducing overhead compared to classical voting schemes. On real-world datasets, these ensembles match or exceed the accuracy of non-windowed neural models with lower computational cost.

Benedikt Bollig, Matthias Függer, Thomas Nowak +1

Université Paris-Saclay, CNRS, ENS Paris-Saclay, LMF, Gif-sur-Yvette, France · Institut Universitaire de France, Paris, France · Direction Générale de l’Armement, Paris, France

Next-Action PredictionEnsemble LearningProcess MiningLanguage ModelingNext-Event Prediction
AskPDF