Generative Modeling for Cascading Extremes via Autoregressive Transformers

Miguel de Carvalho; Myung Won Lee; Jordan Richards; Sotirios Sabanis

stat.MLNov 3, 2026

Generative Modeling for Cascading Extremes via Autoregressive Transformers

Authors: Miguel de Carvalho, Myung Won Lee, Jordan Richards, Sotirios Sabanis

Organizations: School of Mathematics and Maxwell Institute for Mathematical Sciences University of Edinburgh · CIDMA–Department of Mathematics Universidade de Aveiro · National Technical University of Athens Athena/Archimedes Research Centre

Abstract

Extreme events often occur sequentially, from earthquake-tsunami cascades to successive extreme losses in financial markets. We introduce an extremal token representation that encodes multivariate observations as a categorical sequence. A quiet token records the absence of threshold exceedances; otherwise, the token identifies the exceeding component with the largest marginal probability-scale score. The representation is invariant to componentwise strictly increasing transformations. It preserves the identity and temporal ordering of dominant exceedances but discards their magnitudes and the identities of nondominant simultaneous exceedances. A decoder-only transformer estimates the history-dependent next-token kernel on this representation, and recursive sampling generates future token sequences from the induced joint predictive law. We amortize our generative model by pretraining across simulated process families with varying parameters and dependence persistence. Next-token predictions are evaluated against exact conditional probabilities for held-out categorical processes and particle filter references for heavy-tailed vector autoregressive processes. Applied to ten-second returns for five semiconductor stocks, the pretrained model supports zero-shot transfer, while a few epochs of adaptation improve agreement between observed and generated token frequencies and transition patterns. These results support task-specific tokenization and simulation-based pretraining for modeling which dimensions dominate multivariate extremes.

Figures & tables

Panel of Figure 1 — Figure 1: Autoregressive transformer schematic based on the probabilistic setup in Section 2.1 .

Figure 2: Left: Relative frequency histograms comparing observed marginal token frequencies with those generated by the optimal-shot model at $T=1$ . Right: Table 1 reports the marginal probabilities and the total variation distance at each stage of transfer learning and the non-pretrained model. Values in bold are the marginal probabilities closest in absolute distance to the corresponding observed empirical values, or the lowest estimates of the total variation distance.

Figure 3: Empirical (left) and optimal-shot model (centre) transition probabilities, with tilewise absolute differences (right).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

$p$	$\bm{\pi}$	$\widehat{\bm{\pi}}$	$\overline{d}_{\mathrm{KL}}(\bm{\pi},\widehat{\bm{\pi}})$	$\overline{d}_{\mathrm{TV}}(\bm{\pi},\widehat{\bm{\pi}})$	$1-\pi_{0}$	$1-\widehat{\pi}_{0}$
1	$(0.929,0.014,0.015,0.014,0.014,0.014)$	$(0.928,0.014,0.015,0.015,0.014,0.014)$	$0.0009$	$0.0089$	$0.071$	$0.072$
2	$(0.928,0.014,0.014,0.014,0.014,0.014)$	$(0.928,0.014,0.014,0.015,0.014,0.014)$	$0.0012$	$0.0105$	$0.072$	$0.072$
3	$(0.924,0.015,0.015,0.015,0.015,0.015)$	$(0.927,0.015,0.015,0.015,0.015,0.015)$	$0.0008$	$0.0083$	$0.076$	$0.073$
4	$(0.927,0.015,0.015,0.015,0.015,0.015)$	$(0.927,0.014,0.015,0.015,0.015,0.015)$	$0.0007$	$0.0084$	$0.073$	$0.073$
5	$(0.932,0.014,0.014,0.014,0.014,0.014)$	$(0.930,0.014,0.014,0.014,0.014,0.014)$	$0.0010$	$0.0091$	$0.068$	$0.070$
6	$(0.929,0.014,0.014,0.014,0.014,0.014)$	$(0.929,0.014,0.014,0.014,0.014,0.014)$	$0.0009$	$0.0092$	$0.071$	$0.071$

Appendix

Table 2: Conditional probability recovery on the test set for categorical process (top subtable) and vector autoregressive process (bottom subtable). For each order

p

\bm{\pi}

and

\widehat{\bm{\pi}}

are the mean conditional probability vectors across the test set. Here,

\text{d}_{\mathrm{KL}}

and

\text{d}_{\mathrm{TV}}

are computed for each history and then averaged within each order. The final two columns report the corresponding averaged active event probabilities.

Figure 4: Scatter plots of active mass $1-\pi_{0}$ against $1-\widehat{\pi}_{0}$ for categorical DGP (orange) and VAR DGP (blue), by true order $p$ . The diagonal line marks exact calibration.

Figure 5: Within DGP conditional probability error heatmaps for categorical DGP. Each plot corresponds to testing DGP of true order $p$ ; rows are distinct histories and columns are the subsequent values in the extremal vocabulary. Each tile shows the absolute conditional probability error and each tick on the left hand side is the testing history.

$\mathcal{C}_{t}$	$A_{t+1}$	$\ell=1$	$\ell=5$	$\ell=10$
$(0,0,0,2,2,0,0,0,4,2)$	2	$(0.0353,0.0119,0.0041,0.0113,0.0029,0.0134)$	$(0.0575,0.0157,0.0057,0.0157,0.0145,0.0173)$	$(0.0570,0.0132,0.0059,0.0123,0.0105,0.0151)$
$(0,0,0,0,3,0,0,2,3,3)$	2	$(0.0264,0.0174,0.0294,0.0142,0.0131,0.0111)$	$(0.0670,0.0250,0.0117,0.0164,0.0200,0.0172)$	$(0.0595,0.0223,0.0155,0.0195,0.0178,0.0154)$
$(3,0,0,0,0,2,0,0,1,3)$	3	$(0.0276,0.0116,0.0148,0.0051,0.0137,0.0056)$	$(0.0070,0.0245,0.0032,0.0037,0.0094,0.0012)$	$(0.0037,0.0215,0.0039,0.0018,0.0099,0.0021)$
$(0,0,0,0,3,2,0,1,4,2)$	2	$(0.0720,0.0134,0.0300,0.0320,0.0150,0.0116)$	$(0.0035,0.0041,0.0189,0.0208,0.0329,0.0008)$	$(0.0154,0.0012,0.0258,0.0177,0.0301,0.0032)$
$(0,0,0,0,0,4,0,1,5,4)$	5	$(0.0362,0.0239,0.0114,0.0063,0.0161,0.0106)$	$(0.0125,0.0098,0.0032,0.0011,0.0217,0.0028)$	$(0.0056,0.0111,0.0042,0.0000,0.0181,0.0028)$
$(1,0,0,0,0,2,0,0,3,3)$	2	$(0.0104,0.0087,0.0195,0.0128,0.0043,0.0087)$	$(0.0132,0.0120,0.0119,0.0042,0.0065,0.0109)$	$(0.0181,0.0178,0.0104,0.0059,0.0069,0.0099)$

Appendix

Table 3: Context specific recovery of the fixed

10

th-order conditional distribution. For each context

\mathcal{C}_{t}

A_{t+1}

is the realized next token and the vectors contain the tokenwise absolute conditional probability errors

|\widehat{\pi}_{j}-\pi^{\mathrm{MC}}_{j}|

with conditioning length

\ell=1,5,

and

10

for the corresponding Fig. 6 .

Figure 6: Within-DGP fixed-target diagnostic for the $p\!=\!10$ VAR DGP. Each row represents one of the selected contexts in Table 3 , each column represents an extremal token, and each plot corresponds to a context length supplied to the transformer. The color scale is absolute conditional probability error.

$p$	$1$	$2$	$3$	$4$	$5$
Median $\mathrm{MCSE}$	$0.000333$ ( $1.90$ )	$0.000323$ ( $1.81$ )	$0.000332$ ( $2.01$ )	$0.000284$ ( $1.87$ )	$0.000329$ ( $1.98$ )
95th percentile $\mathrm{MCSE}$	$0.001682$ ( $8.17$ )	$0.001511$ ( $7.55$ )	$0.001598$ ( $7.13$ )	$0.001463$ ( $6.59$ )	$0.001519$ ( $7.61$ )
$p$	$6$	$7$	$8$	$9$	$10$
Median $\mathrm{MCSE}$	$0.000275$ ( $1.92$ )	$0.000287$ ( $1.88$ )	$0.000327$ ( $1.81$ )	$0.000356$ ( $1.90$ )	$0.000313$ ( $1.92$ )
95th percentile $\mathrm{MCSE}$	$0.001380$ ( $6.61$ )	$0.001352$ ( $6.90$ )	$0.001291$ ( $6.72$ )	$0.001457$ ( $6.75$ )	$0.001433$ ( $6.96$ )

Appendix

Table 4: MCSE of the particle filter approximation of next-token probabilities. The median and 95th percentile are computed across all DGP, context and token probabilities for each order, and relative MCSE is expressed as a percentage in parentheses. The MCSE is computed from

B=3

independent complete filtering-and-prediction repeats as the between-repeat sample standard deviation divided by

\sqrt{B}

Metric	Target	Fixed empirical
Mean PIT	$0.5000$	0.4999 [0.4985, 0.5013]
PIT variance	$0.08333$	0.08333 [0.08268, 0.08398]
Full-distribution QQ RMSE	$0$	0.00182 [0.00122, 0.00265]
Upper-tail probability, $\Pr(U>0.98)$	$0.0200$	0.01997 [0.01908, 0.02092]
Upper-tail QQ RMSE	$0$	0.000232 [0.000168, 0.000353]

Appendix

Table 5: Monte Carlo robustness of the probability-scale marginal transformation and the induced extremal token structure.

Figure 7: Intraday profile of dominant-exceedance tokens assigned to NVDA, TSM, AVGO, MU, and AMD on 23rd March 2020 (COVID-19 market crash) under the

q=0.98

specification. Each cell reports the number of ten-second dominant-exceedance tokens aggregated within the corresponding minute.

Figure 8: Marginal Hill estimator with its $95\%$ confidence interval for each stock. The estimator is evaluated at every $1{,}000$ values of $k$ spanning the largest $1$ to $15{,}000$ upper-tail observations out of 1.19 million.

Figure 9: Number of non-quiet tokens per day from 2nd January 2020 to 24th July 2026 for the five selected semiconductor stocks.

Metric	Previous Token	Zero-shot	One-shot	Few-shot	Optimal-shot	Non-pretrained	Student- $t$ VAR
Rowwise Total Variation ( $\text{d}_{\mathrm{TV}}$ )	$\mathrm{Quiet}$	$0.1147$	$0.0796$	$0.0200$	$\mathbf{0.0252}$	$0.1147$	$0.1887$
	$\mathrm{NVDA}$	$0.0756$	$0.1163$	$0.1319$	$\mathbf{0.0757}$	$0.0756$	$0.2384$
	$\mathrm{TSM}$	$0.0759$	$0.0714$	$0.1049$	$\mathbf{0.0590}$	$0.0759$	$0.1830$
	$\mathrm{AVGO}$	$0.0701$	$0.1127$	$0.0806$	$\mathbf{0.0409}$	$0.0701$	$0.2252$
	$\mathrm{MU}$	$0.0580$	$0.1031$	$0.0307$	$\mathbf{0.0246}$	$0.0580$	$0.2143$
	$\mathrm{AMD}$	$0.0451$	$0.0836$	$0.0906$	$\mathbf{0.0444}$	$0.0451$	$0.1727$

Appendix

Table 6: Comparison of transition probability metrics (

\text{d}_{\mathrm{TV}}

\text{d}_{\mathrm{KL}}

, and relative discrepancy,

1-\bm{\pi}^{\mathrm{gen}}/\bm{\pi}^{\mathrm{obs}}

) across checkpoints and the Student-

t

VAR(

10

) baseline at temperature

T=1

. Lower values indicate closer agreement with empirical transition probabilities.

Figure 10: Observed and generated next-token transition structure. Each panel reports the absolute transition probability error. Rows correspond to temperatures $T\!=\!1.1$ , $1$ , and $0.95$ , from top to bottom; columns correspond to the zero-shot, one-shot, few-shot, optimal-shot, and non-pretrained models, from left to right.

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 524 papers

SurF: A Generative Model for Multivariate Irregular Time Series Forecasting
May 13, 2026Mohammad R. Rezaei, Tejas Balaji, Rahul G. KrishnanIrregular Time-Series Modeling Multivariate Time Series Forecasting
Department of Computer Science University of Toronto Vector Institute Toronto, ON, Canada
Conditional Attribute Estimation with Autoregressive Sequence Models
May 13, 2026Erica Stutz, Giacomo Marino, Daniella Meeker +2Next-Token Prediction Controllable Text Generation
Department of Biomedical Informatics and Data Science Yale University New Haven, CT 06510 · Department of Biostatistics Yale University New Haven, CT 06510 · Department of Biomedical Informatics and Data Science, Department of Pediatrics Yale University New Haven, CT 06510
E4GEN: Event-level Explainable Extreme-Enhanced Time-series Generation
Jun 1, 2026Lin Jiang, Dahai Yu, Ximiao Li +1Time Series Generation Generative Modeling
Florida State University

May 13, 2026J/K move · Enter open · S save

SurF: A Generative Model for Multivariate Irregular Time Series Forecasting

Mohammad R. Rezaei, Tejas Balaji, Rahul G. Krishnan

Department of Computer Science University of Toronto Vector Institute Toronto, ON, Canada

Irregular Time-Series Modeling Multivariate Time Series Forecasting Temporal Point Processes Generative Modeling Time-Series Foundation Models

Open paper AskPDF arXiv

Irregularly sampled multivariate event streams remain a difficult modality for generative modeling: tokenization-based approaches break down when inter-event intervals vary by orders of magnitude. We (i) propose SurF, a generative model that uses the Time Rescaling Theorem (TRT) as a learnable bijection between event sequences and i.i.d.\ unit-rate exponential noise, enabling a single model to be trained across heterogeneous event-stream datasets; (ii) three efficient parameterizations of the cumulative intensity that scale to long sequences; and (iii) a Transformer-based encoder for multi-dataset pretraining. On six real-world benchmarks, SurF achieves the best reported time RMSE on Earthquake, Retweet, and Taobao, and is within trial-level noise of the strongest specialist on the remaining three. Under a strict leave-one-out protocol, the held-out checkpoint beats every classical and neural-autoregressive baseline on

5/6

datasets and beats every baseline on Amazon and Earthquake, an initial step toward foundation models over asynchronous event streams (Code is available at https://github.com/MrRezaeiUofT/SurF).

Abstract

Generative Modeling

Explore similar work

SurF: A Generative Model for Multivariate Irregular Time Series Forecasting

Conditional Attribute Estimation with Autoregressive Sequence Models

E4GEN: Event-level Explainable Extreme-Enhanced Time-series Generation