Generative Modeling for Cascading Extremes via Autoregressive Transformers

Miguel de Carvalho; Myung Won Lee; Jordan Richards; Sotirios Sabanis

stat.MLNov 3, 2026

Generative Modeling for Cascading Extremes via Autoregressive Transformers

Authors: Miguel de Carvalho, Myung Won Lee, Jordan Richards, Sotirios Sabanis

Organizations: School of Mathematics and Maxwell Institute for Mathematical Sciences University of Edinburgh · CIDMA–Department of Mathematics Universidade de Aveiro · National Technical University of Athens Athena/Archimedes Research Centre

Abstract

Extreme events often occur sequentially, from earthquake-tsunami cascades to successive extreme losses in financial markets. We introduce an extremal token representation that encodes multivariate observations as a categorical sequence. A quiet token records the absence of threshold exceedances; otherwise, the token identifies the exceeding component with the largest marginal probability-scale score. The representation is invariant to componentwise strictly increasing transformations. It preserves the identity and temporal ordering of dominant exceedances but discards their magnitudes and the identities of nondominant simultaneous exceedances. A decoder-only transformer estimates the history-dependent next-token kernel on this representation, and recursive sampling generates future token sequences from the induced joint predictive law. We amortize our generative model by pretraining across simulated process families with varying parameters and dependence persistence. Next-token predictions are evaluated against exact conditional probabilities for held-out categorical processes and particle filter references for heavy-tailed vector autoregressive processes. Applied to ten-second returns for five semiconductor stocks, the pretrained model supports zero-shot transfer, while a few epochs of adaptation improve agreement between observed and generated token frequencies and transition patterns. These results support task-specific tokenization and simulation-based pretraining for modeling which dimensions dominate multivariate extremes.

Figures & tables

Panel of Figure 1 — Figure 1: Autoregressive transformer schematic based on the probabilistic setup in Section 2.1 .

Figure 2: Left: Relative frequency histograms comparing observed marginal token frequencies with those generated by the optimal-shot model at $T=1$ . Right: Table 1 reports the marginal probabilities and the total variation distance at each stage of transfer learning and the non-pretrained model. Values in bold are the marginal probabilities closest in absolute distance to the corresponding observed empirical values, or the lowest estimates of the total variation distance.

Figure 3: Empirical (left) and optimal-shot model (centre) transition probabilities, with tilewise absolute differences (right).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

$p$	$\bm{\pi}$	$\widehat{\bm{\pi}}$	$\overline{d}_{\mathrm{KL}}(\bm{\pi},\widehat{\bm{\pi}})$	$\overline{d}_{\mathrm{TV}}(\bm{\pi},\widehat{\bm{\pi}})$	$1-\pi_{0}$	$1-\widehat{\pi}_{0}$
1	$(0.929,0.014,0.015,0.014,0.014,0.014)$	$(0.928,0.014,0.015,0.015,0.014,0.014)$	$0.0009$	$0.0089$	$0.071$	$0.072$
2	$(0.928,0.014,0.014,0.014,0.014,0.014)$	$(0.928,0.014,0.014,0.015,0.014,0.014)$	$0.0012$	$0.0105$	$0.072$	$0.072$
3	$(0.924,0.015,0.015,0.015,0.015,0.015)$	$(0.927,0.015,0.015,0.015,0.015,0.015)$	$0.0008$	$0.0083$	$0.076$	$0.073$
4	$(0.927,0.015,0.015,0.015,0.015,0.015)$	$(0.927,0.014,0.015,0.015,0.015,0.015)$	$0.0007$	$0.0084$	$0.073$	$0.073$
5	$(0.932,0.014,0.014,0.014,0.014,0.014)$	$(0.930,0.014,0.014,0.014,0.014,0.014)$	$0.0010$	$0.0091$	$0.068$	$0.070$
6	$(0.929,0.014,0.014,0.014,0.014,0.014)$	$(0.929,0.014,0.014,0.014,0.014,0.014)$	$0.0009$	$0.0092$	$0.071$	$0.071$

Appendix

Table 2: Conditional probability recovery on the test set for categorical process (top subtable) and vector autoregressive process (bottom subtable). For each order

p

\bm{\pi}

and

\widehat{\bm{\pi}}

are the mean conditional probability vectors across the test set. Here,

\text{d}_{\mathrm{KL}}

and

\text{d}_{\mathrm{TV}}

are computed for each history and then averaged within each order. The final two columns report the corresponding averaged active event probabilities.

Figure 4: Scatter plots of active mass $1-\pi_{0}$ against $1-\widehat{\pi}_{0}$ for categorical DGP (orange) and VAR DGP (blue), by true order $p$ . The diagonal line marks exact calibration.

Figure 5: Within DGP conditional probability error heatmaps for categorical DGP. Each plot corresponds to testing DGP of true order $p$ ; rows are distinct histories and columns are the subsequent values in the extremal vocabulary. Each tile shows the absolute conditional probability error and each tick on the left hand side is the testing history.

$\mathcal{C}_{t}$	$A_{t+1}$	$\ell=1$	$\ell=5$	$\ell=10$
$(0,0,0,2,2,0,0,0,4,2)$	2	$(0.0353,0.0119,0.0041,0.0113,0.0029,0.0134)$	$(0.0575,0.0157,0.0057,0.0157,0.0145,0.0173)$	$(0.0570,0.0132,0.0059,0.0123,0.0105,0.0151)$
$(0,0,0,0,3,0,0,2,3,3)$	2	$(0.0264,0.0174,0.0294,0.0142,0.0131,0.0111)$	$(0.0670,0.0250,0.0117,0.0164,0.0200,0.0172)$	$(0.0595,0.0223,0.0155,0.0195,0.0178,0.0154)$
$(3,0,0,0,0,2,0,0,1,3)$	3	$(0.0276,0.0116,0.0148,0.0051,0.0137,0.0056)$	$(0.0070,0.0245,0.0032,0.0037,0.0094,0.0012)$	$(0.0037,0.0215,0.0039,0.0018,0.0099,0.0021)$
$(0,0,0,0,3,2,0,1,4,2)$	2	$(0.0720,0.0134,0.0300,0.0320,0.0150,0.0116)$	$(0.0035,0.0041,0.0189,0.0208,0.0329,0.0008)$	$(0.0154,0.0012,0.0258,0.0177,0.0301,0.0032)$
$(0,0,0,0,0,4,0,1,5,4)$	5	$(0.0362,0.0239,0.0114,0.0063,0.0161,0.0106)$	$(0.0125,0.0098,0.0032,0.0011,0.0217,0.0028)$	$(0.0056,0.0111,0.0042,0.0000,0.0181,0.0028)$
$(1,0,0,0,0,2,0,0,3,3)$	2	$(0.0104,0.0087,0.0195,0.0128,0.0043,0.0087)$	$(0.0132,0.0120,0.0119,0.0042,0.0065,0.0109)$	$(0.0181,0.0178,0.0104,0.0059,0.0069,0.0099)$

Appendix

Table 3: Context specific recovery of the fixed

10

th-order conditional distribution. For each context

\mathcal{C}_{t}

A_{t+1}

is the realized next token and the vectors contain the tokenwise absolute conditional probability errors

|\widehat{\pi}_{j}-\pi^{\mathrm{MC}}_{j}|

with conditioning length

\ell=1,5,

and

10

for the corresponding Fig. 6 .

Figure 6: Within-DGP fixed-target diagnostic for the $p\!=\!10$ VAR DGP. Each row represents one of the selected contexts in Table 3 , each column represents an extremal token, and each plot corresponds to a context length supplied to the transformer. The color scale is absolute conditional probability error.

$p$	$1$	$2$	$3$	$4$	$5$
Median $\mathrm{MCSE}$	$0.000333$ ( $1.90$ )	$0.000323$ ( $1.81$ )	$0.000332$ ( $2.01$ )	$0.000284$ ( $1.87$ )	$0.000329$ ( $1.98$ )
95th percentile $\mathrm{MCSE}$	$0.001682$ ( $8.17$ )	$0.001511$ ( $7.55$ )	$0.001598$ ( $7.13$ )	$0.001463$ ( $6.59$ )	$0.001519$ ( $7.61$ )
$p$	$6$	$7$	$8$	$9$	$10$
Median $\mathrm{MCSE}$	$0.000275$ ( $1.92$ )	$0.000287$ ( $1.88$ )	$0.000327$ ( $1.81$ )	$0.000356$ ( $1.90$ )	$0.000313$ ( $1.92$ )
95th percentile $\mathrm{MCSE}$	$0.001380$ ( $6.61$ )	$0.001352$ ( $6.90$ )	$0.001291$ ( $6.72$ )	$0.001457$ ( $6.75$ )	$0.001433$ ( $6.96$ )

Appendix

Table 4: MCSE of the particle filter approximation of next-token probabilities. The median and 95th percentile are computed across all DGP, context and token probabilities for each order, and relative MCSE is expressed as a percentage in parentheses. The MCSE is computed from

B=3

independent complete filtering-and-prediction repeats as the between-repeat sample standard deviation divided by

\sqrt{B}

Metric	Target	Fixed empirical
Mean PIT	$0.5000$	0.4999 [0.4985, 0.5013]
PIT variance	$0.08333$	0.08333 [0.08268, 0.08398]
Full-distribution QQ RMSE	$0$	0.00182 [0.00122, 0.00265]
Upper-tail probability, $\Pr(U>0.98)$	$0.0200$	0.01997 [0.01908, 0.02092]
Upper-tail QQ RMSE	$0$	0.000232 [0.000168, 0.000353]

Appendix

Table 5: Monte Carlo robustness of the probability-scale marginal transformation and the induced extremal token structure.

Figure 7: Intraday profile of dominant-exceedance tokens assigned to NVDA, TSM, AVGO, MU, and AMD on 23rd March 2020 (COVID-19 market crash) under the

q=0.98

specification. Each cell reports the number of ten-second dominant-exceedance tokens aggregated within the corresponding minute.

Figure 8: Marginal Hill estimator with its $95\%$ confidence interval for each stock. The estimator is evaluated at every $1{,}000$ values of $k$ spanning the largest $1$ to $15{,}000$ upper-tail observations out of 1.19 million.

Figure 9: Number of non-quiet tokens per day from 2nd January 2020 to 24th July 2026 for the five selected semiconductor stocks.

Metric	Previous Token	Zero-shot	One-shot	Few-shot	Optimal-shot	Non-pretrained	Student- $t$ VAR
Rowwise Total Variation ( $\text{d}_{\mathrm{TV}}$ )	$\mathrm{Quiet}$	$0.1147$	$0.0796$	$0.0200$	$\mathbf{0.0252}$	$0.1147$	$0.1887$
	$\mathrm{NVDA}$	$0.0756$	$0.1163$	$0.1319$	$\mathbf{0.0757}$	$0.0756$	$0.2384$
	$\mathrm{TSM}$	$0.0759$	$0.0714$	$0.1049$	$\mathbf{0.0590}$	$0.0759$	$0.1830$
	$\mathrm{AVGO}$	$0.0701$	$0.1127$	$0.0806$	$\mathbf{0.0409}$	$0.0701$	$0.2252$
	$\mathrm{MU}$	$0.0580$	$0.1031$	$0.0307$	$\mathbf{0.0246}$	$0.0580$	$0.2143$
	$\mathrm{AMD}$	$0.0451$	$0.0836$	$0.0906$	$\mathbf{0.0444}$	$0.0451$	$0.1727$

Appendix

Table 6: Comparison of transition probability metrics (

\text{d}_{\mathrm{TV}}

\text{d}_{\mathrm{KL}}

, and relative discrepancy,

1-\bm{\pi}^{\mathrm{gen}}/\bm{\pi}^{\mathrm{obs}}

) across checkpoints and the Student-

t

VAR(

10

) baseline at temperature

T=1

. Lower values indicate closer agreement with empirical transition probabilities.

Figure 10: Observed and generated next-token transition structure. Each panel reports the absolute transition probability error. Rows correspond to temperatures $T\!=\!1.1$ , $1$ , and $0.95$ , from top to bottom; columns correspond to the zero-shot, one-shot, few-shot, optimal-shot, and non-pretrained models, from left to right.

Generative Modeling

Generative modeling learns data distributions to create new samples, from images and text to molecules, simulations, and structured records.

See all 524 papers

May 13, 2026

SurF: A Generative Model for Multivariate Irregular Time Series Forecasting

Irregularly sampled multivariate event streams remain a difficult modality for generative modeling: tokenization-based approaches break down when inter-event intervals vary by orders of magnitude. We (i) propose SurF, a generative model that uses the Time Rescaling Theorem (TRT) as a learnable bijection between event sequences and i.i.d.\ unit-rate exponential noise, enabling a single model to be trained across heterogeneous event-stream datasets; (ii) three efficient parameterizations of the cumulative intensity that scale to long sequences; and (iii) a Transformer-based encoder for multi-dataset pretraining. On six real-world benchmarks, SurF achieves the best reported time RMSE on Earthquake, Retweet, and Taobao, and is within trial-level noise of the strongest specialist on the remaining three. Under a strict leave-one-out protocol, the held-out checkpoint beats every classical and neural-autoregressive baseline on

5/6

datasets and beats every baseline on Amazon and Earthquake, an initial step toward foundation models over asynchronous event streams (Code is available at https://github.com/MrRezaeiUofT/SurF).

Mohammad R. Rezaei, Tejas Balaji, Rahul G. Krishnan

Department of Computer Science University of Toronto Vector Institute Toronto, ON, Canada

Irregular Time-Series Modeling Multivariate Time Series Forecasting Temporal Point Processes Generative Modeling Time-Series Foundation Models

Ask PDF

May 13, 2026

Conditional Attribute Estimation with Autoregressive Sequence Models

Generative models are often trained with a next-token prediction objective, yet many downstream applications require the ability to estimate or control sequence-level properties. Next-token prediction can lead to overfitting of local patterns during training, underfitting of global structure, and requires significant downstream modifications or expensive sampling to guide or predict the global attributes of generated samples at inference time. Here, we introduce Conditional Attribute Transformers, a novel method for jointly estimating the next-token probability and the value of an attribute conditional on each potential next token selection. This framework enables three critical capabilities within a single forward pass, without modification of the input sequence: (1) per-token credit assignment across an entire sequence, by identifying how each token in a sequence is associated with an attribute's value; (2) counterfactual analysis, by quantifying attribute differences conditional on alternative next token choices; (3) steerable generation, by decoding sequences based on a combination of next-token and attribute likelihoods. Our approach achieves state of the art performance on sparse reward tasks, improves next-token prediction at sufficient model sizes, estimates attribute probabilities orders of magnitude faster than sampling, and can guide decoding of autoregressive sequence models on a range of language tasks.

Erica Stutz, Giacomo Marino, Daniella Meeker +2

Department of Biomedical Informatics and Data Science Yale University New Haven, CT 06510 · Department of Biostatistics Yale University New Haven, CT 06510 · Department of Biomedical Informatics and Data Science, Department of Pediatrics Yale University New Haven, CT 06510

Next-Token Prediction Controllable Text Generation Autoregressive Language Modeling Counterfactual Reasoning in Language Models Autoregressive Generation Conditional Generative Modeling+1

Ask PDF

Jun 1, 2026

E4GEN: Event-level Explainable Extreme-Enhanced Time-series Generation

Generating realistic time series is essential for scientific research and real-world applications. However, existing methods often emphasize overall distributional fidelity while failing to faithfully capture extreme events. To advance existing research, we propose E4GEN, an explainable diffusion framework for extreme event-aware time-series generation. E4GEN provides systematic insights into when, what, and how to control extreme-event generation through three key components. First, E-Activator learns the dataset-adaptive extreme-control signal activation step during the denoising process without interfering with regular temporal components, including trend and seasonality. Second, E-Predictor determines what control signal to enforce through Self-Driven Semantic Prediction, where each sample derives its own control signal by inferring latent extreme-event information during generation. It also includes a novel Data-Conditioned Training, Noise-Initiated Sampling mechanism to address the issue of unavailable training labels. Third, E-Control specifies how to control extreme-event generation through a trainable Extreme Control Network, which transforms the semantic control signal into layer-wise signals and injects it into the denoising process. We evaluate E4GEN on six datasets with 17 metrics, and extensive experiments show that E4GEN outperforms state-of-the-art models across multiple dimensions, including overall fidelity, extreme-event fidelity, and downstream utility.

Lin Jiang, Dahai Yu, Ximiao Li +1

Florida State University

Time Series Generation Generative Modeling Diffusion Models

Ask PDF

Abstract

Generative Modeling

Explore similar work

SurF: A Generative Model for Multivariate Irregular Time Series Forecasting

Conditional Attribute Estimation with Autoregressive Sequence Models

E4GEN: Event-level Explainable Extreme-Enhanced Time-series Generation