Scaling Collider Event Generation with Residual-Quantized Tokens

Dan Godi; Dmitrii Kobylianskii; Eilam Gross

hep-exSep 30, 2026

Scaling Collider Event Generation with Residual-Quantized Tokens

Authors: Dan Godi, Dmitrii Kobylianskii, Eilam Gross

Organizations: Weizmann Institute of Science, Rehovot, Israel

Abstract

Full detector simulation and reconstruction of collider events are projected to become major bottlenecks at the High-Luminosity Large Hadron Collider, motivating the development of fast, ML-based surrogates. At the same time, LLMs have driven fast progress in generative discrete modeling: autoregressive transformers trained on tokenized data now represent the state of the art across a range of generative tasks. We extend the discrete modeling paradigm by introducing a particle-level generative model trained on residual-quantized full-event data. We demonstrate the ability of this model family to perform conditional generation from detector-stable particles; we study its scaling behavior across a range of dataset and model sizes, characterize the effects of repeated data exposure and demonstrate that token-level loss systematically predicts downstream physical fidelity. These results provide an empirical framework for scalable collider full-event generation based on residual-quantized representations.

Figures & tables

Dataset	$p_{\mathrm{T}}^{\min}$ – $p_{\mathrm{T}}^{\max}$ [GeV]	Type	Training [evts]	Testing [evts]
$H\rightarrow 4\ell$	–	Out-of-distribution		$50{,}000$
$t\bar{t}$	–	In distribution	$9{,}361{,}338$	$28{,}263$
QCD	470–600	In distribution	$3{,}737{,}811$	$11{,}285$
QCD	600–800	In distribution	$3{,}462{,}168$	$10{,}452$
QCD	1000–1400	Out-of-distribution		$50{,}000$

Table 1 : Datasets used for training and testing, in-distribution testing slices sum up to

50,000

events. Parton-level

p_{T}

ranges are shown for QCD samples.

Figure 1 : Reconstructed features residual distribution, and particle-class confusion matrix. Vertex residuals are only computed for charged particles.

Dataset	Usage	Perplexity	Usage	Perplexity	Usage	Perplexity	Usage	Perplexity
	$m=1$		$m=2$		$m=3$		$m=4$
QCD + $t\bar{t}$	58.5%	4042	99.9%	5575	99.9%	4894	99.9%	4723
$H\rightarrow 4\ell$	58.2%	3338	99.3%	5382	99.4%	4797	99.6%	4662
QCD 1000	58.5%	3897	99.9%	5652	100.0%	4981	100.0%	4871

Table 2 : Codebook usage and perplexity (PPL) at each residual depth

m

, for the three test datasets.

Figure 2 : Spectra of particle residual features for all three testing datasets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Figure 3 : Spectra of event-level residual features for all three testing datasets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Figure 4 : Spectra of residuals of jet-level features for all three testing datasets. The residuals are computed against matched truth jets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Name	$\boldsymbol{d_{\text{model}}}$	Layers	Heads	$\boldsymbol{d_{\text{head}}}$	$\boldsymbol{d_{\text{ff}}}$	Total Params
10M	256	8	8	32	512	$9.83$ M
22M	384	10	8	48	768	$21.67$ M
41M	512	12	8	64	1,024	$40.75$ M
97M	768	14	12	64	1,536	$96.71$ M
187M	1,024	16	16	64	2,048	$186.87$ M
1B	2,048	24	32	64	4,096	$1{,}046.92$ M

Table 3 : Model configurations used in the scaling study. Each model is trained with eight data and repetition settings, giving 48 configurations.

Data fraction	Unique events	Epochs	PF tokens	Total tokens
3.3%	$546{,}523$	$1,\ 10,\ 30$	$324$ M	$749$ M
33.3%	$5{,}514{,}918$	$1,\ 3,\ 12$	$3.27$ B	$7.56$ B
100%	$16{,}561{,}317$	$1,\ 4$	$9.82$ B	$22.7$ B

Table 4 : Data and repetition settings used in the scaling study. Each setting is applied to all six model sizes. Data fractions are subsampled from the full dataset, in order to preserve the same statistical properties.

Law	Functional form	$\alpha$	$\beta$	$a$	$b$	MAPE [%]	$R^{2}$
Chinchilla	$L_{\infty}+AN^{-\alpha}+BU^{-\beta}$	$0.42\,{\scriptstyle\pm 0.18}$	$0.62\,{\scriptstyle\pm 0.15}$	$0.60\,{\scriptstyle\pm 0.14}$	$0.40\,{\scriptstyle\pm 0.14}$	3.55	0.9695
Skaling	$L_{\infty}+\left(AN^{-\alpha}+BU^{-\beta}\right)^{k}$	$0.241\,{\scriptstyle\pm 0.052}$	$0.345\,{\scriptstyle\pm 0.070}$	$0.589\,{\scriptstyle\pm 0.048}$	$0.411\,{\scriptstyle\pm 0.048}$	1.17	0.9961
Prescriptive	$L_{\infty}+AN^{-\alpha}+B[U(1+R)]^{-\beta}+PR^{\delta}\left(\frac{N}{U^{\gamma}}\right)^{\kappa}$	$0.410\,{\scriptstyle\pm 0.092}$	$0.598\,{\scriptstyle\pm 0.036}$	$0.593\,{\scriptstyle\pm 0.074}$	$0.407\,{\scriptstyle\pm 0.074}$	2.47	0.9690
Custom	$L_{\infty}+\left(AN^{-\alpha}+B[U(1+R)]^{-\beta}\right)^{k}+PR^{\delta}\left(\frac{N}{U^{\gamma}}\right)^{\kappa}$	$0.273\,{\scriptstyle\pm 0.017}$	$0.349\,{\scriptstyle\pm 0.016}$	$0.561\,{\scriptstyle\pm 0.015}$	$0.439\,{\scriptstyle\pm 0.015}$	0.79	0.9967

Table 5 : Fits of the neural scaling laws. Chinchilla and Skaling are fitted to the 18 single-epoch runs, the repetition-aware laws to all 48 runs. Uncertainties are

1\sigma

errors over refits with one run held-out.

Figure 5 : Left: each trained model’s NTP loss against its parameter count. The lines show the custom fitted law’s prediction for the given combination of dataset size and repetition. Right: compute-guided fit-predicted optimal scaling behaviors and isoFLOP curves minima. The methodology for recovering these minima is discussed in the Appendix C.2 .

Figure 6 : Measured Sliced Wasserstein Distance for all valid (selection criteria in Appendix C.3 ) runs, and fitted law in the form of Equation 4 . We include both final and intermediate checkpoints and show the fitted power law parameters for each feature.

Figure 7 : Per-feature

W_{1}

Wasserstein Distance between the PF and FS valid (selection criteria in Appendix C.3 ) residual distributions. We include both final and intermediate checkpoints and show the fitted power law parameters for each feature.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	$\log(1+p_T)$	$\eta$	$v_{x}$	$v_{y}$	$v_{z}$
mean $\mu$	1.13	0.00	0.206	0.331	0.34
std. $\sigma$	0.83	1.29	0.691	0.702	5.73

Appendix

Table 6 : Fixed feature standardization.

p_{T}

is transformed as

\log(1+p_T)

before standardization;

\phi

is replaced by

(\sin\phi,\cos\phi)

Model	Transformer	Code proj.	Token head	Card. head	Embedding tables	$N$
10M	5.244	0.004	2.106	0.169	2.305	9.827
22M	14.747	0.007	3.155	0.302	3.457	21.668
41M	31.460	0.009	4.204	0.468	4.609	40.750
97M	82.578	0.013	6.302	0.899	6.914	96.706
187M	167.775	0.017	8.400	1.461	9.218	186.871
1B	1006.638	0.035	16.792	5.018	18.436	1046.918

Appendix

Table 7 : Parameter breakdown of the generator models (in millions).

N

is the total used for fitting.

Model	$\boldsymbol{B}$	evt/s	ms/evt	$\boldsymbol{B}$	evt/s	ms/evt
	A100			H200
10M	1,024	126.5	7.9	2,048	265.1	3.8
22M †	512	15.1	66.1	1,024	26.8	37.4
41M	256	62.5	16.0	512	132.5	7.5
97M	256	38.2	26.2	256	72.7	13.8
187M	128	22.0	45.4	256	48.8	20.5

Appendix

Table 8 : Generation throughput of the best run of each model size, measured on

50{,}000

held-out events. Times include quantization, prefill, autoregressive decode and the RQ-VAE decoding. Each model is tested at the biggest power-of-two batch size

B

to fit in GPU memory with an

8\%

margin. The models are tested on NVIDIA A100 SXM4 (

80

GB) and NVIDIA H200 (

141

GB) GPUs.

	Chinchilla	Skaling	Prescriptive	Custom
$L_{\infty}$	$2.67\pm 0.40$	$3.191\pm 0.082$	$2.80\pm 0.11$	$3.255\pm 0.015$
$A$	$1300\pm 4100$	$34\pm 31$	$900\pm 1200$	$55\pm 14$
$\alpha$	$0.42\pm 0.18$	$0.241\pm 0.052$	$0.410\pm 0.092$	$0.273\pm 0.017$
$B$	$(0.9\pm 2.2)\times 10^{4}$	$106\pm 93$	$7400\pm 3900$	$114\pm 24$
$\beta$	$0.62\pm 0.15$	$0.345\pm 0.070$	$0.598\pm 0.036$	$0.349\pm 0.016$
$k$	–	$2.39\pm 0.32$	–	$2.379\pm 0.072$

Appendix

Table 9 : Fitted parameters of the loss scaling laws. Chinchilla and Skaling are fitted to the 18 single-epoch runs, Prescriptive and Custom to all 48 runs;

N

and

U

are in parameters and unique events. Uncertainties are

1\sigma

errors over refits with one held-out run.

Figure 8 : IsoFLOP slices fit through the measured loss values. For every slice we show the minima and the compute value.

Figure 9 : Usage distributions across the testing datasets. Codes are ordered on the $x$ axis according to their usage rank, from most to least used.

Figure 10 : Spectra of particle-level features for all three testing datasets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Figure 11 : Spectra of jet-level features for all three testing datasets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Figure 12 : Spectra of event-level features for all three testing datasets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Event residuals
Residual	Unit	10M	22M	41M	97M	187M	1B	RQ-VAE	Noise
$\Delta H_{T}/H_{T}$	–	0.0212	0.0154	0.0102	0.00538	0.00526	0.00262	0.00144	0.00132
$\Delta E_{x}^{\mathrm{miss}}$	GeV	17.3	9.98	6.69	3.81	2.97	1.81	0.363	0.271
$\Delta E_{y}^{\mathrm{miss}}$	GeV	17.1	10.3	6.82	3.77	3.09	1.79	0.251	0.347
$\Delta E_{T}^{\mathrm{miss}}$	GeV	22.9	13.2	8.60	4.62	3.78	2.09	0.439	0.491
$\Delta N/N$	–	0.0195	0.0131	0.00940	0.0130	0.00629	0.00513	0.00793	0.00509

Appendix

Table 10 : Per-feature

W_{1}

distance between the FS and PF residual distributions, in the units of each feature, for the models trained on the full dataset for

4

epochs, on the held-out in-distribution test set (

50{,}000

events). The RQ-VAE column is the tokenizer reconstruction of an independent

50{,}000

-event sample with the same process mix, and the noise column the distance between two such independent PF samples. The best model in each row is in bold.

Explore similar work

May 27, 2026hep-ph

Neural Scaling Laws for Jet Generation

Recently observed empirical scaling laws describe the performance of foundation-type models as three independent key quantities -- dataset size, compute, and model parameters -- are modified. Extracting these scaling laws informs the training of large complex models for which the tuning of hyperparameters in traditional ways is not feasible. This work for the first time explores if scaling laws can also be observed for the task of particle jet generation -- both relevant as a pre-training objective for foundation models and as in-situ simulation by itself. We indeed replicate the key logarithmic scaling law behavior for model-size scaling. Beyond studying the next token prediction validation loss of the generative model, we also study the sliced Wasserstein distance of five physical quantities that are not immediately available to the model during training. Our study shows that this quantity is monotonically related to the next token prediction validation loss, meaning that this loss is indeed a good proxy for the physics performance. For the scaling with dataset size and compute, we observe substantially weaker scaling behavior of both the loss and the sliced Wasserstein distance. We analyze this behavior by introducing the concept of a learnable window, and argue that autoregressive next token prediction on jet constituents exhibits comparatively rapid saturation relative to language-model studies. We discuss possible origins of this behavior, including the stochastic nature of QCD radiation and differences between generative and supervised learning tasks in collider physics.

Oz Amram, Darius A. Faroughy, Tjarko Gerdes +5

1Fermi National Accelerator Laboratory, Batavia, IL 60510, USA · 2NHETC, Dept. of Physics and Astronomy, Rutgers University, Piscataway, NJ 08854, USA · Institute for Experimental Physics, Universität Hamburg, Luruper Chaussee 149, 22761 Hamburg, Germany +1

Scaling Laws Scaling Autoregressive Token Prediction Downstream Fine-Tuning

Ask PDF

Jun 14, 2026physics.ins-det

GPT-Based Fast Simulation of CLAS12 Detector Hits via Conditional Autoregressive Generation

Modern particles physics experiments have demonstrated an increasing need for fast, high-fidelity detector simulation as detector components have improved and subsequent computational requirements approach the limits of available resources. Recently, deep generative models have emerged as a promising alternative to traditional Monte-Carlo methods, with recent works drawing inspiration from large language models (LLMs) and self-supervised next-token prediction methods. In this work, we present an application of a GPT-style autoregressive transformer as a fast surrogate model for the calorimeter inside the CLAS12 experiment at the Thomas Jefferson National Accelerator Facility. The model is conditioned on incident momentum and generates realistic detector hits autoregressively across all nine calorimeter layers as sequences of strip, ADC, and TDC tokens. We demonstrate that the model faithfully reproduces hit multiplicity, spatial distributions, energy deposits, and the energy-momentum response of the electromagnetic calorimeter. The generator achieves inference rates exceeding 700 events per second on a single GPU, providing a substantial speedup over traditional Geant4-based simulations while maintaining physics fidelity essential for high-luminosity experimental programs.

Cole Granger, James Giroux, Richard Tyson +2

William & Mary, Department of Data Science, Williamsburg, VA 23185, USA · University of Glasgow, School of Physics and Astronomy, Glasgow G12 8QQ,14 UK · Thomas Jefferson National Accelerator Facility, Newport News, VA 23606, USA +1

Autoregressive Generation Autoregressive Transformers Surrogate Models Momentum

Ask PDF

Jan 16, 2026physics.ins-det

AllShowers: One model for all calorimeter showers

Accurate and efficient detector simulation is essential for modern collider experiments. To reduce the high computational cost, various fast machine learning surrogate models have been proposed. Traditional surrogate models for calorimeter shower modeling train separate networks for each particle species, limiting scalability and reuse. We introduce AllShowers, a unified generative model that simulates calorimeter showers across multiple particle types using a single generative model. AllShowers is a continuous normalizing flow model with a Transformer architecture, enabling it to generate complex spatial and energy correlations in variable-length point cloud representations of showers. Trained on a diverse dataset of simulated showers in the highly granular ILD detector, the model demonstrates the ability to generate realistic showers for electrons, photons, and charged and neutral hadrons across a wide range of incident energies and angles without retraining. In addition to unifying shower generation for multiple particle types, AllShowers surpasses the fidelity of previous single-particle-type models for hadronic showers. Key innovations include the use of a layer embedding, allowing the model to learn all relevant calorimeter layer properties; a custom attention masking scheme to reduce computational demands and introduce a helpful inductive bias; and a shower- and layer-wise optimal transport mapping to improve training convergence and sample quality. AllShowers marks a significant step towards a universal model for calorimeter shower simulations in collider experiments.

Thorsten Buss, Henry Day-Hall, Frank Gaede +2

Institut für Experimentalphysik, Universität Hamburg, Hamburg, Germany · Deutsches Elektronen-Synchrotron DESY, Hamburg, Germany

Generative Models Surrogate Models

Ask PDF