Scaling Collider Event Generation with Residual-Quantized Tokens

Dan Godi; Dmitrii Kobylianskii; Eilam Gross

hep-exSep 30, 2026

Scaling Collider Event Generation with Residual-Quantized Tokens

Authors: Dan Godi, Dmitrii Kobylianskii, Eilam Gross

Organizations: Weizmann Institute of Science, Rehovot, Israel

Abstract

Full detector simulation and reconstruction of collider events are projected to become major bottlenecks at the High-Luminosity Large Hadron Collider, motivating the development of fast, ML-based surrogates. At the same time, LLMs have driven fast progress in generative discrete modeling: autoregressive transformers trained on tokenized data now represent the state of the art across a range of generative tasks. We extend the discrete modeling paradigm by introducing a particle-level generative model trained on residual-quantized full-event data. We demonstrate the ability of this model family to perform conditional generation from detector-stable particles; we study its scaling behavior across a range of dataset and model sizes, characterize the effects of repeated data exposure and demonstrate that token-level loss systematically predicts downstream physical fidelity. These results provide an empirical framework for scalable collider full-event generation based on residual-quantized representations.

Figures & tables

Dataset	$p_{\mathrm{T}}^{\min}$ – $p_{\mathrm{T}}^{\max}$ [GeV]	Type	Training [evts]	Testing [evts]
$H\rightarrow 4\ell$	–	Out-of-distribution		$50{,}000$
$t\bar{t}$	–	In distribution	$9{,}361{,}338$	$28{,}263$
QCD	470–600	In distribution	$3{,}737{,}811$	$11{,}285$
QCD	600–800	In distribution	$3{,}462{,}168$	$10{,}452$
QCD	1000–1400	Out-of-distribution		$50{,}000$

Table 1 : Datasets used for training and testing, in-distribution testing slices sum up to

50,000

events. Parton-level

p_{T}

ranges are shown for QCD samples.

Figure 1 : Reconstructed features residual distribution, and particle-class confusion matrix. Vertex residuals are only computed for charged particles.

Dataset	Usage	Perplexity	Usage	Perplexity	Usage	Perplexity	Usage	Perplexity
	$m=1$		$m=2$		$m=3$		$m=4$
QCD + $t\bar{t}$	58.5%	4042	99.9%	5575	99.9%	4894	99.9%	4723
$H\rightarrow 4\ell$	58.2%	3338	99.3%	5382	99.4%	4797	99.6%	4662
QCD 1000	58.5%	3897	99.9%	5652	100.0%	4981	100.0%	4871

Table 2 : Codebook usage and perplexity (PPL) at each residual depth

m

, for the three test datasets.

Figure 2 : Spectra of particle residual features for all three testing datasets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Figure 3 : Spectra of event-level residual features for all three testing datasets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Figure 4 : Spectra of residuals of jet-level features for all three testing datasets. The residuals are computed against matched truth jets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Name	$\boldsymbol{d_{\text{model}}}$	Layers	Heads	$\boldsymbol{d_{\text{head}}}$	$\boldsymbol{d_{\text{ff}}}$	Total Params
10M	256	8	8	32	512	$9.83$ M
22M	384	10	8	48	768	$21.67$ M
41M	512	12	8	64	1,024	$40.75$ M
97M	768	14	12	64	1,536	$96.71$ M
187M	1,024	16	16	64	2,048	$186.87$ M
1B	2,048	24	32	64	4,096	$1{,}046.92$ M

Table 3 : Model configurations used in the scaling study. Each model is trained with eight data and repetition settings, giving 48 configurations.

Data fraction	Unique events	Epochs	PF tokens	Total tokens
3.3%	$546{,}523$	$1,\ 10,\ 30$	$324$ M	$749$ M
33.3%	$5{,}514{,}918$	$1,\ 3,\ 12$	$3.27$ B	$7.56$ B
100%	$16{,}561{,}317$	$1,\ 4$	$9.82$ B	$22.7$ B

Table 4 : Data and repetition settings used in the scaling study. Each setting is applied to all six model sizes. Data fractions are subsampled from the full dataset, in order to preserve the same statistical properties.

Law	Functional form	$\alpha$	$\beta$	$a$	$b$	MAPE [%]	$R^{2}$
Chinchilla	$L_{\infty}+AN^{-\alpha}+BU^{-\beta}$	$0.42\,{\scriptstyle\pm 0.18}$	$0.62\,{\scriptstyle\pm 0.15}$	$0.60\,{\scriptstyle\pm 0.14}$	$0.40\,{\scriptstyle\pm 0.14}$	3.55	0.9695
Skaling	$L_{\infty}+\left(AN^{-\alpha}+BU^{-\beta}\right)^{k}$	$0.241\,{\scriptstyle\pm 0.052}$	$0.345\,{\scriptstyle\pm 0.070}$	$0.589\,{\scriptstyle\pm 0.048}$	$0.411\,{\scriptstyle\pm 0.048}$	1.17	0.9961
Prescriptive	$L_{\infty}+AN^{-\alpha}+B[U(1+R)]^{-\beta}+PR^{\delta}\left(\frac{N}{U^{\gamma}}\right)^{\kappa}$	$0.410\,{\scriptstyle\pm 0.092}$	$0.598\,{\scriptstyle\pm 0.036}$	$0.593\,{\scriptstyle\pm 0.074}$	$0.407\,{\scriptstyle\pm 0.074}$	2.47	0.9690
Custom	$L_{\infty}+\left(AN^{-\alpha}+B[U(1+R)]^{-\beta}\right)^{k}+PR^{\delta}\left(\frac{N}{U^{\gamma}}\right)^{\kappa}$	$0.273\,{\scriptstyle\pm 0.017}$	$0.349\,{\scriptstyle\pm 0.016}$	$0.561\,{\scriptstyle\pm 0.015}$	$0.439\,{\scriptstyle\pm 0.015}$	0.79	0.9967

Table 5 : Fits of the neural scaling laws. Chinchilla and Skaling are fitted to the 18 single-epoch runs, the repetition-aware laws to all 48 runs. Uncertainties are

1\sigma

errors over refits with one run held-out.

Figure 5 : Left: each trained model’s NTP loss against its parameter count. The lines show the custom fitted law’s prediction for the given combination of dataset size and repetition. Right: compute-guided fit-predicted optimal scaling behaviors and isoFLOP curves minima. The methodology for recovering these minima is discussed in the Appendix C.2 .

Figure 6 : Measured Sliced Wasserstein Distance for all valid (selection criteria in Appendix C.3 ) runs, and fitted law in the form of Equation 4 . We include both final and intermediate checkpoints and show the fitted power law parameters for each feature.

Figure 7 : Per-feature

W_{1}

Wasserstein Distance between the PF and FS valid (selection criteria in Appendix C.3 ) residual distributions. We include both final and intermediate checkpoints and show the fitted power law parameters for each feature.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

	$\log(1+p_T)$	$\eta$	$v_{x}$	$v_{y}$	$v_{z}$
mean $\mu$	1.13	0.00	0.206	0.331	0.34
std. $\sigma$	0.83	1.29	0.691	0.702	5.73

Appendix

Table 6 : Fixed feature standardization.

p_{T}

is transformed as

\log(1+p_T)

before standardization;

\phi

is replaced by

(\sin\phi,\cos\phi)

Model	Transformer	Code proj.	Token head	Card. head	Embedding tables	$N$
10M	5.244	0.004	2.106	0.169	2.305	9.827
22M	14.747	0.007	3.155	0.302	3.457	21.668
41M	31.460	0.009	4.204	0.468	4.609	40.750
97M	82.578	0.013	6.302	0.899	6.914	96.706
187M	167.775	0.017	8.400	1.461	9.218	186.871
1B	1006.638	0.035	16.792	5.018	18.436	1046.918

Appendix

Table 7 : Parameter breakdown of the generator models (in millions).

N

is the total used for fitting.

Model	$\boldsymbol{B}$	evt/s	ms/evt	$\boldsymbol{B}$	evt/s	ms/evt
	A100			H200
10M	1,024	126.5	7.9	2,048	265.1	3.8
22M †	512	15.1	66.1	1,024	26.8	37.4
41M	256	62.5	16.0	512	132.5	7.5
97M	256	38.2	26.2	256	72.7	13.8
187M	128	22.0	45.4	256	48.8	20.5

Appendix

Table 8 : Generation throughput of the best run of each model size, measured on

50{,}000

held-out events. Times include quantization, prefill, autoregressive decode and the RQ-VAE decoding. Each model is tested at the biggest power-of-two batch size

B

to fit in GPU memory with an

8\%

margin. The models are tested on NVIDIA A100 SXM4 (

80

GB) and NVIDIA H200 (

141

GB) GPUs.

	Chinchilla	Skaling	Prescriptive	Custom
$L_{\infty}$	$2.67\pm 0.40$	$3.191\pm 0.082$	$2.80\pm 0.11$	$3.255\pm 0.015$
$A$	$1300\pm 4100$	$34\pm 31$	$900\pm 1200$	$55\pm 14$
$\alpha$	$0.42\pm 0.18$	$0.241\pm 0.052$	$0.410\pm 0.092$	$0.273\pm 0.017$
$B$	$(0.9\pm 2.2)\times 10^{4}$	$106\pm 93$	$7400\pm 3900$	$114\pm 24$
$\beta$	$0.62\pm 0.15$	$0.345\pm 0.070$	$0.598\pm 0.036$	$0.349\pm 0.016$
$k$	–	$2.39\pm 0.32$	–	$2.379\pm 0.072$

Appendix

Table 9 : Fitted parameters of the loss scaling laws. Chinchilla and Skaling are fitted to the 18 single-epoch runs, Prescriptive and Custom to all 48 runs;

N

and

U

are in parameters and unique events. Uncertainties are

1\sigma

errors over refits with one held-out run.

Figure 8 : IsoFLOP slices fit through the measured loss values. For every slice we show the minima and the compute value.

Figure 9 : Usage distributions across the testing datasets. Codes are ordered on the $x$ axis according to their usage rank, from most to least used.

Figure 10 : Spectra of particle-level features for all three testing datasets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Figure 11 : Spectra of jet-level features for all three testing datasets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Figure 12 : Spectra of event-level features for all three testing datasets. All histograms are normalized to unity; rows correspond to different datasets/processes while columns correspond to different observables.

Event residuals
Residual	Unit	10M	22M	41M	97M	187M	1B	RQ-VAE	Noise
$\Delta H_{T}/H_{T}$	–	0.0212	0.0154	0.0102	0.00538	0.00526	0.00262	0.00144	0.00132
$\Delta E_{x}^{\mathrm{miss}}$	GeV	17.3	9.98	6.69	3.81	2.97	1.81	0.363	0.271
$\Delta E_{y}^{\mathrm{miss}}$	GeV	17.1	10.3	6.82	3.77	3.09	1.79	0.251	0.347
$\Delta E_{T}^{\mathrm{miss}}$	GeV	22.9	13.2	8.60	4.62	3.78	2.09	0.439	0.491
$\Delta N/N$	–	0.0195	0.0131	0.00940	0.0130	0.00629	0.00513	0.00793	0.00509

Appendix

Table 10 : Per-feature

W_{1}

distance between the FS and PF residual distributions, in the units of each feature, for the models trained on the full dataset for

4

epochs, on the held-out in-distribution test set (

50{,}000

events). The RQ-VAE column is the tokenizer reconstruction of an independent

50{,}000

-event sample with the same process mix, and the noise column the distance between two such independent PF samples. The best model in each row is in bold.

Explore similar work

Neural Scaling Laws for Jet Generation
May 27, 2026Oz Amram, Darius A. Faroughy, Tjarko Gerdes +5Scaling Laws Scaling
1Fermi National Accelerator Laboratory, Batavia, IL 60510, USA · 2NHETC, Dept. of Physics and Astronomy, Rutgers University, Piscataway, NJ 08854, USA · Institute for Experimental Physics, Universität Hamburg, Luruper Chaussee 149, 22761 Hamburg, Germany +1
GPT-Based Fast Simulation of CLAS12 Detector Hits via Conditional Autoregressive Generation
Jun 14, 2026Cole Granger, James Giroux, Richard Tyson +2Autoregressive Generation Autoregressive Transformers
William & Mary, Department of Data Science, Williamsburg, VA 23185, USA · University of Glasgow, School of Physics and Astronomy, Glasgow G12 8QQ,14 UK · Thomas Jefferson National Accelerator Facility, Newport News, VA 23606, USA +1
AllShowers: One model for all calorimeter showers
Jan 16, 2026Thorsten Buss, Henry Day-Hall, Frank Gaede +2Generative Models Surrogate Models
Institut für Experimentalphysik, Universität Hamburg, Hamburg, Germany · Deutsches Elektronen-Synchrotron DESY, Hamburg, Germany

May 27, 2026, hep-phJ/K move · Enter open · S save

Neural Scaling Laws for Jet Generation

Oz Amram, Darius A. Faroughy, Tjarko Gerdes, Anna Hallin+4

1Fermi National Accelerator Laboratory, Batavia, IL 60510, USA · 2NHETC, Dept. of Physics and Astronomy, Rutgers University, Piscataway, NJ 08854, USA · Institute for Experimental Physics, Universität Hamburg, Luruper Chaussee 149, 22761 Hamburg, Germany+1

Scaling Laws Scaling Autoregressive Token Prediction Downstream Fine-Tuning

Open paper AskPDF arXiv

Recently observed empirical scaling laws describe the performance of foundation-type models as three independent key quantities -- dataset size, compute, and model parameters -- are modified. Extracting these scaling laws informs the training of large complex models for which the tuning of hyperparameters in traditional ways is not feasible. This work for the first time explores if scaling laws can also be observed for the task of particle jet generation -- both relevant as a pre-training objective for foundation models and as in-situ simulation by itself. We indeed replicate the key logarithmic scaling law behavior for model-size scaling. Beyond studying the next token prediction validation loss of the generative model, we also study the sliced Wasserstein distance of five physical quantities that are not immediately available to the model during training. Our study shows that this quantity is monotonically related to the next token prediction validation loss, meaning that this loss is indeed a good proxy for the physics performance. For the scaling with dataset size and compute, we observe substantially weaker scaling behavior of both the loss and the sliced Wasserstein distance. We analyze this behavior by introducing the concept of a learnable window, and argue that autoregressive next token prediction on jet constituents exhibits comparatively rapid saturation relative to language-model studies. We discuss possible origins of this behavior, including the stochastic nature of QCD radiation and differences between generative and supervised learning tasks in collider physics.

Abstract

Explore similar work

Neural Scaling Laws for Jet Generation

GPT-Based Fast Simulation of CLAS12 Detector Hits via Conditional Autoregressive Generation

AllShowers: One model for all calorimeter showers