Methods used in mechanistic interpretability have revealed human-aligned concepts inside language and image models. In this paper, we search for physics concepts inside a neural a generative collider physics model trained on real LHC data (ShellFlow). We discover features inside ShellFlow that represent resonances, such as the Z, J/ψ, and Υ. The network was never told which muons come from a resonance, it was only trained to generate the kinematics of the final-state particles. We use these features to alter aspects of the Z boson: we change its mass between ∼80 and 105 GeV, and removing such a feature from the neural network produces a broader Z peak. In both cases the rest of the invariant mass spectrum remains mostly unchanged. The feature was built from the gradients of the neural network, and its selectivity was built using pattern recognition methods. A generative particle physics model trained on unlabeled data can be modified internally to change a physical quantity.
Figures & tables
Figure 1: Feynman diagram of the production of a μ−μ+ pair, through an intermediate vector boson V=J/ψ,Υ,Z,γ . X denotes the proton remnants and any additional radiation.
Figure 2: One transformer block of ShellFlow , block L (dotted frame). The steering vector is added to the residual stream at the output of the block. Inset: the output tokens of one event; only the μ− and μ+ tokens receive the steering vector, every other token is left unchanged.
Class
mμμ [GeV]
Events used for the gradients
Z
[81.19,101.19]
181,987
J/ψ
[2.9,3.3]
224,901
Υ
[9.2,10.6]
50,528
rest
elsewhere above 0.3
542,584
total
1,000,000
Table 1: The selection criteria of our classes of muon pairs, defined from the invariant mass of the generated pair of one seed. All events have exactly one pair of opposite-sign muons. The Υ class contains all three Υ resonances, as ShellFlow cannot properly generate three different resonances.
Figure 3: The steering of the Z boson mass. Top: dimuon invariant mass of one million muon pairs, compared with real data (grey), baseline generation of ShellFlow (black, dashed), and the results from adding a steering vector at block 1 with α=±1,±2 (blue to green). Middle: steered over baseline. Bottom: zoom-in over the different peaks: J/ψ , Υ , and Z bosons.
Figure 4: Muon kinematics under Z steering. 1,000,000 validation events of every final state, the Z steering vector at block 1 added to every muon: transverse momentum, pseudorapidity, and azimuth of the muons. Real data (grey), baseline generation (black, dashed), and steered with α=−2,−1,+1,+2 (blue to green); ratio panels: steered over baseline, drawn in every bin with baseline muons, each panel on its own scale. Every other particle type is in App. D.1 .
Figure 5: Fitted pole and width for different α as steered in block L=1 , from the lineshape fit of App. E . (a) Pole mass M . (b) Width Γ . The point at α=0 is the baseline fit. Dotted: the baseline value times exp(α∑kGˉZ(tk)⊤ξZ(tk)) , with the sum equal to 0.122, as predicted by Eq. ( 9 ). Error bars: standard deviation over three seeds of the sampler.
Figure 6: The Z steering quality features as a function of flow time t . (a) Gradient response of the Z , SZ(tk) , (b) the cosine similarity between the class-mean gradient of the Z and the steering vector, cos(GˉZ(tk),ξZ(tk)) , and (c) the largest eigenvalue of Eq. ( 18 ) for the Z , λZ(tk) .
L
∑kSZ(tk)
⟨cos(GˉZ,ξZ)⟩k
⟨λZ⟩k
0
1.4×10−1
1.4×10−1
0.92
1
1.2×10−1
1.5×10−1
0.91
2
7.7×10−2
1.4×10−1
0.91
3
4.2×10−2
1.1×10−1
0.93
4
1.1×10−2
5.1×10−2
0.95
5
2.6×10−5
2.4×10−4
0.93
Table 2: The Z steering vector at each block L . ∑kSZ(tk) is the first-order shift of logmμμ of the Z per unit α (Eq. ( 21 )), and ⟨⋅⟩k is the average over the 200 Euler steps.
Figure 7: Result of the generated muon spectrum from the ablation in Eq. ( 24 ) (blue), compared with the output from ShellFlow (black, dashed), and real data (grey). The inset shows the Z peak in linear scale, and the lower panel the ratio of ShellFlow to the ablation.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: The steering of the Z boson with the difference-of-means steering vector, as in Fig. 3 . Top: dimuon invariant mass of one million muon pairs, compared with real data (grey), baseline generation of ShellFlow (black, dashed), and the results from adding the steering vector at block 1 with α=±0.25,±0.5 . Middle: steered over baseline. Bottom: zoom-in over the J/ψ , Υ , and Z peaks.
Figure 9: The same as Fig. 8 for the Fisher steering vector, with α=±3,±4 (blue to green).
Figure 10: Fitted pole (a) and width (b) of the Z boson against α for the three steering vectors at block 1, as in Fig. 5 ; dotted: the first-order prediction of each. The Fisher steering vector skews the Z peak, so its fits describe the peak less well.
Channels
Architecture
Training
embedding
MLP
heads
blocks
parameters
FLOPs
data
events
epochs
lr
GPUs
ShellFlow
128
512
4
6
3.02M
5.5M
1LMET30 + 2to4lep
783M
30
6⋅10−4
8 A100
ShellFlow-B
128
512
4
6
3.02M
5.5M
2to4lep
268M
100
3⋅10−4
4 A100
ShellGATr
32 v + 96 s
128 v + 384 s
4
6
3.09M
6.2M
2to4lep
268M
100
3⋅10−4
4 B200
Appendix
Table 3: The three networks. ShellGATr counts its channels in four-vectors (v) and scalars (s). FLOPs: one forward pass per particle, measured at 8 particles per event.
Figure 11: Dimuon invariant mass of ShellFlow-B for the Z steering vector at block L=1 , for α=±1,±2 .
Figure 12: The same as Fig. 11 for ShellGATr , for α=±2,±4 .
Figure 13: Fitted pole (a) and width (b) against α for the Z steering vector at block L=1 of the three networks, as in Fig. 5 , mean of three seeds. Dotted: the baseline value of each network times exp(α∑kSZ(tk)) .
∑kSZ(tk)
⟨cos(GˉZ,ξZ)⟩k
⟨λZ⟩k
ShellFlow
0.122
0.147
0.91
ShellFlow-B
0.140
0.108
0.92
ShellGATr
0.047
0.111
0.91
Appendix
Table 4: The Z steering vector at block L=1 of the three networks, as in Table 2 .
Figure 14: Kinematics of the electrons, photons, jets, large-R jets, and hadronic taus after the steering on muon tokens at block L=1 that moves the Z boson peak. Same events and steering as in Fig. 4 : 1,000,000 validation events. Columns: transverse momentum, pseudorapidity, azimuth, energy, and, for the massive particles, mass; 60 bins per panel. Real data (grey), baseline generation (black, dashed), and steered with α=−2,−1,+1,+2 (blue to green). Ratio panels: steered over baseline.
Figure 15: The Z steering vector at block L=0 (left, α=±1,±2 ) and L=3 (right, α=±2,±3 ), as in Fig. 3 . Top: dimuon invariant mass of one million muon pairs, real data (grey), baseline generation of ShellFlow (black, dashed), and steered (blue to green). Middle: steered over baseline. Bottom: zooms on the J/ψ , Υ , and Z .
Figure 16: Fitted pole and width against α for the Z steering vector at blocks L=0 , 1, and 3, as in Fig. 5 . Dotted: the baseline value times exp(α∑kSZ(tk)) of each block (Table 2 ).
Figure 17: Steered generation compared to the baseline ShellFlow generation for different values of α=−2,−1,+1,+2 . The filled regions show the output of the steered model, and the red contour lines show the predicted steering of Eq. ( 20 ). The star is the result from Eq. ( 21 ).
Figure 18: The steering of the J/ψ mass, as in Fig. 3 : dimuon invariant mass of one million muon pairs, compared with real data (grey), baseline generation of ShellFlow (black, dashed), and the results from adding the J/ψ steering vector at block 1 with α=±1,±2 (blue to green).
Figure 19: The same as Fig. 18 for the Υ steering vector.
Class C
∑kSC(tk)
Z
0.122
J/ψ
0.063
Υ
0.123
Appendix
Table 5: ∑kSC(tk) of each class at block L=1 .
Figure 25Figure 26Figure 27
Figure 20: Lineshape fits of the Z peak. Real data (grey), baseline generation of ShellFlow (black, dashed), and the generation steered at block 1 with α=−2,−1,+1,+2 (blue to green), one seed. Each sample is drawn in the 100 bins of its own fit window, with the fitted lineshape as a thin line. Bottom: pulls, (ni−νi)/νi .
Figure 21: The remaining parameters of the lineshape fit against α , for the Z steering vector at block 1: (a) detector resolution σdet , (b) luminosity exponent ρ , and (c) deviance per degree of freedom. Mean over three seeds, with their standard deviation as bars. Dotted in (a): the baseline value times exp(α∑kGˉZ(tk)⊤ξZ(tk)) , a pure shift of logmμμ , as in Fig. 5 . At α=−3 the fit sets σdet to zero and puts all of the broadening into Γ .
Sample
M [GeV]
Γ [GeV]
σdet [GeV]
ρ
D/dof
PDG
91.1879±0.0020
2.4955±0.0023
–
–
–
.
Real data
90.979±0.009
2.87±0.03
1.60±0.02
−0.32±0.06
1.3
Baseline generation
90.573±0.009
2.41±0.02
1.74±0.02
−1.01±0.07
2.0
.
α=−2
70.299±0.011
4.37±0.03
1.25±0.03
0.76±0.05
15.1
Appendix
Table 6: Lineshape fits of the Z peak. Values are the mean over three seeds; uncertainties are statistical, from ΔD=1 . D/dof is the deviance per degree of freedom (95). PDG: world average of the Particle Data Group ( Particle Data Group, 2026 ) .
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.
Mechanistic interpretability seeks to reverse engineer a trained neural network by identifying the minimal subset of internal components. We perform a mechanistic interpretability analysis of the Particle Transformer architecture, trained on the Top Quark Tagging reference dataset, with the goal of identifying the computational circuit responsible for jet classification and characterizing the physical content of its internal representations. Combining zero ablation, path patching with two complementary on-manifold corruption strategies and linear probing of the residual stream, we identify a sparse six-head circuit that recovers the great majority of the full model performance while admitting a clean source-relay-readout interpretation. In this circuit, a single early layer head serves as the primary causal source, a cluster of middle-layer heads acts as relays selectively attending to hard pairwise substructure and a single late-layer head reads out the aggregated signal. Linear probes show that the residual stream is preferentially aligned with the energy correlator basis over the N-subjettiness basis. Within the energy correlator basis, the model preferentially encodes 2-prong substructure observables over the 3-prong observables. A per-layer trained probe further reveals that the apparent single step commitment of the model to a classification decision in the first class attention block is in fact a basis rotation, with the discriminating signal already saturating in the particle attention stack. These results demonstrate that mechanistic interpretability methods developed for natural language models can be used for jet physics classifiers and indicate that gradient descent may rediscover physically meaningful aspects of jet tagging without supervision.
Saurabh Rai, Sanmay Ganguly
Department of Physics, Indian Institute of Technology, Kanpur. Uttar-Pradesh 208016, India
In this work we demonstrate that a single transformer-based generative model can capture Standard Model structure spanning five decades of invariant mass, from the sub-GeV regime to the TeV continuum, a range that no single Monte Carlo sample covers. To achieve this we design ShellFlow, a Riemannian conditional flow matching model that, given the recorded event composition, generates each particle on its on-shell manifold. Its only physics priors are the on-shell condition and the invariant-mass formula. The model is trained on ∼109 real pp collision events from the ATLAS Open Data 13~TeV release and told nothing else. From a single training run, the model learns to reproduce all of the following: intra-particle kinematics, the dilepton resonances (J/ψ, Υ, Z) at their PDG positions, the leptonic Weinberg angle, the W and top-quark masses, and inter-particle correlations that enter no training objective. A substantial fraction of the Standard Model is thus learnable directly from recorded collision data.
Midori Kato, Kevin A. Urquía-Calderón, Inar Timiryasov +1
Niels Bohr Institute, University of Copenhagen, Jagtvej 155A, DK-2200, Copenhagen, Denmark
Neural networks (NNs) are inherently multidimensional classifiers that learn complex, non-linear relationships among input observables. While their flexibility enables unprecedented performance in high-energy physics (HEP) analyses, it also makes them sensitive to small variations in their inputs. Consequently, the propagation and estimation of systematic uncertainties in NN-based models remain an open challenge. There are indications that uncertainties derived in control regions or from nominal variations of input features can underestimate the true model uncertainty, potentially leaving biases unaccounted for. Inspired by insights from adversarial-attack studies in machine learning, we explore how subtle perturbations, fully consistent with the experimental uncertainties on the input observables, can lead to substantial changes in NN outputs, while keeping the one-dimensional and correlated input distributions nearly unchanged. Using a set of representative HEP tasks, including event classification and object identification, and testing across a variety of network architectures, we demonstrate that networks can be systematically "fooled" at significant rates within the allowed uncertainty envelopes. Building on this observation, we introduce a quantitative framework to probe and measure the hidden sensitivity of neural networks to realistic experimental variations, providing a practical path to evaluate and control their systematic uncertainty in physics analyses.
Lucie Flek, Philipp Alexander Jungs, Akbar Karimi +6
Bonn-Aachen Institute of Technology, University of Bonn, Germany · Physics Institute III A, RWTH Aachen University, Germany · Institute of Physics, University of Bonn, Germany +1