Model-Agnostic Search for New Physics in the Presence of Imperfect Background by Smearing the Data

Soheun Yi; John Alison; Mikael Kuusela

stat.APOct 29, 2026

Model-Agnostic Search for New Physics in the Presence of Imperfect Background by Smearing the Data

Authors: Soheun Yi, John Alison, Mikael Kuusela

Organizations: Department of Statistics and Data Science, Carnegie Mellon University · Department of Physics, Carnegie Mellon University

Abstract

In the search for new particles in high-energy physics, it is necessary to first define the signal region (SR) enriched in signal events, and then compare events in this region with the expected background distribution. While most existing search methods rely on prior domain knowledge of either the signal or the background, there has been less effort to relax both assumptions simultaneously, as methods that weaken one assumption typically rely on strong knowledge of the other. We propose Model-Agnostic Search with Smearing and Imperfect Background (MASSIB), a method that addresses this challenge by learning the density ratio of events relative to a misspecified background distribution using modern tools from machine learning, and defining the SR as the region with high density ratio. Recognizing that a naive application of this idea introduces significant bias in background estimation when interpolating from the control region to the SR, MASSIB mitigates this issue by introducing a "smeared" density ratio, which can be learned efficiently through data-level manipulation and used to define an SR that allows for accurate background estimation. By applying it to various types of simulated Large Hadron Collider (LHC) events with realistic background mismodeling, we demonstrate that MASSIB identifies a data-driven SR that achieves high signal concentration consistently across different types of signals while avoiding bias in background estimation. The method does this without making strong assumptions about either the signal or background distribution. We then show that this robustness extends to the validity and strong statistical power of the resulting hypothesis tests for signal detection, demonstrating the potential of this approach for future model-agnostic searches at the LHC.

Figures & tables

Fig 1: Classification of methods by how strong their assumptions on the signal and background are. Traditional methods, including Higgs boson detection ( ATLAS, 2012 ; CMS, 2012 ) , rely on strong assumptions about both the signal and background. Some recent methods relax the assumptions on either the signal ( Chakravarti et al., 2023 ) or the background ( Manole et al., 2024 ) , but still impose a strong assumption on the other component. In contrast, this work introduces a new approach built upon weaker assumptions on both the signal and background.

Panel of Figure 2 — Fig 2: A contingency table explaining the ABCD method. The four regions $A$ , $B$ , $C$ , and $D$ are defined based on whether an event belongs to the imperfect background or the target events and whether it is in the SR or CR. Given that the imperfect background does not include any signal events and that the two axes are uncorrelated, we can estimate the number of background events in region $D$ by $N_{D}\approx N_{B}N_{C}/N_{A}$ .

Fig 3: An overview of MASSIB. The procedure begins by estimating the density ratio $\gamma$ between the data and imperfect background (IB) in the first split of the dataset. Then, the representation $\zeta$ is smeared and used to learn the density ratio $\widetilde{\gamma}$ between the smeared data and smeared IB in the same split. The SR and CR are then defined by thresholding on $\psi=\gamma/\widetilde{\gamma}$ . Next, the density ratio of the data and IB in the CR, denoted as $\gamma_{\text{CR}}$ , is estimated. Finally, the estimated background $\gamma_{\text{CR}}p_{\text{IB}}$ is compared with the data distribution $p_{\text{data}}$ within the SR via hypothesis tests.

Panel of Figure 4 — Fig 4: Abstraction of the Four-vs-Three (FvT) classifier architecture ( Manole et al., 2024 ) . $x$ denotes the original input features, $\zeta=\zeta(x)$ is the representation, and $\widehat{\gamma}$ is the output of the classifier predicting the density ratio of $4b$ events to $3b$ events.

Fig 5: Histograms of classifier outputs trained on a dataset with signal ratio

\epsilon=0.01

. Even when trained on the same training dataset, due to the stochasticity of the training, a classifier may or may not capture signal events by identifying a region where the

\mathrm{HH}\to 4b

signal events dominate over the background

4b

events, as illustrated in the left and right panels.

Fig 6: Proportion of signal events (

S({\mathcal{X}}_{s})

) versus that of

4b

events (

P_{4b}({\mathcal{X}}_{s})

) in the SR (

{\mathcal{X}}_{s}

) with varying thresholds for ensembles of

\widehat{\psi}

. The signal ratio is set to

\epsilon=0.01

and the noise scales are set to

\eta=\infty

(i.e.,

\widehat{\psi}=\widehat{\gamma}

is an ensemble maximum of density ratios) and

\eta=2.0

. The maximum and mean of

\widehat{\psi}

for

15

different training instances are used to define the SR. The plot illustrates that the ensemble maximum is more effective than the ensemble mean in enriching the signal events in the SR for both smearing and non-smearing cases.

Panel of Figure 7 — Fig 7: In the signal-free case ( $\epsilon=0$ ), defining the SR and CR by thresholding $\widehat{\gamma}$ reduces their overlap in terms of $\gamma$ (first column, top two panels). Consequently, a classifier trained in the CR cannot reliably learn the high values of $\gamma$ occurring in the SR: $\widehat{\gamma}_{\mathrm{CR}}$ is calibrated in the CR but poorly calibrated in the SR (first column, bottom panel). Defining the regions by thresholding $\widehat{\psi}$ , rather than $\widehat{\gamma}$ , increases the overlap and improves calibration in the SR (second column). At the smaller noise scale $\eta=0.1$ , $\widehat{\psi}$ is close to $1$ across most of the range of $\widehat{\gamma}$ , producing still greater overlap between the SR and CR (third column). In the calibration panels, the red curve plots the event-weighted mean of $\widehat{\gamma}_{\mathrm{CR}}$ against the event-weighted mean of $\widehat{\gamma}$ within each of $50$ equal-count (quantile) bins of $\widehat{\gamma}$ , and the dashed black line is the identity.

Fig 8: Quantiles of

\widehat{\psi}

versus the normalized difference between the true and estimated

4b

event counts in the SR in the signal-free case (

\epsilon=0

). Events in the SR are binned into

64

quantiles of

\widehat{\psi}

, and the mean

\pm

standard deviation of the normalized difference between the sum of weights of background

4b

events and its estimate are shown across

50

replications. When

\widehat{\psi}=\widehat{\gamma}

(left), bias grows with

\widehat{\psi}

, inducing poor calibration of

\widehat{\gamma}_{\text{CR}}

in the SR. When smearing with noise scale

\eta=2.0

(right), the bias is reduced and the calibration of

\widehat{\gamma}_{\text{CR}}

in the SR is improved. Here,

\eta

denotes the noise scale used for smearing; see Algorithm 2 .

Fig 9: Histogram and ratio of estimated background (i.e., reweighted

3b

) and true

4b

background. The choice of space on which to train the density ratio in the CR impacts the estimated background distribution (i.e., counts of reweighted

3b

events in each bin) for

4b

events. Learning the density ratio on the original space (left) provides a reasonable estimate of the background

4b

distribution. Training the density ratio on the representation space (right), on the other hand, introduces a strong bias toward higher values in the presence of signal events.

Fig 10: t-SNE plots of original features (left) and representations (right) in the CR with colors based on types of events. Signal events in the representation space are more concentrated in a specific area compared to the original space, leading to a larger bias in the background estimator when it is learned based on the representations.

Fig 11: A toy example (left panel) shows why the choice of noise scale

\eta

should not be too small. When

\eta

is small, the smeared density ratio

\widetilde{\gamma}

still preserves the high-frequency signal feature (third row, left panel), so thresholding

\psi=\gamma/\widetilde{\gamma}

fails to identify a region rich in signal events. Indeed, choosing a smaller

\eta

causes signal events to be less concentrated in the tail of

\psi

(first row, right panel). Consequently, signal events are dominated by background events over the entire range of

\psi

(second row, right panel), resulting in a loss of power for the hypothesis test.

Fig 12: The SR efficiency of the non-resonant

\mathrm{HH}\to 4b

signal with signal ratio

\epsilon=0.01

(left) and

\epsilon=0.02

(right) for different noise scales

\eta

. Dashed lines represent the average measured across

100

repeated experiments, and the shaded regions represent the corresponding standard deviations. The noise scales give comparable concentration overall. On the left side of the curves, where

P_{4b}({\mathcal{X}}_{s})

is small and the SR consists of the highest-

\psi

events,

\eta=0.5

shows the least concentration and overtakes the larger scales only as the SR widens.

Fig 13: Comparison with the baseline SR proposed by Bryant (2018) . Dashed lines represent the average measured across

100

repeated experiments, and the shaded region represents the standard deviation of our method. While the baseline, which has access to a priori knowledge of the location of the signal, shows better performance when

P_{4b}({\mathcal{X}}_{s})

is large, our method matches or exceeds it once

P_{4b}({\mathcal{X}}_{s})

is small.

Fig 14: Percentage of experiments rejecting the null hypothesis of ( 2 ) with non-resonant

\mathrm{HH}\to 4b

signal for the hypothesis test at significance level

\alpha=0.05

with noise scale

\eta=2.0

. Error bars are Clopper–Pearson

95\%

intervals based on the

100

repetitions.

\epsilon=0

corresponds to the null case, for which the observed rejection rates are below the nominal level

\alpha=0.05

Fig 15: The SR efficiency of the resonant

X\to\mathrm{HH}\to 4b

signal at

400\,\mathrm{GeV}

(left) and non-resonant

\mathrm{ZH}\to 4b

(right) with signal ratios

\epsilon=0.0075

and

\epsilon=0.03

, respectively. Dashed lines represent the average measured across

100

repeated experiments, and the shaded regions represent the corresponding standard deviations. The plot demonstrates the adaptivity of our method to different types of signals and its robustness to the choice of the noise scale.

Fig 16: Percentage of experiments rejecting the null hypothesis of ( 2 ) with resonant

X\to\mathrm{HH}\to 4b

signal at

400\,\mathrm{GeV}

for the hypothesis test at significance level

\alpha=0.05

with noise scale

\eta=2.0

. Error bars are Clopper–Pearson

95\%

intervals based on the

100

repetitions. The power is significant when the signal ratio

\epsilon

is greater than or equal to

0.0075

Fig 17: Percentage of experiments rejecting the null hypothesis of ( 2 ) with non-resonant

\mathrm{ZH}\to 4b

signal for the hypothesis test at significance level

\alpha=0.05

with noise scale

\eta=2.0

. Error bars are Clopper–Pearson

95\%

intervals based on the

100

repetitions. The power rises steeply between

\epsilon=0.02

and

\epsilon=0.03

, ranging from

0.87

1.00

\epsilon=0.03

and reaching

1.00

\epsilon=0.05

$\eta$	$s_{\text{SR}}$	$\epsilon=0$	$\epsilon=0.005$	$\epsilon=0.0075$	$\epsilon=0.01$	$\epsilon=0.02$
0.5	$0.05$	$0.00$	$0.00$	$0.02$	$0.42$	$0.75$
	$0.1$	$0.00$	$0.00$	$0.01$	$0.43$	$0.83$
	$0.15$	$0.00$	$0.00$	$0.03$	$0.46$	$0.88$
	$0.2$	$0.01$	$0.01$	$0.04$	$0.52$	$0.93$
1.0	$0.05$	$0.01$	$0.01$	$0.02$	$0.50$	$0.98$
	$0.1$	$0.00$	$0.01$	$0.03$	$0.67$	$0.99$

Table 1: Empirical rejection proportions over

100

repeated experiments for the test of ( 2 ) with non-resonant

\mathrm{HH}\to 4b

signal at significance level

\alpha=0.05

\eta

is the noise scale used in learning the smoothed density ratio

\widetilde{\gamma}

, and

\epsilon

is the signal ratio.

\epsilon=0

corresponds to the null case, for which the observed rejection proportions are below the nominal level.

High-Energy Physics

High-energy physics studies the fundamental particles and forces of nature through collider, neutrino, cosmic-ray, and theory-driven experiments.

See all 164 papers

Sep 29, 2026

Searching for BSM Experimental Signatures with Large Lagrangian Models

The search for physics Beyond the Standard Model (BSM) is generally limited not by the supply of theory descriptions but by the lack of discriminating experimental observations. A case in point is dark matter, where the overwhelming gravitational evidence only goes so far in distinguishing between models within a vast theory space. Exploring the space of testable model signatures may help identify overlooked experimental observables and indicate the utility of future experiments. A challenge is designing a search through model signatures outside what is found in the literature. Our primary contribution is hAIthem, a framework that combines the self-guided exploration of reinforcement learning (RL) with the broad literature-derived knowledge of LLMs. We build an RL agent that learns to find which portions of a theory's high-dimensional parameter space are not excluded under some subset of constraints by playing a Battleship-style "game" against a suite of phenomenology tools. The agent is built as a Large Lagrangian Model (LLaM), an autoregressive transformer that reads a tokenized Lagrangian, is pretrained at scale (here on ~1 billion tokens from ~10,000 Lagrangians), and is fine-tuned in a live environment. The framework then constructs a decision tree that separates RL-found regions using observables computed with established tools, and passes the remaining degenerate regions to a set of LLM agents that compete to produce realistic signatures. In this proof of concept, RL-search outperforms an evolutionary-algorithm baseline, finding more viable regions with greater physical diversity. In a restricted space of single dark scalar multiplet models, we find that hAIthem proposes interesting combinations of previously studied observables, such as the application of a halo-independent kinematic ratio to paleo-detectors.

Ibrahim Elsharkawy, Victoria Knapp-Perez, Wahid Bhimji +1

Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada · NERSC, Lawrence Berkeley National Laboratory, Berkeley, California, USA · Department of Physics and Astronomy, University of California, Irvine, CA 92697 +3

AI for Science High-Energy Physics Large Language Model-Guided Scientific Discovery

Ask PDF

Jun 11, 2026

Conformal calibration and look-elsewhere effect in anomaly detection for new-physics searches

Machine-learned anomaly detection is reshaping searches for new physics, but it has outrun the statistics used to interpret it. A raw anomaly score has no calibrated meaning, a model that scans many regions inflates the look-elsewhere effect, and the asymptotic significances the field relies on are blind to the background mismodelling that anomaly detectors are especially prone to. We propose a calibration layer, built on conformal prediction, that turns any anomaly score into a defensible significance with distribution-free, finite-sample guarantees. Conformal prediction converts scores into valid local p-values, weighted and Mondrian variants repair the sideband-to-signal-region exchangeability failures that resonant searches suffer, and a Gross-Vitells step carries the result through to a look-elsewhere-aware global significance. The layer does two things at once. It exposes miscalibration that the standard pipeline cannot see, and it corrects it without retraining the detector. On public LHC Olympics data, a classifier develops a substructure-mass correlation that makes sideband-calibrated background p-values anti-conservative. Taken at face value, this manufactures a

\sim 46σ

excess from background sculpting alone, which the label-free weighted correction removes, restoring an honest null. When run as a blind wide-mass bump hunt, the standard asymptotic and unweighted procedures fabricate

\gtrsim10σ

excesses and

\approx5σ

excesses even in signal-free windows, while the conformal layer raises no false alarms and its global false-positive rate is verified on background-only pseudoexperiments. The result is an auditable, detector-agnostic path from an uncalibrated score to a trials-factor-aware significance, ready to be folded into experimental anomaly searches.

Jack Y. Araz, Michael Spannowsky

Department of Physics and Astronomy, University College London, London, WC1E 6B, UK · Department of Engineering, City St. George’s, University of London, London, EC1V 0HB, UK · Institute for Theoretical Physics, Campus Süd, Karlsruhe Institute of Technology (KIT), D-76128 Karlsruhe, Germany +1

Post-Hoc Calibration Conformal Prediction Anomaly Detection

Ask PDF

Aug 13, 2026

Unknown Unknowns: Model Misspecification in Machine Learning for Physics

Machine learning is now a central tool for solving inverse problems in particle physics and astronomy. Models are trained on simulation and deployed on real data, raising the question not just of whether they fit, but of whether they are wrong in ways we did not anticipate: the unknown unknowns. This challenge of model misspecification is not unique to machine learning. In physics, misspecification is sometimes exactly what we want to find: new discoveries appear as failures of existing models. At other times, we want such effects absorbed into the analysis without biasing the measurement. A robust analysis is one that absorbs the misspecifications we are not interested in, while preserving sensitivity to the ones we are. Machine learning can both amplify misspecification and provide new tools to address it. We discuss the challenges of model misspecification, diagnostics for detecting it, and strategies for mitigation. No single diagnostic can confirm that a model is correctly specified: detection and mitigation are two halves of an iterative loop, in which a battery of complementary diagnostics is applied, the model is updated, and the process repeated. Robustness against unknown unknowns is ultimately less about any single technique than about a disposition: a willingness to suspect one's own model, and to design analyses that can survive being wrong in ways one did not anticipate.

Juan Cruz-Martinez, Carolina Cuesta-Lazaro, Alexander Held +1

Departamento de Física Atómica, Molecular y Nuclear, Universidad de Sevilla, E-41080 Sevilla, Spain · Center for Computational Astrophysics, Flatiron Institute, 162 Fifth Avenue, New York, NY 10010, USA · Institute for Advanced Study, 1 Einstein Drive, Princeton, NJ 08540, USA +2

Model Misspecification Scientific ML

Ask PDF

Abstract

High-Energy Physics

Explore similar work

Searching for BSM Experimental Signatures with Large Lagrangian Models

Conformal calibration and look-elsewhere effect in anomaly detection for new-physics searches

Unknown Unknowns: Model Misspecification in Machine Learning for Physics