Shapes are not enough: CONSERVAttack and its use for finding vulnerabilities and uncertainties in machine learning applications
Authors: Philip Bechtle, Lucie Flek, Philipp Alexander Jung, Akbar Karimi, Timo Saala, Alexander Schmidt, Matthias Schott, Philipp Soldin, +2 more
Organizations: Institute of Physics, University of Bonn, Germany. · Bonn-Aachen Institute of Technology, University of Bonn, Germany. · Physics Institute III A, RWTH Aachen University, Germany. · Physics Institute III B, RWTH Aachen University, Germany.
In High Energy Physics, as in many other fields of science, the application of machine learning techniques has been crucial in advancing our understanding of fundamental phenomena. Increasingly, deep learning models are applied to analyze both simulated and experimental data. In most experiments, a rigorous regime of testing for physically motivated systematic uncertainties is in place. The numerical evaluation of these tests for differences between the data on the one side and simulations on the other side quantifies the effect of potential sources of mismodelling on the machine learning output. In addition, thorough comparisons of marginal distributions and (linear) feature correlations between data and simulation in "control regions" are applied. However, the guidance by physical motivation, and the need to constrain comparisons to specific regions, does not guarantee that all possible sources of deviations have been accounted for. We therefore propose a new adversarial attack - the CONSERVAttack - designed to exploit the remaining space of hypothetical deviations between simulation and data after the above mentioned tests. The resulting adversarial perturbations are consistent within the uncertainty bounds - evading standard validation checks - while successfully fooling the underlying model. We further propose strategies to mitigate such vulnerabilities and argue that robustness to adversarial effects must be considered when interpreting results from deep learning in particle physics.
Figures & tables
Figure 1: Comparison of three distinct distributions between clean and adversarial events in an attack setting. The plot on the left shows the feature resulting in the minimal change in Jensen Shannon distance, the one in the middle the feature corresponding to the median Jensen Shannon Distance, and the plot on the right the feature exhibiting the largest change in the Jensen Shannon Distance for the given attack run.
Figure 2: Comparison of the correlation matrices between the clean events (left) and adversarial events (right) resulting from the attack.
Task
pmin
ϵstep
nit
α
β
maxJSDt
maxΔFt
nc
Higgs
0.0005
0.0005
10
4.0
1.0
0.006
0.0002
True
TTvsWW
0.005
0.01
10
6.5
1.0
0.003
0.003
True
Table 1: Attack parameters for both the Higgs and the TT vs. WW jet-tagging tasks.
Figure 3: Fooling ratio of the adversarial attack on the Higgs dataset over 10 runs. The average performance of the attack is shown as a dashed line, and its standard deviation as a dotted line.
Figure 4: Average Jensen Shannon distance of the adversarial attack on the Higgs dataset over 10 runs. The average performance of the attack is shown as a dashed line, and its standard deviation as a dotted line.
Figure 5: Average Frobenius Norm of the adversarial attack on the Higgs dataset over 10 runs. The average performance of the attack is shown as a dashed line, and its standard deviation as a dotted line.
Figure 6: Flowchart showing the workflow used in the data augmentation pipeline.
Figure 7: Comparison of the areas under the Receiver-Operating-Curves—on the base model (grey) and the adversarially augmented model (red)—on the Higgs data set over 10 independent runs. The respective average results are shown in dashed lines.
Figure 8: Flowchart showing the workflow used in the adversarial training pipeline.
Figure 9: Comparison of the fooling ratios of adversarial attacks on the baseline model (grey) vs the average-case adversarially trained network (red) and the best-case adversarially trained network (blue) on the Higgs dataset.
Figure 10: Flowchart showing the workflow used in the adversarial detector pipeline.
Figure 11: Comparison of the initial fooling ratios (red) and the corrected fooling ratios (grey) found in 10 independent runs of the adversarial detector pipeline on the Higgs dataset.
Figure 12: Comparing the expected amounts of events misclassified across multiple runs (grey) to the observed amount (red). The red stars depict statistically significant differences between observed and expected. This plot was generated using the TT vs. WW dataset.
Figure 13: Detector accuracies on both clean non-simulated clean events (blue) and simulated clean events(red) using an adversarial detector network trained solely on clean and adversarial simulated events, dependent on the classification threshold for the TT vs. WW dataset.
Figure 14: Point clouds of sub-samples points from both the clean signal (blue) and clean background (red) events for the Donut toy dataset.
Figure 15: Comparison of the feature distributions between clean signal vs. clean background events (top row) and clean signal vs. adversarial background events (bottom row).
Figure 16: Point clouds of sub-samples points from both the clean signal (blue) and adversarial background (yellow) events for the Donut toy dataset.
Figure 17: Point clouds of sub-samples points from both the clean signal (blue) and adversarial background events that successfully fool the baseline network (yellow) for the Donut toy dataset.
Figure 18: Point clouds of sub-samples points from both the clean signal correctly classified by the adversarial detector network as such (blue), and adversarial background events that successfully fool the baseline network but are not correctly identified as such by the detector network (yellow) for the Donut toy dataset.
Figure 19: Comparison of the initial fooling ratios (red) and the corrected fooling ratios (grey) found in 5 independent runs of the Distance correlation restricted adversarial detector pipeline on the Higgs dataset.
Figure 20: Comparison of the initial fooling ratios (red) and the corrected fooling ratios (grey) found in 5 independent runs of the limited data Pearson correlation restricted adversarial detector pipeline on the Higgs dataset.
Figure 21: Area under the Receiver-Operating-Curve for varying sample sizes of training events used for the Pearson correlation restricted adversarial detector network.
Figure 22: Comparison of the averageg Areas under the Receiver-Operating-Curve for the distance correlation restricted attack (red) and the limited data Pearson restricted attack (grey) across 5 adversarial detector runs.
Figure 23: Flowchart showing the proposed workflow.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 24: Fooling ratios of the restricted attack on the Higgs dataset over 10 independent runs. The grey points indicate the total fooling ratio across all events, the blue points correspond to the signal (unrestricted) region, and the red points to the control (restricted) region.
Figure 25: Average Jensen–Shannon distance of the adversarial attack evaluated on the control (restricted) region of the Higgs dataset over 10 runs.
Figure 26: Average Frobenius norm of the adversarial attack evaluated on the control (restricted) region of the Higgs dataset over 10 runs.
Higgs - Signal-Background Classifier
Layer
Nodes
Activation Function
Input Layer
30
-
Dense
300
ReLU
BatchNormalization
-
-
Dense
102
ReLU
BatchNormalization
-
-
Appendix
Table 2: Architecture of the baseline signal-background classifier for the Higgs dataset.
Higgs - Adversarial Detector Model
Layer
Nodes
Activation Function
Input Layer
30
-
Dense
300
ReLU
BatchNormalization
-
-
Dense
102
ReLU
BatchNormalization
-
-
Appendix
Table 3: Architecture of the adversarial detector MLP for the Higgs dataset.
TT vs. WW - Jet Tagger
Layer
Nodes
Activation Function
Input Layer
87
-
Dense
300
ReLU
BatchNormalization
-
-
Dense
102
ReLU
BatchNormalization
-
-
Appendix
Table 4: Architecture of the baseline signal-background classifier for the TT vs. WW Jets dataset.
TT vs. WW - Adversarial Detector Model
Layer
Nodes
Activation Function
Input Layer
87
-
Dense
300
ReLU
BatchNormalization
-
-
Dense
102
ReLU
BatchNormalization
-
-
Appendix
Table 5: Architecture of the adversarial detector MLP for the TT vs. WW Jets dataset.
Figure 27: Average fooling ratio of the adversarial attack on the TT vs. WW dataset over 10 runs. The average performance of the attack is shown as a dashed line, and its standard deviation as a dotted line.
Figure 28: Average Jensen Shannon distance of the adversarial attack on the TT vs. WW dataset over 10 runs. The average performance of the attack is shown as a dashed line, and its standard deviation as a dotted line.
Figure 29: Average Frobenius Norm of the adversarial attack on the TT vs. WW dataset over 10 runs. The average performance of the attack is shown as a dashed line, and its standard deviation as a dotted line.
Figure 30: Comparison of the areas under the Receiver-Operating-Curves—on the base model (grey) and the adversarially augmented model (red)—on the Higgs data set over 10 independent runs. The respective average results are shown in dashed lines.
Figure 31: Comparison of the fooling ratios of adversarial attacks on the baseline model (grey) vs the average-case adversarially trained network (red) and the best-case adversarially trained network (blue) on the TT vs. WW dataset across 5 runs.
Figure 32: Comparison of the initial fooling ratios (red) and the corrected fooling ratios (grey) found in 10 independent runs of the adversarial detector pipeline on the TT vs. WW dataset.
Adversarial Attack Hyperparameters
Parameter Name
Type
Description
x
Numpy Array
Input data to be attacked.
y
Numpy Array
True labels for the input data.
model
str
Path to the trained model.
min_change
float
Minimum allowed change per feature in each step.
step
float
Step size for candidate feature changes.
Appendix
Table 6: User-configurable hyperparameters for the adversarial attack.
Higgs Attack Hyperparameter Choice
Parameter Name
Value
min_change
0.0005
step
0.0005
n_iterations
10
num_bins
200
alpha
4.0
Appendix
Table 7: Parameter values used for adversarial attack for the Higgs dataset.
Higgs Attack Hyperparameter Choice
Parameter Name
Value
min_change
0.002
step
0.006
n_iterations
10
num_bins
200
alpha
6.5
Appendix
Table 8: Parameter values used for the augmentation pipeline for the Higgs dataset.
Higgs Attack Hyperparameter Choice
Parameter Name
Value
min_change
0.003
step
0.02
n_iterations
10
num_bins
100
alpha
6.0
Appendix
Table 9: Parameter values used for the adversarial detector pipeline for the Higgs dataset for training and test samples.
Higgs Attack Hyperparameter Ranges
Parameter Name
Value / Range
min_change
[0.001, 0.005] (uniform)
step
[0.005, 0.03] (uniform)
n_iterations
10
num_bins
100
alpha
[3.0, 10.0] (uniform)
Appendix
Table 10: Parameter ranges used for generating training adversaries in the iterative retraining pipeline (randomized per retraining iteration).
Higgs Attack Hyperparameter Choice (Test)
Parameter Name
Value
min_change
0.003
step
0.01
n_iterations
10
num_bins
100
alpha
8.0
Appendix
Table 11: Parameter values used for generating test adversaries in the iterative retraining pipeline.
DisCo Attack Hyperparameter Choice
Parameter Name
Value
min_change
0.015
step
0.06
n_iterations
10
num_bins
30
alpha
4.0
Appendix
Table 12: Parameter values used for the adversarial attack in the DisCo attack experiment on the Higgs Dataset.
DisCo Detector Attack Hyperparams. (Train)
Parameter Name
Value
min_change
0.07
step
0.08
n_iterations
10
num_bins
30
alpha
4.0
Appendix
Table 13: Parameter values used for generating training adversaries in the DisCo detector experiment.
DisCo Detector Attack Hyperparams. (Test)
Parameter Name
Value
min_change
0.01
step
0.05
n_iterations
10
num_bins
30
alpha
4.0
Appendix
Table 14: Parameter values used for generating test adversaries in the DisCo detector experiment.
Limited Data Attack Hyperparameters
Parameter Name
Value
min_change
0.005
step
0.02
n_iterations
10
num_bins
30
alpha
6.0
Appendix
Table 15: Parameter values used for generating training and test adversaries in the for the limited data Pearson correlation experiment.
TTvsWW Attack Hyperparameter Choice
Parameter Name
Value
min_change
0.005
step
0.01
n_iterations
10
num_bins
200
alpha
6.5
Appendix
Table 16: Parameter values used for adversarial attack for the TT vs. WW dataset.
TTvsWW Attack Hyperparameter Choice
Parameter Name
Value
min_change
0.003
step
0.02
n_iterations
10
num_bins
100
alpha
6.5
Appendix
Table 17: Parameter values used for the augmentation pipeline for the TT vs. WW dataset.
TTvsWW Attack Hyperparameter Choice
Parameter Name
Value
min_change
0.01
step
0.01
n_iterations
10
num_bins
100
alpha
6.0
Appendix
Table 18: Parameter values used for the adversarial detector pipeline for the TT vs. WW dataset for training and test samples.
TTvsWW Attack Hyperparam. Ranges (Train)
Parameter Name
Value / Range
min_change
[0.001, 0.005] (uniform)
step
[0.005, 0.03] (uniform)
n_iterations
10
num_bins
100
alpha
[3.0, 10.0] (uniform)
Appendix
Table 19: Parameter ranges used for generating train adversaries in the TT vs. WW iterative retraining pipeline (randomized per retraining iteration).
TTvsWW Attack Hyperparameters (Test)
Parameter Name
Value
min_change
0.002
step
0.005
n_iterations
10
num_bins
100
alpha
8.0
Appendix
Table 20: Parameter values used for generating test adversaries in the TT vs. WW iterative retraining pipeline.
DonutDummy Attack Hyperparameters (Test)
Parameter Name
Value
min_change
0.001
num_candidates
150
n_iterations
10
num_bins
70
alpha
6.0
Appendix
Table 21: Parameter values used for generating test adversaries in the DonutDummy experiment.
DonutDummy Detector Attack Hyperparams.
Parameter Name
Value
min_change
0.001
num_candidates
150
n_iterations
10
num_bins
60
alpha
6.0
Appendix
Table 22: Parameter values used for generating training and test adversaries in the DonutDummy adversarial detector pipeline experiment.
Machine learning is now a central tool for solving inverse problems in particle physics and astronomy. Models are trained on simulation and deployed on real data, raising the question not just of whether they fit, but of whether they are wrong in ways we did not anticipate: the unknown unknowns. This challenge of model misspecification is not unique to machine learning. In physics, misspecification is sometimes exactly what we want to find: new discoveries appear as failures of existing models. At other times, we want such effects absorbed into the analysis without biasing the measurement. A robust analysis is one that absorbs the misspecifications we are not interested in, while preserving sensitivity to the ones we are. Machine learning can both amplify misspecification and provide new tools to address it. We discuss the challenges of model misspecification, diagnostics for detecting it, and strategies for mitigation. No single diagnostic can confirm that a model is correctly specified: detection and mitigation are two halves of an iterative loop, in which a battery of complementary diagnostics is applied, the model is updated, and the process repeated. Robustness against unknown unknowns is ultimately less about any single technique than about a disposition: a willingness to suspect one's own model, and to design analyses that can survive being wrong in ways one did not anticipate.
Juan Cruz-Martinez, Carolina Cuesta-Lazaro, Alexander Held +1
Departamento de Física Atómica, Molecular y Nuclear, Universidad de Sevilla, E-41080 Sevilla, Spain · Center for Computational Astrophysics, Flatiron Institute, 162 Fifth Avenue, New York, NY 10010, USA · Institute for Advanced Study, 1 Einstein Drive, Princeton, NJ 08540, USA +2
The requirement of generating predictions that exactly fulfill the fundamental symmetry of the corresponding physical quantities has profoundly shaped the development of machine-learning models for physical simulations. In many cases, models are built using constrained mathematical forms that ensure that symmetries are enforced exactly. However, unconstrained models that do not obey rotational symmetries are often found to have competitive performance, and to be able to learn to a high level of accuracy an approximate equivariant behavior with a simple data augmentation strategy. In this paper, we introduce rigorous metrics to measure the symmetry content of the learned representations in such models, and assess the accuracy by which the outputs fulfill the equivariant condition. We apply these metrics to two unconstrained, transformer-based models operating on decorated point clouds (a graph neural network for atomistic simulations and a PointNet-style architecture for particle physics) to investigate how symmetry information is processed across architectural layers and is learned during training. Based on these insights, we establish a rigorous framework for diagnosing spectral failure modes in ML models. Enabled by this analysis, we demonstrate that one can achieve superior stability and accuracy by strategically injecting the minimum required inductive biases, preserving the high expressivity and scalability of unconstrained architectures while guaranteeing physical fidelity.
Michelangelo Domina, Joseph William Abbott, Paolo Pegolo +2
Laboratory of Computational Science and Modeling, Institut des Matériaux, École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland
Machine learning (ML) models are susceptible to various security, privacy, and fairness risks. Adversaries with different characteristics (i.e., objectives, knowledge, and capabilities) can collude by executing one attack to amplify others. Existing work lacks a systematic framework to explore collusion among adversaries, and to study the implications of the adversaries' characteristics. We present a framework covering collusion (a) between train- and inference-time adversaries, and (b) among inference-time adversaries. Our framework accounts for factors enabling collusion between adversaries. We propose a guideline to conjecture about the potential for collusion using enabling factors. We use it to explain prior work, conjecture about unexplored collusions, and empirically validate five such cases. Finally, we discuss how adversaries' characteristics influence the potential for collusion.
Vasisht Duddu, Lipeng He, Asim Waheed +1
University of Waterloo · KTH Royal Institute of Technology