Attack logics, not outputs: Concept-based Property Attacks for Shortcut Elimination in Neural Networks

Raik Dankworth; Felix Mächtle; Thomas Eisenbarth; Gesina Schwalbe

cs.CROct 26, 2026

Attack logics, not outputs: Concept-based Property Attacks for Shortcut Elimination in Neural Networks

Authors: Raik Dankworth, Felix Mächtle, Thomas Eisenbarth, Gesina Schwalbe

Organizations: Institute of Software Engineering and Programming Languages University of Luebeck · Institute of IT Security University of Luebeck

Abstract

Deep neural networks (NNs) for computer vision are vulnerable to adversarial attacks, such that minuscule malicious changes to inputs may induce unintuitive outputs. One key approach to verify and mitigate such robustness issues is to falsify expected output behavior, which enables local robustness guarantees and adversarial training. Due to the black-box nature of NNs, current attacks only falsify a class of the final output, such as flipping from $\text{stop_sign}$ to $\neg \text{stop_sign}$ . In this work we argue that robustness analysis should focus more generally on logical inconsistencies in concept-level reasoning rather than solely on output misclassification. We formalize this perspective through concept-based properties, i.e., constraints over human-interpretable concepts such as $\text{red} \land \text{octogonal} \implies \text{stop_sign}$ and study the falsification of such properties as a principled form of adversarial attack. Furthermore, an implementation of concept-based properties on already trained NNs is proposed using techniques from concept-based explainable artificial intelligence (C-XAI). For that we formally prove that attacks on concept-based properties are expected to have a reduced search space compared to simple class falsification, whilst arguably be more aligned with intuitive robustness targets. The practical effectiveness of our approach for logically informed adversarial training is shown in an empirical study across multiple models and datasets, including both vision and a security-relevant code processing task. Our results demonstrate that concept-level falsification generalizes classical adversarial attacks: Retraining on our novel attacks maintains robustness gains as compared to classical adversarial retraining whilst better eliminating semantic biases.

Figures & tables

Figure 1: Post-hoc Concept Extraction: Two neurons for concept output ( blue and green neurons) are post-hoc added to the trained NN ( gray ) using newly trained connections to hidden layers 1 / 3 .

Backbone		No retraining	PGD	R $\downarrow$	R $\uparrow$
MNIST	LENET-5	$.988_{\pm.003}$	$.997_{\pm.001}$	$.998_{\pm.000}$	$.994_{\pm.001}$
	M3	$.992_{\pm.001}$	$.998_{\pm.001}$	$.998_{\pm.001}$	$.992_{\pm.002}$
	ResNet-8	$.986_{\pm.005}$	$.998_{\pm.000}$	$.998_{\pm.000}$	$.994_{\pm.001}$
	ViT	$.916_{\pm.012}$	$.940_{\pm.011}$	$.937_{\pm.010}$	$.937_{\pm.015}$
	VGG-16	$.992_{\pm.001}$	$.986_{\pm.006}$	$.913_{\pm.253}$	$.853_{\pm.346}$
GTSRB	LENET-5	$.968_{\pm.006}$	$.973_{\pm.001}$	$.975_{\pm.001}$	$.964_{\pm.004}$

Table 1: F1 scores on original data after adversarial retraining. Each model is retrained using adversarial examples generated by the corresponding method. Values are reported as mean

\pm

standard deviation across five runs. R

\downarrow

achieves performance comparable to PGD, while the wrong attack direction R

\uparrow

is consistently lower. In the biased setting, R

\downarrow

yields significant improvements over PGD for the majority of models.

Backbone		No retraining	PGD	R $\downarrow$	R $\uparrow$
MNIST	LENET-5	$.102_{\pm.084}$	$.997_{\pm.001}$	$.938_{\pm.033}$	$.919_{\pm.025}$
	M3	$.373_{\pm.114}$	$.998_{\pm.001}$	$.995_{\pm.001}$	$.987_{\pm.004}$
	ResNet-8	$.110_{\pm.013}$	$.998_{\pm.000}$	$.994_{\pm.002}$	$.985_{\pm.005}$
	ViT	$.430_{\pm.102}$	$.927_{\pm.012}$	$.910_{\pm.014}$	$.923_{\pm.016}$
	VGG-16	$.000_{\pm.000}$	$.984_{\pm.008}$	$.909_{\pm.252}$	$.850_{\pm.345}$
GTSRB	LENET-5	$.057_{\pm.012}$	$.906_{\pm.003}$	$.822_{\pm.009}$	$.822_{\pm.011}$

Table 2: F1 scores on PGD adversarial examples evaluated across differently retrained models. PGD adversarial examples strongly degrade performance on non-retrained models, indicating high attack success. Models retrained with R

\downarrow

and R

\uparrow

generalize well to unseen PGD adversarial examples, although PGD-retrained models perform best due to exposure during training. Values are reported as mean

\pm

standard deviation across five runs.

Panel of Figure 2 — Figure 2: Adversarial examples generated by PGD, R $\downarrow$ , and R $\uparrow$ for the same random selected input sample.

	Acc.	Prec.	Rec.	F1
Baseline	$.649_{\pm.008}$	$.658_{\pm.038}$	$.514_{\pm.094}$	$.569_{\pm.046}$
Std. AA	$.589_{\pm.037}$	$.683_{\pm.140}$	$.407_{\pm.284}$	$.423_{\pm.167}$
ConPAtt	$.650_{\pm.003}$	$.645_{\pm.033}$	$.551_{\pm.091}$	$.587_{\pm.040}$

Table 3: Test metrics for the baseline LLM and its two adversarially fine-tuned variants on the code vulnerability detection task. Mean

\pm

std over five seeds.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Samples of the GTSRB concept dataset.

	Attack	LENET-5	M3	ResNet8	VGG-16	ViT
MNIST	PGD	$.898_{\pm.084}$	$.627_{\pm.114}$	$.890_{\pm.013}$	$1.0_{\pm.000}$	$.570_{\pm.102}$
	R $\uparrow$	$.174_{\pm.028}$	$.413_{\pm.093}$	$.893_{\pm.011}$	$.970_{\pm.007}$	$.511_{\pm.096}$
	R $\downarrow$	$.401_{\pm.261}$	$.381_{\pm.142}$	$.896_{\pm.011}$	$.861_{\pm.002}$	$.355_{\pm.065}$
GTSRB	PGD	$.943_{\pm.012}$	$.886_{\pm.024}$	$.997_{\pm.002}$	$1.0_{\pm.000}$	$1.0_{\pm.000}$
	R $\uparrow$	$.865_{\pm.026}$	$.913_{\pm.017}$	$1.0_{\pm.000}$	$1.0_{\pm.000}$	$1.0_{\pm.000}$
	R $\downarrow$	$.689_{\pm.010}$	$.767_{\pm.033}$	$.994_{\pm.007}$	$.999_{\pm.000}$	$.992_{\pm.004}$

Appendix

Table 4: Attack success rate with respect to misclassification for standard attacks and their ConPAtt variants across datasets and architectures. Results are reported as mean

\pm

standard deviation over multiple runs.

Retrained	LeNet-5	M3	ResNet-8	VGG-16	ViT
No retraining	$.988_{\pm.003}$	$.992_{\pm.001}$	$.986_{\pm.005}$	$.992_{\pm.001}$	$.916_{\pm.012}$
PGD	$.997_{\pm.001}$	$.998_{\pm.001}$	$.998_{\pm.000}$	$.986_{\pm.006}$	$.940_{\pm.011}$
PGD $\downarrow$	$.998_{\pm.000}$	$.998_{\pm.001}$	$.998_{\pm.000}$	$.913_{\pm.253}$	$.937_{\pm.010}$
PGD $\uparrow$	$.994_{\pm.001}$	$.992_{\pm.002}$	$.994_{\pm.001}$	$.853_{\pm.346}$	$.937_{\pm.015}$
FGSM	$.991_{\pm.001}$	$.991_{\pm.002}$	$.989_{\pm.009}$	$.986_{\pm.004}$	$.939_{\pm.004}$
FGSM $\downarrow$	$.992_{\pm.002}$	$.992_{\pm.002}$	$.993_{\pm.002}$	$.847_{\pm.344}$	$.945_{\pm.011}$

Appendix

Table 5: F1 scores on original data after adversarial retraining. Each model is retrained using adversarial examples generated by the corresponding method. Values are reported as mean

\pm

standard deviation across five runs.

AE	Retrained	LeNet-5	M3	ResNet-8	VGG-16	ViT
PGD	No retraining	$.102_{\pm.084}$	$.373_{\pm.114}$	$.110_{\pm.013}$	$.000_{\pm.000}$	$.430_{\pm.102}$
	PGD	$.997_{\pm.001}$	$.998_{\pm.001}$	$.998_{\pm.000}$	$.984_{\pm.008}$	$.927_{\pm.012}$
	R $\downarrow$	$.938_{\pm.033}$	$.995_{\pm.001}$	$.994_{\pm.002}$	$.909_{\pm.252}$	$.910_{\pm.014}$
	R $\uparrow$	$.919_{\pm.025}$	$.987_{\pm.004}$	$.985_{\pm.005}$	$.850_{\pm.345}$	$.923_{\pm.016}$
FGSM	No retraining	$.270_{\pm.071}$	$.703_{\pm.065}$	$.098_{\pm.000}$	$.102_{\pm.003}$	$.213_{\pm.061}$
	FGSM	$.989_{\pm.002}$	$.989_{\pm.005}$	$.990_{\pm.004}$	$.989_{\pm.005}$	$.916_{\pm.009}$

Appendix

Table 6: F1 scores on adversarial examples of PGD, FGSM and VMIFGSM evaluated across differently retrained models. Values are reported as mean

\pm

standard deviation across five runs.

Concept	Retrained	LeNet-5	M3	ResNet-8	VGG-16	ViT
Circle	No retraining	$.803_{\pm.042}$	$.896_{\pm.014}$	$.921_{\pm.030}$	$.962_{\pm.002}$	$.766_{\pm.018}$
	PGD	$.824_{\pm.040}$	$.909_{\pm.013}$	$.981_{\pm.005}$	$.906_{\pm.018}$	$.825_{\pm.018}$
	R $\downarrow$	$.813_{\pm.043}$	$.915_{\pm.021}$	$.981_{\pm.004}$	$.908_{\pm.031}$	$.826_{\pm.020}$
	R $\uparrow$	$.818_{\pm.038}$	$.923_{\pm.014}$	$.981_{\pm.004}$	$.883_{\pm.139}$	$.815_{\pm.030}$
Crossing	No retraining	$.540_{\pm.031}$	$.896_{\pm.039}$	$.816_{\pm.044}$	$.913_{\pm.007}$	$.539_{\pm.050}$
	PGD	$.517_{\pm.066}$	$.848_{\pm.046}$	$.967_{\pm.008}$	$.714_{\pm.074}$	$.623_{\pm.072}$

Appendix

Table 7: F1 scores on original MNSIT concept data before and after adversarial retraining, reported as mean

\pm