RePAIR: Inverting Backdoor Attacks in Semantic Segmentation

Arnold Brosch; Kira Maag

cs.CVOct 27, 2026

RePAIR: Inverting Backdoor Attacks in Semantic Segmentation

Authors: Arnold Brosch, Kira Maag

Organizations: Heinrich Heine University Düsseldorf

Abstract

Despite significant advances in computer vision, semantic segmentation models remain vulnerable to backdoor attacks, posing a security risk. These attacks redirect pixels of a chosen victim class to an attacker-specified target class when a trigger is present, while preserving clean predictions. Existing defenses are largely derived from image classification, while dedicated segmentation defenses remain scarce and typically require retraining or model modification. We introduce RePAIR, a post-hoc, label-free, attack-unaware defense based on the observation that backdoor-induced shifts can be stronger in the classifier head than in internal representations, leaving recoverable victim-class semantics. RePAIR infers the victim-target relation from an unlabeled, potentially contaminated image pool, constructs a safety-approved semantic reference, and localizes suspicious pixels by comparing head predictions with internal semantic evidence. It then applies feature-space and output corrections toward the inferred victim semantics, while conservative localization and output constraints limit changes to benign predictions. We model the attack as adversarial probability transport and repair as its approximate reversal, deriving bounds and sufficient conditions for localization, residual attack success, clean-side changes, defense-induced target predictions, and attack-unaware pair identification. Experiments across segmentation architectures and attack settings show that RePAIR substantially reduces attack success while largely preserving clean performance.

Figures & tables

Panel of Figure 1 — Figure 1: From left to right: poisoned image, prediction under the backdoor attack, prediction after applying our defense and ground truth segmentation.

Panel of Figure 2 — Figure 2: Targeted backdoor as directed victim-to-target transport. The prediction may move toward the target while internal representations retain victim evidence, which RePAIR uses for localization and repair. The forward operator and its coupling describe the same directed mass transfer at the marginal and joint-measure levels, respectively.

Model	Method	IBA (NNI)	IBA (PRL)	ConSeg	OFBA	BadSeg-O2B	BadSeg-O2O	BadSeg-B2O	BadSeg-B2B	mIoU-A	mIoU-B
	no defense	$88.56$	$97.41$	$86.23$	$60.83$	$95.49$	$97.03$	$96.87$	$96.82$	$71.14_{-7.28}^{+6.49}$	$73.28_{-10.01}^{+6.00}$
	fine-tuning	$56.36$	$12.35$	$84.54$	$6.31$	$13.42$	$\underline{81.91}$	$86.36$	$97.69$	$35.86_{-16.03}^{+14.73}$	$37.65_{-21.43}^{+18.41}$
Deep-	pruning	$89.60$	$98.13$	$90.07$	$71.36$	$96.63$	$96.55$	$97.58$	$97.25$	$70.13_{-6.58}^{+6.82}$	$72.37_{-9.91}^{+6.00}$
LabV3+	Seg.-Cleanse	$\mathbf{0.36}$	$\mathbf{0.89}$	$86.23$	$\mathbf{0.75}$	$\underline{3.31}$	$94.20$	$96.85$	$\underline{36.95}$	$71.14_{-7.36}^{+6.49}$	$74.14_{-7.31}^{+5.30}$
	ABL	$83.94$	$94.55$	$\mathbf{1.12}$	$\underline{0.78}$	$17.29$	$90.36$	$\underline{86.19}$	$92.17$	$36.66_{-6.48}^{+4.39}$	$40.97_{-4.08}^{+5.46}$
	RePAIR (ours)	$\underline{1.18}$	$\underline{1.01}$	$\underline{1.51}$	$1.10$	$\mathbf{2.35}$	$\mathbf{1.00}$	$\mathbf{0.46}$	$\mathbf{7.86}$	$73.83_{-7.77}^{+3.97}$	$74.50_{-5.69}^{+3.17}$

Table 1: Cityscapes results across backdoor attacks. ASR is the primary metric, with lower values indicating better defense performance. mIoU-A and mIoU-B are averaged across the eight attacks and reported with their minimum and maximum deviations from the mean.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Result	Main condition	Consequence
Lemma 1	local directed transport	pair margin decreases
Cor. 2	positive initial pair margin	pairwise preference flips
Lemma 3	fixed feasible transport amount	local inverse exists
Lemma 4	feasible one-edge shift	coupling representation
Lemma 5	evidence definition	surviving victim evidence
Prop. 6	pair-correct state and additive semantic arm	$\eta_{\mathrm{loc}}\leq\eta_{\mathrm{ref}}$

Appendix

Table 2: Role and scope of the theoretical results.

Defense	Parameter	Value
Fine-tuning	Clean data fraction	$10\%$
	Epochs	10
Pruning	Pruning ratio	$10\%$
	Criterion	Least-active final-layer channels
ABL	Isolation epochs	20
	Isolation ratio $p$	0.01

Appendix

Table 3: Hyperparameters of the benchmark defenses.

Parameter	Variable	Value
Shared
Image-disjoint safety/calibration folds	$K_{\mathrm{fold}}$	$4$
Minimum safety-approved images	$N_{\mathrm{safe}}^{\min}$	$24$
Minimum semantic class coverage	$\gamma_{\mathrm{cov}}$	$75\%$
Semantic probability budget	$\delta_{\mathrm{prob}}$	$0.05$
DeeplabV3+

Appendix

Table 4: Principal global state-construction and inference parameters for Cityscapes.

Model	Method	IBA (NNI)	IBA (PRL)	ConSeg	OFBA	BadSeg-O2B
DeepLabV3+	no defense	$3.02$	$3.22$	$2.88$	$2.40$	$2.58$
SegFormer	no defense	$2.51$	$2.38$	$4.93$	$2.48$	$2.43$

Appendix

Table 5: ADE20K Attack Results Undefended attack success rate (ASR, %). RePAIR is not applied because the evaluated attacks do not establish a meaningful backdoor effect.

Model	Attack	No defense	RePAIR
DeeplabV3+	IBA (NNI)	$12.21$	$5.41$
	IBA (PRL)	$15.03$	$5.45$
	ConSeg	$16.12$	$7.05$
	OFBA	$99.71$	$8.51$
	BADSEG-O2B	$92.14$	$19.67$
SegFormer	IBA (NNI)	$4.83$	$0.12$

Appendix

Table 6: PASCAL VOC 2012. Undefended and defended attack success rate (ASR, %).

Model	mIoU-A	mIoU-B
DeeplabV3+ / ResNet-50	$66.24_{-4.99}^{+2.71}$	$58.21_{-2.05}^{+1.48}$
SegFormer / MiT-B5	$84.45_{-3.34}^{+1.28}$	$76.17_{-0.62}^{+0.22}$

Appendix

Table 7: PASCAL VOC 2012 segmentation utility. Mean IoU on clean inputs (mIoU-B) and triggered inputs (mIoU-A).

Backdoor Defense

Backdoor defense develops methods to detect, remove, or contain hidden trigger-conditioned behaviors in machine-learning models.

See all 50 papers

SemPurify: Semantics-Aware Data Purification for Black-box Backdoor Defense
Oct 16, 2026Jiawei Yu, Syed Irfan Ali Meerza, Yi Wu +2Backdoor Defense
University of Georgia · Virginia Commonwealth University · University of Oklahoma +1
OmniPatch: A Universal Adversarial Patch for ViT-CNN Cross-Architecture Transfer in Semantic Segmentation
Mar 21, 2026Aarush Aggarwal, Akshat Tomar, Amritanshu Tiwari +1Adversarial Patch Attacks Adversarial Attacks
Data Science Group, Indian Institute of Technology Roorkee, India
Beyond Small Patches: Black-Box Detection and Purification of Diverse Backdoor Triggers
Sep 2, 2026Ahmed Abdelnaby, Mohamed ElmahallawyBackdoor Detection Backdoor Defense
School of Engineering and Applied Sciences, Washington State University, Richland, WA 99354, USA

Oct 16, 2026J/K move · Enter open · S save

SemPurify: Semantics-Aware Data Purification for Black-box Backdoor Defense

Jiawei Yu, Syed Irfan Ali Meerza, Yi Wu, Amir Sadovnik+1

University of Georgia · Virginia Commonwealth University · University of Oklahoma+1

Backdoor Defense

Open paper AskPDF arXiv

Backdoor attacks have emerged as a critical security threat in machine learning, where adversaries embed triggers within poisoned training samples to manipulate a model's predictions on targeted inputs while leaving its general behavior unaffected. This stealthy nature makes backdoor attacks difficult to detect, as the model functions normally until activated by a specific malicious trigger. Current purification-based defenses use image transformations (e.g., blurring) to disrupt trigger patterns but often degrade image quality. Even with advanced restoration techniques like diffusion models, these approaches can still alter visual or semantic information, reducing utility for downstream tasks. To address this, we propose the first semantics-aware data purification framework for black-box backdoor defense, SemPurify, which explicitly incorporates multi-level semantics (i.e., low-level semantic embeddings and high-level image tags) within a text-to-image (T2I) framework to guide image restoration during purification. Using this semantic information, the T2I model subsequently refines the transformed samples, helping preserve visual details and semantic information during restoration. Extensive experiments spanning multiple backdoor scenarios, i.e., classification models, CLIP-style encoders, and generative models, demonstrate that SemPurify reliably removes backdoors while effectively preserving semantic information for downstream tasks.

Abstract

Backdoor Defense

Explore similar work

SemPurify: Semantics-Aware Data Purification for Black-box Backdoor Defense

OmniPatch: A Universal Adversarial Patch for ViT-CNN Cross-Architecture Transfer in Semantic Segmentation

Beyond Small Patches: Black-Box Detection and Purification of Diverse Backdoor Triggers