Privacy-Preserving Object Detection for Vision Transformer-Based Models

Homare Sueyoshi; Kiyoshi Nishikawa; Hitoshi Kiya

cs.CRAug 21, 2026

Privacy-Preserving Object Detection for Vision Transformer-Based Models

Authors: Homare Sueyoshi, Kiyoshi Nishikawa, Hitoshi Kiya

Organizations: Tokyo Metropolitan University 6-6, Asahigaoka, Hino, Tokyo, Japan

Abstract

We propose a novel object detection method that enables us to protect sensitive visual information of test images. Previous studies considering visual information protection focus on image classification tasks. This paper proposes an object detection method using perceptual encryption for the first time. The proposed method can achieve almost the same accuracy as that of models without any protection by utilizing the embedding structure of the Vision Transformer (ViT) and a domain adaptation technique with keys. In experiments, the effectiveness of the proposed method is verified in terms of accuracy and visual protection under the use of ViTdet, which is a ViT-based object detection model.

Figures & tables

Panel of Figure 4 — (a) Non-encrypted images

Privacy-Preserving ML

Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.

See all 433 papers

Corrupting Attention: Evasion-Based Adversarial Attacks on Encoder Attention in Detection Transformers
Aug 7, 2026Ridma Jayasundara, Shaheer Mohamed, Tharindu Fernando +6Adversarial Attacks Adversarial Robustness
Signal Processing, Artificial Intelligence and Vision Technologies Group (SAIVT), Queensland University of Technology · Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · School of Computing, National University of Singapore +1
Vision Language Model Helps Private Information De-Identification in Vision Data
Jun 8, 2026Tiejin Chen, Pingzhi Li, Kaixiong Zhou +2VLM Adaptation Privacy Protection in VLMs
Arizona State University · University of North Carolina at Chapel Hill · North Carolina State University
Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT
Jul 8, 2026Giulia Marchiori Pietrosanti, Giulio Rossolini, Giorgio ButtazzoAdversarial Patch Attacks Adversarial Attacks
Department of Excellence in Robotics and AI, Sant’Anna School of Advanced Study, Italy

Aug 7, 2026J/K move · Enter open · S save

Corrupting Attention: Evasion-Based Adversarial Attacks on Encoder Attention in Detection Transformers

Ridma Jayasundara, Shaheer Mohamed, Tharindu Fernando, Harshala Gammulle+5

Signal Processing, Artificial Intelligence and Vision Technologies Group (SAIVT), Queensland University of Technology · Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore · School of Computing, National University of Singapore+1

Adversarial Attacks Adversarial Robustness Detection Transformer Attention-Based Adversarial Attacks Adversarial Attacks on Object Detectors

Open paper AskPDF arXiv

Adversarial vulnerabilities remain a major concern for the safe deployment of neural networks, particularly in object detection, a core task embedded in many safety-critical systems. Detection transformers have emerged as leading object detectors, yet their adversarial robustness remains comparatively underexplored. Most existing attacks target the detection output rather than the attention mechanism that makes these models distinctive. In this paper, we introduce the first attack that directly optimizes an encoder-attention objective under an imperceptible, bounded

\ell_\infty

perturbation. Rather than introducing an attacker-owned sink token through a visible patch, it drives the model's own attention toward a corrupted target. We argue that encoder attention concentrates the model's spatial reasoning, so corrupting it propagates through the detection pipeline more disruptively than perturbing the detection output alone. Our attack reduces DETR-R50 mAP on COCO from 42.1 to 0.97, a

\sim 4\times

reduction in resulting mAP over the strongest existing attack under an identical perturbation budget and iteration count. We further show that this vulnerability is not specific to a particular corruption objective: across four qualitatively distinct targets, dispersion, re-ranking, permutation, and peak-suppression, detection consistently drops below 3 mAP, suggesting that the weakness arises from disrupting the attention structure itself rather than from any single target. Finally, we demonstrate that the attack generalizes across attention formulations, reducing DINO-Swin-L from 56.8 to 1.44 mAP against 7.3 for the strongest prior attack, establishing state-of-the-art on both dense and deformable attention.

Abstract

Privacy-Preserving ML

Explore similar work

Corrupting Attention: Evasion-Based Adversarial Attacks on Encoder Attention in Detection Transformers

Vision Language Model Helps Private Information De-Identification in Vision Data

Adversarial Decoys: Misdirecting Attention-Based Defenses in ViT