Transferable Multiform Attack for Cross-Modal Person Re-Identification
Organizations: School of Informatics, Xiamen University · Department of Computing, The Hong Kong Polytechnic University
Abstract
Cross-modal person re-identification (ReID) adversarial attacks face a transferability challenge: existing methods may exhibit source-pair overfitting, where learned perturbations become overly dependent on the feature statistics of the source modality pair, thereby limiting their transferability under heterogeneous domain shifts. We argue that a key challenge is reducing source-pair-specific bias while maintaining attack effectiveness. To address this, we propose Multiform Attack (MA). The framework first learns a universal perturbation via Mahalanobis-guided gradient optimization. Since this perturbation may still retain source-pair-specific bias, the second stage performs sparse refinement through attention-shift-filtered multi-objective evolutionary search, leveraging heterogeneous source distributions in transfer settings. By separating universal perturbation learning from sparse evolutionary refinement, MA aims to reduce reliance on source-pair-specific patterns and improve transferability to unseen domains. Extensive experiments demonstrate strong attack effectiveness on cross-modal ReID benchmarks and improved transferability in the evaluated held-out-domain settings, with additional evaluations across model architectures.
Figures & tables
| Settings | Visible to Thermal | Thermal to Visible | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Venue | mAP | mAP | ||||||
| AGW baseline Ye et al. (2022) | TPAMI 2022 | 70.05 | 86.21 | 91.55 | 66.37 | 70.49 | 87.21 | 91.84 | 65.90 |
| M-FGSM attack Bai et al. (2020) | TPAMI 2020 | 29.34 | 52.90 | 61.44 | 23.35 | 23.64 | 40.36 | 48.61 | 18.57 |
| LTA attack Gong et al. (2022) | CVPR 2022 | 12.65 | 25.24 | 34.02 | 12.80 | 10.51 | 22.93 | 31.79 | 9.74 |
| ODFA attack Zheng et al. (2023) | IJCV 2023 | 28.57 | 51.42 | 60.58 | 21.84 | 17.26 | 33.27 | 42.92 | 15.27 |
| Col.+Del. attack Yang et al. (2023b) | TPAMI 2023 | 5.12 | 16.83 | 22.10 | 4.94 | 4.92 | 14.47 | 23.04 | 4.86 |
| Settings | Visible to Infrared | Infrared to Visible | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Venue | mAP | mAP | ||||||
| DEEN baseline Zhang and Wang (2023) | CVPR 2023 | 62.53 | 90.31 | 94.73 | 65.84 | 54.96 | 84.92 | 90.91 | 62.95 |
| FGSM attack Goodfellow et al. (2015) | ICLR 2015 | 54.64 | 78.23 | 88.76 | 50.32 | 49.18 | 73.42 | 86.12 | 47.89 |
| PGD attack Madry et al. (2018) | ICLR 2018 | 49.14 | 70.34 | 81.54 | 45.76 | 44.25 | 68.39 | 80.12 | 42.81 |
| M-FGSM attack Bai et al. (2020) | TPAMI 2020 | 28.48 | 64.92 | 75.12 | 32.88 | 25.64 | 61.45 | 78.31 | 30.46 |
| LTA attack Gong et al. (2022) | CVPR 2022 | 15.16 | 56.42 | 67.53 | 21.47 | 19.54 | 58.25 | 70.72 | 24.86 |
| Cross-Dataset Method r=1 (%) mAP (%) SYSU CnMix CMPS SYSU CnMix MUA SYSU CnMix Ours SYSU Sketch CMPS SYSU Sketch MUA SYSU Sketch Ours | Cross-Dataset Method r=1 (%) mAP (%) CnMix SYSU CMPS CnMix SYSU MUA CnMix SYSU Ours Sketch SYSU CMPS Sketch SYSU MUA Sketch SYSU Ours |
| (a) Visible to Non-visible | (b) Non-visible to Visible |
| No. | Maha. | Grad. | Evol. | Struct. | mAP | rank-1 |
|---|---|---|---|---|---|---|
| 0 | 65.84 | 62.53 | ||||
| 1 | 8.54 | 12.35 | ||||
| 2 | 6.88 | 9.47 | ||||
| 3 | 3.43 | 2.49 | ||||
| 4 | 3.01 | 1.97 |
| Parameter | Value | Description |
|---|---|---|
| 8 | perturbation budget (pixel scale 0–255) | |
| Learning rate for gradient update | ||
| 0.9 | Momentum decay coefficient | |
| 30 | Number of gradient iterations in Stage 1 | |
| 0.3 | Margin in the triplet loss (Eq. 3) | |
| Diagonal regularization for the covariance matrix (Eq. 5) |
| Training Type | MUA attack | Our attack | ||
|---|---|---|---|---|
| Rank-1 (%) | mAP (%) | Rank-1 (%) | mAP (%) | |
| Normal Training | 1.85 | 3.21 | 0.83 | 1.08 |
| Adv. Training (MUA attack) | 34.53 | 30.31 | 27.14 | 24.36 |
| Adv. Training (Our attack) | 40.37 | 35.22 | 38.35 | 33.41 |
| Experiment | Protocol | Source dataset(s) | Auxiliary dataset(s) | Source / surrogate model access | Target dataset / model | Label access |
|---|---|---|---|---|---|---|
| RegDB V T / T V | Restricted | RegDB | None | AGW on RegDB | RegDB / AGW | Source labels only |
| LLCM V I / I V | Restricted | LLCM | None | DEEN on LLCM | LLCM / DEEN | Source labels only |
| SYSU CnMix | Heterogeneous transfer | SYSU | RegDB, Sketch | Models trained on accessible source/auxiliary domains | CnMix / held-out CnMix model | Source/auxiliary labels only |
| SYSU Sketch | Heterogeneous transfer | SYSU | RegDB, CnMix | Models trained on accessible source/auxiliary domains | Sketch / held-out Sketch model | Source/auxiliary labels only |
| CnMix SYSU | Heterogeneous transfer | CnMix | RegDB, Sketch | Models trained on accessible source/auxiliary domains | SYSU / held-out SYSU model | Source/auxiliary labels only |
| Sketch SYSU | Heterogeneous transfer | Sketch | RegDB, CnMix | Models trained on accessible source/auxiliary domains | SYSU / held-out SYSU model | Source/auxiliary labels only |
| Method | Rank-1 (%) | mAP (%) |
|---|---|---|
| MA | 8.87 0.25 | 8.13 0.19 |
| MA | 6.42 0.19 | 5.74 0.14 |
| ResNet Swin | Swin VGG | VGG ResNet | ||||
|---|---|---|---|---|---|---|
| Rank-1 | mAP | Rank-1 | mAP | Rank-1 | mAP | |
| 2 | 42.9 | 39.1 | 35.7 | 32.4 | 38.2 | 35.6 |
| 4 | 30.6 | 26.3 | 20.3 | 18.1 | 26.1 | 23.2 |
| 8 | 10.7 | 9.1 | 7.0 | 6.3 | 12.4 | 10.5 |
| 16 | 6.7 | 5.2 | 3.1 | 2.5 | 5.3 | 4.2 |
Transfer-Based Adversarial Attacks
Transfer-based adversarial attacks craft inputs on accessible models to fool different, often black-box, target models.
See all 90 papers