Adversarial Patch Generation for Visual-Infrared Dense Prediction Tasks via Joint Position-Color Optimization

He Li; Wenyue He; Weihang Kong; Xingchen Zhang

cs.CVFeb 27, 2026

Adversarial Patch Generation for Visual-Infrared Dense Prediction Tasks via Joint Position-Color Optimization

Authors: He Li, Wenyue He, Weihang Kong, Xingchen Zhang

Organizations: School of Artificial Intelligence (School of Software), Yanshan University, Qinhuangdao 066004, China. · Fusion Intelligence Laboratory, Department of Computer Science, University of Exeter, EX4 4RN, United Kingdom.

Abstract

Multimodal adversarial attacks for dense prediction remain largely underexplored. In particular, visual-infrared (VI) perception systems introduce unique challenges due to heterogeneous spectral characteristics and modality-specific intensity distributions. Existing adversarial patch methods are primarily designed for single-modal inputs and fail to account for crossspectral inconsistencies, leading to reduced attack effectiveness and poor stealthiness when applied to VI dense prediction models. To address these challenges, we propose a joint position-color optimization framework (AP-PCO) for generating adversarial patches in visual-infrared settings. The proposed method optimizes patch placement and color composition simultaneously using a fitness function derived from model outputs, enabling a single patch to perturb both visible and infrared modalities. To further bridge spectral discrepancies, we introduce a crossmodal color adaptation strategy that constrains patch appearance according to infrared grayscale characteristics while maintaining strong perturbations in the visible domain, thereby reducing cross-spectral saliency. The optimization procedure operates without requiring internal model information, supporting flexible black-box attacks. Extensive experiments on visual-infrared dense prediction tasks demonstrate that the proposed AP-PCO achieves consistently strong attack performance across multiple architectures, providing a practical benchmark for robustness evaluation in VI perception systems.

Figures & tables

Fig. 1: Comparison of existing adversarial attack work and this study. The security of visual–infrared dense prediction models, remains underexplored compared with single-modal settings. This study aims to fill this gap.

Fig. 2: Comparison between traditional adversarial patches (“Trad-Patch”) and the proposed patch (“AP-PCO”). Traditional methods determine the patch position based on a single forward pass of the target model and do not optimize color for multimodal data, which limits their effectiveness and stealthiness in visual–infrared settings. In contrast, our approach performs iterative, gradient-free global search to jointly determine position and color, and incorporates a cross-modal color reuse strategy to achieve stronger attacks and better stealthiness.

Fig. 3: Framework of cross-modal adversarial patches with position-color joint optimization. The initial population consists of a series of randomly generated circular samples. Then, through mutation, crossover, and boundary handling, a child population with diverse positions and colors is generated. A fitness function is then applied for cross-modal evaluation to compare the parent and child populations and repeat the iterative loop, ultimately obtaining the optimal patch deployed for dense prediction tasks.

Fig. 4: Variation trajectory of patch positions across iterations. Yellow markers denote the patch center coordinates of all individuals in each generation, while red markers indicate the center coordinate of the individual with the highest fitness in that generation. As iterations progress, the population gradually converges toward crowd-dense regions and ultimately stabilizes at the optimal patch location.

	Effectiveness					Stealthiness
	GAME(0) $\uparrow$	GAME(1) $\uparrow$	GAME(2) $\uparrow$	GAME(3) $\uparrow$	RMSE $\uparrow$	PSNR_RGB $\uparrow$	SSIM_RGB $\uparrow$	PSNR_T $\uparrow$	SSIM_T $\uparrow$
Clean	13.7001	18.3601	22.1256	28.6380	24.4166	-	-	-	-
PAP	14.9798	20.9912	26.6463	33.4813	25.1726	23.5981	0.9768	23.7474	0.9747
APAM	14.6624	19.1693	23.1393	30.0921	25.9723	28.0505	0.9822	26.7175	0.9738
AP-PCO(Ours)	40.5543	51.2453	56.7172	63.6817	45.1786	25.6450	0.9832	28.2151	0.9850

TABLE I: Comparison of experimental results with existing methods. Single-modal adversarial patch methods adapted to the cross-modal BL+IADM crowd counting model. “PSNR_RGB” and “PSNR_T” denote the PSNR values for visible images and infrared images, respectively. “SSIM_RGB” and “SSIM_T” denote the SSIM values for visible images and infrared images, respectively. Bold indicates the best results.

Model	Settings	Effectiveness					Stealthiness
Model	Settings	GAME(0) $\uparrow$	GAME(1) $\uparrow$	GAME(2) $\uparrow$	GAME(3) $\uparrow$	RMSE $\uparrow$	PSNR_RGB $\uparrow$	SSIM_RGB $\uparrow$	PSNR_T $\uparrow$	SSIM_T $\uparrow$
BL+IADM	Clean	13.70	18.36	22.13	28.64	24.42	-	-	-	-
	Random	14.20	18.80	22.80	29.32	24.36	27.01	0.99	31.31	0.99
	AP-PCO (Ours)	40.55	51.25	56.72	63.68	45.18	25.65	0.98	28.22	0.99
CAGNet	Clean	11.15	14.43	18.19	23.37	17.66	-	-	-	-
	Random	11.41	14.66	18.41	23.56	18.12	27.04	0.99	31.28	0.99

TABLE II: Attack Effectiveness and Stealthiness of Our Method on Different Models for the Crowd Counting Task. “Clean” Denotes the Original Image, “Random” Denotes Images With Patches Added Using Randomly Sampled Positions and Colors.

Fig. 5: Visualizations of patch attacks for the crowd counting, semantic segmentation, and image fusion tasks. As can be seen, the proposed method shows strong attack performance across three VI dense prediction tasks.

Fig. 6: Comparison of ablation experiment results for the crowd counting task (BL+IADM model), the semantic segmentation task (Openress) and the image fusion task (UNFusion). In this figure, “clean” denotes the original image; “random” denotes images with randomly added patches in both positions and color; “position” denotes images with patches optimized only for positions while color remains random; and “position-color” denotes images with patches optimized for both positions and color.

Model	Settings	Effectiveness		Stealthiness
Model	Settings	mIoU $\downarrow$	Recall $\downarrow$	PSNR _RGB $\uparrow$	SSIM _RGB $\uparrow$	PSNR _T $\uparrow$	SSIM _T $\uparrow$
Open ress	Clean	24.41	31.63	-	-	-	-
	Random	6.99	10.64	18.65	0.86	30.69	0.98
	AP-PCO (Ours)	6.69	9.90	20.05	0.86	31.88	0.98
FEA Net	Clean	52.52	70.68	-	-	-	-
	Random	50.32	68.06	21.50	0.86	36.12	0.99

TABLE III: Attack effectiveness and Stealthiness of the Proposed Method on Different Models for the Semantic Segmentation Task.

Model	Setting	Qabf $\downarrow$	PSNR $\downarrow$	SSIM $\downarrow$	VIFF $\downarrow$	CC $\downarrow$
UNFusion	Clean	0.52	13.89	0.65	0.74	0.61
	Random	0.51	13.88	0.64	0.72	0.60
	AP-PCO (Ours)	0.50	13.50	0.64	0.72	0.56
Res2Fusion	Clean	0.44	15.84	0.68	0.61	0.62
	Random	0.44	15.74	0.67	0.60	0.62
	AP-PCO (Ours)	0.43	15.00	0.67	0.59	0.56

TABLE IV: Attack Effectiveness and Stealthiness of the Proposed Method on Different Models for the Image Fusion Task.

	GAME(0) $\uparrow$	GAME(1) $\uparrow$	GAME(2) $\uparrow$	GAME(3) $\uparrow$	RMSE $\uparrow$
clean	13.70 $\pm$ 19.99	18.36 $\pm$ 19.90	22.13 $\pm$ 20.35	28.64 $\pm$ 22.21	24.42 $\pm$ 20.06
AP-PCO(Ours)	40.55 $\pm$ 19.73	51.25 $\pm$ 19.83	56.72 $\pm$ 20.29	63.68 $\pm$ 21.13	45.18 $\pm$ 20.00

TABLE V: Stability analysis of adversarial attack performance for BL+IADM crowd counting model (mean ± std).

	Effectiveness		Stealthiness
	mIoU $\downarrow$	Recall $\downarrow$	PSNR_RGB $\uparrow$	SSIM_RGB $\uparrow$	PSNR_T $\uparrow$	SSIM_T $\uparrow$
Not used	36.06	42.86	20.4394	0.8543	26.2563	0.9802
Used	39.62	56.78	20.7228	0.8544	33.3521	0.9835

TABLE VI: Effect of the Cross-modal Color Parameter Reuse Strategy on the Semantic Segmentation Task.

Modal	Crowd Counting					Semantic Segmentation		Image Fusion
Modal	GAME(0) $\uparrow$	GAME(1) $\uparrow$	GAME(2) $\uparrow$	GAME(3) $\uparrow$	RMSE $\uparrow$	mIoU $\downarrow$	Recall $\downarrow$	Qabf $\downarrow$	PSNR $\downarrow$	SSIM $\downarrow$	VIFF $\downarrow$	CC $\downarrow$
Visible	21.3622	25.8889	30.4938	36.2902	30.7622	6.97	9.96	0.4347	15.0853	0.6672	0.5956	0.5668
Infrared	36.9179	47.8553	54.7607	62.1525	43.8502	7.56	10.75	0.4347	15.1014	0.6710	0.5963	0.5695
VI	40.5543	51.2453	56.7172	63.6817	45.1786	6.69	9.90	0.4309	14.9952	0.6659	0.5912	0.5600

TABLE VII: Effect of Different Attack Modalities of Three Dense Prediction Tasks. “Visible” and “Infrared” Denote Attacks Applied to the Visible or Infrared Modality Only, While “VI” Denotes Attacks Applied to Both Modalities Simultaneously.

	Effectiveness					Stealthiness
$\alpha$	GAME(0) $\uparrow$	GAME(1) $\uparrow$	GAME(2) $\uparrow$	GAME(3) $\uparrow$	RMSE $\uparrow$	PSNR_RGB $\uparrow$	SSIM_RGB $\uparrow$	PSNR_T $\uparrow$	SSIM_T $\uparrow$
1	39.5844	53.0996	61.2997	68.8205	44.4267	25.2850	0.9792	28.1450	0.9792
0.5	39.2131	51.5339	61.2194	69.1666	46.6531	26.4535	0.9800	28.8502	0.9799
0.1	30.8848	43.2991	50.7023	58.2660	38.2710	32.8543	0.9855	33.6389	0.9837

TABLE VIII: Effect of Different Fitness Function Weights on the Crowd Counting Task.

	Effectiveness		Stealthiness
$\alpha$	mIoU $\downarrow$	Recall $\downarrow$	PSNR_RGB $\uparrow$	SSIM_RGB $\uparrow$	PSNR_T $\uparrow$	SSIM_T $\uparrow$
1	6.69	9.90	20.0540	0.8612	31.8827	0.9834
0.5	6.88	10.36	24.7938	0.8734	59.0620	0.9996
0.1	6.88	10.36	24.7985	0.8736	59.2910	0.9996

TABLE IX: Experimental results on different fitness function weights for the semantic segmentation task.

Fig. 7: Visualization of patch radius parameters attack performance. As the radius of the adversarial patch increases, its occlusion ratio in the image gradually rises, resulting in a significant degradation of stealthiness in cross-modal scenarios.

Fig. 8: Visualization results of adversarial patches against physical attacks in crowd counting, semantic segmentation, and image fusion tasks. As can be seen, the proposed method shows strong attack performance across three VI dense prediction tasks.

	Effectiveness					Stealthiness
	GAME(0) $\uparrow$	GAME(1) $\uparrow$	GAME(2) $\uparrow$	GAME(3) $\uparrow$	RMSE $\uparrow$	PSNR_RGB $\uparrow$	SSIM_RGB $\uparrow$	PSNR_T $\uparrow$	SSIM_T $\uparrow$
Clean	13.7001	18.3601	22.1256	28.6380	24.4166	-	-	-	-
r=20	20.3511	25.2802	29.4354	35.0948	29.6377	29.7634	0.9922	31.1696	0.9901
r=40	25.8050	30.3045	33.6848	38.7906	34.7603	22.8577	0.9864	25.3934	0.9854
r=80	34.1544	39.5820	42.4527	47.1011	44.3972	17.4165	0.9518	20.2550	0.9475

TABLE X: Experimental results on the patch radius parameter of the BL+IADM model for the crowd counting task.

	Effectiveness					Stealthiness
	GAME(0) $\uparrow$	GAME(1) $\uparrow$	GAME(2) $\uparrow$	GAME(3) $\uparrow$	RMSE $\uparrow$	PSNR_RGB $\uparrow$	SSIM_RGB $\uparrow$	PSNR_T $\uparrow$	SSIM_T $\uparrow$
Clean	13.7001	18.3601	22.1256	28.6380	24.4166	-	-	-	-
Color=1	25.8050	30.3045	33.6848	38.7906	34.7603	22.8577	0.9864	25.3934	0.9854
Color=2	25.2305	30.2875	33.9155	39.0918	33.5994	23.8335	0.9816	25.8799	0.9797
Color=5	28.8294	34.7558	39.1652	45.1211	36.9115	24.4387	0.9836	27.1367	0.9848
Color=10	39.5943	51.1300	56.8469	63.9860	44.1738	25.6450	0.9832	28.2151	0.9850

TABLE XI: Experimental results on the patch color number parameter of the BL+IADM model for the crowd counting task.

	Effectiveness		Stealthiness
	mIoU $\downarrow$	Recall $\downarrow$	PSNR_RGB $\uparrow$	SSIM_RGB $\uparrow$	PSNR_T $\uparrow$	SSIM_T $\uparrow$
Clean	24.41	31.63	-	-	-	-
Color=1	6.65	9.96	19.1507	0.8645	29.8579	0.9833
Color=2	6.71	9.94	19.3025	0.8625	30.3935	0.9829
Color=5	6.69	9.92	19.7081	0.8614	30.8039	0.9835
Color=10	6.69	9.90	20.0540	0.8612	31.8827	0.9834

TABLE XII: Experimental results on the patch color number parameter of the openress model for the semantic segmentation task.

	Qabf $\downarrow$	PSNR $\downarrow$	SSIM $\downarrow$	VIFF $\downarrow$	CC $\downarrow$
Clean	0.4434	15.8424	0.6784	0.6132	0.6249
Color=1	0.4305	15.0364	0.6662	0.5912	0.5618
Color=2	0.4309	14.9952	0.6659	0.5912	0.5600
Color=5	0.4393	15.2879	0.6613	0.6005	0.5847
Color=10	0.4393	15.2915	0.6615	0.6002	0.5843

TABLE XIII: Experimental results on the patch color number parameter of the Res2Fusion model for the image fusion task.

	GAME(0)	GAME(1)	GAME(2)	GAME(3)	RMSE
clean	13.7001	18.3601	22.1256	28.6380	24.4166
JPEG	30.0835	40.1116	45.5558	52.6953	36.6276
MF	30.2246	41.7568	47.1867	54.3790	36.6903
AP-PCO(Ours)	40.5543	51.2453	56.7172	63.6817	45.1786

TABLE XIV: Experimental Results of Defense Methods on the Crowd Counting Task.

Adversarial Patch Attacks

Adversarial patch attacks use localized visual patterns to manipulate machine-learning predictions and actions, including in physical environments.

See all 49 papers

Hiding in Plain Sight: An Effective Physical Adversarial Patch Attack against Visual-Infrared Fused Face Detection
Jul 25, 2026Qiucheng Yu, Tao Ni, Yihe Zhou +2Adversarial Patch Attacks Face Recognition
City University of Hong Kong · King Abdullah University of Science and Technology (KAUST)
Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability
May 21, 2026Xiang Chen, Yuxian Dong, Chao Li +6VLM Robustness Adversarial Patch Attacks
Stealthy in Semantics, Antagonistic in Space: Attacking Visible-Infrared Object Detectors via Object-Level Misalignment
Sep 16, 2026Yueqi Zhu, Qi Ming, Guo Cheng +6Physical Adversarial Attacks Adversarial Patch Attacks
Beijing University of Technology · Peking University · National Key Laboratory of Space Integrated Information System, Institute of Software, Chinese Academy of Sciences +1

Jul 25, 2026J/K move · Enter open · S save

Hiding in Plain Sight: An Effective Physical Adversarial Patch Attack against Visual-Infrared Fused Face Detection

Qiucheng Yu, Tao Ni, Yihe Zhou, Jiayimei Wang+1

City University of Hong Kong · King Abdullah University of Science and Technology (KAUST)

Adversarial Patch Attacks Face Recognition Physical Adversarial Attacks

Open paper AskPDF arXiv

Deep learning-based visual-infrared fused face detection models are increasingly deployed across a wide range of applications, yet they remain susceptible to adversarial patch attacks. Most prior attacks target either the visual or the infrared image alone in the digital domain, which renders them ineffective against fused models in the physical world. Moreover, many of these methods are readily noticeable, as their patch patterns deviate substantially from those seen in the real world. In this paper, we introduce VIPatch (Visual-Infrared Patch), a novel physical adversarial patch attack that produces inconspicuous, realistic, and natural-looking patches for facial images. Specifically, VIPatch crafts a gradient-color mask together with a band-aid sticker across both the visual and infrared images, and jointly optimizes these two elements; the resulting digital patches further guide the fabrication of their physical counterparts. Experimental results show that VIPatch achieves competitive attack success rates (over 90%) in both the digital and physical domains, while keeping the patches unobtrusive to human observers.

Abstract

Adversarial Patch Attacks

Explore similar work

Hiding in Plain Sight: An Effective Physical Adversarial Patch Attack against Visual-Infrared Fused Face Detection

Exposing Vulnerabilities in Visible-Infrared VLMs: A Unified Geometric Adversarial Framework with Cross-Task Transferability

Stealthy in Semantics, Antagonistic in Space: Attacking Visible-Infrared Object Detectors via Object-Level Misalignment