In-the-Wild Camouflage Attack on Vehicle Detectors through Controllable Image Editing

Xiao Fang; Yiming Gong; Stanislav Panev; Celso de Melo; Shuowen Hu; Shayok Chakraborty; Fernando De la Torre

Figure 1: Overview. Given a real image, our pipeline stylizes the target vehicle based on either its immediate surroundings (image-level) or a visual concept present in the overall scene (scene-level), producing stealthy adversarial examples. The numbers on the bounding boxes indicate detector confidence scores, and the absence of a box indicates that the vehicle is not detected.

Figure 2: Overview of our pipeline. As shown in (a) and (b), the pipeline consists of a No-Box Attack stage and a White-Box Attack stage. In (a), the ControlNet is fine-tuned to stylize vehicles using a reference region while preserving geometry and background through structure, style, and background supervisions (

L_{\text{struct}}

,

L_{\text{s}}

,

L_{\text{b}}

). (b) further optimizes the model against a detector

\mathcal{M}_{\text{det}}

by incorporating an additional adversarial loss

L_{\text{adv}}

and a color-consistency loss

L_{\text{c}}

. (c) summarizes the conditions provided to ControlNet under the image-level and scene-level settings, and (d) illustrates the style loss

L_{\text{s}}

that aligns vehicle latent features with the reference area.

Method	Faster-RCNN					ViTDet
	image-level		scene-level		Inf. Latency (s) $\downarrow$	image-level		scene-level		Inf. Latency (s) $\downarrow$
	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	Inf. Latency (s) $\downarrow$	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	Inf. Latency (s) $\downarrow$
Normal	98.3	-	98.3	-	-	97.8	-	97.8	-	-
AdvCAM [ 9 ]	90.5	0.968	88.6	0.963	21.8	84.1	0.964	80.8	0.963	21.8
Diff-PGD [ 44 ]	73.2	0.966	66.4	0.959	32.0	62.3	0.961	59.6	0.96	32.4
Ours	18.3	0.972	27.5	0.961	7.00	13.7	0.972	11.1	0.964	7.67

Table 1: Quantitative comparison on the LINZ Dataset for stylization-based camouflage attacks . We report the

\mathrm{AP}_{50}

, SSIM, and average sampling time.

Method	Faster-RCNN					ViTDet
	image-level		scene-level		Inf. Latency (s) $\downarrow$	image-level		scene-level		Inf. Latency (s) $\downarrow$
	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	Inf. Latency (s) $\downarrow$	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	Inf. Latency (s) $\downarrow$
Normal	85.6	-	85.6	-	-	91.4	-	91.4	-	-
AdvCAM [ 9 ]	64.7	0.680	66.4	0.651	21.8	70.5	0.678	68.3	0.650	21.8
Diff-PGD [ 44 ]	61.1	0.692	64.6	0.664	33.2	68.5	0.677	68.5	0.663	34.2
Ours	15.0	0.850	16.6	0.837	7.32	19.2	0.849	12.5	0.840	8.15

Table 2: Quantative comparison on the COCO Dataset for stylization-based camouflage attacks . We report the

\mathrm{AP}_{50}

, SSIM, and average sampling time.

Method	Type	LINZ		COCO
		Faster-RCNN	ViTDet	Faster-RCNN	ViTDet
		$\mathrm{AP}_{50}(\%)$	$\mathrm{AP}_{50}(\%)$	$\mathrm{AP}_{50}(\%)$	$\mathrm{AP}_{50}(\%)$
Normal	-	98.3	97.8	85.6	91.4
ToG [ 5 ]	Noise	83.5	82.8	72.5	80.0
DiffAttack [ 1 ]	Noise	98.0	97.5	77.7	83.2
NAP [ 16 ]	Patch	89.2	88.6	75.4	87.1

Table 3: Quantitative comparison with non-stylization camouflage attacks. “Noise” and “Patch” denote adding random perturbations and patches on vehicles to conduct camouflage attack. We report the average

\mathrm{AP}_{50}

of both strategies.

Figure 3: Qualitative comparison with other methods. The first two rows show results from the COCO dataset, and the last two rows are from the LINZ dataset. Within each dataset, the first row corresponds to the image-level strategy, and the second row corresponds to the scene-level strategy. Scene types are indicated on the left. In the “lake” scene, boats are stylized toward the water, while in the “parking lot” scene, cars are stylized toward trees. All camouflaged images are composed with real images background.

Dataset	Surrogate	Faster-RCNN	ViTDet	YOLOv5	YOLOv8	MLLM
LINZ	Normal	98.3	97.8	97.4	95.4	96.3
LINZ	Faster-RCNN	-	33.7	32.3	19.3	10.7
LINZ	ViTDet	11.4	-	38.6	25.2	11.3
COCO	Normal	85.6	91.4	88.3	90.7	87.5
COCO	Faster-RCNN	-	40.0	30.2	39.0	31.8
COCO	ViTDet	38.5	-	34.4	39.6	33.0

Table 4: Evaluation of camouflages across various models. We report the average of image-level and scene-level strategies using

\mathrm{AP}_{50}

for detectors, and classification accuracy for MLLMs.

Figure 4: Projector-based physical experiment. (a) Real images in digital space. (b) Photos captured from real images. (c) Reference areas used for style guidance. (d) Camouflaged images generated from the captured photos. (e) Photos taken after projecting the camouflaged images back onto the 3D physical models.

Figure 5: Effectiveness of background supervision.

Figure 6: Effectiveness of color-consistency loss. (a) Real images. (b) Reference areas used for style guidance. (c) Camouflaged images generated in the “No-Box Attack” stage. Camouflaged images with

L_{\text{c}}

. (d) Camouflaged images without

L_{\text{c}}

.

Figure S7 : Workflow for selecting style reference areas for the scene-level camouflage generation strategy. The process consists of four steps. In the first step, we query MoonDream for initial scene types. In the second step, we refine these categories using GPT-4o and recollect labels under five representative scene groups. In the third step, we query object-level information in each image. In the last step, we select a concept in each scene and synthesize an exemplar image containing that concept. We then segment the target concept with SAM 2. The extracted region serves as the reference area for scene-level stylization. Red circles denote we use the “house” concept in “Residential” area as an example to illustrate how we generate reference area in the last step.

Figure S8 : Qualitative evaluation of SD v3.5 on COCO. We visualize scene-level camouflage generation results. Each row corresponds to a scene type. Numbers above the bounding boxes indicate detector confidence, and the absence of a box indicates a missed detection. All camouflaged vehicles are composited onto the original real-image backgrounds.

Figure S9 : Qualitative evaluation of SD v3.5 on LINZ. We visualize image-level camouflage generation results.

Method	Faster-RCNN					ViTDet
	image-level		scene-level		Inf. Latency (s) $\downarrow$	image-level		scene-level		Inf. Latency (s) $\downarrow$
	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	Inf. Latency (s) $\downarrow$	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	Inf. Latency (s) $\downarrow$
Normal	98.3	-	85.6	-	-	97.8	-	91.4	-	-
Ours (SD v1.5)	18.3	0.972	16.6	0.837	7.32	13.7	0.972	12.5	0.840	8.15
Ours (SD v3.5)	17.0	0.978	27.9	0.845	4.86	8.8	0.978	29.7	0.844	4.88

Table S5 : Quantative comparison between SD v1.5 and v3.5. The image-level strategy is evaluated on the LINZ dataset, while the scene-level strategy is evaluated on the COCO dataset. We report the

\mathrm{AP}_{50}

, SSIM, and average sampling time per image.

Dataset	Target	Strategy	No-Box Attack					White-Box Attack
Dataset	Target	Strategy	$L_{\text{struct}}$	$L_{\text{s}}$	$L_{\text{b}}$	lr	iter	$L_{\text{struct}}$	$L_{\text{s}}$	$L_{\text{b}}$	$L_{\text{c}}$	$L_{\text{adv}}$	lr	iter
COCO	Faster-RCNN	Scene-Level	5.0	1.0	0.0	5e-6	12000	5.0	1.0	0.0	2.0	1.0	4e-6	15000
COCO	Faster-RCNN	Image-Level	2.5	2.5	1.0	2e-6	15000	4.0	1.0	1.0	2.0	1.0	2.5e-6	20000
COCO	ViTDet	Scene-Level	5.0	1.0	0.0	5e-6	12000	5.0	1.0	0.0	2.0	1.0	4e-6	15000
COCO	ViTDet	Image-Level	2.5	2.5	1.0	2e-6	15000	10.0	2.5	1.0	2.5	1.0	4e-6	20000
LINZ	Faster-RCNN	Scene-Level	4.0	1.0	0.0	2.5e-6	10000	10.0	2.5	0.0	2.5	1.0	2e-6	12000

Table S6 : Training parameters of each experiment . “lr” denotes learning rate. “iter” denotes the total number of training iterations. “

L_{\text{struct}}

,

L_{\text{s}}

,

L_{\text{b}}

,

L_{\text{c}}

,

L_{\text{adv}}

” denotes the coefficents of structure preservation loss, style loss, background reconstruction loss, color-consistency loss, and adversarial loss, respectively.

Figure S10 : More Visualizations on the COCO dataset. We visualize scene-level camouflage generation results. We present examples in urban, rural, road, sky , and lake scenes, where they are associated with building, grass, tree, sky , and water as representative concepts.

Figure S11 : More Visualizations on the COCO dataset. We visualize image-level camouflage generation results.

Figure S12 : Visualizations on the LINZ dataset. We present examples in residential, industrial, agricultural, parking lot, and highway scenes, where they are matched with house, building, field, tree , and grass as representative concepts.

Figure S13 : More Visualizations on the LINZ dataset. We visualize image-level camouflage generation results.

Dataset	Method	Faster-RCNN		ViTDet
Dataset	Method	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$	$\mathrm{AP}_{50}(\%)\downarrow$	SSIM $\uparrow$
COCO	Normal	85.6	-	91.4	-
COCO	Ours	16.6	0.837	12.5	0.840
COCO	Ours*	21.1	0.846	20.2	0.845
LINZ	Normal	98.3	-	97.8	-
LINZ	Ours	27.5	0.961	11.1	0.964

Table S7 : Quantitative evaluation of alternative scene–concept pairings for the scene-level strategy on COCO and LINZ. “Ours” denotes the scene–concept groups described in Sec. 4.1 , while “Ours*” adopts the alternative set evaluated in Sec. H.3 .

Figure S14 : Visualizations on the COCO dataset. We present examples in urban, rural, road, sky , and lake scenes, where they are matched with with platform, tree, road, cloud , and beach as representative concepts.

Figure S15 : Visualizations on the LINZ dataset. We present examples in residential, industrial, agricultural, parking lot, and highway scenes, where they are matched with with grass, sidewalk, parking lot, dirt , and parking lot as representative concepts.

Method	Faster-RCNN		ViTDet
Method	image-level	scene-level	image-level	scene-level
Normal	85.6	85.6	91.4	91.4
Baseline	15.0	16.6	19.2	12.5
Denoise	29.8	28.2	36.4	27.4
Smooth	21.4	25.4	25.5	23.0

Table S8 : Evaluation under preprocessing-based defense strategies on the COCO dataset. “Normal” denotes clean images, “Baseline” denotes adversarial images without defense, “Denoise” applies Non-local Means denoising, and “Smooth” applies bilateral filtering.

Figure S16 : Comparing our pipeline with one-stage variants .

Dataset	Method	Faster-RCNN		ViTDet
Dataset	Method	image-level	scene-level	image-level	scene-level
COCO	Normal	85.6	85.6	91.4	91.4
COCO	W/O $L_{\text{adv}}$	67.2	65.1	73.6	72.1
COCO	Ours	18.3	27.5	13.7	11.1
LINZ	Normal	98.3	98.3	97.8	97.8
LINZ	W/O $L_{\text{adv}}$	92.4	77.2	85.2	52.9

Table S9 : Effectiveness of adversarial loss.

Figure S17 : Effectiveness of structure preservation loss.

Figure S18 : Effectiveness of style loss.

Figure S19 : Human Evaluation of stealthiness comparing with state-of-the-art methods.

Figure S20 : Human Evaluation of stealthiness across ablation variants.

Figure S21 : Projector-based physical setup for the LINZ dataset. (a) Device configuration with the projector and camera aligned to minimize parallax. (b) A real image is projected onto a whiteboard with a 3D-printed car model placed at the corresponding location. (c) Alternative view showing attachment of the printed model to the board.

Figure S22 : Projector-based physical setup for the COCO dataset. (a) Hardware configuration. (b) 3D-printed scene reconstructed from two COCO images via single-view reconstruction. (c-d) Real images projected onto the corresponding 3D-printed models.

Figure S23 : Transferability of scene-level camouflage across locations in COCO dataset. For each scene type, we composite the vehicle into five additional backgrounds to create synthetic scenes while keeping the vehicle appearance unchanged. In each scene group, the first row shows detection results on clean vehicles, and the second row shows results on camouflaged vehicles. “Original” denotes the original COCO scene, whereas “Synthetic” refers to the newly composed scenes with replaced backgrounds.

Figure S24 : Projector-based physical experiment for the LINZ dataset. The first three rows present examples produced using the image-level strategy, and the last three rows illustrate the scene-level strategy. (a) Real images in digital space. (b) Photos captured from real images. (c) Reference areas used for style guidance. (d) Camouflaged images generated from the captured photos. (e) Photos taken after projecting the camouflaged images back onto the 3D car models.

Figure S25 : Projector-based physical experiment for the COCO dataset. The first three rows present examples produced using the image-level strategy, and the last three rows illustrate the scene-level strategy. (a) Real images in digital space. (b) Photos captured from real images projected on the 3D-printed scenes. (c) Reference areas used for style guidance. (d) Camouflaged images generated from the captured photos. (e) Photos taken after projecting the camouflaged images back onto the 3D-printed scenes.

Figure S26 : Comparison of camouflaged vehicles and their corresponding style reference areas in HSV hue space.

Figure S27 : Failure cases of the image-level camouflage generation strategy on the COCO dataset.

Figure S28 : Image-level user study guideline.

Figure S29 : Image-level example question on the COCO dataset.

Figure S30 : Image-level example question on the LINZ dataset.

Figure S31 : Scene-level user study guideline.

Figure S32 : Scene-level example question on the COCO dataset.

Figure S33 : Scene-level example question on the LINZ dataset.

Figure S34 : Example question on the COCO dataset.

Figure S35 : Example question on the LINZ dataset.

In-the-Wild Camouflage Attack on Vehicle Detectors through Controllable Image Editing

Abstract

Figures & tables

Adversarial Attacks on Object Detectors

Structured Adversarial Camouflage via Voronoi Diagrams

AdvTiles: Physical Adversarial Camouflage Clothing against Person Detectors via Learnable Tiles

Detectors Learn the Wrong Thing: Shortcut-Resistant Adversarial Training Against Physically Realizable Attacks

Abstract

Adversarial Attacks on Object Detectors

Explore similar work

Structured Adversarial Camouflage via Voronoi Diagrams

AdvTiles: Physical Adversarial Camouflage Clothing against Person Detectors via Learnable Tiles

Detectors Learn the Wrong Thing: Shortcut-Resistant Adversarial Training Against Physically Realizable Attacks