Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders

Tao Yang; Jianying Zhou

cs.CVOct 8, 2026

Does Target Alignment Mean Target Recovery? An Evidence-Ladder Study of Adversarial Claims on Contrastive Encoders

Authors: Tao Yang, Jianying Zhou

Organizations: Singapore University of Technology and Design

Abstract

Adversarial attacks on vision-language models optimize an image toward a text target, then cite the attacked model's similarity score as evidence of success. We ask whether that score - victim-space target alignment (VTS) - predicts recovery of the target by an independent model. We first build a measurement instrument: supervised judges outside the attacked geometry, real-target blend controls, shuffled-target negatives, and a reference level derived from a 50% target-image blend. Two preregistered studies then compare six contrastive encoders under a matched attack at three perturbation budgets. Robustly trained encoders (FARE, TeCoA, PMG, TRADES) transfer substantially more independent evidence than vanilla CLIP or SigLIP; all eight contrasts reject at the bootstrap floor. However, no cell reaches the blend-derived reference level. The three best cells fall within its replication band, leaving practical recovery undecided. Within robust encoders, per-sample alignment gain correlates with evidence gain ( $ρ= 0.24-0.51$ ); within vanilla CLIP the correlation is consistent with zero. Across encoders we find no monotone alignment-evidence relation. VTS is therefore informative only within a fixed robust encoder, and we provide a reporting protocol in its place.

Figures & tables

Level	Construct	Admissible evidence	Does not establish
L0	Target intent	WordNet/ontology map	any change in the image
L1	Victim alignment	frozen-target cosine/margin	independent recognition
L2	Independent recovery	validated external judges	human perception
L3	Downstream propagation	caption/VQA/retrieval endpoints	human perception
L4	Human grounding	preregistered human study	universal meaning

Table 1: Evidence levels and what each does not establish.

Panel of Figure 1 — Figure 1: Evidence levels. Adjacent levels do not imply one another.

Figure 2: Instrument dose–response on the R50 judge: ITE (left axis) and top-1 recovery (right axis) vs. blend dose. The reference level equals the ITE at the

\alpha{=}0.5

dose.

Quantity	Value
Primary judge	ResNet-50 (ImageNet-1k, supervised)
Gate A (PC1/PC2/negative controls)	PASS
Held-out ITE at the anchor dose	$5.71$ [95% CI $5.12$ , $6.32$ ]
Held-out ITR top-5 increase	$0.324$ [95% CI $0.268$ , $0.384$ ]
Dose–response (Spearman)	$\rho=1.00$ (monotone)
Frozen margin $S_{\min}$	$5.3711$

Table 2: Study A: validation of the L2 independent-evidence instrument (held-out split,

n{=}250

). The pre-specified reference level is anchored at the

\alpha^{*}{=}0.50

blend dose (half of each source image’s pixels replaced by a real target-class image) and frozen before Study B.

Contrast	Pairs	Mean difference	Holm reject
FARE-vanilla @ $\epsilon{=}$ 16	300	+1.83	yes
FARE-vanilla @ $\epsilon{=}$ 64	300	+2.60	yes
TeCoA-vanilla @ $\epsilon{=}$ 16	300	+2.55	yes
TeCoA-vanilla @ $\epsilon{=}$ 64	300	+4.45	yes
FARE-TeCoA @ $\epsilon{=}$ 16	300	-0.72	no
FARE-TeCoA @ $\epsilon{=}$ 64	300	-1.85	no

Table 3: Study B boundary contrasts (one-sided robust

>

base on paired ITE differences,

n{=}300

pairs each; class-cluster bootstrap; step-down Holm over the 6-contrast family,

\alpha{=}0.05

Cell (study, victim, $\epsilon$ )	mean	CI low	$\geq\!5.371$	$\geq\!4.65$
B: TeCoA@64	5.24	4.74	no	yes flip
C: TeCoA@64	4.98	4.35	no	yes flip
C: PMG@64	4.90	4.42	no	yes flip
C: TRADES@64	4.30	3.85	no	no
All remaining 23 cells: mean ITE $<4.0$ , negative under both anchors

Table 4: Anchor-sensitivity of the L2-positive-at-the-anchor verdict (all 27 confirmatory cells). A cell counts iff mean ITE

\geq

anchor AND CI lower

>0

(CI condition holds in every cell). Frozen anchor

5.371

; replicated anchor dose

4.65

. Exactly three cells sit inside the band and flip; every other cell is negative under both.

Figure 3: Within-encoder slopes at

\epsilon{=}64/255

across six encoders (Study-C cohort,

n{=}300

per panel). X-axis ranges vary; line angles are not directly comparable.

Victim	$\epsilon$	$n$	ITE mean	ITE sd	CI 95 lower	ITR top-5	VTS
vanilla	4/255	300	0.1653	0.6131	0.1015	0.0033	0.4298
vanilla	16/255	300	0.611	1.6177	0.4464	0.01	0.4784
vanilla	64/255	300	0.7935	1.9026	0.5988	0.0133	0.4879
FARE	4/255	300	0.3117	0.8091	0.2361	0.0067	0.2362
FARE	16/255	300	2.4412	2.7311	2.1131	0.08	0.3499
FARE	64/255	300	3.3974	3.2656	3.025	0.19	0.3842

Table 5: Study B (confirmatory): independent target evidence ITE

{}_{S_{a}}

per cell, pooled over mapping seeds 101/202/303 (

n{=}300

/cell; R50 judge; held-out ImageNet-100 rows). Rule: L2-positive at the anchor

\iff

mean

\geq S_{\min}=5.371

and class-cluster bootstrap 95% CI lower bound

>0

. ITR = anchor-in-top-5 rate. VTS = victim-space target similarity after attack (L1 diagnostic).

Between-victim means (van/FARE/TeCoA):
Victim	$\rho$ ( $\epsilon{=}4$ )	$\rho$ ( $\epsilon{=}16$ )	$\rho$ ( $\epsilon{=}64$ )
vanilla	-0.041	-0.016	-0.001
FARE	+0.314	+0.368	+0.236
TeCoA	+0.313	+0.512	+0.409
VTS $0.465/0.323/0.262$ vs ITE $0.52/2.05/2.90$ ;
$\rho=-1.0$ ( $n{=}3$ , descriptive)

Table 6: Two-level diagnosticity of victim-space alignment (confirmatory). Within-cell: Spearman

\rho

between per-sample

\Delta

VTS and ITE

{}_{S_{a}}

(all permutation

p\leq 0.0005

except vanilla, ns). Between-victim: victim-mean VTS vs victim-mean ITE. Pooled cell-level

\rho

shown to exhibit the Simpson cancellation that pooled correlations conceal.

$\epsilon$	Metric	vanilla	FARE	TeCoA	PMG	TRADES	SigLIP
4/255	ITE mean	0.1961	0.3288	0.2786	0.3893	0.3748	0.0833
	CI 95 lower	0.1288	0.2417	0.1987	0.3094	0.2954	0.0196
	ITR top-5	0.01	0.0033	0.0067	0.01	0.0033	0.0
	VTS	0.4325	0.2342	0.1846	0.2032	0.2075	0.2367
16/255	ITE mean	0.5428	2.3106	2.9692	3.0684	2.8698	0.2907
	CI 95 lower	0.3955	1.9795	2.5086	2.6842	2.4941	0.1533

Table 7: Study C per-cell results by budget (confirmatory;

n{=}300

/cell; R50 judge; cf. Holm-4 and H_C2/H_C3 in the main text). No cell is L2-positive at the anchor.

Victim	$\epsilon$	ITE mean			VTS
		frozen	PGD-100	C&W-100	frozen	PGD-100	C&W-100
vanilla	16/255	0.6571	0.3098	0.3626	0.4789	0.5723	0.4898
vanilla	64/255	0.7479	0.6111	0.5009	0.4883	0.63	0.5029
FARE	16/255	2.569	2.4876	2.0176	0.3483	0.4254	0.3705
FARE	64/255	3.3824	2.991	1.5034	0.383	0.5423	0.4411
TeCoA	16/255	3.1555	3.4647	3.2887	0.2762	0.3296	0.3

Table 8: Recipe-frontier check (exploratory split; identical cohort, anchors, and judge to the calibration pilot). ITE under the preregistered recipe (frozen-20) vs standard stronger optimizers at 100 iterations. No variant crosses

S_{\min}=5.371

in any cell; the preregistered recipe attains the best ITE in five of six victim–budget combinations. PGD-100 attains higher victim-side alignment (VTS) than the recipe while transferring less evidence.

Explore similar work

Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models
Jun 4, 2026Liangsheng Liu, Si Chen, Jiamin Wu +5VLM Robustness Adversarial Attacks on VLMs
University of Science and Technology of China · The Chinese University of Hong Kong · Zhejiang University +2
Attack Ensembles Expose a Safety-Utility Trade-off in Black-Box Guard Defenses Against Encoded VLM Jailbreaks
Jul 29, 2026Haoyu Zhang, Zhuoxi Wang, Shibo Zheng +7VLM Robustness Adversarial Attacks on VLMs
Northeastern University
It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $ε\leq 4/255$
Sep 29, 2026Binchi Zhang, Apurva Narayan, Atrisha SarkarVLM Robustness Adversarial Attacks on VLMs
University of Western Ontario

Jun 4, 2026, cs.CVJ/K move · Enter open · S save

Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models

Liangsheng Liu, Si Chen, Jiamin Wu, Weiwei Feng+4

University of Science and Technology of China · The Chinese University of Hong Kong · Zhejiang University+2

VLM Robustness Adversarial Attacks on VLMs

Open paper AskPDF arXiv

Vision-Language Models (VLMs), such as CLIP, have shown strong zero-shot generalization but remain highly vulnerable to adversarial perturbations, posing serious risks in real-world applications. Test-time defenses for VLMs have recently emerged as a promising and efficient approach to defend against adversarial attacks without requiring costly large-scale retraining. In this work, we uncover a surprising phenomenon: under diverse input transformations, adversarial images in CLIP's feature space consistently shift along a dominant direction, in contrast to the dispersed patterns of clean images. We hypothesize that this dominant shift, termed the Defense Direction, opposes the adversarial shift, pointing features back toward their correct class centers. Building on this insight, we propose Directional Bias-guided Defense (DBD), a test-time framework that estimates the Defense Direction and employs a DB-score-based two-stream reconstruction strategy to recover robust representations. Experiments on 15 datasets demonstrate that DBD not only achieves SOTA adversarial robustness while preserving clean accuracy, but also reveals the counterintuitive result that adversarial accuracy can even surpass clean accuracy. This demonstrates that adversarial perturbations inherently encode directional priors about the true decision boundary.

Abstract

Explore similar work

Adversarial Attacks Already Tell the Answer: Directional Bias-Guided Test-time Defense for Vision-Language Models

Attack Ensembles Expose a Safety-Utility Trade-off in Black-Box Guard Defenses Against Encoded VLM Jailbreaks

It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at ε≤4/255ε\leq 4/255ε≤4/255

It Takes Little to Rewrite Perception: Targeted Semantic Substitution in Vision-Language Models at $ε\leq 4/255$