FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models

Jiaming Zhang; Zherui Li; Siqi Guo; Junhao Dong; Kun Wang; Wei Yang Bryan Lim

cs.CVOct 15, 2026

FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models

Authors: Jiaming Zhang, Zherui Li, Siqi Guo, Junhao Dong, Kun Wang, Wei Yang Bryan Lim

Organizations: College of Computing and Data Science Nanyang Technological University, Singapore

Abstract

As vision-language models (VLMs) transition from frozen-encoder pipelines to native multimodal architectures, the visual channel increasingly serves as both a perceptual input and a carrier of rendered text that competes with user prompts. Current single-attack, pixel-centric evaluations fail to capture this complexity: they collapse distinct visual attack surfaces into a single scalar and ignore how architectural shifts alter vulnerability. We introduce PaRMA, an evidence-based auditing framework that defines pathway-conditioned risk over two primary image-level interventions: pixel perturbation and rendered-text insertion. Instead of arbitrary attack enumeration, PaRMA uses pathway-specific diagnostics to expose architecture-specific blind spots. Notably, we uncover a see-but-not-deceived signature where native VLMs attend heavily to typographic overlays but visually reject them, and we identify an objective mismatch between encoder-level and answer-level attacks in deep-fusion models. To exploit these findings, we instantiate FusionAudit, introducing two diagnostic-motivated attacks: MANTIS (margin-aware pixel perturbation) and Nat-Evid (host-congruent semantic insertion). Evaluating 18 open-weight VLMs across a novel nativeness taxonomy, we demonstrate that legacy single-attack rankings (e.g., PGD) are unstable for native models. Our diagnostic-driven attacks substantially raise the estimated residual risk, showing that PaRMA provides a falsifiable, evidence-driven auditing methodology for uncovering vulnerabilities missed by standard benchmarks.

Figures & tables

Figure 1: The PaRMA framework and FusionAudit pipeline. Native VLMs are evaluated over two visual pathways: pixel perturbation and rendered-text insertion. Baseline attacks (Vis-PGD, End-PGD, Typo-Evid) yield initial failures and diagnostic readouts. These diagnostics surface architecture-associated failure signatures (e.g., the see-but-not-deceived signature), guiding the generation of diagnostic-motivated augmented attacks (MANTIS, Nat-Evid) to compute the final falsifiable residual risk $\widehat{R}^{+}$ .

		Pixel pathway				Rendered-text pathway			Audit summary
Model	Clean	Vis-PGD	End-PGD	MANTIS	$\delta$	Typo-Evid	Nat-Evid	$\delta$	$\widehat{R}^{0}$	$\widehat{R}^{+}$
$\mathrm{L}_{1}$ : alignment-only / projection-based models
SmolVLM2-2.2B	$61.2$	$52.7$	$80.4$	$99.5$ (+19.1)	$13.4$	$17.2$	$30.9$ (+13.7)	$22.2$	86.7	$100.0$ (+13.3)
BLIP2-OPT-2.7B	$19.8$	$20.2$	$43.1$	$63.0$ (+19.9)	$25.9$	$69.4$	$4.4$ ( $-$ 65.0)	$1.0$	83.5	$91.2$ ( 0 +7.7)
InstructBLIP-Vicuna-7B	$26.5$	$60.5$	$70.5$	$98.0$ (+27.5)	$11.8$	$10.8$	$6.0$ ( 0 $-$ 4.8)	$3.0$	87.7	$99.5$ (+11.8)
LLaVA-1.5-7B	$30.7$	$47.5$	$82.0$	$94.8$ (+12.8)	$9.5$	$8.5$	$20.0$ (+11.5)	$15.2$	87.6	$97.2$ ( 0 +9.6)

Table 1: Pathway-Complete Robustness Audit. Results are aggregated over the three-dataset audit set (ChartQA, DocVQA, and MME). All ASRs are percentages conditioned on each model’s clean-correct set (

n_{c}

). Gray parentheticals show the absolute ASR gain over the paired baseline (e.g., MANTIS vs. End-PGD). Crucially, the

\delta

columns report the exclusive-flip rate : the fraction of

n_{c}

broken by the augmented attack that survived all corresponding baseline attacks (e.g., MANTIS

\setminus

(Vis-PGD

\cup

End-PGD)). Light gray columns mark the final audit-summary risks. Level means are unweighted macro-averages.

Panel of Figure 2 — Figure 2: MANTIS Stability and Ablation. Evaluated on a balanced 300-sample subset across all 18 models. (a) Min-max ASR variance across three seeds, centered on the per-model mean. (b) Clean-correct ASR as PGD steps increase at $\varepsilon=8/255$ . (c) Clean-correct ASR across varying $\ell_{\infty}$ radii at $t=40$ . The rightmost bars in each panel show the 18-model mean.

Level	$P_{\mathrm{read}}$	$P_{\mathrm{use}}$	RUG
$\mathrm{L}_{1}$	38.9	22.4	+16.5
$\mathrm{L}_{2}$	68.6	15.9	+52.7
$\mathrm{L}_{3}$	81.0	11.6	+69.5

Table 2: Read–Use Gap. Values are unweighted level means over the 18-model audit.

Panel of Figure 3 — Figure 3: Attention-Level Evidence for the See-but-not-Deceived Signature. (Left) Level-mean Overlay Attention Ratio (OAR). In $\mathrm{L}_{2}$ and $\mathrm{L}_{3}$ , OAR more than doubles over clean ( $+138\%$ and $+135\%$ ), while Typo-Evid ASR falls with fusion level. (Right) Qualitative attention map for Qwen3.5-2B on an MME sample. Attention concentrates on the injected banner (blue box), while the model keeps the visual answer and outputs "no".

Figure 4: Generator Sensitivity of Nat-Evid. Model-macro ASR across four generated variants.

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 443 papers

Breaking the weakest link to evade vision language models
Aug 19, 2026Ilan Zini, Boussad Addad, Katarzyna KapustaVLM Robustness Adversarial Attacks on VLMs
ESILV student and Thales intern · Thales, Thales cortAIx Labs, France
Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
Apr 6, 2026Jia Chengyu, AprilPyone MaungMaung, Huy H. Nguyen +2VLM Robustness VLM Evaluation
Zhejiang University of Technology, Hangzhou, Zhejiang, China · National Institute of Informatics, Tokyo, Japan
BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models
Nov 24, 2025Juncheng Li, Yige Li, Hanxun Huang +5VLM Evaluation VLM Robustness
Fudan University · Singapore Management University · The University of Melbourne

Aug 19, 2026J/K move · Enter open · S save

Breaking the weakest link to evade vision language models

Ilan Zini, Boussad Addad, Katarzyna Kapusta

ESILV student and Thales intern · Thales, Thales cortAIx Labs, France

VLM Robustness Adversarial Attacks on VLMs Adversarial Attacks

Open paper AskPDF arXiv

Vision Language Models (VLMs) have recently emerged as a critical component of multimodal AI systems, enabling joint reasoning over visual and textual inputs in real-world and safety-critical applications. Despite their growing deployment, the robustness of VLMs against adversarial threats remains insufficiently explored, particularly in the context of evasion attacks targeting multimodal alignment. In this work, we investigate the vulnerability of VLMs to adversarial perturbations applied to visual inputs and study two attack settings: untargeted attacks, where the goal is to disrupt the model's interpretation of the original image, and targeted attacks, where the adversary aims to force the model to generate a specific semantic description unrelated to the original image. To efficiently generate adversarial examples, we propose a gradient-based attack method that performs optimization exclusively on the vision encoder of the VLM rather than on the entire multimodal architecture. This design significantly reduces the computational cost and resource requirements of the attack while maintaining strong effectiveness. We evaluate our approach on several open-source VLMs, including Qwen2.5-VL, Granite-Vision, FastVLM, and Phi-3.5-Vision, and show that small, human-imperceptible perturbations can substantially alter the textual interpretation produced by the models. Our findings highlight the vulnerability of modern VLMs to adversarial manipulation and emphasize the need for improved robustness and security mechanisms in multimodal AI systems.

Abstract

VLM Robustness

Explore similar work

Breaking the weakest link to evade vision language models

Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models