Text-Anchored Semantic Perturbations for Transferable Jailbreak Attacks on Multimodal Large Language Models

Wenyun Li; Guiping Cao; Xiangyuan Lan; Zheng Zhang

cs.CLAug 23, 2026

Text-Anchored Semantic Perturbations for Transferable Jailbreak Attacks on Multimodal Large Language Models

Authors: Wenyun Li, Guiping Cao, Xiangyuan Lan, Zheng Zhang

Organizations: Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Pazhou Laboratory (Huangpu), Guangzhou, China

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress in vision-language interaction, yet their safety alignment remains vulnerable to jailbreak attacks. A key challenge is that safety behavior learned in the textual space does not reliably transfer to fused cross-modal representations, leaving multimodal inputs exploitable through latent semantic cues. We propose Text-Anchored Semantic Perturbation Attack (TA-SPA), a black-box jailbreak framework that optimizes transferable perturbations in a text-anchored semantic space. TA-SPA integrates Text-Anchored Semantic Factorization (TASF), which encourages the separation of cross-modal semantic factors from modality-specific residuals, with Semantic-Preserving Augmentation (SPA), which diversifies harmful target anchors while preserving semantic consistency. Experiments show strong attack effectiveness and transfer to commercial MLLMs, with competitive performance under representative defenses. Additional controls and probing support the intended factorization without implying perfect disentanglement, motivating representation-level safety alignment beyond input-level filtering.

Figures & tables

Figure 1: An overview of our proposed TA-SPA. TASF encourages text-anchored semantic intent to be distinguished from modality-specific residuals, guiding perturbations toward transferable cross-modal semantics. SPA further diversifies residual variations while keeping the semantic factor fixed, reducing overfitting to superficial visual or lexical cues.

MLLM	Method	AdvBench			MM-SafetyBench			VAJM
		D-ASR	G-ASR	Tox.	D-ASR	G-ASR	Tox.	D-ASR	G-ASR	Tox.
LLaVA-7B	imgJP (arXiv’24)	55.5	49.5	0.32	67.5	54.5	0.46	55.0	38.0	0.28
	FigStep (AAAI’25)	80.5	72.0	0.55	84.0	55.5	0.49	72.5	66.0	0.45
	HADES (ECCV’24)	84.5	76.5	0.62	80.5	75.5	0.54	77.5	72.0	0.48
	IDEATOR (ICCV’25)	77.5	65.5	0.47	75.0	60.0	0.46	68.5	56.5	0.38
	MML (ACL’25)	72.5	66.5	0.36	66.0	58.5	0.42	60.0	48.5	0.28

Table 1: Attack effectiveness across victim MLLMs and harmful-query benchmarks. D-ASR and G-ASR denote dictionary-based and GPT-based attack success rates, respectively. Tox. is measured by the Google Perspective API.

VLLM	Method	AdvBench
		D-ASR	G-ASR	Tox.
MiniGPT-4	GCG	43.5	56.0	0.29
	AutoDAN	66.0	58.5	0.32
	PAIR	74.5	65.0	0.43
	TAP	75.0	68.0	0.50
	Ours	80.5	75.0	0.61

Table 2: Comparison with text-based jailbreak attacks on AdvBench.

Method	GPT-4o	Gemini	ERNIE	Avg.
imgJP	38.0	48.7	29.8	38.8
FigStep	47.5	58.6	28.5	44.9
HADES	35.4	65.3	25.4	42.0
IDEATOR	45.5	64.2	38.6	49.4
MML	34.2	49.2	27.3	36.9
Ours	52.3	68.9	35.7	52.3

Table 3: Transferability to commercial MLLMs measured by GPT-based ASR (%) on MM-SafetyBench. Higher is better.

Variant	G-ASR
CLIP target matching	18.2
+ TASF	23.4
+ Target paraphrases	43.1
+ Source repulsion	55.6
Full TA-SPA	60.0

Table 4: Ablation study on MiniGPT-4. We report average GPT-based ASR (%) over three benchmarks.

Variant	G-ASR
Clean proxy ( $\delta=0$ )	45.0
Random bounded noise	46.0
Direct CLIP matching	18.2
Full TA-SPA	60.0

Table 5: Matched perturbation controls on MiniGPT-4.

Feature	Harm-category macro-F1
Semantic factor $s$	0.75
Residual factor $u$	0.46

Table 6: Linear-probe validation using frozen TASF features.

Surrogate architecture	ASR
SigLIP	50.0
BLIP	55.0
CLIP	60.0

Table 7: Transfer results with different surrogate architectures.

Judge	ASR
Human majority vote	77.0
Qwen3	65.0
DeepSeek	68.0

Table 8: Judge-calibration results with human and independent LLM assessments.

Defense	Attack	LLaVA-7B	MiniGPT-4	InstructBLIP
VLGuard	FigStep	13.4	24.6	18.9
	HADES	9.8	17.9	13.2
	Ours	18.4	26.7	25.5
ProEAT	FigStep	23.7	28.5	19.4
	HADES	17.5	24.6	18.0
	Ours	26.4	27.1	23.8

Table 9: Comparison against defense countermeasures measured by GPT-based ASR (%). Higher is better.

Figure 4: Training loss (log) of TASF representation learning.

Figure 5: Qualitative visualization results of TA-SPA on multimodal jailbreak examples.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Sensitivity of TA-SPA to the source-repulsion weight

\beta

Figure 7: Effect of paraphrase count

m

on attack success rate (ASR) and runtime.

Panel of Figure 8 — Figure 8: Additional qualitative jailbreak examples on commercial GPT-4o APIs.

Adversarial Attacks on Unified Multimodal Models

See all 23 papers

MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs
Feb 28, 2026Yilian Liu, Xiaojun Jia, Guoshun Nan +6Multimodal Jailbreak Attacks Adversarial Attacks on Unified Multimodal Models
Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore · Guilin University of Electronic Technology, China
DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputs
May 18, 2026Wenzhuo Xu, Zhipeng Wei, Zonghao Ying +4Multimodal Large Language Models LLM Safety
AI Security Lab, Beijing, China · International Computer Science Institute, CA, USA · UC Berkeley, CA, USA +1
Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models
Dec 21, 2024Yanxu Mao, Peipei Liu, Tiehan Cui +3Adversarial Attacks on LLMs Multimodal Large Language Models
School of Computer Science and Technology, Shanghai Jiao Tong University, Shanghai, China · Zhongguancun Laboratory, Beijing, China · School of Software, Henan University, Kaifeng, China +1

Feb 28, 2026J/K move · Enter open · S save

MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs

Yilian Liu, Xiaojun Jia, Guoshun Nan, Jiuyang Lyu+5

Beijing University of Posts and Telecommunications, China · Nanyang Technological University, Singapore · Guilin University of Electronic Technology, China

Multimodal Jailbreak Attacks Adversarial Attacks on Unified Multimodal Models LLM Safety Multimodal Large Language Models

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have achieved remarkable performance but remain vulnerable to jailbreak attacks that can induce harmful content and undermine their secure deployment. Previous studies have shown that introducing additional inference steps, which disrupt security attention, can make MLLMs more susceptible to being misled into generating malicious content. However, these methods rely on single-image masking or isolated visual cues, which only modestly extend reasoning paths and thus achieve limited effectiveness, particularly against strongly aligned commercial closed-source models. To address this problem, in this paper, we propose Multi-Image Dispersion and Semantic Reconstruction (MIDAS), a multimodal jailbreak framework that decomposes harmful semantics into risk-bearing subunits, disperses them across multiple visual clues, and leverages cross-image reasoning to gradually reconstruct the malicious intent, thereby bypassing existing safety mechanisms. The proposed MIDAS enforces longer and more structured multi-image chained reasoning, substantially increases the model's reliance on visual cues while delaying the exposure of malicious semantics and significantly reducing the model's security attention, thereby improving the performance of jailbreak against advanced MLLMs. Extensive experiments across different datasets and MLLMs demonstrate that the proposed MIDAS outperforms state-of-the-art jailbreak attacks for MLLMs and achieves an average attack success rate of 81.46% across 4 closed-source MLLMs. Our code is available at this link.

Abstract

Adversarial Attacks on Unified Multimodal Models

Explore similar work

MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMs

DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputs

Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Models