ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Models

Wenzheng Jiang; Xuankun Rong; Yuanzhao Zhai; Dawei Feng; Huaimin Wang

cs.AIAug 21, 2026

ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Models

Authors: Wenzheng Jiang, Xuankun Rong, Yuanzhao Zhai, Dawei Feng, Huaimin Wang

Organizations: College of Computer Science and Technology, National University of Defense Technology · State Key Laboratory of Complex & Critical Software Environment · School of Computer Science, Wuhan University

Abstract

While multimodal large language models (MLLMs) extend model capabilities beyond text, they also make safety alignment increasingly challenging. Multimodal safety alignment methods must address cross-modal jailbreaks, safety-awareness failures, and over-sensitive refusals. However, existing methods often rely on retraining or internal-state inspection, limiting their applicability to deployed closed-source MLLMs and motivating test-time safety alignment. We analyze this setting and identify two key obstacles, utility dominance and reasoning inertia, which cause models to overlook latent risks or follow malicious reasoning trajectories. Guided by these insights, we propose ReFrame, a training-free multimodal input reframing framework where two agents share a lightweight locally deployed MLLM: the evidence-generation agent constructs complementary risk and utility evidence, and the rewrite-and-routing agent converts it into a safe proxy prompt and image-routing decision before calling the downstream MLLM, without modifying it or accessing its internal information. Experiments across multiple MLLMs and benchmarks show that ReFrame improves jailbreak defense, safety awareness, and oversensitivity reduction while preserving multimodal utility.

Figures & tables

Figure 1: Limitations of existing multimodal safety alignment methods, highlighting the necessity of black-box approaches at test time.

Figure 2: Utility Dominance in test-time Multimodal Safety Alignment. Adding a reminder enhances safety awareness and redirects responses from unsafe to safe.

Figure 3: Overall pipeline of ReFrame. Given a multimodal input, the evidence-generation agent first generates a risk card and a utility card (Section 4.1 ). The rewrite-and-routing agent then synthesizes the cards into a safe proxy prompt and an image-routing decision before calling the downstream MLLM (Section 4.2 ).

	Jailbreak Defense						Safety Awareness		Oversensitivity		Average
Method	MM-Safety		MML-mirror		MML-base64		SIUO		MOSSBench		Overall
	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$	$\bar{\mathcal{S}}~\uparrow$	$\bar{\mathcal{U}}~\uparrow$
GPT-4.1	96.82	37.17	23.36	49.56	22.34	46.85	69.63	72.42	99.46	83.95	62.32	57.99
+ Self-Reminder [NMI’23]	99.47	54.76	56.33	58.17	53.93	58.60	84.28	79.70	99.80	78.44	78.76	65.93
+ ECSO [ECCV’24]	98.41	72.34	33.31	57.86	37.21	51.52	65.66	77.50	98.82	88.72	66.68	69.59
+ AMIA [EMNLP’25]	98.99	41.46	33.85	47.68	43.38	48.26	71.88	72.45	99.36	84.69	69.49	58.91

Table 1: Comparison of ReFrame with test-time baselines across three black-box MLLMs and five safety benchmarks, reporting safety score

\mathcal{S}

and utility score

\mathcal{U}

for jailbreak defense, safety awareness, and oversensitivity. Bold and underline indicate the best and second-best performance. The green values denote absolute gains over the corresponding original downstream model. Please see analysis in Section 5.2 .

	POPE	Math	Physics	Computer	Overall
Method	$\mathcal{A}~\uparrow$				$\bar{\mathcal{A}}~\uparrow$
GPT-4.1	35.80	35.00	33.33	45.00	37.28
+ Self-Reminder	36.40	33.33	36.67	46.67	38.27
+ ECSO	71.40	5.00	6.67	6.67	22.44
+ AMIA	62.20	28.33	35.00	36.67	40.55
+ EchoSafe	53.00	31.67	43.33	41.67	42.42

Table 2: Normal-task performance of ReFrame and baselines on GPT-4.1 across POPE and three MMMU subsets, reporting accuracy

\mathcal{A}

and

\bar{\mathcal{A}}

. Bold and underline indicate the best and second-best performance. Please see analysis in Section 5.2 .

Figure 4: Ablation on risk and utility cards in ReFrame, showing their contribution to safe proxy rewriting on GPT-4.1 across four benchmarks. Please see details in Section 5.3 .

	MML-mirror		SIUO		MOSSBench
Backbone	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$
GPT-4.1	23.36	49.56	69.63	72.42	99.46	83.95
w/ Qwen3VL-4B	99.67	97.22	88.30	92.00	99.83	96.04
w/ Qwen3VL-8B	99.80	98.72	92.12	94.76	99.93	95.90
w/ Qwen3.5-9B	98.54	93.48	86.08	91.75	100.00	94.95

Table 3: Scalability across different local MLLM for ReFrame with GPT-4.1 as the downstream model, highlighting its generalizability. See details in Section 5.4 .

	MML-mirror		SIUO		MOSSBench
Method	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$	$\mathcal{S}~\uparrow$	$\mathcal{U}~\uparrow$
Qwen3.6-35B	30.08	46.06	74.66	82.88	99.08	92.68
+ ReFrame	99.97 $\uparrow$ 69.89	96.81 $\uparrow$ 50.75	83.80 $\uparrow$ 9.14	89.76 $\uparrow$ 6.88	100.00 $\uparrow$ 0.92	93.80 $\uparrow$ 1.12
Qwen3VL-235B	14.27	39.42	73.01	82.61	98.69	93.29
+ ReFrame	99.91 $\uparrow$ 85.64	98.46 $\uparrow$ 59.04	80.12 $\uparrow$ 7.11	89.28 $\uparrow$ 6.67	99.93 $\uparrow$ 1.24	94.33 $\uparrow$ 1.04

Table 4: Scalability on open-source models demonstrates ReFrame is not limited to closed-source models. Please see details in Section 5.4 .

Figure 5: Case study of prompt rewriting in ReFrame across three safety benchmarks and two normal benchmarks, illustrating how multimodal inputs are compiled into downstream-model inputs. See details in Section 5.5 .

Figure 6: Runtime-performance trade-off of test-time methods , comparing runtime, safety, and normal-task performance. Please see details in Section 5.6 .

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Reasoning Inertia in test-time Multimodal Safety Alignment. By elaborately designing and obfuscating prompts to guide the model to reason step by step and follow instructions for jailbreaking, MML (bottom) Wang et al. (2025d) is an enhanced version of Figstep (top) Gong et al. (2025) .

Method	Safety ( $\uparrow$ )	Normal ( $\uparrow$ )	Runtime ( $\downarrow$ )
Self-Reminder	72.35	38.27	0.00
ECSO	68.14	22.44	1.50
AMIA	64.20	40.55	0.57
EchoSafe	84.89	42.42	1.85
ReFrame (Ours)	96.20	45.95	0.84

Appendix

Table 5: Comparison of safety effectiveness, normal-task effectiveness, and per-sample runtime of different safety methods. Safety effectiveness is computed as the average of

\bar{\mathcal{S}}

and

\bar{\mathcal{U}}

. Normal-task effectiveness is reported as

\bar{\mathcal{A}}

. Runtime is measured in seconds per sample on a single A100 GPU.

MLLM Safety

See all 28 papers

Harnessing Textual Refusal Directions for Multimodal Safety
Jun 30, 2026Moreno D'Incà, Nicu Sebe, Massimiliano ManciniLanguage Model Safety Evaluation Multimodal Large Language Models
University of Trento
Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correction
May 18, 2026Jiahe Guo, Xiangran Guo, Jiaxuan Chen +6Multimodal Large Language Models LLM Safety
Harbin Institute of Technology · Huawei Technologies Co., Ltd
COMIC: Reference-Aware Safety Gating for Multimodal Large Language Models
Aug 18, 2026Md Abdullahil Oaphy, Anhao Xiang, Zongxing Xie +3MLLM Safety LLM Safety
Kennesaw State University, Kennesaw, GA, USA

Jun 30, 2026J/K move · Enter open · S save

Harnessing Textual Refusal Directions for Multimodal Safety

Moreno D'Incà, Nicu Sebe, Massimiliano Mancini

University of Trento

Language Model Safety Evaluation Multimodal Large Language Models Multimodal Multi-Turn Safety Alignment Activation Steering MLLM Safety

Open paper AskPDF arXiv

Safety in Multimodal Large Language Models (MLLMs) depends on unsafe multimodal data for post-training alignment or activation steering. However, such data is hard to collect, must be gathered anew for each modality, and its collection and distribution raise ethical and legal concerns. In this work, we relax this constraint and investigate whether textual refusal directions, extracted directly from the LLM backbone, generalize across modalities (i.e., image, video). Preliminary findings confirm this ability, though effectiveness is conditioned by layer selection, steering strength, and cross-modal alignment, with the latter causing safe multimodal inputs to be spuriously steered toward refusal. Building on this, we introduce Modality-Agnostic Refusal Steering (MARS), a lightweight training-free approach that injects multimodal safety without the need for multimodal safety data. MARS corrects modality misalignment via activation centering, adaptively scales steering strength within a geometrically defined trust region, and selects the optimal intervention layer, operating at the first generated token. Evaluated on five SOTA MLLMs across safety, utility, and video jailbreak, MARS achieves consistent safety gains while preserving utility up to over-refusal. MARS reveals that safety-relevant structure is shared across modalities and that textual refusal directions are a powerful and underexplored foundation for multimodal alignment.

Abstract

MLLM Safety

Explore similar work

Harnessing Textual Refusal Directions for Multimodal Safety

Safety Geometry Collapse in Multimodal LLMs and Adaptive Drift Correction

COMIC: Reference-Aware Safety Gating for Multimodal Large Language Models