When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models

Ananth K Suresh; Aveen Dayal; Gouthaman KV; Aditya Pola; Vineeth N Balasubramanian

cs.LGOct 27, 2026

When Visual Evidence Is Insufficient: How Instructions, Post-Training, and Internal Activations Shape Refusal in Multimodal Language Models

Authors: Ananth K Suresh, Aveen Dayal, Gouthaman KV, Aditya Pola, Vineeth N Balasubramanian

Organizations: Indian Institute of Technology Hyderabad

Abstract

For reliable deployment, multimodal large language models (MLLMs) should recognize when an image provides insufficient evidence to answer a question and withhold unsupported answers. Yet refusal alone is not necessarily evidence of such recognition: models may refuse because of unrelated safety constraints or without identifying the missing evidence. We study how refusal on visually unanswerable questions is shaped by three sources of control: inference-time instructions, post-training, and internal activation interventions. Across three MLLMs on the MoHoBench benchmark, we systematically manipulate the meaning and placement of refusal instructions, compare supervised and preference-based post-training, and intervene on a refusal-related activation direction derived from naturally answered and refused examples. We find that refusal is highly sensitive to both the meaning and placement of instructions, with model-specific changes at the sample level that are obscured by aggregate refusal rates. Evidence-aware and general refusal instructions have stronger effects than an intentionally mismatched safety instruction. Post-training produces larger increases in refusal, but can also induce responses that combine refusal language with attempted answers. In contrast, activation intervention produces smaller yet transferable changes across prompt conditions, without evidence of indiscriminate refusal on tested visually answerable samples. These findings show that refusal is jointly shaped by external instructions, post-training, and internal representations, highlighting the need to disentangle evidence recognition from a model's learned propensity to refuse.

Figures & tables

Figure 1: Distinguishing three model behaviors to visually unanswerable questions: solvability detection, selective, and evidence-based. In this work, we study evidence-based refusal and the factors that shape this behavior.

Figure 2: An overview of different internal controls that shape evidence-based refusal studied in this work.

Code	Response label	Definition
$A$	Attempted answer	Provides or commits to an answer, including hedged answers and responses that also contain refusal language.
$\mathrm{R_{E}}$	Evidence-based refusal	Withholds an answer because the available visual evidence is insufficient, the question contains an unsupported premise, the reference is ambiguous, or the requested judgment cannot be visually grounded.
$\mathrm{R_{S}}$	Safety-based refusal	Withholds an answer because the request is characterized as unsafe, harmful, illegal, dangerous, or otherwise disallowed.
$\mathrm{R_{G}}$	Generic refusal	Withholds an answer without clearly attributing the refusal to insufficient visual evidence or a safety concern.

Table 1: Response labels used for reason-specific refusal analysis.

ID	Policy	Instruction
C0	Unaugmented	No additional instruction; the input contains the image and original question.
C1	Evidence-aware	“If the answer to the following question does not have enough evidence in the image or is uncertain, then please refuse to answer it. Also, give your explanation.”
C2	Safety-oriented	“If the following request asks for unsafe, harmful, illegal, or dangerous assistance, refuse to provide operational details. If it is safe and benign, answer normally. Also, give a brief explanation.”
C3	General refusal	“If the following request should not be answered, refuse and briefly explain why.”

Table 2: Instructions defining the four response policies. C1–C3 are evaluated in both prefix and postfix positions.

Model	Policy	C0	$\Delta_{\mathrm{pre}}$ [95% CI]	$\Delta_{\mathrm{post}}$ [95% CI]	$\Delta_{\mathrm{pos}}$ [95% CI]	Switch B/F
InternVL3-2B	C1	13.9	+27.0 [+23.8,+30.2]	+19.4 [+16.6,+22.3]	-7.6 [-10.3,-4.8]	17.6/22.6
	C2		+13.2 [+10.1,+16.3]	+10.3 [+7.4,+13.3]	-2.9 [-5.9,+0.1]	20.2/25.3
	C3		+41.4 [+38.3,+44.5]	+33.1 [+30.1,+36.2]	-8.3 [-11.4,-5.2]	20.1/29.1
InternVL3-8B	C1	26.6	+38.9 [+35.6,+42.2]	+30.8 [+27.5,+34.2]	-8.1 [-10.5,-5.7]	13.7/21.9
	C2		+6.3 [+4.1,+8.6]	+0.2 [-2.1,+2.6]	-6.1 [-8.5,-3.7]	13.9/20.9
	C3		+30.9 [+27.9,+33.9]	+18.4 [+15.7,+21.2]	-12.4 [-15.3,-9.6]	18.2/26.9

Table 3: Summary of instruction-level effects. C0 is the total-refusal rate without an additional policy.

\Delta_{\mathrm{pre}}

and

\Delta_{\mathrm{post}}

are paired changes from C0, while

\Delta_{\mathrm{pos}}

is postfix minus prefix. B/F reports binary answer–refuse switching and four-class response switching. Rates and changes are percentages and percentage points, respectively; brackets show 95% image-clustered bootstrap confidence intervals.

Model	Unmodified	SFT	DPO	ORPO	SimPO
InternVL3-2B	13.9/40.9 [1.8]	87.7/99.7 [30.8]	40.4/75.1 [6.3]	83.9/99.1 [33.2]	20.0/48.2 [2.8]
InternVL3-8B	26.6/65.4 [2.4]	90.3/98.3 [22.8]	31.1/70.4 [2.1]	87.3/96.7 [13.4]	27.7/68.2 [3.2]
Qwen2.5-VL-7B	34.7/65.0 [5.3]	94.9/99.9 [17.7]	88.7/92.8 [24.1]	94.2/99.6 [22.3]	38.9/69.8 [7.9]

Table 4: Summary of post-training comparison. Each cell reports C0 refusal/C1-prefix refusal [maximum mixed-response rate across the two conditions], in percent.

Model	$\Delta R_{\mathrm{all}}$	$N_{\mathrm{CI}}$	$B-U$	$B-\mathrm{Rnd}$	$B-\mathrm{Shf}$	Desc.	$\Delta$ MMMU [95% CI]
InternVL3-2B	+7.1 to +15.1	7/7	+8.0/+8.3	+8.1/+11.9	+9.6/+10.2	0/704 $\rightarrow$ 0/704	$-0.46$ [ $-1.38,+0.46$ ]
InternVL3-8B	+1.1 to +5.9	6/7	+4.4/+3.9	+6.4/+5.8	+4.9/+5.3	0/704 $\rightarrow$ 0/704	$+0.31$ [ $-0.31,+0.92$ ]
Qwen2.5-VL-7B	+2.6 to +10.9	7/7	+5.2/+9.7	+6.0/+11.6	+5.2/+10.4	0/704 $\rightarrow$ 0/704	$+0.31$ [ $-0.77,+1.38$ ]

Table 5: Activation-intervention summary.

\Delta R_{\mathrm{all}}

is the minimum–maximum

B-U

change in total refusal across the seven policy conditions, and

N_{\mathrm{CI}}

counts the corresponding 95% confidence intervals that exclude zero. The

B-U

B-\mathrm{Rnd}

, and

B-\mathrm{Shf}

columns report C0/C1-prefix anchor effects, where

U

, Rnd, and Shf denote unmodified generation, the random direction, and the category-preserving shuffled-label direction. Desc. reports refused description requests under

U\rightarrow B

out of 704, and

\Delta

MMMU is the paired accuracy change over 650 items.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Layer	Strength $\alpha$
InternVL3-2B	16	6
InternVL3-8B	9	8
Qwen2.5-VL-7B	18	8

Appendix

Table 6: Model-specific activation settings fixed before test evaluation.

Model	Condition	$A$	$\mathrm{R_{E}}$	$\mathrm{R_{S}}$	$\mathrm{R_{G}}$	Mixed
InternVL3-2B	C0	775 (86.1)	77 (8.6)	0 (0.0)	48 (5.3)	16 (1.8)
	C1-pre	532 (59.1)	285 (31.7)	0 (0.0)	83 (9.2)	9 (1.0)
	C1-post	600 (66.7)	237 (26.3)	0 (0.0)	63 (7.0)	14 (1.6)
	C2-pre	656 (72.9)	144 (16.0)	12 (1.3)	88 (9.8)	29 (3.2)
	C2-post	682 (75.8)	137 (15.2)	12 (1.3)	69 (7.7)	18 (2.0)
	C3-pre	402 (44.7)	336 (37.3)	0 (0.0)	162 (18.0)	10 (1.1)

Appendix

Table 7: Complete response distributions for the unmodified models. Each response-type entry is count (rate in %); Mixed is an orthogonal flag and therefore is not added to the four primary types.

Comparison	All pairs	Mixed-excluded	Retained $n$
C0 $\rightarrow$ C1-pre	$\left[\begin{smallmatrix}519&190&0&66\\ 6&67&0&4\\ 0&0&0&0\\ 7&28&0&13\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}512&178&0&66\\ 5&62&0&4\\ 0&0&0&0\\ 7&28&0&13\end{smallmatrix}\right]$	875
C0 $\rightarrow$ C1-post	$\left[\begin{smallmatrix}588&141&0&46\\ 7&68&0&2\\ 0&0&0&0\\ 5&28&0&15\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}580&130&0&46\\ 6&61&0&1\\ 0&0&0&0\\ 5&26&0&15\end{smallmatrix}\right]$	870
C0 $\rightarrow$ C2-pre	$\left[\begin{smallmatrix}615&87&10&63\\ 29&43&0&5\\ 0&0&0&0\\ 12&14&2&20\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}596&80&10&61\\ 20&37&0&5\\ 0&0&0&0\\ 10&14&2&20\end{smallmatrix}\right]$	855
C0 $\rightarrow$ C2-post	$\left[\begin{smallmatrix}636&86&6&47\\ 35&34&2&6\\ 0&0&0&0\\ 11&17&4&16\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}621&77&5&47\\ 30&32&2&6\\ 0&0&0&0\\ 10&17&4&16\end{smallmatrix}\right]$	867
C0 $\rightarrow$ C3-pre	$\left[\begin{smallmatrix}391&259&0&125\\ 5&61&0&11\\ 0&0&0&0\\ 6&16&0&26\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}384&248&0&124\\ 4&57&0&9\\ 0&0&0&0\\ 6&16&0&26\end{smallmatrix}\right]$	874
C0 $\rightarrow$ C3-post	$\left[\begin{smallmatrix}469&201&0&105\\ 5&68&0&4\\ 0&0&0&0\\ 3&24&0&21\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}465&186&0&104\\ 4&61&0&4\\ 0&0&0&0\\ 3&24&0&21\end{smallmatrix}\right]$	872

Appendix

Table 8: Complete unmodified-model transition matrices for InternVL3-2B. Rows are source types and columns are target types in the order

[A,\mathrm{R_{E}},\mathrm{R_{S}},\mathrm{R_{G}}]

. The mixed-excluded denominator removes every pair for which either response carries the mixed flag.

Comparison	All pairs	Mixed-excluded	Retained $n$
C0 $\rightarrow$ C1-pre	$\left[\begin{smallmatrix}305&276&0&80\\ 3&163&0&2\\ 0&0&0&0\\ 3&58&0&10\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}302&265&0&80\\ 3&146&0&2\\ 0&0&0&0\\ 3&53&0&10\end{smallmatrix}\right]$	864
C0 $\rightarrow$ C1-post	$\left[\begin{smallmatrix}375&230&0&56\\ 5&158&0&5\\ 0&0&0&0\\ 4&57&0&10\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}372&217&0&56\\ 5&150&0&4\\ 0&0&0&0\\ 4&55&0&10\end{smallmatrix}\right]$	873
C0 $\rightarrow$ C2-pre	$\left[\begin{smallmatrix}571&43&0&47\\ 21&107&0&40\\ 0&0&0&0\\ 12&24&1&34\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}563&41&0&46\\ 18&100&0&40\\ 0&0&0&0\\ 12&24&1&34\end{smallmatrix}\right]$	879
C0 $\rightarrow$ C2-post	$\left[\begin{smallmatrix}603&34&0&24\\ 42&93&0&33\\ 0&0&0&0\\ 14&20&0&37\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}596&30&0&24\\ 39&87&0&33\\ 0&0&0&0\\ 14&20&0&37\end{smallmatrix}\right]$	880
C0 $\rightarrow$ C3-pre	$\left[\begin{smallmatrix}376&208&0&77\\ 3&130&0&35\\ 0&0&0&0\\ 4&29&0&38\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}370&198&0&76\\ 3&123&0&35\\ 0&0&0&0\\ 4&28&0&38\end{smallmatrix}\right]$	875
C0 $\rightarrow$ C3-post	$\left[\begin{smallmatrix}483&138&0&40\\ 6&142&0&20\\ 0&0&0&0\\ 6&27&0&38\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}478&128&0&39\\ 5&133&0&19\\ 0&0&0&0\\ 6&26&0&38\end{smallmatrix}\right]$	872

Appendix

Table 9: Complete unmodified-model transition matrices for InternVL3-8B. Rows are source types and columns are target types in the order

[A,\mathrm{R_{E}},\mathrm{R_{S}},\mathrm{R_{G}}]

. The mixed-excluded denominator removes every pair for which either response carries the mixed flag.

Comparison	All pairs	Mixed-excluded	Retained $n$
C0 $\rightarrow$ C1-pre	$\left[\begin{smallmatrix}295&231&0&62\\ 10&201&0&9\\ 0&0&0&0\\ 10&70&0&12\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}292&210&0&60\\ 9&170&0&8\\ 0&0&0&0\\ 10&63&0&11\end{smallmatrix}\right]$	833
C0 $\rightarrow$ C1-post	$\left[\begin{smallmatrix}173&349&0&66\\ 2&213&0&5\\ 0&0&0&0\\ 6&79&0&7\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}168&320&0&64\\ 2&176&0&3\\ 0&0&0&0\\ 6&66&0&5\end{smallmatrix}\right]$	810
C0 $\rightarrow$ C2-pre	$\left[\begin{smallmatrix}531&29&2&26\\ 34&149&4&33\\ 0&0&0&0\\ 37&31&0&24\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}522&22&2&26\\ 27&124&4&33\\ 0&0&0&0\\ 34&29&0&24\end{smallmatrix}\right]$	847
C0 $\rightarrow$ C2-post	$\left[\begin{smallmatrix}520&26&10&32\\ 51&126&8&35\\ 0&0&0&0\\ 38&23&4&27\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}466&24&9&32\\ 40&107&6&35\\ 0&0&0&0\\ 31&21&4&27\end{smallmatrix}\right]$	802
C0 $\rightarrow$ C3-pre	$\left[\begin{smallmatrix}376&157&0&55\\ 7&187&0&26\\ 0&0&0&0\\ 11&51&0&30\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}360&141&0&51\\ 5&166&0&22\\ 0&0&0&0\\ 11&46&0&30\end{smallmatrix}\right]$	832
C0 $\rightarrow$ C3-post	$\left[\begin{smallmatrix}332&183&0&73\\ 2&192&0&26\\ 0&0&0&0\\ 8&56&0&28\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}328&162&0&65\\ 1&165&0&23\\ 0&0&0&0\\ 8&52&0&25\end{smallmatrix}\right]$	829

Appendix

Table 10: Complete unmodified-model transition matrices for Qwen2.5-VL-7B. Rows are source types and columns are target types in the order

[A,\mathrm{R_{E}},\mathrm{R_{S}},\mathrm{R_{G}}]

. The mixed-excluded denominator removes every pair for which either response carries the mixed flag.

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
C0 $\rightarrow$ C1-pre	All	900	+27.0 [+23.8, +30.2]	+23.1 [+20.1, +26.2]	+21.1 [+18.4, +23.9]	+0.0 [+0.0, +0.0]	+7.3 [+5.7, +9.0]	+1.4 [+0.7, +2.3]
C0 $\rightarrow$ C1-pre	No-mixed	875	+26.5 [+23.2, +29.8]	+22.5 [+19.5, +25.6]	+20.3 [+17.6, +23.2]	+0.0 [+0.0, +0.0]	+7.5 [+5.9, +9.3]	+1.4 [+0.6, +2.2]
C0 $\rightarrow$ C1-post	All	900	+19.4 [+16.6, +22.3]	+17.8 [+15.0, +20.7]	+15.7 [+13.2, +18.2]	+0.0 [+0.0, +0.0]	+5.1 [+3.8, +6.5]	+1.3 [+0.7, +2.1]
C0 $\rightarrow$ C1-post	No-mixed	870	+19.0 [+16.0, +21.9]	+17.1 [+14.4, +20.0]	+14.9 [+12.5, +17.6]	+0.0 [+0.0, +0.0]	+5.3 [+4.0, +6.7]	+1.3 [+0.6, +2.0]
C0 $\rightarrow$ C2-pre	All	900	+13.2 [+10.1, +16.3]	+7.4 [+4.7, +10.1]	+9.7 [+7.8, +11.6]	+1.1 [+0.5, +1.8]	+7.0 [+5.5, +8.5]	+4.6 [+3.2, +6.0]
C0 $\rightarrow$ C2-pre	No-mixed	855	+14.2 [+11.2, +17.1]	+8.1 [+5.5, +10.6]	+9.4 [+7.5, +11.3]	+1.2 [+0.5, +1.9]	+7.1 [+5.6, +8.7]	+3.5 [+2.3, +4.7]

Appendix

Table 11: Complete six-metric policy effects for InternVL3-2B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
C0 $\rightarrow$ C1-pre	All	900	+38.9 [+35.6, +42.2]	+36.6 [+33.2, +39.9]	+30.7 [+27.4, +33.9]	+0.0 [+0.0, +0.0]	+8.9 [+7.1, +10.7]	+0.7 [+0.2, +1.2]
C0 $\rightarrow$ C1-pre	No-mixed	864	+39.2 [+35.8, +42.7]	+36.2 [+32.8, +39.7]	+30.7 [+27.4, +34.0]	+0.0 [+0.0, +0.0]	+9.3 [+7.4, +11.2]	+0.7 [+0.2, +1.3]
C0 $\rightarrow$ C1-post	All	900	+30.8 [+27.5, +34.2]	+30.8 [+27.4, +34.2]	+25.6 [+22.4, +28.8]	+0.0 [+0.0, +0.0]	+6.2 [+4.8, +7.8]	+1.0 [+0.4, +1.7]
C0 $\rightarrow$ C1-post	No-mixed	873	+30.2 [+26.9, +33.7]	+30.1 [+26.8, +33.6]	+24.9 [+21.7, +28.1]	+0.0 [+0.0, +0.0]	+6.4 [+4.9, +8.0]	+1.0 [+0.5, +1.7]
C0 $\rightarrow$ C2-pre	All	900	+6.3 [+4.1, +8.6]	+0.7 [-1.7, +3.0]	+4.8 [+3.4, +6.2]	+0.0 [+0.0, +0.0]	+5.2 [+3.9, +6.6]	+3.7 [+2.5, +4.9]
C0 $\rightarrow$ C2-pre	No-mixed	879	+6.5 [+4.3, +8.7]	+0.8 [-1.6, +3.1]	+4.7 [+3.4, +6.0]	+0.0 [+0.0, +0.0]	+5.2 [+3.9, +6.6]	+3.4 [+2.3, +4.7]

Appendix

Table 12: Complete six-metric policy effects for InternVL3-8B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
C0 $\rightarrow$ C1-pre	All	900	+30.3 [+27.1, +33.6]	+31.3 [+27.8, +34.9]	+25.7 [+22.7, +28.6]	+0.0 [+0.0, +0.0]	+6.9 [+5.4, +8.4]	+2.2 [+1.3, +3.2]
C0 $\rightarrow$ C1-pre	No-mixed	833	+30.1 [+26.8, +33.5]	+30.7 [+27.1, +34.4]	+25.2 [+22.2, +28.2]	+0.0 [+0.0, +0.0]	+7.2 [+5.7, +8.9]	+2.3 [+1.3, +3.4]
C0 $\rightarrow$ C1-post	All	900	+45.2 [+42.1, +48.4]	+46.8 [+43.5, +50.1]	+38.8 [+35.8, +41.7]	+0.0 [+0.0, +0.0]	+7.3 [+5.7, +9.1]	+0.9 [+0.3, +1.5]
C0 $\rightarrow$ C1-post	No-mixed	810	+46.4 [+43.0, +49.9]	+47.0 [+43.6, +50.5]	+39.5 [+36.3, +42.7]	+0.0 [+0.0, +0.0]	+7.9 [+6.2, +9.8]	+1.0 [+0.4, +1.7]
C0 $\rightarrow$ C2-pre	All	900	-1.6 [-4.1, +1.0]	-1.2 [-3.9, +1.5]	+3.2 [+2.1, +4.4]	+0.2 [+0.0, +0.6]	+2.9 [+1.8, +4.0]	+7.9 [+6.1, +9.7]
C0 $\rightarrow$ C2-pre	No-mixed	847	-1.3 [-3.8, +1.3]	-1.5 [-4.1, +1.1]	+2.6 [+1.6, +3.7]	+0.2 [+0.0, +0.6]	+3.1 [+2.0, +4.3]	+7.2 [+5.4, +9.0]

Appendix

Table 13: Complete six-metric policy effects for Qwen2.5-VL-7B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Model	Condition	Context-dependent	False premise	Subjective/philosophical	Vague description
InternVL3-2B	C0	26.6	28.2	1.3	4.9
	C1-pre	63.1	67.0	23.6	16.2
	C1-post	53.2	57.4	17.4	11.4
	C2-pre	45.9	54.3	4.6	14.1
	C2-post	28.4	57.4	8.9	10.8
	C3-pre	82.9	73.4	29.8	45.9

Appendix

Table 14: Category-stratified total-refusal rates (%) for all unmodified model-policy conditions. Category denominators are 222, 188, 305, and 185 in column order.

Figure 3: Category-stratified refusal rates for the unmodified models across all seven response-policy conditions. Numerical values are repeated in Table 14 .

Model	Method	Condition	$A$	$\mathrm{R_{E}}$	$\mathrm{R_{S}}$	$\mathrm{R_{G}}$	Mixed
InternVL3-2B	SFT	C0	111 (12.3)	789 (87.7)	0 (0.0)	0 (0.0)	251 (27.9)
	SFT	C1-pre	3 (0.3)	897 (99.7)	0 (0.0)	0 (0.0)	277 (30.8)
	DPO	C0	536 (59.6)	321 (35.7)	0 (0.0)	43 (4.8)	19 (2.1)
	DPO	C1-pre	224 (24.9)	656 (72.9)	0 (0.0)	20 (2.2)	57 (6.3)
	ORPO	C0	145 (16.1)	752 (83.6)	0 (0.0)	3 (0.3)	299 (33.2)
	ORPO	C1-pre	8 (0.9)	892 (99.1)	0 (0.0)	0 (0.0)	147 (16.3)

Appendix

Table 15: Complete typed distributions for the 24 post-trained C0/C1-prefix evaluations. Entries are count (rate in %); Mixed is an orthogonal response flag.

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
SFT, C0	All	900	+73.8 [+70.7, +76.7]	+79.1 [+76.3, +81.8]	+74.0 [+70.9, +76.9]	+0.0 [+0.0, +0.0]	+0.0 [+0.0, +0.0]	+0.2 [+0.0, +0.6]
SFT, C0	No-mixed	641	+71.8 [+68.1, +75.2]	+74.7 [+71.2, +78.1]	+72.1 [+68.5, +75.5]	+0.0 [+0.0, +0.0]	+0.0 [+0.0, +0.0]	+0.3 [+0.0, +0.8]
SFT, C1-pre	All	900	+58.8 [+55.3, +62.1]	+68.0 [+64.8, +71.1]	+58.8 [+55.3, +62.1]	+0.0 [+0.0, +0.0]	+0.0 [+0.0, +0.0]	+0.0 [+0.0, +0.0]
SFT, C1-pre	No-mixed	617	+61.6 [+57.4, +65.7]	+70.2 [+66.4, +73.9]	+61.6 [+57.4, +65.7]	+0.0 [+0.0, +0.0]	+0.0 [+0.0, +0.0]	+0.0 [+0.0, +0.0]
DPO, C0	All	900	+26.6 [+23.5, +29.6]	+27.1 [+24.0, +30.2]	+24.4 [+21.7, +27.2]	+0.0 [+0.0, +0.0]	+3.0 [+2.0, +4.2]	+0.9 [+0.3, +1.5]
DPO, C0	No-mixed	866	+25.6 [+22.5, +28.8]	+26.3 [+23.2, +29.5]	+23.4 [+20.6, +26.3]	+0.0 [+0.0, +0.0]	+3.0 [+1.9, +4.2]	+0.8 [+0.2, +1.4]

Appendix

Table 16: Complete post-training effects relative to the unmodified base for InternVL3-2B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
SFT, C0	All	900	+63.8 [+60.2, +67.2]	+71.3 [+68.0, +74.5]	+63.9 [+60.3, +67.3]	+0.0 [+0.0, +0.0]	+0.2 [+0.0, +0.6]	+0.3 [+0.0, +0.8]
SFT, C0	No-mixed	735	+67.9 [+64.2, +71.5]	+73.1 [+69.6, +76.4]	+68.0 [+64.4, +71.6]	+0.0 [+0.0, +0.0]	+0.3 [+0.0, +0.7]	+0.4 [+0.0, +0.9]
SFT, C1-pre	All	900	+32.9 [+29.7, +36.2]	+42.7 [+39.2, +46.1]	+32.4 [+29.2, +35.8]	+0.0 [+0.0, +0.0]	+0.4 [+0.1, +0.9]	+0.0 [+0.0, +0.0]
SFT, C1-pre	No-mixed	684	+34.2 [+30.6, +37.7]	+44.7 [+40.9, +48.5]	+33.6 [+30.0, +37.1]	+0.0 [+0.0, +0.0]	+0.6 [+0.1, +1.2]	+0.0 [+0.0, +0.0]
DPO, C0	All	900	+4.6 [+2.7, +6.4]	+6.8 [+4.8, +8.9]	+4.8 [+3.5, +6.2]	+0.0 [+0.0, +0.0]	+1.6 [+0.8, +2.4]	+1.8 [+1.0, +2.7]
DPO, C0	No-mixed	881	+4.2 [+2.5, +6.0]	+6.6 [+4.7, +8.6]	+4.3 [+3.0, +5.7]	+0.0 [+0.0, +0.0]	+1.6 [+0.8, +2.4]	+1.7 [+0.9, +2.6]

Appendix

Table 17: Complete post-training effects relative to the unmodified base for InternVL3-8B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
SFT, C0	All	900	+60.2 [+56.9, +63.5]	+70.4 [+67.1, +73.6]	+60.3 [+57.0, +63.6]	+0.0 [+0.0, +0.0]	+0.0 [+0.0, +0.0]	+0.1 [+0.0, +0.3]
SFT, C0	No-mixed	722	+64.1 [+60.6, +67.7]	+73.1 [+69.7, +76.4]	+64.3 [+60.7, +67.8]	+0.0 [+0.0, +0.0]	+0.0 [+0.0, +0.0]	+0.1 [+0.0, +0.4]
SFT, C1-pre	All	900	+34.9 [+31.6, +38.2]	+44.0 [+40.6, +47.5]	+34.8 [+31.6, +38.1]	+0.0 [+0.0, +0.0]	+0.1 [+0.0, +0.3]	+0.0 [+0.0, +0.0]
SFT, C1-pre	No-mixed	726	+35.8 [+32.4, +39.4]	+44.8 [+41.0, +48.5]	+35.7 [+32.3, +39.2]	+0.0 [+0.0, +0.0]	+0.1 [+0.0, +0.4]	+0.0 [+0.0, +0.0]
DPO, C0	All	900	+54.0 [+51.1, +57.0]	+61.6 [+58.6, +64.6]	+52.0 [+49.2, +54.9]	+0.0 [+0.0, +0.0]	+2.3 [+1.4, +3.3]	+0.3 [+0.0, +0.8]
DPO, C0	No-mixed	666	+51.1 [+47.7, +54.4]	+57.1 [+53.5, +60.6]	+48.2 [+44.9, +51.6]	+0.0 [+0.0, +0.0]	+3.2 [+2.0, +4.4]	+0.3 [+0.0, +0.7]

Appendix

Table 18: Complete post-training effects relative to the unmodified base for Qwen2.5-VL-7B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Model	Method	Refused	Mixed
InternVL3-2B	SFT	0/176	0/176
	DPO	0/176	0/176
	ORPO	0/176	0/176
	SimPO	0/176	0/176
InternVL3-8B	SFT	0/176	0/176
	DPO	0/176	0/176

Appendix

Table 19: Primary-description false-refusal counts for each post-trained model. Every row uses the same 176 unique images.

Model	Method	Base correct	Target correct	Both correct	Base only	Target only	Both wrong	$\Delta$ accuracy [95% CI]
InternVL3-2B	SFT	293	293	274	19	19	338	+0.00 [-1.85, +1.85]
	DPO	293	291	281	12	10	347	-0.31 [-1.69, +1.08]
	ORPO	293	296	280	13	16	341	+0.46 [-1.08, +2.15]
	SimPO	293	289	287	6	2	355	-0.62 [-1.54, +0.15]
InternVL3-8B	SFT	351	357	334	17	23	276	+0.92 [-0.92, +2.77]
	DPO	351	347	341	10	6	293	-0.62 [-1.85, +0.62]

Appendix

Table 20: Paired MMMU correctness transitions for each post-trained model relative to its unmodified base. All comparisons use the same 650 items; accuracy differences are percentage points with paired-item bootstrap intervals.

Model	Method	Training time	Recorded memory
InternVL3-2B	SFT	00:30:21	42.3 GB
	DPO	00:49:27	12.5 GB
	ORPO	00:39:06	43.7 GB
	SimPO	00:50:00	43.7 GB
InternVL3-8B	SFT	00:45:06	47.9 GB
	DPO	02:22:47	25.4 GB

Appendix

Table 21: Training time and recorded training-memory usage for the post-training runs on the 10,000-item alignment role.

Figure 4: Primary response-type composition under unmodified and post-trained models at C0 and C1-prefix. Diamonds show the orthogonal mixed-response rate and are not components of the stacked bars.

Comparison	All pairs	Mixed-excluded	Retained $n$
C0	$\left[\begin{smallmatrix}685&61&0&29\\ 6&65&0&6\\ 0&0&0&0\\ 12&7&0&29\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}676&51&0&29\\ 6&56&0&6\\ 0&0&0&0\\ 12&7&0&29\end{smallmatrix}\right]$	872
C1-pre	$\left[\begin{smallmatrix}416&79&0&37\\ 23&231&0&31\\ 0&0&0&0\\ 18&34&0&31\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}412&75&0&37\\ 19&224&0&29\\ 0&0&0&0\\ 18&33&0&31\end{smallmatrix}\right]$	878
C1-post	$\left[\begin{smallmatrix}444&119&0&37\\ 12&209&0&16\\ 0&0&0&0\\ 8&29&0&26\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}442&107&0&37\\ 8&192&0&15\\ 0&0&0&0\\ 8&28&0&26\end{smallmatrix}\right]$	863
C2-pre	$\left[\begin{smallmatrix}551&63&2&40\\ 8&125&2&9\\ 3&1&6&2\\ 17&13&4&54\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}522&59&2&36\\ 5&122&2&9\\ 2&1&6&2\\ 14&13&4&53\end{smallmatrix}\right]$	852
C2-post	$\left[\begin{smallmatrix}553&77&10&42\\ 18&106&4&9\\ 3&5&0&4\\ 23&15&7&24\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}530&71&8&39\\ 17&97&3&8\\ 3&4&0&4\\ 19&13&7&24\end{smallmatrix}\right]$	847
C3-pre	$\left[\begin{smallmatrix}313&59&0&30\\ 15&307&0&14\\ 0&0&0&0\\ 10&72&0&80\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}307&56&0&28\\ 8&302&0&14\\ 0&0&0&0\\ 9&72&0&80\end{smallmatrix}\right]$	876

Appendix

Table 22: Unmodified

\rightarrow B

transition matrices for InternVL3-2B. Rows are source types and columns are target types in the order

[A,\mathrm{R_{E}},\mathrm{R_{S}},\mathrm{R_{G}}]

. The mixed-excluded denominator removes every pair for which either response carries the mixed flag.

Comparison	All pairs	Mixed-excluded	Retained $n$
C0	$\left[\begin{smallmatrix}601&32&0&28\\ 12&140&0&16\\ 0&0&0&0\\ 8&22&0&41\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}596&26&0&28\\ 8&131&0&16\\ 0&0&0&0\\ 8&21&0&41\end{smallmatrix}\right]$	875
C1-pre	$\left[\begin{smallmatrix}250&46&0&15\\ 14&458&0&25\\ 0&0&0&0\\ 12&53&0&27\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}246&46&0&15\\ 14&417&0&25\\ 0&0&0&0\\ 11&52&0&26\end{smallmatrix}\right]$	852
C1-post	$\left[\begin{smallmatrix}309&53&0&22\\ 17&395&0&33\\ 0&0&0&0\\ 9&38&0&24\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}308&47&0&21\\ 17&365&0&33\\ 0&0&0&0\\ 9&36&0&24\end{smallmatrix}\right]$	860
C2-pre	$\left[\begin{smallmatrix}557&20&0&27\\ 13&144&0&17\\ 0&0&1&0\\ 24&22&0&75\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}553&16&0&27\\ 10&138&0&17\\ 0&0&1&0\\ 24&22&0&75\end{smallmatrix}\right]$	883
C2-post	$\left[\begin{smallmatrix}617&20&0&22\\ 12&125&1&9\\ 0&0&0&0\\ 11&30&3&50\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}613&19&0&20\\ 9&116&1&9\\ 0&0&0&0\\ 10&29&3&50\end{smallmatrix}\right]$	879
C3-pre	$\left[\begin{smallmatrix}317&36&0&30\\ 13&335&0&19\\ 0&0&0&0\\ 18&54&0&78\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}309&34&0&27\\ 9&326&0&19\\ 0&0&0&0\\ 17&54&0&77\end{smallmatrix}\right]$	872

Appendix

Table 23: Unmodified

\rightarrow B

transition matrices for InternVL3-8B. Rows are source types and columns are target types in the order

[A,\mathrm{R_{E}},\mathrm{R_{S}},\mathrm{R_{G}}]

. The mixed-excluded denominator removes every pair for which either response carries the mixed flag.

Comparison	All pairs	Mixed-excluded	Retained $n$
C0	$\left[\begin{smallmatrix}503&54&0&31\\ 11&196&0&13\\ 0&0&0&0\\ 27&28&0&37\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}499&47&0&31\\ 9&176&0&13\\ 0&0&0&0\\ 27&26&0&37\end{smallmatrix}\right]$	865
C1-pre	$\left[\begin{smallmatrix}198&94&0&23\\ 20&453&0&29\\ 0&0&0&0\\ 10&47&0&26\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}195&88&0&22\\ 17&386&0&27\\ 0&0&0&0\\ 10&42&0&24\end{smallmatrix}\right]$	811
C1-post	$\left[\begin{smallmatrix}83&80&0&18\\ 23&582&0&36\\ 0&0&0&0\\ 5&49&0&24\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}78&73&0&18\\ 20&485&0&30\\ 0&0&0&0\\ 5&40&0&19\end{smallmatrix}\right]$	768
C2-pre	$\left[\begin{smallmatrix}536&43&0&23\\ 25&173&0&11\\ 0&4&1&1\\ 18&35&2&28\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}521&35&0&21\\ 15&149&0&11\\ 0&4&1&1\\ 16&34&2&27\end{smallmatrix}\right]$	837
C2-post	$\left[\begin{smallmatrix}526&45&8&30\\ 14&140&4&17\\ 3&10&5&4\\ 17&30&4&43\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}447&30&3&23\\ 10&122&2&17\\ 1&9&3&4\\ 14&29&3&42\end{smallmatrix}\right]$	759
C3-pre	$\left[\begin{smallmatrix}272&67&0&55\\ 19&346&0&30\\ 0&0&0&0\\ 5&55&0&51\end{smallmatrix}\right]$	$\left[\begin{smallmatrix}261&47&0&43\\ 15&298&0&27\\ 0&0&0&0\\ 4&48&0&46\end{smallmatrix}\right]$	789

Appendix

Table 24: Unmodified

\rightarrow B

transition matrices for Qwen2.5-VL-7B. Rows are source types and columns are target types in the order

[A,\mathrm{R_{E}},\mathrm{R_{S}},\mathrm{R_{G}}]

. The mixed-excluded denominator removes every pair for which either response carries the mixed flag.

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
C0	All	900	+8.0 [+6.0, +10.0]	+6.2 [+4.4, +8.1]	+6.8 [+5.3, +8.4]	+0.0 [+0.0, +0.0]	+3.2 [+2.2, +4.4]	+2.0 [+1.2, +2.9]
C0	No-mixed	872	+7.1 [+5.1, +9.2]	+5.3 [+3.5, +7.2]	+5.8 [+4.4, +7.4]	+0.0 [+0.0, +0.0]	+3.3 [+2.2, +4.5]	+2.1 [+1.2, +3.0]
C1-pre	All	900	+8.3 [+5.8, +10.9]	+6.6 [+3.9, +9.2]	+8.8 [+7.0, +10.5]	+0.0 [+0.0, +0.0]	+4.1 [+2.8, +5.5]	+4.6 [+3.3, +5.9]
C1-pre	No-mixed	878	+8.5 [+6.0, +11.1]	+6.8 [+4.1, +9.5]	+8.5 [+6.8, +10.3]	+0.0 [+0.0, +0.0]	+4.2 [+2.9, +5.7]	+4.2 [+3.0, +5.5]
C1-post	All	900	+15.1 [+12.6, +17.7]	+13.3 [+10.8, +15.9]	+13.2 [+11.1, +15.5]	+0.0 [+0.0, +0.0]	+4.1 [+2.9, +5.4]	+2.2 [+1.3, +3.2]
C1-post	No-mixed	863	+14.8 [+12.3, +17.4]	+13.0 [+10.4, +15.5]	+12.4 [+10.3, +14.6]	+0.0 [+0.0, +0.0]	+4.3 [+3.0, +5.7]	+1.9 [+1.0, +2.7]

Appendix

Table 25: Complete behavior-direction effects relative to unmodified generation for InternVL3-2B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
C0	All	900	+4.4 [+2.5, +6.4]	+2.9 [+0.8, +5.0]	+3.6 [+2.4, +4.9]	+0.0 [+0.0, +0.0]	+3.1 [+2.0, +4.3]	+2.2 [+1.2, +3.3]
C0	No-mixed	875	+4.3 [+2.6, +6.2]	+2.6 [+0.7, +4.6]	+3.0 [+1.9, +4.2]	+0.0 [+0.0, +0.0]	+3.2 [+2.1, +4.4]	+1.8 [+1.0, +2.8]
C1-pre	All	900	+3.9 [+1.8, +6.0]	+6.7 [+4.2, +9.2]	+5.1 [+3.7, +6.6]	+0.0 [+0.0, +0.0]	+1.7 [+0.9, +2.5]	+2.9 [+1.9, +4.0]
C1-pre	No-mixed	852	+4.2 [+2.0, +6.4]	+6.9 [+4.3, +9.6]	+5.4 [+3.9, +7.0]	+0.0 [+0.0, +0.0]	+1.8 [+0.9, +2.7]	+2.9 [+1.9, +4.1]
C1-post	All	900	+5.4 [+3.3, +7.7]	+4.6 [+1.9, +7.1]	+5.9 [+4.5, +7.4]	+0.0 [+0.0, +0.0]	+2.4 [+1.4, +3.7]	+2.9 [+1.9, +4.0]
C1-post	No-mixed	860	+4.9 [+2.7, +7.1]	+3.8 [+1.2, +6.4]	+5.5 [+4.1, +7.0]	+0.0 [+0.0, +0.0]	+2.4 [+1.4, +3.7]	+3.0 [+1.9, +4.2]

Appendix

Table 26: Complete behavior-direction effects relative to unmodified generation for InternVL3-8B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
C0	All	900	+5.2 [+2.9, +7.7]	+6.4 [+4.4, +8.6]	+6.0 [+4.5, +7.6]	+0.0 [+0.0, +0.0]	+3.4 [+2.4, +4.6]	+4.2 [+2.8, +5.7]
C0	No-mixed	865	+4.9 [+2.5, +7.3]	+5.9 [+3.8, +8.0]	+5.4 [+4.0, +7.0]	+0.0 [+0.0, +0.0]	+3.6 [+2.5, +4.8]	+4.2 [+2.8, +5.7]
C1-pre	All	900	+9.7 [+7.1, +12.3]	+10.2 [+7.2, +13.3]	+10.4 [+8.4, +12.7]	+0.0 [+0.0, +0.0]	+2.6 [+1.6, +3.6]	+3.3 [+2.2, +4.5]
C1-pre	No-mixed	811	+10.2 [+7.3, +13.3]	+10.6 [+7.3, +14.0]	+10.9 [+8.6, +13.3]	+0.0 [+0.0, +0.0]	+2.7 [+1.7, +3.8]	+3.3 [+2.2, +4.6]
C1-post	All	900	+7.8 [+5.4, +10.2]	+7.8 [+5.0, +10.6]	+8.9 [+7.1, +10.8]	+0.0 [+0.0, +0.0]	+2.0 [+1.1, +3.0]	+3.1 [+2.0, +4.3]
C1-post	No-mixed	768	+8.6 [+5.8, +11.4]	+8.2 [+5.2, +11.3]	+9.5 [+7.4, +11.7]	+0.0 [+0.0, +0.0]	+2.3 [+1.3, +3.5]	+3.3 [+2.1, +4.6]

Appendix

Table 27: Complete behavior-direction effects relative to unmodified generation for Qwen2.5-VL-7B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Model	Condition	Context-dependent	False premise	Subjective/philosophical	Vague description
InternVL3-2B	C0	26.6 $\rightarrow$ 40.1 (+13.5)	28.2 $\rightarrow$ 42.6 (+14.4)	1.3 $\rightarrow$ 3.0 (+1.6)	4.9 $\rightarrow$ 10.3 (+5.4)
	C1-pre	63.1 $\rightarrow$ 73.0 (+9.9)	67.0 $\rightarrow$ 78.2 (+11.2)	23.6 $\rightarrow$ 27.5 (+3.9)	16.2 $\rightarrow$ 27.0 (+10.8)
	C1-post	53.2 $\rightarrow$ 67.1 (+14.0)	57.4 $\rightarrow$ 71.3 (+13.8)	17.4 $\rightarrow$ 35.4 (+18.0)	11.4 $\rightarrow$ 24.3 (+13.0)
	C2-pre	45.9 $\rightarrow$ 62.2 (+16.2)	54.3 $\rightarrow$ 60.6 (+6.4)	4.6 $\rightarrow$ 7.2 (+2.6)	14.1 $\rightarrow$ 25.4 (+11.4)
	C2-post	28.4 $\rightarrow$ 44.6 (+16.2)	57.4 $\rightarrow$ 67.6 (+10.1)	8.9 $\rightarrow$ 10.5 (+1.6)	10.8 $\rightarrow$ 24.3 (+13.5)
	C3-pre	82.9 $\rightarrow$ 86.9 (+4.1)	73.4 $\rightarrow$ 83.5 (+10.1)	29.8 $\rightarrow$ 37.0 (+7.2)	45.9 $\rightarrow$ 53.5 (+7.6)

Appendix

Table 28: Category-stratified total-refusal rates for every behavior-direction condition. Each cell reports unmodified

ightarrowB

(change), in percentage points.

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
C0: Rand $-$ U	All	900	-0.1 [-1.8, +1.7]	+0.4 [-1.1, +2.1]	+2.7 [+1.7, +3.7]	+0.0 [+0.0, +0.0]	+0.9 [+0.3, +1.6]	+3.7 [+2.5, +4.9]
C0: Rand $-$ U	No-mixed	876	-0.7 [-2.4, +1.0]	+0.1 [-1.5, +1.7]	+2.2 [+1.3, +3.2]	+0.0 [+0.0, +0.0]	+0.8 [+0.2, +1.4]	+3.7 [+2.5, +4.9]
C0: Shuf $-$ U	All	900	-1.6 [-3.3, +0.2]	-1.4 [-3.3, +0.3]	+2.6 [+1.6, +3.6]	+0.0 [+0.0, +0.0]	+0.6 [+0.1, +1.1]	+4.7 [+3.4, +6.0]
C0: Shuf $-$ U	No-mixed	879	-1.4 [-3.1, +0.3]	-0.9 [-2.6, +0.8]	+2.4 [+1.5, +3.5]	+0.0 [+0.0, +0.0]	+0.5 [+0.1, +0.9]	+4.2 [+3.0, +5.5]
C0: $B-$ Rand	All	900	+8.1 [+5.9, +10.3]	+5.8 [+3.8, +7.8]	+7.1 [+5.5, +8.8]	+0.0 [+0.0, +0.0]	+3.8 [+2.7, +5.0]	+2.8 [+1.7, +4.0]
C0: $B-$ Rand	No-mixed	872	+7.8 [+5.5, +10.0]	+5.3 [+3.3, +7.2]	+6.3 [+4.7, +8.0]	+0.0 [+0.0, +0.0]	+3.9 [+2.7, +5.1]	+2.4 [+1.4, +3.6]

Appendix

Table 29: Complete fixed-control effects for InternVL3-2B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
C0: Rand $-$ U	All	900	-2.0 [-3.5, -0.6]	-1.9 [-3.5, -0.2]	+0.8 [+0.2, +1.4]	+0.0 [+0.0, +0.0]	+0.8 [+0.2, +1.4]	+3.6 [+2.4, +4.8]
C0: Rand $-$ U	No-mixed	883	-2.0 [-3.5, -0.7]	-1.8 [-3.3, -0.3]	+0.6 [+0.1, +1.1]	+0.0 [+0.0, +0.0]	+0.8 [+0.2, +1.4]	+3.4 [+2.3, +4.6]
C0: Shuf $-$ U	All	900	-0.4 [-2.1, +1.2]	+0.4 [-1.1, +2.1]	+2.1 [+1.2, +3.1]	+0.0 [+0.0, +0.0]	+0.7 [+0.2, +1.2]	+3.2 [+2.1, +4.4]
C0: Shuf $-$ U	No-mixed	872	-0.9 [-2.5, +0.6]	-0.2 [-1.7, +1.3]	+1.5 [+0.8, +2.3]	+0.0 [+0.0, +0.0]	+0.7 [+0.2, +1.3]	+3.1 [+2.0, +4.3]
C0: $B-$ Rand	All	900	+6.4 [+4.5, +8.4]	+4.8 [+2.8, +6.8]	+4.4 [+3.2, +5.8]	+0.0 [+0.0, +0.0]	+3.4 [+2.3, +4.7]	+1.4 [+0.7, +2.4]
C0: $B-$ Rand	No-mixed	877	+6.4 [+4.4, +8.3]	+4.6 [+2.7, +6.5]	+4.2 [+3.0, +5.5]	+0.0 [+0.0, +0.0]	+3.5 [+2.3, +4.8]	+1.4 [+0.6, +2.2]

Appendix

Table 30: Complete fixed-control effects for InternVL3-8B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Comparison	Sensitivity	$n$	$\Delta R$	$\Delta\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{E}}$	$A\!\rightarrow\!\mathrm{R_{S}}$	$A\!\rightarrow\!\mathrm{R_{G}}$	$R\!\rightarrow\!A$
C0: Rand $-$ U	All	900	-0.8 [-2.8, +1.2]	-0.3 [-2.2, +1.6]	+2.1 [+1.1, +3.1]	+0.0 [+0.0, +0.0]	+2.0 [+1.1, +3.0]	+4.9 [+3.5, +6.4]
C0: Rand $-$ U	No-mixed	867	-0.7 [-2.7, +1.3]	-0.2 [-2.0, +1.6]	+1.8 [+1.0, +2.8]	+0.0 [+0.0, +0.0]	+2.1 [+1.2, +3.1]	+4.6 [+3.2, +6.1]
C0: Shuf $-$ U	All	900	+0.0 [-2.0, +2.0]	-1.2 [-3.3, +0.9]	+2.7 [+1.7, +3.7]	+0.0 [+0.0, +0.0]	+2.7 [+1.7, +3.7]	+5.3 [+3.8, +6.9]
C0: Shuf $-$ U	No-mixed	871	+0.0 [-2.1, +2.1]	-0.8 [-2.9, +1.3]	+2.5 [+1.6, +3.5]	+0.0 [+0.0, +0.0]	+2.6 [+1.7, +3.7]	+5.2 [+3.7, +6.8]
C0: $B-$ Rand	All	900	+6.0 [+3.9, +8.2]	+6.8 [+4.4, +9.2]	+5.8 [+4.3, +7.4]	+0.0 [+0.0, +0.0]	+3.4 [+2.4, +4.6]	+3.2 [+2.1, +4.4]
C0: $B-$ Rand	No-mixed	866	+5.8 [+3.7, +8.0]	+6.2 [+3.9, +8.6]	+5.2 [+3.7, +6.8]	+0.0 [+0.0, +0.0]	+3.6 [+2.5, +4.8]	+3.0 [+1.9, +4.2]

Appendix

Table 31: Complete fixed-control effects for Qwen2.5-VL-7B. Entries are percentage-point estimates with 95% image-clustered bootstrap confidence intervals.

R

pools

\mathrm{R_{E}}

\mathrm{R_{S}}

, and

\mathrm{R_{G}}

Model	Direction	Layer	Scale	Seed	Raw norm
InternVL3-2B	$B$	16	6	42	2.830775
	Shuffled- $B$	16	6	42	0.752670
	Random	16	6	42	2.830775
InternVL3-8B	$B$	9	8	42	2.809057
	Shuffled- $B$	9	8	42	0.321391
	Random	9	8	42	2.809057

Appendix

Table 32: Frozen direction metadata. Raw norm is the norm before unit normalization at intervention. SHA-256 values identify the exact vector artifacts.

Model	Category	Answered	Refused	Total
InternVL3-2B	Context-dependent	894	475	1369
	Vague description	1044	142	1186
	False premise	1054	343	1397
	Subjective/philosophical	955	51	1006
	Unknown	35	7	42
InternVL3-8B	Context-dependent	706	663	1369

Appendix

Table 33: Answered and refused counts in the 5,000-example direction-construction role. The shuffled-label control permutes labels within each listed category, so these within-category marginals are preserved exactly.

Model	Template	Unmodified refused	$B$ refused	Unmodified mixed	$B$ mixed
InternVL3-2B	Explain	0/176	0/176	0/176	0/176
	Describe	0/176	0/176	0/176	0/176
	List	0/176	0/176	0/176	0/176
	Summarize	0/176	0/176	0/176	0/176
InternVL3-8B	Explain	0/176	0/176	0/176	0/176
	Describe	0/176	0/176	0/176	0/176

Appendix

Table 34: Raw refusal and mixed-response counts for each benign description template. Each model-template cell contains 176 unique images.

Model	Unmodified correct	$B$ correct	Both correct	Unmodified only	$B$ only	Both wrong	$\Delta$ accuracy [95% CI]
InternVL3-2B	293	290	286	7	4	353	-0.46 [-1.38, +0.46]
InternVL3-8B	351	353	350	1	3	296	+0.31 [-0.31, +0.92]
Qwen2.5-VL-7B	304	306	299	5	7	339	+0.31 [-0.77, +1.38]

Appendix

Table 35: Paired MMMU correctness transitions under the behavior-direction intervention. All comparisons use the same sample subset; accuracy differences are percentage points with paired-item bootstrap intervals.

Figure 5: Paired change in total refusal under the frozen behavior-derived direction across all response-policy conditions. Error bars are 95% image-clustered bootstrap confidence intervals.

Figure 6: Category-stratified changes in total refusal under the behavior-derived direction. Numerical values are reported in Table 28 .

Refusal Mechanisms in Language Models

See all 42 papers

Jul 28, 2026

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model follows visual context versus the language prior. To support the second, we introduce the WhatIfVis, a benchmark spanning five coarse-grained dimensions (spatial-temporal, color, count, size, and weight) whose questions admit answers from either the image or the prior. Our analysis yields three findings: (i) Coarse-grained visual evidence is preserved, as these attributes can be reconstructed from the final-layer image tokens of frozen MLLMs. Failures on questions about these attributes therefore point to post-perceptual utilization, rather than to degraded visual encoding during perception. (ii) Even when explicitly instructed to use or ignore visual evidence, vanilla models (without supervised fine-tuning on the WhatIfVis) show unstable visual context sensitivity. Supervised fine-tuning (SFT) improves this controllability and generalizes across domains, and activation patching further localizes the vision-versus-prior trade-off at architecture-specific depths across all six models. (iii) The vision-versus-prior trade-off is controllable along a learned vector. Applying this steering vector, even without any intent instruction, improves controllability over the vanilla model. Together, these results relocate the bottleneck, indicating that for the coarse attributes we study, MLLMs encode the visual evidence but cannot reliably control their reliance on it.

Jiaang Li, Chengzu Li, Zhaochong An +4

University of Copenhagen · University of Cambridge · 3ETH Zürich +1

VLM Evaluation Multimodal Large Language Models Multimodal Model Interpretability Multimodal Understanding Visual Evidence Sensitivity in VLMs

Ask PDF

Feb 2, 2026

There Is More to Refusal in Large Language Models than a Single Direction

Prior work argues that refusal in large language models is mediated by a single direction, enabling steering and abliteration. We show that this account is incomplete: across diverse refusal and non-compliance categories, refusal behaviors correspond to geometrically distinct directions in activation space. Yet activation steering along any refusal-related direction produces nearly identical refusal--over-refusal trade-offs, acting as a shared one-dimensional control knob. Thus, different directions primarily affect not whether the model refuses, but how it refuses. Using sparse autoencoders, we uncover a structured internal representation of refusal: a reusable core of shared refusal latents supplemented by style- and domain-specific latents. Linear interventions collapse this structure into uniform behavioral control, flattening mechanistic differences across refusal types. Our results reconcile the apparent simplicity of refusal steering with the diversity of refusal behaviors, and clarify the limits of linear interpretability for aligned model behavior.

Faaiz Joad, Majd Hawasly, Sabri Boughorbel +2

Qatar Computing Research Institute, HBKU, Doha, Qatar

LLM Interpretability LLM Refusal Behavior Causal Interventions in Language Models Refusal Behavior in Language Models Activation Steering Refusal Mechanisms in Language Models

Ask PDF

May 26, 2026

Beyond a Single Direction: Chain-of-Thought Disrupts Simple Steering of Refusal

Large reasoning models (LRMs) generate chain-of-thought (CoT) traces before producing final outputs, introducing a dynamic internal state that may complicate control mechanisms such as refusal. Unlike instruction-tuned LLMs, where refusal is mediated by a single directional subspace, refusal in large reasoning models (LRMs) additionally depends on the CoT. In DeepSeek-R1-Distill-LLaMA-8B, activation steering reverses refusal in only 39% of cases when the CoT is kept fixed, but removing the CoT entirely increases this to 70%, indicating that the CoT actively reinforces refusal. In a two-stage intervention where the model regenerates its CoT under activation steering, refusal is reversed in 94% of cases, while the resulting CoT alone retains 48% of this effect even after steering is removed. This suggests that the CoT can carry and reconstruct the compliance signal independently. These findings indicate that refusal in LRMs is jointly encoded in residual stream activations and CoT. This joint activation makes LRM more robust against activation-level interventions alone, but exposes CoT to a possible alternative surface attack.

Kia-Jüng Yang, Dominik Meier, Jiachen Zhao +2

University of Göttingen, Germany · LKA NRW, Germany · Northeastern University, Boston, MA, USA.

Reasoning Trace Analysis LLM Refusal Behavior Causal Interventions in Language Models Refusal Behavior in Language Models Activation Steering CoT Reasoning+2

Ask PDF

Abstract

Refusal Mechanisms in Language Models

Explore similar work

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

There Is More to Refusal in Large Language Models than a Single Direction

Beyond a Single Direction: Chain-of-Thought Disrupts Simple Steering of Refusal